Work with more than text — image understanding (vision), image generation, speech-to-text, and text-to-speech. Then handle the parts unique to binary inputs: validation, encoding, timeouts, cost, caching, and moderation. Every lesson is runnable Python.
Before you start
You will call real multimodal APIs from Python. Vision uses the Anthropic API; image generation, transcription, and speech use OpenAI because Anthropic does not offer those modalities. Have both keys ready — the patterns port to any provider that offers each modality.
Image Understanding (Vision)
Send an image to a model and get text back — a description, an answer, or structured data extracted from it. Encode the image, ask a precise question, and pull out fields.
Image Generation
Turn a text prompt into an image. Call an image model, save the bytes, and write prompts that actually control the result — subject, style, and composition.
Speech-to-Text (Transcription)
Turn audio into text you can search, summarize, or act on. Transcribe a file, then feed the transcript straight into a model to build a voice-driven feature.
Text-to-Speech
Give your app a voice. Turn text into natural-sounding audio, pick a voice and format, and stream it so playback starts before the whole clip is ready.
Input Validation & Handling Files
Multimodal inputs are big, binary, and user-supplied — validate before you spend a single token. Check type and size, encode correctly, and set the right timeouts.
Multimodal in Production
Multimodal calls cost 10–100× a text call and take seconds. Deduplicate inputs, cache deterministic outputs, run heavy work async, and moderate what you generate.