Send an image to a model and get text back — a description, an answer, or structured data extracted from it. Encode the image, ask a precise question, and pull out fields.
Why: a vision model takes an image plus text in the same message and answers about the picture — the image is just another content block alongside your prompt. When: use it to describe, classify, answer questions about, or extract data from images. Where: images are sent as base64 (inline) or by URL, and they count as input tokens based on their size.
MESSAGE = [ image block ] + [ text block: "What is in this photo?" ]
| |
the picture your question about it
|
MODEL -> text answer
Bigger images = more input tokens. Resize before sending when you can.Why: you read the file, base64-encode it, and put it in the message as an image block with its media type — then ask anything in a text block. When: inline base64 is simplest for local files; use a URL source for images already on the web. Where: the media_type must match the real format (image/jpeg, image/png) or the call fails.
import base64, anthropic
client = anthropic.Anthropic()
with open("receipt.jpg", "rb") as f:
data = base64.standard_b64encode(f.read()).decode()
resp = client.messages.create(
model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user", "content": [
{"type": "image", "source": {
"type": "base64", "media_type": "image/jpeg", "data": data}},
{"type": "text", "text": "What store is this receipt from, and what's the total?"},
]}],
)
print(resp.content[0].text)Why: the real power is turning a photo of a receipt, form, or label into JSON your code can use — combine vision with the structured-output trick (a forced tool call). When: any time an image feeds a database or workflow, demand a schema, not a paragraph. Where: validate the fields afterward — vision can misread digits, so treat the output as untrusted.
tool = {
"name": "record_receipt",
"description": "Record the fields extracted from a receipt image.",
"input_schema": {
"type": "object",
"properties": {
"store": {"type": "string"},
"total": {"type": "number"},
"date": {"type": "string", "description": "ISO 8601"},
},
"required": ["store", "total", "date"],
},
}
resp = client.messages.create(
model="claude-opus-4-8", max_tokens=512,
tools=[tool], tool_choice={"type": "tool", "name": "record_receipt"},
messages=[{"role": "user", "content": [
{"type": "image", "source": {
"type": "base64", "media_type": "image/jpeg", "data": data}},
{"type": "text", "text": "Extract the receipt fields."},
]}],
)
print(resp.content[0].input) # {'store': ..., 'total': ..., 'date': ...}