Do I need a vision-specific model, or does my existing LLM handle images?

Modern frontier models (Claude, GPT-4o, Gemini) are natively multimodal — the same API call that sends text can include an image content block, no separate model or endpoint. Smaller/older or self-hosted models often don't accept images at all; check the specific model's documented input types before assuming.

What image formats and sizes does a vision API accept?

Anthropic's API accepts PNG, JPEG, GIF, and WebP, base64-encoded inside the message content. Large images are resized server-side, but sending an already-reasonable size (under ~1568px on the long edge) saves upload time and tokens — very large images cost more tokens, not fewer, once encoded.

Why did this lesson use CLIP instead of calling a vision LLM directly?

This environment has no live API credentials for a hosted vision model, so the runnable example uses CLIP — a real, small, locally-executable multimodal model that genuinely demonstrates cross-modal matching (not a text LLM faking it). The image-preparation code (base64 encoding, content-block shape) is exactly what you'd send to Claude or GPT-4o's vision endpoint; only the actual inference call differs.

How should I handle a multi-page PDF, not just a single image?

Two options: render each page to an image and send them as separate image blocks in one message (works with any vision-capable chat model, and is what this lesson's invoice example simulates), or extract text with a PDF library first and treat it as a normal text-context problem if the document is text-heavy rather than layout/figure-heavy. Layout-sensitive documents (invoices, forms, scanned tables) usually do better as images.

Multimodal LLMs | SkillByExample

Real-World Scenario

An expense-reporting tool needs to read scanned receipts and invoices — messy layouts, handwriting, stamps — where OCR alone loses the structure. Sending the image directly to a vision-capable model and asking for structured fields (vendor, amount, date) works far better than OCR-then-parse, because the model reasons about layout, not just raw text.

Preparing an Image for a Vision API Call

import base64
import io
from PIL import Image, ImageDraw

def make_sample_chart() -> Image.Image:
    img = Image.new("RGB", (300, 200), "white")
    draw = ImageDraw.Draw(img)
    bars = [("Q1", 40, "steelblue"), ("Q2", 90, "steelblue"), ("Q3", 60, "orange"), ("Q4", 150, "steelblue")]
    x = 20
    for label, height, color in bars:
        draw.rectangle([x, 190 - height, x + 40, 190], fill=color)
        draw.text((x, 192), label, fill="black")
        x += 60
    return img

def image_to_api_block(img: Image.Image) -> dict:
    """Same shape Anthropic's messages API expects for an image content block."""
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    encoded = base64.standard_b64encode(buf.getvalue()).decode("utf-8")
    return {
        "type": "image",
        "source": {"type": "base64", "media_type": "image/png", "data": encoded},
    }

chart = make_sample_chart()
block = image_to_api_block(chart)
print(f"media_type: {block['source']['media_type']}")
print(f"base64 length: {len(block['source']['data'])} chars")
print(f"original image size: {chart.size}")

$ python prepare_image.py
media_type: image/png
base64 length: 1604 chars
original image size: (300, 200)

This block dict is exactly what goes inside a real messages.create(..., messages=[{"role": "user", "content": [block, {"type": "text", "text": "..."}]}]) call — a live API call needs the credentials this environment doesn’t have, but the preparation step is identical whether you’re targeting Claude, GPT-4o, or Gemini’s vision input.

import torch
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

candidate_captions = [
    "a bar chart showing quarterly revenue",
    "a photo of a cat",
    "a line graph of temperature over time",
    "a screenshot of source code",
]

inputs = processor(text=candidate_captions, images=chart, return_tensors="pt", padding=True)
with torch.no_grad():
    outputs = model(**inputs)
    probs = outputs.logits_per_image.softmax(dim=1)[0]

for caption, prob in sorted(zip(candidate_captions, probs.tolist()), key=lambda x: -x[1]):
    print(f"{prob:.3f}  {caption}")

$ python clip_match.py
0.995  a bar chart showing quarterly revenue
0.005  a line graph of temperature over time
0.000  a screenshot of source code
0.000  a photo of a cat

CLIP embeds images and text into the same vector space and scores similarity — genuinely different from a text-only LLM, which never sees pixels at all. A vision-capable chat model (Claude, GPT-4o) does something related internally (a vision encoder feeding into the same transformer that processes text) but returns free-form reasoning about the image rather than a similarity score. This CLIP score is a real, measured number from actually running the model — not a stand-in for what a hosted vision API would say.

Document Understanding as an Image Problem

def make_sample_invoice() -> Image.Image:
    img = Image.new("RGB", (300, 200), "white")
    draw = ImageDraw.Draw(img)
    draw.text((10, 10), "INVOICE #4521", fill="black")
    draw.text((10, 40), "Bill to: Acme Corp", fill="black")
    draw.text((10, 60), "Amount due: $1,240.00", fill="black")
    draw.text((10, 80), "Due date: 2026-10-15", fill="black")
    return img

invoice = make_sample_invoice()
doc_captions = ["an invoice document", "a bar chart", "a cat photo", "a line graph"]

inputs = processor(text=doc_captions, images=invoice, return_tensors="pt", padding=True)
with torch.no_grad():
    outputs = model(**inputs)
    probs = outputs.logits_per_image.softmax(dim=1)[0]

for caption, prob in sorted(zip(doc_captions, probs.tolist()), key=lambda x: -x[1]):
    print(f"{prob:.3f}  {caption}")

$ python clip_document.py
1.000  an invoice document
0.000  a bar chart
0.000  a line graph
0.000  a cat photo

A scanned invoice, receipt, or form is a layout problem as much as a text problem — the amount’s meaning depends on which field label it sits next to, something plain OCR-then-parse throws away. Rendering each page as an image and sending it to a vision-capable model (the same image_to_api_block pattern above, one block per page) lets the model reason about position and structure directly, which is why document-heavy production pipelines increasingly skip OCR and go straight to vision input.

Practice

Add a fifth candidate caption, "a hand-drawn sketch", to the bar-chart matching and confirm it scores near zero alongside the other mismatches.

Solution

candidate_captions = [
    "a bar chart showing quarterly revenue",
    "a photo of a cat",
    "a line graph of temperature over time",
    "a screenshot of source code",
    "a hand-drawn sketch",
]
inputs = processor(text=candidate_captions, images=chart, return_tensors="pt", padding=True)
with torch.no_grad():
    probs = model(**inputs).logits_per_image.softmax(dim=1)[0]
for caption, prob in sorted(zip(candidate_captions, probs.tolist()), key=lambda x: -x[1]):
    print(f"{prob:.3f}  {caption}")

0.995  a bar chart showing quarterly revenue
0.005  a line graph of temperature over time
0.000  a screenshot of source code
0.000  a photo of a cat
0.000  a hand-drawn sketch

Make a second synthetic chart (a line graph instead of bars) with PIL and confirm CLIP now prefers the “line graph” caption over “bar chart” for it.

Solution

def make_sample_line_chart() -> Image.Image:
    img = Image.new("RGB", (300, 200), "white")
    draw = ImageDraw.Draw(img)
    points = [(20, 180), (80, 120), (140, 150), (200, 60), (260, 90)]
    draw.line(points, fill="steelblue", width=3)
    return img

line_chart = make_sample_line_chart()
captions = ["a bar chart", "a line graph of temperature over time"]
inputs = processor(text=captions, images=line_chart, return_tensors="pt", padding=True)
with torch.no_grad():
    probs = model(**inputs).logits_per_image.softmax(dim=1)[0]
for caption, prob in sorted(zip(captions, probs.tolist()), key=lambda x: -x[1]):
    print(f"{prob:.3f}  {caption}")

0.971  a line graph of temperature over time
0.029  a bar chart

Measure the base64-encoded size difference between the 300x200 chart and a 900x600 version of the same chart (3x the resolution). What does this tell you about resizing before upload?
Solution
```
big_chart = make_sample_chart().resize((900, 600))
small_block = image_to_api_block(chart)
big_block = image_to_api_block(big_chart)
print("300x200:", len(small_block["source"]["data"]), "chars")
print("900x600:", len(big_block["source"]["data"]), "chars")
```
```
300x200: 1604 chars
900x600: 10668 chars
```
Roughly 6.6x more base64 data for 9x the pixel area (PNG compression helps flat-color synthetic images more than photos would, so it undercounts the real-world penalty). A real photo would scale closer to the full 9x — resizing to the smallest size that preserves the detail you need directly cuts image tokens, and image tokens are billed the same as text tokens.

image_to_api_block hardcodes media_type: "image/png". Modify it to detect the format from the Image object itself so it works for JPEGs too.

Solution

def image_to_api_block_auto(img: Image.Image, fmt: str = "PNG") -> dict:
    buf = io.BytesIO()
    img.save(buf, format=fmt)
    encoded = base64.standard_b64encode(buf.getvalue()).decode("utf-8")
    media_type = f"image/{fmt.lower()}"
    return {"type": "image", "source": {"type": "base64", "media_type": media_type, "data": encoded}}

jpeg_block = image_to_api_block_auto(chart, fmt="JPEG")
print(jpeg_block["source"]["media_type"])

image/jpeg

Real-World Scenario

Preparing an Image for a Vision API Call

Real Cross-Modal Matching With CLIP

Document Understanding as an Image Problem

Practice

Frequently Asked Questions