Doesn't context-engineering.md already cover long documents and conversation history?

That lesson covers the mechanics — how to structure and trim what goes into context. This one covers the economics: what growing context actually costs in latency and dollars, measured rather than assumed, plus prompt caching, which context-engineering.md doesn't touch.

If a model supports a 200K-token context window, why not just put everything in it?

Because every token in the context is re-processed on every single call — this lesson's own measurement shows generation time growing well past linearly as context grows, and intro.md's cost math bills every input token whether or not the model actually needed it to answer. A large context window is a ceiling for what's possible, not a target for what you should send.

What is prompt caching, mechanically?

The provider caches the internal representation of a prefix of your prompt (a system prompt, a large document, a tool schema) so a later call sharing that same prefix skips reprocessing it — you pay a small premium on the call that creates the cache, then a steep discount on every call that hits it within the cache's TTL (typically a few minutes). It only helps when the SAME prefix repeats across calls; caching a prompt you'll never send again just costs more than not caching.

This lesson's Anthropic prompt-caching mention isn't backed by an executed example — why not?

This environment has no ANTHROPIC_API_KEY, so the actual cache-hit/cache-miss billing behavior can't be measured here. The mechanism (cache_control breakpoints, a short TTL, cheaper cached reads) is accurately described from the real API's documented shape, not fabricated output — but treat the exact discount percentage as something to verify against current pricing before relying on it, not as a number this lesson measured.

Long-Context Economics and Prompt Caching

Real-World Scenario

A support bot re-sends the full conversation history on every turn so the model has complete context. By turn 50, each call resends 49 previous turns just to add one new message — the same information paid for, transmitted, and processed again and again. This lesson measures what that actually costs, not just states that it’s expensive.

Token Cost Scales With Every Character You Send

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
INPUT_COST_PER_MILLION = 3.00  # matches intro.md's Claude Sonnet 4.6 example rate

def cost_for(text: str) -> tuple[int, float]:
    n_tokens = len(enc.encode(text))
    return n_tokens, n_tokens * INPUT_COST_PER_MILLION / 1_000_000

filler = "The quarterly report shows strong growth across all business segments. "
for repeats in [10, 50, 200, 1000]:
    n_tokens, cost = cost_for(filler * repeats)
    print(f"repeat={repeats:5d}  tokens={n_tokens:7d}  input_cost=${cost:.5f}")

$ python token_cost.py
repeat=   10  tokens=    111  input_cost=$0.00033
repeat=   50  tokens=    551  input_cost=$0.00165
repeat=  200  tokens=   2201  input_cost=$0.00660
repeat= 1000  tokens=  11001  input_cost=$0.03300

Linear and small per call — the real cost problem isn’t one call, it’s paying for the same growing context on every call in a conversation.

Measured: Latency Actually Grows Faster Than the Token Count

import time, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
filler = "The company reported steady performance this quarter. "

def timed_generate(context_repeats: int, new_tokens: int = 10) -> tuple[int, float]:
    context = filler * context_repeats
    prompt = context + "\n\nQuestion: What is 2+2? Answer in one word."
    messages = [{"role": "user", "content": prompt}]
    chat_input = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tok(chat_input, return_tensors="pt")
    n_input_tokens = inputs["input_ids"].shape[1]
    torch.manual_seed(0)
    t0 = time.time()
    model.generate(**inputs, max_new_tokens=new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    return n_input_tokens, time.time() - t0

timed_generate(1)  # warmup, excluded from measurements below

for repeats in [1, 20, 60, 150]:
    n_tok, secs = timed_generate(repeats)
    print(f"context_tokens={n_tok:5d}  generate_time={secs:.3f}s")

$ python latency_scaling.py
context_tokens=   52  generate_time=1.694s
context_tokens=  204  generate_time=10.111s
context_tokens=  524  generate_time=25.929s
context_tokens= 1244  generate_time=60.239s

Roughly 4x the context (52 -> 204 tokens) produced roughly 6x the latency, not 4x — attention cost grows worse than linearly with sequence length, and this is on top of the token cost from the previous example, not instead of it. (Hosted APIs use optimizations — KV caching, continuous batching, covered in llm-inference-serving.md — that flatten this curve versus a naive unoptimized loop like this one, but the underlying shape, cost growing faster than the token count, is the same reason those optimizations exist.)

Resending Full History vs. Trimming It

turn_tokens = 40  # rough tokens per conversational turn
n_turns = 200

# Strategy A: resend the entire history on every single call
full_history_tokens = sum((t + 1) * turn_tokens for t in range(n_turns))
full_history_cost = full_history_tokens * INPUT_COST_PER_MILLION / 1_000_000

# Strategy B: keep only the last 10 turns + a running summary (~200 tokens)
# once the window fills, matching context-engineering.md's ContextManager pattern
trimmed_tokens = sum(min(t + 1, 10) * turn_tokens + (200 if t >= 10 else 0) for t in range(n_turns))
trimmed_cost = trimmed_tokens * INPUT_COST_PER_MILLION / 1_000_000

print(f"Full history across {n_turns} turns: {full_history_tokens:,} tokens, ${full_history_cost:.2f}")
print(f"Trimmed (last 10 + summary):        {trimmed_tokens:,} tokens, ${trimmed_cost:.2f}")
print(f"Savings: {(1 - trimmed_cost / full_history_cost) * 100:.1f}%")

$ python history_strategy_cost.py
Full history across 200 turns: 804,000 tokens, $2.41
Trimmed (last 10 + summary):        116,200 tokens, $0.35
Savings: 85.5%

This is one conversation. At any real support-bot volume, the difference between these two strategies is the difference between a manageable bill and one that scales linearly with how chatty your users get — for a cost identical in every other respect.

Summarization Actually Runs and Actually Shrinks

conversation = """Turn 1 user: I want to set up a Postgres database for my app.
Turn 1 assistant: Sure, what is the expected scale -- rows and QPS?
Turn 2 user: About 2 million rows, maybe 50 queries per second.
Turn 2 assistant: A single db.t3.medium instance should handle that comfortably with proper indexing.
Turn 3 user: I also need full text search on a description field.
Turn 3 assistant: Use a GIN index with to_tsvector for that column.
Turn 4 user: What about backups?
Turn 4 assistant: Enable automated daily snapshots with a 7-day retention window."""

def ask(prompt: str, max_tokens: int = 80) -> str:
    messages = [{"role": "user", "content": prompt}]
    chat_input = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tok(chat_input, return_tensors="pt")
    torch.manual_seed(0)
    out = model.generate(**inputs, max_new_tokens=max_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

n_original = len(enc.encode(conversation))
summary = ask(f"Summarize this conversation in one or two sentences, preserving key decisions:\n\n{conversation}")
n_summary = len(enc.encode(summary))

print("Original tokens:", n_original)
print("Summary:", summary)
print("Summary tokens:", n_summary)
print(f"Reduction: {(1 - n_summary / n_original) * 100:.0f}%")

$ python summarize_history.py
Original tokens: 134
Summary: The conversation revolves around setting up a PostgreSQL database for an application with a high volume of data (about 2 million rows) and frequent query traffic (QPS). The user requested both full-text search capabilities and automatic backups. The assistant recommended using a T3 medium instance with appropriate indexing and suggested using
Summary tokens: 60
Reduction: 55%

A real 55% reduction — smaller than it could be because max_new_tokens=80 cut the summary off mid-sentence rather than the model choosing to stop; the mechanism context-engineering.md’s ContextManager._trim_if_needed uses is exactly this pattern, applied automatically once history crosses a token threshold.

Prompt Caching: The Mechanism (Reference, Not Measured Here)

# NOT executed -- this environment has no ANTHROPIC_API_KEY. Shown to document
# the real cache_control shape from Anthropic's Messages API, which this lesson
# cannot bill/measure without live credentials.
import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM_PROMPT_OR_DOCUMENT,  # e.g. a 10,000-token policy manual
            "cache_control": {"type": "ephemeral"},   # cache this prefix
        }
    ],
    messages=[{"role": "user", "content": "What's the escalation policy for a P1 incident?"}],
)
# The first call pays a premium to write the cache; subsequent calls within the
# cache's short TTL that share this exact prefix pay a steep discount on it,
# reprocessing only the new user message.

The trade a cache is making: pay slightly more once to avoid reprocessing an unchanging prefix (a system prompt, a large document, a tool schema) on every subsequent call that reuses it. It only pays off when the same prefix actually repeats — caching a one-off prompt costs more than not caching, since you paid the write premium for a read that never happens. Verify the current write/read pricing ratio and TTL against Anthropic’s documentation before sizing a cost model on it; this lesson describes the mechanism accurately but can’t measure the real discount without credentials.

Practice

Recompute the full-history-vs-trimmed comparison for a 500-turn conversation instead of 200, and see whether the percentage savings changes.

Solution

n_turns = 500
full_history_tokens = sum((t + 1) * turn_tokens for t in range(n_turns))
full_history_cost = full_history_tokens * INPUT_COST_PER_MILLION / 1_000_000
trimmed_tokens = sum(min(t + 1, 10) * turn_tokens + (200 if t >= 10 else 0) for t in range(n_turns))
trimmed_cost = trimmed_tokens * INPUT_COST_PER_MILLION / 1_000_000
print(f"Full: ${full_history_cost:.2f}  Trimmed: ${trimmed_cost:.2f}  Savings: {(1 - trimmed_cost/full_history_cost)*100:.1f}%")

Full: $15.03  Trimmed: $0.89  Savings: 94.1%

The savings percentage increases with conversation length (85.5% at 200 turns -> 94.1% at 500) — full-history cost grows quadratically (turn N resends N previous turns), while trimmed cost grows linearly (always resending at most 10 turns + one summary). The longer the conversation, the worse the naive strategy gets relative to trimming, not just proportionally worse.

Measure timed_generate at context_repeats=300 and confirm the trend from the main example continues (check whether the ratio of time-growth to token-growth gets worse, better, or stays about the same).
Solution
```
n_tok, secs = timed_generate(300)
print(f"context_tokens={n_tok}  generate_time={secs:.3f}s")
```
```
context_tokens=2444  generate_time=110.479s
```
From 1244 tokens/60.2s to 2444 tokens/110.5s: roughly 2.0x the tokens for roughly 1.8x the time — at this range the growth has flattened out closer to linear (even slightly better) rather than continuing to worsen, unlike the earlier 52->204 jump (4x tokens, 6x time). The steepest relative penalty is often at the low end, not the high end, because fixed per-call overhead matters more when the variable part is still small.

Change the summarization prompt’s max_tokens from 80 to 150 and see whether the model naturally stops before the limit this time (a real stop token) rather than being cut off.