LLM Context Engineering

Real-World Scenario

A router support bot has a system prompt, a manual, a five-turn conversation and a status API. The user’s latest message says the light is blinking white, and the manual says never to unplug the router while it does. The model gives the right answer only if both of those sentences reach it in the same request, within a token budget. Context engineering is the code that makes sure they do.

Every example here is plain Python with no model call: what you control is the text you send. Token counts use an estimate of about four characters per token, which is close enough to budget with. The tokenization lesson measures real counts with a real tokenizer.

The Data

A small manual, a conversation, and the user’s new question. Save this as data.py.

SYSTEM = ("You are the support assistant for the Vela X2 home router. Answer only from the "
          "documents provided. If they don't contain the answer, say so. Cite documents by id.")

DOCS = {
    "reset":    "To factory-reset the Vela X2, hold the recessed RESET button for 10 seconds "
                "until the power light blinks amber. All settings, including the Wi-Fi password, "
                "are erased. The router restarts in about two minutes.",
    "firmware": "Firmware updates install automatically between 02:00 and 04:00 local time. "
                "To update manually, open the admin page at 192.168.8.1, choose System, then "
                "Update. Do not unplug the router while the light blinks white.",
    "lights":   "Power light colours: solid white means normal operation, blinking white means "
                "a firmware update is installing, blinking amber means a reset is in progress, "
                "and solid red means the router cannot reach the internet.",
    "guest":    "The guest network is off by default. Turn it on under Wi-Fi, then Guest. Guest "
                "devices can reach the internet but not other devices on your network.",
}

HISTORY = [("user", "Hi, my internet stopped working this morning."),
           ("assistant", "Sorry to hear that. What colour is the power light on the router?"),
           ("user", "It was red, so I unplugged it and plugged it back in."),
           ("assistant", "Thanks. Did the light change after the restart?"),
           ("user", "Yes, now it's blinking white and has been for a few minutes.")]

QUESTION = "Is it safe to unplug it again to restart it?"

The answer depends on two pieces: the firmware document (“do not unplug the router while the light blinks white”) and the last history turn (“now it’s blinking white”).

The Failure: Truncating to Fit

The obvious approach is to join everything and cut it to the budget.

from data import SYSTEM, DOCS, HISTORY, QUESTION

def tokens(text):                        # rough estimate: about 4 characters per token
    return -(-len(text) // 4)

if __name__ == "__main__":
    prompt = "\n\n".join([SYSTEM, *DOCS.values(),
                          *(f"{role}: {text}" for role, text in HISTORY), f"user: {QUESTION}"])
    print(f"full prompt: {tokens(prompt)} tokens (estimated)")

    budget = 250
    cut = prompt[: budget * 4]               # "just truncate it to fit"
    print(f"truncated:   {tokens(cut)} tokens")
    print("question still there? ", QUESTION in cut)
    print("last 90 chars:", repr(cut[-90:]))

$ python naive.py
full prompt: 332 tokens (estimated)
truncated:   250 tokens
question still there?  False
last 90 chars: 'ernet but not other devices on your network.\n\nuser: Hi, my internet stopped working this m'

The cut landed in the first history turn, so the user’s question is gone, and so is the turn saying the light is blinking white. The model would receive a manual and the words “Hi, my internet stopped working this m” and have to guess what is being asked. Truncation removes whatever is last, and in a chat prompt the last thing is the most important. Save this file as naive.py, since later examples import tokens from it.

A Budgeted Context Builder

Treat the context as a list of items with priorities. The system prompt and the question are mandatory; the rest is added in priority order while it fits, and dropped whole when it doesn’t.

import re
from data import SYSTEM, DOCS, HISTORY, QUESTION
from naive import tokens

STOPWORDS = set("a an and the is it its to of for in on has been now yes s again few".split())

def words(text):
    return set(re.findall(r"[a-z]+", text.lower())) - STOPWORDS

def build(budget):
    must = [SYSTEM, f"user: {QUESTION}"]                     # never dropped
    left = budget - sum(tokens(t) for t in must)
    turns = [f"{r}: {t}" for r, t in HISTORY]
    query = words(QUESTION + " " + turns[-1])                # what the user is asking about now
    ranked = sorted(DOCS, key=lambda d: -len(words(DOCS[d]) & query))
    kept, report = [], []
    for kind, name, text in ([("history", f"turn {i}", t) for i, t in reversed(list(enumerate(turns)))]
                             + [("doc", d, DOCS[d]) for d in ranked]):
        fits = tokens(text) <= left
        left -= tokens(text) if fits else 0
        kept += [(kind, name, text)] if fits else []
        report.append(f"{'kept ' if fits else 'DROP '} {kind:7} {name:9} {tokens(text):3} tok")
    return must, kept, report, left

if __name__ == "__main__":
    must, kept, report, left = build(budget=250)
    print("\n".join(report))
    print(f"unused budget: {left} tokens")

$ python builder.py
kept  history turn 4     17 tok
kept  history turn 3     15 tok
kept  history turn 2     15 tok
kept  history turn 1     19 tok
kept  history turn 0     13 tok
kept  doc     firmware   54 tok
kept  doc     lights     54 tok
DROP  doc     reset      53 tok
DROP  doc     guest      37 tok
unused budget: 9 tokens

The question and system prompt are always in. History goes in newest first, so if anything is dropped it is the oldest turn. Documents are ranked by word overlap with the question plus the latest turn, which puts firmware and lights ahead of reset, and those two fit. Nothing is cut mid-sentence. Word overlap is the crudest possible ranker. The reranking lesson replaces it with a real one, but the budgeting logic stays the same.

Formatting Documents So They Can Be Cited

The model needs to tell documents, conversation and question apart. Tags with ids do that, and let you ask for citations.

from html import escape
from builder import build

def assemble(budget=250):
    must, kept, _, _ = build(budget)
    system, question = must
    docs = [(name, text) for kind, name, text in kept if kind == "doc"]
    turns = [text for kind, _, text in kept if kind == "history"][::-1]   # back to oldest-first
    doc_block = "\n".join(f'<doc id="{name}">{escape(text)}</doc>' for name, text in docs)
    user = (f"<documents>\n{doc_block}\n</documents>\n\n"
            f"<conversation>\n" + "\n".join(turns) + "\n</conversation>\n\n"
            f"{question}\nAnswer in two sentences and cite the doc ids you used.")
    return system, user

if __name__ == "__main__":
    system, user = assemble()
    print("SYSTEM:", system, "\n")
    print(user)
    from naive import tokens
    print(f"\n{tokens(system) + tokens(user)} tokens sent against a 250-token budget")

$ python assemble.py
SYSTEM: You are the support assistant for the Vela X2 home router. Answer only from the documents provided. If they don't contain the answer, say so. Cite documents by id. 

<documents>
<doc id="firmware">Firmware updates install automatically between 02:00 and 04:00 local time. To update manually, open the admin page at 192.168.8.1, choose System, then Update. Do not unplug the router while the light blinks white.</doc>
<doc id="lights">Power light colours: solid white means normal operation, blinking white means a firmware update is installing, blinking amber means a reset is in progress, and solid red means the router cannot reach the internet.</doc>
</documents>

<conversation>
user: Hi, my internet stopped working this morning.
assistant: Sorry to hear that. What colour is the power light on the router?
user: It was red, so I unplugged it and plugged it back in.
assistant: Thanks. Did the light change after the restart?
user: Yes, now it's blinking white and has been for a few minutes.
</conversation>

user: Is it safe to unplug it again to restart it?
Answer in two sentences and cite the doc ids you used.

280 tokens sent against a 250-token budget

Everything the answer needs is present: the firmware warning, the colour table, and the turn that says the light is blinking white. The html.escape call matters when documents come from users or the web: a document containing </doc> would otherwise end its own block early. With a real chat API you would send the conversation as separate messages rather than as text; the tags are the equivalent for documents.

The last line is a real bug. The builder budgeted 250 tokens of content, but tags, labels and the instruction line added 30 more, so the request is over budget. Formatting overhead is easy to forget because it isn’t “content”. Practice task 1 fixes it.

Putting the Best Evidence at the Edges

Research on long contexts (Liu et al., 2023, “Lost in the Middle”) found models use information at the start and end of the context more reliably than information in the middle. When you send several chunks, place them accordingly.

def edges(ranked):
    """Best chunk first, second-best last, weakest in the middle of the context."""
    front, back = [], []
    for i, chunk in enumerate(ranked):
        (front if i % 2 == 0 else back).append(chunk)
    return front + back[::-1]

ranked = ["firmware", "lights", "reset", "guest", "wifi-bands"]   # best match first
print("ranked:", ranked)
print("placed:", edges(ranked))

$ python edges.py
ranked: ['firmware', 'lights', 'reset', 'guest', 'wifi-bands']
placed: ['firmware', 'reset', 'wifi-bands', 'guest', 'lights']

The best chunk opens the document block and the second-best closes it. The weakest ones sit in the middle, where a lapse in attention costs least. This can’t be measured without a model, so treat it as a cheap default rather than a fix. The larger effect comes from sending fewer, better chunks, so that nothing important is in the middle of a long context at all.

Chunking: Where the Split Falls Decides What Can Be Found

Retrieval can only return whole chunks, so a fact split across two chunks may never be retrieved intact. Three chunkers run over the same manual:

import re
from data import DOCS

manual = "\n\n".join(f"## {name.title()}\n{text}" for name, text in DOCS.items())
sentences = [s for text in DOCS.values() for s in re.split(r"(?<=[.!?])\s+", text)]

def fixed(text, size=150):
    return [text[i:i + size] for i in range(0, len(text), size)]

def by_sentence(text, size=150):
    chunks, current = [], ""
    for sentence in re.split(r"(?<=[.!?])\s+", text):
        if current and len(current) + len(sentence) > size:
            chunks.append(current)
            current = ""
        current = f"{current} {sentence}".strip()
    return chunks + [current]

def by_heading(text):
    return [part.strip() for part in re.split(r"\n(?=## )", text)]

if __name__ == "__main__":
    for name, chunker in [("fixed 150", fixed), ("sentences", by_sentence), ("headings", by_heading)]:
        chunks = chunker(manual)
        broken = sum(not any(s in c for c in chunks) for s in sentences)
        mixed = sum("## " in c[1:] for c in chunks)            # crosses a section heading
        orphan = sum(not c.startswith("## ") for c in chunks)     # no heading to say what it is about
        print(f"{name:9}  {len(chunks)} chunks  split sentences: {broken}  "
              f"cross a heading: {mixed}  no heading: {orphan}")

    print(repr(fixed(manual)[1]))

$ python chunking.py
fixed 150  6 chunks  split sentences: 4  cross a heading: 3  no heading: 5
sentences  7 chunks  split sentences: 0  cross a heading: 0  no heading: 3
headings   4 chunks  split sentences: 0  cross a heading: 0  no heading: 0
'i-Fi password, are erased. The router restarts in about two minutes.\n\n## Firmware\nFirmware updates install automatically between 02:00 and 04:00 local'

Fixed 150-character chunks cut 4 sentences in half, and 3 of the 6 chunks straddle a section boundary. The printed chunk starts mid-word (“i-Fi password”) and ends in the firmware section: an embedding of that chunk is half about resets and half about updates. Sentence packing fixes the cuts, but 3 of its chunks start mid-section with no heading, so a chunk reading “To update manually…” doesn’t say what product or topic it belongs to. Splitting on headings gives 4 clean, self-describing chunks here. In a real manual, sections run to pages, so practice task 3 combines the two.

Trimming Tool Results Before They Enter the Context

Tool results are context too, and APIs return far more than the question needs.

import json
from naive import tokens

status = {   # what a router status API really returns: mostly irrelevant to the user's question
    "device": {"model": "Vela X2", "serial": "VX2-7A91-0042", "hw_rev": "B3", "uptime_s": 412},
    "firmware": {"running": "3.8.1", "installing": "3.9.0", "progress_pct": 64, "channel": "stable"},
    "led": {"power": "blinking_white", "wifi_2g": "on", "wifi_5g": "on", "wan": "off"},
    "wan": {"ip": None, "gateway": None, "dns": ["1.1.1.1", "8.8.8.8"], "mtu": 1500},
    "clients": [{"mac": f"a4:83:e7:1c:{i:02x}:9f", "rssi": -40 - i, "band": "5g"} for i in range(12)],
}

def trim(result):
    fw, led = result["firmware"], result["led"]
    return {"power_light": led["power"], "firmware_installing": fw["installing"],
            "progress_pct": fw["progress_pct"], "internet_connected": result["wan"]["ip"] is not None}

if __name__ == "__main__":
    raw, small = json.dumps(status), json.dumps(trim(status))
    print(f"raw result:     {tokens(raw)} tokens")
    print(f"trimmed result: {tokens(small)} tokens")
    print(small)

$ python tool_result.py
raw result:     264 tokens
trimmed result: 29 tokens
{"power_light": "blinking_white", "firmware_installing": "3.9.0", "progress_pct": 64, "internet_connected": false}

The raw status response alone is 264 estimated tokens, more than this request’s whole 250-token budget, and most of it is MAC addresses and DNS servers. The trimmed version is 29 tokens and contains exactly the facts the question turns on: the firmware update is 64% through, and the router isn’t online yet. It also renames fields in plain language and turns "ip": null into "internet_connected": false, so the model doesn’t have to interpret a null.

Practice

1. Make assemble stay within its budget. Measure the formatting overhead and rebuild with a smaller content budget until the full request fits.

   Solution

   import re
   from assemble import assemble
   from naive import tokens
   
   def assemble_within(budget):
       system, user = assemble(budget)
       overhead = tokens(system) + tokens(user) - budget     # tags, labels, instructions
       while overhead > 0:
           system, user = assemble(budget - overhead)
           overhead = tokens(system) + tokens(user) - budget
       return system, user
   
   system, user = assemble_within(250)
   print(tokens(system) + tokens(user), "tokens sent")
   print("docs kept:", re.findall(r"<doc id=\"(\w+)\"", user))

   $ python sol_overhead.py
   221 tokens sent
   docs kept: ['firmware']

   The request now fits, at 221 tokens. It fits with room to spare because items are dropped whole: shaving the 30 tokens of overhead meant losing the whole lights document. That is the real trade-off with a tight budget. Either raise the budget, or shorten the documents themselves (trimming, as the tool result was) instead of dropping them.

2. Run build(180). With history added first, the five turns use most of the budget, and the only document that still fits is the irrelevant guest one. Add documents first and compare.

   Solution

   Swap the order of the two lists in the for statement, so documents come before history:

   for kind, name, text in ([("doc", d, DOCS[d]) for d in ranked]
                            + [("history", f"turn {i}", t) for i, t in reversed(list(enumerate(turns)))]):

   $ python sol_docs_first.py
   history first, budget 180:
   kept  history turn 4     17 tok
   kept  history turn 3     15 tok
   kept  history turn 2     15 tok
   kept  history turn 1     19 tok
   kept  history turn 0     13 tok
   DROP  doc     firmware   54 tok
   DROP  doc     lights     54 tok
   DROP  doc     reset      53 tok
   kept  doc     guest      37 tok
   docs first, budget 180:
   kept  doc     firmware   54 tok
   kept  doc     lights     54 tok
   DROP  doc     reset      53 tok
   DROP  doc     guest      37 tok
   kept  history turn 4     17 tok
   DROP  history turn 3     15 tok
   DROP  history turn 2     15 tok
   DROP  history turn 1     19 tok
   DROP  history turn 0     13 tok

   Documents first keeps firmware and lights, plus the newest turn, which is the one that says the light is blinking white. That is enough to answer. The history-first run shows a second bug in greedy filling: after the big relevant documents failed to fit, the small irrelevant guest doc slipped into the gap. A real builder should skip documents whose relevance score is 0, whatever their size. Neither fixed order is right for every request. Budgets per section (for example half for documents, a quarter for history) are the usual compromise, and the long-context lesson shows how to summarize old turns instead of dropping them.

3. Heading chunks can be pages long in a real manual. Write a chunker that splits on headings first, then packs sentences within each section, prefixing every piece with its section heading.

   Solution

   from chunking import manual, by_heading, by_sentence
   
   def by_heading_then_sentence(text, size=150):
       chunks = []
       for section in by_heading(text):
           heading, body = section.split("\n", 1)
           chunks += [f"{heading}\n{piece}" for piece in by_sentence(body, size)]
       return chunks
   
   chunks = by_heading_then_sentence(manual, size=120)
   print(len(chunks), "chunks, all with a heading:", all(c.startswith("## ") for c in chunks))
   for c in chunks[2:4]:
       print(repr(c))

   $ python sol_hybrid.py
   8 chunks, all with a heading: True
   '## Firmware\nFirmware updates install automatically between 02:00 and 04:00 local time.'
   '## Firmware\nTo update manually, open the admin page at 192.168.8.1, choose System, then Update.'

   Every chunk now carries its heading, so “To update manually…” is unambiguously about firmware. No chunk crosses a section, and no sentence is cut. Repeating the heading costs a few tokens per chunk, which is cheap compared with retrieving a chunk nobody can place.

4. You won’t write a trim() for every tool. Add a generic guard that caps any tool result at 60 estimated tokens and tells the model what was cut.

   Solution

   import json
   from naive import tokens
   from tool_result import status
   
   def cap(result, max_tokens=60):
       """Generic guard for tools you haven't written a trim() for."""
       text = json.dumps(result)
       if tokens(text) <= max_tokens:
           return text
       kept = text[: max_tokens * 4]
       return f"{kept}...[truncated {tokens(text) - tokens(kept)} tokens; ask for a specific field]"
   
   print(cap({"power_light": "blinking_white"}))
   print(cap(status))

   $ python sol_cap.py
   {"power_light": "blinking_white"}
   {"device": {"model": "Vela X2", "serial": "VX2-7A91-0042", "hw_rev": "B3", "uptime_s": 412}, "firmware": {"running": "3.8.1", "installing": "3.9.0", "progress_pct": 64, "channel": "stable"}, "led": {"power": "blinking_white", "wifi_2g": "on...[truncated 204 tokens; ask for a specific field]

   Small results pass through untouched. The status dump is cut, and the marker tells the model it saw a partial result and can ask for a specific field. That beats silently cutting, which makes the model treat a fragment as the whole answer. Here the cut happened to keep the firmware progress but lost wan, which is why a hand-written trim() is still worth it for the tools you call most.