AI Agent Memory Systems

Real-World Scenario

A travel assistant is told early in a chat that the user is allergic to peanuts. Ten messages later the user asks it to pick a seafood restaurant, and it suggests one famous for satay, because the allergy is no longer in the context it receives. The next week the user comes back and the assistant doesn’t know their name. Neither failure is the model’s fault. The model only ever sees the text sent with each request, and deciding what goes in that text is the memory system.

Every example below uses reply, a stand-in for an LLM that answers only from lines present in its context. That constraint is close to how a real model behaves with memory — a fact missing from the prompt is gone, though a real model will often guess or hallucinate instead of cleanly saying it doesn’t know, which the stand-in always does. No API key is needed.

A Model That Only Knows Its Context

Save the stand-in as model.py.

import re

STOP = set("a an the is my i me what which do you to of in and am are it for was".split())

def words(text):
    return set(re.findall(r"[a-z]+", text.lower())) - STOP

def reply(context, question):
    """Stands in for an LLM: it can only use facts that are present in its context."""
    best, score = None, 0
    for line in context:
        overlap = len(words(line) & words(question))
        if overlap > score:
            best, score = line, overlap
    return f"(from context) {best}" if best else "I don't know that."

if __name__ == "__main__":
    history = ["user: My name is Priya.", "user: I live in Pune.", "user: I am allergic to peanuts."]
    print(reply(history, "What is my name?"))
    print(reply([], "What is my name?"))

$ python model.py
(from context) user: My name is Priya.
I don't know that.

The first call has the fact in its context and answers. The second has an empty context and can’t, however many times the user said it before. Every memory technique in this lesson is a way of getting the right lines into that first argument.

Short-Term Memory: The Sliding Window Forgets

The conversation, saved as conversation.py:

TURNS = [
    "My name is Priya.",
    "I am allergic to peanuts.",
    "I live in Pune.",
    "I'm planning a weekend trip to Goa.",
    "Trains or flights, which is quicker?",
    "Let's book the Friday evening train.",
    "Find a hotel near the beach.",
    "Something under 4000 rupees a night.",
    "Add a dinner reservation on Saturday.",
    "Pick a seafood place with good reviews.",
]

The simplest short-term memory sends only the newest turns, because the context has a budget.

from conversation import TURNS
from model import reply

def window(turns, keep=4):
    return [f"user: {t}" for t in turns[-keep:]]      # only the newest turns fit

if __name__ == "__main__":
    context = window(TURNS)
    print("context sent:", *context, sep="\n  ")
    for q in ["Which foods am I allergic to?", "What is my name?", "How much per night can I spend?"]:
        print(f"{q:32} -> {reply(context, q)}")

$ python window.py
context sent:
  user: Find a hotel near the beach.
  user: Something under 4000 rupees a night.
  user: Add a dinner reservation on Saturday.
  user: Pick a seafood place with good reviews.
Which foods am I allergic to?    -> I don't know that.
What is my name?                 -> I don't know that.
How much per night can I spend?  -> (from context) user: Something under 4000 rupees a night.

The budget question works, because that turn is recent. The name and the allergy were said in turns 1 and 2, fell out of the four-turn window, and are gone, just as the user asks for a seafood place. Nothing errors and nothing warns; the model simply never sees the allergy. Fixed windows fail silently, and they fail on exactly the facts people state once at the start.

Summarized Memory: Keep the Facts, Drop the Chatter

Before turns leave the window, extract what should outlive them.

import re
from conversation import TURNS
from model import reply
from window import window

FACT_PATTERNS = [r"my name is \w+", r"i am allergic to \w+", r"i live in \w+"]

def summarize(dropped):
    """Stands in for an LLM summarizer: keep durable facts, drop the chatter."""
    facts = []
    for turn in dropped:
        for pattern in FACT_PATTERNS:
            facts += re.findall(pattern, turn.lower())
    return [f"remembered: {fact}" for fact in facts]

summary = summarize(TURNS[:-4])                       # everything the window dropped
context = summary + window(TURNS)
print(*summary, sep="\n")
for q in ["Which foods am I allergic to?", "What is my name?", "How much per night can I spend?"]:
    print(f"{q:32} -> {reply(context, q)}")

$ python summary.py
remembered: my name is priya
remembered: i am allergic to peanuts
remembered: i live in pune
Which foods am I allergic to?    -> (from context) remembered: i am allergic to peanuts
What is my name?                 -> (from context) remembered: my name is priya
How much per night can I spend?  -> (from context) user: Something under 4000 rupees a night.

All three questions now work. The window still carries recent detail, and three short facts carry what the dropped turns established. In production the summarizer is itself an LLM call, prompted to extract names, preferences, constraints and decisions. The regex here plays that part so the result is predictable. The shape is the same either way: facts go in, chatter is dropped, and the facts ride along on every later request.

Summaries lose detail by design, so decide what counts as durable. Identity, safety constraints, and decisions the user has confirmed should survive. Small talk shouldn’t.

Long-Term Memory: Persisting Across Sessions

Summaries die with the session. Facts the agent should know next week go to a store. Here that is SQLite, and each command below is a separate process, like separate sessions.

import sqlite3, sys
from datetime import datetime

db = sqlite3.connect("memory.db")
db.execute("""CREATE TABLE IF NOT EXISTS memory (user TEXT, key TEXT, value TEXT,
              saved_at TEXT, active INTEGER)""")

def remember(user, key, value):
    """Latest value wins; the old one is kept as history, not deleted."""
    db.execute("UPDATE memory SET active = 0 WHERE user = ? AND key = ? AND active = 1", (user, key))
    db.execute("INSERT INTO memory VALUES (?, ?, ?, ?, 1)",
               (user, key, value, datetime.now().isoformat(timespec="seconds")))
    db.commit()

def recall(user):
    return db.execute("SELECT key, value FROM memory WHERE user = ? AND active = 1 ORDER BY key",
                      (user,)).fetchall()

def history(user, key):
    return db.execute("SELECT value, saved_at, active FROM memory WHERE user = ? AND key = ?"
                      " ORDER BY saved_at, rowid", (user, key)).fetchall()

if __name__ == "__main__":
    command, user, *rest = sys.argv[1:]
    if command == "remember":
        remember(user, *rest)
    else:
        print(recall(user) if command == "recall" else history(user, rest[0]))

$ rm -f memory.db
$ python store.py remember priya city Pune
$ python store.py remember priya allergy peanuts
$ python store.py recall priya
[('allergy', 'peanuts'), ('city', 'Pune')]
$ python store.py remember priya city Bengaluru
$ python store.py recall priya
[('allergy', 'peanuts'), ('city', 'Bengaluru')]
$ python store.py history priya city
[('Pune', '2026-09-27T18:32:42', 0), ('Bengaluru', '2026-09-27T18:32:42', 1)]

Each recall runs in a fresh process and still finds what earlier processes saved: that is long-term memory. When the user moves, remember marks Pune inactive and inserts Bengaluru, so recall returns one current city, and history still shows the change and when it happened. Practice task 2 shows what an append-only store does instead.

In an agent, remember and recall are exposed as tools. The model calls remember when the user states a durable fact, and your code calls recall at the start of each session to put the current facts into the context.

Retrieving Relevant Memories, and Pinning Critical Ones

A long-lived user accumulates hundreds of memories, too many to send every time. Retrieve by relevance, but not only by relevance.

from model import words

MEMORIES = [   # (text, pinned) - pinned facts must reach the model on every request
    ("Name is Priya", True),
    ("Allergic to peanuts", True),
    ("Prefers window seats on trains", False),
    ("Lives in Bengaluru, moved from Pune", False),
    ("Likes seafood restaurants near the beach", False),
    ("Works night shifts on weekdays", False),
    ("Hotel budget is 4000 rupees a night", False),
    ("Has a dog called Biscuit", False),
    ("Prefers trains over flights for short trips", False),
    ("Birthday is 14 March", False),
]

def top_k(query, k=3, pin=False):
    scored = sorted(MEMORIES, key=lambda m: -len(words(m[0]) & words(query)))
    chosen = [m[0] for m in scored[:k] if words(m[0]) & words(query)]
    pinned = [m[0] for m in MEMORIES if m[1] and m[0] not in chosen] if pin else []
    return pinned + chosen

query = "Book a seafood restaurant for dinner near the hotel"
print("top 3 by relevance: ", top_k(query))
print("with pinned facts:  ", top_k(query, pin=True))

$ python relevant.py
top 3 by relevance:  ['Likes seafood restaurants near the beach', 'Hotel budget is 4000 rupees a night']
with pinned facts:   ['Name is Priya', 'Allergic to peanuts', 'Likes seafood restaurants near the beach', 'Hotel budget is 4000 rupees a night']

Relevance scoring picked the seafood preference and the hotel budget, which share words with the request, and returned only two because memories with no shared words are dropped rather than padded in. It did not pick the peanut allergy, the one memory that matters most for this booking, because “peanuts” shares no word with “seafood restaurant”. Embedding-based retrieval narrows this gap (see the embeddings lesson) but can’t guarantee it. So pin facts whose absence would cause harm, and send them on every request regardless of the query. Pinning costs a few tokens; missing an allergy costs far more.

Practice

1. window.py keeps a fixed number of turns, but turns vary in length. Keep as many of the newest turns as fit a token budget (about 4 characters per token), and compare budgets of 40 and 60 tokens.

   Solution

   from conversation import TURNS
   
   def tokens(text):
       return -(-len(text) // 4)
   
   def window_by_budget(turns, budget=40):
       kept, used = [], 0
       for turn in reversed(turns):                # newest first
           if used + tokens(turn) > budget:
               break
           kept.insert(0, turn)
           used += tokens(turn)
       return kept, used
   
   for budget in (40, 60):
       kept, used = window_by_budget(TURNS, budget)
       print(f"budget {budget}: kept {len(kept)} turns ({used} tokens), oldest kept: {kept[0]!r}")

   $ python sol_budget.py
   budget 40: kept 4 turns (36 tokens), oldest kept: 'Find a hotel near the beach.'
   budget 60: kept 6 turns (54 tokens), oldest kept: 'Trains or flights, which is quicker?'

   A 40-token budget happens to keep the same four turns as before, and 60 tokens keeps six. Neither reaches back to the name and the allergy, which is why a budget window still needs the fact summary. The budget also has to be shared with the system prompt and the pinned facts, so size the window last.

2. Replace the UPDATE ... active = 0 step with a plain insert, as many first versions do, then recall the user’s city after they move.

   Solution

   import sqlite3
   
   db = sqlite3.connect(":memory:")
   db.execute("CREATE TABLE memory (user TEXT, key TEXT, value TEXT)")
   for value in ["Pune", "Bengaluru"]:                     # the user moved; naive store just appends
       db.execute("INSERT INTO memory VALUES ('priya', 'city', ?)", (value,))
   
   rows = db.execute("SELECT value FROM memory WHERE user = 'priya' AND key = 'city'").fetchall()
   print("recalled:", rows)
   print("what the agent would say:", f"You live in {rows[0][0]}.")

   $ python sol_append.py
   recalled: [('Pune',), ('Bengaluru',)]
   what the agent would say: You live in Pune.

   Both cities come back with nothing saying which is current, and the agent quotes the first row: the old address. Worse, the order of rows without an ORDER BY isn’t guaranteed, so the answer can change between runs. Keyed memories with one active value per key avoid the ambiguity.

3. Add forget(user, key) to store.py’s functions for a user who asks for their phone number to be removed. Check that it is gone from both recall and history.

   Solution

   from store import db, remember, recall, history
   
   def forget(user, key):
       """A deletion request means delete, history included; deactivating is not enough."""
       n = db.execute("DELETE FROM memory WHERE user = ? AND key = ?", (user, key)).rowcount
       db.commit()
       return n
   
   remember("priya", "phone", "98450 12345")
   print("before:", recall("priya"))
   print("rows deleted:", forget("priya", "phone"))
   print("after: ", recall("priya"), " history:", history("priya", "phone"))

   $ python sol_forget.py
   before: [('allergy', 'peanuts'), ('city', 'Bengaluru'), ('phone', '98450 12345')]
   rows deleted: 1
   after:  [('allergy', 'peanuts'), ('city', 'Bengaluru')]  history: []

   The phone number is gone from current memory and from history. Deactivating would have hidden it from the agent but kept it in the database, which doesn’t satisfy a deletion request. The other memories are untouched. In a real system, also delete the fact from summaries, logs and any vector index that holds a copy.