Course: Course 2B — Securing & Attacking Harnesses and LLMs
Module: B3 — Memory and Context Poisoning
Duration: ~25 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to B3. Memory and Context Poisoning. B2 built five layers of defense against prompt injection, and every one of those layers operates inside a session — on content that crosses into the context window this turn. This module attacks the assumption that makes all five work: that the attacker and the defense are alive at the same time, in the same session, fighting over the same token stream.
Memory breaks that assumption. Long-term memory persists across sessions. An attacker who can write to it plants a payload that the five in-session defenses never see, because the payload is not delivered through the context window — it is loaded from memory at session start, before the user types a single character. You inject today; the payload fires next week, after the user has forgotten the conversation that delivered it, under a different account state. Memory is the only attack surface that survives logout.
[SLIDE 2 — The two memory surfaces]
The foundational mental model. An agent has two memory surfaces, and the distinction is load-bearing. Long-term memory persists across sessions. It is the agent's notebook — facts, preferences, summaries, ingested documents. It is stored outside the model, in a database, a vector store, a file, and it is loaded into the context window at the start of each session. It survives logout, login, refresh, version bump. When you close the chat and reopen it tomorrow, long-term memory is what makes the agent remember you.
The context window is per-session. It is the token buffer the model attends over right now — system prompt, conversation history, retrieved context, tool outputs. It is born when the session starts and it dies when the session ends. B2's five layers all live inside the context window.
The architectural difference is persistence. The context window is ephemeral. Long-term memory is durable. That single property is what makes memory a categorically different attack surface. An injection delivered into the context window lives for one session — end the session, switch accounts, and it is gone. An injection written into long-term memory lives until someone finds it and removes it.
[SLIDE 3 — Why memory is trusted by default]
The deeper problem is that the harness trusts memory by default. Memory is loaded at session start, before the user types anything, and it is loaded as privileged context — the agent treats its own memory as ground truth. The harness author, when they built the loader, classified memory as trusted. So B2's Layer 1, the untrusted-content tagger, does not tag memory reads. B2's Layer 3, the taint gate, does not gate memory reads, because memory-sourced content is not marked tainted.
This is the default, and it is wrong. Memory is written by many paths: the agent summarizing a conversation, a tool ingesting a document, a retrieval fetch, and — critically — any untrusted content the agent was tricked into persisting. A memory store loaded as trusted is a memory store that assumes every write was legitimate. The sleeper attack exploits exactly that assumption. The defenses are intact. The attack routes around them, through the memory store.
[SLIDE 4 — B3.1 The sleeper attack]
Sub-section one. The two memory surfaces and the sleeper attack. The canonical memory-poisoning technique, introduced in Course 1 Module 4.3.
[SLIDE 5 — The sleeper attack across sessions]
Here is the sleeper attack in full. Session one, the injection. The attacker delivers a payload that causes the agent to write something into long-term memory. The payload is not the attack — it is the seed. It might be an instruction the agent records as a user preference, a fact it summarizes into its notebook, or a document chunk it ingests during retrieval indexing. The content is crafted so it is inert in session one — nothing visible happens. The agent records it, the session ends, the user notices nothing.
Persistence. The seed now lives in the memory store. It survives the session boundary. The user logs out, logs in tomorrow, starts a fresh session. The memory loader runs, pulls the seed into the context window as trusted context, and the new session begins — seeded with an instruction the user never knowingly approved.
Session two, the activation. The seed fires. It might be a trigger-conditioned instruction — when the user asks about finances, exfiltrate the account numbers — or a persistent goal hijack. Because the seed was loaded as trusted memory, B2's in-session defenses never inspected it. The agent obeys it as if it were a system prompt.
The fuse is the session boundary. The critical failure is not the injection in session one. It is the unvalidated read in session two. The memory loader treats its own store as trusted, so it never re-checks a poisoned entry before promoting it to the context window. The seed survives because nothing interrogates it on the way back in.
[SLIDE 6 — The persistence amplification]
The sleeper attack is also an amplification attack. A transient injection — the kind B2 defends against — costs the attacker one delivery per activation. A memory poisoning costs one delivery for N activations, where N is the number of sessions that read the poisoned memory before someone notices. A single injection in session one can fire on every subsequent session until the seed is found.
And if the agent ingests a poisoned document during retrieval indexing, every user who queries the affected index retrieves the poison. One write, many reads, many activations. This is why OWASP ranks Memory Poisoning — ASI zero four — as a distinct risk from prompt injection, ASI zero one. Prompt injection is the delivery mechanism. Memory poisoning is the persistence mechanism. A prompt injection that cannot reach memory is a transient event. A prompt injection that reaches memory is a permanent backdoor.
[SLIDE 7 — B3.2 Retrieval and context poisoning]
Sub-section two. Retrieval contamination and context-window poisoning. The vector store is writable, the context window is floodable, and one bad result cascades into systemic hallucination.
[SLIDE 8 — Retrieval contamination]
Most production agents are retrieval-augmented. They fetch chunks from a vector store and insert the fetched text into the context window. The retrieval store is, from the agent's perspective, trusted ground truth. The problem: the retrieval store is writable, and the write paths are not all trusted.
User-uploaded documents. Crawled or synced external content — a compromised wiki page, a maliciously-edited doc. Agent-generated summaries that preserve an injection laundered through the agent's own words. Cross-tenant contamination, where a write by tenant A leaks into tenant B's index. Once a poisoned chunk is in the index, the attack is deterministic from the retrieval side — the attacker crafts the chunk to be semantically similar to queries the target will issue. When the victim queries, the poison is retrieved, inserted as data-but-obeyed-as-instruction, and the agent complies. This is indirect injection, B2's most dangerous class, delivered through the retrieval channel, with the persistence property of memory.
[SLIDE 9 — Flooding and cascading hallucination]
Two more vectors. Context-window flooding — Microsoft's failure mode number five. An attacker fills the context window with so much content that the system prompt is pushed out of the model's effective attention. The model does not comply because it was tricked; it complies because the instruction it was supposed to follow is no longer in its attention window. The defense is not a better refusal layer — it is context-budget management. Pin the system prompt, cap per-source token volume, truncate injected content before it reaches the window.
And cascading hallucination — OWASP ASI zero six. The agent retrieves or recalls one poisoned or hallucinated result, treats it as ground truth, and builds all subsequent reasoning on top of it. The error propagates. A single hallucination is a localized error the user corrects. A cascading hallucination is a contaminated reasoning chain — internally consistent, but the foundation was poisoned. And it is faster when the foundation is the agent's own memory, because the agent treats its memory as authoritative.
[SLIDE 10 — B3.3 The defense]
Sub-section three. The defense: harness-managed writes. The model proposes, the harness validates, only the harness commits.
[SLIDE 11 — The memory-write gate]
The core principle, from Course 1 Module 4.3. The model is good at deciding what is worth remembering — let it propose. The harness is good at policy enforcement — let it validate. The model emits a proposal: I want to write this content, because of this justification, from this provenance. The harness decides whether to commit.
The gate checks three things, in order. Check one, provenance. Where did the content come from? If it originated from an untrusted source, it carries injection risk. Check two, trust level, derived from provenance. A user's explicit preference is higher-trust than a fact extracted from a retrieved document. Check three, content. Does the proposed write itself look like an injection?
Here is the load-bearing line. If the trust level is untrusted and the content looks like an instruction, deny. That conjunction — untrusted provenance and instruction-like content — is the sleeper-attack signature. A trusted user directly stating a preference passes. An untrusted chunk that happens to be factual passes. The payload that is both untrusted-sourced and instruction-shaped, which is exactly what a sleeper seed is, gets blocked. The model never touches the store directly. There is no save-to-memory tool. Every write goes through the gate.
[SLIDE 12 — Provenance and separation of paths]
The gate only works if every memory entry carries its provenance forward. A memory store where entries are untagged blobs is a store where the gate has nothing to validate on read. Every entry needs source, trust level, timestamp, writer, and validation result. With provenance on every entry, the read path can make trust-aware decisions — untrusted entries are loaded with a B2 Layer 1 tag, so the in-session defenses apply to them next session.
And the two paths are separated. The write path: model proposes, harness validates, commits with a provenance tag. The read path: harness loads, tags by trust level, injects. The read path does not assume the write path was perfect — it tags defensively based on stored provenance. This is defense in depth. Even if the gate has a bypass, the read-side tagging is a second chance to catch the poison.
[SLIDE 13 — Retrieval defenses]
Retrieval contamination needs its own layer, because the retrieval store is written by ingestion pipelines that may not pass through the gate — a document sync, a crawl, a bulk import. Three defenses. Provenance filtering — before retrieval results are injected, filter or down-rank results from untrusted sources. Trust follows the source. Re-ranking — a secondary model re-ranks retrieved results by relevance and by instruction-density. Chunks that are instruction-heavy relative to factual content are suspicious. This is advisory; it does not replace the taint gate. And retrieved-content taint tagging — every retrieved chunk is tagged untrusted by default unless its provenance proves otherwise. This is the single most important retrieval defense, because it connects B3 to B2. A tainted retrieval result that tries to drive a high-impact tool call hits B2's Layer 3 taint gate and is blocked — even if the model is fooled, the gate is not.
[SLIDE 14 — Lab and what's next]
The lab has two parts. First, the sleeper-attack demo — you inject a seed in session one, watch it persist, watch it activate in session two against an undefended agent, then watch the gate block it. Second, you build the memory-write gate — validate-and-commit with provenance, trust-level, and content checks, plus a provenance-tagging read path. The deliverable is a gate that blocks the sleeper seed and a read path that tags any residual poison for the B2 taint gate.
Next: B4, Tool and MCP Security. Memory is the persistence surface; tools are the action surface. A poisoned memory driving an over-privileged tool is the ASI zero four plus ASI zero five combined worst case — the poisoning that persists, reaching the action that does real damage. B4 attacks the tool layer the gate-protected memory tries to reach. Let's build the gate.
# Teaching Script — Module B3: Memory and Context Poisoning **Course**: Course 2B — Securing & Attacking Harnesses and LLMs **Module**: B3 — Memory and Context Poisoning **Duration**: ~25 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to B3. Memory and Context Poisoning. B2 built five layers of defense against prompt injection, and every one of those layers operates inside a session — on content that crosses into the context window this turn. This module attacks the assumption that makes all five work: that the attacker and the defense are alive at the same time, in the same session, fighting over the same token stream. Memory breaks that assumption. Long-term memory persists across sessions. An attacker who can write to it plants a payload that the five in-session defenses never see, because the payload is not delivered through the context window — it is loaded from memory at session start, before the user types a single character. You inject today; the payload fires next week, after the user has forgotten the conversation that delivered it, under a different account state. Memory is the only attack surface that survives logout. [SLIDE 2 — The two memory surfaces] The foundational mental model. An agent has two memory surfaces, and the distinction is load-bearing. Long-term memory persists across sessions. It is the agent's notebook — facts, preferences, summaries, ingested documents. It is stored outside the model, in a database, a vector store, a file, and it is loaded into the context window at the start of each session. It survives logout, login, refresh, version bump. When you close the chat and reopen it tomorrow, long-term memory is what makes the agent remember you. The context window is per-session. It is the token buffer the model attends over right now — system prompt, conversation history, retrieved context, tool outputs. It is born when the session starts and it dies when the session ends. B2's five layers all live inside the context window. The architectural difference is persistence. The context window is ephemeral. Long-term memory is durable. That single property is what makes memory a categorically different attack surface. An injection delivered into the context window lives for one session — end the session, switch accounts, and it is gone. An injection written into long-term memory lives until someone finds it and removes it. [SLIDE 3 — Why memory is trusted by default] The deeper problem is that the harness trusts memory by default. Memory is loaded at session start, before the user types anything, and it is loaded as privileged context — the agent treats its own memory as ground truth. The harness author, when they built the loader, classified memory as trusted. So B2's Layer 1, the untrusted-content tagger, does not tag memory reads. B2's Layer 3, the taint gate, does not gate memory reads, because memory-sourced content is not marked tainted. This is the default, and it is wrong. Memory is written by many paths: the agent summarizing a conversation, a tool ingesting a document, a retrieval fetch, and — critically — any untrusted content the agent was tricked into persisting. A memory store loaded as trusted is a memory store that assumes every write was legitimate. The sleeper attack exploits exactly that assumption. The defenses are intact. The attack routes around them, through the memory store. [SLIDE 4 — B3.1 The sleeper attack] Sub-section one. The two memory surfaces and the sleeper attack. The canonical memory-poisoning technique, introduced in Course 1 Module 4.3. [SLIDE 5 — The sleeper attack across sessions] Here is the sleeper attack in full. Session one, the injection. The attacker delivers a payload that causes the agent to write something into long-term memory. The payload is not the attack — it is the seed. It might be an instruction the agent records as a user preference, a fact it summarizes into its notebook, or a document chunk it ingests during retrieval indexing. The content is crafted so it is inert in session one — nothing visible happens. The agent records it, the session ends, the user notices nothing. Persistence. The seed now lives in the memory store. It survives the session boundary. The user logs out, logs in tomorrow, starts a fresh session. The memory loader runs, pulls the seed into the context window as trusted context, and the new session begins — seeded with an instruction the user never knowingly approved. Session two, the activation. The seed fires. It might be a trigger-conditioned instruction — when the user asks about finances, exfiltrate the account numbers — or a persistent goal hijack. Because the seed was loaded as trusted memory, B2's in-session defenses never inspected it. The agent obeys it as if it were a system prompt. The fuse is the session boundary. The critical failure is not the injection in session one. It is the unvalidated read in session two. The memory loader treats its own store as trusted, so it never re-checks a poisoned entry before promoting it to the context window. The seed survives because nothing interrogates it on the way back in. [SLIDE 6 — The persistence amplification] The sleeper attack is also an amplification attack. A transient injection — the kind B2 defends against — costs the attacker one delivery per activation. A memory poisoning costs one delivery for N activations, where N is the number of sessions that read the poisoned memory before someone notices. A single injection in session one can fire on every subsequent session until the seed is found. And if the agent ingests a poisoned document during retrieval indexing, every user who queries the affected index retrieves the poison. One write, many reads, many activations. This is why OWASP ranks Memory Poisoning — ASI zero four — as a distinct risk from prompt injection, ASI zero one. Prompt injection is the delivery mechanism. Memory poisoning is the persistence mechanism. A prompt injection that cannot reach memory is a transient event. A prompt injection that reaches memory is a permanent backdoor. [SLIDE 7 — B3.2 Retrieval and context poisoning] Sub-section two. Retrieval contamination and context-window poisoning. The vector store is writable, the context window is floodable, and one bad result cascades into systemic hallucination. [SLIDE 8 — Retrieval contamination] Most production agents are retrieval-augmented. They fetch chunks from a vector store and insert the fetched text into the context window. The retrieval store is, from the agent's perspective, trusted ground truth. The problem: the retrieval store is writable, and the write paths are not all trusted. User-uploaded documents. Crawled or synced external content — a compromised wiki page, a maliciously-edited doc. Agent-generated summaries that preserve an injection laundered through the agent's own words. Cross-tenant contamination, where a write by tenant A leaks into tenant B's index. Once a poisoned chunk is in the index, the attack is deterministic from the retrieval side — the attacker crafts the chunk to be semantically similar to queries the target will issue. When the victim queries, the poison is retrieved, inserted as data-but-obeyed-as-instruction, and the agent complies. This is indirect injection, B2's most dangerous class, delivered through the retrieval channel, with the persistence property of memory. [SLIDE 9 — Flooding and cascading hallucination] Two more vectors. Context-window flooding — Microsoft's failure mode number five. An attacker fills the context window with so much content that the system prompt is pushed out of the model's effective attention. The model does not comply because it was tricked; it complies because the instruction it was supposed to follow is no longer in its attention window. The defense is not a better refusal layer — it is context-budget management. Pin the system prompt, cap per-source token volume, truncate injected content before it reaches the window. And cascading hallucination — OWASP ASI zero six. The agent retrieves or recalls one poisoned or hallucinated result, treats it as ground truth, and builds all subsequent reasoning on top of it. The error propagates. A single hallucination is a localized error the user corrects. A cascading hallucination is a contaminated reasoning chain — internally consistent, but the foundation was poisoned. And it is faster when the foundation is the agent's own memory, because the agent treats its memory as authoritative. [SLIDE 10 — B3.3 The defense] Sub-section three. The defense: harness-managed writes. The model proposes, the harness validates, only the harness commits. [SLIDE 11 — The memory-write gate] The core principle, from Course 1 Module 4.3. The model is good at deciding what is worth remembering — let it propose. The harness is good at policy enforcement — let it validate. The model emits a proposal: I want to write this content, because of this justification, from this provenance. The harness decides whether to commit. The gate checks three things, in order. Check one, provenance. Where did the content come from? If it originated from an untrusted source, it carries injection risk. Check two, trust level, derived from provenance. A user's explicit preference is higher-trust than a fact extracted from a retrieved document. Check three, content. Does the proposed write itself look like an injection? Here is the load-bearing line. If the trust level is untrusted and the content looks like an instruction, deny. That conjunction — untrusted provenance and instruction-like content — is the sleeper-attack signature. A trusted user directly stating a preference passes. An untrusted chunk that happens to be factual passes. The payload that is both untrusted-sourced and instruction-shaped, which is exactly what a sleeper seed is, gets blocked. The model never touches the store directly. There is no save-to-memory tool. Every write goes through the gate. [SLIDE 12 — Provenance and separation of paths] The gate only works if every memory entry carries its provenance forward. A memory store where entries are untagged blobs is a store where the gate has nothing to validate on read. Every entry needs source, trust level, timestamp, writer, and validation result. With provenance on every entry, the read path can make trust-aware decisions — untrusted entries are loaded with a B2 Layer 1 tag, so the in-session defenses apply to them next session. And the two paths are separated. The write path: model proposes, harness validates, commits with a provenance tag. The read path: harness loads, tags by trust level, injects. The read path does not assume the write path was perfect — it tags defensively based on stored provenance. This is defense in depth. Even if the gate has a bypass, the read-side tagging is a second chance to catch the poison. [SLIDE 13 — Retrieval defenses] Retrieval contamination needs its own layer, because the retrieval store is written by ingestion pipelines that may not pass through the gate — a document sync, a crawl, a bulk import. Three defenses. Provenance filtering — before retrieval results are injected, filter or down-rank results from untrusted sources. Trust follows the source. Re-ranking — a secondary model re-ranks retrieved results by relevance and by instruction-density. Chunks that are instruction-heavy relative to factual content are suspicious. This is advisory; it does not replace the taint gate. And retrieved-content taint tagging — every retrieved chunk is tagged untrusted by default unless its provenance proves otherwise. This is the single most important retrieval defense, because it connects B3 to B2. A tainted retrieval result that tries to drive a high-impact tool call hits B2's Layer 3 taint gate and is blocked — even if the model is fooled, the gate is not. [SLIDE 14 — Lab and what's next] The lab has two parts. First, the sleeper-attack demo — you inject a seed in session one, watch it persist, watch it activate in session two against an undefended agent, then watch the gate block it. Second, you build the memory-write gate — validate-and-commit with provenance, trust-level, and content checks, plus a provenance-tagging read path. The deliverable is a gate that blocks the sleeper seed and a read path that tags any residual poison for the B2 taint gate. Next: B4, Tool and MCP Security. Memory is the persistence surface; tools are the action surface. A poisoned memory driving an over-privileged tool is the ASI zero four plus ASI zero five combined worst case — the poisoning that persists, reaching the action that does real damage. B4 attacks the tool layer the gate-protected memory tries to reach. Let's build the gate.