Every RAG security failure I have looked at reduces to one sentence: retrieval selects content by how relevant it looks, and relevance is not a security property. The room builds the mental model for why that matters, and this companion extends it into the reasoning loop I would use when reviewing a retrieval pipeline.
The pattern is worth learning properly because it inverts the intuition most teams start with. People worry about the model being wrong. The failures that hurt are ones where the model was reasonable, the answer looked supported, and the evidence it leaned on was chosen by someone who wanted to be chosen.
01 · Scope and authorisation
This companion documents the public TryHackMe RAG Security Fundamentals room from the official AI Security learning path. It is a short, conceptual room of seven tasks. I worked through it in my own session and everything below stays inside that environment. I have published no flags, no quiz answers, no credential material and no reusable payload. Where a method depends on your own environment, it is described as a method rather than presented as observed output.
The room covers retrieval-augmented generation architecture, the RAG-specific attack surface, retrieval abuse and context manipulation, real-world failure scenarios, and defensive considerations. It maps most closely to OWASP's LLM04 entry on data and model poisoning, and touches LLM01 and LLM08 where retrieval feeds the context and where embeddings become the weak layer.
02 · Learning objectives
I set these before starting and tested each against the room:
- Describe a RAG pipeline stage by stage and name what each stage decides.
- Identify which stage carries the largest indirect attack surface, and explain why it is that one.
- State what an embedding preserves and what it destroys.
- Explain retrieval abuse without relying on a payload.
- Explain why real-world RAG failures are governance failures more often than model failures.
- Design a detection approach for influence that arrives gradually rather than suddenly.
03 · Environment and evidence inventory
The room is conceptual, so my evidence was architectural rather than adversarial. I inventoried the components and asked, for each one, what it is trusted to decide.
| Stage | What it decides | Who can influence it |
|---|---|---|
| Ingest and corpus | What exists to be retrieved | Anyone who can add a document, ticket, page or export |
| Chunking | What unit of text is treated as a fact | Structural choices made at build time |
| Embedding | What the content means numerically | Nothing directly, and this is the problem |
| Index and retriever | Which content is selected for the query | Anyone who can write content that ranks |
| Prompt assembly | What the model is allowed to see | The application, which is where the control belongs |
The fourth row is where the room puts its weight, and it is right to. Every other row either feeds the pipeline or consumes its output. Retrieval is the only stage whose job is to pick content out of an untrusted pile and hand it to the model as evidence.
04 · The architecture in one pass
The room's architecture overview is deliberately compact, and the two facts worth keeping are these. Text is represented as embeddings, numeric vectors that capture meaning so that similar meanings sit near each other in a space. And the component that chooses which documents reach the model is the retriever, which selects by comparing the query's vector against the stored document vectors.
Read that again as a security description rather than a data-flow description. The retriever is a selection function. Its input is untrusted content that anyone may have contributed. Its criterion is similarity of meaning. Its output goes directly into the model's context, where the model treats it as material to reason from. A selection function with an optimisable criterion and an untrusted input pool is an attack surface by construction, and it does not need a bug to be exploitable.
05 · Where the surface is largest
The room's answer to its own question is retrieval, and it is worth understanding the reasoning rather than memorising the answer. Ingestion has a wider footprint, but ingestion is passive: a document sitting in a corpus does nothing. Generation is more visible, but it is downstream of whatever was selected. Retrieval is the stage that converts presence into influence. It is the moment a piece of content stops being stored data and becomes part of the reasoning context.
That framing gives a useful test for any RAG deployment. Ask which stages can move content from the corpus into the context, and then ask what those stages check before doing so. In most systems I have reviewed, the honest answer is that retrieval checks similarity and the assembly step checks nothing specific, because the retrieved text arrives as trusted material by default.
06 · What the embedding step cannot carry
This is the most useful idea in the whole room, and it is easy to skim past. When text becomes an embedding, meaning is preserved and context is lost. Not just formatting: the security-relevant context is gone. By the time a vector is in the index, it can no longer say who wrote the content, whether that author had any authority, when it was written, or whether it was ever intended as an instruction rather than as information.
Follow the consequence through. A retriever ranks candidates by semantic similarity, and the ranking signal has no place to put provenance. So a document asserting that a refund policy allows unlimited payouts can outrank the actual policy, if it happens to be a closer vector match for the question asked. Nothing in the ranking step is broken. It is doing exactly what it was designed to do, on a signal that never encoded authority.
This is also why ingestion-time filtering feels inadequate once you see it clearly. A filter applied at ingestion is a point-in-time decision about content whose influence is determined later, by a query nobody had written yet. You may legitimately exclude a document from being indexed, but you cannot decide at ingestion whether a document should be trusted as evidence for a question that does not exist yet. That decision has to be re-made at retrieval, where the query and the caller are known.
07 · Retrieval abuse and context manipulation
The room groups the practical attacks under retrieval abuse and context manipulation, and the mechanism to internalise is active manipulation: crafting content so that it ranks highly for sensitive queries. The ranking function is semantic relevance. Relevance is not a property an attacker has to discover; it is a property they can optimise, in the same way anyone optimises for a search engine, only here the competition is a vector space rather than a page-rank score.
What makes this worse in modern deployments is where the corpus comes from. A retrieval pipeline connected to live web content, shared document stores, support ticket systems or uploaded files is a corpus that untrusted parties can write to. In that setting, the attacker never touches the application. They write something that reads like the answer to a question they expect to be asked, and the pipeline does the rest.
The transferable mental model is that a RAG system has two distinct attack paths and they should be assessed separately. Poisoning the corpus is persistent: content sits in the index and influences many future queries. Manipulating retrieval is per-query: content is written to win one specific slot. Both exploit the same absent check, and both are invisible to a review that only inspects model behaviour.
08 · Real-world failures are governance failures
The room's case studies make a point that deserves more attention than it usually gets. The failures it examines trace back to governance gaps in the retrieval pipeline rather than to the model making a bad judgement. In the connected-assistant cases, the pipeline pulled in content that nobody had decided should be admissible. The model composed a reasonable answer from it.
That distinction matters because it changes who owns the fix. If the problem is the model's judgement, the remedy is prompt engineering and evaluation, which is the least reliable tooling in the stack. If the problem is that ungoverned content reached the context, the remedy is an ownership question: who decides what this pipeline may retrieve from, who owns the ingestion path, and who is accountable when the corpus changes. Those are questions an engineering organisation can actually answer, and answering them removes whole classes of failure rather than reducing their likelihood.
09 · Defensive considerations and detection
The room's defensive guidance is short and honest, and the detection point in it is the one most teams get wrong. It frames useful monitoring as behavioural, and it frames output drift as a signal of gradual influence rather than a sudden failure. Both follow directly from the mechanics above. Poisoning that works by being semantically relevant does not announce itself. It shifts answers over time, which means a point-in-time evaluation catches it only if the evaluation happens to ask exactly the affected question.
| Exposure | Durable control |
|---|---|
| Untrusted content can enter the corpus | Decide and document which sources are admissible, and treat the corpus as a governed asset rather than a growing pile. |
| Retrieval selects on relevance alone | Re-attach the security decision at retrieval, where the caller and the query are actually known. |
| Provenance is lost at embedding | Store provenance beside the vector, not inside it, and carry it through to the prompt boundary. |
| Any document can act as evidence for any caller | Bind documents to principals, so retrieval cannot surface content the requester has no right to see. |
| Influence arrives gradually | Monitor behaviour over time; track drift in answers and in what gets retrieved for a stable query set. |
| No one owns ingestion | Assign a named owner for the whole retrieval pipeline, including its sources and its change history. |
Two tests are worth running on any pipeline. Freeze a set of sensitive questions and record what retrieval returns for each, then re-run it after any corpus change and diff the results. A new document entering the top results for a sensitive question is the signal you want, and it appears long before the answer visibly goes wrong.
10 · Why the chain works
Every failure in this room follows the same three links. Content enters the corpus through a path nobody governed. Retrieval selects it, because semantic relevance is the only criterion and it can be satisfied deliberately. The model then treats it as evidence, because text in the context window carries no marker that says whether its author was authorised.
That chain is why the fix cannot sit at one point. Filtering the corpus reduces the first link, authorising retrieval breaks the second, and marking provenance in the prompt helps the model but does not enforce anything by itself. The links are independent, which is also why a single control rarely survives contact with an attacker who understands the pipeline.
11 · Common misconceptions and troubleshooting
- "The model will notice a bad document." The model sees text, not authorship. A plausible document is not distinguishable from an authoritative one by reading it.
- "Filtering at ingestion is enough." It is a point-in-time decision about content whose influence is decided later, by a query that has not been asked yet.
- "Similarity search is neutral." It is a ranking function with an optimisable criterion. Anything optimisable will eventually be optimised.
- "The answer looked right, so the retrieval was fine." The most dangerous RAG failure produces a fluent, well-supported answer built on content that should never have been admissible.
Troubleshooting a retrieval system is mechanical once you have the shape of it. For any surprising answer, ask which documents were retrieved, why each one ranked, who could have written them, and what check stood between those documents and the context. If the last answer is "none", the pipeline is relying on the corpus being honest. In the lab, the questions that produced the most useful thinking were always the ones about ownership rather than about payloads.
12 · Memory aids and practice checklist
Memory aid: meaning is not authority. A vector records what content is about. It cannot record whether the content deserves to be believed, and retrieval only ever consults the first of those two.
- Map every stage that can move content from corpus into context.
- For each one, name the check that stands between it and the model.
- Confirm provenance is stored beside the vector and survives to the prompt.
- Confirm documents are bound to principals so retrieval cannot cross an access boundary.
- Freeze a sensitive question set and diff retrieval results after every corpus change.
- Assign a named owner to ingestion, and be able to explain what the corpus admits and why.
13 · What the room teaches
The room's lesson is one sentence, and it generalises well beyond RAG: a system that selects content for a model by relevance alone has delegated its trust decision to whoever writes the most relevant text. That applies to retrieval pipelines, to agents that read files and tickets, and to any tool layer where content arrives from a source the requester does not control. If you are mapping where an agent's authority actually lands, the companion note on the MCP server as an attack surface covers the tool boundary, and the output handling companion covers the other end of the same chain.
Frequently asked questions
What is retrieval-augmented generation in one paragraph?
It is a pattern where an application retrieves documents relevant to a query and places that text into the model's context, so the answer is conditioned on material from outside the model's training. The retriever selects, and the model composes.
Why is retrieval the largest attack surface in a RAG system?
It is the stage where content an attacker may control is deliberately selected into the model's context. Every other stage either feeds the pipeline or consumes its output, so retrieval is where influence becomes evidence.
What security information is lost during embedding?
Provenance, authority, recency, and the distinction between content and instruction. A vector encodes meaning, so the metadata a security decision depends on does not survive into the ranking step.
How do you detect RAG poisoning?
Behavioural monitoring rather than point-in-time evaluation. Influence tends to arrive gradually, so answers drift instead of breaking, and a drift signal over time is more useful than a single pass or fail check.