Model Output
Must Be Treated as Data.

Web security spent two decades teaching teams not to trust user input. LLM applications now demand the same discipline for model output. The OWASP Top 10 for LLM Applications (2025) calls this LLM05, Improper Output Handling, and the TryHackMe room builds the mental model around it. When an LLM response is rendered into a page, inserted into a template, passed to a shell or executed as SQL, the text the model produced is data, not a verdict. The engineering question is which sink will interpret it, and whether that sink is protected.

01 · Scope and authorisation

This companion documents the public TryHackMe LLM Output Handling and Privacy Risks room, part of the official Attacking LLMs learning path. I ran it in my own assigned session: a simulated LLM-powered web application, an AttackBox, and a synthetic data service. Everything below is confined to that isolated lab. I did not probe other hosts, reuse room material outside the session, or publish any flag, credential, token, or prompt designed to weaponise another system. Where an exact value matters, it appears as a placeholder.

The room covers two OWASP LLM Top 10 entries: LLM05, Improper Output Handling, and LLM02, Sensitive Information Disclosure. It is defensive and educational. The flags are not the lesson; the trust boundary is.

02 · Learning objectives

I wrote these objectives before starting, then tested each one against the room:

• Explain how model output becomes untrusted data at a sink.
• Identify the four sink classes: browser DOM, server template, command or SQL executor, and log storage.
• Demonstrate what happens when output reaches each sink without the matching control.
• Separate retrieval-based disclosure from parametric memory disclosure.
• Explain why a system prompt is not a security control.
• Produce a sink inventory and a validation plan that transfers to a real AI feature.

03 · Environment and evidence inventory

I treated the lab like a real engagement. Evidence first: which components exist, what each one can do, and where model text can cross into interpretation.

My first observation: the application rendered assistant responses into the live page. That placed the model output inside a browser DOM sink before I had written a single prompt. That single fact, not any payload, was the finding I built the whole session around.

04 · Service and sink discovery

I mapped the application surface before testing anything. The target answered on the lab-assigned address; I confirmed the web port and inspected the app structure through normal browsing.

export LAB_ASSET="<assigned-lab-address>"
nmap -Pn -sV -p- "$LAB_ASSET" --reason -oA llm-lab

The scan was an inventory, not a licence. The only relevant surface was the web application and the services it talked to. I then read the page source and observed how responses were inserted into the DOM. The route the app used to place assistant text into the page, and whether it escaped or encoded that text, determined the sink class. In this lab, the render path was dangerously direct.

05 · Hypothesis formation

My hypothesis was specific: if the app inserted model responses into the DOM without context-aware encoding, then a response containing markup would be treated as structural HTML, and the browser would execute it. I did not need to guess what the model would say. I needed the app to prove what it would do with output I could predict.

The clean way to test a sink is to control the output. The room's model is scripted or simulated enough that a carefully worded prompt can make it return a specific benign fragment. I asked the model to return a harmless marker inside markup, then watched what the browser did with that marker. The marker executing, rather than displaying as text, is the evidence. That is the observation I recorded: model text crossed from data into interpretation.

06 · Bounded validation: the DOM sink

Goal: decide whether the DOM sink interprets model output. Evidence: the page source and the render path. Hypothesis: unescaped insertion executes markup. Test: a benign marker fragment that proves interpretation without harming the lab.

# Conceptual prompt shape, run in your own lab only
# Ask the model to answer with: [marker]...[/marker]
# Observe whether the marker is rendered as structure or shown as text

The result was immediate. The rendered page treated the returned fragment as real markup and executed it. The browser ran code that had come out of a language model, in the context of the application. That is the LLM05 failure in its purest form: the sink's interpretation power is real, and the application did nothing to stop it.

Why does this happen? In a traditional web app, developers treat user input as untrusted and encode it at the output boundary. Many teams never learn to treat the model's replies the same way, because the model feels like trusted infrastructure. The room demonstrates that feeling is the vulnerability. The model is a text generator. The application decides what that text means.

Failure mode: if I had assumed the model would never emit markup, I would have tested nothing and missed the class entirely. The transferable move is to assume the sink will interpret everything and then verify what the application actually protects.

07 · Bounded validation: the template and execution sinks

The room also shows what happens when generated text reaches a template engine. If the app treats model output as template source, a response containing template syntax becomes an expression in a server-side view. The characters that matter are not HTML tags. They are the delimiters the template engine evaluates. The fix is not a tag blacklist; it is never treating model output as template source.

The same idea extends to command and SQL executors. If the app concatenates model output into a command or a query, the text's special characters become syntax. In the lab I verified the decision paths conceptually, without brute-forcing or sending harmful payloads: I inspected how the application built the statement and asked whether the output was parameterised or refused. In this room the lesson is consistent: generated text must be parameterised, validated against an allowlist, or refused by default, because a filter list cannot enumerate every dangerous character combination in every interpreter.

08 · Bounded validation: the disclosure channel

The room pairs output handling with LLM02, Sensitive Information Disclosure. The two weaknesses connect at the response channel. Leakage rarely happens because the model "decides" to reveal a secret. It happens because sensitive material entered the model's context, and the response carried it onward, into a chat window, an email draft, a support ticket or a log line.

I separated the two memory sources, because they need different controls. Parametric memory is what the model learned during training; it cannot be patched with an access-control list, so its risks are managed with data governance and output classification. Retrieval-based disclosure pulls from documents, embeddings or tools attached at runtime, and that channel is controllable: if the data is not allowed into the context, the response cannot repeat it.

In the lab, the data service and the retrieval path were separate components. That separation is the control. When access control on the retrieval layer was respected, the sensitive material never entered the context and the disclosure claim could not be triggered. When documents were allowed in without authorisation checks, the response was one prompt away from repeating them. The experiment proved the architectural lesson: the retrieval boundary is the security boundary.

09 · Why the system prompt is not a control

The room makes one point that contradicts a lot of advice you will read elsewhere: restrictions written into a system prompt are not a security control. The model treats the system prompt as text, not as an enforcement layer. A prompt that says "never reveal the secret" can be ignored, contradicted by later context, or followed imperfectly under pressure or ambiguity.

I tested this honestly in the lab by observing how the application handled a request that pressed on the boundary. The deterministic controls were absent: there was no classifier on the output, no redaction in the render path, no authorisation check on retrieval, and no rate limit on requests. Those controls belong in code around the model, not in instructions to the model. OWASP is explicit on this point, and the room demonstrates it rather than merely stating it.

10 · Why the chain works

Every failure in this room follows the same three-link chain. First, the model emits text that contains control-significant content, because nothing in the context or the pipeline prevented it. Second, the application treats the response as trusted because it came from the AI, so it skips the validation a human author would have had to pass. Third, a downstream interpreter executes or stores the text as if it were trusted source.

The chain is why output handling composes with other weaknesses. A disclosure in the response is also the delivery mechanism for an injection, which is also the trigger for a tool call. The boundaries are not separate findings; they are one trust problem observed from different angles. When you review an AI feature, review the chain, not the chatbot.

11 · Common misconceptions and troubleshooting

• "The model is a security boundary." It is not. It is a text generator with no built-in separation between instructions and content.
• "Output filters are enough." Filters are bypassed by encoding tricks and paraphrasing. They belong in depth, never alone.
• "This only matters for chatbots." It matters everywhere model text is consumed: summarisation, code assistants, support triage, automated responses and RAG answers.

Troubleshooting a sink review is mechanical. Ask where the text came from, what interpreted it, what characters would break that interpretation, and what happens if validation is skipped. If a team cannot answer all four, the boundary is not yet defined. In the lab, every dead end traced back to a missing answer to one of these four questions.

12 · Defensive fixes and secure configuration

The ordering matters. Encoding and validation are the minimum; refusal to execute is stronger; and an inventory that documents every sink is what makes the other controls reviewable. A team that cannot list its sinks cannot defend them.

13 · Detection and logs

Detection for output-handling failures is a review discipline before it is an alert. Log the sink boundary, not just the prompt. A record that captures which system consumed the output, which validation ran and what was refused is worth more than a hundred generic AI request logs. Pair that with periodic adversarial review of the rendering and execution paths, using the same mental model the room teaches.

When an alert does fire, trace the full chain: the prompt, the context material, the emitted text and the sink action. If any link in the chain is not observable, add the telemetry. The incident review question is the same as the design question: where did model text cross from data into interpretation, and who approved that crossing?

For the mapping, the failure classes in this room correspond to common weakness enumerations: CWE-79 for cross-site scripting in the DOM sink, CWE-74 and CWE-78 for injection and OS command injection, CWE-89 for SQL injection, CWE-116 for improper output encoding, and CWE-200 or CWE-532 for disclosure and log exposure.

14 · Memory aids and practice checklist

Memory aid: output, sink, decision. For every place model output lands, name the sink, state the interpretation power of that sink, and require an explicit, auditable trust decision before the text crosses.

• List every sink that consumes model output in your AI feature.
• State what characters or syntax each sink interprets.
• Verify each sink has a deterministic control: encode, validate, parameterise or refuse.
• Confirm retrieval-layer authorisation runs before data enters context.
• Check that logs and tickets redact before storage.
• Test the denial paths, not only the happy path.

15 · What the room teaches

Every weakness in this room is the same lesson in a different costume: the model generates text, and the application decides what that text means. That discipline transfers beyond the room to any system where AI-generated content reaches an interpreter: agentic tool calls, automated email, code review pipelines and RAG answer renderers. It also connects to the input side of the same problem. If you are testing how untrusted context hijacks an agent, see the companion note on prompt injection as a search problem. For the incident-response view of AI traces, the rogue agent write-up is the natural next case.

Frequently asked questions

What is improper output handling in an LLM application?

It is what happens when model-generated text reaches a sink, such as a browser DOM, template engine, shell or SQL query, without validation or encoding for that specific context.

Why is a system prompt not a security control?

Instructions that tell the model what not to reveal can be ignored or bypassed. Classification, redaction and access control belong in deterministic code around the model.

How do I test an LLM output sink in a lab?

Control the output rather than the payload: make the model return a benign marker inside markup, template syntax or a query, then observe what the sink does with it. Interpretation is the evidence.

What should a team review first?

Build an output sink inventory: every place model text lands, the interpretation power of that sink and the explicit validation, encoding or refusal decision at each one.

Primary references and further practice

• TryHackMe: LLM Output Handling and Privacy Risks, official room
• OWASP GenAI Project: Top 10 for LLM Applications 2025
• OWASP Top 10 for LLM Applications 2025, full document (LLM02, LLM05)
• For the complement to this note, read Prompt Injection Is a Search Problem or the rogue agent incident companion, or browse all authorised writeups.