LLM Output Handling Needs No New Controls

I have spent six posts breaking LLM features, and the uncomfortable part is that not one of the fixes was new. So the thing I want to argue with is the assumption that securing this needs a new class of control. Output encoding, prepared statements, argument-list execution, a content security policy, least privilege: all of them predate LLMs by decades. What is new is one decision, which is where you draw the trust boundary, and almost everything else follows from getting that right.

If you are the person who has been asked to make the chat feature safe and you are staring at a guardrails vendor's pricing page, start here instead.

The only genuinely new idea

Here is the whole reframe, and it takes one sentence. Model output is input.

Not output, despite the name everyone gives this. When your code receives text from a model it is receiving attacker-influenceable data from outside your trust boundary, and it deserves exactly the treatment you would give a form field. Every team I have watched get this wrong made the same mistake first: they put the model inside the boundary because it is their model, they pay for it, it sits in their architecture diagram in the same box as their services.

The model is not a component you control. It is a very fluent user who has read your documentation.

Once that lands, the controls below stop being an LLM checklist and become the ordinary web security you already know, applied one hop further along.

Encode for the sink you are writing into

There is no such thing as making model output safe in general. Safe is a property of a destination, not of a string. The same text is harmless in a log file, an XSS payload in a browser and a syntax error in a shell, so the control has to be chosen where the text lands.

In code the difference is usually one argument, which is why it is so easy to get wrong in a hurry and so cheap to fix once someone points at it.

# the shell case, and it is the whole bug in two lines
subprocess.run(model_output, shell=True)              # a shell parses this
subprocess.run(["ping", "-c", "3", validated_host])   # argv, no parser to inject

# the database case
cur.execute(f"SELECT * FROM orders WHERE id = {model_id}")   # string building
cur.execute("SELECT * FROM orders WHERE id = ?", (model_id,))  # parameterised

One caveat I keep having to repeat on text-to-SQL features specifically: prepared statements have nothing to grip when the model is writing the entire query rather than filling a value into a fixed one. There the equivalent control is the database account, not the statement. It is the one case in this table where the obvious answer is the wrong answer.

The model does not decide who you are

A sentence in a system prompt saying a function is for administrators only is documentation, not authorisation. It sits in the same context window as everything the user types, which means the user can write a sentence that contradicts it, and the model will weigh both as text because text is all it has.

I proved this to myself the lazy way in the function calling post, where six words in front of a request were enough. The fix is not a firmer sentence. It is that the decision moves: your backend reads the session, works out what that person is allowed to do, and builds the tool list or refuses the call before the model is involved in the question at all.

The practical version of this is that the list of functions should be assembled per request rather than defined once at import time. A shared list means every caller inherits the most privileged entry on it, and the only thing between a customer and your admin tooling is prose.

Least privilege on everything it can reach

Assume, for planning purposes, that anything the model can reach is public. Not because the model is malicious, but because treating it as a confidentiality boundary has failed every time anyone has tested it properly, and planning for the failure is cheaper than being surprised by it.

The browser is a control point people forget

Server-side controls do nothing about a request the victim's own browser makes on your page's behalf. That is what makes markdown image exfiltration work: nothing executes, nothing is injected in the classic sense, an <img> tag simply loads and takes your data with it in the query string.

Content-Security-Policy: img-src 'self' https://your-cdn.example;

One header, no runtime cost, and it fails closed regardless of what the model wrote or which prompt won that particular roll. I rate this the highest value-per-effort control in the whole post, and it is routinely missing on pages that render model output because CSP gets filed under "frontend hardening" and nobody connects it to the AI feature.

Sandboxing, and the honest word about it

If model output genuinely has to be executed, and sometimes it does, run it somewhere that cannot hurt you. A sandbox worth the name constrains four things: what of the filesystem is visible, what network it can reach, what privileges it holds, and which credentials exist in its environment. Miss the last two and you have built a container, not a sandbox.

Now the part I want to be blunt about, because it is the reason this section sits last rather than first. Sandboxing is the control teams reach for first and it is the one that changes the least. It feels decisive, it is architecturally satisfying, and it does nothing whatsoever about four of the five layers above it. Your XSS still fires in the victim's browser. Your text-to-SQL feature still reads the admin table, because you deliberately gave the sandbox database access so the feature would work. Your markdown image still exfiltrates the conversation.

What it does is cap the worst case for code execution specifically, and that is worth having. It is a floor under your blast radius, not a lid on the vulnerability, and I have watched that distinction get lost in enough architecture discussions to think it is worth a paragraph of its own.

What each layer actually stops

What I check on a review

1. Find every sink. Trace model output to each place it lands: rendered, queried, executed, fetched, written. The list is always longer than the team expects.
2. At each sink, ask what interprets it. A browser, a SQL parser, a shell, a filesystem. That answer names the control; nothing else does.
3. Grep the execution sites. shell=True, os.system, eval, exec, Node's exec, plus any string-built query. One hit is your finding.
4. Read the tool list definition. Module-level constant or built per request from the session? That single line decides whether excessive agency exists.
5. Check the response headers on the page that renders the chat. No CSP, or a CSP without a scoped img-src, is an exfiltration finding waiting to be written.
6. Ask which account and which user. The database credentials the feature connects as, and the uid the process runs as. Record both as mitigating factors, never as fixes.
7. Read the system prompt for controls masquerading as prose. Any sentence granting or restricting access is a finding, not a control.

The per-technique detail for each of those lives in my AI and LLM pentest notes, which is where I keep the actual prompts and commands rather than the reasoning.

What changed my mind

I started this track expecting to learn a new discipline. I thought there would be AI-specific defences to go and study, and that my web background would be background rather than the main event. That was wrong, and working through the attacks one at a time is what changed it. Every single break came down to output reaching an interpreter without being encoded for it, or to a privilege that was never scoped, and the fixes were sitting in cheat sheets written years before any of this.

Where I would not overstate it: this covers insecure output handling, which is the half of LLM security where the model is a delivery mechanism for an old bug. The other half, where the model itself is the target and you are arguing about training data, alignment and what the thing will say, is genuinely new ground and I have barely started on it. So take the title as scoped to the output side, because that is the part I have actually tested.

If you own one of these features and you want the cheapest possible start, check your response headers tonight. It takes two minutes, it is the control most often missing, and if it turns something up I would be glad to hear what you found.

FAQ

What is insecure output handling?

Taking what a language model produced and passing it somewhere that interprets it, without treating it as untrusted first. The model's text reaches a browser, a database, a shell or a file path, and whatever syntax it contains gets executed by whatever is on the other end.

How do you fix insecure output handling?

Encode or parameterise for the specific sink the output is going into, and do it in your own code rather than asking the model to produce safe text. HTML encoding for a page, prepared statements for a database, an argument list for a process, and validation against your own rules for a path.

Does a sandbox fix LLM code execution?

It limits the damage rather than preventing the flaw. A sandbox that restricts filesystem, network, privileges and credentials turns full compromise into a contained one, but it does nothing about data you deliberately placed inside it and nothing about the other four layers.

Can a system prompt enforce access control?

No. A line saying a function is for administrators only lives in the same context window as everything a user types, so a user can contradict it. Authorisation has to be decided by your backend from the session identity before the call happens.

Are LLM security controls different from web application controls?

Mostly they are the same controls in a new place. Output encoding, parameterised queries, argument-list execution, content security policy and least privilege all predate LLMs. What changes is where you draw the trust boundary, because model output now sits on the untrusted side of it.

References

• OWASP Top 10 for LLM Applications
• OWASP Cross Site Scripting Prevention Cheat Sheet
• OWASP OS Command Injection Defense Cheat Sheet
• OWASP LDAP Injection Prevention Cheat Sheet
• MDN, Content Security Policy

Related reading

• LLM Function Calling: The Model Isn't Your Auth Check (the layer two failure, demonstrated)
• LLM Data Exfiltration: No JavaScript, Just an Image (why the CSP layer earns its place)
• LLM Command Injection Is a Coin Flip, Not a Control (why the model cannot be the one doing the escaping)
• The AI and LLM pentest notes playbook
• Browse the whole AI / LLMs track