There's a document sitting in a repo somewhere in your company right now. It's called system_prompt.md or agent_policy.txt. It's somewhere between 3,000 and 15,000 words long. It contains your refund limits, your tone of voice rules, your escalation paths, your data handling requirements, the four things the agent must never promise, the GDPR paragraph legal insisted on, and a section near the bottom that says "always ask for confirmation before taking any irreversible action."
Someone in a compliance meeting asked "how do we stop the agent doing X?" and the answer was "we've added it to the prompt." Everyone nodded. The ticket got closed.
That document is not a control. It's a wish list with good formatting.
A recent arXiv paper, HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following, put numbers on something that anyone who has shipped an agent into production already suspected: long policy documents do not reliably govern model behaviour. As the context grows, adherence to any individual instruction degrades. Rules compete with each other. The instructions you care about most get quietly outvoted by whatever is nearest the model's attention - usually the user's last message, or the tool output that just came back, or the three paragraphs you added most recently.
This isn't a prompt engineering problem you can fix with better prose. It's a category error about where controls live.
Why prose can never be a control
Here's the thing that separates a control from a preference: a control has a defined failure mode.
When a database rejects a write because of a foreign key constraint, you know exactly what happened. The transaction failed, loudly, at a specific line, with a specific error. You can alert on it. You can count it. You can prove to an auditor that it fired 1,412 times last quarter and blocked every one of them.
When a language model reads "never issue refunds above €500" in paragraph 47 of a system prompt and then issues a €900 refund, nothing failed. The system worked exactly as designed. A probability distribution over tokens produced tokens. There's no exception, no log line, no boundary that was crossed - because there was no boundary. There was a suggestion, weighted against several thousand other tokens, some of which happened to be a very persuasive customer.
Three mechanics drive this, and they're worth understanding properly because they determine what you can and can't delegate to the prompt:
Attention is finite and shared. Every instruction you add dilutes the influence of every other instruction. A 40-page handbook doesn't give you 40 pages of governance; it gives you an averaged blur of priorities. The rule you added on day one is competing with the rule you added on day ninety, and the model has no idea which one your general counsel would die on a hill for.
Position matters more than importance. Instructions near the start and end of context carry more weight than instructions buried in the middle. Your most critical constraint may be sitting in the exact worst place for it to survive. And the model has no concept of "this section overrides that section" unless you engineer precedence explicitly - natural language has no PRIORITY: CRITICAL keyword that the transformer respects.
Conflict resolution is statistical, not hierarchical. Real policy documents contradict themselves. Any handbook long enough to be useful contains tension: be helpful vs. be cautious, resolve quickly vs. verify identity, be concise vs. include the required disclosure. Humans resolve this with judgement and the knowledge that some rules get you fired. A model resolves it by picking whichever path has more probability mass in the current context.
I wrote about the underlying constraint here before in The AI Context Window Problem. Bigger context windows made this worse, not better, because they made the bad approach feel viable. When you could only fit 4,000 tokens you were forced to be architectural. Now you can paste the whole employee handbook in, so people do.
The governance ladder
The useful mental model is a ladder, and every rule in your policy document needs to be placed on exactly one rung.
Rung 1 - Impossible. The agent physically cannot do it, because the capability doesn't exist in its tool surface. There is no issue_refund tool with an unbounded amount parameter. There is no tool that returns raw PII. This is the strongest form of governance and it requires zero trust in the model.
Rung 2 - Rejected. The agent can attempt it, but a deterministic layer refuses. Server-side validation on tool calls, permission checks against the actual authenticated user, state machine transitions that aren't legal from the current state. The model proposes; code disposes.
Rung 3 - Reviewed. The action is queued for a human with genuine authority and genuine context to reject it. Not a "are you sure? [Yes]" dialog, which is theatre. An actual queue with an actual owner and an SLA.
Rung 4 - Suggested. It's in the prompt. Tone, formatting, how to phrase a decline, when to offer alternatives, style. This rung is fine. This is what prompts are genuinely good at.
Anything on rung 4 that your legal or finance team believes is on rung 1 or 2 is where you'll get hurt. The audit question I'd ask any team shipping an agent is brutally simple: for each rule in your policy document, show me the line of code that enforces it. If the answer is "it's in the prompt," that rule doesn't exist.
Tool schemas are your real policy document
The most common architectural mistake is treating tool definitions as an API surface for the model, rather than as the enforcement boundary. If a tool accepts an amount, that amount will eventually be wrong. If a tool accepts a customer_id, the model will eventually pass a different customer's ID - not maliciously, just because it appeared in a transcript three turns ago.
The fix is to design tools so that the dangerous parameters aren't the model's to supply.
// WRONG: the model decides who and how much. // The €500 limit lives in the prompt. Which means it doesn't exist. const issueRefund = tool({ name: "issue_refund", parameters: z.object({ customerId: z.string(), amountCents: z.number(), reason: z.string(), }), execute: async ({ customerId, amountCents, reason }) => payments.refund(customerId, amountCents, reason), });
// RIGHT: identity comes from the session, not the transcript. // The limit is a constraint, not a sentence. const issueRefund = tool({ name: "issue_refund", parameters: z.object({ orderId: z.string().uuid(), reason: z.enum(["damaged", "late_delivery", "wrong_item", "other"]), amountCents: z.number().int().positive(), }), execute: async (args, ctx: AgentContext) => { // 1. Authorisation: does THIS session own THIS order? const order = await orders.findForCustomer(args.orderId, ctx.session.customerId); if (!order) throw new ToolRefusal("ORDER_NOT_OWNED_BY_SESSION"); // 2. Policy as arithmetic, not as prose. const cap = policy.refundCapCents(ctx.session.tier); if (args.amountCents > cap) { return { status: "escalated", queuedFor: "human_review", reason: `Requested ${args.amountCents} exceeds cap ${cap}`, }; } // 3. Idempotency: the key must describe the INTENT, never the attempt. // Anything per-turn or per-retry here defeats the purpose - the retry // arrives with a fresh key and you pay twice. return payments.refund({ orderId: order.id, amountCents: args.amountCents, idempotencyKey: `refund:${order.id}:${args.reason}:${args.amountCents}`, }); }, });
Note what happened to the over-limit case. It doesn't throw a scary error the model might try to work around. It returns a legitimate, structured outcome - escalated - that the agent can explain to the user. Governance and good UX aren't in conflict here. The refusal is a first-class product state.
State machines beat vibes
The second structural fix is to stop treating the conversation as the state. Agents that "remember" where they are in a process via transcript are agents that can be talked out of any step. Identity verification, in particular, cannot live in dialogue history. It has to live in a row in a table.
const TRANSITIONS: Record<CaseState, CaseState[]> = { intake: ["identity_pending"], identity_pending: ["identity_verified", "abandoned"], identity_verified: ["remediation_proposed", "escalated_human"], remediation_proposed: ["executing", "escalated_human"], executing: ["closed", "escalated_human"], closed: [], escalated_human: ["closed"], }; export function guard(current: CaseState, next: CaseState, toolName: string) { if (!TRANSITIONS[current].includes(next)) { throw new ToolRefusal(`ILLEGAL_TRANSITION ${current} -> ${next} via ${toolName}`); } }
Five lines of logic that no amount of "ignore previous instructions" can argue with. Every tool that mutates the world calls guard first. The model can be as confused, jailbroken or hallucinatory as it likes; the process still can't skip verification.
The contrarian bit: your eval suite is not a control either
Here's where I'll disagree with the current consensus. The industry's answer to unreliable prompt adherence has been to build enormous evaluation suites - hundreds of adversarial test cases, red-team prompts, scored rubrics, dashboards showing 97.4% policy compliance.
Evals are valuable. They are not governance. An eval suite tells you the probability of a violation in a sample of scenarios you thought of. A control tells you the violation is impossible. Those are fundamentally different guarantees, and confusing them is how teams end up defending a 97% pass rate to a regulator who only cares about the 3%.
The same goes for the reflex of adding "and never do X" to the prompt after every incident. That loop is why the handbook is 40 pages in the first place. Each addition marginally dilutes everything already there. You are trading a known specific failure for a diffuse increase in general unreliability, and you can't measure the trade. The correct response to a policy violation is almost never a prompt edit. It's a missing validator.
My blunt advice: cap your system prompt. Pick a budget - 800 words is generous for most agents - and defend it. Anything that doesn't fit isn't important enough to be prose, which means it's important enough to be code. That's the same discipline behind cutting half the feature list to find an MVP. Fewer behaviours, actually enforced, beat a hundred behaviours politely requested.
You cannot govern an agent on top of an ungovernable system
This is the part that gets skipped, and it's the expensive part.
Every control above assumes things about the layer underneath. orders.findForCustomer assumes your data model can express ownership. The idempotency key assumes your payment path honours idempotency keys. The escalation path assumes there's an audit trail that lets a human reconstruct what the agent did and why. The state machine assumes you have a durable store of case state rather than inferring it from logs.
If your existing API has endpoints that trust a customer_id in the request body, no handbook saves you - you've just given a very fast, very creative, tireless client access to that weakness. If you have no audit trail, you cannot investigate the incident, which means you cannot close it, which means you cannot ship the next version with confidence. If your write paths aren't idempotent, an agent retry loop becomes a duplicate-charge event.
This is exactly the pattern behind our order of operations: stabilise first, improve second, add AI last. Not because AI is dangerous in the abstract, but because an agent is an amplifier. It amplifies capability and it amplifies every structural weakness in the base with equal enthusiasm. Bolt one onto a shaky foundation and you pay twice - once for the AI, once to rebuild the base underneath it while the AI is live and customers are watching. I went into the infrastructure side of this in more detail in Why Your AI Product Might Fail Before You Write a Line of Code.
The practical readiness test isn't "do we have a policy document." It's four questions. Can every write path identify who authorised it? Can every write path be safely retried? Can we reconstruct any decision after the fact from stored data, not just from chat logs? And can any critical action be blocked by something other than the model's own judgement? Four yeses and you're ready to give an agent real capability. Anything less and you're ready to give it read-only access and a very narrow tool surface, which is a perfectly respectable place to start.
One source of truth, two outputs
The good news is you don't have to maintain policy in two places. Write it once, as data, then generate both the enforcement and the description.
export const REFUND_POLICY = { capsByTier: { standard: 20_000, premium: 50_000 }, requiresIdentityVerified: true, allowedReasons: ["damaged", "late_delivery", "wrong_item", "other"], } as const; // Enforcement (rung 2) export const refundCapCents = (tier: Tier) => REFUND_POLICY.capsByTier[tier]; // Description (rung 4) — derived, never hand-written export const refundPromptFragment = (tier: Tier) => ` Refunds: max ${(refundCapCents(tier) / 100).toFixed(0)} EUR without human review. Identity must be verified first. Above the cap, explain that a specialist will review within one business day. `.trim();
Now the prompt and the validator cannot drift. When finance changes the cap, one constant changes and both the enforcement and the agent's explanation update together. The prompt's job is reduced to what it's genuinely good at: helping the model behave sensibly inside a boundary it cannot cross.
The strategic takeaway
Agents are being sold to your board as configurable. They aren't. You can't configure a probability distribution. You can only constrain it, and constraint is an engineering deliverable with a cost and a timeline, not a document review.
So when the next governance meeting asks how you're controlling the agent, the answer worth giving isn't a page count. It's a list of things the agent is structurally incapable of doing, and the tests that prove it. Everything else - the tone, the phrasing, the empathy, the graceful decline - belongs in the prompt, where it's cheap and where being 95% right is genuinely fine.
Write less handbook. Write more validators. The model was never going to read the handbook anyway.
Book a Free Rescue Call
