Securing Systems That Think

Why it matters now

Defending a system that reasons is a different problem from defending one that executes. The attack surface is no longer only the code path — it is the context window, the tool permissions, and every place untrusted text can reach a decision.

This session brought defenders and builders into the same room, which is rarer than it should be. The disagreements were the useful part.

How the threat model shifts

Classic controls still apply, but three assumptions break the moment a model is in the loop.

What changes

1. 1.Input is code. Anything the model reads can steer it. Retrieved documents are executable in practice, even when they are inert on disk.
2. 2.Permissions compound. An agent inherits every tool you grant it, and chains them in orders you did not anticipate.
3. 3.Identity blurs. Requests arrive on behalf of a user, an agent, or an agent acting for another agent. Audit needs all three.
4. 4.Failure is quiet. A compromised agent usually keeps working. It simply works toward someone else's goal.

Pitfalls to avoid

• Prompt-level defences alone. Instructions are guidance, not a boundary. Enforce limits where the tool call executes.
• Over-broad tool scopes. Grant the narrowest capability that completes the task, scoped per run rather than per agent.
• No kill switch. You need a way to revoke an agent's credentials mid-run without redeploying the service.

Frequently asked questions

Conclusion

The consensus in the room: treat the agent as an untrusted client with useful judgment. Sandbox it accordingly, and the rest of your existing security practice still works.