Six months ago I started paying attention to how developers talk about AI agents they've built and shipped. Something came up again and again.
They'd describe what the agent does. They'd mention a bug or an unexpected output. Then someone would ask: "What stops it from doing that again?" And the answer, more often than not, was some version of: "I haven't really thought about that."
This isn't negligence. It's a tooling gap. There's a lot of writing about AI safety at the research level — papers on alignment, interpretability, RLHF. There's almost nothing at the level of: "Here is a code pattern you can add to your agent today that reduces the chance it does something you didn't intend."
So I'm building that.
The project is called AI Oversight Patterns. It's an open-source catalog of software engineering patterns for maintaining human control over AI agents. Each pattern targets a specific failure mode that shows up in real deployments. Each one comes with a description of when to use it, a Python implementation, and a breakdown of failure modes and tradeoffs.
Three patterns are live right now:
Human Approval Gate — Before executing any irreversible action (send email, delete record, submit payment), the agent generates a plain-language summary of what it's about to do and waits for a human yes/no. The agent proposes. The human decides.
Action Scope Limiter — At startup, you define a whitelist of what the agent is allowed to do. That list is enforced in code, not just in the system prompt. If an action isn't on the list, it can't happen. No amount of clever prompting changes that.
Audit Log Checkpoint — Before every action, the agent writes a structured log entry: what it's about to do, why it chose that action, what alternatives it considered, and how confident it is. Append-only. Useful for debugging, for compliance, and for improving the system over time.
I'm planning 20 patterns total. The remaining 17 cover things like rollback checkpoints, confidence threshold pauses, blast radius limiters, multi-agent scope boundaries, and graceful uncertainty escalation.
The goal is not to make AI agents slower or more annoying to build. It's to give developers a concrete reference for the specific moments where adding a checkpoint is worth the friction.
Repo: https://github.com/Focus1010/ai-oversight-patterns
I'm also running a short survey for people building with LLM APIs. Three questions about whether you've added anything like this to your agents and whether a reference like this would be useful. Responses are helping me prioritize which patterns to build first.
If you're building with LLMs — especially anything agentic — I'd appreciate your input: https://dev.to/focus1010/quick-question-for-people-building-with-llm-apis-3-questions-2-min-3mf9
A few things I'm genuinely curious about from people who've shipped agents:
- Have you ever had an agent do something unintended in production? What happened?
- Do you think about oversight when you build agents, or does it feel like overkill for your use case?
- Is there a failure mode you've personally encountered that isn't covered by the three patterns above?
Happy to discuss any of this in the comments.
Top comments (0)