DEV Community: Samuel Omisakin

Developers are shipping AI agents without any oversight mechanisms, I'm building a pattern library to fix that

Samuel Omisakin — Thu, 16 Apr 2026 01:32:10 +0000

Six months ago I started paying attention to how developers talk about AI agents they've built and shipped. Something came up again and again.

They'd describe what the agent does. They'd mention a bug or an unexpected output. Then someone would ask: "What stops it from doing that again?" And the answer, more often than not, was some version of: "I haven't really thought about that."

This isn't negligence. It's a tooling gap. There's a lot of writing about AI safety at the research level — papers on alignment, interpretability, RLHF. There's almost nothing at the level of: "Here is a code pattern you can add to your agent today that reduces the chance it does something you didn't intend."

So I'm building that.

The project is called AI Oversight Patterns. It's an open-source catalog of software engineering patterns for maintaining human control over AI agents. Each pattern targets a specific failure mode that shows up in real deployments. Each one comes with a description of when to use it, a Python implementation, and a breakdown of failure modes and tradeoffs.

Three patterns are live right now:

Human Approval Gate — Before executing any irreversible action (send email, delete record, submit payment), the agent generates a plain-language summary of what it's about to do and waits for a human yes/no. The agent proposes. The human decides.

Action Scope Limiter — At startup, you define a whitelist of what the agent is allowed to do. That list is enforced in code, not just in the system prompt. If an action isn't on the list, it can't happen. No amount of clever prompting changes that.

Audit Log Checkpoint — Before every action, the agent writes a structured log entry: what it's about to do, why it chose that action, what alternatives it considered, and how confident it is. Append-only. Useful for debugging, for compliance, and for improving the system over time.

I'm planning 20 patterns total. The remaining 17 cover things like rollback checkpoints, confidence threshold pauses, blast radius limiters, multi-agent scope boundaries, and graceful uncertainty escalation.

The goal is not to make AI agents slower or more annoying to build. It's to give developers a concrete reference for the specific moments where adding a checkpoint is worth the friction.

Repo: https://github.com/Focus1010/ai-oversight-patterns

I'm also running a short survey for people building with LLM APIs. Three questions about whether you've added anything like this to your agents and whether a reference like this would be useful. Responses are helping me prioritize which patterns to build first.

If you're building with LLMs — especially anything agentic — I'd appreciate your input: https://dev.to/focus1010/quick-question-for-people-building-with-llm-apis-3-questions-2-min-3mf9

A few things I'm genuinely curious about from people who've shipped agents:

Have you ever had an agent do something unintended in production? What happened?
Do you think about oversight when you build agents, or does it feel like overkill for your use case?
Is there a failure mode you've personally encountered that isn't covered by the three patterns above?

Happy to discuss any of this in the comments.

Need all hand on this short survey

Samuel Omisakin — Thu, 16 Apr 2026 00:39:22 +0000

Samuel Omisakin

Apr 16

Quick question for people building with LLM APIs (3 questions, 2 min)

#ai #openai #claude #productivity

Comments

2 min read

Quick question for people building with LLM APIs (3 questions, 2 min)

Samuel Omisakin — Thu, 16 Apr 2026 00:37:37 +0000

I'm building an open-source reference called AI Oversight Patterns, a catalog of software patterns for keeping humans in control of AI agents. Things like approval gates before irreversible actions, action whitelists, audit logs, that kind of thing.

Before I go further, I want to make sure I'm solving a real gap and not just something that seems important to me. Three quick questions:

1. Are you currently building or maintaining an application that uses an LLM API (OpenAI, Anthropic, Gemini, etc.)?

Yes, actively building
Yes, it's in production
Was building, now paused
No, but planning to

2. Have you implemented any mechanism specifically to keep humans in control of what your AI agent can do? For example: an approval step before a sensitive action, a whitelist of what the agent is allowed to do, a log of what the agent decided and why.

Yes, I have something like this
No, I haven't thought about it much
No, I thought about it but it felt like overkill
I rely on the model's training to self-limit

3. If a public GitHub repo existed with 20 documented patterns like these, each with a code example and a description of failure modes, would you use it?

Yes, I'd use it as a reference
Maybe, depends on the quality
Probably not, I'd build my own approach
I don't think oversight mechanisms are necessary for my use case

Drop your answers in the comments. Any extra context you have is also welcome, I'm especially curious about the "I thought about it but felt like overkill" responses.

Repo (work in progress): https://github.com/Focus1010/ai-oversight-patterns