At Sharkly, people rarely fix routine bugs by hand. Most bugs go directly to an AI agent.
Sharkly is the product our team is building, and we use it to run our own product and engineering workflow.
Our dedicated agent for bug fixes and small engineering tasks is called Sherlock. It can discuss an issue, inspect the relevant repository, trace the implementation, run tests in a real development environment, and verify UI changes in a browser. When visual behavior changes, it also captures screenshots as evidence.
According to our internal data, Sherlock completed 13% of all Sharkly tasks over a recent 15-day period. Of the latest 100 tasks assigned to it, 90 were completed successfully.
Sherlock runs on a shared development Computer using the oh-my-pi Runtime and the GPT 6 Astra model. We assign it dozens of bugs and small improvements every day. More importantly, we trust it to handle them without constant supervision.
That trust does not come from assuming the model is always right. It comes from putting the agent inside a strict, inspectable workflow.
A Real Bug Fix from Start to Finish
Consider a recent issue in Sharkly. The temporary-directory list in the Clear Build Cache dialog could grow beyond the available space, but the list was not scrollable.
Oliver discovered the problem and posted it in our internal Slack channel. Emma then mentioned the Sharkly Task Assistant and asked it to turn Oliver’s report into a structured task.
Emma assigned the new task to Sherlock. Sherlock spent 14 minutes reproducing the issue, tracing the implementation, and locating the root cause.
Sherlock then presented three implementation decisions that required human input before development could begin.
Emma selected the recommended options and sent the explicit approval token:
1A 2A 3A [START DEVELOPMENT]
Only then did Sherlock begin changing the product code.
After another 37 minutes, it reported that implementation and verification were complete. It provided three concrete items for review:
- A code pull request
- A preview URL
- Screenshots of the corrected behavior
Emma reviewed the change and its evidence. Once the fix was confirmed, Emma sent the second approval token:
[SUBMIT FOR TESTING]
That token authorized Sherlock to merge the change into the development branch and move the task into the automated testing workflow.
Emma made decisions at only two points:
- Approve the proposed solution and allow product code changes.
- Approve the completed change and allow it to enter the testing pipeline.
Everything between those gates was handled autonomously. There was no need for a long back-and-forth conversation about every file, command, or implementation detail.
That is how one engineer can supervise dozens of small fixes without becoming the bottleneck.
The Platform Behind Sherlock
Sharkly is the team-level project management and execution layer for AI coding agents. People plan work in Tasks and Sprints, assign it to Agents or Crews, and run those agents safely in parallel on their own Computers.
Sharkly separates an Agent from its Runtime. The Agent carries its identity, instructions, Skills, and operating rules. The Runtime is the coding tool that performs the work on a selected Computer, such as Claude Code, Codex, or another compatible tool. This separation lets a team choose its execution tools while keeping task context, approvals, run history, and evidence in one shared workspace.
Sharkly also connects Agents to the team’s operating process. Automations can start work when a task changes status, while explicit approval points keep high-impact actions under human control. Agents research, execute, test, and report. People set direction, grant authority, and accept the result.
For Sherlock, Sharkly provides the task, conversation, execution history, and approval gates. A shared development Computer provides the repository, dependencies, browser, and other tools, while the configured Runtime executes Sherlock’s instructions.
Sherlock’s 10-Step Workflow
Sherlock follows a fixed workflow with ten stages. It cannot skip ahead simply because a solution looks obvious.
1. Initialize the Task
Sherlock checks the task ID, title, description, labels, and priority. It confirms that it is running inside the task workspace provided by Sharkly and selects the correct baseline branch.
An urgent task might use main. A high-priority task might use staging. Other tasks normally follow the repository’s default integration branch.
2. Diagnose the Problem
The diagnosis stage is limited to reproducing the issue, collecting evidence, and identifying the root cause. Sherlock is not allowed to modify product code during this stage, even when the fix appears straightforward.
Its findings must be written to diagnosis.md. Every later proposal must be traceable back to evidence recorded in that file.
3. Draft a Solution
After identifying the root cause, Sherlock creates a proposed solution. It explains what should change, which behaviors must remain unchanged, and which unresolved decisions require human input.
If the change affects user-facing copy, Sherlock first reads the repository’s terminology and localization rules. It then prepares several candidate phrasings instead of inventing text without context.
The proposal is stored in idea.md and refined as feedback arrives.
4. Wait for Development Approval
This is the first human gate.
Sherlock may understand the problem and have a complete solution, but it still does not have permission to change product code. It must wait for the exact [START DEVELOPMENT] token.
Messages such as “continue,” “looks good,” or “go ahead” do not unlock the next stage. Natural language is too easy to misinterpret when authorization matters.
5. Explore and Implement
After approval, Sherlock performs a focused implementation exploration. It reads the relevant files, traces call sites, checks existing tests, and learns the repository’s established conventions.
The exploration produces findings.md and plan.md. Sherlock then implements the approved solution. Any necessary departure from the plan is documented in plan-delta.md.
6. Run a Clean-Room Review
Finishing the code does not mean the task is complete.
Sherlock first synchronizes with the authoritative baseline branch. It then performs a clean-room review, temporarily setting aside the assumptions it used during implementation and evaluating the change as an independent reviewer would.
The review asks questions such as:
- Does the change address the actual root cause?
- Have all affected call sites been migrated?
- Do the tests cover the meaningful risks?
- Does any obsolete code path remain?
- Did the implementation introduce unrelated changes?
7. Patch Blocking Issues
If the review finds a blocking problem, Sherlock enters a patch stage. It fixes the issue, synchronizes with the baseline again, and repeats the clean-room review.
This loop can run at most six times. If blocking issues remain after the sixth review, the automated workflow stops and requests human intervention.
The limit prevents the agent from running indefinitely in a cycle where it writes, reviews, and repeatedly repairs its own code.
8. Verify Real Behavior
After the review passes, Sherlock runs the repository’s declared test suite. It also verifies the behavior that actually changed.
A successful test command is only one piece of evidence. UI changes require proof from the real interface. Backend changes need evidence from APIs, logs, queries, or executed commands that reflect runtime behavior.
The verification standard is simple: show that the feature works, not merely that a command exited with status code zero.
9. Wait for Testing Approval
Sherlock assembles a testing package containing the implementation summary, code diff, automated decisions, test results, preview links, and visual or runtime evidence.
It then stops at the second human gate and waits for [SUBMIT FOR TESTING].
This token authorizes Sherlock to create the final commit, push the code, perform the target-branch operation, and update the Sharkly task status.
10. Submit the Change for Testing
Finally, Sherlock merges the change into the development branch and moves the task to Ready for Testing. A testing agent can then continue the workflow.
The Core Design: A State Machine, Not a Persona
Sherlock’s instructions are not a conventional role prompt that says, “You are an expert software engineer.”
They define an engineered, single-agent defect-resolution state machine. The result is closer to an executable development procedure than a chatbot personality.
Sherlock can work independently for most of the task, but it cannot decide when to start modifying product code. It also cannot decide when to push or merge the result. Those transitions belong to people.
Every stage is recorded in a persistent state.json file. Sherlock must read and validate that file before taking action, which makes the current state explicit and recoverable.
The design has some important parts.
1. A Narrow Role and Clear Boundaries
Sherlock is allowed to work only on the Sharkly task currently assigned to it.
It may not search for unrelated work, create replacement tasks, delegate the task to another agent, or perform actions that belong to a later stage.
These constraints reduce ambiguity. The agent does not need to decide what kind of work it should do. It only needs to complete the current task within the permitted workflow.
2. An Explicit Order of Authority
When information conflicts, Sherlock follows a defined priority order:
- Agent instructions and explicit human decisions
- Repository-level guidance such as
AGENTS.md, documentation, and branch policies - The task description and approved proposal
- The source code and observed runtime evidence
The important principle is that repository facts override generic model knowledge.
A model may remember how a framework is usually configured, but that does not prove that the current repository follows the same convention. If the repository declares its own test commands or branch policy, Sherlock must follow them.
When documentation, code, and runtime behavior disagree, Sherlock must expose the conflict. It cannot silently choose the most convenient interpretation.
3. Persistent State
After initialization, state.json is the single source of truth for the workflow’s current stage.
The file records:
- Task information
- The active stage
- Pending human approvals
- Approved decisions
- Stage artifacts
- Verification status
- Transition history
Every valid state transition records the previous stage, the next stage, the reason for the transition, and the relevant files and hashes. A transition that requires approval also stores the approval token and the exact version of the proposal that was approved.
This persistence allows Sherlock to recover after an interrupted session. A new run does not guess what happened from chat history. It reloads the state and stage artifacts from disk.
If state.json is missing, corrupted, or internally inconsistent, Sherlock stops and reports a state error. It cannot manually advance the stage or reconstruct a missing approval from an informal chat message.
4. Two Human Approval Gates
The two approval tokens control two different categories of high-impact action.
[START DEVELOPMENT] authorizes writes to product code. It confirms that a person has reviewed and accepted the proposed solution.
[SUBMIT FOR TESTING] authorizes commits, pushes, target-branch operations, and task-status changes. It confirms that a person has reviewed the implementation and its evidence.
This divides risk into two parts:
- The risk of changing the code
- The risk of sending that change into the team’s shared delivery process
Sherlock can complete a large amount of autonomous work between the gates, but it cannot cross either boundary by itself.
5. Required Stage Artifacts
Each stage must produce a concrete artifact:
-
diagnosis.mdexplains how the issue was reproduced, identifies the root cause, and records the supporting evidence. -
idea.mddescribes the proposed solution and the choices approved by the user. -
findings.mdrecords implementation exploration. -
plan.mdexplains what will change and how the change will be verified. -
plan-delta.mddocuments deviations from the approved plan. -
review-report.mdcontains the clean-room review results. -
ship-summary.mdcontains the final delivery and verification information.
Sherlock cannot claim that a stage is complete when its required artifact is missing. A chat message saying “diagnosis complete” has no authority by itself.
The state transition is valid only when both the state record and the required artifact exist.
6. Stage-Specific Skills
Sherlock uses seven specialized Skills in addition to its main instructions:
lattice-diagnoselattice-idealattice-explorelattice-implementlattice-reviewlattice-verifylattice-deliver
The main instruction set controls stages, permissions, and transition rules. Each Skill defines how work should be performed inside a specific stage.
Before entering a stage, Sherlock must read the relevant Skill in full and record that action in the state history. The resulting artifact must also demonstrate that the Skill was followed. Recording “Skill read” without producing compliant work is not enough.
This separation keeps the central state machine focused while allowing each stage to have detailed, testable operating rules.
7. Evidence-Based Verification and Delivery
The verification stage has a deliberately high standard of evidence.
Passing tests does not prove that the complete feature works in its real environment. A user-visible change must be checked in the actual interface. Backend changes must be supported by API responses, logs, queries, or command output.
Before submission, Sherlock pushes its working branch to a dedicated remote review branch and provides a comparison link. The reviewer can inspect both the code diff and the associated verification evidence.
8. A Consistent Response Format
Every Sherlock response must include:
- The work that was actually completed
- The current stage
- The next action required from a person
Each response also ends with a status footer containing the task, current stage, pending approval token, review count, and artifact paths.
This makes it difficult for a vague progress update to hide the true workflow state.
What We Learned
Building Sherlock required several architectural choices and many iterations. A few lessons had a much larger impact than we expected.
One Agent or Multiple Agents?
Sharkly supports multi-agent collaboration through Crews, so we seriously considered splitting the workflow across several specialized agents.
We ultimately chose a single-agent state machine.
For routine bug fixing, today’s models are capable enough to handle diagnosis, implementation, review, and verification within one controlled context. A multi-agent design would add coordination overhead, latency, and cost without consistently improving the result.
Crews are more useful when a process has several clearly separated specialties and the steps are only loosely coupled. Bug fixing is often the opposite. The diagnosis, implementation, and verification all depend heavily on the same repository context and evidence.
Our practical rule is to start with one agent plus strong Skills. We introduce multiple agents only when the work has genuinely separable responsibilities.
Choosing a Runtime
The key architectural choice was to keep Sherlock separate from the Runtime that executes its work. We tested Claude Code, Codex CLI, OpenCode, and oh-my-pi against the same real tasks.
For Sherlock’s unattended state-machine workflow, we chose oh-my-pi because it provided direct control over instructions and tools, replayable and recoverable sessions, and lightweight concurrency on a shared development Computer.
This was a workload-specific decision, not a universal ranking. An interactive coding assistant or a different Agent may be better served by another Runtime, while Sharkly keeps the surrounding task, approval, and evidence workflow consistent.
Lessons from Iterating on the Instructions
Sherlock’s workflow did not emerge fully formed. We found many failure modes before it became reliable.
Use Skills That Have Already Been Proven
An agent’s reliability is limited by the quality of its operating procedures.
An untested instruction can produce a different path on every run, which makes failures difficult to diagnose. We converted each stage into a Skill that had been exercised repeatedly. Sherlock’s job is to follow those Skills precisely.
New Skills first run on low-risk tasks. They enter the production workflow only after their behavior becomes predictable.
Keep Human Gates to a Minimum
Early versions of Sherlock had seven human approval gates. The agent stopped every few steps, repeatedly interrupting the person supervising it.
We gradually reduced the workflow to two gates: permission to change product code and permission to submit the result for testing.
Fewer gates made the remaining gates more meaningful. People now make decisions at a small number of information-rich moments instead of approving fragments of work throughout the process.
Never Treat Casual Language as Authorization
Phrases such as “continue,” “fine by me,” and “do it” cannot advance the state machine.
Natural language changes meaning with context, and an agent may mistake a casual acknowledgment for approval. Explicit tokens make authorization unambiguous. They also create an auditable record of who approved which version of a proposal.
Put Screenshots Where the Decision Happens
For UI changes, screenshots are mandatory. They must also appear directly in the conversation, not merely as file paths that someone has to locate manually.
The evidence used to make a decision should be visible at the decision point. When a reviewer can compare the behavior before and after the fix immediately, approval becomes faster and more reliable.
Evidence hidden several directories away is effectively missing evidence.
Trust Comes from the Process
Why should we trust an AI agent to fix bugs?
Sherlock’s answer is not that the model is exceptionally smart. The answer is that the process is exceptionally strict.
The state machine controls where the agent can go. Required artifacts preserve evidence of what it did. Two human gates protect the actions with the greatest impact. Clean-room review and real-environment verification provide a quality floor. Every stage is inspectable, interruptible, and recoverable.
The workflow also changes the role of the engineer. People no longer need to watch every command the agent runs. They focus on two high-value decisions: whether the proposed solution is right and whether the completed change is ready for testing.
Human attention moves from supervising activity to exercising judgment. That is what makes it practical for one engineer to oversee dozens of small tasks at the same time.
Bug fixing is only the beginning. We are applying the same principles, including narrow boundaries, persistent state, human approval gates, and evidence chains, to feature development, testing, and release workflows.
We will share more of those AI-native engineering practices as they mature.
Top comments (2)
The 90/100 is the number that will get quoted, and it is conditional on the routing policy that sent those tasks to Sherlock. Since it takes 13% of tasks, the end-to-end figure is closer to 12% of all bugs fixed without a human - still good, but a different claim, and the one a buyer is actually paying for.
Two other splits would make the 90% legible. Who signs off "completed" - the agent's own passing tests, or a human reading the diff? Those are very different numbers. And the reopen rate: of the 90, how many are back within 30 days. A fix that does not hold is not a completed task, it is a delayed one.
Really impressive work!