The most revealing result in my Issue Desk rehearsal came from a pull request that passed its tests. The comparison report also flagged a network call, credential-reading code, and a blocked connection attempt.
I had deliberately planted that behavior in a practice PR. I wanted to see whether the workflow would give a maintainer enough evidence to question a green test result. It did.
That rehearsal brought together the two problems I built Issue Desk to address: the volume of work around an open-source issue, and the care required to decide what happens next.
A busy issue can collect dozens of “assign me” comments, several proposed approaches, and competing implementations. Someone still has to read the plans, check whether they refer to real files, establish a direction, compare the code, and reply to contributors. Each task is manageable on its own. Together, they become a queue that one maintainer has to carry.
For the WeMakeDevs Flip the Switch hackathon, I built Issue Desk: three Claude Code agents working through Switch in Mattermost. They organize the evidence and prepare the replies. The maintainer chooses the direction and approves the configured GitHub write commands.
Giving each agent a specific responsibility
Issue Desk has three agents with deliberately narrow jobs.
desk coordinates the work. It lists busy issues, opens a room for an issue when asked, and reads linked rooms to produce a digest of what arrived, what was decided, and what needs attention.
scout reads the claims. It uses a scorecard to distinguish bare requests from concrete plans, identify near-duplicates, and check whether the files a plan names exist. It recommends promising approaches and drafts a direction comment for review.
bench compares the implementations. It reads the agreed direction, runs competing PRs through an isolated test runner, and reports test results, files outside the agreed scope, and risk flags. After the maintainer chooses a PR, it drafts feedback for the other authors.
The division follows the work: establish what contributors propose, agree on what should be built, then compare the implementations against that agreement. Each agent has a clear source of context and a clear stopping point.
A real issue, copied into a private rehearsal
I rehearsed on a private copy of Flamki/stellarmind#183, with 55 claim comments. The mirror replaced usernames with labels such as contributor-N and made mentions inert. All rehearsal comments went to my sandbox repository; nothing was posted to the original project.
I then added four purpose-built practice PRs: a good implementation, a buggy implementation, one that wandered outside the intended scope, and the test-passing PR with suspicious network and credential behavior. These were fixtures I controlled, so the result demonstrates the workflow on known cases rather than a detection rate on unknown submissions.
The interaction begins in the hub:
@desk sweep
@desk open <issue-number>
desk lists the issues drawing claims and PRs. When asked to open one, it calls Switch's run_template to create an issue room, then link_rooms to connect it to the hub. The template's kickoff wakes scout without another prompt from me.
In the rehearsal, the scorecard classified the 55 comments as 42 bare requests, 12 plans, and one follow-up. It also checked the file paths named in plans against the repository. That gave me something concrete to inspect before choosing an approach.
Those labels are heuristics. A short comment says little about someone's ability, and naming an existing file does not establish that a proposal is correct. The scorecard helps focus the maintainer's reading; its recommendations still require judgment.
scout saves the full evidence as a Switch room document called Scorecard. It saves the proposed direction separately as Direction, which I can revise in chat before anything is posted.
The handoff that makes the workflow useful
The Direction document is the connection between claim triage and PR comparison. It ends with three explicit fields:
Scope: <allowed file patterns>
Install: <dependency installation command, or none>
Tests: <test command, including the expected new tests>
This detail came from a failure during development. The repository's default npm test command would have passed the buggy practice PR because it did not exercise the relevant new test file. A successful command was giving an incomplete answer.
Now scout proposes the scope and commands from repository evidence, asks when something is uncertain, and updates the document when the maintainer changes the direction. bench reads those settings and stops to ask if any are missing.
The shared document keeps the agreement visible. I can correct it once and have the next agent use the revision, rather than reconstructing the context in another session.
When I ask @bench compare, the four practice PRs produce a report like this:
| Practice PR | Test result | Other evidence |
|---|---|---|
| Intended implementation | Passed | No configured risk flags |
| Buggy implementation | Failed | Relevant tests exposed the bug |
| Broader rewrite | Passed | Files outside scope; dependency changes |
| Suspicious implementation | Passed | Network call, credential access, blocked connection attempt |
The table gives the maintainer several dimensions to assess together. A passing suite, an unexpected CI change, and a network attempt deserve different kinds of attention.
The runner copies each PR into a scratch directory and uses Docker for execution. The test phase has no network access or host credentials, restricted resources, and only the scratch copy mounted. Dependency installation happens separately, with network access and install scripts disabled.
That installation step still has limitations: PR-controlled dependency URLs and package-manager configuration can affect what gets fetched or executed. The risk checks are pattern-based, too. An unflagged PR still needs a diff review. The README documents these boundaries.
Where a draft becomes an external action
After reviewing scout's draft, I can ask it to post. The configured gh issue comment command raises a Switch approval card in the same thread. A reply such as A1 1 allows that request once; A1 3 declines it.
I exercised both paths against the sandbox. Approving posted the comment. Declining left GitHub unchanged.
The same pattern applies to bench's contributor feedback. It shows the drafts first, then requests approval for each comment. Choosing a preferred PR does not merge it or close the others; those actions are outside the agents' assigned workflow and have deny rules.
After the comparison, I tell bench which PR I prefer and review its notes for the other authors. Back in the hub, @desk digest brings the linked room's progress into three lists: what came in, what was decided, and what is waiting on me. That closes the loop from a crowded issue to a documented decision and reviewed replies.
Putting the approval beside the draft makes the decision easier to inspect. The evidence, proposed wording, and requested command remain in the conversation where the work happened.
The implementation uses command-prefix rules, however, and the rooms run in auto mode. An unlisted command form can escape those rules. I describe this as a best-effort approval gate and keep the rehearsal in a private repository. The local doctor command checks that the expected rules are present; it cannot prove that every possible command is covered.
What Switch contributed
Switch supplied the coordination layer that made these pieces usable together.
The hub template creates the agents and their initial room. The issue template gives each issue a consistent room, snapshot, instructions, and kickoff. An agent can run that template itself when the maintainer asks to open an issue. Linked rooms let desk gather a digest, while room documents carry the evidence and agreed direction between specialists.
Addressing policies also matter: desk responds to its owner, while scout and bench can respond to the owner and other agents. That supports handoffs inside the issue room without making every agent answer every hub message.
My command-line tools handle GitHub data and test execution. Switch connects those tools to an ongoing conversation with shared context and reviewable actions. That is the part of the build that turns separate commands into a maintainer workflow.
The bugs the rehearsal exposed
The first template run left an agent looking at a literal {repo} placeholder. In Switch Console 0.40.1 on macOS, the version I tested, parameters were substituted into room instructions but not agent instructions. I moved the repository and paths into the room context and made the agent briefs point there. Earlier validation would make this easier to diagnose.
Approval replies had another rough edge: the system processed A1 1, but the agent also received it as a normal message and could respond with confusion. My recording script initially assumed every card would be A1; later cards in the room had different IDs. Both experiences made clearer approval events an obvious improvement.
The largest data-quality lesson came from testing beyond the demo. A read-only run across 289 issue threads and 1,217 comments from two other public Stellar Wave repositories found 568 bot notices—about 47% of the comments. The scorecard was treating notices such as “application accepted” and “issue completed” as contributor claims.
I added filtering for comments authored by GitHub apps in both scoring and mirroring. Bots using ordinary user accounts can still slip through, and the remaining classifications have not been manually validated. The broader run exposed a mistake that the demo thread, which had no bots, could never reveal.
What I would build next
The clearest performance improvement is dependency caching keyed to the lockfile. Comparing four PRs took roughly 11–14 minutes in rehearsals, with much of that spent installing dependencies. A digest also currently requires a prompt; scheduled sweeps would make it more useful as a regular maintenance routine.
Before expanding beyond the practice setup, I would strengthen write enforcement with repository-limited credentials and a command-aware gate. I would also manually evaluate the claim classifications on a labeled sample. The current build was tested with one maintainer on a local Switch server and bundled Mattermost; a hosted team setup remains untested.
Try the workflow
The project is open source at JemIIahh/issue-desk. Start with Switch Console, Docker, Claude Code, an authenticated GitHub CLI, and uv:
git clone https://github.com/JemIIahh/issue-desk
cd issue-desk
uv sync
uv run issue-desk doctor
Follow the README to create a private practice repository, mirror the issue, seed the four PRs, and import the two Switch templates. From there, @desk sweep begins the rehearsal.
The moment I would watch closely is the handoff from scout to bench: the direction I reviewed becomes the scope used to compare the code. By the time contributor replies reach an approval card, the room contains the evidence behind them.
Build and AI disclosure
The underlying triage-kit library—including GitHub reading, claim scoring, the isolated test runner, mirroring, and the practice fixtures—was written before the hackathon window. The initial design and a short Switch feasibility spike also predate the window. The Issue Desk templates, agent configuration and instructions, approval rules, CLI wrapper, tests, and project documentation were developed during the window, as detailed in the repository.
Claude Code assisted with implementation from a written design and plan, with tests written first for the command-line code. The agents running inside Issue Desk are also Claude Code agents. I made the product decisions, set the rules, and authorized outward actions. This article was revised with AI assistance using the project documentation and rehearsal notes.
The demo recording uses scripted maintainer typing and approval replies for repeatability. Agent responses and sandbox GitHub comments are real. The recording is a workflow demonstration, not a measurement of human decision time.
Top comments (0)