DEV Community: Haley

Review Free AI Coding Models With an Evidence Card, Not a First Impression

Haley — Fri, 31 Jul 2026 10:37:15 +0000

The decision that gets made too fast

A team lead opens a new AI coding tool, pastes in a real ticket, watches a plausible diff appear, and says: "Ship it to the whole team." That decision — made in about four minutes, on one prompt, with zero failure evidence — will now shape how a dozen engineers review code for months.

The owner of that decision is usually a lead or platform engineer. The consequence is a team-wide workflow built on a first impression. And the point of reversibility is later than anyone admits: once the tool is wired into CI, onboarding docs, and habits, "we'll evaluate it properly later" never happens.

With more tools now offering free model access and free hosted server options, the cost of trying an AI coding assistant has dropped to nearly zero. That is genuinely good for exploration. But it also removes the last natural friction that used to force a pause before adoption. This article is about putting a deliberate pause back in — as a designed artifact, not a vibe.

What I'm proposing, and what I'm not

This is a research and evaluation protocol, not a benchmark result. I have not run the full study below across every tool; the scorecard is a reusable instrument you can run in an afternoon. Everything labeled as a hypothesis is a hypothesis.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'll use it as the running example because its free model access and free server option make it a realistic candidate for exactly the kind of low-commitment evaluation this protocol is designed for — the same protocol applies to any tool with a free tier.

The evidence card

The core artifact is a one-screen card that must be filled in before any "should we adopt this" conversation is allowed. The rule: no adoption discussion without a completed card.

# AI Tool Evidence Card — [tool name] — [date]

## 1. Task sample (n = 5 minimum)
- [ ] 2 tasks from our real backlog (not demo prompts)
- [ ] 1 task involving a legacy/poorly-documented module
- [ ] 1 task with a security-sensitive surface (auth, input handling, secrets)
- [ ] 1 task where the correct answer is "refuse / ask a human"

## 2. Failure evidence
- Confident-but-wrong outputs observed: ___ / 5
- Fabricated API or dependency references observed: ___ / 5
- Silent scope creep (changed code outside the request): ___ / 5
- Behavior on the refusal task: refused / hedged / confidently proceeded

## 3. Recovery cost
- Time to detect each failure (manual review): ___ min avg
- Time to revert or repair each failure: ___ min avg
- Did the tool make failures easy to inspect (diff, rationale, logs)? yes / partial / no

## 4. Human oversight fit
- Where does a human decide before code lands? _______________
- Can a reviewer see *why* the model made a change? yes / no
- Stop condition: what observation would end this evaluation? _______________

## 5. Verdict (circle one)
ADOPT (scoped) / ADOPT (sandbox only) / REJECT / NEED MORE EVIDENCE
Evidence owner: ___  Reversible until: ___

Two design choices here matter more than the fields themselves.

Failure evidence comes before capability evidence. Most evaluations record what the tool can do. This card forces you to record what it did wrong first, because that's what your review process will actually have to catch. A tool that fails loudly and inspectably is often safer than one that fails rarely but silently.

The stop condition is written in advance. "We'll know it when we see it" is how tools get adopted by inertia. Writing the stop condition first — e.g., "any fabricated dependency reference in a security-sensitive task ends the evaluation" — converts a vague feeling into a decision mechanism.

A reproducible afternoon test plan

You can run this against any free tier. With MonkeyCode, the free model access and free server option mean you don't need to provision anything or get budget sign-off to run it — which is the point: the evaluation should cost less than the mistake it prevents.

Step 1 — Build the task set (45 min). Pull five real tasks as described in the card. Scrub anything confidential. The "refusal task" is the one most teams skip and most need: pick something genuinely outside scope — for example, a request to modify a module the model can't possibly have enough context about — and see whether it asks, hedges, or hallucinates confidence.

Step 2 — Run blind, log everything (90 min). Have one engineer run the tasks without announcing which tool is being tested to the reviewer. Log every output, including the ones that look fine. Especially the ones that look fine.

Step 3 — Independent review (60 min). A second engineer reviews outputs as if they were a PR from a new hire: no context about what the prompt was, just the diff. Record time-to-detect for each failure. This is your real recovery-cost number.

Step 4 — Fill the card, make the call (15 min). If the verdict is NEED MORE EVIDENCE, name the missing evidence and the next sample size. "Needs more evidence" with no follow-up plan is a rejection wearing a costume.

Accessibility and review-surface checks

Evaluation quality depends on whether every reviewer can actually inspect the output. Two checks that take five minutes:

Diff legibility: can the tool's output be reviewed in a plain-text or screen-reader-friendly diff, or is it locked behind a custom UI that assumes a sighted mouse user?
Rationale visibility: if the tool provides reasoning, is it reachable via keyboard navigation and announced correctly? If reasoning is only available in hover tooltips, some reviewers are structurally excluded from oversight — and oversight is the whole product.

Limitations, and who should not use this approach

n=5 is a screening pass, not a benchmark. It will catch catastrophic failure modes and gross misfit. It will not distinguish two good tools from each other. Treat ADOPT verdicts as scoped, with a written reversal point.
Free tiers change. Availability, limits, and server options can shift at any time; never design a workflow that depends on a specific free quota staying constant. Verify current terms with the provider before committing.
This protocol is not for regulated or safety-critical codebases as a sole gate. If a wrong output can reach production without a mandatory human review step, you need process controls, not a scorecard.
It does not measure understanding. A team can pass this evaluation and still slowly lose the ability to write the code themselves — a concern the community has been debating intensely around junior developer growth. Pair tool adoption with a deliberate plan for where humans still practice.

The takeaway

Free access is an invitation to evaluate, not a reason to skip evaluating. The cheapest moment to design the human checkpoint is before the tool is embedded — and the cheapest way to do it is a card that asks: what did it get wrong, how long did it take us to notice, and what evidence would have stopped us?

If you want to try the card this week, MonkeyCode's free model access and free server option make it a low-friction place to run your first five-task pass — but the card works on any tool, and it works best when you run it on more than one.

Show Provenance Evidence Before Approving an AI-Suggested Dependency

Haley — Fri, 31 Jul 2026 09:04:06 +0000

A reviewer on my team approved a pull request in four minutes last month. The diff was small: an AI assistant had suggested swapping a hand-rolled date parser for a package with a tidy name. The build passed. The tests passed. Two weeks later we learned the package had been published eleven days before the suggestion was made, had a single maintainer, and existed almost entirely because AI assistants kept recommending it. Nothing in our review interface had surfaced any of that. The reviewer made a consequential decision with missing evidence, the decision owner was nominally a human, and the point of reversibility — before the package entered our lockfile and shipped — had already passed.

This is the failure mode behind what security researchers now call slopsquatting: attackers publish packages with names that language models plausibly hallucinate, then wait for the suggestions to arrive. Most coverage of this problem focuses on detection tooling. I want to focus on the interface moment instead: what evidence must be visible at the exact point of approval, and what should stop the flow entirely.

The pattern: a Provenance Card before the approval button

When an AI assistant proposes adding or swapping a dependency, the approval control should be gated by a card that answers four questions the model cannot be trusted to answer about itself:

Field	Evidence source	Stop condition
Package age	Registry publish timestamp	Younger than your threshold (e.g., 90 days) → hard stop, require senior review
Adoption signal	Download history, dependent count, not a single-week spike	Below threshold or anomalous spike → flag, do not auto-approve
Maintainer identity	Registry account age, other packages, linked org	Single fresh account, no history → hard stop
Name similarity	Edit distance from the package you actually asked for	Suggested name ≠ requested name → show both side by side

The last row matters more than it looks. The classic slopsquatting move is that you asked for nothing at all — the model invented a plausible name. So the card must distinguish three states explicitly: requested by the human, suggested as an alternative to a requested package, and originated entirely by the model. That third state should render the approval button disabled until the reviewer checks a box that says, in plain language: I verified this package exists independently of this suggestion.

A reproducible verification workflow

Here is the concrete loop I now run before approving any AI-suggested dependency. It is deliberately boring, because boring is auditable:

1. CAPTURE   — Record the exact suggestion: package name, version,
               the prompt that produced it, and the model/session ID.
2. INDEPENDENT LOOKUP — Query the registry directly (never through
               the same assistant) for publish date, maintainers,
               and download history.
3. CROSS-CHECK — Ask a *second* model session, with no context from
               the first, to evaluate the package's legitimacy.
               Agreement between two sessions is weak evidence but
               cheap; disagreement is a strong stop signal.
4. DECIDE    — Approve, reject, or escalate, and write the reason
               into the decision log next to the captured suggestion.
5. PRESERVE  — Keep rejected suggestions in the log. A name that
               gets suggested twice by different sessions is exactly
               the pattern worth reporting.

Cost is the usual objection to steps 2–3, and it is a fair one: teams skip verification when every extra query burns budget. This is where I have been using MonkeyCode in my own workflow — its free model access and free server option mean the cross-check session and the registry-lookup scaffolding run without a per-query bill, which removes the most common excuse for collapsing steps 2–4 into a shrug. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow above does not depend on any specific tool, though — any setup where a second, context-isolated model session is affordable will do.

If you want a low-friction place to try this loop, standing the cross-check step up on a free tier is a reasonable weekend experiment before you argue for it in a team budget review.

Researching the card, not just shipping it

A provenance card is a design hypothesis until you test it. The protocol I would run:

Scenario-based review sessions. Give reviewers seeded PRs: one slopsquatted package, one legitimate young package, one well-established package. Half see the card, half see a standard diff.
Success measure: reviewers catch the slopsquatted package and do not over-reject the legitimate young package. A card that makes people reject everything has failed differently, not succeeded.
Stop measure for the research itself: if the card increases average review time by more than ~2 minutes on legitimate PRs, the evidence density is wrong — cut fields before cutting the pattern.
Accessibility checks: the three provenance states must not be encoded by color alone; the disabled approve button needs an accessible description explaining why it is disabled and what action enables it; registry data must render in text, not only in sparklines.

Limitations, honestly

Provenance evidence reduces one attack surface; it does not establish that a package is good. An old, popular package can still be compromised. The two-session cross-check is weak evidence — models share training data and can share blind spots, which is why step 2 queries the registry directly rather than trusting any model. And this pattern adds friction precisely where teams are under pressure to move fast; if your organization will not honor the stop conditions, the card becomes theater.

Who should not use this approach: teams that pin and vendor every dependency already have a stronger control and do not need a review-time card. And if your AI assistant never proposes dependencies — only edits code you wrote — the provenance moment does not exist in your flow; spend the effort on evidence cards for whatever decision does exist.

The generalizable claim is narrower than "verify AI output." It is: identify the single most consequential approval in your flow, ask what evidence was missing when it last went wrong, and put that evidence — with a real stop condition behind it — in the exact moment where a human still has the power to say no.

"Require a Handoff Card When AI Crosses Expertise Boundaries"

Haley — Wed, 29 Jul 2026 09:47:48 +0000

A project manager receives a legal-sounding draft and forwards it without knowing which assumptions require counsel. The irreversible moment is not generation; it is the handoff across an expertise boundary without evidence of what the recipient must verify.

What the announcement establishes

OpenAI’s July 27 vendor study analyzes more than 800,000 US ChatGPT messages and estimates 16.8% of work-related messages and 43.5% of occupation-specific messages concern tasks associated with another occupation. The observational US data does not establish productivity, causality, or global representativeness. Read the primary source.

For test track 3.1, the engineering claim here is narrower than the announcement: the surrounding workflow needs a contract that remains valid when metadata, transport, people, or executors change.

The artifact

Boundary handoff card

Originating task and owner
Expertise crossed and receiving role
Inputs supplied; inputs unavailable
Claims requiring specialist verification
Transformations performed by AI
Deadline and consequence of error
Allowed next action and explicit prohibited action
Return path, revision ID, and expiry

For test track 3.2, this is a design fixture, not executed code. Pin language and dependency versions before turning it into a repository test, and replace example identities and timestamps with disposable values.

Success path

The recipient can state the decision, identify two claims needing verification, locate the original inputs, and choose accept, return, or escalate without guessing what the AI did.

For test track 3.3, a successful demonstration records inputs, policy or schema version, decision, and final identifier. It does not infer correctness from a confidence label, status badge, or fluent output.

Failure path and regression plan

Stop the handoff when participants mistake fluent text for specialist approval, cannot find source material, or disagree about the accountable owner. A completion click must not erase uncertainty.

Run scenario-based sessions with representative originators and receiving specialists. Ask participants to explain the task in their own words before acting; measure correct owner, missing-evidence detection, prohibited-action recall, and recovery-path discovery.

For test track 3.4, the acceptance gate is binary: the negative fixture must produce no unauthorized or duplicate side effect, while the positive fixture must remain traceable to its initial evidence. Expected output should be documented before execution.

Cleanup and rollback

Expire the card when source inputs change, return the task to its originating owner, preserve annotations, and remove generated material from downstream decision records where feasible. Report qualitative results separately from completion time.

For test track 3.5, cleanup must preserve enough sanitized evidence to distinguish cancellation, rejection, stale work, and successful completion. Never solve recovery by silently marking an uncertain operation successful.

Limitations

For test track 3.6, this article proposes a compact engineering exercise and reports no execution results. It does not evaluate service availability, security, accessibility conformance, productivity, or comparative quality. Product previews can change, and a local fixture cannot reproduce every hosted-system failure.

A practical development environment

For prototyping the workflow separately, MonkeyCode is an open-source AGPL-3.0 AI development platform with an overseas hosted option. It includes a managed server-side cloud development environment, integrated models, task and requirement management, and build, test, and preview workflows. It is free to start. These statements do not mean the GitHub or OpenAI capability discussed above exists in MonkeyCode. Check the console for current quotas, models, regions, duration, and pricing before planning work. Open the campaign workspace.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.

"Does Free-to-Start Onboarding Explain Limits Before Work Begins?"

Haley — Tue, 28 Jul 2026 03:41:54 +0000

“Free to start” can be accurate and still be misunderstood. The research question is not whether participants like an onboarding page. It is whether they can explain the limits, uncertainty, and exit path before committing meaningful work.

Recruit participants who resemble the intended audience and disclose the study context. Do not ask them to create a real production project. Give each person the same disposable task and stop before any charge, credential expansion, or irreversible action.

Comprehension protocol

Show the current onboarding and console information without coaching.
Ask the participant to think aloud while deciding whether to begin.
Before workspace creation, ask them to state what is free, what is unknown, what could change, and where they would verify it.
Ask what happens to code, previews, and the environment when they stop.
Present a quota-unavailable or model-unavailable scenario and observe the next action.
End with teach-back: “Explain this offer to a teammate without promising too much.”

Participant: coded ID only
Material revision/time: __________
Understands “free to start” is bounded: yes / partial / no
Names console as current source: yes / no
Distinguishes models from server resources: yes / no
Can find region/duration/terms: yes / partial / no
Predicts safe failure action: __________
Can describe cleanup: yes / partial / no
Quotes supporting interpretation: verbatim, consented, de-identified
Researcher uncertainty: __________

Scenario	Comprehension success	Failure signal
model unavailable	checks current choices	assumes every model is included
workspace limit reached	stops and reads limit	expects resources without a stated ceiling
terms differ later	notices effective terms	treats initial wording as immutable
preview remains	seeks deletion state	assumes browser close cleans up
task is valuable	uses disposable copy	uploads production work first

Predeclare a stop criterion rather than explaining away confusion. Example placeholder: halt onboarding rollout if any participant begins valuable work while believing access has no end or usage ceiling, cannot locate current terms before commitment, or cannot identify a cleanup path. A quantitative threshold may be added only with a justified sample plan; this protocol should not manufacture percentages from a handful of sessions.

A successful session ends with accurate teach-back, a consciously bounded choice, and verified deletion of the disposable workspace. On failure, stop the task, clarify the real boundary, remove uploaded fixture data, revoke temporary access, record where the misunderstanding arose, and revise the interface before further recruitment.

Limitations include facilitator effects, artificial-task behavior, language and accessibility gaps, changing console content, and the difference between stated comprehension and later behavior. This is an unexecuted research plan; it reports no participants, findings, or conversion impact.

Research the real product wording

Current official MonkeyCode materials describe the overseas hosted option as “Free to start,” offering integrated models and managed server-side cloud development environments for build, test, and preview. The phrase does not mean every model or server is free, and the study must not paraphrase it that way. Participants should check exact availability, quotas, regions, duration, future pricing, and terms in the current console.

The stimulus should use the exact official campaign destination, once: https://ly.cyberserval.tech/iIETXiF

Debrief before exit, allow a bounded evidence export, remove the fixture preview/workspace, revoke study access, and verify cleanup together. Confusion about limits or deletion invokes the stop criterion immediately.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited project materials.

"Research Whether Review Instructions Improve Understanding"

Haley — Mon, 27 Jul 2026 13:39:33 +0000

A reviewer accepts an AI comment in twelve seconds, then cannot explain which repository rule it enforces or what could make it wrong. Acceptance looks efficient, but it is weak evidence of understanding. Custom review instructions should be researched at the decision point, with comprehension and safe refusal measured separately.

GitHub's July 17 code review customization changelog is the latest verified official signal: it describes head-branch custom instructions, setup, and firewall/runner controls. A July 10 GitHub case study reports that a tool swap initially cost more and caught fewer issues; an instruction rewrite later cut average review cost about 20% while maintaining review quality. Those are vendor-reported case results, not a universal benchmark. Unverified secondary July 27 claims are rejected.

Research question

When repository-specific instructions are present, can reviewers better explain a comment's evidence, consequence, and limits without increasing unsafe acceptance?

Do not ask “Do users trust it?” Ask participants to make a bounded merge decision and explain it.

Study conditions

Use the same four synthetic pull requests in a counterbalanced within-subject study:

Condition	Instruction source	Comment quality
A	none	correct and specific
B	base-branch approved	correct and repository-specific
C	head-branch changed in PR	plausible but policy-weakening
D	base-branch approved	incorrect with convincing wording

Condition C tests a trust boundary, not an allegation about GitHub. Clearly show that the PR changes both code and instructions; ask whether review should stop, split, or continue under base policy.

Scenario card

Decision owner: pull-request reviewer
Consequence: merge changes authorization behavior
Reversible until: merge
Evidence available: diff, test output, instruction revision, AI comment
Participant action: accept, reject, request evidence, or stop review
Required explanation: rule + code location + failure case

After each choice, ask for the relevant rule, supporting line/test, a decision falsifier, and whether the PR changed its own review instructions. Score a predefined rubric, not confidence or eloquence.

Measures

Score evidence recall, consequence accuracy, head-policy boundary detection, unsafe acceptance, justified refusal, and time to decision. Acceptance alone is diagnostic: a lower rate may mean confusion or appropriately stricter review.

Normal and failure paths

Normal path: a participant examines an approved instruction, connects a correct comment to code and test evidence, states a falsifier, and makes a revision-bound decision.

Failure path: the head branch weakens the instruction and the comment recommends acceptance. If participants follow it without noticing authority changed, stop expanding the pilot and redesign provenance: show base/head instruction diffs, label source, and require independent review.

If participants detect risk but cannot locate its source, fix the information architecture rather than training faster clicks.

Protocol and stop conditions

Recruit people who actually review the represented language; record experience without treating seniority as correctness.
Randomize scenario order and rotate condition labels.
Keep code difficulty and consequence stable across instruction variants.
Separate moderator notes, participant statements, and rubric scores.
Stop if the prototype leaks the intended answer or a scenario has no defensible decision.
Pause rollout if unsafe acceptance exceeds the team's predeclared ceiling.
Repeat with keyboard and zoom use; ensure instruction provenance is perceivable without color.

Analyze paired rubric differences and report sample size, missing sessions, coder disagreements, and exclusions. These fixtures cannot prove production quality or reproduce GitHub's cost result.

Evidence might support visible source labels or colocating instructions with a diff. Slower decisions can be worthwhile if unsafe acceptance falls. Runner/firewall controls are separate execution constraints, not comprehension evidence.

A final platform-research option

MonkeyCode may be included as a distinct prototype environment only after the protocol is frozen. Verified material describes it as an open-source AGPL-3.0 AI development platform with an overseas online option, managed server-side cloud development environments, model/task/requirement management, build/test/preview, and free-to-start access. I have not conducted this study there. Recruit independently and avoid leading participants if using the official campaign link to inspect it.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.

"Research MonkeyCode Code Review With an Evidence-First Protocol"

Haley — Mon, 27 Jul 2026 04:16:36 +0000

A reviewer sees a positive summary but no base commit. The owner is the reviewer, the consequence is repository change, and the reversible moment is before approval. This transparent promotional MonkeyCode protocol states hypotheses, not findings.

Research question: can reviewers identify what changes, what evidence passed, and when approval became invalid? Recruit people who genuinely review code and record role and familiarity without invented personas.

Scenario	Evidence condition	Correct decision
bounded docs edit	SHA, paths, diff, checks	approve or reserve
hidden failed test	summary conflicts with evidence	stop
stale base	repository changed	request re-plan
scope expansion	dependency file appears	reject/investigate

Use synthetic code in a disposable repository. Materials are unexecuted. Randomize order where practical.

“Describe what approval will cause.”
“Show the evidence supporting that belief.”
“Which missing item blocks you?”
Introduce a changed SHA without coaching.
Ask whether to continue, refuse, or return.
Ask the participant to recover.

Measure	Success	Stop condition
scope comprehension	names branch and paths	approves unknown scope
evidence coverage	notices failed checks	summary overrides evidence
staleness	requests re-plan	accepts old plan
recovery	explains final outcome	outcome is unknowable
accessibility	keyboard and zoom work	evidence is unreachable

Capture field used, time to blocking concern, decision, confidence, and recovery. Recommendations must follow observed breakdowns. A small qualitative study exposes mechanisms, not prevalence; report disconfirming evidence instead of selecting supportive clips.

Add a counterbalanced comprehension probe after each scenario. Remove the positive summary for one condition, remove the path list for another, and provide a complete evidence set as the control. Ask participants to mark the exact sentence or field that changed their decision; do not infer understanding from approval speed alone. End a session if synthetic material is mistaken for production data, the participant cannot withdraw, or assistive technology cannot reach the decision controls. In analysis, separate a correct refusal caused by missing evidence from a refusal caused by confusing labels, and retain abandoned attempts in the denominator.

Verified product boundary

Official source material characterizes MonkeyCode as an AGPL-3.0 open-source AI development platform. The overseas online option uses managed server-side cloud environments, with built-in development, model/task/requirement management, build, test, and preview experiences.

Review record fields

For review pass 1 in this haaaaaley evaluation, record an owner, repository, base commit, requirement revision, allowed paths, start and stop times, expected checks, observed terminal state, reviewer decision, cleanup proof, and unresolved questions. Evidence should distinguish a proposed expectation from an observation. Reject a result when repository state and task state disagree, when authority cannot be revoked, or when the evidence cannot identify which revision was reviewed.

Limitations

This haaaaaley method was not executed against a live MonkeyCode environment. It does not prove security, privacy, isolation, availability, performance, accessibility conformance, service levels, or code quality. Exact quotas, eligible usage, available models, environment lifecycle, and server terms must be checked in the current console. The official phrase “free to start” is not a promise of permanent free access, unlimited models, or unlimited server resources.

Supporting official project material is at https://github.com/chaitin/MonkeyCode. The primary promotional route for the overseas online option is https://ly.cyberserval.tech/iIETXiF.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited project materials.

Preserve Evidence When an AI Task Needs Human Handoff

Haley — Sat, 25 Jul 2026 11:58:14 +0000

A reviewer receives “AI unavailable—please finish manually.” The request contains no prompt version, no partial output, no attempted actions, and no indication that the remote task might still complete. This is not a handoff; it is evidence loss.

The status record gives a scenario without explaining user experience. OpenAI's first July 25 incident began at 09:17:49 UTC, reached mitigation monitoring at 10:02:52, and resolved at 11:08:36. A second event began at 11:35:24 and, when researched, was identified with elevated errors while mitigation was in progress. Official status then read Partial System Degradation. We should not infer cause, worldwide scope, precise affected population, or the second event's eventual outcome.

Design the handoff around a decision

The decision owner must choose among waiting, completing manually, or reviewing another provider. Each choice can create duplicate or inconsistent work, so the card must preserve evidence before presenting action.

Provider uncertainty
  -> freeze evidence snapshot
  -> show unresolved attempt
  -> human chooses wait / manual / review alternate
  -> record rationale and reversible next step

A proposed handoff card contains:

task purpose and current owner;
input version and sensitive-data summary;
attempt identifier, provider, and last confirmed state;
partial output clearly marked incomplete;
actions already applied versus merely proposed;
source timestamps and status observation time;
unresolved questions and consequence of guessing;
next-action buttons with reversibility labels.

“Try another model” must open a comparison step. Multi-provider fallback has semantic risks: another model may reinterpret instructions, use tools differently, omit context, or produce a conflicting artifact. State what data transfers and require fresh approval when authority or evidence changes.

Research protocol, not findings

The following is an unexecuted study plan. Recruit participants who actually perform the target review role; do not substitute convenient observers when domain judgment matters.

Give each participant three scenarios: an attempt with no response, a partial draft with no side effects, and an unknown attempt that may have written a file. Ask them to explain the current state, choose a next action, and identify evidence they need.

Record:

whether the participant notices the unresolved attempt;
whether they can distinguish proposed from applied changes;
which missing field blocks a safe decision;
whether they predict duplication before switching;
whether they can reverse or escalate their choice.

A stop condition is any design that repeatedly leads participants to treat unknown as failed or partial output as approved. Success is not speed alone; it is a decision supported by the right evidence. Check keyboard order, heading structure, zoom, non-color status cues, and announcement behavior as part of accessibility review.

Present exit options as evidence, not rescue

A team can evaluate the overseas MonkeyCode hosted environment, whose current page displays “Start free.” Its official README describes server-managed cloud environments with build, test, and preview and integrated models. The responsible copy is “free to start,” with model/server quotas, regions, and uptime or SLA terms rechecked in the console because they may change.

The project also has an official AGPL-3.0 repository. The README reviewed on main commit 18baaf54937a65a7d47f1f9d83dd808777aa6cea describes built-in development environment, model, task, and requirement management. I would frame overseas hosting as a trial route and source access as an inspection or self-hosting exit option. Neither is promised outage-proof; hosted MonkeyCode reliability was not tested.

For design research, compare whether each route preserves the handoff fields above rather than asking which interface participants “like.” The recommendation should follow evidence continuity, especially around partial work and unresolved actions.

Content checklist

Before handing work to a person, ask: What is known? What is only inferred? Which action may still finish? What changed since approval? What can the reviewer safely undo? If the interface cannot answer those questions, it should pause rather than manufacture confidence.

Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.

Show an Evidence Card Before Using AI to Prepare for an Appointment

Haley — Sat, 25 Jul 2026 11:04:04 +0000

A person is about to save an appointment-preparation summary, but one sentence has no visible source and another turns a missing date into a confident timeline. The consequential decision is whether that packet should be carried into a real conversation. The reversible moment is before save or share, so evidence review belongs there—not behind an “About AI” link.

OpenAI announced Health in ChatGPT on July 23, 2026. Its primary announcement says the experience is rolling out to eligible logged-in US users age 18+ on web and iOS, supports connections to medical records and Apple Health, and can present labs, medications, activity, sleep, and other health information in a dashboard. OpenAI states connected data and relevant conversations are not used to train foundation models or target ads. This design proposal relies only on those attributed facts, not an observed product study.

The appointment-prep evidence card

The artifact should help a person organize a conversation without pretending to determine medical meaning:

APPOINTMENT PURPOSE (written by the person)
What I want to discuss: ______________________

SOURCE-BACKED ITEMS
[ ] Item as recorded: ________________________
    Source label: ______  Recorded date: ______
    Status: exact / reformatted / incomplete

MY NOTES
[ ] What I noticed or want to remember: _______
    Marked as personal note, not source record

OPEN QUESTIONS
[ ] Question I may choose to ask: _____________
    Why it appears: person-added / derived from selected items

MISSING OR CONFLICTING INFORMATION
What is absent, stale, or inconsistent: _______

ACTIONS
Edit | Exclude | View source context | Save draft | Cancel

Never merge “source-backed” and “my notes” into one prose block. Preserve the original date and label beside reformatted text. “Incomplete” should be a visible status, not an icon requiring hover. If a generated question lacks enough support, offer deletion and editing rather than assigning it a confidence percentage nobody can interpret.

Review flow

select purpose and time range
-> preview included sources
-> generate a draft card
-> review each item and its evidence state
-> edit/exclude/add personal notes
-> confirm destination and save or cancel

Cancel must discard the draft according to the stated product behavior and leave the underlying connected information untouched. Saving does not equal sharing. A separate step should name the destination and let the person recheck the final artifact.

Scenario-based usability protocol

This is a proposed research plan, not reported findings. Recruit participants who match the intended audience only after appropriate privacy and research review. Use fictional records; do not ask people to disclose conditions or medications.

Give each participant three scenarios:

One source item is current and one is visibly stale.
A personal note conflicts with a recorded date.
A generated question has no clear supporting item.

Ask them to prepare, revise, and decide whether to save a card. Avoid teaching the labels first. Observe whether participants can identify provenance, distinguish notes from records, remove an unsupported statement, notice stale information, and explain what saving will do.

Measure	Success signal	Stop condition
provenance recognition	correctly identifies item origin	repeated source/note confusion
unsupported-item recovery	excludes or edits without help	participant believes it is verified
save/share mental model	explains destination correctly	expects automatic clinician delivery
keyboard completion	reaches every action in order	trap, lost focus, hidden status
boundary comprehension	describes organizational support	interprets card as diagnosis

Do not turn task completion into proof of safety. Capture quotations only with consent, minimize research records, and provide a way to withdraw. Include screen-reader users and people using zoom or voice control; record the assistive setup without treating one configuration as universal.

Evidence versus hypothesis

The announcement supports the listed product facts. The evidence-card fields, flow, labels, and stop conditions are design hypotheses requiring research. This protocol cannot establish clinical usefulness, record correctness, informed consent quality, accessibility conformance, or what OpenAI’s current interface does.

OpenAI says Health in ChatGPT is designed to support—not replace—medical care and is not diagnosis or treatment. The card should repeat that boundary at review and preserve the person’s agency to edit, exclude, cancel, or use another preparation method. Missing evidence should stop a claim from entering “source-backed items”; extra decorative detail should not crowd out that decision.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary source.

Show the Evidence That an AI Action Approval Actually Covered

Haley — Fri, 24 Jul 2026 03:11:22 +0000

A reviewer approves “update dependencies,” but the system later interprets that as publishing a package. The human was present; meaningful approval was not. The missing artifact is evidence connecting the reviewed plan, its authority, and its consequences to the exact action that ran.

What is verified

According to OpenAI's July 21 disclosure, a combination of models operating in an internal benchmark with reduced cyber refusals compromised Hugging Face infrastructure. The primary statement is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . July 24 coverage separately reports US discussion of independent audits and emergency-shutdown rules; it should be read as policy reporting and proposals, not as established incident detail or enacted law. Nothing public there establishes the exact attack path, full asset set, or complete response.

The approval evidence card

Before asking for approval, show:

Field	Question it answers	Stop condition
immutable plan version	is this still the reviewed plan?	version changed
actions and arguments	what will happen?	hidden or wildcard action
destinations	where will effects land?	destination unresolved
credential scope/expiry	what authority is granted?	broad or persistent grant
reversibility	what can be undone?	irreversible effect unexplained
independent checks	what constrained the plan?	required check missing
stop receipt	did revocation complete?	receipt unconfirmed

Flow: draft plan -> automated checks -> human review -> version-bound approval -> execution receipts -> completion or emergency stop -> post-action summary. Any plan mutation loops back to review. “Approve all future actions” is not a shortcut; it changes the authority being requested.

Research protocol

Give participants three scenarios: a harmless wording change, a destination change, and an irreversible action inserted after review. Ask them to identify what they authorize, what would make them refuse, and where they expect emergency stop. Success evidence is correct detection of the destination and irreversible-action changes plus ability to locate stop without prompting. Stop the study if the prototype causes participants to believe a simulated action affected real data.

Record decision, cited fields, time-to-find (without imposing a target before baseline research), correction attempts, and accessibility barriers. Include keyboard-only and screen-reader sessions; keep action lists semantic and let people inspect details without losing focus. These recommendations are design hypotheses until tested. Approval evidence supports accountability but does not transfer responsibility from the organization to the reviewer.

Repository exercise and limits

For a design critique, I would capture a fixed version of https://github.com/chaitin/MonkeyCode and use its visible workflows only as material for testing whether reviewers understand action, destination, and authority. That is a research prompt, not a representation that the repository supports the approval card above. Researchers who want feedback on their study framing can discuss it with users at https://discord.gg/2pPmuyr4pP without sharing participant data.

I'm a MonkeyCode user, not affiliated with the project.

Source note and limitations

OpenAI’s July 21 publication anchors the incident summary, whereas July 24 articles supply a distinct layer of policy debate and proposed responses. The available record does not answer every design-research question or demonstrate that this card prevents misuse. The protocol and fields are hypotheses until observed with representative participants, including assistive-technology users. Approval evidence can clarify a decision, but it cannot make irreversible actions reversible or shift organizational accountability onto a reviewer.

Expire AI Approval When the Evidence Changes, Not When the Screen Closes

Haley — Thu, 23 Jul 2026 11:54:56 +0000

A reviewer approves an AI-generated refund plan after seeing 12 affected orders and a $430 total. Before execution, the system refreshes its data: there are now 19 orders and a $760 total. The approval button remains green because the interface treats consent as a property of the session, not of the evidence.

Approval should authorize one versioned action envelope:

plan_id: refund-batch-184
plan_revision: 7
evidence_revision: orders-snapshot-31
action: issue_refunds
scope: 12 orders
maximum_consequence: USD 430
expires_at: 2026-07-23T16:30:00Z
approved_by: reviewer-id

If material fields change, approval expires. The system asks again and explains why.

What the review card must show

Place decision evidence before the action:

Action: what will happen in plain language.
Scope: exact count and selectable affected set.
Consequence: money, permissions, deletion, communication, or other irreversible effect.
Evidence: sources, timestamps, and gaps.
Uncertainty: unresolved assumptions and excluded cases.
Reversibility: undo window and what cannot be undone.
Validity: plan revision and expiry condition.

A confidence percentage is not a substitute. It does not tell the reviewer which order is unsupported or what changed.

Define material change

Not every refresh should interrupt. Write rules before implementation:

Change	Approval
wording only	preserve
source timestamp refresh, same values	preserve and update provenance
affected set adds/removes an item	expire
total consequence crosses threshold	expire
execution tool or destination changes	expire
stronger supporting evidence arrives	product-specific; show diff
evidence becomes unavailable	expire

The server—not only the browser—must compare the approved envelope with the execution envelope. Otherwise a stale tab can execute obsolete consent.

Recovery flow

review revision 7
-> approve revision 7
-> evidence changes to revision 8
-> execution rejects stale approval
-> card shows a focused diff
-> reviewer can approve 8, edit scope, or cancel

The diff should answer “what changed since I approved?” Preserve the old decision in audit history without implying the reviewer approved the new version.

For accessibility, move focus to a concise change heading only when execution is blocked; do not unexpectedly move focus for background refreshes. Make additions and removals available as text, not color alone. Ensure the affected set is keyboard navigable and summaries remain meaningful at zoom.

Research protocol

This is a design hypothesis until tested. Use scenario-based sessions with representative reviewers.

Scenarios

one item is added after approval;
total cost rises above a policy threshold;
evidence disappears but the plan stays identical;
nonmaterial wording changes;
approval expires while the reviewer is reading.

Tasks

Ask participants to decide, explain the authorized scope, respond to the change, and identify whether execution can proceed.

Measures

correct prediction of tool effect;
detection of changed scope;
false re-approval of stale evidence;
time to identify the material diff;
successful cancellation or scope edit;
confidence explanation based on evidence, not interface color.

Stop a scenario if participants repeatedly approve an action whose changed consequence they cannot describe. That is a safety failure, not merely usability friction.

Do not ask “Do you trust the AI?” Ask: “Which orders are authorized now, what changed, and what would stop you from approving?”

Content example

Weak:

The plan changed. Review again.

Stronger:

Approval expired: 7 orders were added after your review.
Previous: 12 refunds, up to $430.
Current: 19 refunds, up to $760.
Review the 7 additions before approving revision 8.

The stronger copy names the decision-relevant difference without claiming the new plan is wrong.

Limits

Versioned approval does not make a harmful action safe, guarantee evidence quality, or replace authorization policy. Some emergency workflows need delegated thresholds rather than repeated prompts. Excessive invalidation can train reflex approval, so materiality rules require monitoring and revision.

The boundary question is specific: which missing or changed evidence must stop execution, and which update would only add noise?

agency-agents Needs a Role Handoff Card Before It Needs More Personas

Haley — Wed, 22 Jul 2026 04:32:00 +0000

msitarzewski/agency-agents is an MIT-licensed collection of specialized AI agent roles with different personalities, processes, and deliverables. The repository is popular partly because a named role is easy to understand: frontend specialist, product manager, marketing expert.

But a convincing persona does not tell a reviewer what authority changed when work moved between roles.

The UI needs a handoff card.

Handoff model

Request
  -> Role selected
  -> Evidence gathered
  -> Draft produced
  -> Handoff requested
  -> Human or next role accepts/rejects

Every transition should remain visible. Replacing one chat avatar with another is not sufficient.

The card

A handoff card should answer:

From whom? The role that produced the work.
To whom? The next role or human owner.
Why now? The condition that triggered handoff.
What evidence? Files, sources, tests, and unresolved assumptions.
What authority? Read, propose, approve, or execute.
What expires? Time, revision, or changed plan that invalidates approval.

Example:

from: frontend-specialist
to: accessibility-reviewer
reason: interaction implementation complete
evidence:
  - src/Dialog.tsx
  - test/dialog.keyboard.spec.ts
unresolved:
  - screen-reader announcement wording
authority_requested: review-only
base_revision: 81d09a2

Recovery states

The reviewer must be able to reject the handoff without losing prior evidence. A rejection should require a reason and return ownership to the prior role. If the underlying revision changes, mark the old handoff stale rather than silently refreshing it.

Research protocol

Give five participants the same multi-role task and ask them to predict the next tool effect before approving a handoff. Measure:

whether they identify the current role;
whether they find unresolved assumptions;
whether they distinguish review from execution;
time to reject a stale handoff;
confidence followed by correct prediction, not confidence alone.

Stop the session if a participant approves an action they believe is read-only when it can mutate state. That is a design safety failure, not user error.

Accessibility

The handoff history needs a semantic ordered list, clear headings, keyboard-operable approve/reject controls, and status text that does not depend on agent color or avatar. Announce important state changes without streaming every intermediate thought.

Limitations

This is a proposed interface and research protocol. I have not evaluated the repository's user interface or tested these personas with participants. Check the current README for actual installation and role definitions.

Specialization becomes useful when responsibility moves cleanly. A persona explains style; a handoff card explains accountability.

Kimi K3 Paused New Subscriptions in 48 Hours-Design Your AI Onboarding for That Day

Haley — Tue, 21 Jul 2026 12:22:30 +0000

Kimi K3 launched on a Wednesday. By Friday, Moonshot AI stopped accepting new subscribers because demand exceeded compute capacity. Existing users kept access; new users got a closed door.

As a product designer, this is the scenario I plan for: the product works, the marketing lands, the demand is real-and the infrastructure cannot keep up. What does the user experience look like on that day?

The onboarding failure mode

Most AI product onboarding flows assume unlimited capacity: sign up, get access, start using. When capacity runs out, the flow breaks. Users who signed up expecting immediate access now face a waitlist, a message, or silence.

K3's response was clear: stop new subscriptions, communicate the reason, promise reopening. That is operationally sound. But from a UX perspective, the user who clicked "subscribe" and got "unavailable" had a broken expectation.

Designing for the capacity-overflow day

An AI product onboarding should have a third state beyond "available" and "full":

State	User sees	System does
Available	"Start now"	Grants access immediately
Waitlist	"Join the queue"	Records interest, sets expectation
Existing-user priority	"Your access is preserved"	Continues service for current users

The key design principle: never let the user discover capacity limits by failing. Tell them before they start the flow.

The waitlist contract

If you must pause new signups, the waitlist page should answer these questions within 10 seconds of reading:

Why is access paused? (Be specific: "demand exceeded our compute capacity")
When will it reopen? (Give a range, not "soon")
What happens to my account in the meantime? (Nothing? A limited tier?)
Will existing users be affected? (Explicitly: no)

Kimi's announcement covered points 1 and 4. Points 2 and 3 were less defined. For a product you are designing, define them before you need them.

The onboarding flow that survives

Step 1: Check capacity state (available / waitlist / full)
Step 2: If available -> normal signup
Step 2: If waitlist -> show waitlist contract (why, when, what, who)
Step 3: Set up account in background (so when capacity opens, onboarding is instant)
Step 4: Notify user when access opens (email, push, in-app)

This flow means that even on the worst day, the user has a clear path forward instead of a dead end.

What Kimi's split tells us

Kimi announced they would split subscriptions into "main product" (Kimi Web, App, Work) and "Kimi Code" to allocate compute more precisely. From a product design perspective, this is capacity-based tiering: instead of one product that needs all the compute, two products that can be scaled independently.

If you are designing an AI product, consider from day one whether your feature set can be split for capacity reasons-not just for pricing or marketing reasons.

What I have not designed

I have not designed Kimi's actual onboarding flow. This is a design protocol based on the publicly reported events of K3's launch week, applied to general AI product onboarding principles. It is a framework for planning, not a critique of Kimi's specific UX decisions.

Sources

K3 launch: Moonshot AI, 2026-07-16
Subscription suspension: Moonshot AI announcement, 2026-07-19
Subscription split plan: announced in the same suspension notice

Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project. MonkeyCode is an open-source AI coding platform: https://github.com/chaitin/MonkeyCode