Info Inlet

Posted on Aug 24

ChatGPT Plugin Rejected? Here's What the Review Actually Checks

#programming #ai #openai #chatgpt

We submitted our app to OpenAI's directory on a Thursday. Nine days later, this arrived:

One or more of your test cases did not produce correct results. Please re-run all submitted test cases and align tool behavior/output with the documented expected outcomes. Ensure the same test cases pass consistently on both ChatGPT web and mobile.

That is the entire rejection. No tool named, no transcript, no log line. One paragraph and a link to appeal if we thought they'd made a mistake.

They had not made a mistake. Every symptom in that paragraph was real, and finding out why took a full day of running our own tools against production like a stranger. There were four failures: three server bugs, and one that was in the test cases themselves. Two of the three bugs meant tools that had never worked once, in production, for anyone, since the day they shipped.

We build Xenition, an AI workspace, and the submission was our MCP server — 49 tools that let ChatGPT create and edit real artifacts in a user's workspace: documents, decks, boards, ledgers, notes, spreadsheets. If you're preparing a submission this month, this article is the thing I wish I'd read the week before we hit Submit — every bug named, and the one review technique that would have caught all four.

I'll name our own bugs exactly rather than paraphrase them. None of it is product-specific.

Why this review is harder to pass than an app store review

A normal store reviewer taps through your UI. Your UI is a thing you have looked at ten thousand times.

A directory reviewer does something you have almost certainly never done: they take your written test cases, hand the prompts to a model, and watch what your tools actually return — then do it again on a second client. That's the part that broke us. The review surface isn't your product. It's:

A cold account. No state you seeded by hand while debugging.
A one-turn tool call. No follow-up. Whatever your tool returns, that's the whole conversation.
Two clients that word things differently. ChatGPT web and ChatGPT mobile do not phrase the same user intent identically, so your tool receives two different strings for one test case.
Someone reading the model's sentence, not your JSON.

Every one of our four failures lives in that list. None of them was reachable from our own app.

Bug 1: the tools read an empty copy of everything

Our artifact store has a List that selects content and meta as NULL. That is not a bug — it's the query behind the library grid, which renders cards. A card needs a title, a type and a timestamp; shipping every document's full body to draw a wall of tiles would be absurd.

Then the workspace tools arrived — list_tasks, add_task, query_ledger, add_ledger_entry, search_notes — and their lookup helper went through that same List. Every one of those callers then parses the document it just asked for.

Here is what a user of a perfectly healthy workspace got:

Call	Reality	What the tool said
`list_tasks`	a board with 30 cards	`0 tasks`
`add_task`	same board	"the board is empty"
`add_ledger_entry`	a ledger with real invoices	success — having saved a brand-new ledger over the real one

Read that last row again, because it's the one that scared me. Enter an invoice for Acme, then one for Globex, and you are left with a ledger containing only Globex. Not an error. Not a warning. A success message, and the user's data replaced with a single row. The empty parse looked exactly like an empty document, so "add a row to the existing ledger" and "create a ledger with one row in it" became the same code path.

The dates are the humbling part. The List change that started blanking content landed on 20 July. These tools shipped on top of it on 10 August. We submitted on 13 August. They never worked, not once, in production — and three days of our own use didn't reveal it, because nothing in our own app calls the workspace through MCP.

The fix is four lines and a rename:

// ListFull is List with content and meta loaded. Callers that have to READ what is inside each
// artifact — the board's cards, the ledger's rows — need this one, not List.
func (s *Store) ListFull(ctx context.Context, userID, workspaceID, typeFilter, search, projectID string) ([]Artifact, error) {

and at the call site, with the reason written down so nobody quietly swaps it back:

// ListFull, not List: every caller here reads the artifact's document — the board's cards, the
// ledger's rows, the note's body. List blanks content for the library grid.
return s.artifacts.ListFull(ctx, uid, "", typ, query, "")

The lesson generalises well past MCP:

A read path optimised for one caller is a data-loss bug waiting for its second caller.

List returning blank content was a feature for the grid. It became silent destruction the moment a writer used it to decide whether something already existed. If a query drops fields for performance, the name has to say so — List vs ListFull — because the type signature doesn't. Both return []Artifact. Both compile. Only one of them tells the truth about what's inside.

Bug 2: our clarifying question had nobody to ask

Our orchestrator has a clarify gate. Vague brief in, questions back out — good behaviour in a chat window, where a human answers and generation continues.

An MCP tool call is one turn. There is no human in it. So a thin brief made create_slides come back in about two seconds with:

the engine did not return a deck

Correct, technically. The engine had returned a set of questions and there was nobody to answer them, so the tool reported the only thing it could see.

Now the detail that turns this from an ordinary bug into the exact wording of the rejection: whether the gate fires depends on how the brief reads, and ChatGPT phrases the same user intent differently on web than on mobile. Same test case, same account, same minute — passes on one client, fails on the other. "Ensure the same test cases pass consistently on both ChatGPT web and mobile" was not boilerplate. It was a description of our bug, written by someone who had just watched it happen.

The fix is to tell the engine there is nobody home:

// NoClarify because there is nobody here to answer. On a vague brief the orchestrator's clarify
// gate returns questions instead of an artifact, and an MCP call is one turn.
req := engine.Request{Message: brief, UserID: uid, Skill: skill, NoClarify: true}

A flag is a promise, though, and promises get broken by the next person to touch the
orchestrator — so there is also a guard behind it, and the comment on the guard is the part
worth copying:

// Belt-and-braces: NoClarify above should stop this, but if the gate ever fires again the
// caller gets something it can act on rather than "the engine did not return a deck".

If the gate does fire, the tool now returns the questions themselves. A model that receives
three questions can ask the user them; a model that receives "the engine did not return a
deck" can only apologise.

Any interactive gate in a pipeline becomes a hang or a lie when the caller is a machine. Clarify loops, consent prompts, "are you sure?", rate-limit backoff that waits for a retry someone has to press — find every one of them before a reviewer does.

Bug 3: an unknown id reported as "Still working…"

Small bug, worst optics of the three.

check_agent_run polls a background agent. Ask it about a run id that does not exist and it answered:

Still working…

For any id. Forever. A typo'd id, a made-up id, an id from a different account — all of them "still working". A model that receives that will tell the user to be patient, then tell them again, and the user waits for a job that was never created.

Fixed by making a 404 an error rather than an absence of information, and the error names the recovery:

no agent run found with id "nope" — use the missionId that run_agent returned

The general form: "not found" and "not finished" must never collapse into the same reply. One of them is a completed fact about the world; the other is a request to wait. Any status endpoint that answers pending when the truth is nonexistent will strand somebody.

Problem 4: our test cases depended on each other

This one had no server fix, and it's the one I'd bet most submissions get wrong.

Our eight cases looked reasonable in the portal. Case 1 created a deck. Case 2 searched the workspace for the deck case 1 had created — and the expected-outcome text said, helpfully, run case 1 first.

A reviewer who runs them in a different order, in a fresh chat, or on the second device sees case 2 fail. And they are right to. We had written a suite that only passes if it is executed as a script by someone who read our footnotes, and then handed it to strangers on two clients.

Rewritten, every case stands alone. These are the rules we wrote at the top of the file so the next set can't regress:

Self-contained. Seed the demo workspace so the read cases have something to find without a write case running first.
Specific briefs. A vague brief is what fired the clarify gate. NoClarify fixes it server-side, but a specific brief also keeps the output stable enough to describe.
Expected outcomes describe shape, not exact words. A deck's title comes from a model and differs run to run. "A 5-slide deck is created and rendered" passes every time; asserting a literal title does not.
No dependence on which artifact is "most recent" unless the outcome is phrased to allow any of them.
Every case verified against production, from a cold account, on both clients.

An example of the difference, from the same tool:

Version	Expected outcome
Rejected	Returns the deck created in case 1, titled "Orbit Sales Deck".
Resubmitted	`search_artifacts` returns the workspace's sales-onboarding documents as a results list, each row with an "Open" link back into the app. Read-only: nothing is created or changed.

The second one is falsifiable by a stranger. That's the only property that matters.

The part I got right by accident: annotations

While preparing the submission we annotated all 49 tools — readOnlyHint, destructiveHint, openWorldHint — and had to write a justification for each hint on each tool. 147 short paragraphs. It felt like paperwork.

It wasn't, because of one line in the spec that I would have bet money against:

// DestructiveHint and OpenWorldHint are *bool in the Go SDK, and the spec's default when they
// are ABSENT is true.

Absent means destructive. Absent means open-world. Before we filled these in, every tool we shipped — search_artifacts, a pure read, included — was advertising itself to every client as a destructive, open-world operation. Not a cosmetic problem: clients use these hints to decide what needs a confirmation dialog and what can run unattended.

We settled it with constructors instead of scattered literals, one per behaviour class, each carrying the definition in its comment:

// hintsRead: reads the user's own workspace and nothing else.
func hintsRead() *mcp.ToolAnnotations {
    no := false
    return &mcp.ToolAnnotations{ReadOnlyHint: true, DestructiveHint: &no, OpenWorldHint: &no}
}

// hintsAdd: adds something new to the workspace; nothing that already exists is replaced.
func hintsAdd() *mcp.ToolAnnotations {
    no := false
    return &mcp.ToolAnnotations{DestructiveHint: &no, OpenWorldHint: &no}
}

// hintsAct: hands work to a system whose effects this call does not bound — a background agent, or
// a pending action against a connected third-party app. Destructive because what runs is open-ended.
func hintsAct() *mcp.ToolAnnotations {
    yes := true
    return &mcp.ToolAnnotations{DestructiveHint: &yes, OpenWorldHint: &yes}
}

Constructors, not package-level vars, so no two tools can ever share and mutate one annotations value.

Writing 147 justifications also forced the question what does this tool actually do onto tools nobody had asked it about in a while, and three of ours turned out to be annotated the opposite of what their names suggest:

create_app is read-only. It persists nothing. It builds a pre-filled deep link into the builder.
check_3d_model is not read-only. When the poll succeeds it writes the finished mesh onto the artifact.
approve_action is destructive and open-world; deny_action is neither. Approving releases a queued action into a third-party app. Denying closes a request locally.

If a reviewer ever questions our annotations, it will be those three, and the justification text already explains each one. Annotate by what the call does, never by the verb in its name.

The demo account is part of the submission

Two things here cost me hours and would cost you the same.

A duplicate row can be load-bearing. Our seeded workspace held two documents with the same title, so a search case returned the same name twice. It looked sloppy, so I deleted the newer copy — and the grounded-answer case went flaky. It had been returning the same six-step answer every run; afterwards one run answered properly and the next said the provided passages do not describe the onboarding process. Removing the duplicate had thinned the grounding corpus below whatever threshold that answer needed.

Flaky is the precise failure we'd just been rejected for. The copy went back, the answer was consistent across three runs again, and the doc now says so in writing:

A duplicate row reads better than an unreliable answer.

Clean out what your own debugging left behind. Verifying all 49 tools against production left the demo account full of junk — a "Dev Board", "A Budget", four near-identical sales decks. A reviewer opening that account should find a workspace that looks like a real user's, not the wreckage of a test sweep.

Portal friction, so you can plan around it

Four things that cost time and are nobody's bug:

The tool list is virtualised. Page-reading tools only ever return part of it. Filling in each tool's fields one tool at a time was the only reliable way through.
The SPA has a load race. Navigating straight to a deep-linked section sometimes renders ask an organization admin to assign you a role with the api.apps.read permission, and the Skills tab briefly showed no uploaded skill at all. Both times a full reload through the plugin list showed the real state. It is not a permissions problem — I nearly filed a support ticket about a role I already had.
Scan Tools needs an OAuth authorization first. It opens your own consent page and somebody has to sign in there before the scan will run.
The skill safety scan is slow. The portal warns up to two hours. Don't schedule your submit for the last hour of the day.

And one thing worth knowing before you plan a launch: approval does not publish. When it passes, the portal unlocks a publish option and a human presses it. The go-live moment stays yours.

The pre-submit checklist

Everything above, as the list I'll actually run next time:

[ ] Run every tool against production, from an account with no state you created by hand.
[ ] For each read tool: does it see content, or an optimised-away copy of it? Check against a known row count.
[ ] For each write tool: does "I found nothing" ever become "so I'll create a fresh one"? That's a data-loss path.
[ ] Does any tool sit in front of an interactive gate — clarify, consent, confirm — that has no human to answer it?
[ ] Does any status tool report an unknown id as pending?
[ ] Do the test cases pass in any order, in a fresh chat, on web and mobile?
[ ] Does any expected outcome assert model-generated wording? Assert shape instead.
[ ] Are destructiveHint / openWorldHint set explicitly on every tool? Absent means true.
[ ] Is each annotation justified by what the call does, not by its name?
[ ] Is the demo account seeded, and cleaned of what your own testing left behind?
[ ] Did you read the model's sentence for each case, not just your JSON?

What I'd tell someone submitting next week

The rejection email is one paragraph and it will feel unfair. Take it literally instead of personally: "did not produce correct results" and "consistently on both web and mobile" were, in our case, precise technical descriptions of three bugs and a broken suite. We didn't appeal, because there was nothing to appeal — they had run our tools as a user and seen our product lie to them.

The uncomfortable takeaway isn't about any directory. It's this: an MCP server is a second product, and it has its own users, its own state, and its own bugs. Ours had two tools that had never once worked in production, and our own app was structurally incapable of noticing, because our app doesn't call itself through MCP. Every hour of that review found something real.

At the time of writing this, the fixes are live on production, re-verified there — list_tasks reads 30 and goes to 31 after add_task, the ledger keeps both invoices, an unknown run id is an error — and the resubmission hasn't gone in yet. If the second attempt teaches me anything new, I'll write that one too.

If you've built a harness that calls your own MCP server the way a client does — cold account, one turn, both clients, reading the model's sentence rather than your JSON — I'd like to read about it. That's the piece I'm still missing, and it's the only thing that would have caught all four of these before a stranger did.

I work on Xenition — one AI workspace for documents, decks, code, apps and media, free to start, on web, desktop and both app stores. Its MCP server has 49 tools. Two of them, until recently, had never worked.

DEV Community