DEV Community: Yuli Dai

Scope Desk: "one more thing" needs a scope review

Yuli Dai — Tue, 13 Oct 2026 04:48:33 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

The agreement says one English landing page with a contact form. A new message says, "Please add online payments and a French version."

Neither request sounds huge in a chat. Both add work. Before talking about a price, someone has to put the original agreement beside the new message and work out what changed.

I built Scope Desk for that step. It's a small command-line tool for a freelance developer reviewing client requests. You choose the project, enter the agreed scope, and give it a new message. An open-weight Nemotron model drafts three kinds of suggestions: included work, extra work, and things that need clarification. Each suggestion quotes the request it came from. Included work also has to point to an existing scope item.

The result stays a draft. You read it, explicitly approve it, and export a plain-text scope review to edit before sharing. It doesn't set a price, agree to do the work, or send a message to a client.

The potential buyer is a freelancer juggling small fixed-scope jobs. The value would be having the agreement, new requests and review record together when deciding what to discuss with a client. That's a business hypothesis: I haven't had a real customer try it, and nobody has paid for it. The current prototype is free, MIT-licensed, and still needs user feedback.

Demo

Watch or download the 72-second demo.

The video is a labelled text replay of real saved synthetic model results and actual command-line checks. It is not a live screen recording or a new inference run. The approval in the demonstration was scripted; it doesn't establish that a human reviewed the answer.

It shows three development examples:

Payments and a French version added to the agreed English landing page: two extra-work suggestions.
A revision to the third blog post, with one revision per post already agreed: included work.
"Can you improve it?": clarification, because the request doesn't say what to improve.

It also shows an unapproved draft refusing export, an approved review exporting successfully, and a repeated approval after reopening the database returning the saved receipt without duplicating it.

These are three synthetic development probes, not an accuracy benchmark or evidence that it handles every contract correctly.

Release page and limitations.

Code

Scope Desk on GitHub

The project and repository were created on October 13, after this challenge's opening. This is new code, separate from my earlier TheOne projects. Codex assisted the implementation, tests, documentation and demonstration.

Python 3.11 or later and its standard library are enough for the CLI and SQLite storage. The README has the commands for creating a project, analysing a request, reading the draft, approving it and exporting the review. The model call requires your own provider access; inference is not promised to be free.

Eight offline tests use mocked provider responses. The three genuine model calls are documented separately in the synthetic development evidence. The published implementation's CI passed. Keys and private databases are excluded from the repository.

How I Built It

The model used in the actual calls was nvidia/nemotron-3-super-120b-a12b, served by Nebius Token Factory. Each request contains only the selected project's scope and current message. Other projects don't go into that request.

The response has to match a small JSON schema. The application then checks the quotes against the input and the scope references against the selected project. An unknown reference or an incomplete response is rejected. Those checks establish where the words came from; they cannot establish that the model's interpretation is right. That's why the draft still needs review.

I also wanted the record to survive ordinary interruptions. Approval writes a receipt to SQLite. Repeating that approval after reopening the database returns the same receipt. If the agreed scope changes while inference is running, the result is discarded. A stale unapproved draft cannot be approved against a changed scope.

The persistent prototype budget allows ten attempted model calls per database, including failures. A process restart doesn't reset it. The default request is bounded to 900 output tokens and a 45-second timeout. There is no fake-answer fallback when the key is missing or the provider fails.

This is a single-operator prototype. Its database isn't encrypted and it has no multi-user authentication. A review flag is a deliberate action, not proof that someone actually read the text. The exported review still needs editing before anyone uses it with a client.

Why Does Open Innovation Matter?

The open-weight model does the central task: it interprets a request against an agreement. The original application code is open too, so the prompt, validation, review gate and storage can all be inspected and changed.

The model and endpoint are settings rather than assumptions buried in the workflow. That gives me a place to try another compatible model without rewriting how approval or receipts work. A compatible local endpoint is supported by the client, but I haven't tested local inference. All three actual model calls used the hosted service.

That distinction matters here. SQLite keeps the records on the operator's machine, but the selected scope and message still go to the configured provider. This build does not keep all client data offline, and I haven't measured a cost or quality advantage over another model. The useful part of the open approach so far is control over the workflow and the ability to investigate or replace its parts.

The next useful test is a freelancer trying their own messy, ambiguous requests and telling me where the suggestions stop helping. I would do that with their consent and without publishing their client data.

Prize Categories

Overall challenge entry. No partner category is claimed.

Two models sent a moving bill to the wrong chat

Yuli Dai — Sun, 11 Oct 2026 02:37:51 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

I'm building TheOne, a DSH plugin where you talk in one main chat and each topic has a separate background session. Before the assistant answers, it has to decide where your message belongs.

I wanted to isolate that decision. Given a catalog of three topics and recent conversation, can a model choose an existing topic, recognize a new matter, or ask for clarification?

The test has 36 synthetic cases, six in each group:

Explicit topic references
Resuming a topic from recent history
Overlapping words and facts across topics
Starting a genuinely new matter
Ambiguous references that require clarification
Routing instructions quoted inside untrusted content

The topics are a thesis paper, a monthly household budget, and an apartment move. Each case makes one model request. The catalog order rotates, and neither the expected label nor the group name appears in the prompt. The model must return a topic ID, NEW, or CLARIFY. Scoring is exact matching after case and whitespace normalization. Extra explanations score zero; there is no LLM judge.

The cases and implementation were authored with AI assistance. A separate offline harness checks scoring with scripted responses. Those checks make no model calls and are not part of the model results below.

Models Tested

Kaggle's initial task execution used Gemini 3.7 Flash. I then ran GPT-5.4 nano as a lightweight alternative on the same frozen cases and scoring policy. These are single runs, not repeated trials or a search for the best prompt.

I also scheduled Qwen 3 Next 80B Instruct. Update, October 11: Kaggle reports that run ended with HTTP 429 because the model was experiencing heavy load. I am not assigning it an accuracy score or treating the service failure as a routing failure. I have not started a replacement run.

Findings

Group	Gemini 3.7 Flash	GPT-5.4 nano
Explicit topic	6/6	6/6
Resumption	6/6	6/6
Overlapping facts	5/6	5/6
New matter	6/6	6/6
Clarification	6/6	5/6
Quoted instructions	6/6	6/6
Total	35/36 (97.2%)	34/36 (94.4%)

Both models missed this message:

Add the 480 EUR moving bill as a one-off expense in my monthly budget.

Both returned the apartment-move topic. The expected destination is the household budget: the user is asking to categorize an expense. The routing policy explicitly distinguishes moving logistics from monthly spending.

The paired case made this easier to inspect:

The mover's quote is 480 EUR. Compare it with renting a truck for moving day.

That request belongs to the apartment move, and both models routed it correctly. The number and subject overlap, but the requested action changes the destination. My interpretation is that the moving-related content outweighed the budget operation in the failed case. Two examples cannot establish a general mechanism, but they give me a specific failure to test further.

GPT-5.4 nano also picked the thesis topic for “Use the smaller number,” even though the supplied history mentioned lowering both the participant count and the grocery allowance without selecting either. Under the stated policy, that should be CLARIFY.

The six quoted-instruction cases passed for both models. That is useful as a basic check, but the outer requests are explicit and the quoted attacks are simple. I would not call this evidence of general prompt-injection resistance.

The completed traces record about $0.0368 for Gemini and $0.0026 for GPT-5.4 nano. Those are recorded inference costs for these runs, not a price comparison for general workloads. The failed Qwen run is excluded. Execution used Kaggle's free allowance; no paid subscription was started.

The scores are close to the ceiling. One case changes the total by 2.78 percentage points, so the one-case gap is not enough to declare a broadly better model. The concrete mistakes tell me more than the ranking.

Next I would expand the paired cases while keeping the operation distinct from the subject, then test natural ambiguous dialogue without phrases such as “neither is selected.” I would also test cases where a single message genuinely needs two projects, and define the expected behavior before running the models.

This is a small public diagnostic of base-model decisions under an explicit policy. It does not measure production TheOne accuracy, represent real user conversations, or provide a contamination-proof held-out test.

My Benchmark

The result summarizer checks all 36 case traces against the source labels and the parent score before producing the tables. Kaggle run IDs are 4630860 (Gemini) and 4630864 (GPT-5.4 nano).

If you've built a router like this, how do you decide which topic owns a message that mentions one project but asks for work in another?