DEV Community: Hamza Mohammad Khan

Reward Desk: When a Payout Claim Is Missing Its Proof

Hamza Mohammad Khan — Sun, 04 Oct 2026 12:47:55 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange

What I Built

Built by hkay.

Reward Desk is a small workspace for people trying to earn money through product bounties. An opportunity advertising "$500 available" and a message saying "submission received" can both feel like progress. Neither proves that money arrived.

I wanted a desk that could explain the difference. Programs hold participation conditions and advertised rewards. Cases hold milestone events. Each event points to a separate proof document. The interface checks those relationships before including a value in counted income.

The demonstration contains a complete $96 payout, a $75 award still awaiting delivery and redemption, a submitted report, a rejected report, a service-credit reward, and a claimed $40 payout with its delivery evidence missing. That last $40 stays outside the total even though a later redemption event exists.

Every record is fictional. No private bounty report, email, chat, receipt code, or real earnings information is included.

Demo

Open Reward Desk — no login required.

The initial view reads published records from Sanity. The connection label says "Sanity · published content," while the banner explicitly identifies all records as synthetic. The totals should show $96 counted, $75 awarded but not yet redeemed, and one evidence gap.

Try the following:

Select A payout claim missing its receipt. Its timeline shows the absent delivery document and explains why the later redemption claim cannot fill that gap.
Select Try sample scenario. This opens a separately labelled, local sample workspace. Select the missing-receipt case and attach its sample delivery receipt. The fictional counted total changes from $96 to $136. Reset sample restores the starting state. These experiments do not write to Sanity.
Open Opportunities to compare an open program, a closed program, unconfirmed eligibility, and a noncash reward. A large advertised maximum does not make an opportunity actionable.
Use the case search, stage filter, and Export preview. The export keeps counted amounts separate and excludes evidence descriptions.

If the Content Lake cannot be reached, the app visibly announces that it has fallen back to sample data. An empty live dataset stays empty instead of quietly becoming an invented live workspace.

Code

Public source repository

The application uses Next.js, React, the Sanity JavaScript client, and TypeScript Studio schemas. The frontend is a static export on GitHub Pages; it queries Content Lake in the browser on each load. It needs no frontend API token.

The repository includes the three schemas, GROQ query, synthetic NDJSON seed, reward checks, automated tests, and a verification script that compares the published dataset with every approved seed field. The docs/ folder contains the deployed export.

My Build Process

I used Codex to generate and refine the application. The brief was to make the evidence chain visible and keep advertised, awarded, delivered, and redeemed amounts distinct. That is a paraphrase of the design intent, not a quotation from an exported prompt transcript.

The useful constraint was a deliberately awkward example: a case that claims redemption but lacks its delivery receipt. A single status field could call it "redeemed" and make the dashboard look healthy. The event-and-proof model instead lets the interface preserve the claim while explaining why it does not count.

The schema separates three kinds of facts:

rewardProgram records current scope, eligibility, reward type, advertised ceiling, policy URL, and deadline.
rewardCase references a program and stores milestone events.
rewardProof belongs to a specific case and records an evidence kind, public-safe description, value, and unit.

The frontend validates the chain independently of Studio. It rejects missing or wrongly typed evidence, evidence belonging to another case, conflicting values, future or out-of-order events, duplicate milestones, and contradictory review outcomes. A closed program can still retain an earlier valid payout. Service credits remain visible without becoming USD income.

The first working version used bundled examples. That was enough to develop the interface, but it was not evidence of a live Sanity integration. Completion therefore required signing into the owned project, checking that the target dataset was empty, importing exactly 4 programs, 6 cases, and 19 proofs, and reading them back through the same published-content query used by the app. The verification compared every seed field and the expected totals, rather than checking only whether the API returned HTTP 200.

Deployment exposed two practical problems. Next's Turbopack build rejected a dependency-directory junction in the local Windows workspace; the supported webpack build produced the static export. Then Sanity rejected browser reads from the new public hostname. Adding the exact https://hkay26.github.io CORS origin with credentials disabled fixed the public read. Neither step required putting an API token into the client.

Eighteen automated checks passed, along with the static production build and the Studio TypeScript check. The deployed browser view was checked against the expected Sanity totals. The sample repair is kept in a separate mode so testing a "what if" never edits the published dataset.

This version uses Content Lake, GROQ, document references, and Studio schema validation. It does not use App SDK or Sanity Workflows. The milestone process is represented as content; it does not automatically move real reports through an external program.

There are important limits. Linking a proof document cannot authenticate a receipt or prove an issuer paid. This version supports one settled payout per case, not partial deliveries, refunds, currency conversion, or reversals. Those need additional event types and reconciliation rules. The public demonstration is intentionally separate from any future private-data workflow.

Sanity Project Details

Project ID: cogop6sj
Dataset: production (public, synthetic demonstration records only)
Document types: rewardProgram, rewardCase, rewardProof
Published records: 4 programs, 6 cases, 19 proofs
Verification: October 4, 2026; all published seed fields matched and the expected $96 / $75 / one-gap result passed.

Inspect the public document counts.

The synthetic missing-delivery example intentionally uses a weak reference to an absent document. Its absence is the demonstration: a later claim does not create the missing evidence.

Reward Evidence: an AI can track the money and still misread the opportunity

Hamza Mohammad Khan — Sat, 03 Oct 2026 21:34:38 +0000

Prepared for the Kaggle Benchmarking Challenge. Developed with OpenAI Codex assistance; the DEV authorship setting is Fully Autonomous.

On 40 synthetic paid-work cases, GPT-5.4 nano correctly extracted every cash receipt but missed nine availability labels. The money ledger and the decision to pursue an opportunity need separate checks. Gemini 3.7 Flash matched the full rubric on all 40 cases; gpt-oss-20b matched 30. This is a small diagnostic, not a general ranking.

What I Benchmarked

A paid-work assistant needs to distinguish an offer, an eligibility condition, a chance of winning and money actually received. A dollar sign alone answers none of those questions.

Reward Evidence contains 40 original synthetic cases in 20 matched pairs. Each supplies numbered sentences and an evaluation date. The model extracts seven fields: reward form, maximum individual cash award, currency, availability, selection method, confirmed cash receipts in integer minor units, and receipt currency. It also identifies the supporting sentences.

Pairs distinguish cash from service credit, prize pools from individual awards, pending from delivered payments, old from current terms, available from exclusively claimed work, and pending verification from explicit ineligibility. Eight cases contain longer distractor passages. These are invented examples with no private reports, correspondence, customer data or copied program policies.

Evaluation design

Two variants share the specification, cases and shuffled order (seed 271828). Plain requests extraction; grounded adds reminders about current terms, individual awards, conditions and delivered money. Every case gets a fresh chat, and the code requests temperature 0. The SDK passes that value only for models marked as supporting temperature; effective provider settings were not independently verified. Each comparison requires 80 requests.

Exact correctness requires all seven fields and the complete expected evidence set. The grader compares decimal cash amounts, rejects duplicate JSON keys and wrong types, and checks evidence bounds. Request errors reduce coverage. Unsupported-cash and invented-receipt counts cover schema-valid answers, so invalid schemas are shown alongside them. Fifteen offline controls check the grader and runner; those are separate from model results.

A correction I had to make

The first pilot scored 37/40 for both prompts under my original rubric. Reviewing every case revealed two unsupported gold labels: an offer without selection terms had been labeled per accepted result, and I had carried a superseded announcement's selection terms into its replacement. Both should be UNSPECIFIED. A third case overlapped CLAIMED and INELIGIBLE; the shared instructions needed explicit category precedence.

I preserved the original dataset, prompts and all 80 pilot responses. Before another evaluation, I froze v2 with those two gold corrections and clearer shared definitions. Source sentences, case order, cash amounts, receipt labels and evidence sets stayed unchanged. This is post-pilot rubric repair, not a preregistered gold standard or proof of better prompting. The original 37/40 is not evidence of three established model reasoning failures.

Models Tested

The initial prompt comparison used google/gemini-3.7-flash. On October 3, 2026, plain ran 11:50:19–11:51:53 UTC; grounded followed 11:51:53–11:53:53 UTC. All 80 planned requests completed. Kaggle showed $0.39 used from the free $10 daily quota after the pilot and v2 comparison; no money was spent.

The v2 dataset SHA256 is 08e5fe18c05520bf1d6e18316c01641fe209822e34d1ccb1a2bb4ced0425f2bd. Offline regrading confirmed every saved grade and the planned case order. Grounded was designated the publication task before v2 results were known; plain is the control.

After Gemini reached a ceiling, I recorded an additional-model plan for GPT-5.4 nano and gpt-oss-20b on the same grounded task. These models were selected after the Gemini result, not preregistered before the pilot. Both first attempts stopped before evaluation: my setup assertion was fixed to Gemini's identifier. Those failures are retained. Task version 2 reads the model selected by Kaggle and runs only grounded; it preserves the exact v2 cases, prompts and grader. Each declared model then received one complete 40-case evaluation, with no performance-based reruns.

The completed task-version-2 executions use these observed public SDK identifiers:

Model identifier	UTC start–finish, October 3	Cases completed
`google/gemini-3.7-flash`	12:23:16–12:24:41	40/40
`openai/gpt-5.4-nano-2026-03-17`	12:26:53–12:27:33	40/40
`openai/gpt-oss-20b`	12:26:58–12:37:44	40/40

These saved-task executions are separate from the earlier Gemini prompt comparison and publication preflight. Repeated runs are retained separately, not pooled into a larger independent sample. All raw synthetic responses from the three version-2 task executions were preserved and independently regraded offline.

Findings

Same grounded task, three models

Model	Exact correct / 40	Both-correct pairs / 20	Invalid schemas	Availability correct / 40	Cash receipt amount correct / 40
Gemini 3.7 Flash	40	20	0	40	40
GPT-5.4 nano	25	8	0	31	40
gpt-oss-20b	30	12	2	38	38

All 120 requests completed with zero request errors. Every schema-valid response kept the cash receipt amount and currency correct. The two gpt-oss-20b schema failures were an empty response and a response with string evidence IDs instead of integer IDs. Under the strict grader, an invalid schema earns no field credit; its 38/40 receipt score therefore does not mean two accepted responses invented money.

Nano's nine availability mismatches were separate from its correct ledger. For example, an explicitly eligible, funded task with a future deadline became ELIGIBILITY_REQUIRED. A delivered USD 60 award was correctly recorded as 6000 cents, but the still-open program became CLAIMED. The amount of money received did not establish an exclusive assignment.

Nano omitted three stated cash maxima, including an EUR 20 offer. gpt-oss-20b missed the INR 10,000 cash award inside a mixed software-and-cash headline, returning a null amount and USD currency. Both returned UNSPECIFIED selection for a cash contest whose individual split was unstated: the per-person amount was unknown, but the contest selection method was still specified.

The unsupported-cash and invented-receipt diagnostics were zero for all three models. Those narrow counts would miss the omissions and availability errors above. Zero invented income is useful, but insufficient for choosing paid work.

Exact evidence requirements also affect the result. Nano had all seven extraction fields correct in 26 cases, versus 25 exact cases including evidence; gpt-oss-20b had 32 versus 30. These gaps describe mismatches with the author-defined complete citation set, not necessarily unsupported extraction fields.

Initial Gemini prompt comparison

Variant	Completed / 40	Exact correct / 40	Both-correct pairs / 20	Invalid schemas	Unsupported cash values	Invented cash receipts
Plain	40	40	20	0	0	0
Grounded	40	40	20	0	0	0

All seven field accuracies and evidence matches were 100% in both v2 runs, with zero request errors. The extra grounded reminder showed no measurable benefit on this dataset.

Three concrete distinctions survived both Gemini prompts:

A USD 2,500 pool with five stated USD 500 winners produced an individual maximum of 500. With an unspecified split, both prompts returned a null individual amount and retained USD as the cash currency. A pool did not become a promised personal payout.
An announced USD 60 award produced zero confirmed cash receipts. Its paired bank-delivery record produced 6000 USD cents. The advertised amount stayed 60: offer terms and receipts are different fields.
A delivered, redeemed merchant gift card still produced zero cash receipts because its source explicitly said no cash was transferred. That does not mean the card lacks value; this field measures cash, not every economic benefit.

The rubric review matters too. A scorer that demands unsupported selection terms can mark justified restraint as an error. Evidence discipline applies to the benchmark author as well as the model.

Limits

Gemini reached a ceiling on this small constructed diagnostic. The observed three-model differences do not establish a general ranking or estimate accuracy across real paid-work programs. Related pairs are not 40 independent observations. Status and delivery are often explicit; documents are English and the receipt ledger is USD-only. Evidence sets are author-defined, and observed responses informed the disclosed rubric repair. Only Gemini received both prompt variants; the additional models received grounded only. Provider behavior and effective generation settings were not independently controlled.

Independent label review, replicated comparisons, additional currencies, and permitted natural documents with less explicit status statements are worthwhile future measurements, not completed results here. A practical assistant should track advertised terms, eligibility, submission stage, award and delivered payment separately, with source evidence for each decision.

My Benchmark

Open Reward Evidence on Kaggle. The collection uses grounded task version 2, with average task scores shown on its leaderboard.

The cases and grader are original work. The runner uses the Kaggle Benchmarks SDK and its official quick start. Pilot and v2 artifacts are retained separately so the corrected result does not erase the original measurement.