DEV Community: Saifullah

Default Flag: an exoplanet agent that won't mix papers

Saifullah — Mon, 05 Oct 2026 00:15:25 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Default Flag answers derived exoplanet questions (density, insolation, equilibrium temperature, habitable-zone status) from one self-consistent published parameter set per planet, and shows, as a labelled counterfactual, the different answer you get by mixing values from several papers.

Live demo, no login: https://default-flag.vercel.app · Code: https://github.com/unicorn-9-spec/default-flag · Sanity project: 0eu544dk, dataset production (public)

The NASA Exoplanet Archive keeps every published solution for a planet as its own row and flags one as the default. It also publishes a composite table with one row per planet, filled parameter by parameter from whichever paper its rules pick. The archive itself calls that table "a more complete, though not necessarily self-consistent, set of parameters". Answers built on the composite row can combine a mass from one paper with a radius from another, or with a radius the archive calculated from that mass.

Default Flag never does that. Every number comes from deterministic code reading exactly one parameter set (plus the stellar values published with it). The composite answer appears only behind a "What the composite table would give" toggle.

Kepler-139 d. Weiss et al. 2024 gives a density of 5.29 g/cm³. Their mass is M·sin i, which the archive calls a lower limit, so the answer says the density is a lower limit. The composite row takes its mass from Lammers & Winn 2025 instead and gives 2.40 g/cm³: less than half, from two papers that never published together.
Proxima Cen b. No paper in the archive measures its radius, so Default Flag refuses and names the missing field. The composite row answers anyway, 5.49 g/cm³, from a radius the archive calculated from the minimum mass.
ups And c (a compounding case). The default set refuses for lack of a radius. The composite gives 13.19 g/cm³ from a true mass (McArthur et al. 2010) and a radius the archive calculated from that same mass: a density built on itself.

Four-arm evaluation

40 frozen questions (15 density, 10 insolation or equilibrium temperature, 5 habitable zone, 5 provenance, 5 refusals). Ground truth comes from the archive's default row in the raw snapshot CSV, independent of Sanity and the model. Same model, step budget and compute tool for every arm; only retrieval differs. Model gemini-3.7-flash, version 3.7-flash-08-2026.

Arm	Accuracy	Mixed-provenance rate	Correct refusals	Valid citations
Structured agent (Sanity Context)	98% (39/40)	0% (0/29)	5/5	98%
Semantic search	93% (37/40)	3% (1/32)	3/5	100%
Keyword (BM25)	98% (39/40)	3% (1/31)	4/5	100%
No content	45% (18/40)	35% (7/20)	3/5	0%

What the numbers say, plainly:

Structured agent: never mixed papers, and refused all five questions the data can't support.
Search baselines: they read the same structured documents, where each parameter set is one self-contained document, and they mixed far less than I expected: 3%. Their failure was different. Where the default set can't answer, they quietly answered from another paper's set instead of saying so. That's a provenance error the mixed-provenance metric doesn't count.
No content: the model mixed or invented its inputs 35% of the time.
The structured agent's one miss was an answer its own output guard withheld.

Why it's new

Plenty of agents surface contradictions between sources. This one prevents a specific, measurable error, mixing values from different papers into one derived number, and measures how often each retrieval approach makes it. Astronomy's per-paper parameter sets make the error checkable: every input value an arm used is matched back to the snapshot rows.

Demo

Judge path:

Click Density of Kepler-139 d? and read the answer and its provenance strip.
Turn on What the composite table would give.
Click Density of Proxima Cen b? and get a refusal that names the missing field.
Open /eval for the four-arm table.
Open /how-it-works for the Knowledge Base section.

The other pages:

/planet/<slug> puts every published parameter set side by side.
/eval has a drill-down per question.
/how-it-works has the endpoints, schema, snapshot checksums and the Knowledge Base evidence.

Every page has loading states and a banner naming the failing component (Sanity Context, dataset or model), never a silent fallback. The agent API is rate-limited.

Code

https://github.com/unicorn-9-spec/default-flag (README with CI badge)

ingest/: verify (SHA-256 manifests) → select → normalise → import. Snapshot files live in the repo; a second run makes zero changes.
agent/:
- physics.ts: density, insolation, T_eq, Kopparapu 2014 habitable zone, 10,000 seeded split-normal Monte Carlo draws;
- compute and composite_counterfactual tools;
- the output guard;
- the MCP wiring.
web/: the Next.js site.
eval/: questions, truth, four arms, scoring, results.

CI runs on every push:

snapshot checksums;
an ingest determinism check;
typecheck and lint;
38 unit tests, including physics against the archive's own insolation and temperature columns, guard tests and the offline evaluation subset;
GROQ validation (zero violations);
the Knowledge Base budget (≤ 150);
the build;
Playwright plus axe-core on the judge path, desktop and mobile.

Code computes, the model explains. The output guard is code, not prompt. It rejects an answer if:

any number in it doesn't appear in a dataset or compute result from that turn (Knowledge Base text is never accepted as a number source);
any document id or Knowledge Base path it cites wasn't read that turn;
any quote isn't exact.

It retries once with the list of violations, then withholds the answer.

What didn't work (from the build log)

The spec's premise was wrong on real data. I assumed composite radii would contradict default radii; across 6,375 planets they never do, because the archive keeps the default value whenever it has one. The real mixing is gap-filling (1,573 calculated radii) and minimum-mass to true-mass swaps (100 planets).
My first flagship, HD 10180 c, fell apart under checking. Its composite mass carries a limit flag with no uncertainties, and it has no radius at all.
The spec's dotted document ids (pset.k2-18-b.<ref>) would have made every document private: Sanity serves ids containing a dot only to authenticated users.
Context's dataset mode stayed "Not ready" until a Studio was deployed. sanity schema deploy alone wasn't enough.
The endpoint name was auto-filled from its title (default-flag-data), and it can't be changed.
Context allows one source per dataset, so the two planned Knowledge Base queries became one.
My own output guard rejected correct answers three times. It read "16th–84th percentile" ordinals as unsourced numbers, then rejected "1240" as a rounding of 1244. Each fix came with a regression test. The first evaluation run is kept in the repo next to the rerun.
A stale next start served HTML pointing at deleted CSS, and my accessibility test "passed" on the unstyled page.

Known limitations

Selection: 31 planets and 129 parameter sets, chosen by explicit rules (ingest/data/selection.json), not a population study.
Albedo: fixed Bond albedo of 0.3, a modelling assumption.
Habitable zone: limits are extrapolated below 2600 K (TRAPPIST-1), and the answer says so.
Limit flags: the archive's limit-flag sign convention isn't defined in the documentation I snapshotted, so any flagged value is refused.
Prose claims: the guard checks numbers, ids and quotes, not free-prose claims. One live answer added "detected via radial velocity", which is true but came from the model, not a tool.

How I Used Sanity

Structured content. The archive snapshot is modelled so that provenance is a property of the data, not of the prompt:

planet: references to its parameterSets, its defaultParameterSet, and a compositeSnapshot holding every composite value with its reference, or "Calculated Value".
parameterSet: one archive row: values with asymmetric errors and the archive's limit flag, massKind (Mass or M·sin i), and the stellar solution published with that row.
stellarSolution, publication, star.
snapshot: raw file, SHA-256, row count, retrieval time.
constant / threshold / hzLimit: each with its source and the location within it.
derivedAnswer: every input references its parameter set.

Studio validation and CI GROQ checks enforce one default per planet and single-set inputs for every derived answer. Document ids are deterministic and hyphenated, so the dataset stays anonymously readable.

Sanity Context: two MCP endpoints, because an endpoint serves one mode:

default-flag-data (dataset/GROQ mode over production). The agent discovers its tools with tools/list at runtime and gets initial_context, schema_explorer, groq_query and array_field_reader. It uses them to find the planet and list its parameter sets. It writes GROQ like *[_type == "planet" && name match "Kepler-139*"]{defaultParameterSet, "sets": parameterSets[]->{_id, defaultFlag, "paper": publication->citation}}. It then calls the local compute(setId, quantity) tool, which can only read that one set.
defaultflag-kb (Knowledge Base mode, Knowledge Base kbhX0D4yDJok), exposing initial_context, knowledge_base_search and knowledge_base_read. It is pointed at:
- one GROQ query that gives every planet two documents, its default parameter set and its composite row, with matching field names (62 documents);
- six archive documentation pages, uploaded as files: the default-set FAQ, how the composite table is calculated, the table overview, the column definitions, the acknowledgement page, and the DOI page.

That's 68 documents against the 150 budget, checked in CI. The agent searches it, then reads entries by path, resolved at runtime because paths change on rebuild. It uses them to explain the rules: why the default set governs, and what a minimum mass means. It never takes a planet value from it.

The Knowledge Base conflict. For Kepler-139 d the two documents disagree: 4.658 M⊕ (M·sin i, Weiss et al. 2024, the default) against 2 M⊕ (Lammers & Winn 2025, composite).

Context did not raise it in the Issues tab. The Knowledge Base Purpose states the rule: "the self-consistent single-source set governs; composite values are recorded only as explicitly labelled alternatives". The build applied it directly. The planet entries hold a "Default parameter sets (preferred for derived quantities)" table and a separate "Composite table (mixed-source alternatives)" table, and say outright that Kepler-139 d's composite mass differs from its default.

The answer that changed. Before that source existed, the Knowledge Base held only methodology and could say nothing about Kepler-139 d. Now, asked "Which mass should I use for Kepler-139 d, and why does the composite table give a different one?", the agent answers:

4.66 M⊕ from Weiss et al. 2024;
the composite's 2 M⊕ comes from a different paper;
citing kb:parameter_sets/default.

The computed density stays 5.29 g/cm³, by design: numbers come only from the dataset and deterministic code.

Where the advantage comes from. knowledge_base_search is itself keyword (BM25) search, so the structured agent's edge over the BM25 baseline isn't better Knowledge Base retrieval. It comes from GROQ over typed documents plus a compute tool that can only read one parameter set.

Sanity Project Details

Project ID: 0eu544dk
Dataset: production (public read, no token needed)
Public GROQ query: open it in the browser (the first five planets and their default papers, no token)
Hosted Studio: https://default-flag.sanity.studio
Knowledge Base: kbhX0D4yDJok
Context endpoints: default-flag-data (dataset mode) and defaultflag-kb (Knowledge Base mode)

No login anywhere on the demo site.

Agent Session

The whole build was done with Claude Code, and every step is in the public build log: prompts that worked, prompts that failed, where it got stuck and how it got unstuck. See docs/BUILD_LOG.md and the agent/coordinator exchange in docs/REPORT.md.

Data sources and licences

NASA Exoplanet Archive: Planetary Systems (doi:10.26133/NEA12) and Planetary Systems Composite Parameters (doi:10.26133/NEA13) tables, retrieved 2026-10-03, snapshot checksums in the repo.
Kopparapu et al. 2014 (habitable-zone coefficients, Table 1).
Rogers 2015 (1.6 R⊕ rocky threshold).
IAU 2015 Resolution B3 and IAU 2012 Resolution B2 (nominal solar constants, astronomical unit).

This research has made use of the NASA Exoplanet Archive, which is operated by the California Institute of Technology, under contract with the National Aeronautics and Space Administration under the Exoplanet Exploration Program.

Where Need Meets Nothing: finding Florida's aid deserts with Snowflake

Saifullah — Sun, 06 Sep 2026 22:54:37 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

Generosity has a targeting problem. Money follows attention, attention follows disasters, and disasters get covered where the cameras already are. The places that quietly need the most are often the places nobody is looking.

So I went looking for them. Two Florida counties:

County	FEMA disasters (10yr)	Poverty rate	Doctors per 10k
Alachua	21	21.2%	351.0
Glades	14	17.2%	20.7

Alachua has more disasters and more poverty than Glades. It also has 17 times the doctor density. Alachua contains Gainesville and a teaching hospital. Glades contains the western shore of Lake Okeechobee and not much else.

That gap is invisible in any single dataset. FEMA knows where the storms landed. The Census knows who is poor. The national provider registry knows where the doctors are. Nobody joins them — so nobody can point at a map and say there, send help there.

Aid Desert Finder joins those three datasets into one county-level view and ranks all 67 Florida counties by how badly need outruns resources. It answers the question a donor actually has: not "where is it bad," but "where is it bad and nobody is already there."

The answer for Florida: 9 counties, about 258,000 people, averaging 22 disaster declarations in ten years, one in five residents below the poverty line, and roughly half the state's median doctor density.

Demo

Live map → https://aid-desert-finder.vercel.app

The panhandle and the rural interior light up. The coasts and the university towns do not.

Hovering a county — on the map or in the table — dims the rest of the state and pulls up its numbers. The toggle asks the same 67 counties four different questions.

Code

Saifullahakjsnd / where-need-meets-nothing

Aid Desert Finder - joins FEMA disasters, Census poverty, and NPPES provider data in Snowflake to rank Florida counties where high need meets thin resources

Where Need Meets Nothing — Aid Desert Finder

Live map → https://aid-desert-finder.vercel.app

Finds the Florida counties where disaster exposure and poverty are high and healthcare resources are thinnest — the places most likely to be overlooked by donors, because attention follows cameras rather than need.

Built for the DEV Weekend Challenge: Generosity Edition. Category: Best Use of Snowflake.

County	FEMA disasters (10yr)	Poverty	Doctors per 10k
Alachua	21	21.2%	351.0
Glades	14	17.2%	20.7

Alachua has more disasters and more poverty than Glades, and 17× the doctor density. That gap is invisible in any one dataset. This joins three of them.

What it does

Ranks all 67 Florida counties on a composite need score — 35% disaster frequency, 35% poverty rate, 30% scarcity of doctors per capita, each as a percentile rank across the state. The result is an interactive choropleth plus a natural-language query box.

Headline finding: 9…

View on GitHub

Everything is there: the SQL that builds the joined view, the Cortex Analyst semantic model, the Streamlit in Snowflake app, and the generator for the standalone page.

How I Built It

Three federal datasets, zero ETL

The whole thing runs on the Snowflake Public Data (Free) Marketplace share. No pipelines, no downloads, no CSVs — three federal datasets already sitting there as queryable views:

FEMA_DISASTER_DECLARATION_AREAS_INDEX — which counties each declared disaster touched
AMERICAN_COMMUNITY_SURVEY_TIMESERIES — ACS poverty and population estimates
NPPES_PROVIDER_ADDRESSES — every registered healthcare practitioner's practice location

What makes the join tractable is that all three share one GEO_ID spine (Data Commons style — Florida is geoId/12, Alachua County is geoId/12001). No FIPS wrangling, no ZIP-to-county crosswalk scraped off a university website. The join is GEO_ID = GEO_ID.

Even the map geometry is in there. GEOGRAPHY_CHARACTERISTICS carries coordinates_geojson rows — real county polygons — so the choropleth uses zero external geodata. ST_SIMPLIFY at a 300m tolerance takes 4.5MB of boundaries down to 190KB with no visible difference at state zoom.

Asking it in English

Cortex Analyst handles the plain-English layer over a deliberately tiny semantic model — one view, six measures. The interesting part is what it does unprompted:

Asked "Which counties had disasters but fewest doctors?", it chose providers_per_10k over the raw headcount on its own — because the semantic model's description of that field says it's the right one for per-capita questions. Nothing in the question mentioned per-capita anything.

It generalises past its verified queries too: "more than 1 in 5 people in poverty" became poverty_rate_pct > 20, and "show me counties where fewer than 60 doctors per 10k" composed correctly with it. The whole semantic model is 108 lines of YAML.

Three traps that produced convincing wrong answers

The first working version gave numbers that looked entirely plausible and were wrong. Each of these only surfaced by checking output against reality rather than trusting it:

1. Doctors counted twice. The provider registry stores three address rows per practitioner — Mailing, Primary Practice, Secondary Practice. Counting all of them credits a doctor to both the county they live in and the county they work in. Gadsden County fell from 542 doctors to 436 once filtered to primary practice.

2. Two counties that structurally could not have doctors. Union County reported zero practitioners. Not "few" — zero, in a county of 15,700 people. Union has no ZIP children in the geography hierarchy, so a ZIP-based rollup can never assign it anyone. It ranked as a perfect aid desert because it was invisible. All 67 counties do have city children, so the fix was a city-level fallback. Union actually has 70.

3. A view built on temp tables. My first draft assembled the join in TEMPORARY tables and created a view over them. That compiles fine and breaks the moment the session ends.

The check that caught the rest: county populations sum to 22.4M against Florida's actual ~22.6M. Had that been off by millions, something in the join was silently dropping rows.

Scoring, and one map bug

The composite is a percentile rank across the 67 counties — 35% disaster frequency, 35% poverty rate, 30% scarcity of doctors per capita. Percentile ranks rather than divide-by-max, because Miami-Dade has 85,714 doctors and against that maximum every rural county rounds to zero.

The same skew bit the map. My first choropleth used equal-interval colour bins, and since Alachua sits at 351 doctors per 10k against a median of 118, nearly every county collapsed into the same two shades. Quantile bins fixed it — each of the seven steps now holds roughly a seventh of the counties, and the distribution is actually readable.

Honest limits

It's one state. The geography lookup resolves Florida by name, so another state is a one-line change — but the score is percentile-ranked within the state, so cross-state comparison needs a different normalisation.
"Doctor" is doing a lot of work. The registry counts every individual practitioner — nurses, therapists, chiropractors. Filtering by taxonomy to primary-care physicians would sharpen this a lot.
Practice location isn't service area. A county with no doctors next door to a hospital is less isolated than one 60 miles from anything. Drive-time isochrones would capture that; county lines don't.
Disaster counts aren't severity. A 22-declaration county might have had 22 minor events. FEMA's public-assistance dollars are in the same share and would weight this properly.

Prize Categories

Best Use of Snowflake. Three federal datasets joined through the free public data share with no ETL, Cortex Analyst for the natural-language layer, Streamlit in Snowflake for the in-platform app, and the county boundary geometry pulled from the same share so the map needs no external geodata.

Credits: the Cortex Analyst REST call pattern (send_snow_api_request) is adapted from Snowflake's own Cortex Analyst quickstart. Everything else was written for this challenge, in the challenge window. Built with Claude Code as a pair — it caught trap #2 by questioning a result that looked wrong instead of accepting it. Every figure in this post is a live query result.