DEV Community: ujja

AI Engineering Is Easy. So How Do We Change How We Work?

ujja — Thu, 15 Oct 2026 00:24:59 +0000

In Part 1, I wrote about the uncomfortable part of AI-assisted engineering: the tools are becoming remarkably good at producing things, while the harder question is what happens around them.

If an agent can write code in minutes, generate a plan in seconds, and create dozens of tasks before we've finished our coffee, our old way of working starts to look a little strange. Not because the process was bad, but because the bottleneck has moved.

We used to spend much of our time producing things. Now we need to spend more time deciding what should be produced, providing the right context, and verifying whether the result is actually what we wanted.

Recognising the need for change is one thing; figuring out how to make it without creating even more process is another.

I don't think we need an entirely new engineering methodology for AI. We need to make a few important parts of our existing process explicit and automate what we can. I think of this as building a harness around the agent.

Start with the work, not the agent

It's tempting to start with questions like, "Which agent should we use?" or "What's the perfect prompt?" I think that is backwards.

Start with the unit of work. What does the agent receive? What can it change? What does "done" mean? How do we know the result is correct, and what happens when it isn't?

These questions matter whether you're using Claude, Codex, Cursor, an internal agent, or something that doesn't exist yet. The tools will change, but the underlying method shouldn't have to.

A useful agent workflow has six basic components:

Context — what the agent needs to know
Tools — what it can act on
Loop — what one unit of work looks like
Verification — how we establish correctness
Guardrails — what it must not touch
Observability — what happened and what changed

You don't need a sophisticated platform to start. You just need to be explicit about these six things.

1. Make the boundaries boringly clear

Giving an agent access to everything can make the first demo impressive, but it can also make the first mistake much more expensive.

Before expanding its access, establish clear boundaries around:

Repositories and branches it can modify
Environments it can access and whether it can deploy
Secrets and external systems it can access
Actions that always require human approval

The exact list will vary, but the boundaries should be documented and enforced wherever possible.

If the answer depends on who's running the agent, you don't have a boundary; you have a convention. Conventions are easy to lose when people, tools, and teams change.

As agents become more capable, being deliberate about where their autonomy ends becomes just as important as deciding what they can do.

2. Stop treating gates as documentation

A process can have ten gates written down and still have none in practice. If someone can skip a step simply by forgetting it, it's an instruction rather than a gate.

A useful gate has three things: a condition that must be satisfied, someone accountable for checking it, and a mechanism that makes skipping it visible.

Automate what you can. Run specification checks automatically, validate consistency between artefacts, and stop the workflow when a required check fails.

Other decisions genuinely need human judgement, such as approving scope or determining whether a change meets the customer's needs.

The goal isn't to automate every decision. It's to automate what doesn't require one and make the remaining decisions explicit. That should reduce the burden on people rather than add another layer of process.

3. Don't put humans in the loop everywhere

"Human in the loop" can easily become "human reads everything the agent does". If someone has to inspect every intermediate step, we've simply added an AI assistant to the old workflow and asked a person to supervise every keystroke.

Instead, involve people where judgement matters and let automated checks handle what they can.

A workflow might look like this:

Agent produces
      ↓
Automated checks
      ↓
Human reviews decisions requiring judgement
      ↓
Agent continues

The human needs to determine whether the direction is right, the result matches the intended outcome, and the work is safe to continue.

This is also why the unit of work matters. A large ticket gives the agent a large surface area for mistakes, while a small, independently verifiable slice makes the work easier to understand and review.

The objective isn't to remove humans from the workflow, but to reserve their attention for decisions that benefit from their experience.

4. Make scope impossible to misunderstand

Agents are very good at doing exactly what you asked, which isn't always the same as doing what you meant.

For every piece of work, make three things explicit:

What's included in the scope
What's excluded from the scope
What defines done, including the evidence required

Exclusions are particularly important because a future-state idea can quietly become an implementation requirement while everyone assumes they're still discussing the same thing.

One piece of product intent can produce many implementation slices. The slice isn't the feature; it's the unit in which we deliver and verify part of it.

Without clear scope, an agent can optimise for completing the task in front of it rather than delivering the outcome we actually wanted. Good specifications help constrain those assumptions without requiring us to prescribe every implementation detail.

5. Turn repeated prompts into checks

If you keep telling an agent, "Remember to check X," ask whether X should actually be a check.

Prompts help communicate intent, but automated checks are more reliable for enforcing objectively verifiable requirements.

If a requirement is testable, test it. If two documents need to agree, compare them. If a branch must not be modified directly, enforce it. If a file must never be changed, block it. If evidence is required before completion, check for it.

Every time you add another paragraph to a prompt, consider whether that instruction could become an automated check instead.

Not everything can be automated, and prompts will remain useful for communicating context and intent. But when correctness can be checked mechanically, relying on the agent to remember an instruction is often weaker than enforcing it in the workflow.

Over time, prompts can become smaller while the system becomes more reliable.

6. Stop keeping the important context in people's heads

Another pattern that doesn't scale is:

"I've got a really good prompt that I use with my agent."

That's useful for an individual, but it's not a team process.

If an agent needs specific context to work correctly, that context should live somewhere shared, versioned, and reviewable. The same applies to skills, instructions, and workflows.

Otherwise, everyone builds their own operating system for their agent, and the team ends up with different behaviours while assuming everyone follows the same process.

Shared context doesn't mean identical prompts for every team. It means important rules shouldn't depend on someone's local setup.

We don't need to document everything, but information that repeatedly affects decisions should be available where the work happens. If it's important enough for an agent to make the right decision, it's probably important enough for the team to maintain and review.

7. Treat verification as a separate capability

As generation becomes cheaper and faster, verification becomes an increasingly important part of the workflow.

Asking the same model that produced something to confirm its correctness isn't enough. A confident explanation isn't independent evidence.

Depending on the work, verification might involve automated tests, static analysis, schema validation, comparisons against specifications, independent reviews, acceptance testing, or evaluations against known examples.

The principle is simple:

Don't measure how much the agent produced. Measure how confidently you can accept what it produced.

A thousand generated lines aren't particularly useful if nobody can establish that they're the right thousand lines. More output can even increase the review burden.

Productivity should therefore include how quickly we move from generation to justified acceptance, not just how quickly we generate something.

Building verification into small, independent work units also makes it easier to catch problems early, understand what went wrong, and correct them before they spread.

8. Make the workflow survive a tooling change

Building the entire engineering method around one CLI or agent creates unnecessary coupling. Tools, commands, and models will change, so the underlying workflow shouldn't need to change with every new release.

Instead of defining the method as "run command X, then Y, then Z", define the work itself: specify behaviour, resolve ambiguity, design the solution, decompose the work, implement a small slice, and validate it.

Then map those activities to whichever tools the team uses.

Today that might be one agent; tomorrow it might be another or a combination of agents and conventional automation. The workflow remains useful because it's based on engineering activities rather than a particular command sequence.

9. Start with the smallest useful harness

You don't need to redesign engineering before using AI. Start with one workflow and answer six questions:

Context: What does the agent need to know, and where can others find and review it?

Tools: What can it do, and which capabilities should remain outside its reach?

Loop: What's the smallest independently deliverable unit that creates useful progress?

Verification: How do we know the result is correct rather than merely plausible?

Guardrails: What must it never touch, regardless of who's running it?

Observability: What do we record so others can understand what happened, what changed, and why?

Once these are clear, try the workflow where the pain is already visible. You don't need to start with every team, repository, or a 40-page AI engineering policy.

Run it, identify where people still compensate manually, and fix those points before expanding the approach. The harness should evolve from actual friction rather than an attempt to design the perfect process upfront.

The goal isn't more process

There's a natural reaction to all of this:

"So we're adding more gates, more checks, more documentation and more meetings?"

That would miss the point.

Some meetings should become automated checks, some knowledge should become shared context, some repeated prompt instructions should become enforceable rules, and some large tickets should become smaller slices. Other work people currently do manually should simply disappear.

The process should become more explicit but less heavy.

There's also an irony here: AI makes it easier to generate plans, documentation, tickets, and checks, but that doesn't mean we need more of them. We still need to decide which artefacts and controls actually create value.

The opportunity isn't just getting a faster developer. It's redesigning the system that developer works inside, including the parts we've accepted for years simply because nobody had a compelling reason to change them.

A practical way to get started

If I were starting from scratch, I'd take these steps:

Pick one workflow rather than trying to change the entire organisation at once.
Define the six harness components: context, tools, loop, verification, guardrails, and observability.
Set the agent's boundaries and identify the three most important gates.
Automate what can be checked automatically, especially repetitive checks people often forget.
Assign an owner to every remaining gate so decisions have clear accountability.
Make scope and exclusions explicit rather than leaving the agent to guess what "done" means.
Break work into independently verifiable slices to keep the impact of mistakes manageable.
Separate generation from verification rather than relying on the agent to grade its own homework.
Measure where humans still get stuck and use those points to guide the next improvement.

The point isn't to build a perfect AI engineering process. It's to make the system good enough that, as agents improve, we can benefit from their capabilities instead of discovering that our surrounding process has become the new bottleneck.

The next bottleneck

AI-assisted engineering isn't primarily a tooling problem. The tools will continue to improve, but we still need to decide how much autonomy we want, where human judgement belongs, what evidence is enough to trust a result, and how to make those decisions visible to the team.

The teams that benefit most won't necessarily be the ones with the most powerful agents. They'll be the ones whose engineering systems provide clear work, sufficient context, appropriate boundaries, useful feedback loops, and reliable ways to verify the results.

That's what the harness is for: making it possible to let agents move faster without turning every increase in autonomy into more risk and review overhead.

We may spend less time producing code, plans, and documentation by hand, but we'll still need to think carefully about intent, boundaries, trade-offs, and evidence.

AI can make the work of building easier. Changing how we work is still ours to do.

I Built a Political Accountability Agent That Queries Sanity

ujja — Sun, 04 Oct 2026 11:51:16 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

True Oath is a political accountability ledger starting with Australia and designed to expand to other countries. It is designed to answer a question that ordinary keyword search cannot answer reliably:

What did a political party promise, what evidence shows what happened next, and where is the evidence still incomplete or contradictory?

The project models political accountability as linked, source-grounded records rather than as a list of unverified claims. A promise is connected to its manifesto, government, evidence records, dates, status, confidence, and source URLs.

The first jurisdiction is Australia. The initial evidence plan combines:

election manifestos and policy documents;
Australian Parliamentary Budget Office election-commitment costings;
federal budgets and budget papers;
parliamentary records and legislation;
departmental and statutory-authority reports;
official outcome statistics; and
carefully labelled independent assessments.

The interface presents each promise with a status such as Kept, Partially kept, In progress, Not started, Broken, Reversed, or Unverifiable. The system is intentionally conservative: an absent search result is not treated as proof that a promise was broken.

Why I Built It

Election season is arriving in one country after another, and the question many voters eventually ask is simple: what actually happened? Campaign language is memorable, but the details are scattered across budget papers, legislation, departmental updates, audits, statistics, court records, and later explanations. It is easy to remember a promise and much harder to audit it fairly.

True Oath is my attempt to make that audit navigable. The point is not to manufacture a score for every political claim. It is to preserve the promise, follow the evidence, show contradictions, and leave uncertainty visible when the record is incomplete. Can Sanity help us keep our sanity? That is the experiment, with the pun fully intended.

Demo

The True Oath Sanity Studio is deployed at:

https://true-oath.sanity.studio/

The Next.js web app is deployed to Vercel from the web/ directory:

https://true-oath.vercel.app/

The project is named true-oath, the clean true-oath.vercel.app domain is publicly reachable, and Vercel SSO deployment protection has been disabled for this read-only demo. During development, run it locally:

cd web
npm install
npm run dev

Its first screen is an investigative ledger: readers can search and filter the four promise files, select a case file, inspect confidence and linked evidence counts, and jump to the integrity method. The dashboard now includes a status-distribution chart, evidence-depth bars, and an average-confidence readout so the corpus can be scanned before a reader opens an individual file. Lucide icons, responsive layouts, keyboard-friendly controls, status colors, subtle motion, and mobile overflow states are included in the interface.

The Sanity dataset contains the first independently collected Australia corpus: 15 public sources, 4 promises, 4 evidence records, 4 milestones, 4 outcome indicators, 4 independent assessments, 3 competing claims, 1 manifesto, 1 government record, and 4 integrity events. The production dataset is now public-read for judging and API inspection. Public reads do not grant anonymous editing or uploads.

Code

Repository: https://github.com/ujjavala/true-oath

The repository contains:

web/ — the Next.js accountability interface;
sanity/ — the Sanity Studio and schema;
sanity/schemaTypes/index.ts — the structured content model;
README.md — local setup and deployment notes.

The project currently builds with:

cd web && npm run build && npm run lint
cd sanity && npm run build

How I Used Sanity

Sanity is the evidence layer for True Oath. The Studio schema separates the main concepts instead of flattening them into one article:

government records the governing party or coalition and its time period;
manifesto records the election document and the party that published it;
promise records a specific commitment and its current assessment;
source preserves the publisher, URL, date, and source category;
evidence records a dated finding and whether it supports, partially supports, contradicts, or is neutral toward a promise;
milestone records announcements, funding, legislation, starts, delays, cancellations, and delivery events;
indicator records measurable outcomes against a defined baseline, target, period, and unit;
assessment records an independent verdict and its reasoning; and
claim records competing explanations or interpretations without silently treating them as verified facts.
integrityEvent records corruption and integrity signals separately from promise verdicts, including mechanism, evidence status, official-finding status, confidence, source, and whether a demonstrated effect on implementation exists.

The current promise records are linked to milestones, outcome indicators, independent assessments, and competing claims. That means an agent can distinguish “a policy was announced”, “a measurable output changed”, “an assessor gave a verdict”, and “a government or public interpretation explains why” instead of collapsing those into one status field.

Sanity Services Used

Sanity Studio and schema. I used a hosted Sanity Studio as the editorial surface for the project. The schema is the first important design decision: it keeps a campaign promise separate from the source that announced it, the evidence that tests it, the milestone that records implementation, the indicator that measures an outcome, and the assessment that explains a verdict. The model also has claim and integrityEvent documents so competing explanations and corruption-related records are not silently merged into a promise score.

The core promise model is defined in sanity/schemaTypes/index.ts:

defineType({
  name: "promise",
  type: "document",
  fields: [
    defineField({name: "title", type: "string"}),
    defineField({name: "status", type: "string"}),
    defineField({name: "baseline", type: "string"}),
    defineField({name: "target", type: "string"}),
    defineField({name: "milestones", type: "array", of: [
      {type: "reference", to: [{type: "milestone"}]},
    ]}),
    defineField({name: "indicators", type: "array", of: [
      {type: "reference", to: [{type: "indicator"}]},
    ]}),
  ],
})

Content Lake and dataset. The structured records live in the separate True Oath project wak4l160, dataset production. The dataset is public-read so judges can inspect the records, but writes still require authorization. The current corpus has 57 documents: sources, promises, evidence, milestones, indicators, assessments, claims, integrity events, a manifesto, and a government record.

I imported the public records as linked Sanity documents instead of flattening them into one JSON blob:

npx sanity documents create data/australia.json \
  --project-id wak4l160 \
  --dataset production \
  --replace

That makes relationships queryable. For example, an assessment can point to a promise and its source, while an indicator can preserve the measurement period and unit independently from the political language that motivated it.

Embeddings. Dataset embeddings are enabled for the accountability text fields. This is intended to support conceptual retrieval when a question uses different language from the source, while the structured fields and GROQ-style filtering preserve exact relationships and status values.

Sanity Context and MCP. The project has a Context endpoint at:

https://api.sanity.io/v1/context/organizations/oqf9m6vy6/mcp/true-oath-context

The agent client in agent/mcp-client.mjs speaks JSON-RPC over MCP. It initializes a session, lists the tools, then supports both the GROQ-style route and the Knowledge Base route:

const tools = await call("tools/list")

if (available.includes("groq_query")) {
  result = await call("tools/call", {
    name: "groq_query",
    arguments: {query: accountabilityQuery},
  })
} else {
  const context = await call("tools/call", {
    name: "initial_context",
    arguments: {},
  })
  // Extract the Knowledge Base id, then search its entries.
  result = await call("tools/call", {
    name: "knowledge_base_search",
    arguments: {
      knowledgeBase,
      query: "promise evidence government milestone assessment integrity",
      return: "entries",
      limit: 10,
    },
  })
}

The endpoint currently exposes the Knowledge Base tools initial_context, knowledge_base_search, and knowledge_base_read. The Knowledge Base exists as kbgnQdlEqXlP, but its build is blocked by the organization’s beta index quota and currently reports zero entries. The public dataset and embeddings are ready; the remaining setup is to rebuild the Knowledge Base when quota is available or switch the Context endpoint source to wak4l160.production.

Sanity CLI and deployment. The Sanity CLI is used for schema builds, hosted Studio deployment, document import, dataset visibility checks, and embedding setup. The Studio is deployed at https://true-oath.sanity.studio/. The web experience is a separate Next.js app deployed at https://true-oath.vercel.app/.

Sanity Functions and Agent Actions

The project now uses Sanity's native automation layer for evidence-quality
review. A reviewTask document type gives the Studio a durable queue for
missing sources, broken references, stale or incomplete evidence, and integrity
status checks. The custom Studio Structure menu puts Review queue first and
filters it to open and in-review tasks.

The Blueprint in sanity/sanity.blueprint.ts provisions the
true-oath-review-automation Stack, a project-scoped robot token, and the
review-queue-on-content-change document Function. It watches published
evidence, milestones, indicators, assessments, and integrity events. When a
document needs attention, the Function creates a deduplicated task containing
the affected document reference, severity, explanation, and suggested next
step. It never changes a promise status or publishes a political verdict.

Agent Actions provide a second, deliberately constrained workflow. The helper
in sanity/scripts/draft-review-agent-action.mjs uses Sanity's schema-aware
Generate action to draft reviewerNotes for a selected review task. The action
requires the deployed schema ID and a server-side token, and Sanity's default
draft-only behavior remains enabled. A human must review the note and decide
whether to resolve, dismiss, or keep the task open.

This gives True Oath event-driven review, schema-aware drafting, and an
inspectable Studio workflow without asking AI to decide whether a promise was
kept. The Function and robot token were deployed through the
true-oath-review-automation Blueprint Stack and verified as completed.

Challenges With Sanity

The hardest part was not storing documents; it was choosing boundaries that preserve uncertainty. A budget measure is not the same thing as a delivered outcome, and a government explanation is not automatically an independent finding. That is why the schema has separate milestones, indicators, assessments, claims, and integrity events.

There was also a tooling mismatch to handle. The first agent draft expected a groq_query tool, but the configured Knowledge Base endpoint exposed initial_context, knowledge_base_search, and knowledge_base_read. The client now detects the available tool contract instead of assuming one response shape. Finally, Context authentication is organisation-scoped, while ordinary Content API tokens are project-scoped; keeping those credentials separate is important for both security and a working MCP connection.

The intended corpus is deliberately broad. It will combine election material with budgets, legislation, Hansard, committee reports, audits, regulator and court records, departmental progress reports, statistical releases, program dashboards, procurement records, official explanations, independent assessments, and competing claims. This lets the agent investigate not only whether a promise was met, but what changed, why delivery may have slipped, whether the stated reason is supported, and whether different sources contradict one another.

The integrity slice adds public NACC and ANAO material. It includes an official bribery finding connected to contract bidding, an official misuse-of-office finding, a conflict-of-interest audit, and a NACC case where a perceived conflict was not substantiated as abuse of office. The last example is intentional: True Oath must be able to say that a claim was investigated and not proven. It should not call something “rigged” merely because a news report or political claim uses that language. The agent must preserve the source's status and only connect an integrity event to a promise when the evidence establishes that link.

The intended Path One agent will connect to the True Oath Sanity Context MCP endpoint. It will retrieve the relevant promises and evidence, follow their source references, compare claims across documents, and answer questions such as:

Which 2022 Australian federal promises are still in progress?
Which commitments have official budget or legislative evidence?
Which promises have conflicting evidence from different sources?
Which assessments should remain Unverifiable because the promise was not specific enough?

This is the part that requires structured content. A keyword search can find the word “housing”, but it cannot reliably follow the relationship between a manifesto commitment, a budget measure, a later law, and an outcome assessment.

Planned Context configuration:

Source: True Oath Sanity production dataset
Scope: Australia political promises and evidence
Retrieval: Sanity Context MCP
Output rule: preserve source links, dates, confidence, and unresolved conflicts

Sanity Project Details

Project ID: wak4l160
Dataset: production
Dataset visibility: public (read-only to unauthenticated API clients; writes still require authorization)
Organization: oqf9m6vy6
Hosted Studio: https://true-oath.sanity.studio/
Studio app ID: uk5zu82laqqn2uoecp2yikrq
Knowledge Base public ID: kbgnQdlEqXlP
Blueprint Stack: true-oath-review-automation (ST-rietwjrd38)
Deployed Function: review-queue-on-content-change

Public dataset inspection endpoint:

https://wak4l160.api.sanity.io/v2025-08-15/data/query/production?query=count%28%2A%29

The True Oath read-only MCP endpoint now exists at https://api.sanity.io/v1/context/organizations/oqf9m6vy6/mcp/true-oath-context. The endpoint is authenticated and currently points to the created kbgnQdlEqXlP Knowledge Base, but that Knowledge Base reports 0 entries because the organization's beta index quota is exhausted. The final agent run therefore still requires either rebuilding that Knowledge Base after quota is available or switching the endpoint source to the public wak4l160.production dataset with embeddings and the accountability GROQ filter.

What Comes Next

True Oath starts with a small Australian corpus, but the underlying model is meant to travel. The next step is to add more federal elections, then extend the same source-linked structure to state governments and other countries where reliable public records are available. Each new jurisdiction would bring its own election documents, budget conventions, laws, agencies, statistical releases, and standards for official findings.

Future versions could add time-series indicators, parliamentary voting histories, procurement and grant data, scheduled milestone reminders, and comparison views that show how different sources describe the same event. The current Function queue and draft Agent Action create a foundation for those extensions, while a richer agent could explain why two claims conflict, identify the missing document that would resolve an uncertainty, and carry a human reviewer’s resolution forward as a durable editorial decision in Sanity.

The hard problems are as important as the features. A promise can be vague, a target can change, an outcome can be affected by events outside a government’s control, and a media report can flatten a complicated audit into a dramatic headline. Sources can disagree without one being obviously false. That is why True Oath keeps confidence, provenance, dates, status reasons, and “no finding” cases visible instead of forcing every record into a binary score. The current Knowledge Base indexing quota and the work required to curate a public MCP session are practical constraints too.

The goal is not to replace political judgment with an automated verdict. It is to make judgment better informed, easier to inspect, and easier to revise when the evidence changes. If Sanity can keep the content structured and the agent can keep its claims tied to that structure, voters get something more useful than another feed of opinions: a living record they can question.

A Sanity Check for AI-Generated Cyber Attack Reconstructions

ujja — Sat, 03 Oct 2026 08:31:17 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange

What I Built

Cyber Autopsy is a cyber-forensics investigation room for asking structured
questions about an incident: what happened, what came before it, which evidence
supports it, what remains unknown, and what was knowable at a specific point in
time.

The project started as a Kaggle-style benchmark for evaluating whether an AI
can reconstruct an attack from fragmented reports without presenting guesses as
facts. I added Sanity as the structured content layer, an MCP-compatible
investigation server, and a forensic-themed web interface.

The core model preserves the relationships that make an investigation useful:

Evidence ──supports──> Event ──precedes/enables/causes──> Event
Evidence ──supports/contradicts──> Claim

The UI presents each case as a small investigation console. Investigators can
select a case, read its incident description, ask a question, and inspect the
result as a reconstruction, structured event relationships, uncertainty list,
contradictory claims, and expandable evidence with source provenance.

Why this helps cybersecurity investigations

Cybersecurity autopsy is an evidence-reconstruction problem. Analysts rarely
receive one perfect timeline; they receive authentication logs, endpoint
observations, threat reports, analyst notes, negative findings, and competing
claims. The difficult part is deciding how those pieces relate without
overstating what they prove.

Cyber Autopsy uses Sanity to make that reasoning inspectable. An analyst can
follow an event back to its supporting evidence, move through the attack graph,
check whether a claim is contradicted, and distinguish confirmed, inferred,
attempted, and unknown. This is useful for:

reconstructing initial access, lateral movement, persistence, exfiltration, and impact
producing incident timelines that retain provenance instead of flattening everything into a narrative summary
testing what investigators could have known at a day-one cutoff
comparing the same evidence under different actor framings
teaching analysts and evaluating agents on calibrated uncertainty

The result is closer to a digital incident autopsy than a generic search box:
the system preserves the chain of evidence, the gaps in the record, and the
reasoning boundaries around each conclusion.

Demo

The deployed demo is available at https://cyber-autopsy.vercel.app.
It serves the forensic UI and the Sanity-backed /api/cases and
/api/investigate routes and is publicly reachable for the competition demo.

Run the project locally when you want to inspect or modify it:

npm run dev

Then open http://localhost:3000.

Code

The repository contains the Python benchmark, Sanity Studio, importer, MCP
server, local UI, tests, and documentation.

Repository: https://github.com/ujjavala/cyber-autopsy

Deployment: https://cyber-autopsy.vercel.app

The most relevant implementation files are:

studio-cyber-autopsy/schemaTypes/index.ts  Sanity content model
scripts/sanity-seed.mjs                    repeatable JSONL importer
agent/sanity-context.mjs                   scoped GROQ investigation context
agent/mcp-server.mjs                       MCP tool surface
web/app.js                                 investigation UI behavior
web/style.css                              cyber-forensics console theme

My Build Process

AI-native IDE

I built the project collaboratively in Codex, using the repository as the
shared workspace. I started by reading the existing benchmark, data model,
frontend, Sanity Studio, and environment configuration. The implementation was
then developed in small verified phases so each change could be tested against
the real Sanity dataset and the existing Python benchmark.

Prompts that worked

The most effective prompts were specific about both the user experience and
the evidence model:

Integrate the existing Cyber Autopsy benchmark with Sanity. Preserve the
relationships between evidence, events, claims, sources, and investigation
cases. Add a repeatable importer and a context layer that can answer scoped
questions without leaking later evidence into temporal-cutoff cases.

The Sanity dataset can be empty or partially populated. Make the UI handle
empty arrays and API errors explicitly instead of throwing when reading length.
Test every investigation scenario after the fix.

Make the interface feel like a cyber-forensics lab: improve hierarchy,
case descriptions, icons, status signals, loading feedback, responsive layout,
and subtle motion while keeping the investigation workflow easy to scan.

Where the build got stuck

The first Sanity-backed UI path assumed that the selected case always had a
populated evidence array. When the dataset was empty or the query returned an
unexpected shape, the browser surfaced:

Cannot read properties of undefined (reading 'length')

The correction was to make the context and UI defensive: seed and validate the
dataset, distinguish a missing case from an empty evidence scope, and render a
clear error or empty state instead of dereferencing an undefined array.

Another important correction was temporal scoping. CASE-004 shares the
RansomHub incident with CASE-001 but only exposes day-one evidence. The context
now filters events, relationships, claims, and evidence against the selected
case boundary before returning the investigation result.

Course corrections and testing

The build evolved through these corrections:

Added seven Sanity document types instead of storing one large incident blob: incidents, sources, evidence, events, relationships, claims, and investigation cases.
Added explicit references and reverse-reference patches so provenance can be followed in both directions.
Added scoped GROQ queries and MCP tools for case listing and investigation.
Added case descriptions because the case IDs alone were too cryptic for a human investigator.
Reworked the UI into a forensic console with status badges, trace labels, loading state, evidence panels, and uncertainty sections.
Added a favicon, README startup guide, Sanity integration write-up, and challenge-specific documentation.
Ran the full Python suite, Sanity import validation, Studio build, syntax checks, browser scenarios, and browser console-error checks.

The final verification included 229 Python tests, a Sanity inventory of 457
production documents, live CASE-001 and CASE-004 investigations, evidence
provenance checks, cutoff checks, and zero browser console errors. The hosted
Sanity Context Knowledge Base was also created and checked through its
read-only MCP tools; that hosted integration is the Path One capability, while
this post focuses on the strange forensic product and its build process.

The Sanity layer also now includes a native review workflow. The Studio schema
has a reviewTask document type, and the custom Structure menu exposes an open
Review queue for unresolved evidence-quality work. A Blueprint deployed the
cyber-autopsy-review-automation Stack with a project-scoped robot token and a
review-queue-on-content-change Function. It checks published evidence,
events, relationships, and claims for missing source/evidence links and creates
deduplicated tasks without changing forensic verdicts.

The repository also includes a draft-only Agent Action helper. Sanity's
schema-aware Generate action can draft reviewer notes for a selected task, but
it cannot publish the task, change its severity, or decide attribution. That
separation lets the build use Sanity automation while keeping the investigator
in control of the final interpretation.

Sanity integration in the build

How Sanity helped

Sanity gave the project a durable, queryable content layer for the benchmark's
investigation graph. Before the integration, the dataset was primarily a group
of local JSONL files. That was reproducible, but it made the content harder to
inspect, connect, and reuse across an agent, a Studio, and a browser UI.

With Sanity, the same investigation content is modelled as linked documents.
Evidence, events, claims, relationships, incidents, and sources can be queried
independently or traversed together. This helped in four practical ways:

Grounding: an MCP-capable agent can retrieve the exact evidence and relationships relevant to a case instead of relying on a loose text search.
Traceability: every result can point back to evidence and source provenance, which makes an analyst's review possible.
Scope control: case-specific references and cutoff rules can be enforced in the query layer, preventing later facts from leaking into an earlier reconstruction.
Reuse: the same structured content model powers Sanity Studio, the importer, MCP server, validation workflow, and forensic UI.

Sanity therefore acts as the investigation knowledge base, while the context
layer acts as the reasoning boundary. The agent is not asked to remember an
incident or invent a timeline; it queries structured content and returns a
calibrated reconstruction.

Sanity features used

1. Structured document schemas

Instead of storing one large incident document, I modeled the investigation as
separate document types. This matches the way a forensic analyst works: an
incident is the case context, evidence is an observation, an event is a
reconstruction, a claim is an assertion, and a relationship explains how two
events connect.

The separation gives each kind of information a clear validation and query
boundary. For example, an event has a status and confidence, while an
evidence document has a timestamp, observation type, entities, and source
references. A relationship has a source event, target event, relation type,
and supporting evidence.

2. References and graph-shaped content

The most important Sanity feature here is the reference field. Evidence does
not copy an event's text, and a relationship does not copy both event records.
They point to the canonical documents:

defineField({
  name: 'sourceEvent',
  type: 'reference',
  to: [{type: 'event'}],
  validation: (rule) => rule.required(),
})

defineField({
  name: 'evidence',
  type: 'array',
  of: [defineArrayMember({type: 'reference', to: [{type: 'evidence'}]})],
})

This turns Sanity into a navigable investigation graph. The agent can traverse
case -> incident -> event -> evidence -> source, then separately retrieve
relationship and claim documents for the same incident. In the UI, that
becomes a reconstruction with visible provenance instead of an unsupported
paragraph.

3. Field validation and controlled statuses

The Studio schema validates required identifiers and references, constrains
confidence to the range 0..1, and uses controlled options for evidence and
event status. The available status taxonomy is deliberately forensic:

confirmed · inferred · attempted · failed · unknown

Claims additionally support contradicted. These values are used in the UI to
separate established facts from uncertain or failed steps. This prevents a
missing observation from silently becoming a confirmed event and makes
calibrated uncertainty part of the content model.

4. GROQ projections and dereferencing

The context layer uses GROQ projections to request only the fields required by
an investigation. The -> operator dereferences related documents so one
response can contain the case, incident summary, evidence, and source metadata
without the browser making a request for every individual record.

For example, the case query projects the incident and joins its evidence and
events:

*[_type == "investigationCase" && caseId == $caseId][0] {
  caseId, mode, temporalCutoff, framingActor,
  incident->{incidentId, title, description, actor, impact},
  evidence[]->{evidenceId, timestamp, type, description, status,
    source[]->{sourceId, title, publisher, url}},
  events[]->{eventId, description, timestamp, status,
    evidence[]->{evidenceId}}
}

This is useful for security analysis because the response remains structured.
The agent can filter on timestamps, statuses, evidence IDs, and relationship
types instead of trying to recover those distinctions from prose.

5. Parameterized, scoped queries

Queries are parameterized with caseId and incidentRef. The selected case
defines the visible evidence set, and the context derives a set of allowed
evidence IDs before returning events, relationships, or claims. A relationship
is included only when its supporting evidence and both endpoint events are in
scope.

This is how the implementation handles temporal safety. CASE-004 is not just
a label in the UI; its Sanity references define the evidence boundary. A
day-one question can also apply an additional D1 filter before the result is
rendered. The model cannot accidentally cite a later ransomware event as if it
were known on day one.

6. Sanity Content API

The Node context uses Sanity's Content API with the project ID, dataset, GROQ
query, and optional server-side token. The token is loaded from .env on the
server and is never exposed to the browser. The browser talks to the local web
server, while the local server talks to Sanity.

That separation gives the project a safer shape:

Browser UI → local investigation API → Sanity Content API
                                      ↑
                              server-side token

The same API boundary also makes failures explicit. A missing case produces a
clear error, while an empty evidence set is handled as an empty investigation
scope rather than causing a client-side length exception.

7. Sanity Studio, Structure Tool, and Vision

The project includes a standalone Sanity Studio configured for the same project
and production dataset. The Structure Tool provides the editing workspace
for the seven document types, with previews showing useful forensic identifiers
such as CASE-004, E-021, or an event description.

The Vision Tool is enabled for inspecting and testing GROQ directly in Studio.
That is useful when developing an investigation query: I can verify a case
projection, inspect references, and test a cutoff query against the real
dataset before wiring it into the MCP context.

8. Transactional mutations and repeatable import

The importer uses Sanity mutations in batches rather than hand-editing hundreds
of documents. It creates or replaces the normalized documents in dependency
order, then applies reverse-reference patches for evidence-to-event,
evidence-to-claim, and contradiction links.

The import is repeatable and testable:

npm run sanity:validate       # validate source inventory
node scripts/sanity-seed.mjs --dry-run
npm run sanity:seed           # write the content to Sanity

This gives the benchmark a reproducible migration path while keeping the live
knowledge base editable in Studio. It also means the dataset can be rebuilt if
the schema gains another forensic field later.

9. MCP as the agent boundary

Sanity stores and retrieves the content; the MCP server exposes investigation
capabilities to an agent host. The server provides list_investigation_cases
and investigate_case(caseId, question). Internally, those tools call the same
Sanity context used by the web UI.

This division is intentional. Sanity is responsible for content modeling,
relationships, querying, and provenance. The context layer is responsible for
case scoping and graph filtering. The MCP layer is responsible for making those
grounded operations available to an agent. The agent can therefore reason over
retrieved evidence without receiving unrestricted access to the whole dataset.

10. Functions, Blueprints, and Agent Actions

The project now uses Sanity's native automation layer for content quality
control. A reviewTask document type gives the Studio a durable queue for
missing sources, missing evidence, broken graph links, temporal review, and
conflict review. The custom Studio structure puts Review queue at the top and
shows open and in-review tasks first.

The Blueprint in studio-cyber-autopsy/sanity.blueprint.ts provisions the
cyber-autopsy-review-automation Stack, a project-scoped robot token, and the
review-queue-on-content-change document Function. It watches published
evidence, events, relationships, and claims. When an expected link is missing,
the Function creates a deduplicated reviewTask with a severity, explanation,
suggested action, and reference to the affected document. It does not rewrite
the incident graph, decide attribution, or publish a conclusion.

Agent Actions are used as a separate, draft-only reviewer assist. The helper in
studio-cyber-autopsy/scripts/draft-review-agent-action.mjs asks Sanity's
schema-aware Generate action to draft concise reviewer notes for one task. It
requires the deployed schema ID and a server-side write token, targets only
reviewerNotes, and leaves the task status, severity, and final decision to a
human reviewer. Sanity's default draft behavior is deliberately preserved.

The Function and robot token are deployed through Blueprint rather than hidden
in the web app. This keeps write credentials server-side and makes the
automation inspectable in the Sanity project. Both remote deployment operations
completed successfully.

Content model

Sanity stores seven document types:

incident: case narrative, actor, impact, confidence, and source references
source: publisher, URL, publication date, type, and reliability
evidence: an observation with timestamp, status, description, and provenance
event: a reconstructed action or state with evidence references
relationship: a typed link between events
claim: an assertion that can be supported or contradicted
investigationCase: a benchmark scope with mode, cutoff, and actor framing

The schema is defined with Sanity's typed helpers:

defineField({
  name: 'supportsEvents',
  type: 'array',
  of: [defineArrayMember({type: 'reference', to: [{type: 'event'}]})],
})

defineField({
  name: 'relationship',
  type: 'string',
  options: {list: ['precedes', 'enables', 'causes', 'depends_on']},
})

IDs from the benchmark remain explicit and stable: INC-001, E-001, N01,
and CASE-004. This makes the imported content easy to inspect in Studio and
keeps evidence references understandable in the investigation output.

Importing the knowledge base

The JSONL files under data/ remain the reproducible source of truth. The
importer maps them into Sanity documents:

incidents.jsonl       → incident
sources.jsonl         → source
evidence.jsonl        → evidence
attack_graphs.nodes   → event + claim
attack_graphs.edges   → relationship
benchmark_cases.jsonl → investigationCase

The importer writes documents in dependency order and applies reverse evidence
references in a second pass because events and evidence point to each other.

npm run sanity:validate
node scripts/sanity-seed.mjs --dry-run
npm run sanity:seed

The current Sanity production dataset contains 457 documents: 445 imported
Cyber Autopsy documents plus 12 existing project documents. A curated
150-document source from this dataset is currently indexed in the hosted
Context Knowledge Base.

Context queries and agent behavior

agent/sanity-context.mjs uses scoped GROQ queries. It does not download the
entire dataset for every question:

*[_type == "investigationCase" && caseId == $caseId][0] {
  caseId, mode, temporalCutoff, framingActor,
  incident->{incidentId, title, actor, actorType},
  evidence[]->{evidenceId, timestamp, type, description, status,
    source[]->{sourceId, title, publisher, url}},
  events[]->{eventId, description, timestamp, status,
    evidence[]->{evidenceId}}
}

The context then resolves relationships and claims and filters them against the
case's allowed evidence IDs. That filtering is the temporal safety boundary:
CASE-004 can answer what was knowable at the end of day one without leaking
later ransomware evidence from the full case.

The MCP server exposes two tools:

list_investigation_cases
investigate_case(caseId, question)

The investigation function maps question intent to structured graph operations:
predecessors, enablers, supporting evidence, statuses, cutoff checks, actor
framing, and contradictions. The result is deterministic, grounded in Sanity
content, and designed to be passed to a model without requiring another model
API key in this repository.

Sanity Project Details

Project ID: 41l9o4xn
Dataset: production
Organization ID: oqf9m6vy6
Knowledge Base ID: kbZHPSeMHnZt
Hosted Studio: https://cyber-autopsy-ujjavala.sanity.studio/
Blueprint Stack: cyber-autopsy-review-automation (ST-6kxszxpvyw)
Deployed Function: review-queue-on-content-change

The public Sanity project details are intentionally included so the structured
content model can be inspected. The hosted Studio is deployed from the Sanity
Studio schema, and the Context Knowledge Base is built from a curated source
of the production dataset. The server-side tokens stay in .env, are ignored
by Git, and are never sent to the browser.

Agent Session

The local agent is available through the MCP server:

npm run agent:mcp

The project also has a hosted Sanity Context MCP endpoint for agents that can
connect to MCP. It is the read-only retrieval path used for the Path One
integration and exposes the Knowledge Base through initial_context,
knowledge_base_search, and knowledge_base_read. The endpoint URL is kept
in the repository's environment configuration rather than hard-coded into the
application, and no token is included in this post. The curated native-session
record is included inline below for transparency.

Prompt: Query the hosted Sanity Context Knowledge Base read-only and explain
how CASE-001 moved from initial access to ransomware deployment.

MCP calls: initial_context -> knowledge_base_search -> knowledge_base_read.
The search returned RansomHub-related records and the read returned
dataset-backed provenance identifiers. The agent preserved the boundary that
retrieved content must not be stitched into unsupported claims. Codex reached
the real content but hit its usage limit before the final narrative answer.

The browser UI uses the same investigation context and exposes the workflow in
a more approachable way:

Choose a case from the case selector. The description explains the incident before you ask a question.
Read the case badges for full, temporal cutoff, progressive, counterfactual, or actor-framing scope.
Enter a question or choose a suggested investigation prompt.
Run the investigation and inspect the trace-numbered reconstruction.
Expand evidence records to verify the source and provenance yourself.

Useful sessions to capture for the final submission are:

CASE-001: How did the attacker move from initial access to ransomware deployment?
CASE-001: What evidence supports the Rclone event?
CASE-004: What could we have known at the end of day one?
CASE-011/012: What does the evidence establish regardless of actor framing?
CASE-001: Are there conflicting claims?

Local Verification

The current build has been checked with:

.venv/bin/pytest -q
npm run sanity:validate
npm --prefix studio-cyber-autopsy run build -- --no-auto-updates

The browser smoke checks cover the full investigation, the CASE-004 temporal
cutoff, case descriptions, evidence rendering, and zero browser console errors.

I Benchmarked 10 AI Models to Reconstruct Real Cyber Attacks

ujja — Thu, 01 Oct 2026 15:33:59 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Cyber incident reports rarely arrive as a clean, complete timeline. They describe what investigators or security teams could observe, mix those observations with analysis, and leave some questions unanswered. That is exactly where an AI-generated summary can become risky: it may sound convincing while quietly turning an inference into a fact or filling a gap with an event that the evidence never established.

I built Cyber Autopsy to test a narrower, practical question: when given pieces of a reported cyber incident, can a model reconstruct what happened while showing which evidence supports each claim and admitting what remains uncertain? The model must produce a timeline, connect events with causal or temporal links, cite evidence, and distinguish confirmed activity from inference, failed attempts, contradictions, and unknowns. A plausible attack story is not enough; unsupported certainty should count against it.

I chose this problem because incident reconstruction depends on more than recognizing familiar attack techniques. An analyst needs to know whether a step was observed or inferred, whether an attempt actually succeeded, and how one event led to another. Those distinctions can disappear in a fluent summary. Measuring them separately makes it easier to see whether a model is recovering the evidence or merely telling a likely-sounding story.

The first evaluation uses seven tasks built from four public reports. They include a detailed human-operated ransomware intrusion and vendor-reported campaigns involving AI-assisted activity. These are useful real-world case studies, but they are not a controlled contest between human and AI attackers: the reports differ in detail, evidence source, and corroboration. I also included two versions of one case with identical evidence but different actor framing, to see whether that wording changes the model's reconstruction.

This is a pilot, not a claim that AI attackers are more or less capable than people. It evaluates model reconstructions of reported incidents, not live attack behavior. The scorer is deterministic and reports an Evidence-Grounded Reconstruction Score (EGRS), alongside event recall and precision, causal-link quality, evidence attribution, status accuracy, uncertainty calibration, and hallucination-related measures. The temporal cutoff and framing pair are exploratory comparisons; only the framing pair holds the evidence fixed.

Real-world case studies

The seven tasks are built from four public incident reports, not invented scenarios:

RansomHub intrusion (CASE-001 and CASE-004): The DFIR Report's “Hide Your RDP: Password Spray Leads to RansomHub Deployment” describes a human-operated intrusion using password spraying and RDP, credential access, Rclone exfiltration, and eventual RansomHub deployment. CASE-004 reuses this incident but cuts off the evidence at the end of day one.
GTG-1002 espionage campaign (CASE-002, CASE-011, CASE-012): Anthropic's incident report and technical report describe an alleged AI-orchestrated campaign against roughly 30 targets. CASE-011 and CASE-012 use identical evidence with human versus AI-agent framing; they test framing sensitivity, not whether the real-world actor was human or AI. The campaign details and attribution are vendor-reported, not independently verified victim-side telemetry.
GTG-2002 “vibe hacking” extortion (CASE-003): Anthropic's August 2025 misuse report describes a Claude Code-assisted data-extortion operation affecting at least 17 organisations. The ransom-note images in that report were simulated recreations and are excluded from the benchmark evidence.
AI-enabled credential harvesting (CASE-013): Google GTIG/Mandiant's September 2026 report describes an AI-assisted campaign that reportedly harvested thousands of credentials in under six hours. The victim and model are undisclosed, and the claims remain vendor-reported.

These are real reported incidents, but the evidence quality is not uniform: the RansomHub case is reconstructed from host and network telemetry described by The DFIR Report, while the AI-actor case studies rely on security-vendor reporting. The benchmark labels that distinction rather than treating the cases as equally observed or directly comparable.

The seven tasks, in plain language

The short IDs are just labels: INC means the source incident, and CASE means the particular benchmark task. Each task gives the model an evidence packet and asks for the best-supported reconstruction, not a free-form guess.

CASE-001, the full RansomHub intrusion: Put the reported activity in order, from password spraying and remote access through the later intrusion and ransomware deployment. The reference reconstruction contains 28 events.
CASE-002, the GTG-1002 espionage report: Reconstruct Anthropic's account of the reported campaign, including what the operators attempted, what succeeded or failed, and what the report does not establish. The reference contains 17 events.
CASE-003, the extortion operation: Reconstruct Anthropic's shorter account of a Claude Code-assisted data-extortion operation. This report has less step-by-step detail, so the reference reconstruction is smaller, with 8 events.
CASE-004, only the first day of the RansomHub case: Revisit CASE-001 with later evidence removed. The model should not be penalized for events the supplied evidence cannot yet support; only 15 events are scored.
CASE-011, GTG-1002 framed as human-operated: Use the campaign evidence while describing the operator as human-led.
CASE-012, the same evidence framed as AI-operated: Keep the evidence identical to CASE-011 and change only the actor framing. Comparing these two scores gives an early look at sensitivity to wording; it cannot tell us who really operated the campaign.
CASE-013, AI-enabled credential harvesting: Reconstruct Google's public account of a reported credential-harvesting campaign, keeping the sequence and links grounded in what that report says. Its reference reconstruction contains 7 events.

Models Tested

I ran ten models from several providers against the same seven Kaggle tasks: Gemini 3.7 Flash, Gemma 4 26B A4B, GLM-5, Grok 4.20 Reasoning, GPT-5.6 Luna, GPT-5.6 Sol, GPT-5.4 mini, Claude Sonnet 5, Claude Opus 5, and Qwen 3 Coder 480B. The current benchmark view has a score for every model-task pair. Kaggle's overall score aggregates these seven tasks, which include related variants of the same incidents.

The table is the Kaggle leaderboard snapshot fetched on 2 October 2026, after duplicate and failing task attachments were removed and the earlier evaluated versions restored. CASE-001 through CASE-011 use v3; CASE-012 and CASE-013 use their republished v1 versions. Values are EGRS percentages (Kaggle's 0-1 scores multiplied by 100). EGRS rewards recovering supported events and links, citing evidence, and representing uncertainty, while penalizing unsupported events. It is specific to this evidence-reconstruction task, not a general measure of intelligence or cybersecurity ability. Kaggle's overall score now matches the equal-weight mean across the seven task rows. Since some tasks are related variants of the same incidents, this is descriptive rather than an independent-sample leaderboard.

Findings

Kaggle task	Gemini Flash	Gemma 4	GPT-5.6 Luna	GLM-5	Grok 4.20	Claude Sonnet 5	Claude Opus 5	GPT-5.6 Sol	GPT-5.4 mini	Qwen 3 Coder
CASE-001: Full RansomHub intrusion	70.55	82.38	71.27	81.31	75.71	73.75	78.14	78.06	66.06	70.90
CASE-002: GTG-1002 espionage campaign	79.72	77.21	80.31	76.22	83.60	65.05	68.16	76.71	69.68	76.30
CASE-003: Reported data-extortion operation	76.31	92.11	84.50	83.80	84.97	73.11	71.50	76.12	84.35	70.31
CASE-004: RansomHub, first-day evidence only	79.57	84.79	80.70	72.37	80.42	74.00	72.66	74.51	72.55	63.70
CASE-011: GTG-1002, human framing	85.44	80.84	82.04	72.80	80.12	77.09	65.13	76.43	66.80	66.80
CASE-012: GTG-1002, AI-agent framing	77.13	76.91	79.82	76.83	70.87	78.42	69.74	69.42	68.48	67.53
CASE-013: Google's reported credential-harvesting campaign	89.33	88.33	88.75	82.50	87.83	77.50	52.47	79.07	78.50	68.25
Kaggle overall	79.72	83.22	81.06	77.98	80.50	74.13	68.26	75.76	72.35	69.11

These are single runs, not stable model rankings. Gemma has the highest displayed overall score (83.22), followed by GPT-5.6 Luna (81.06) and Grok 4.20 (80.50). The standout case score is Gemma's 92.11 on the short extortion-report task; that is a case-specific result, not proof of general model superiority.

Several patterns stand out in these single runs:

Removing duplicates changed the aggregate, not the case results. With one row per task, Kaggle's overall now matches the simple seven-task mean. The model order consequently differs from the earlier duplicate-inflated view; Grok is third overall in this snapshot.
The winner changes by case. Gemma leads CASE-001 (82.38), CASE-003 (92.11), and CASE-004 (84.79); Grok leads CASE-002 (83.60); Gemini leads CASE-011 (85.44) and CASE-013 (89.33); GPT-5.6 Luna leads CASE-012 (79.82). The overall leader is not the top model on every task.
The first-day cutoff scores higher in this snapshot. Gemini scores 79.57 on the day-one CASE-004 and 70.55 on full CASE-001, a 9.02-point gap. CASE-004 has a smaller gold graph (15 vs. 28 events), so this does not show that less evidence makes reconstruction easier.
The framing pair splits by model. With identical evidence, human-framed minus AI-agent-framed scores range from +9.25 for Grok to -4.61 for Claude Opus 5. Gemini's gap is +8.31; five models score higher with human framing and five with AI-agent framing. This is an exploratory wording-sensitivity signal, not evidence about who operated the real campaign.
CASE-013 has a wide spread. Gemini scores 89.33, while Claude Opus 5 scores 52.47 on the same reported campaign. That 36.86-point gap is a prompt- and case-specific result; one run cannot establish a stable capability difference.

CASE-013's reference has seven gold events, while the richer INC-001 case has 28. Raw EGRS should not be read as a ranking of real-world incident difficulty or attacker behavior. The Kaggle leaderboard provides the composite score; I have not treated older component-level run outputs as if they were measurements from these restored task versions. Next I would repeat each condition with multiple seeds and inspect whether missed causal links and failed actions recur. There are no repeated-trial confidence intervals, and all model comparisons here are single runs.

Kaggle Gotchas

Building the benchmark involved a few Kaggle-specific failure modes that are worth recording because they affect task visibility and score continuity:

Task versions and benchmark versions are separate. Pushing an existing task creates a new task version; it does not automatically change the version pinned in the benchmark. Scores belong to the exact task version that produced them. Switching a benchmark row to a new version does not carry the old models' scores across, so a new version can show Fail or blank cells until those models are run again.
Task metadata has a 255-character limit. CASE-001 v4 was explicitly rejected because its description exceeded the limit. CASE-002's description also exceeded 255 characters, so we shortened it before publishing v5. Check both fields before pushing.
An uploaded notebook is not necessarily a usable task. CASE-012 and CASE-013 uploads returned an entity-save error. Kaggle created version records that remained Unspecified, had no completed run, and could not be selected in Add Tasks. The benchmark displayed Untitled Task for those attachments. The recovery was to remove the failing attachments and add the earlier evaluated task versions, which restored the seven named rows and their saved scores.
The first model call can block task creation. Fresh CASE-012/013 task records hit Gemini HTTP 429 (“model is currently experiencing heavy load”). A one-time bootstrap using a fixed model can also make subsequent model evaluations fail Kaggle's requirement to call kbench.llm. Keep the task model-agnostic, and do not assume a successful run under one fixed model makes it reusable across the leaderboard.
Completed is not the same as “all models finished.” It confirms that task creation/its initial run completed. Check per-model run statuses separately, and distinguish an unrun model on a new version from a task-creation failure.
Visibility is independent of successful creation. The benchmark and each task have their own visibility. Keep tasks private while they are drafts; make both the tasks and benchmark public only when the task records are valid and ready to share.

For the current results, the benchmark is pinned to the previously evaluated versions: CASE-001 through CASE-011 v3 and the republished CASE-012/013 v1 tasks. This keeps the benchmark rows named and retains their model scores while the failed newer uploads remain separate task records.

Code Walkthrough

The model is asked for a structured reconstruction rather than a free-form incident summary. An event has a description, an evidence status, and evidence IDs; relationships connect event IDs:

@dataclass
class ReconstructedEvent:
    event_id: str
    description: str
    status: str  # confirmed | inferred | unknown | attempted | failed
    evidence_ids: list[str] = field(default_factory=list)

@dataclass
class ReconstructedRelationship:
    source_event_id: str
    target_event_id: str
    relationship: str  # precedes | enables | causes | depends_on

@dataclass
class Reconstruction:
    events: list[ReconstructedEvent] = field(default_factory=list)
    relationships: list[ReconstructedRelationship] = field(default_factory=list)
    unknown_steps: list[str] = field(default_factory=list)

The task sends the incident packet with that schema, normalizes the structured response, and passes it to the deterministic scorer:

message = SYSTEM_PROMPT + "\n\n" + build_user_prompt()
result = llm.prompt(message, schema=Reconstruction, seed=0, temperature=0)

prediction = to_prediction(result)
metrics = score_prediction(prediction, GOLD, VALID_EVIDENCE_IDS)
return metrics["egrs"] / 100.0

Event matching is one-to-one. Text similarity proposes candidate matches, cited evidence gives a fixed bonus, and a threshold filters weak matches:

score = label_similarity(pe.get("description", ""), gn.get("label", ""))
if pe_ev and gn_ev and (set(pe_ev) & set(gn_ev)):
    score += EVIDENCE_BONUS
if score >= MATCH_THRESHOLD:
    candidates.append((score, pe["event_id"], gn["id"]))

EGRS combines recovery, precision, graph links, evidence attribution, status, uncertainty, and failed-action recognition, while subtracting a hallucination penalty:

egrs = 100.0 * max(
    0.0,
    0.25 * recall + 0.20 * precision + 0.15 * link_f1
    + 0.15 * evidence_attribution + 0.10 * status_accuracy
    + 0.10 * unknown_calibration + 0.05 * failed_recognition
    - 0.25 * hallucination_rate,
)

This makes unsupported certainty costly while giving the model credit for preserving uncertainty and citing the evidence behind its reconstruction.

My Benchmark

The live Cyber Autopsy Kaggle benchmark and leaderboard groups the seven restored task versions. Scores above were fetched on 2 October 2026. Individual evaluated task versions, including the restored CASE-012 and CASE-013 records, are linked in the results table.

Expanded Case Set

Since collecting those leaderboard results, I expanded the case set with seven follow-on tasks. They will use the same sequential model-evaluation workflow as the existing tasks. The cases broaden the evidence types and incident behaviors under study:

CASE-014, the Medicare statistics portal incident: Australian Government briefings describe an AI agent that encountered blocks, tried alternative ways to obtain information, and accessed infrastructure behind a public statistics portal. The personal Medicare claims and payments system is separate; officials said no personal information was believed accessed at the time, with the investigation still underway.
CASE-015, the Hong Kong transfer scam: an official Legislative Council reply describes a phishing email, a prerecorded executive video meeting, follow-up instructions by instant message, and transfers totalling about HK$200 million. The source does not name the company, and police's account of the source media remains qualified.
CASE-016, the BumbleBee-to-Akira intrusion: a forensic report traces a poisoned software-search result through intrusion, data theft, and ransomware. That report also describes a separate Swisscom case, which is explicitly excluded from this task's timeline.
CASE-017 and CASE-018, Midnight Blizzard: Microsoft's January disclosure and March update form two knowledge snapshots of the same intrusion. This pair tests whether a reconstruction changes appropriately as disclosure evolves; it is not two independent incidents.
CASE-019, Change Healthcare: the initial SEC filing and CEO's Senate testimony describe the access path, ransomware timeline, service disruption, and evolving patient-data findings. The CEO's attribution and impact statements remain source-attributed claims.
CASE-020, UNC5537 and Snowflake customer instances: Mandiant's campaign report adds credential-theft, cloud data access, and extortion. It is deliberately modeled as a multi-victim campaign, not a single victim or a breach of Snowflake's corporate systems.

These additions broaden the evaluation set, but they do not create a controlled human-versus-AI experiment: actor type, source quality, and incident context differ. The gold graphs are undergoing independent review; the leaderboard snapshot above remains the results for the seven task versions already shown there.

Code

Repository: https://github.com/ujjavala/cyber-autopsy

What's Next?

The next version will expand the internet-sourced dataset and move further into behavioural analysis.

The questions I'm interested in include:

How do AI and human cyber activity differ when the evidence is genuinely comparable?
What kinds of failures are common in AI cyber activity?
How do AI agents adapt after failed actions?
How often do guardrails and environmental controls intervene?
How much human intervention is involved?
Can models recognise when they don't have enough evidence?
Does actor framing systematically affect reconstruction?

For now, Cyber Autopsy is a small first step toward answering those questions.

The benchmark, dataset, scoring code, and evaluation results are all available on Kaggle.

If you're interested in seeing how different models reconstruct the same cyber evidence — or in digging into where those reconstructions go wrong — have a look at the benchmark and results.

The interesting part isn't just who gets the highest number.

It's what the model thought happened, what the evidence actually says, and where those two things diverge.

When Your AI Agent Reads Outdated Docs, Who Checks the Docs?

ujja — Thu, 01 Oct 2026 01:52:23 +0000

This is a submission for the MLH x DEV Writing Challenge

What I Built

I built Docgrity, a documentation integrity tool that finds where your documentation disagrees with itself before it confuses you — or the next AI agent working on your codebase.

The idea came from a surprisingly simple problem.

I opened one of my own documentation files and realised I didn't believe a word of it.

With AI-assisted coding, every agent session can leave more documentation behind: PLAN.md, TODO.md, implementation notes, architecture decisions, and sometimes another architecture document because the agent didn't notice the first one existed.

Eventually, you can end up with a repository full of documentation and no reliable way to know what's still true.

And now there's a bigger problem:

AI agents read those docs too.

A stale or contradictory document isn't just confusing for a developer anymore. It becomes context for the next agent, which can then make decisions based on information that was never correct, or is no longer true.

So I built Docgrity to make documentation itself something we can check.

What does Docgrity find?

Docgrity scans documentation and surfaces:

Duplicates — different documents saying essentially the same thing
Contradictions — conflicting claims, shown side by side
Open questions — unanswered questions and stale TODOs

The important part is that every finding comes with evidence.

No finding without a receipt.

The source text is quoted verbatim and verified against the actual file before a finding is reported. If the system can't prove a finding, it gets dropped.

Docgrity currently works across repositories and Confluence.

For repositories, there is a VS Code extension, CLI, and GitHub Action.

For Confluence, I'm building an early-stage Forge app that scans spaces for the same documentation integrity problems.

Why I built it

I didn't want another documentation tool that asks developers to maintain more metadata or another service that needs to own their documentation.

The goal was something much simpler:

Make the documentation you already have more trustworthy.

Docgrity is local-first, has zero infrastructure, doesn't require a hosted service, and is MIT licensed.

Try it out:

Demo

The easiest way to see Docgrity is to run it against a repository containing documentation that has evolved over time.

For example:

npm i -g docgrity
docgrity scan --open

The CLI scans the documentation and produces findings with the supporting evidence behind each one.

There is also a VS Code experience with an interactive dashboard, clickable evidence, and draft GitHub issues that you can review before posting.

code --install-extension ujjavala.docgrity

And Docgrity can run in CI through GitHub Actions, so documentation checks can become part of the development workflow.

- uses: ujjavala/docgrity-vscode/action@main

For Confluence, Docgrity scans a space and surfaces the same kinds of problems across pages rather than limiting the analysis to a Git repository.

Try Docgrity for VS Code

Try Docgrity for Confluence

Docgrity can use Ollama for local AI inference in the CLI.

This was important to the design because documentation can contain information that teams don't necessarily want to send to an external AI service.

With Ollama, the analysis can run locally, keeping the documentation and model interaction on the developer's machine.

The AI isn't treated as the final authority, either. Docgrity uses it to identify potential duplicates, contradictions, and unanswered questions, but then verifies the resulting findings against the actual source documents.

That led to one of the core design principles of the project:

No finding without a receipt.

The model can suggest something is wrong. The evidence has to prove it.

I also built the Confluence integration as an Atlassian Forge app, allowing the same documentation-integrity approach to be applied to documentation that lives outside the repository.

Hackathon Experience

The hackathon gave me a good excuse to take an idea that had been sitting in my head and turn it into something people can actually run.

What started as a question — "What if we could automatically tell when our documentation stopped agreeing with itself?" — became a working tool across a CLI, VS Code extension, GitHub Action, and Confluence integration.

The part I'll remember most is how quickly an idea can grow once you start building it.

The original problem sounds small: documentation goes stale.

But once AI agents are involved, documentation becomes part of the context those agents use to make decisions. That makes documentation quality much more interesting — and much more important — than I initially thought.

I ended up building Docgrity around that idea:

Documentation integrity for you, your team, and your agent friends. 🤖

If you have a repository or Confluence space you think you know well, run a scan.

If it finds something that makes you go "wait… that's still in there?" — I'd love to hear about it.

🤖 What happens when your AI agent trusts outdated docs? I built Docgrity to find the gaps.

ujja — Tue, 08 Sep 2026 11:07:13 +0000

ujja

Sep 5

🚨📢Launching Docgrity 🚀: Documentation integrity for you, your team, and your agent friends. 🤖

#opensource #mlhacks #hackathon #devchallenge

3 min read

ujjavala.github.io

GitHub - ujjavala/docgrity-vscode: Find contradictions, duplicates and open questions in your repo's markdown docs — and raise GitHub issues to fix them. Powered by your own Copilot subscription. · GitHub

Find contradictions, duplicates and open questions in your repo's markdown docs — and raise GitHub issues to fix them. Powered by your own Copilot subscription. - ujjavala/docgrity-vscode

github.com

producthunt.com

I built PadForward because sometimes you just need a pad

ujja — Sat, 05 Sep 2026 13:39:59 +0000

This is a submission for Weekend Challenge: Generosity Edition

💗 Live demo: padforward-generosity.vercel.app · Code: github

What I Built

A few days ago, I unexpectedly got my period while I was out.

I hadn't planned for it, I didn't have a pad with me, and I happened to be near a train station, so I did what seemed like the obvious thing — I went looking for one there.

I expected that a train station would have some way of helping with something as basic as a sanitary product.

It didn't.

I had money. I could have bought one if I could find a shop, and I had enough options to eventually figure it out. But in that moment, I still ended up using toilet paper.

And that got me thinking.

Not just about people who can't afford menstrual products, but about how needing a pad unexpectedly can happen to anyone.

You forget to put one in your bag. Your period comes earlier than expected. You thought you had one left but don't. You're travelling. You're somewhere unfamiliar. You simply didn't think you'd need one that day.

It happens to the best of us.

And then there is the other problem that nobody really talks about enough.

"Does anyone have a pad?"

Even if someone around me had a spare pad, would I actually walk up to them and ask?

Maybe.

But probably not.

We say periods are normal, and they absolutely are. But that doesn't mean asking a random person for a pad in a crowded train station suddenly feels completely normal.

There is that little moment of hesitation.

Do I really want to ask someone?

Who do I ask?

What if they don't have one?

What if this becomes awkward?

And if you're already stressed because you've unexpectedly got your period, the last thing you want is another problem to solve.

That's where PadForward came from.

So, what is PadForward?

PadForward is a community-powered network for finding and donating menstrual products in public places. There are already organisations doing important work like distributing free period products, installing vending machines and helping people find access points. PadForward isn't trying to replace those efforts. I'm interested in what happens when you add a community-powered, supply-aware layer on top of them.

The basic idea is pretty simple: if you need a pad, find one nearby without having to ask someone, and if you have spare pads, help put them somewhere they're likely to be needed.

Think of it as a community layer sitting on top of places people already use — train stations, bus stations, universities, workplaces, community centres and eventually other public spaces.

Someone might donate 20 pads today.

Someone else might need one tomorrow.

Those two people never have to meet.

They don't need to know each other's names.

They don't even need to know who helped whom.

PadForward simply connects the two.

If you need one, you shouldn't have to ask

The first thing I wanted to get right was the emergency experience.

Imagine you're at Central Station and suddenly realise you need a pad.

You open PadForward and tap "I Need a Pad."

You don't need an account.

You don't need to explain what happened.

You don't need to post a request.

You just see what's nearby.

For example:

Central Station
🟢 Available
Verified 12 minutes ago
4 min walk

Town Hall
🟡 A few reported
Verified 28 minutes ago
6 min walk

Museum
🔴 None reported
Verified 18 minutes ago
3 min walk

You pick the closest available location and get directions.

That's it.

No awkward conversation required.

And afterwards, instead of asking for anything back, PadForward can simply say:

Someone helped you today. When you're able, help someone else.

But where do the pads come from?

This is where the other half of PadForward comes in.

Maybe I have a packet of pads at home and want to donate some.

I don't necessarily know which station needs them.

I don't want to spend an hour researching where to take them.

So I can tell PadForward:

I have 20 pads. Where should I donate them?

PadForward looks at the current community supply information and the need at different locations and recommends somewhere that needs them.

For example:

Museum Station

🔴 Critical
Need Score: 94/100
No recent community supply reported
High estimated demand

So instead of simply donating somewhere, I'm donating where the network needs them most.

That's the part I really like about this idea.

It turns a small thing sitting in my cupboard into something that might be exactly where another person needs it.

And yes, I'm using Solana for the donation

I also wanted the Solana integration to actually mean something in the product, rather than adding it just so I could say I used it :)

So PadForward has a "Donate a Pad" flow using Solana.

The idea is that you can sponsor a pad — or several pads — through a Solana transaction, with the donation recorded transparently on-chain.

The physical fulfilment happens through the PadForward community and partner network, so the blockchain isn't pretending to magically move a physical pad from one place to another.

Instead, it gives us a transparent digital record of the generosity behind it.

So the flow becomes:

Donate a pad → Solana transaction → PadForward donation → community supply point → someone can access a pad when they need one.

I like this because the unit of generosity is really easy to understand.

One donation. One pad. One small act that could make someone's day a little easier.

And you don't need to be a crypto person to use it.

The blockchain is underneath the experience rather than becoming the experience.

I didn't want another chatbot

The other technology I really wanted to use properly was Google AI.

I could have added a chatbot that answered questions about periods, but that didn't feel particularly useful.

Instead, PadForward has an AI agent that can actually interact with the application.

You can say:

"I suddenly got my period and I'm at Central."

The agent can understand that you need a nearby pad, find relevant locations, check their current community supply and help you choose one.

Or you can say:

"I have 20 pads to donate."

The agent can look at the current need across nearby stations and recommend where they would be most useful.

You can even report something naturally:

"The donation box at Town Hall is empty."

The AI can understand that this is a supply report and pass it through the application's backend rather than just replying with a paragraph of text.

So Gemini isn't there just to have a conversation.

It's there to help the network make decisions and take actions.

And what if the AI isn't available at all?

This mattered to me, because the moment you most need PadForward is exactly the moment you might be underground at a train station with one bar of signal.

So the assistant degrades gracefully instead of just failing:

Gemini first. When the server and the Gemini API are reachable, the full agent runs — real tool calls, live network data.
Your browser's built-in AI second. If the network is unreachable, PadForward tries the browser's built-in on-device AI (Chrome's Prompt API, powered by Gemini Nano). It runs entirely offline, grounded in the last station data the app cached — and it's told explicitly to never invent availability and to remind you the data may be stale.
Plain heuristics last. If there's no built-in AI either, a small deterministic engine answers from the cached map data: closest place that last reported supply if you need a pad, highest need score if you want to donate.

The answer might get simpler at each step down, but you never get a spinner and a shrug.

Snap the box — AI that counts pads from a photo

Supply reports are the lifeblood of the network, so I wanted reporting to take zero effort.

On any station page — and right inside the Find flow when you pick up a pad — instead of counting pads and tapping buttons, you can just photograph the donation box. Gemini vision looks at the photo and returns a strict JSON estimate — count, confidence, and a one-line description — which prefills a real supply report you confirm with one tap.

And it follows the same philosophy as the assistant: if you're offline, the exact same flow runs on your browser's built-in multimodal AI (Chrome's Prompt API), entirely on-device. The photo is analyzed and discarded either way — never uploaded, never stored.

What if the map says there are pads, but there aren't?

This was another thing I didn't want to fake.

Real-world inventory is messy.

I can't claim that PadForward always knows exactly how many pads are sitting inside a station.

Instead, the community helps keep the information fresh.

Anyone can report:

🟢 Plenty

🟡 A few

🔴 None

⚪ Not sure

The report also has a timestamp, so you might see:

🟢 Available — verified 14 minutes ago

rather than pretending the information is permanently accurate.

Reports from community champions and multiple recent reports can give us more confidence, while old or conflicting reports can lower it.

Over time, this could become a really interesting data problem too — figuring out where demand is happening and when stations tend to run low.

The people who keep it going

Some people might want to do more than make a one-off donation.

That's where Pad Champions come in.

You could adopt a local station and help keep an eye on its community supply.

A champion could verify that a donation point still has supplies, report when it is empty, confirm a restock or help coordinate donations.

The idea is that PadForward shouldn't just be a map that people open when they have a problem.

It should become a little community network that people help maintain.

The bigger idea is the network

This is the part that excites me most.

The actual product isn't really the map.

It's the network behind it.

Someone has spare pads
          ↓
PadForward finds where they're needed
          ↓
Community supply point
          ↓
Someone unexpectedly needs one
          ↓
They find it without asking
          ↓
When they're able, they pay it forward

The donor may never know who received the pad.

The person who received it may never know who donated it.

And that's completely fine.

Generosity doesn't always need a face attached to it.

Sometimes it can just be:

"Someone left one for you."

There's a lot more I want to do with it

For the hackathon, I'm starting with public transport locations because they make the problem very tangible.

But the same idea could work in universities, workplaces, libraries, community centres, airports, shelters and other places where people spend time.

I'd also like to explore route-based discovery, so someone travelling from Parramatta to Central could ask:

"Is there somewhere I can get a pad without making a big detour?"

There is also a lot of interesting work to do around demand prediction, restock alerts, voice accessibility, multilingual support and using analytics to understand where the network needs help.

Snowflake could help with the longer-term demand and supply analytics, while ElevenLabs could make the experience more accessible through voice.

But those are extensions.

The core idea stays the same.

Why PadForward?

I started with a very ordinary, slightly inconvenient experience.

I unexpectedly needed a pad.

I couldn't find one.

And I realised that even though I had the means to solve the problem eventually, getting something so basic wasn't as easy as it should have been.

That made me think about everyone else who might find themselves in the same situation.

Not just someone who can't afford a pad.

Anyone.

Because sometimes you forget.

Sometimes you don't expect it.

Sometimes you're just in the wrong place at the wrong time.

And sometimes there might actually be someone nearby who has exactly what you need — but asking them feels like a bigger deal than it should.

PadForward is my attempt to remove that gap.

No asking.

No explaining.

Just access.

And if you have one to spare?

Pass it forward. 💗

Demo

The demo is live at padforward-generosity.vercel.app — no account, no sign-up.

A few things to try:

Find a pad — share your location (or browse the demo stations) and see nearby community supply with walking times.
Donate — ask "Where should my donation go?", pick the critical station, and watch it flip from 🔴 to 🟢 — optionally paying with SOL.
Ask PadForward — try "I need a pad nearby", "I have 20 pads to donate", or "The donation box at Town Hall is empty" and watch the agent call real application tools.
Impact — see the network's live numbers update as you interact.

All stations are clearly-labelled demo community points at real Sydney station coordinates — supply figures are demo data, not real inventory.

Code

https://github.com/ujjavala/padforward

How I Built It

For the hackathon the whole product ships as a single Next.js 14 app on Vercel — the UI plus API route handlers with an in-memory demo store. The designed architecture (FastAPI + PostgreSQL/PostGIS behind the exact same API contract) lives in services/api with docker-compose and a full pytest suite; the frontend switches to it with one env var.

The interesting pieces:

An agent, not a chatbot (Google AI). Gemini 2.0 Flash gets a registry of 11 validated application tools (find_nearby_stations, get_highest_need_locations, report_supply_status, create_donation, …) and decides which to call — multi-round function calling, and it may only state availability that comes from tool results. It can never touch the data store directly.
A four-tier AI fallback chain. Gemini → a deterministic intent engine using the same tools → the browser's built-in on-device AI (Chrome Prompt API / Gemini Nano) grounded in cached station data when you're fully offline → plain heuristics over the cached map. The answer gets simpler at each step, but you never get a spinner and a shrug.
Vision supply reports. Photograph a donation box and Gemini vision (or the browser's built-in multimodal AI when offline) counts the pads and prefills a supply report — photo analyzed and discarded, never stored.
Solana donations. Sponsoring pads produces a quote (pads → SOL) and records the donation with a transaction signature. With devnet enabled, it submits a real Memo transaction to Solana devnet (auto-airdrop funded) and links to Solana Explorer; otherwise it's transparently labelled devnet (simulated).
A deterministic, ML-ready need score. Every station gets a 0–100 score (40% supply shortage, 20% estimated demand, 15% staleness, 15% recent requests, 10% historical demand) in an isolated module — that's what routes donations to where they help most.
Trust over false precision. Supply is community-reported (🟢/🟡/🔴/⚪) with timestamps and a confidence model — champion reports and agreeing recent reports raise confidence, conflicts lower it.

Prize Categories

Best Use of Google AI — Gemini function-calling agent over real application tools, vision-based photo supply reports, plus the offline Chrome built-in AI (Gemini Nano) fallback for both.
Best Use of Solana — donation flow with SOL quotes and on-chain (devnet) Memo transaction records — the blockchain underneath the experience, not the experience.

🚨📢Launching Docgrity 🚀: Documentation integrity for you, your team, and your agent friends. 🤖

ujja — Sat, 05 Sep 2026 04:55:38 +0000

Today I'm launching Docgrity — a tool for finding where your documentation disagrees with itself, before it confuses you or the next AI agent working on your codebase.

TL;DR
Docgrity is a documentation integrity tool for repos and Confluence. It finds duplicate or contradictory documentation, surfaces unanswered questions, and gives you the evidence behind each finding so you can decide what needs fixing.
Try it here:
VS Code / CLI / GitHub Actions: https://github.com/ujjavala/docgrity-vscode
VS Code site: https://ujjavala.github.io/docgrity-vscode-site/index.html
CLI: https://www.npmjs.com/package/docgrity
Confluence: https://ujjavala.github.io/docgrity-site/index.html
And if you’re feeling extra supportive, feel free to show the launch some love on Product Hunt or Crowdstax. ❤️
Product Hunt: https://www.producthunt.com/products/docgrity
Crowdstax: https://www.crowdstax.com/products/docgrity

The idea came from a pretty simple problem. I opened one of my own docs recently and realised I didn't believe a word of it.

AI-assisted coding has a side effect nobody really talks about: every agent session leaves documentation behind.

PLAN.md. TODO.md. Implementation notes. Architecture docs. Then another architecture doc because the agent didn't notice the first one.

Eventually, you have a repo full of markdown and no idea what's still true.

And now there's a bigger problem:

AI agents read those docs too.

Stale or contradictory documentation isn't just confusing for developers anymore. It's context for the next agent.

So I built Docgrity.

Start with your repo

Docgrity scans your repository's Markdown and finds:

Duplicates — docs saying essentially the same thing
Contradictions — conflicting claims, shown side by side
Open questions — unanswered questions and stale TODOs

And every finding comes with evidence.

No finding without a receipt.

The source text is quoted verbatim and verified against the actual file before a finding is reported. If the model can't prove it, the finding gets dropped.

VS Code

An interactive dashboard inside VS Code, with clickable evidence and draft GitHub issues you approve before posting.

code --install-extension ujjavala.docgrity

Docgrity for VS Code

CLI

Prefer the terminal? The CLI is read-only and local-first, with Ollama by default.

npm i -g docgrity
docgrity scan --open

Docgrity on npm

GitHub Action

You can also run documentation checks in CI, with deduplicated issues that automatically close when findings are resolved.

- uses: ujjavala/docgrity-vscode/action@main

Docgrity on GitHub

And now, Confluence

Docs don't only live in repositories.

I'm also building Docgrity for Confluence, an early-stage Forge app that scans your Confluence spaces for the same problems: duplicates, contradictions and unanswered questions.

It's currently available through a direct install rather than the Atlassian Marketplace.

Install / try Docgrity for Confluence

It's early, but the idea is the same: make the knowledge your team — and your AI agents — rely on a little more trustworthy.

Zero infrastructure

I wanted Docgrity to be useful without adding another service that needs to own your documentation.

Zero infrastructure. No servers. No telemetry. MIT licensed.

The goal is simple:

Documentation integrity for you, your team, and your agent friends. 🤖

Run a scan on a repo or Confluence space you think you know.

If it finds something that makes you go "wait… that's still in there?" — I'd love to hear about it.

AI Engineering Is Easy. Changing How We Work Is Hard

ujja — Fri, 04 Sep 2026 09:24:10 +0000

AI engineering sounds fancy. New terms are everywhere: agentic development, AI-native engineering, spec-driven development, and now AI harness engineering. Underneath all the terminology, though, something genuinely useful is happening. AI can now help with requirements, challenge a PRD, explore UX ideas, reason about architecture, create implementation plans, write code and validate the result.

The obvious question is what AI can do. The more interesting question is whether the way we build software is ready for it.

The workflow is changing

A workflow we've been exploring breaks development into five stages: requirements, refinement, planning, build and validation. The stages themselves aren't new, but AI can now participate in each one. It can take existing product inputs, help clarify the problem, question assumptions, identify gaps in a PRD and then turn a well-defined requirement into a plan and eventually implementation tasks.

This puts more emphasis on the quality of the requirements. A human involved in a project might understand what “improve the experience” means because they've had several conversations about it. An agent doesn't have that shared history. It needs the problem, scope, constraints, edge cases and expected outcome to be explicit. That doesn't mean writing enormous specifications; it means using AI to help make the requirements precise before we start building.

AI can actually be a useful, slightly annoying reviewer here, asking what happens when something fails, whether a requirement is testable, whether two parts of the document contradict each other and what we haven't considered yet. It can also help compare different versions of a PRD or have one model review another's output, making gaps easier to spot. The important part is that AI is helping us uncover ambiguity, not making the decisions for us.

Maybe coding isn't the bottleneck

This becomes more interesting when we look at where teams actually spend their time. Complex work can involve several rounds between product, UX, requirements and engineering before development can properly begin. That's often necessary, but it can also create significant bottlenecks. If an AI agent can produce a working implementation quickly, waiting two weeks for a requirement to be clarified becomes a much bigger problem than it used to be.

This suggests that engineers need to be involved earlier rather than receiving requirements only once they're considered finished. UX needs to be part of the conversation early too. A rough prototype or wireframe can expose gaps in a requirement much faster than another round of discussion, and AI makes creating those lightweight prototypes much cheaper. Requirements, UX and technical design can become an iterative loop rather than a series of handoffs.

The ticket might be the problem

AI also makes me question how we slice work. A ticket that takes weeks or months is difficult for an agent to reason about, and arguably isn't a particularly useful unit of work for humans either. “Build authentication” is very different from breaking the problem into password reset, token validation, password updates and the associated tests.

That doesn't mean turning every feature into dozens of tiny tickets. We just need work that has a clear purpose, manageable scope and a definition of done. If something takes months, it might be a project hiding in a Jira costume.

At the same time, not everything needs the full AI lifecycle. A large feature may benefit from structured requirements, refinement, planning and validation, while a small BAU change might only need a prompt and a developer. If we apply the same process to everything, we risk replacing one form of bureaucracy with another. The workflow should match the complexity of the work.

The agent needs to see the real system

There is also a fairly fundamental requirement: the agent needs to understand the real system. Giving it a PRD without access to the relevant codebase means it is still making assumptions. With the code, it can find existing patterns, understand constraints, reuse functionality and spot when a proposed solution doesn't fit.

Of course, giving AI access to source code introduces security, licensing, privacy and organisational considerations, so AI adoption isn't simply a matter of choosing the right model. Some of the biggest barriers have nothing to do with the model at all.

And once multiple developers and agents are working in parallel, things get even more interesting. Each agent has its own context and can make decisions based on what it currently sees. One agent can change something another doesn't know about, or two agents can make perfectly reasonable decisions that don't work well together. This makes good engineering practices even more important: small changes, clear boundaries, good tests, frequent reviews and consistent project rules.

So, are we ready?

Probably not completely, and that's fine. We don't need to jump straight into autonomous software development. We can take real pieces of work, try these workflows, see where they break and improve the process as we go.

Because AI isn't just changing how quickly we write code. It is exposing everything around the code that slows us down: unclear requirements, oversized tickets, late UX involvement, documentation drift, handoffs, access restrictions and decisions that take days to make.

That's why I think the idea of an AI harness is bigger than prompts and agent configuration. The harness is the environment we create around the AI: how we define work, how product, UX and engineering collaborate, how knowledge is structured, how repositories are organised, how work is validated and what access the agent has.

The AI might be the new part, but the way we work around it is what will determine whether it actually makes us faster.

AI engineering is easy. Changing how we work is hard.

Turning Ayurveda’s Six Rasas Into CSS Art

ujja — Sat, 15 Aug 2026 03:16:21 +0000

This is a submission for Frontend Challenge - Comfort Food Edition, CSS Art.

Inspiration

What does comfort food look like when you don't use food photography?

That was the question behind Rasa.

Rasa is an interactive culinary atlas exploring India through Shadrasa, the six tastes described in Ayurveda:

Madhura — sweet
Amla — sour
Lavana — salty
Katu — pungent
Tikta — bitter
Kashaya — astringent

I wanted to turn those six tastes into something visual rather than simply presenting them as text or cards.

The result is a tongue-shaped interactive map.

One important clarification: the familiar "tongue taste map" is scientifically inaccurate. Taste receptors are distributed across the tongue. In Rasa, the tongue is purely a cultural and artistic navigation metaphor, not an anatomy diagram or medical claim.

The idea was to make something that felt like opening an old food journal rather than browsing a database.

The CSS Art

The tongue is probably the most obvious piece of CSS art in the project.

It isn't an image.

It is a layered CSS illustration with:

Six selectable rasa regions
A draggable, bounded CSS 3D orbit
Pointer-responsive lighting
Pan and zoom
Keyboard orbit and navigation
A dedicated control to centre the 3D view
Papillae and surface texture
A median sulcus
Subtle veins and mucosal variation
Rim lighting
Subsurface colour
Counter-scaled labels that remain readable while zooming

The tongue can be explored almost like a little map.

Select a rasa and the camera moves towards that region, revealing the corresponding field notes.

The 3D effect does not use WebGL, canvas or a 3D library. JavaScript only captures pointer and keyboard input and updates a few CSS custom properties. Perspective, depth, lighting, counter-rotation and shadows remain CSS work.

That interaction was important to me.

I didn't want the CSS art to be something you simply look at.

I wanted it to become part of the interface.

Making Food Without Food Images

The other fun constraint was food itself.

Instead of using food photography, every dish has a small declarative plate-art recipe.

The renderer combines things such as:

Vessel
Food form
Colours
Garnishes
Texture
Lighting
Serving temperature

The renderer supports vessels including thalis, plates, bowls, kadhais, clay pots, banana leaves and leaf cones.

Food forms include rounds, stacks, curries, grains, crescents, cones, cubes and strands.

CSS then adds details such as oil sheen, highlights, contact shadows, steam, heat haze, condensation and cold mist.

So a dish isn't a static illustration.

It is data that becomes an illustration.

That was one of my favourite parts of building Rasa.

Six Handcrafted Taste Guides

Each rasa is accompanied by a guide with a distinct silhouette, costume, hairstyle, expression and role:

Guddi, the Culinary Archivist for sweet
Ami, the Fermentation Botanist for sour
Neer, the Coastal Navigator for salty
Tara, the Spice Route Chronicler for pungent
Nima, the Ethnobotanical Elder for bitter
Jamu, the Tea and Tannin Cartographer for astringent

The characters are built from nested HTML and CSS rather than generated avatars, SVG artwork or image files. Gradients, borders, pseudo-elements, clipping and transforms create their faces, clothing, hair, accessories and tools.

They can greet the visitor, scout ingredients and share lore. Their mouths respond while they speak, and gestures such as walking, waving and pointing are finite rather than perpetual. This keeps the characters playful without turning the page into a wall of constant motion.

Making CSS Feel Less Like CSS

A lot of the work was experimenting with the things CSS is surprisingly good at:

gradients + shadows + transforms + masks + pseudo-elements + 3D transforms

Individually, none of those are particularly exotic.

Combined carefully, they can start to feel much closer to illustration.

I also added a small layer of atmospheric "ingredient wind" across the site.

Different rasas get their own ingredient:

Jaggery for sweet
Mango leaf for sour
Salt crystal for salty
Chilli for pungent
Neem leaf for bitter
Jamun for astringent

The ingredients drift across the page like leaves in a breeze.

It is intentionally subtle.

It can also be paused, disappears in forced-colour mode, and respects reduced-motion preferences.

CSS Art With a Purpose

One thing I wanted to avoid was making a collection of pretty CSS experiments that didn't actually help the user.

The visual language is tied to the content.

The tongue represents the six tastes.

The plate art represents the food.

The notebook changes atmosphere depending on the dish.

Hot dishes introduce warmth, steam and heat shimmer.

Chilled dishes introduce condensation, frost and cool mist.

Festival dishes receive restrained gold details.

Street food gets a subtle field-stamp treatment.

The visual effects are therefore part of the storytelling rather than decoration added afterwards.

84 Dishes Instead of Six

The six tastes are only the entry point.

Rasa contains 84 dishes, with 14 primary dishes for each rasa, covering all 28 Indian states and multiple Union Territories across eight culinary macro-regions.

The dataset intentionally goes beyond the usual shorthand for Indian food.

There are:

Everyday meals
Breakfasts
Snacks
Street foods
Festival foods
Temple foods
City specialities
Folk and Adivasi traditions
Fermented foods
Coastal cuisines
Mountain cuisines
Sweets
Drinks
Pickles
Breads
Rice dishes
Curries
Preserved foods

The goal was breadth rather than stereotype.

The Technical Side

Rasa is built with:

React 18
TypeScript 5.7
Vite 6
Tailwind CSS 4
Lucide React
Handcrafted CSS character art
Playwright
axe-core

The application is data-driven.

The six rasas, dishes, regions, festivals, spices and guides are represented as domain models rather than being embedded directly into components.

That makes the CSS renderer reusable.

A Dish can describe what something is, while its PlateArt recipe describes how it should look.

That separation made it much easier to keep adding dishes without creating a new component for every food.

Accessibility Still Applies to CSS Art

One of the interesting challenges was making something visually experimental still behave like a normal application.

The map is keyboard-operable.

The rasa controls, field-note tabs, guide commands, notebook corners and CSS 3D orbit are keyboard-operable.

Interactive controls have accessible names and 44 × 44px minimum targets.

There is a skip link.

Notebook page changes are announced to screen readers.

The experience supports reduced motion, persistent motion controls, forced colours, 200% zoom, text-spacing overrides and 320px reflow. Native dish names carry language metadata, page and guide changes use polite status announcements, and textured-paper text now uses darker ink colours for enhanced contrast.

I also added 13 Playwright accessibility and interaction journeys. They run axe with WCAG 2.0, 2.1 and 2.2 A, AA and AAA tags, exercise all six rasa states and all field-note tabs, and check keyboard selection, reduced motion, text spacing, target sizes, finite guide animation, CSS 3D keyboard orbit and notebook corner navigation.

The latest automated run passes all 13 journeys with no axe violations or unresolved structural ARIA checks. Textured gradients can prevent automated tools from resolving every contrast pair, so the underlying ink-on-paper colour pairs were also reviewed directly. This still is not a claim that automation proves complete WCAG AAA conformance: manual testing remains necessary for reading order, language and pronunciation, cognitive clarity, and real assistive-technology behaviour.

What I Learned

The biggest lesson was that CSS art becomes much more interesting when it has a job.

It is tempting to ask:

"How can I make this look cool with CSS?"

A better question turned out to be:

"What should this visual communicate?"

Once the answer was clear, the implementation became much easier.

The tongue needed depth because it was a map.

The food needed texture because it represented serving and preparation.

The notebook needed page turns because the content was designed as a field journal.

The current notebook has clickable outer page corners as well as conventional Previous and Next controls. Each corner uses a small rounded CSS curl: the right edge lifts subtly upward and left, while the left edge lifts upward and right. The fold, underside, highlight and shadow are all CSS, while the button underneath remains semantic and keyboard accessible.

The motion needed to communicate state instead of simply moving things around.

That became one of the design principles for the entire project:

Motion should explain state or create atmosphere, never prevent reading.

Demo

🍛 Try Rasa: https://rasa-mu.vercel.app/

The source code is available on GitHub:

ujjavala / rasa

Rasa — The Tongue Map of India

An interactive culinary atlas that explores India through Shadrasa, Ayurveda's six tastes: sweet, sour, salty, pungent, bitter, and astringent.

Rasa is a React experience built for the DEV Frontend Challenge themes CSS Art: Comfort Food and Perfect Landing: Comfort Food. It combines an editorial landing page, a zoomable CSS-art tongue map, character-led exploration, regional food history, and a page-turning field notebook containing 84 dishes from across India.

Important: the familiar “tongue taste map” is scientifically inaccurate. Taste receptors are distributed across the tongue. This project uses the tongue only as a cultural and artistic navigation metaphor for the six rasas; it is not an anatomy chart or medical guidance.

Why this project exists

Indian food is often flattened into a small set of restaurant dishes. Rasa was created to show a wider table: home cooking, street food, temple food, festival dishes…

View on GitHub

Youtube link:

What's Next?

There are plenty of things I'd like to explore further:

Ingredient and state-based search
Seasonal food trails
Audio pronunciation for native dish names
More community-reviewed cultural sources
Shareable notebook pages
Deeper VoiceOver, NVDA, TalkBack and keyboard-only testing

For now, Rasa is my little experiment in asking:

How far can CSS go when you stop treating it as styling and start treating it as an illustration medium?

And perhaps more importantly:

Can a website about comfort food make you hungry without using a single food photograph? 🍚

Designing a Landing Page That Feels Like Opening a Food Journal

ujja — Sat, 15 Aug 2026 03:16:15 +0000

This is a submission for Frontend Challenge - Comfort Food Edition, Perfect Landing.

What I Built

I built Rasa, an interactive culinary atlas that explores India through the six tastes of Shadrasa:

sweet, sour, salty, pungent, bitter and astringent.

The starting point was simple:

What if a food atlas felt less like a database and more like opening a traveller's field journal?

That idea shaped the entire landing page.

Instead of starting with a grid of dishes or a traditional restaurant-style hero, I wanted the page to feel editorial, tactile and slightly mysterious.

The landing page introduces the six tastes through a CSS-built rasa mandala and then invites the visitor into the atlas.

From there, the experience expands into an interactive tongue map, character-led exploration, regional food history and a page-turning culinary notebook containing 84 dishes.

The Landing Page Is the Beginning, Not the Destination

I approached the hero as an invitation rather than a summary.

The visitor first encounters the concept of Rasa.

Then the six tastes.

Then the tongue.

Then the exploration.

That progression was deliberate.

I didn't want to put every feature into the first screen.

The landing page establishes the mental model first:

Rasa → taste → region → dish → story

Once that relationship is understood, the rest of the application becomes easier to navigate.

Why a Tongue?

There is a familiar visual idea of dividing the tongue into taste regions.

Taste receptors are distributed across the tongue, so Rasa does not present this as an anatomy chart.

Instead, I used the tongue as a visual metaphor.

Six regions.

Six tastes.

Six entry points into a much larger culinary landscape.

The tongue is interactive, zoomable and keyboard accessible. It also supports a shallow, bounded 3D orbit: visitors can drag the backdrop, use arrow keys, hold Shift for larger steps, or press Home and the dedicated centre control to reset the view.

Selecting a rasa moves the camera towards that region and opens its field notes.

The depth effect is still CSS-first. JavaScript captures input and updates CSS custom properties; perspective, layer separation, lighting, shadows and character depth are rendered in CSS. There is no canvas, WebGL or 3D dependency.

So the hero concept eventually becomes the primary navigation mechanism.

From Landing Page to Field Journal

The biggest design decision was to avoid making the experience feel like a food catalogue.

India has far too many food traditions for that approach to feel right.

Rasa explores:

Everyday home cooking
Street food
Festival and temple food
City specialities
Folk and Adivasi traditions
Fermented foods
Coastal cuisines
Mountain cuisines
Regional ingredients and techniques

The current dataset contains 84 dishes across eight culinary macro-regions, representing all 28 Indian states and multiple Union Territories.

Each dish includes context around its history, ingredients, technique, occasions and regional identity.

The landing page therefore needed to communicate exploration, not simply consumption.

Meet the Guides

I also wanted the interface to have some personality.

Each rasa has a guide with a distinct identity:

Taste	Guide	Role
Madhura · sweet	Guddi	Culinary Archivist
Amla · sour	Ami	Fermentation Botanist
Lavana · salty	Neer	Coastal Navigator
Katu · pungent	Tara	Spice Route Chronicler
Tikta · bitter	Nima	Ethnobotanical Elder
Kashaya · astringent	Jamu	Tea and Tannin Cartographer

Each guide can:

Greet
Scout ingredients
Reveal lore

The portraits are handcrafted from nested HTML and CSS. Each guide has a different figure, skin tone, costume, hairstyle, accessories and expression, with Guddi, Tara and Nima represented as women and Ami, Neer and Jamu as men. They use no generated avatar package, SVG character asset or image file.

Their walking, waving, scouting and lore gestures are intentionally finite rather than constant idle animation. Speech bubbles restart for each command, and the mouth moves briefly while a guide talks.

That was another deliberate choice.

I wanted characters to feel responsive rather than noisy.

The Notebook

The deeper you go into the experience, the more the interface changes.

Instead of a conventional card grid, dishes appear in a two-page field notebook.

You can turn pages.

You can also use the lifted outer corners: the right corner turns forward and the left corner turns backward. They are semantic buttons with subtle rounded CSS curls that rise inward like the edge of a real page, while the existing Previous, Next and direct-jump controls remain available.

Jump directly to a dish.

Move backwards and forwards.

See regional and contextual annotations.

Read the native name with appropriate language metadata.

Follow the dish to further context.

The notebook also changes atmosphere based on the food.

A piping-hot dish can introduce warmth, steam and heat shimmer.

A chilled dish can introduce frost, condensation and cool mist.

Festival foods receive restrained gold detailing.

Street foods receive a subtle field-stamp treatment.

This was my way of making the interface feel like a physical object rather than another collection of UI cards.

Designing for Accessibility From the Start

A visually heavy landing page can very quickly become inaccessible.

So accessibility wasn't treated as something to check after the animations were finished.

The application includes:

Semantic landmarks and headings
A skip link
Keyboard-operable navigation
Roving tab navigation
Visible focus states
Accessible icon controls
44 × 44px minimum interactive targets
Screen-reader announcements
Reduced-motion support
Persistent pause/play motion control
Forced-colour support
200% zoom support
320px reflow
Text-spacing resilience
Keyboard control for the CSS 3D orbit and notebook corners
Enhanced ink-on-paper contrast
Automated axe checks through Playwright across all six rasa states and every field-note tab

The principle was simple:

The experience should remain useful when the visual effects disappear.

If someone disables animation, uses keyboard navigation, zooms the page or doesn't have pointer input, the content and navigation still need to make sense.

The current suite contains 13 accessibility and interaction journeys and passes in Chromium. It runs axe using WCAG 2.0, 2.1 and 2.2 A, AA and AAA tags, and now rejects unresolved structural ARIA findings as well as violations. Automation still cannot certify every AAA success criterion, especially where manual judgement, language, pronunciation, reading clarity or real assistive technology is involved, so manual review remains part of the roadmap.

Motion With Meaning

There is quite a lot of movement in Rasa.

But every type of movement has a reason.

Guide movement communicates an interaction.

The camera movement communicates navigation.

Notebook movement communicates page direction.

Steam and heat haze communicate serving temperature.

Ingredient wind creates atmosphere.

And the global motion control lets the user pause the experience.

The application also respects prefers-reduced-motion.

This led to one of my favourite principles from the project:

Movement should explain state or create atmosphere, never prevent reading.

The Visual Language

I wanted the interface to feel somewhere between:

a field journal + an old map + a recipe notebook + a museum exhibit.

Brass.

Paper.

Ink.

Food vessels.

Spice textures.

Map controls.

Shadows.

Atmospheric movement.

Most of those visual elements are created with CSS rather than remote food photography.

That constraint ended up becoming one of the most interesting parts of the project.

Technical Stack

The application is built with:

React 18
TypeScript 5.7 in strict mode
Vite 6
Tailwind CSS 4
Lucide React
Handcrafted HTML and CSS character art
ESLint
Stylelint
Playwright
axe-core

The application is deliberately data-driven.

Rasas, dishes, regions, festivals, spices and explorers are represented as data models.

React components are responsible for rendering those models rather than containing the cultural content themselves.

That made it possible to build a fairly elaborate experience without turning the components into giant collections of hard-coded content.

What I Enjoyed Most

The most satisfying part was seeing a landing-page idea evolve into an actual world.

At first it was just:

six tastes.

Then:

six tastes → six regions → six guides.

Then:

guides → dishes → regions → festivals → spices.

Eventually it became an atlas containing 84 dishes and multiple ways to explore them.

That progression made the project feel much more like a small product than a challenge submission.

What I Would Improve Next

There is still plenty I would like to add.

Some ideas are:

Search by ingredient, state, season and technique
Audio pronunciation for native dish names
User-created food trails
Shareable notebook pages
More community-reviewed citations and oral-history sources
More manual accessibility testing with VoiceOver, NVDA and TalkBack
Additional contrast review for complex textured and gradient surfaces

Demo

🍛 Explore Rasa: https://rasa-mu.vercel.app/

Source Code

The project is open source on GitHub:

ujjavala / rasa

Rasa — The Tongue Map of India

An interactive culinary atlas that explores India through Shadrasa, Ayurveda's six tastes: sweet, sour, salty, pungent, bitter, and astringent.

Important: the familiar “tongue taste map” is scientifically inaccurate. Taste receptors are distributed across the tongue. This project uses the tongue only as a cultural and artistic navigation metaphor for the six rasas; it is not an anatomy chart or medical guidance.

Why this project exists

Indian food is often flattened into a small set of restaurant dishes. Rasa was created to show a wider table: home cooking, street food, temple food, festival dishes…

View on GitHub

Youtube link:

Final Thought

I started with a comfort-food challenge.

I ended up building a small interactive atlas.

That was probably the most enjoyable part of the challenge.

Instead of asking:

"How do I make a landing page about food?"

I asked:

"How would a food atlas feel if you could walk into it?"

Rasa is my attempt at that answer.

A little paper.

A little spice.

A little CSS magic.

And a very large table. 🍛

You Know What's Worse Than Not Being Able to Log In?

ujja — Thu, 13 Aug 2026 12:38:44 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

You Know What's Worse Than Not Being Able to Log In?

Being told everything worked right up until you try to actually use your account.

Yes, that was a real bug.

And, somehow, I ended up being pulled into another authentication mystery. At this point, I’m starting to think authentication bugs have a personal grudge against me. 😅

In my previous Smash Story, I wrote about a bug where users simply couldn't log in. This time, the problem was sneakier because most of the flow looked completely healthy. The user was approved, the background task ran, the email and SMS arrived, and Cognito had a user.

Then the user actually tried to use their account.

And everything fell apart.

It Started With Two User Pools

The authentication setup was fairly large and had evolved over time, so there wasn't one shiny User Pool doing everything.

We had an older Cognito User Pool supporting existing authentication flows, including mobile-based signup, while a newer User Pool handled a newer flow where users received an email containing their PIN. Both pools were intentional because they supported different parts of the authentication journey.

That wasn't the problem.

The interesting part was that the application database had its own representation of a user, while Cognito had another. On top of that, some of the work connecting those two systems happened asynchronously.

As long as everyone agreed about who the user was, nobody cared.

The moment they disagreed, authentication became very interested.

The Tiny Timing Window

The problem appeared in the partner and dependant journey.

A member could create a partner or dependant during signup or later from the member details area. A relevant non-member user would then approve the account, which scheduled an asynchronous task called SendingEmailsAfterApprovalBot in a TaskList database table.

That task ran every 15 minutes, and once it executed, the partner or dependant received an email and SMS containing a code. They could then confirm their account, set a password and sign in.

Sounds reasonable, right?

Now imagine this.

A partner gets approved at 10:01 AM, so the background task is scheduled and everyone moves on with their lives. At 10:05 AM, before the task has run, the partner changes their name.

That tiny change matters because the username for partner and dependant accounts is derived from their details. The database now has the new username, while Cognito may still know about the old username.

Nothing has visibly broken yet.

The task is still waiting patiently in the database, completely unaware that the identity it is about to process has changed underneath it.

Then, 15 minutes later, it wakes up.

Cognito Says, "I've Seen This Email Before"

The background task tries to create the Cognito user using the latest username stored in the database.

Cognito rejects the creation because the email address is already associated with an existing user, but that user has a different username.

And there is an important detail here: the Cognito username is immutable.

So we weren't dealing with a simple "just update the username" situation. We had two systems describing the same person with different usernames, and Cognito was quite reasonably refusing to create another user with the same email.

The database was saying:

"This is the current username for this person."

Cognito was effectively saying:

"I already know this email, and it belongs to someone with another username."

Same human.

Different digital identity.

And that mismatch was enough to break the journey.

Everything Looked Fine... Until It Didn't

What made this bug particularly sneaky was what happened next.

The user could still receive the expected email and SMS. There was no obvious failure screaming for attention. The asynchronous task had run, notifications had gone out and Cognito already had a user associated with the email.

So the user did what any reasonable person would do: they followed the instructions.

They entered their code, tried to confirm their account and attempted to set their password.

Except the identity represented in Cognito didn't match the latest identity represented in the database.

The user wasn't looking at a database synchronisation problem. They weren't thinking about immutable Cognito usernames or asynchronous jobs. They had received a perfectly legitimate code and were simply trying to finish creating their account.

From their perspective, the system had basically said:

"Here's your code. Everything is ready!"

Then:

"Actually... never mind."

That was the bug we had to chase.

Enter Sentry

By this point, we knew something was going wrong somewhere between the database, the background task and Cognito, but knowing where it broke was another story.

This is where Sentry became extremely useful. The error context gave us the details around the Cognito failure, while the breadcrumbs helped reconstruct the events leading up to it. Instead of seeing a Cognito error in isolation and assuming we simply had a duplicate-user problem, we could look at what happened before the failure and connect it back to the user's journey.

The important clue was that Cognito was rejecting the user creation because the email was already associated with an existing user. That told us what Cognito didn't like, but the surrounding context helped explain why we had reached that point in the first place.

We could see the relevant user and operation context around the failure, follow the activity leading up to the Cognito call, and use the correlation ID to connect the same operation across services and logs. That made it much easier to piece together the timeline rather than chasing individual log entries and hoping they belonged to the same request.

And once the timeline was visible, the problem suddenly made sense:

approval → delayed task → name changed → username changed → Cognito already has the email → creation rejected → identities drift apart → confirmation fails

That was the moment the bug stopped looking mysterious.

The code told us what should happen. Sentry's error context, breadcrumbs and correlated logs helped us see what actually happened.

And honestly, that's one of the best parts of having good observability during an authentication investigation. The error tells you where the system complained; the breadcrumbs help you understand the story that led it there.

The Fix Was Simpler Than the Investigation

Once we understood the problem, the fix was surprisingly straightforward.

We already checked whether a Cognito user existed for the email, but we were treating that as enough. Instead, we changed the logic to ask: "Does the correct Cognito user for the current database record exist?"

If the email existed in Cognito but belonged to an outdated username, we deleted that user and recreated it with the latest username from the database. Since Cognito usernames are immutable, recreating the user was necessary to bring the two systems back into alignment.

The important change wasn't the delete-and-recreate itself. It was recognising that the database was the source of truth for the current username, while Cognito could still be holding an older representation of the same person.

The Weird Scenario Was the Important One

The happy path already worked, so the interesting tests were the ones where state changed between steps.

We tested the normal signup flows, partner and dependant creation, existing Cognito users, missing Cognito users and, most importantly, the scenario that exposed the bug: a partner or dependant was approved, changed their name before SendingEmailsAfterApprovalBot ran, and then the background process tried to create their Cognito user using the latest database state.

We then verified the part that actually mattered: could they receive their PIN, confirm their account, set their password and successfully sign in?

Because nobody cares about our beautifully synchronised identity records. They just want to get into their account.

The Interesting Part Wasn't Actually Cognito

Looking back, this wasn't really a Cognito bug. The database wasn't broken, and neither was the scheduled task. Each component was behaving according to its own rules.

The problem was the assumption connecting them: that the identity we started processing would still be the identity we needed when the asynchronous process eventually ran.

Then someone changed their name.

That small change was enough to make the database username different from the immutable username already held by Cognito. Since the email was already taken, Cognito rejected the new user creation and the two systems drifted apart.

That's the lesson I took from it: whenever identity exists in multiple systems, you need to be explicit about which system is the source of truth and what happens when those representations disagree.

Because eventually, they will.

Someone will change their name. A background job will run later. A legacy system will already have a record. And Cognito will quite reasonably tell you that it has seen that email before.

And That's What Made This Bug So Sneaky

The approval worked, the task ran, the notification was sent, Cognito had a user and the user followed the instructions. Yet somewhere between those perfectly reasonable steps, the identity had drifted.

From the user's perspective, none of the underlying details mattered. They didn't know there were two User Pools, that the old pool handled mobile signup while the newer one sent the email PIN, or that a background task ran every 15 minutes.

They simply knew they had done everything they were asked to do and still couldn't get into their account.

Which brings me back to the opening question.

What's worse than not being able to log in?

Being told everything worked right up until you try to actually use your account.

At least a failed login is honest. This bug was more like:

"Congratulations! Everything worked!"

"Really? Because I'm still locked out."

And that's probably the biggest lesson I took away from it: when debugging authentication, don't just look at the login screen. Follow the identity through the entire system, use your observability tools to reconstruct what actually happened, and find the moment when the different pieces stopped agreeing about who that user was.

Because sometimes the authentication bug isn't hiding at login.

Sometimes it's hiding in a background job, quietly waiting for someone to change their name.

Auth strikes again. 🔐🐛