What Broke When We Turned Government Open Data Into Agent-Callable Evidence

Bryan Williams — Mon, 24 Aug 2026 05:25:11 +0000

Government open data looks deceptively easy from a distance.

Find an endpoint. Make a request. Normalize the JSON. Ship an API.

That is enough for a demo. It is not enough for a production system that has to answer a consequential question next month, after the publisher changes a field, caps a response, moves an endpoint, or returns an empty page that looks exactly like “there are no records.”

We have been building CivicDataForge around one question:

What has to exist between an official government publisher and a software system—or an AI agent—before the result deserves to be called evidence?

This post is the practical answer we have reached so far. It is not a claim that government data can be made perfectly clean. It is an account of the failure modes we hit, the architecture we adopted, and the boundaries we refuse to erase.

1. A successful HTTP response is not a complete dataset

The first serious failure mode is pagination.

Government-data platforms often impose a maximum record count. A response can be valid JSON, have a 200 status, and still represent only the first page.

ArcGIS feature services expose controls such as resultOffset and resultRecordCount, and may signal that a transfer limit was exceeded. Socrata datasets similarly support paging and query controls. If a collector ignores those contracts, “the source returned 1,000 rows” can quietly become “the jurisdiction has 1,000 records.”

Those are not the same statement.

Our collection rule became:

Discover the publisher's page limit.
Request every page in a stable order.
Track page count, observed row count, and source-reported totals where available.
Reject a monitoring baseline when the selected query is capped, partial, or structurally inconsistent.
Bind the completed scope into a receipt.

The important product decision is step four. A partial page is useful for exploration, but it cannot prove that a record disappeared between two runs.

Official references:

2. “No match” is not a negative legal conclusion

Suppose an address lookup returns no permit record.

It is tempting to return NOT_PERMITTED or ILLEGAL. That is usually stronger than the evidence.

The selected publisher may not cover the full legal regime. The address may be formatted differently. A local permit may exist in another system. The source may be stale. The query may be incomplete. Or there may simply be no published match.

We use an explicit, fail-closed decision vocabulary:

EVIDENCE_FOUND
NO_PUBLISHED_MATCH
REVIEW_REQUIRED
SOURCE_UNAVAILABLE
SCOPE_INCOMPLETE

This vocabulary is deliberately less dramatic than a binary yes/no result. That is the point.

An API should not manufacture certainty because a consumer wants a Boolean. If a downstream workflow requires a Boolean, the customer should own the policy that maps evidence states to an action—and should preserve the original state for review.

3. Provenance has to survive normalization

Normalization is necessary. It is also where evidence can lose its identity.

We normalize names, addresses, dates, identifiers, and source-specific statuses, but retain enough lineage to answer:

Which authority published this record?
Which endpoint, file, or dataset did it come from?
What query and scope produced it?
When was it retrieved?
What did the publisher call this field?
Which transformation produced the normalized value?
Can the record and decision be fingerprinted again?

A minimal evidence envelope looks roughly like this:

{
  "source": {
    "publisher": "issuing authority",
    "url": "https://official.example/dataset",
    "retrieved_at": "2026-08-24T12:00:00Z"
  },
  "scope": {
    "query": "bounded source query",
    "complete": true
  },
  "record": {
    "source_id": "publisher-owned-key",
    "normalized_identifier": "stable-identifier",
    "record_hash": "sha256:..."
  },
  "decision": {
    "state": "EVIDENCE_FOUND",
    "receipt_hash": "sha256:..."
  }
}

The exact fields vary by source. The invariant is that the normalized result does not sever the path back to the official record.

4. Source health and data freshness are different dimensions

A source can be reachable and stale.

It can have a stable schema and return incomplete data. It can publish a recent file whose internal dates are old. It can change content without changing a URL. It can change a schema without failing a request.

So one green check is not enough.

We monitor independent dimensions:

availability
schema compatibility
record-count behavior
source date integrity
freshness, when the publisher exposes a defensible freshness signal
content fingerprint
contract compatibility

When freshness cannot be measured, we say freshness_not_measured. We do not let an availability check imply that the data is current.

This distinction matters because a production monitor should be able to quarantine a bad update instead of replacing a known baseline with an empty or damaged response.

5. Change monitoring needs replay and correction, not just alerts

“Send a webhook when something changes” is only the visible edge of a reliable change system.

The harder questions are:

Was the previous snapshot complete?
Is the new snapshot complete?
Did the source change, or did the collector fail?
Can the event be delivered more than once?
Can the consumer replay it?
What happens after a human corrects an ambiguous match?

Apify webhooks retry failed deliveries. That is helpful, but it means consumers must be idempotent. Our recommended key is the Actor run ID plus event type, with the downstream handler acknowledging quickly and queueing the expensive work.

idempotency_key = actor_run_id + ":" + event_type

We treat a webhook as a notification about an evidence object, not as the evidence object itself. The durable dataset, record hashes, scope, and decision receipt remain the inspectable source of truth.

Official reference: Apify webhook actions and retry behavior.

6. AI agents need selection contracts, not a pile of tools

Publishing an MCP server does not automatically make a tool safe or useful.

An agent still needs to know:

the task the tool supports
the minimum input
the preferred identifier
the source and jurisdiction boundary
the possible output states
what the output does not prove

We expose a stable Streamable HTTP MCP endpoint and machine-readable catalogs, but the more important artifact is the task-to-tool selection contract.

For example:

{
  "task": "verify India company evidence",
  "preferred_identifier": "Corporate Identification Number",
  "minimum_input": "exact CIN preferred",
  "boundary": "research evidence, not automatic KYC or eligibility clearance"
}

The transport gets the tool into the agent. The selection contract helps the agent call the right tool and preserve the result's meaning.

The MCP specification recommends Streamable HTTP for remote servers and defines the transport behavior. Authentication, origin validation, and secret handling still belong to the implementation and client configuration—not to public examples with embedded credentials.

Official reference: MCP transports specification.

7. We stopped thinking of the product as “a scraper”

The architecture now has six explicit services:

Source feasibility and rights review
Normalization and identity mapping
Source health and change monitoring
Evidence delivery and integration
Evidence packets and correction paths
AI-agent tool integration

That framing changes what we build.

A new source does not enter production because we found an endpoint. It needs a source and rights passport, a completeness test, a normalization contract, health probes, a bounded decision contract, and an acceptance-tested delivery path.

Likewise, a connector is not “available” because we can imagine it. Public APIs, datasets, schedules, webhooks, file exports, and the remote MCP endpoint are live interfaces. A customer-specific adapter, SFTP route, or platform-native app remains scoped work until its real downstream path passes an acceptance test.

The service catalog and integration recipes publish that boundary in human- and machine-readable form.

Our mission

CivicDataForge exists to reduce the hours lost between “the government publishes this somewhere” and “a person or system can use it without pretending it says more than it does.”

Our intention is to build a reusable evidence layer across public records: official-source first, inspectable, correction-friendly, and callable by both software and AI agents.

Our want is straightforward: make source quality and uncertainty visible enough that developers do not have to choose between a brittle one-off script and an opaque vendor answer.

We are open to collaboration and grateful for serious support, correction, and feedback. We are not asking for permission to build; we are inviting developers, public servants, data owners, researchers, and AI teams to make the contracts stronger with us.

What we need most from the developer community is adversarial input:

Which government source breaks your workflow today?
Which identifier refuses to match cleanly?
Which publisher silently caps or reshapes results?
Which evidence state does your current API flatten into a misleading Boolean?
Which integration path is missing from the recipes?

Those questions are more valuable than generic feature requests because they expose the real contract the system has to survive. If you bring one, we will meet it with respect, evidence, and a willingness to change what the facts prove should change.

What we would do differently

If we started again, we would make three decisions earlier:

Define failure and uncertainty states before defining the happy-path schema.
Prove complete collection before building change monitoring.
Publish task-selection and evidence-boundary metadata alongside every agent tool.

The code that fetches rows is often the smallest part of the system. The defensible work is keeping a source-linked statement truthful after paging, normalization, retries, source drift, and downstream automation.

That is the layer we are building.

DEV Community: CivicDataForge