DEV Community: Kun Shen

Deriving Break-Even ROAS from Contribution Margin (Instead of Guessing a Target)

Kun Shen — Mon, 17 Aug 2026 14:04:39 +0000

Direct answer: Break-even ROAS is attributable sales divided by the maximum ad spend the order can absorb before profit reaches zero. Compute that ad budget only after VAT, platform fees, product cost, fulfilment, creator commission and expected return loss are represented on the correct bases.

Teams often inherit a rule such as “never run below 3× ROAS.” That threshold may be sensible for one SKU and disastrous for another. The correct threshold is an output of the unit economics.

Define the variables

Let:

G = attributable gross sales shown by the ad measurement system;
V = VAT amount embedded in G, when applicable;
P = platform commission;
C = product and inbound cost;
F = fulfilment, packaging and delivery cost;
A = affiliate/creator cost;
R = expected return loss;
T = target profit per attributed order.

Then:

Maximum ad spend = G - V - P - C - F - A - R - T

For pure break-even, set T = 0.

Break-even ROAS = G / maximum ad spend

This ratio is valid only if G uses the same sales definition as the observed ad-platform ROAS. If the platform reports tax-inclusive gross sales, the numerator should match that definition while VAT remains a cost bridge below it.

A hypothetical test case

Suppose the ad platform attributes EUR 119 of tax-inclusive sales to one order. Assume:

VAT estimate: EUR 19;
platform commission: EUR 10.71;
product and inbound cost: EUR 35;
fulfilment and packaging: EUR 8;
affiliate cost: EUR 5;
expected return loss: EUR 3.

The maximum break-even ad spend is:

119 - 19 - 10.71 - 35 - 8 - 5 - 3 = EUR 38.29

The gross-sales break-even ROAS is:

119 / 38.29 = 3.11

This is an illustrative scenario, not a benchmark. Change any input and the threshold changes.

Implementation sketch

type Economics = {
  attributedGrossSales: number;
  vatAmount: number;
  platformCommission: number;
  productAndInbound: number;
  fulfilment: number;
  affiliate: number;
  expectedReturnLoss: number;
  targetProfit: number;
};

function breakEvenRoas(x: Economics): number | null {
  const maxAdSpend =
    x.attributedGrossSales -
    x.vatAmount -
    x.platformCommission -
    x.productAndInbound -
    x.fulfilment -
    x.affiliate -
    x.expectedReturnLoss -
    x.targetProfit;

  if (maxAdSpend <= 0) return null;
  return x.attributedGrossSales / maxAdSpend;
}

Returning null when pre-ad contribution is zero or negative is intentional. The product cannot buy its way to break-even with more advertising. Showing “0” would hide the failure state.

Tests worth keeping

No headroom: pre-ad contribution is zero; output is not a finite ROAS.
Higher target profit: break-even threshold rises.
Higher expected returns: break-even threshold rises.
Different VAT rate: sales and tax bases remain internally consistent.
No affiliate attribution: creator cost is zero only when evidence supports it.
Rounding: monetary lines round at the documented stage, not opportunistically.

Connect campaign and finance data carefully

Ad dashboards, order exports and settlement reports may use different time zones and attribution windows. A useful diagnostic keeps both the marketing definition and the finance definition. Do not force them into false precision; label late refunds and attribution uncertainty.

Add a target-profit ROAS, not just break-even

Break-even is a boundary, not necessarily an operating target. If a product needs EUR 10 contribution after advertising to fund overhead or inventory risk, include targetProfit: 10 in the same function. The available ad budget shrinks and the required ROAS rises. Reporting both values is useful:

break-even ROAS answers, “At what measured efficiency does contribution reach zero?”
target ROAS answers, “At what measured efficiency does the order preserve the chosen contribution?”

The difference creates a safety buffer for delayed refunds and attribution noise. Teams can alert before the mathematical loss line instead of after it.

Do not average away SKU economics

Campaign-level ROAS can hide a profitable hero SKU subsidising a loss-making product. Calculate the threshold per SKU or economically equivalent group, then compare observed performance on the same attribution definition. A blended number is useful for portfolio reporting only after the underlying thresholds are visible.

An interactive break-even ROAS model can help test the variables, but its defaults are planning assumptions. Account-specific rates, product VAT treatment and actual settlement data remain authoritative.

Sources verified 17 August 2026:

European Commission, VAT rules and rates.
TikTok Shop Seller University, Platform Commission Fee.

Disclosure: I work on TokMargin. This article was prepared with AI-assisted editing and manually checked against the cited primary sources.

Why adi-registration.properties Fails After a Correct Copy-Paste

Kun Shen — Sat, 15 Aug 2026 06:39:53 +0000

adi-registration.properties is an ownership-challenge asset used during certain manual Android package-registration flows. The important rule is simple: use the exact snippet generated by the official console, place the file in the APK assets directory, then sign and submit the intended challenge artifact. Do not invent the keys, reuse another package's file, or “clean up” the generated value.

When a console rejects the challenge, the visible text is often not the problem. The file may have been packaged into the wrong variant, renamed by an editor, omitted by a build rule, or added before a later step produced a different artifact.

Treat the snippet as opaque input

The official console is the source of the challenge value. Keep it exact:

preserve the filename adi-registration.properties;
preserve the generated property names and values;
avoid smart quotes, added comments, and line wrapping;
do not copy a sample token from a tutorial;
do not publish the live snippet in a repository or article.

The task is packaging, not reverse-engineering the challenge format.

Put it in the APK assets root

For a conventional Android application module, the source path is commonly:

app/src/main/assets/adi-registration.properties

Product flavors can change that source location. A flavor-specific asset may live under a path such as app/src/enterprise/assets/, and Gradle merges source sets according to the selected variant. The only decisive evidence is the built APK.

After building, inspect the archive:

unzip -l app-release.apk | grep 'assets/adi-registration.properties'

The expected packaged path is:

assets/adi-registration.properties

If the archive listing does not show that exact entry, the console cannot read what never reached the submitted APK.

The adi-registration.properties packaging checks collect the common source-set, filename, and re-signing mistakes in one checklist. They supplement, rather than replace, the instructions shown for your package in the official console.

Verify the artifact after the final signing step

Do not inspect an intermediate APK and upload a later one. Release pipelines can rebuild, shrink, align, bundle, split, or re-sign artifacts after the file was first added.

Use this sequence:

Add the console-generated file to the intended variant's assets.
Build the challenge APK.
Sign it with the key required by the registration flow.
Inspect that signed APK for the asset entry.
Extract its public signing-certificate fingerprint.
Confirm the package name and certificate are the pair you intend to prove.
Upload that exact file through the official console.

Record a cryptographic hash of the uploaded APK for internal provenance. The APK file hash is not the signing-certificate fingerprint, but it lets the team confirm that the inspected and uploaded files were identical.

Diagnose failures without exposing secrets

A useful support record includes the package name, variant, expected asset path, archive listing, public certificate fingerprint, artifact file hash, build ID, and console error text. It should not include the private signing key, keystore password, Google credentials, or unpublished challenge value.

Common failures map to concrete checks:

Symptom	Check first
Asset missing from APK	Source-set and packaging rules
Wrong package in console	Final `applicationId` and flavor suffix
Key not eligible	Actual signer and official console guidance
File looks correct but fails	Uploaded artifact identity and exact generated snippet
Works locally but not in CI	Workspace path, generated files and clean-build behavior

Remove or retain it deliberately

Follow the current console instructions on whether the challenge asset must remain in later production releases. Do not assume a one-time challenge file is a permanent runtime configuration, and do not remove it before the official flow confirms completion. Record the decision in the release repository so a future cleanup does not happen by accident.

Primary references: Play Console Help's package-name registration instructions and the Android Developer Console package registration guide, rechecked August 15, 2026.

Disclosure: I work on PkgReady, an independent readiness tool. It is not affiliated with or endorsed by Google or Android. This article was prepared with AI assistance and manually reviewed against the linked primary sources.

Stop Comparing the APK File Hash to the Signing Certificate

Kun Shen — Sat, 15 Aug 2026 06:39:11 +0000

An APK can have at least two SHA-256 values that developers casually call a “fingerprint.” The SHA-256 hash of the APK file identifies the exact bytes of that file. The SHA-256 digest of the signing certificate identifies the public certificate used to sign it. Android package registration checks use the certificate fingerprint, not the file hash.

Mixing them creates a frustrating failure mode: the value is valid hexadecimal, the length looks right, and the status check still cannot match the registered signing identity.

Extract the certificate from the final APK

Android's apksigner is the preferred artifact-aware tool because it understands modern APK signature schemes. Run it against the signed APK that will actually be distributed:

apksigner verify --print-certs app-release.apk

Find the line labeled like this:

Signer #1 certificate SHA-256 digest: a32a43cd...

That digest is the public certificate fingerprint. A file checksum command such as sha256sum app-release.apk answers a different question: whether two APK files contain identical bytes.

Google's current Play Console Help also documents a keytool fallback:

keytool -printcert -jarfile app-release.apk

Use an up-to-date Android SDK Build Tools version when possible. An old signing tool may not interpret every modern signature scheme the way your release pipeline does.

Choose the correct artifact

The extraction command is easy; selecting the artifact is the hard part. Check these variables before trusting the result:

release versus debug build;
product flavor and applicationIdSuffix;
local APK versus store-delivered APK;
upload key versus store app-signing key;
universal APK versus split or channel-specific output;
key rotation or signing migration state.

If Google Play re-signs the app through Play App Signing, the certificate on the Play-distributed artifact can differ from the certificate used to upload the bundle. For a registration check about the installed Play app, use the app-signing identity shown by Play or extracted from the relevant distributed artifact—not an upload certificate selected merely because it is nearby.

The APK signing-certificate fingerprint guide summarizes extraction and normalization across these sources. It is an independent workflow aid; the artifact and official console are authoritative.

Normalize only formatting

Tools may display a fingerprint with uppercase letters and colons while an API example uses lowercase text without separators. Normalizing case and removing colons is safe because it changes presentation, not bytes.

printf '%s' "$FINGERPRINT" | tr -d ':' | tr '[:upper:]' '[:lower:]'

After normalization, a SHA-256 certificate digest should contain exactly 64 hexadecimal characters. Do not truncate it, hash the displayed string again, or convert it to SHA-1.

Handle multiple signers explicitly

If apksigner reports more than one signer, preserve each signer index and digest. Do not silently take the first line and discard the rest. Multiple signers may be legitimate for a specific signing history, but the registration workflow needs a deliberate mapping between the certificate being checked and the artifact's signing configuration.

Make the build prove its own identity

A robust release pipeline extracts and records the public certificate immediately after signing. Store these non-secret fields with the build metadata:

application ID;
artifact digest for byte-level provenance;
signing-certificate SHA-256 digest for identity;
signer index if multiple signers exist;
build and source revision;
channel and signing path;
extraction tool version.

This distinction improves more than registration checks. The APK file hash proves you have a particular binary; the certificate digest helps prove that binary belongs to the expected signing lineage. They complement each other but are not interchangeable.

Primary sources: Google's SHA-256 certificate fingerprint guide and Android Developers' apksigner reference, both rechecked August 15, 2026.

Disclosure: I work on PkgReady, an independent Android release-readiness tool. It is not affiliated with or endorsed by Google or Android. This article was prepared with AI assistance and manually reviewed against the linked sources.

Build a Read-Only Android Package Registration Gate

Kun Shen — Sat, 15 Aug 2026 06:38:48 +0000

An Android registration gate should answer two different questions in sequence: “Is this package name registered to any verified developer?” and “Is this package name registered with the public certificate that signed this release?” Combining those questions into one vague green check hides the exact failure a developer needs to fix.

The Android Developer ID Status API is suitable for this job because it is a read-only status service. It does not register a package or mutate keys. The official documentation describes a package-only request and an optional certificateFingerprint query parameter for the certificate-aware request.

Level 1: package-only signal

The package-only call tests whether the application ID is registered. It is useful early in a build, when the final signed artifact may not exist yet.

curl -sS \
  -H "X-Goog-Api-Key: $ANDROID_STATUS_API_KEY" \
  "https://androiddeveloperidstatus.googleapis.com/v1/packages/com.example.app/packageRegistrationStatus:check"

A valid response state is REGISTERED or NOT_REGISTERED. The result does not tell you whether your next release certificate matches. It also does not prove that your organization owns the package. It reports registration state for the supplied identifier.

Level 2: artifact-aware signal

After signing the release artifact, extract the signing certificate's public SHA-256 fingerprint and make the certificate-aware check.

curl -sS \
  -H "X-Goog-Api-Key: $ANDROID_STATUS_API_KEY" \
  "https://androiddeveloperidstatus.googleapis.com/v1/packages/com.example.app/packageRegistrationStatus:check?certificateFingerprint=PUBLIC_SHA256_HEX"

That check can return:

REGISTERED: the package-and-fingerprint pair is registered.
REGISTERED_WITH_ANOTHER_CERTIFICATE_FINGERPRINT: the package is registered, but not with the fingerprint supplied in this request.
NOT_REGISTERED: the supplied lookup is not registered according to the service response.

The distinction is the reason to retain both levels. A package-only green result and an artifact-aware mismatch should produce a signing investigation, not a generic “registration passed” message.

For a compact state and remediation map, see this package-registration status workflow. Treat it as an operational explanation; the API response and official consoles are the source of truth.

Fail the right thing

Status, authentication, quota, and service availability are separate dimensions. The official API guide uses standard Google Cloud error objects and recommends building logic around the canonical status field rather than parsing English error text.

A reasonable gate policy is:

Condition	Release-gate action
`REGISTERED` for final package and certificate	Pass the registration check
`NOT_REGISTERED`	Block protected-channel promotion and assign registration work
Different certificate	Block and start signing-identity diagnosis
`INVALID_ARGUMENT`	Fail the job configuration without retry
`PERMISSION_DENIED`	Fail configuration; verify API enablement and credential scope
`RESOURCE_EXHAUSTED`	Mark the check inconclusive; retry later with backoff
`INTERNAL` or `UNAVAILABLE`	Retry with bounded exponential backoff

Do not translate every non-200 response into NOT_REGISTERED. That produces false negatives precisely when quota or networking is unhealthy.

Keep the API key on the server side

The API supports key credentials, but that does not make a key safe to embed in a browser bundle, APK, build log, or public workflow file. Store it in the CI platform's secret store, send it only from a trusted runner, and mask derived command output that might echo headers.

Restrict the key to the required API where Google Cloud supports that restriction. Rotate it if it appears in logs. Also budget calls: the official page currently publishes a default limit of 1,000 CheckPackageRegistrationStatus requests per project per day. Cache a result only for a deliberately short period, because registration state can change.

Log decisions, not credentials

A useful audit record contains the final package name, a normalized public fingerprint, response state, HTTP status, canonical error status when present, timestamp, workflow run, and source commit. It must not contain the API key, a private signing key, keystore password, or whole APK.

This turns the gate into reproducible evidence. When a later build changes state, the team can compare the exact artifact identity and status response instead of debating which screenshot was current.

Primary reference: Check app registration status with the Android Developer ID Status API, last updated by Android Developers on July 30, 2026 when checked for this article.

Disclosure: I work on PkgReady, an independent implementation aid. It is not affiliated with or endorsed by Google or Android. This article was prepared with AI assistance and manually checked against the linked primary source on August 15, 2026.

Modeling a Phone Upgrade Contract as a State Machine

Kun Shen — Wed, 12 Aug 2026 10:09:48 +0000

An upgrade-cost calculator should not begin with a price field. It should begin with a state.

When a user asks what an iPhone upgrade will cost, the application must know whether the user intends to return the phone, replace it with a new lease, leave early, or keep it. Those are different contract paths, not variations of one total.

Define the terminal states

A minimal model can use four terminal states:

RETURN_AT_TERM
REPEAT_UPGRADE
EARLY_RETURN
PURCHASE_DEVICE

Each state activates a different group of inputs. PURCHASE_DEVICE requires a purchase-option amount and purchase tax. RETURN_AT_TERM does not. EARLY_RETURN may require remaining scheduled payments and a condition assessment. REPEAT_UPGRADE closes one contract and begins a separate application; it should not inherit the new payment as though it were an extension of the old schedule.

Use a schedule, not an average

Do not store only monthlyPayment and termMonths. A schedule is safer:

type Money = { cents: number; currency: 'USD' };

type ScheduledPayment = {
  sequence: number;
  amount: Money;
  dueOn?: string;
  source: string;
};

This structure can represent payment changes, credits that expire, and an individual quote that does not match a public example. Integer cents prevent binary floating-point errors from leaking into consumer totals.

Model unknown as a real state

Public information cannot determine a customer’s local tax, final trade-in value, approval, condition fee, or exact Klarna purchase quote. Those values should be nullable or represented by a tagged union:

type Known<T> =
  | { state: 'known'; value: T; source: string }
  | { state: 'unknown'; reason: string };

Never convert unknown tax to zero. Zero is a verified value; unknown means the result is incomplete. A calculation can still return the supported subtotal plus an explicit list of unresolved fields.

Keep cost buckets separate

The calculation contract should distinguish:

type UpgradeCosts = {
  scheduledLease: Known<Money>;
  leaseTax: Known<Money>;
  appleCare: Known<Money>;
  earlyExitObligation: Known<Money>;
  conditionCharge: Known<Money>;
  purchaseOption: Known<Money>;
  carrierCharges: Known<Money>;
};

The separation is more important than the final addition. AppleCare is optional and separate from the current Apple Upgrade lease. Carrier charges come from a wireless-service relationship, not the Klarna lease. A purchase option belongs only to the ownership path.

The published calculation methodology uses the same principle: show the inputs and boundaries instead of inventing a universal residual percentage.

Attach provenance to volatile fields

Every price or rule that can change should carry:

source URL;
date verified;
device and configuration;
whether it is a public example or an individual quote;
source snapshot or content hash; and
review status.

A value without provenance should not silently replace the last known-good snapshot. Freshness is useful only when the new record passes validation.

Assert exclusions in tests

State-machine tests should assert both what is included and what is excluded:

expect(returnAtTerm.purchaseOption).toBeExcluded();
expect(purchaseDevice.purchaseOption).toBeKnown();
expect(earlyReturn.remainingPayments).not.toBeAssumedZero();
expect(allPaths.carrierCharges.source).not.toBe('apple-lease');

Useful cases include an ordinary end-of-term return, three payments remaining on an early exit, ownership after partial payment credit, and a schedule whose promotional credit expires.

Return a scenario result

The result should identify its boundary:

type ScenarioResult = {
  state: TerminalState;
  knownTotal: Money;
  includedFields: string[];
  excludedFields: string[];
  unresolvedFields: string[];
  verifiedAt: string;
};

This lets the UI say “known return-path total before local tax” instead of presenting an unsupported universal cost.

The architecture rule is simple: choose the ending, activate the fields that belong to that ending, preserve unknowns, and show the evidence attached to every volatile input.

Primary references: Apple Upgrade and Apple’s upgrade and purchase-option guidance.

Disclosure: I maintain PhoneUpgradeCalc. This article was prepared with AI assistance and manually reviewed against the cited primary sources on August 12, 2026.``

How to Build a Crawl Budget That Keeps AI Agents Fast and Predictable

Kun Shen — Sun, 19 Jul 2026 02:52:52 +0000

AI agents often begin with a deceptively simple web-access loop: take a URL, fetch it, extract text, and pass the result to a model. That loop works in a demo. In production, it can become a source of latency spikes, runaway costs, repeated requests, and inconsistent evidence.

A crawl budget is the control system that keeps this work predictable. It is more than a request limit. A useful budget decides which pages deserve attention, how much effort each page may consume, and when the agent should stop.

Start with the job, not the crawler

The correct budget depends on the agent's task. A monitoring agent may revisit a small set of pages on a schedule. A research agent may explore many domains once. A shopping agent may need current prices but can ignore most navigation pages.

Write the task as a small contract before choosing limits:

what evidence must be collected;
how fresh the evidence needs to be;
how many independent sources are required;
the maximum acceptable latency;
the maximum cost per completed task.

This contract prevents the crawler from treating every discovered URL as equally valuable.

Give every request an expected value

A URL should enter the queue with a reason. Useful signals include its relationship to the query, the authority of its host, its distance from a known source, its content type, and the chance that it contains new information.

A simple priority score can combine those signals: priority equals relevance times freshness need times source value, divided by expected cost.

The formula does not need to be mathematically perfect. Its purpose is to make tradeoffs visible. A product specification linked from a manufacturer's page should normally outrank a tag archive discovered five clicks away.

Expected cost should include more than bandwidth. JavaScript rendering consumes more time and compute than a direct HTML fetch. A screenshot adds storage and downstream vision cost. Retries also consume the budget, even when they produce no content.

Use a staged access strategy

The cheapest successful method should win. Start with a normal fetch and examine the result. Escalate to rendering only when the response lacks the content that should be present, relies on client-side navigation, or contains an application shell instead of the requested data.

Search is often a better first step than blind crawling. A targeted search can identify a few relevant pages before the agent spends budget extracting them. For focused discovery, AnyCrawler's search-page endpoint is one example of a workflow that combines result discovery with page-level access.

Screenshots should be deliberate. They are valuable when layout, charts, canvas elements, or visual state are evidence. They should not be the default representation of a text article.

Separate task, host, and page budgets

One global limit is too coarse. Use three layers.

A task budget limits total requests, rendered pages, bytes, elapsed time, and retries for one user goal. A host budget prevents one domain from consuming the entire task. A page budget caps the work spent on a single stubborn URL.

Host-level controls also improve politeness. Limit concurrency per host, respect crawl directives, and add delays when a server returns rate-limit or overload responses. Backoff should consume elapsed-time budget so that the agent cannot wait forever.

Page budgets should define a clear escalation ceiling. For example, allow one fetch, one render attempt if justified, and one retry for a transient failure. Authentication walls, persistent access denials, and repeated empty responses should become explicit outcomes rather than infinite loops.

Deduplicate before spending

Agents frequently encounter the same content through tracking parameters, alternate paths, print views, and redirects. Normalize URLs before enqueueing them. Remove known tracking parameters, resolve relative links, and store the final URL after redirects.

Content fingerprints catch duplicates that URL rules miss. A lightweight hash of normalized main text can prevent the same syndicated article from being processed repeatedly. Keep the source URLs even when content is duplicated; provenance still matters.

Caching should reflect freshness requirements. Stable documentation can be reused longer than a live price or breaking-news page. Record the retrieval time and cache policy next to the extracted evidence so the agent can decide whether reuse is acceptable.

Make stopping a first-class decision

A good agent stops because it has enough evidence, not merely because it has exhausted the web. Define completion signals such as:

the required facts are supported by two independent sources;
new pages have stopped adding unique claims;
remaining queue items fall below a value threshold;
the time or cost reserve is needed for synthesis and verification.

Reserve part of the total budget for verification. Discovering ten pages is not useful if no capacity remains to check their claims, dates, and canonical sources.

Measure outcomes, not just requests

Request counts alone cannot reveal whether a budget works. Track useful pages per task, unique evidence items, duplicate rate, render escalation rate, median and tail latency, bytes transferred, and cost per accepted source.

Also log why pages were skipped or stopped. Reasons such as low relevance, duplicate content, access denied, budget exhausted, and stale cache make later tuning possible.

Review failures by task type. If research tasks often run out of render budget, the discovery stage may be selecting too many application pages. If monitoring tasks repeatedly fetch unchanged documents, caching or conditional requests need improvement.

A practical default policy

A reasonable starting policy is conservative: search first, fetch selected pages, render only on evidence of client-side content, and capture screenshots only for visual claims. Cap per-host concurrency, normalize and deduplicate URLs, reserve verification capacity, and stop when evidence coverage is sufficient.

The exact numbers will change with the product and workload. The structure should remain stable. A crawl budget turns web access from an open-ended exploration into an accountable resource allocation process. That makes agents faster, cheaper, easier to debug, and more respectful of the sites they depend on.

Model an Evidence Chain, Not a Bag of Citations

Kun Shen — Sat, 18 Jul 2026 04:26:39 +0000

AI research products often display citations, but a row of links at the bottom of an answer does not tell you how the answer was built. A citation can be relevant to the topic without supporting the sentence beside it. Several links can repeat the same underlying source. A model can also drop important uncertainty while turning notes into fluent prose.

The data model has to preserve more than URLs. It should preserve the path from the original question to research queries, retrieved material, evidence grouped by claim, and the final report. We call that path an evidence chain.

This article focuses on the database and pipeline boundaries that make such a chain reconstructable.

The common shortcut: one giant JSON result

The fastest implementation is usually a job table with an input question and one JSON column containing everything else. It works until the product needs to answer operational questions:

Which research query found this source?
Which source supported this claim?
Did two citations come from the same domain?
Was this page reused from an earlier crawl?
Which model stage changed the wording?
Can we regenerate the summary without repeating the crawl?

A single blob makes these questions expensive and fragile. Every query becomes a custom JSON traversal, and relationships that should be enforced by keys exist only by convention.

The better approach is to keep the original run as the parent while giving important artifacts their own records.

1. Start with a search run

A search run represents one research question and its lifecycle. It owns the status, timestamps, final report, content metadata, and publication decision.

The run should not pretend that generation is atomic. A real research workflow may expand keywords, search multiple providers, crawl pages, extract page-level facts, aggregate evidence, synthesize a report, classify the content, and calculate an indexability result. Those stages should all reference the same run ID.

This parent key is what lets an operator reconstruct one execution without correlating timestamps across unrelated log streams.

2. Store research queries in order

Generated research queries deserve their own ordered records. Order matters because it shows the strategy the system attempted, and stable indices make retries deterministic.

A minimal query record needs the run ID, a query index, and the query text. You may also want the prompt version that produced it, the provider used, and whether the query was executed or skipped.

Keeping queries separate makes it possible to evaluate query quality independently from answer quality. If a report misses an important perspective, you can tell whether the gap originated in query generation or later retrieval.

3. Treat crawled pages as artifacts

Search results and crawled pages are inputs, not yet evidence. A page artifact should record the run, artifact type, query or keyword, URL, retrieval metadata, and the raw or normalized payload needed by later stages.

That distinction prevents a dangerous shortcut: assuming that every retrieved page supports the answer. Most search results are candidates. Some are duplicates, some only mention the topic, and some contradict the emerging conclusion.

Page artifacts also create a clean caching boundary. A system can reuse a recent crawl for the same URL while still running fresh evidence extraction for a new question. Retrieval freshness and claim relevance are different concerns and should not share one cache key.

4. Model evidence around claims

Evidence becomes useful when it is connected to a claim. An evidence record can contain:

the parent run ID
a stable evidence index
the claim it supports or challenges
the research keyword or query
source title, summary, and URL
normalized domain
an explanation of relevance
nested source details when several passages support one item

The claim field is the critical part. It gives reviewers a unit they can inspect. Instead of asking whether a source is “about the topic,” they can ask whether the source supports the specific statement that will appear in the report.

This structure also enables domain-diversity checks and source counts without parsing rendered Markdown. Quality rules should operate on evidence records before the final page exists.

5. Keep model-call provenance beside content provenance

Content provenance explains where facts came from. Model-call provenance explains how the system transformed those facts. Both are required for reproducibility.

For each stage, retain request messages, structured inputs, the raw provider response, parsed output, model and reasoning configuration, token usage, latency, prompt version, and errors. Link every call to the same run.

With that relationship, a reviewer can move in both directions:

from a sentence in the report to its claim and sources
from a suspicious source to the extraction and synthesis calls that consumed it

This is much more actionable than a generic “generated by AI” label.

6. Save the final report, but do not make it the source of truth

The final report is a presentation artifact. It may be Markdown or HTML, and it may include inline links for readers. But source counts, domain counts, confidence, and evidence sufficiency should come from structured data, not from scraping the report after generation.

That separation has a practical benefit: presentation can change without destroying provenance. You can redesign citation components, generate mobile summaries, or expose an API while keeping the same underlying evidence relationships.

7. Make insufficiency a valid outcome

An evidence chain should be able to end without a publishable answer. If retrieval produces too few sources, domains are not independent, or the evidence conflicts, the run can complete with an insufficiency flag and clear reasons.

This is not a pipeline failure. It is a research result. Treating insufficiency as data prevents the system from filling gaps with more confident prose simply to satisfy a success state.

8. Expose enough of the chain to readers

Internal logs can contain sensitive or operational details, so they should not be dumped into a public page. Readers still benefit from a carefully selected public surface:

visible source links
source titles and domains
claim-oriented evidence groups
correction and attribution channels
methodology and editorial policy pages

The public Omniracle methodology describes how signal discovery, research, evidence, model provenance, and publication gates fit together. The recommended reports show the reader-facing side of that architecture.

A useful mental model

Think of the pipeline as a directed chain:

question → research queries → page artifacts → claim evidence → synthesis calls → final report → publication decision

Each arrow should be represented by a durable relationship, not inferred later from similar text. When a result is challenged, the system can then answer the most important engineering question: not merely “Which links were shown?” but “How did this claim travel from source material into the published answer?”

That is the difference between citation decoration and an auditable research system.

Building Reliable Web Access for AI Agents: Search, Crawl, Markdown, and Screenshots

Kun Shen — Mon, 15 Jun 2026 16:54:41 +0000

AI agents are only as useful as the context they can reach. For many product, research, support, and competitive-intelligence workflows, that context lives on public websites: documentation pages, changelogs, pricing pages, articles, search results, screenshots, and long-tail reference content.

The hard part is not simply "scraping a page." The hard part is giving an agent a repeatable web access layer that can:

search for candidate sources,
fetch static pages cheaply,
render JavaScript-heavy pages when needed,
convert pages into clean markdown,
capture screenshot evidence,
retry safely when upstream sites are slow,
and avoid flooding the model context with irrelevant HTML.

This is where a web scraping API or crawler API becomes more useful than ad hoc browser scripts.

A practical pattern for agent web access

For most AI agent workflows, I like to split web access into four steps.

1. Search first, crawl second

Agents often do better when they first discover likely sources instead of starting with one URL. A search API for AI agents can return public web, news, image, video, or scholar results. The agent can then choose the highest-signal pages to read.

This reduces unnecessary crawling and gives the model a better source set.

2. Use fetch before render

Many pages do not need a headless browser. Documentation, blog posts, landing pages, legal pages, and static HTML often contain the useful content in the initial response.

For those pages, a fetch-based web data extraction API is usually faster, cheaper, and more reliable.

Use browser rendering only when the page depends on client-side JavaScript, hydration, or late network calls.

3. Convert pages to markdown

Raw HTML is noisy. Agents usually need a compact representation:

page title,
main content,
links,
metadata,
selected media,
and readable markdown.

Website to markdown conversion is a simple change that often improves answer quality because the model sees content instead of layout scaffolding.

4. Capture screenshots when trust matters

Text extraction is enough for many tasks, but not all of them. When an agent is checking visual layout, pricing evidence, legal copy, product UI, or compliance-sensitive content, a screenshot API gives a durable record of what the page looked like.

Where AnyCrawler fits

I have been testing AnyCrawler as an agent-facing web access layer. It combines public search, page crawling, markdown extraction, browser rendering, and screenshots behind API endpoints that are easier for agents to call than a full browser automation stack.

The useful part is the routing model:

use fetch crawling for static or content-first pages,
use render crawling for JavaScript-heavy pages,
use screenshots when visual evidence matters,
and use search before crawling when the source URL is not already known.

There is also an open skill package for agent runtimes here:

https://github.com/AnyCrawler-com/AnyCrawler-Skill

Design advice

If you are adding web access to an AI agent, avoid making the browser the first tool for every task. A better default is:

Search if the source is unknown.
Fetch the page if content is likely available in HTML.
Render only when fetch is incomplete.
Convert the result to markdown.
Capture screenshots only when the task needs visual proof.

That structure keeps workflows faster, less expensive, and easier to debug.