One Actor, two sources, three checks: how Codex verified CRE data through Apify MCP

The run was green, but the answer was incomplete

The first run was green. The data was real. But one of the two sources I had requested was completely missing.

I had asked Codex for a small commercial real estate (CRE) market file: office listings for sale in Dallas, with no more than five results from Crexi and five from LoopNet. Codex called my Actor through the hosted Apify MCP server, which implements the Model Context Protocol (MCP). Apify marked the run SUCCEEDED, and the dataset contained five valid Crexi rows.

Stopping there would have produced a confident but wrong answer. The Actor's run-summary.json classified the run as partial. Crexi had succeeded, while LoopNet had timed out with deadline_exceeded and contributed zero rows. The five returned listings were useful as a Crexi shortlist. They were not a completed two-source result.

That mismatch became the point of the experiment. An agent can call a tool correctly, receive structured data, and still misunderstand what the data proves. I needed a contract that went beyond the green platform badge.

I ended up separating the decision into three checks: execution, source coverage, and row validity. The MCP call starts the workflow. The workflow ends only when the agent proves that the output satisfies the request.

A later 360 s bi-source retest reproduced the same mismatch:

What I built and why I exposed it to Codex

I built the CRE Actor from scratch to turn public LoopNet and Crexi listings into one normalized market file. The two sources describe similar properties with different fields, identifiers, location conventions, and failure modes. The Actor accepts a single search contract, runs the selected sources, normalizes their rows, deduplicates them, and writes both a dataset and an ordered comma-separated values (CSV) file.

The output is deliberately explicit about provenance. Every row keeps its source and listing URL. Missing public values stay missing or are marked not public; the Actor does not manufacture broker contacts or financial fields. When a run reaches the summary-writing path, the Actor records an outcome for each selected source. The final summary says whether coverage was complete or partial, how many raw and final rows were produced, and whether a source succeeded, returned an empty result, failed, or timed out. If every selected source fails before usable output exists, no summary may be written; the execution check then treats the run as a technical failure.

I exposed the public CRE Brokerage Intelligence Actor to Codex for a specific reason. I wanted to know whether the input and output contracts I had designed were clear enough for an agent to use without me interpreting every run.

This was not a test of whether Codex could press a Run button. It was a test of delegation. Could the agent select the right Actor, send a bounded job, recover the complete evidence, detect an incomplete answer, and limit its conclusion accordingly?

The real estate case makes the risk easy to see. Zero LoopNet rows can mean no matching inventory, or it can mean a blocked transport. Those are different business conclusions. A tool contract needs to preserve that difference all the way back to the agent.

Connect Codex to one Actor through Apify MCP

The experiment used the Codex command-line interface (CLI) 0.147.0-alpha.1.2 and the official Apify plugin 0.1.0. The plugin registered the full hosted Apify MCP server at https://mcp.apify.com. The server uses Streamable HTTP. I authenticated with OAuth, so no Apify access token had to appear in the project, article, screenshots, or companion repository.

I ran the canaries against a private staging build, 0.3.22, so validation did not happen on the public product. The same tested code was later published as public build 0.3.327. The reader-facing configuration below points to that public Actor; none of the private Actor or run identifiers is required to reproduce the workflow.

Apify MCP prerequisites

You need an Apify account, a Codex client with remote MCP support, and a browser for the OAuth approval. The local verification project requires Node.js 22 or newer and npm. Reproducing the live call also requires a spend ceiling you are willing to authorize; running the sanitized fixtures does not require an Apify account or network access.

For a reproducible audit with a smaller tool surface, the portable configuration preloads the public Actor plus the four run and storage tools used below. The hosted server automatically adds get-actor-output to Actor-related configurations:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=kazkn/commercial-real-estate-brokerage-intel,get-actor-run,get-actor-log,get-dataset-items,get-key-value-store-record"
    }
  }
}

For the tested Codex CLI, the equivalent commands are:

codex mcp add apify --url "https://mcp.apify.com?tools=kazkn/commercial-real-estate-brokerage-intel,get-actor-run,get-actor-log,get-dataset-items,get-key-value-store-record"
codex mcp login apify

The login command opens Apify's browser authorization flow. After OAuth completed, I opened a fresh Codex task so the client would reload its tool catalog. The local CLI then reported the hosted URL, streamable_http, and auth_status: o_auth, with no bearer-token environment variable.

Scoping matters. An agent with dozens of unrelated tools has more room to choose the wrong surface or build an invalid call. The tools query limits this connection to the Actor and the four audit operations required for the job, while get-actor-output remains available automatically.

The initial tool response is also not the whole evidence package. An Actor call can return a preview and storage references, while the final dataset or key-value-store records remain in Apify storage. Codex therefore used get-actor-run and get-actor-log for execution evidence, get-dataset-items for the rows, and get-key-value-store-record for stored input and run-summary.json. Apify's guide to configuring Actors as MCP tools covers the basic connection. My focus here is what the agent must verify after that connection works.

The Actor schema was part of that verification surface. Codex could see the allowed source names, transaction values, booleans, and result limit before forming the call. That eliminated several prompt-level ambiguities, but it did not express the business meaning of a missing source. I kept that meaning in the Actor summary and in the agent-side coverage check instead of trying to encode every operational failure into the input schema.

The Model Context Protocol defines the tool boundary. OAuth protects access. Neither one guarantees that a returned dataset covers the user's request.

Give the agent a bounded CRE job

A verifiable job needs sharper boundaries than “find some Dallas properties.” I used the same filters throughout the experiment:

{
  "city": "Dallas",
  "state": "TX",
  "assetClasses": ["office"],
  "sourcesEnabled": ["loopnet", "crexi"],
  "transactionTypes": ["sale"],
  "maxResultsPerSource": 5,
  "includeListingDetails": false,
  "deduplicate": true,
  "normalizeCapRate": true,
  "monitoringMode": false,
  "outputSortBy": "source_default"
}

Each field makes a later assertion possible. sourcesEnabled defines the coverage obligation. maxResultsPerSource gives each source a separate target instead of allowing one source to fill the entire quota. The state, transaction type, and asset class become row-level invariants. Disabling listing details keeps the job small and avoids enrichment work that the workflow did not need.

I also wrote the acceptance criteria before the first call: five retained rows from each requested source, ten final rows, no timed-out source, and rows matching the verifiable filters. That prevented a smaller but plausible dataset from quietly redefining success. The target was a bounded sample, not every office listing in the market, so “complete” meant complete against this request rather than exhaustive against the web.

Runtime and financial limits sit outside this business input. I approved a strict cumulative ceiling of $0.25 for the experiment. Each call also received its own maximum charge, never greater than the remaining experiment budget. The positive bi-source retest used 1024 MB, a 360 s timeout, and a $0.10 per-call guardrail. No run was automatically retried.

That distinction matters. A row limit controls output volume. A platform timeout bounds wall-clock execution. A charge ceiling limits financial exposure. None substitutes for the others.

Check 1: execution

The execution check answers a narrow question: did the intended run execute through the intended path within its guardrails?

For every run, Codex inspected the terminal status, meta.origin, stored input, build number, memory, timeout, duration, and Apify usage cost. The origin had to be MCP. The stored input had to match the job I had approved. Build, memory, and timeout mattered because a result produced by different runtime settings would not test the same hypothesis.

I exported those facts into a sanitized local fixture. The companion verifier then applies the machine-readable part of the check:

import type { ActorJob, CheckResult, RunMetadata } from './contracts.js';

export function verifyExecution(job: ActorJob, run: RunMetadata): CheckResult {
  const reasons = [
    ...(run.status !== 'SUCCEEDED' ? [`platform_status_not_succeeded:${run.status}`] : []),
    ...(run.meta.origin !== 'MCP' ? [`origin_not_mcp:${run.meta.origin}`] : []),
    ...(run.buildNumber.trim().length === 0 ? ['missing_build_number'] : []),
    ...(run.memoryMbytes <= 0 ? ['invalid_memory'] : []),
    ...(run.timeoutSecs <= 0 ? ['invalid_timeout'] : []),
    ...(run.usageTotalUsd > job.maxTotalChargeUsd
      ? [`cost_cap_exceeded:${run.usageTotalUsd}>${job.maxTotalChargeUsd}`]
      : []),
  ];
  return { status: reasons.length === 0 ? 'pass' : 'fail', reasons };
}

The verifier checks one run against the ceiling in job.json; it does not add costs across retries or external providers. I kept a separate six-run experiment ledger for the cumulative $0.25 ceiling, including the Bright Data estimate.

This check is intentionally strict but incomplete. SUCCEEDED means the Actor process reached a successful terminal state. It does not mean every selected source contributed data. A green status passes execution and only execution.

The order of inspection matters. I first read the immutable run metadata and stored input, then the Actor-authored summary, then the dataset and logs. Starting from the rows alone would bias the review toward whatever happened to be returned. Starting from the approved job keeps the requested scope as the reference point.

There is also one manual provenance step the harness cannot reconstruct: confirming that the local job.json was copied from the run's stored input rather than rewritten afterward. I kept that inspection in the workflow instead of pretending a fixture could prove its own origin.

Check 2: source coverage

The coverage check compares what the job requested with what each source actually delivered.

The Actor writes outcomes like these:

{
  "status": "partial",
  "rawRows": 7,
  "finalRows": 5,
  "outcomes": [
    {
      "label": "crexi:sale",
      "source": "crexi",
      "transactionType": "sale",
      "status": "succeeded",
      "rowCount": 7
    },
    {
      "label": "loopnet:sale",
      "source": "loopnet",
      "transactionType": "sale",
      "status": "timed_out",
      "rowCount": 0,
      "reasonCode": "deadline_exceeded"
    }
  ]
}

Seven raw Crexi rows were collected before the final per-source cap retained five. LoopNet produced none. The important field is not only finalRows: 5; it is the combination of requested sources, source outcomes, and retained rows.

The coverage verifier from the tested companion project encodes that comparison:

import type { ActorJob, CoverageCheck, DatasetRow, RunSummary } from './contracts.js';

export function verifyCoverage(
  job: ActorJob,
  summary: RunSummary,
  dataset: DatasetRow[],
): CoverageCheck {
  const summaryReasons = [
    ...(summary.status !== 'complete' ? [`summary_not_complete:${summary.status}`] : []),
    ...(summary.finalRows !== dataset.length
      ? [`summary_dataset_mismatch:${summary.finalRows}!=${dataset.length}`]
      : []),
  ];
  const sourceReasons = job.input.sourcesEnabled.flatMap((source) => {
    const relevant = summary.outcomes.filter((outcome) => (
      outcome.source === source
      && job.input.transactionTypes.includes(outcome.transactionType)
    ));
    const rowCount = dataset.filter((row) => row.source === source).length;
    return [
      ...(relevant.length === 0 ? [`missing_source_outcome:${source}`] : []),
      ...relevant.flatMap((outcome) => outcome.status === 'succeeded'
        ? []
        : [`source_outcome_not_succeeded:${source}:${outcome.status}`]),
      ...(rowCount < job.input.maxResultsPerSource
        ? [`source_row_shortfall:${source}:${rowCount}<${job.input.maxResultsPerSource}`]
        : []),
    ];
  });
  const reasons = [...summaryReasons, ...sourceReasons];

  return { status: reasons.length === 0 ? 'complete' : 'partial', reasons };
}

The partial fixture produces three useful reasons: the summary is not complete, the LoopNet outcome did not succeed, and LoopNet has 0<5 retained rows. These are stable codes that an agent can act on.

I kept source outcomes separate from final dataset counts because they answer different questions. An outcome's rowCount records what a source task observed before final normalization and caps. The dataset count records what survived into the artifact the agent will use. In the partial run, Crexi observed seven rows and retained five. That is healthy coverage for a target of five. LoopNet observed and retained zero because its task timed out.

The same distinction handles natural empty results. If a source completes its search and explicitly reports empty, the workflow can say that no matching rows were observed under the tested query. If it reports timed_out or failed, the workflow cannot turn zero rows into an inventory claim. Both cases may produce an empty slice of the dataset, but only one supports a market interpretation.

This also prevents a common multi-source shortcut: aggregating counts before checking provenance. Ten rows from Crexi would meet a global target of ten while still failing a five-plus-five request. The coverage contract evaluates each requested source and transaction pair independently. One successful source cannot compensate for a missing one unless the user changes the job.

Most importantly, deadline_exceeded is a technical outcome. It is not evidence that Dallas had no matching LoopNet inventory. The correct agent response is to label the result partial, preserve the valid Crexi subset, and avoid a market-wide conclusion.

For this run, that meant the next action was diagnostic rather than editorial. Codex could use the five Crexi rows only if it labeled them as such. It could inspect the source logs, test a bounded timeout hypothesis, or stop and request human review. It could not publish a two-source market file.

I call this a coverage contract. It is a project method, not an official Apify or Model Context Protocol standard.

Check 3: row validity

Complete source coverage can still hide bad rows. The data check verifies every returned item against the fields that are observable in the dataset.

For this job, each row had to satisfy five rules: state = TX, transaction_type = sale, asset_class = office, a requested source, and unique identity fields. The final dataset contained ten unique source_listing_id values and ten unique listing_url values.

I did not enforce city = Dallas for every LoopNet row. The source's Dallas market page included nearby listings in Irving, DeSoto, and Grand Prairie. Treating those cities as parser errors would ignore how that source defines the result area. The state, market context, transaction, asset class, and provenance remained explicit, so the agent could describe the result as the Dallas market area rather than Dallas city limits.

The verifier reports deterministic reasons such as unexpected_state, unexpected_transaction, unexpected_asset_class, unexpected_source, duplicate_source_listing_id, and duplicate_listing_url. A row failure does not become partial coverage. It becomes a data failure. Keeping those states separate tells the agent whether to reduce the claim, reject rows, or treat the entire execution as unsafe.

The complete fixture is sanitized. Real listing IDs, URLs, street addresses, and contacts are replaced with deterministic placeholders. The shapes, counts, filters, and source outcomes mirror the observed run. Apify datasets hold the live run output; the public companion code needs only enough evidence to reproduce the verification logic.

Public listing data is not closed-deal data or investment advice. It may be incomplete or stale. Anyone reproducing the workflow should review the relevant site's terms, access rules, and robots.txt, and verify material facts against primary property and transaction sources.

What the failed runs taught the agent

The failed runs were useful because each one tested a different explanation. I did not keep retrying until a green badge appeared.

The first bi-source run used a 240 s timeout. It succeeded at the platform level after about 182 seconds, but LoopNet exhausted its source deadline. That established the original coverage failure.

Next, I isolated Crexi and tried a 90 s platform timeout. The Actor failed before collecting any rows. The logs explained why: pulling the container image took about 25 seconds, and the Actor reserves time for writing outputs safely. Only 3 s remained as sourceTaskSoftTimeoutSec. Crexi was not the problem; my timeout budget was.

The same Crexi-only input with 180 s left a 74 s source budget. It collected five rows in less than one second of application work and wrote a complete summary. This was a correction based on observed budget accounting, not an arbitrary larger number.

I then isolated LoopNet with 360 s. The source received about 275 s, yet all transports failed: two bounded Bright Data requests, LoopNet mobile API calls to pds.loopnet.com, public HTTP paths, and the browser path used in that run. The Actor finished with code 91, zero items, and no source summary because its only source failed before producing usable data. That run closed the “the timeout was simply too short” hypothesis.

A second bi-source run at 360 s was still partial. The final bounded run used the same filters and timeout and became complete only when the last browser fallback recovered LoopNet rows after the other transports failed again. That is evidence of one successful recovery, not continuous availability.

The agent's retry policy should therefore be hypothesis-driven. Change one bounded condition, state what the run will distinguish, and stop when the evidence no longer justifies another charge.

The complete run and the real cost

The sixth run closed the bounded happy path. Apify reported SUCCEEDED, and the Actor summary reported complete. Both source outcomes succeeded with seven raw rows each. The final cap retained 5 Crexi and 5 LoopNet rows, for ten unique IDs and ten unique URLs. Every row was in Texas, for sale, and classified as office.

The six Apify runs cost $0.137056 in total. Four runs used two bounded Bright Data requests each. At the experiment's recorded estimate of $0.0015 per request, the external estimate was $0.012. I therefore used the prudent combined total: $0.149056.

That left $0.100944 under the strict $0.25 cumulative ceiling. The Bright Data amount is an estimate, not an Apify charge or a universal provider price. Separating measured platform cost from estimated external cost prevents a tidy total from looking more precise than the evidence.

The result proves one end-to-end Codex → Apify MCP → Actor → storage → verified-dataset workflow. It does not prove market exhaustiveness, continuous LoopNet availability, or future cost. The three preceding LoopNet-involved failures are part of the result, not inconvenient noise to delete from the case study.

Apify Console showing the six existing MCP-origin runs, including their results, costs, durations, and tested build.

Reuse the three-check contract

The companion TypeScript project turns the method into a small local gate. It uses Zod to parse four sanitized JavaScript Object Notation (JSON) files: the requested job, run metadata, Actor summary, and dataset. It never starts an Actor and never contacts Apify.

Run the Apify MCP verifier

The output contract is deliberately small:

type WorkflowVerdict = {
  execution: 'pass' | 'fail';
  coverage: 'complete' | 'partial' | 'technical_failure';
  data: 'pass' | 'fail';
  safeToUse: boolean;
  reasons: string[];
};

safeToUse becomes true only when execution passes, coverage is complete, and the data check passes. A failed execution maps coverage to technical_failure. A successful run with a missing requested source maps to partial. Invalid rows fail the data gate even if both sources succeeded.

Those verdicts map to different agent actions. safeToUse: true allows the bounded artifact to continue downstream with its stated limitations. partial allows only a reduced, source-labeled claim. technical_failure blocks a data conclusion and points to runtime diagnosis. A data failure quarantines the affected artifact until the invalid or duplicated rows are resolved.

Install the locked dependency tree and run the complete verification gate:

npm ci
npm run verify

node dist/src/cli.js examples/job.json \
  fixtures/complete/run.json \
  fixtures/complete/run-summary.json \
  fixtures/complete/dataset.json

node dist/src/cli.js examples/job.json \
  fixtures/partial/run.json \
  fixtures/partial/run-summary.json \
  fixtures/partial/dataset.json

The complete fixture exits 0 and returns:

{
  "execution": "pass",
  "coverage": "complete",
  "data": "pass",
  "safeToUse": true,
  "reasons": []
}

The partial fixture exits 2. That exit code means the files are valid but the result is unsafe for the requested two-source claim. An agent can then preserve a clearly labeled subset, request human review, or report a technical limitation. It should not silently upgrade partial evidence into a complete answer.

The public Apify MCP coverage contract repository contains 15 passing tests, lint, a TypeScript build, a privacy scan, and a lockfile for the tested dependency tree. The fixtures intentionally contain no live storage identifiers, contacts, credentials, private handles, or local paths.

What I would do differently

I would define the coverage contract before the first run. My initial mental model still gave too much weight to the platform status. Writing the three verdicts first would have made the missing-source rule explicit before any data arrived.

I would also budget timeout from the source inward. Cold start, container setup, and output reserves are part of the system. A 90 s platform timeout did not mean the source received 90 seconds; it received three.

I would keep execution, coverage, and validity as separate states in every multi-source Actor. Combining them into one success flag removes the reason an agent needs to choose its next action.

Finally, I would declare both cost and retry policy before the call. Each new run should test a named hypothesis within the remaining ceiling. Stable public reason codes such as deadline_exceeded are what let the agent change scope without reading private logs or inventing a market explanation.

That is the durable part of this experiment: the Actor can fail, recover, or return a subset. The agent's contract should make each state explicit before it turns rows into an answer.