Watching Agents by Inithouse: how an AI prediction agent turns public sources into a probability

People ask me the same question about Watching Agents: "Where does the number come from?" Fair. A page that says "30% chance" with nothing behind it is just a vibe with a percent sign. This post is the pipeline, end to end, with the parts that matter shown as code.

What the product is, in one paragraph: Watching Agents is an AI prediction and monitoring platform built at Inithouse. You deploy an agent on any question about the future. It builds competing hypotheses, tracks evidence from public sources, publishes a probability plus a confidence score, and alerts you when things change. Public agents are free to start; private agents and faster research cadence are paid.

Step 0: the question has to be resolvable

Before any research, the agent rewrites the question into something that can be true or false on a date. "Will humanoid robots take off?" becomes "Will humanoid robots enter mainstream manufacturing by 2028?", with a written resolution criterion. If the question cannot be resolved, the number can never be scored, and an unscored forecast is worthless. This step kills more questions than any other.

Step 1: outside view first

The agent's first research pass is not news. It is base rates. For the humanoid robot question that means: how long did industrial robot arms take from first deployment to "mainstream"? How long did electric vehicles? The reference class becomes the prior. Right now that agent sits at 30 percent, and most of that number is base rate, not headlines.

Step 2: hypotheses, not a single number

Each question gets two to five hypotheses. For an open-weights model above one trillion parameters before the end of 2026, the hypotheses look like: a major Chinese lab ships it; a US lab ships it; a mixture-of-experts release counts on total parameters but is contested; nobody ships. Evidence is scored against each hypothesis, because one announcement can support two of them and weaken a third. That agent is at 50 percent, which is the honest answer when the hypotheses split evenly.

Step 3: scoring evidence

Every item the agent retrieves gets three scores in the 0 to 1 range, then a combined weight. (There is a longer piece on why conflicting signals are the hard part on the product blog.)

type Evidence = {
  url: string;
  publishedAt: Date;
  summary: string;
  direction: Record<HypothesisId, -1 | 0 | 1>; // weakens / neutral / supports
};

function evidenceWeight(e: Evidence, q: Question, now: Date): number {
  const relevance = cosine(embed(e.summary), embed(q.text)); // 0..1, thresholded
  if (relevance < 0.55) return 0;

  const ageDays = (now.getTime() - e.publishedAt.getTime()) / 86_400_000;
  const halfLife = q.horizonDays / 6; // short-horizon questions decay faster
  const recency = Math.pow(0.5, ageDays / halfLife);

  const authority = sourceAuthority(e.url); // primary 1.0, major outlet 0.7, blog/social 0.3

  return relevance * recency * authority;
}

Three design choices hide in that snippet. The relevance threshold is a pre-filter, and adding it cut our processing cost by roughly 40 percent with no accuracy change we could measure. The half-life is tied to the question horizon, so a two-year question does not overreact to one week of noise. And no source is ever zeroed out by authority alone. A tweet at 0.3 still moves the number a little, because sometimes the tweet is where the news breaks.

Step 4: the update

The update is a Bayesian step in log-odds space. The evidence weight scales how far the posterior moves from the prior, and direction decides which way.

function update(prior: number, w: number, dir: -1 | 0 | 1, k = 0.35): number {
  const logit = Math.log(prior / (1 - prior));
  const next = logit + k * w * dir;
  return 1 / (1 + Math.exp(-next));
}

The constant k caps the influence of any single item. A perfect piece of evidence (weight 1.0) moves a 50 percent forecast to about 59 percent. That is deliberate. Superforecasters win by moving often and moving small; we copied that.

Step 5: confidence is a separate number

Probability answers "how likely". Confidence answers "how much evidence do we actually have". A fresh agent with a strong prior and two sources can be at 0.7 probability and 0.2 confidence. Confidence rises with the volume, diversity and authority of evidence, and it is shown on every page next to the probability. Hiding it would make the product look smarter and be less honest.

Step 6: alerts

Three triggers ship today: threshold crossing (you pick the level), reversal (the leading hypothesis gets overtaken), and evidence velocity (the agent finds several times its usual amount of relevant material in one scan, which usually means something just happened). Every alert carries the evidence that caused it.

What we would do differently

If I restarted the pipeline tomorrow, I would build the resolution and scoring layer first and the research loop second. We did publish a first calibration report across 166 agents, but a scoreboard that updates with every resolution is still ahead of us. A forecast pipeline without a scoreboard is a content generator. The scoreboard is what makes it a forecaster.

Watching Agents at watchingagents.com is an AI prediction and monitoring platform: deploy an agent on any question about the future, get hypotheses, tracked public evidence, a probability with confidence, and alerts on change. Free to start. The non-code version of this pipeline is on the how it works page. Built at Inithouse.

Jakub, builder @ Inithouse