How we tuned TypeSafe Jev for log triage without alert storms

TypeSafe Jev is a typed evaluation model available on Vercel AI Gateway. It costs 0.042 dollars per million input tokens, and output tokens are free. Because the price is low, many teams want to use it as a programmable filter on log streams, either to trigger on-call alerts or to drop routine noise before sending logs to larger models.

We tested Jev on a stream of 3,000 synthetic payment and checkout logs, and on 5,000 lines from the Loghub dataset. We ran into several unexpected problems, including alert storms on normal deployments and pre-filtering pipelines that cost more money than sending raw logs directly to GPT-5.6 Luna.

Here is what went wrong, what fixed it, and the code patterns that worked.

The problem with discrete urgency labels

In our first implementation, we asked Jev to classify each incoming log into one of three buckets: page, ticket, or ignore.

This setup produced zero false alarms, but it failed to catch database replication lag. In our test data, a replica fell 47 minutes behind while the primary database continued taking writes. The application logged this event with INFO severity. Because the log level was INFO, Jev selected ticket instead of page.

The model saw the danger in its probability scores. Jev assigned an alert probability between 0.24 and 0.31 to the 47 minute lag, compared to 0.11 for normal 12 second lag. The discrete choice head discarded that difference and output ticket.

We tried fixing this by changing the prompt. We instructed Jev that INFO severity should not prevent an alert if customer data was at risk.

That prompt change caused an alert storm. The discrete choice head became hypersensitive. It generated 189 false pages across 3,000 lines, and 122 of those false pages were completely normal deployment notifications.

Thresholding probability in application code

Prompt changes were too blunt to control the decision boundary. Instead of tuning the prompt, we moved the decision logic into our application code.

We removed the three-way urgency classification. We asked Jev one boolean question: should this log page an engineer right now.

Instead of relying on the boolean true or false answer, we read the continuous probability from the response object. We then set a threshold directly in JavaScript:

import { createJevPager } from 'jevlogs';

const pager = createJevPager({ pageAbove: 0.50 });

const log = {
  service: 'orders-db',
  severityText: 'INFO',
  body: 'Replica lag 47m on primary still accepting writes'
};

const decision = await pager.decide(log);

if (decision.page) {
  await triggerPagerDuty(log, decision.probability);
}

Setting the threshold to 0.50 in code caught all 500 incidents in the dataset, including all 57 replication lag lines. It produced zero false alarms across the 3,000 test logs.

Asking a single boolean question also reduced token counts. The single question prompt cost 0.062 dollars for all 3,000 calls, compared to 0.087 dollars when asking for multiple fields.

Benchmark comparison on 3,000 logs

We compared the probability threshold against traditional severity filtering and OpenAI GPT-5.6 Luna. Luna was called with structured JSON output on the same gateway key.

Trigger method	Page recall	Page precision	False pages	INFO replica lag caught	Gateway spend
ERROR severity filter	49.2 percent	33.0 percent	500	0 of 57	0.00 dollars
Jev version 1 with discrete urgency	88.6 percent	100.0 percent	0	0 of 57	0.072 dollars
Jev version 2 with loosened prompt	100.0 percent	72.6 percent	189	57 of 57	0.087 dollars
Jev version 3 with probability 0.50	100.0 percent	100.0 percent	0	57 of 57	0.062 dollars
GPT-5.6 Luna structured output	96.2 percent	100.0 percent	0	46 of 57	0.320 dollars

Traditional log levels failed as an on-call trigger. Paging on ERROR caught less than half of the incidents and woke up engineers for 500 expected validation errors.

Luna caught 46 of the 57 replication lag incidents, but Azure content filtering dropped 130 log lines that contained search injection strings. Jev processed all lines without errors.

When pre-filtering logs increases your bill

Many developers consider placing Jev in front of a larger language model to drop routine logs and reduce total cost.

The financial outcome depends on the percentage of logs you drop. Consider a stream of one million logs.

If the downstream model is GPT-5.6 Luna at 0.20 dollars per million input tokens and 1.20 dollars per million output tokens, processing 1,000,000 logs with Luna costs 120 dollars.

Jev uses roughly 537 input tokens per log, which costs 22 dollars and 55 cents per million logs.

To break even, Jev must drop at least 18.8 percent of the log stream.

On the Loghub HDFS dataset, Jev was conservative. Its default score kept 99.16 percent of lines, dropping only 0.84 percent. Adding Jev as a pre-filter increased the total bill from 120 dollars to 142 dollars.

If your filter criteria only drop a small fraction of lines, adding a triage model acts as a surcharge.

Caching repeated log templates

System logs consist mostly of static templates with changing IDs, IP addresses, and timestamps.

Before calling Jev, we sanitize the log body by replacing IP addresses and block IDs with fixed strings. We then compute a SHA-256 hash of the sanitized string and check an in-memory cache with a five minute expiration:

import { createHash } from 'node:crypto';

function sanitizeLog(body: string): string {
  return body
    .replace(/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g, '[IP]')
    .replace(/blk_[-0-9]+/g, '[BLOCK]');
}

const cache = new Map<string, number>();

async function getProbability(body: string): Promise<number> {
  const sanitized = sanitizeLog(body);
  const hash = createHash('sha256').update(sanitized).digest('hex');

  if (cache.has(hash)) {
    return cache.get(hash)!;
  }

  const p = await callJev(sanitized);
  cache.set(hash, p);
  return p;
}

On our 2,500-line HDFS sample, 2,412 lines matched an earlier template. This in-memory cache reduced model calls from 2,500 to 88, cutting token consumption from 1,350,308 to 48,019 tokens.

Production rules for log evaluation

First, protect errors locally. If a log arrives with FATAL or CRITICAL severity, route it immediately in code. Do not spend model tokens on records that already require human review.

Second, avoid multi-field schemas. Ask one boolean question to keep token counts low and responses fast.

Third, read the continuous probability. Do not let the model choose discrete urgency buckets. Enforce your cutoffs in application code.

Fourth, separate your archive pipeline from your triage pipeline. Send all raw logs to your storage backend through an independent OpenTelemetry processor so model timeouts never drop audit records.

The code, test datasets, and interactive explorer are available on Hugging Face under reachjalil/jev-luna-pagerduty-trigger.

Comments

[Tae Kim]

The shift from discrete labels to a threshold on the raw probability is exactly the fix we landed on for anomaly triage in a document ingestion pipeline. The discrete head was collapsing confidence gradients we needed, and the model was giving us signal through the probabilities that the label was throwing away. The one thing I'd add to the JavaScript approach you show: log the raw probability on every call, not just the ones that cross the threshold, because a drift in the non-paging distribution usually appears days before you see false alerts again.