DEV Community: jalil laaraichi

How we tuned TypeSafe Jev for log triage without alert storms

jalil laaraichi — Fri, 18 Sep 2026 22:57:27 +0000

TypeSafe Jev is a typed evaluation model available on Vercel AI Gateway. It costs 0.042 dollars per million input tokens, and output tokens are free. Because the price is low, many teams want to use it as a programmable filter on log streams, either to trigger on-call alerts or to drop routine noise before sending logs to larger models.

We tested Jev on a stream of 3,000 synthetic payment and checkout logs, and on 5,000 lines from the Loghub dataset. We ran into several unexpected problems, including alert storms on normal deployments and pre-filtering pipelines that cost more money than sending raw logs directly to GPT-5.6 Luna.

Here is what went wrong, what fixed it, and the code patterns that worked.

The problem with discrete urgency labels

In our first implementation, we asked Jev to classify each incoming log into one of three buckets: page, ticket, or ignore.

This setup produced zero false alarms, but it failed to catch database replication lag. In our test data, a replica fell 47 minutes behind while the primary database continued taking writes. The application logged this event with INFO severity. Because the log level was INFO, Jev selected ticket instead of page.

The model saw the danger in its probability scores. Jev assigned an alert probability between 0.24 and 0.31 to the 47 minute lag, compared to 0.11 for normal 12 second lag. The discrete choice head discarded that difference and output ticket.

We tried fixing this by changing the prompt. We instructed Jev that INFO severity should not prevent an alert if customer data was at risk.

That prompt change caused an alert storm. The discrete choice head became hypersensitive. It generated 189 false pages across 3,000 lines, and 122 of those false pages were completely normal deployment notifications.

Thresholding probability in application code

Prompt changes were too blunt to control the decision boundary. Instead of tuning the prompt, we moved the decision logic into our application code.

We removed the three-way urgency classification. We asked Jev one boolean question: should this log page an engineer right now.

Instead of relying on the boolean true or false answer, we read the continuous probability from the response object. We then set a threshold directly in JavaScript:

import { createJevPager } from 'jevlogs';

const pager = createJevPager({ pageAbove: 0.50 });

const log = {
  service: 'orders-db',
  severityText: 'INFO',
  body: 'Replica lag 47m on primary still accepting writes'
};

const decision = await pager.decide(log);

if (decision.page) {
  await triggerPagerDuty(log, decision.probability);
}

Setting the threshold to 0.50 in code caught all 500 incidents in the dataset, including all 57 replication lag lines. It produced zero false alarms across the 3,000 test logs.

Asking a single boolean question also reduced token counts. The single question prompt cost 0.062 dollars for all 3,000 calls, compared to 0.087 dollars when asking for multiple fields.

Benchmark comparison on 3,000 logs

We compared the probability threshold against traditional severity filtering and OpenAI GPT-5.6 Luna. Luna was called with structured JSON output on the same gateway key.

Trigger method	Page recall	Page precision	False pages	INFO replica lag caught	Gateway spend
ERROR severity filter	49.2 percent	33.0 percent	500	0 of 57	0.00 dollars
Jev version 1 with discrete urgency	88.6 percent	100.0 percent	0	0 of 57	0.072 dollars
Jev version 2 with loosened prompt	100.0 percent	72.6 percent	189	57 of 57	0.087 dollars
Jev version 3 with probability 0.50	100.0 percent	100.0 percent	0	57 of 57	0.062 dollars
GPT-5.6 Luna structured output	96.2 percent	100.0 percent	0	46 of 57	0.320 dollars

Traditional log levels failed as an on-call trigger. Paging on ERROR caught less than half of the incidents and woke up engineers for 500 expected validation errors.

Luna caught 46 of the 57 replication lag incidents, but Azure content filtering dropped 130 log lines that contained search injection strings. Jev processed all lines without errors.

When pre-filtering logs increases your bill

Many developers consider placing Jev in front of a larger language model to drop routine logs and reduce total cost.

The financial outcome depends on the percentage of logs you drop. Consider a stream of one million logs.

If the downstream model is GPT-5.6 Luna at 0.20 dollars per million input tokens and 1.20 dollars per million output tokens, processing 1,000,000 logs with Luna costs 120 dollars.

Jev uses roughly 537 input tokens per log, which costs 22 dollars and 55 cents per million logs.

To break even, Jev must drop at least 18.8 percent of the log stream.

On the Loghub HDFS dataset, Jev was conservative. Its default score kept 99.16 percent of lines, dropping only 0.84 percent. Adding Jev as a pre-filter increased the total bill from 120 dollars to 142 dollars.

If your filter criteria only drop a small fraction of lines, adding a triage model acts as a surcharge.

Caching repeated log templates

System logs consist mostly of static templates with changing IDs, IP addresses, and timestamps.

Before calling Jev, we sanitize the log body by replacing IP addresses and block IDs with fixed strings. We then compute a SHA-256 hash of the sanitized string and check an in-memory cache with a five minute expiration:

import { createHash } from 'node:crypto';

function sanitizeLog(body: string): string {
  return body
    .replace(/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g, '[IP]')
    .replace(/blk_[-0-9]+/g, '[BLOCK]');
}

const cache = new Map<string, number>();

async function getProbability(body: string): Promise<number> {
  const sanitized = sanitizeLog(body);
  const hash = createHash('sha256').update(sanitized).digest('hex');

  if (cache.has(hash)) {
    return cache.get(hash)!;
  }

  const p = await callJev(sanitized);
  cache.set(hash, p);
  return p;
}

On our 2,500-line HDFS sample, 2,412 lines matched an earlier template. This in-memory cache reduced model calls from 2,500 to 88, cutting token consumption from 1,350,308 to 48,019 tokens.

Production rules for log evaluation

First, protect errors locally. If a log arrives with FATAL or CRITICAL severity, route it immediately in code. Do not spend model tokens on records that already require human review.

Second, avoid multi-field schemas. Ask one boolean question to keep token counts low and responses fast.

Third, read the continuous probability. Do not let the model choose discrete urgency buckets. Enforce your cutoffs in application code.

Fourth, separate your archive pipeline from your triage pipeline. Send all raw logs to your storage backend through an independent OpenTelemetry processor so model timeouts never drop audit records.

The code, test datasets, and interactive explorer are available on Hugging Face under reachjalil/jev-luna-pagerduty-trigger.

let Jev score OpenTelemetry logs before a bigger LLM sees them

jalil laaraichi — Thu, 17 Sep 2026 06:07:50 +0000

Health checks. Cache hits. A payment failure hiding in the middle. If every OpenTelemetry log goes into a reasoning model, you pay for noise before the investigation starts.

Jev Logs is a small open-source layer that puts TypeSafe’s Jev in front of those logs. Jev makes the first decision: how useful is this record, how urgent is it, and does it deserve a more expensive model?

I wrote it. MIT licensed. Independent not a TypeSafe, Vercel, or OpenTelemetry product.

reachjalil / jevlogs

Open-source Jev log triage for OpenTelemetry. Score the signal before expensive LLM analysis.

Jev Logs

A little intelligence between your logs and your LLM bill.
Score, prioritize, and route OpenTelemetry logs with Jev. Keep the signal. Keep your stack

Website · Guide · llms.txt · npm · Feedback

Meet your log filter’s smarter friend

Health checks. Cache hits. A payment failure hiding in the middle. Sending every event to a reasoning model adds cost before the investigation even starts.

Jev Logs makes the first decision: how useful is this log, how urgent is it, and does it deserve deeper analysis? It uses TypeSafe’s Jev through the Vercel AI SDK, with a small TypeScript API and an OpenTelemetry exporter wrapper.

A small layer	What you get
Score the signal	A 0–100 diagnostic-value score, priority, and actionable probability.
Keep your pipeline	Wrap your existing exporter; preserve resource, scope, timestamps, and trace context.
Start with visibility	Annotation mode keeps every record and attaches `jev.*` attributes.
Spend

…

View on GitHub

What Jev actually returns

Jev is built for structured choices, not paragraphs. For each log, Jev Logs asks it for:

a diagnostic value (0–100)
a priority (critical, high, normal, low)
an actionable probability
a route: analyze or retain

Your archive still gets every record. The analysis branch only needs the ones Jev (or a rule, or a conservative fallback) says are worth it.

A log may skip deeper analysis only when all three are true: priority is low, value is 25 or below, and actionable probability is under 0.1. Errors, jev.protected records, timeouts, and provider failures stay eligible. Nothing in the SDK deletes your logs.

Try Jev Logs in one command

Offline demo. No key. No network.

npx jevlogs

   0 / 100  low      RETAIN   GET /health returned 200 in 2ms
  25 / 100  low      RETAIN   Cache hit for product:482
 100 / 100  critical ANALYZE  Payment capture failed after three retries
  75 / 100  high     ANALYZE  Database connection pool at 94% capacity

That walkthrough uses fixed answers so you can see the shape. It does not call Jev.

Real Jev, still on your machine:

export AI_GATEWAY_API_KEY=your-vercel-ai-gateway-key

npx jevlogs --live --sample
npx jevlogs --live --file ./app.log --limit 20

--live alone starts a local OTLP HTTP/JSON receiver on http://127.0.0.1:4318/v1/logs. Point your app at it; Jev Logs prints one JSON decision per record and can forward annotated batches to the collector you already run. Node.js 22+.

Score a log from TypeScript

npm install jevlogs

import { createJevLogs } from "jevlogs";

const jev = createJevLogs();
const decision = await jev.triage({
  body: "Database connection pool at 94% capacity",
  severityText: "WARN",
});

console.log(decision);
// value · priority · route · actionableProbability · reason

Skip health checks without spending a Jev call:

const jev = createJevLogs({
  rules: [{ name: "health", match: "^GET /health", route: "retain" }],
});

Already on OpenTelemetry?

Wrap the exporter you have. Annotation mode keeps every log and attaches jev.* attributes.

import {
  LoggerProvider,
  BatchLogRecordProcessor,
  ConsoleLogRecordExporter,
} from "@opentelemetry/sdk-logs";
import { JevLogExporter } from "jevlogs";

const provider = new LoggerProvider({
  processors: [
    new BatchLogRecordProcessor({
      exporter: new JevLogExporter({
        exporter: new ConsoleLogRecordExporter(),
        mode: "annotate",
      }),
      maxExportBatchSize: 16,
    }),
  ],
});

Keep that archive processor. Add a second exporter with mode: "analysis-only" when you actually want to drop low-value records from the LLM path. Annotation alone does not cut the bill — the downstream pipeline has to honor route.

Why Jev, not “another LLM pass”

A second giant completion per log is the thing this is trying to avoid. Jev’s published rate is cheap structured evaluation (TypeSafe lists $0.042/M input, free output). You pay Jev for a small decision, then pay GPT-class analysis only for the selected slice.

The README has an illustrative table: 1M logs/month, if 10% still need analysis, a $1,000 GPT-4.1-style bill models down to about $129 including Jev triage. That is not a measured production result. Measure incident recall on your logs before you filter.

What this is not

No hosted dashboard. No log storage. No Collector plugin. No root-cause write-up. Preview software: jevlogs on npm, TypeScript first, CI on the repo. Jev itself is a hosted model via Vercel AI Gateway; this repo is the integration.

Links

If you try it, I want feedback on the Jev scoring/routing shape and whether wrapping an exporter is the right split vs the local receiver. Issues with sanitized examples are welcome.