DEV Community: marcelotaparelli

Jev 1.13 in Practice: I Tested a Decision Model Against Rules and a Local LLM

marcelotaparelli — Wed, 23 Sep 2026 02:33:26 +0000

Jev 1.13 got every category classification right in my held-out set: 1.0000 category accuracy. The same experiment measured 0.9857 priority accuracy and 0.9571 risk accuracy. Those are results on 70 synthetic tickets; they do not prove that Jev is superior in production.

I decided to test Jev because I already had two reference points in ops-triage-ai: a deterministic rules baseline and a classifier using local Ollama. The question was whether a third paradigm — a typed probabilistic decision model — would produce different results on the same taxonomy.

What changes compared with a chat LLM

I did not use Jev as a free-form conversation and then interpret prose afterward. A separate adapter sent decisions through the OpenRouter Decisions API using typesafe/jev-1.13, with typed responses and probability distributions. All 73 API responses during the experiment resolved to typesafe/jev-1.13-20260917.

The adapter implements the TriageClassifier interface, but remained separate from the Ollama classifier and HybridPolicy. I used Bun native fetch and added no dependencies. Jev was not integrated into the application flow and did not replace the local model.

How I froze the experiment

I used the same taxonomy and the same 70 synthetic labels from the historical held-out benchmark. Development calls happened before the freeze. In commit 4c41e0b, I froze the configuration and evaluation, then ran the held-out set once for the official result.

That separation matters: the final set was not used to tune the model and then measure that tuning on the same set. It is still a small synthetic sample, but the protocol makes clear what was frozen and which run produced the numbers.

Results

Metric	Deterministic	Ollama	Jev 1.13
Category accuracy	0.8286	0.9571	1.0000
Category macro-F1	0.8512	0.9550	1.0000
Priority accuracy	0.9000	0.9143	0.9857
Risk accuracy	0.9571	0.9143	0.9571
HIGH/CRITICAL priority recall	0.7857	1.0000	1.0000
HIGH risk recall	0.5714	0.7143	0.8571

Jev got the full category, priority, and risk tuple right in 66 of 70 cases (0.9429). The historical benchmark did not emit standalone exact-tuple accuracy for the deterministic classifier or Ollama. I therefore do not compare 66/70 with per-field metrics or the hybrid path's exact tuple: they measure different things.

Measured cost and latency

Across Jev's 70 standalone calls, mean latency was 569.4 ms, with p50 at 545.5 ms, p95 at 712.2 ms, and a maximum of 1,142.2 ms. The report recorded 90,229 input tokens, a total reported cost of US$ 0.003789618, and zero API or schema failures.

That cost is the reported amount for this OpenRouter run. The historical Ollama benchmark did not emit cost or standalone latency. Its published 6–7 seconds describe the full hybrid path; that is not an equivalent comparison with a standalone Jev call.

Confidence is not semantic correctness

Jev preserved probability distributions, continuous confidence, and top-1/top-2 margins. That gives more information for evaluating a decision, but it does not guarantee the decision is right: one incorrect HIGH risk classification had 0.86 confidence.

For this observed dataset, exploratory calibration analysis reported Brier scores of 0.0005 for category, 0.0386 for priority, and 0.0800 for risk; top-1 ECE was 0.0054, 0.0329, and 0.0226, respectively. These are statistics over 70 synthetic tickets, with few examples in each range. They do not show that probabilities are calibrated for real tickets.

Selective automation: an exploratory signal

When requiring all three confidence values to reach a threshold of 0.80, 54 of 70 tickets remained covered, and all covered cases had the observed tuple correct. At 0.90, 51 of 70 remained, also with 100% observed exact accuracy.

This does not define a production threshold. The denominator is small, the examples are synthetic, and the result does not establish calibration or performance beyond this set. It is a hypothesis to test with representative data and separately defined review criteria.

What this benchmark does not prove

It does not estimate performance in production or on real tickets.
One official run does not measure run-to-run variance.
70 synthetic examples do not represent the full domain distribution.
The result does not establish that Jev is generally superior to Ollama or rules.
Exploratory calibration metrics do not validate confidence for real decisions.
I did not compare Ollama standalone cost or latency because the historical benchmark did not emit them.

What I would do architecturally

I would keep the current decisions: the deterministic baseline and Ollama remain in the existing evaluation and hybrid policy; Jev stays benchmark/evaluation only. Before any integration, I would expand and diversify the labeled data, repeat the frozen evaluation to measure variance, and define how human review and severity errors factor into acceptance criteria.

The useful result is not “Jev won.” A third paradigm produced different measurable signals in a reproducible run with explicit limits — material for guiding the next evaluation, not for justifying a production replacement.

Evidence

Ops Triage AI case
Repository: ops-triage-ai on GitHub
Jev 1.13 held-out report at the freeze commit: jev-1.13-held-out-4c41e0b.md
Run JSON artifact: jev-1.13-held-out-4c41e0b.json

A Valid JWT Does Not Mean Authorized Access

marcelotaparelli — Sat, 19 Sep 2026 21:12:04 +0000

A security review of Salus brought me back to a question that remains after authentication:

Can this user access this specific resource?

Consider this scenario:

User A creates patient X.
User B signs in and receives a valid JWT.
B calls GET /patients/X.

If the application looks up the patient by patientId alone, B may receive a patient belonging to A. B is authenticated, but is not authorized to access that object. This is Broken Object Level Authorization (BOLA).

Authorization must reach the query

A valid JWT establishes who made the request. The access rule still needs to use that identity to scope the resource lookup:

patientId + currentUser.id → authorized resource

Instead of:

findById(patientId)

the operation could conceptually use:

findByIdAndOwner(patientId, currentUser.id)

The method name is incidental. The access boundary belongs in the rule and the lookup, including reads, updates, and deletes.

What should the API return for someone else's patient?

A 403 Forbidden response can tell B that patient X exists. When even that fact is sensitive, 404 Not Found may be preferable: X is outside the set of resources B is allowed to see.

The choice depends on the API's policy and should be applied consistently. It makes HTTP responses less useful for probing which IDs exist.

Security as testable behavior

The path does not end at login → JWT → middleware. It continues through the operation on each object:

identity → authentication → authorization → ownership → resource access

This is the property that became a Salus security regression test:

User A creates patient X.
User B tries to access X.
Expected result: 404.

The hardening work reproduced this scenario as a failing test (RED): User A creates X, User B receives a valid JWT and tries to access it. Ownership was then introduced, operations were scoped by ownerId, and cross-user access began returning 404 in this context. The regression test now passes (GREEN), alongside JWT verification on patient routes.

TDD and security regression tests protect this behavior as the system evolves.

Security should be verifiable behavior, not just configuration.

Project: Salus on GitHub

Evals: I Stopped Asking Whether the LLM “Looks Good” and Started Measuring

marcelotaparelli — Wed, 16 Sep 2026 16:03:28 +0000

When I started working with LLMs, one of the hardest questions looked deceptively simple:

how do I know the model is actually getting better?

Running a few examples by hand and thinking "that answer looks good" works at first.

But it does not scale.

And, more importantly, it produces no evidence.

That is when I started to understand the role of evals.

What is an eval?

An eval is a structured way of testing the behavior of an AI system.

The idea is fairly simple:

input
+
produced answer
+
expected answer
+
metric
=
evaluation

Instead of subjectively asking whether an answer turned out well, you define up front what you expect and measure the distance between actual behavior and expected behavior.

An article by Martin Fowler on GenAI patterns describes this kind of mechanism as scoring and judging: the model's output goes through a scorer that produces metrics or feedback about the result.

That is exactly the principle I applied in my operational triage project.

Before the LLM, I wrote code

In ops-triage-ai, the system receives tickets and has to determine things like:

category;
priority;
risk;
suggested team.

Before putting an LLM on the problem, I implemented a deterministic classifier with hand-written rules in TypeScript.

Simplified:

ticket
   ↓
deterministic rules
   ↓
classification

That created something extremely valuable:

a baseline.

I now had a concrete implementation to compare any AI-based solution against.

Then I wrote down expected answers

I set aside a collection of tickets and defined in advance what the correct classification for each one should be.

Conceptually something like:

{
  "ticket": {
    "title": "Production API unavailable",
    "description": "Users cannot access the service"
  },
  "expected": {
    "category": "INCIDENT",
    "priority": "CRITICAL",
    "risk": "HIGH"
  }
}

The classifier receives the ticket.

Its output is compared against expected.

And then we compute metrics.

               ┌──────────────────┐
ticket ───────►│    classifier    │
               └────────┬─────────┘
                        │
                        ▼
                 predicted output
                        │
expected output ────────┤
                        ▼
                    scorer
                        │
                        ▼
                    metrics

That scorer can be plain code.

It does not need to be another LLM.

My code became part of the experiment

That is where I found the most interesting idea.

The deterministic code I would normally write to solve the problem also became an experimental reference.

I could run:

dataset
   ├── deterministic classifier
   └── LLM classifier

and compare both on exactly the same examples.

On the project's final held-out set of 70 synthetic tickets, for instance:

Category accuracy

Deterministic: 82.9%
LLM:           95.7%

For HIGH/CRITICAL priority recall:

Deterministic: 78.6%
LLM:           100%

But something even more important happened.

The model did not win everywhere.

On overall risk classification:

Deterministic: 95.7%
LLM:            91.4%

Without an eval, it would have been easy to look at a few good LLM answers and conclude the model was simply better.

The metrics told a more interesting story.

And then the eval started shaping the architecture

At that point the question stopped being:

How do I make the LLM replace my rules?

and became:

Where does each approach work best?

That pushed the project toward a hybrid architecture.

                 ticket
                   │
          ┌────────┴────────┐
          ▼                 ▼
 deterministic           LLM
 classifier           classifier
          │                 │
          └────────┬────────┘
                   ▼
              hybrid policy
                   │
             ┌─────┴─────┐
             ▼           ▼
         decision    human review

The baseline stopped being just an old version of the system.

It started serving as:

a reference;
a divergence signal;
a fallback;
a component of the human-review policy.

The eval did not just measure the architecture.

It helped determine the architecture.

It also changes how you develop with LLMs

Without structured evaluation, the loop tends to look like this:

change the prompt
↓
run a few examples
↓
looks better
↓
deploy

With evals:

change prompt/model/policy
↓
run the dataset
↓
measure results
↓
compare against baseline
↓
analyze regressions
↓
decide

That difference looks small.

But it is the difference between experimenting and simply trusting an impression.

Not every metric needs to come from an LLM

There is a lot of discussion about LLM-as-a-judge, where another model grades the produced answer.

That is useful when the criteria are subjective, such as:

clarity;
relevance;
coherence;
quality of an open-ended answer.

But when there is a verifiable answer, traditional code is usually simpler.

In my case:

predicted.category === expected.category;

already answers an important question.

The evaluation tool should be proportional to the problem.

The main takeaway

I used to think of evals as something that happens after building an AI system.

I see it differently now.

The eval is part of development itself.

It helps answer:

did the model improve?
where did it get worse?
by how much?
in which cases?
compared to what?

And perhaps the most important question:

does this improvement actually justify putting the LLM in this part of the system?

Without that, it is easy to build a convincing demo.

With it, engineering starts to show up.

Keep exploring

The full details — official metrics, trade-offs, and limitations — are in the project case.

Project: ops-triage-ai on GitHub

The LLM Didn't Win Everywhere — and That's What Made the Project Interesting

marcelotaparelli — Tue, 15 Sep 2026 09:33:21 +0000

I closed an important stage of ops-triage-ai, an operational triage
system that combines a deterministic baseline, a local LLM, and a hybrid
policy with human review.

The most interesting result was not simply "the LLM was better."

What the benchmark measured

Evaluation on a frozen held-out set of 70 synthetic tickets, in a single
run. Deterministic baseline versus the local LLM:

Category accuracy: 82.9% → 95.7%
HIGH/CRITICAL priority recall: 78.6% → 100%
HIGH risk recall: 57.1% → 71.4%

And the point that matters: the deterministic baseline still won on
overall risk accuracy — 95.7% against the LLM's 91.4%.

The LLM did not win everywhere. The regression is published alongside the
gains, with nothing smoothed over.

The question changed

Instead of:

"How do we replace rules with AI?"

the architecture answers:

"How do we combine different behaviors in a way that is safe, observable,
and auditable?"

The final architecture

Deterministic baseline as reference and fallback;
Local LLM for semantic interpretation;
Hybrid policy to decide when human review is required;
Audit trail preserving ticket, run, decision, and feedback.

In the final benchmark:

51.4% of cases were routed to human review;
the classifiers disagreed on 32.9% of tickets;
there was no fallback and no LLM failure during the official run.

Review rate describes operations here — how many cases asked for review —
not review precision/recall. The held-out set has no independent ground
truth for that.

Accepted limits

Synthetic dataset, in English, with 70 examples. A single official run,
with no measurement of LLM variance. The system has not been validated
under real production traffic. Hybrid latency sits around 6 to 7 seconds:
fine for asynchronous triage, not for a critical synchronous path. The
full details — official metrics, trade-offs, and the 13 published
limitations — are in the project case.

The takeaway

A useful AI system does not need to trust the model blindly.

Sometimes the best architecture comes precisely from understanding where
the model is better — and where it is not.

Project: https://github.com/marcelotaparelli/ops-triage-ai

Why I Built My Portfolio with Bun + Astro + MDX Instead of a More Complex Stack

marcelotaparelli — Sun, 13 Sep 2026 13:05:47 +0000

Choosing a stack is a product decision before it is a technical one. This
article documents why this portfolio runs on Bun, Astro, and MDX — and,
more importantly, the reasoning that led me to turn down more sophisticated
alternatives. Nothing here is theory: every claim points to something
verifiable in this repository.

The problem

I needed a portfolio that could carry professional authority as a Software
Engineer going deeper into Applied AI Engineering. The requirements were
concrete: bilingual content (Portuguese and English), project cases and
articles with their own identity, solid SEO, high performance, genuine
accessibility, and maintenance simple enough that I would never have to
think about it twice. No backend, no platform team, no infrastructure
budget — just me, the code, and the content.

The question behind the architecture

The guiding question was simple: what is the simplest solution capable of
solving this problem? Not "which stack is trending," and not "which one
shows off the most skill." Technology chosen for hype often charges
interest in complexity. Every layer I added had to justify itself against this specific
problem — a mostly static content site maintained by one person.

Why Bun

Bun is this project's toolchain: package manager, runner for the
development scripts, and runtime for the validation tooling. The dev
server, build, artifact validation, asset measurement, and release-check
scripts are TypeScript executed directly by Bun, with no separate
transpilation layer for tooling. I make no comparative claims I haven't
measured here — the decision was about structural simplicity: one tool
where there would otherwise be several. What is observable is objective
but contextual: in this implementation, on the local environment, the full
build renders every page in about two seconds.

Why Astro

This site's content is predominantly static, so HTML is generated at build
time and served as files — no application server, no database, no
per-request runtime. Astro was chosen for exactly that model: it lets me
write composable editorial content while shipping almost no JavaScript to
the browser. The script embedded in the home page is 277 bytes (198
gzipped) and exists for exactly one job: the mobile menu. Zero bytes of
client-side framework — measured, not estimated. React was never installed
because no problem required React. This model also simplifies SEO: every
page arrives as complete HTML, with its own canonical and alternates.

Why MDX

Articles and cases live versioned alongside the code, in the same
repository, under the same review process. There is no CMS or backend to
maintain, update, or pay for — and no content outside version control. MDX
gives the content structure (schema-validated frontmatter requiring a
bilingual pair, slug, category, and review status), which makes every piece
of writing an artifact as reviewable as any other code. Publishing means
merging and building; rolling back means reverting a commit.

Why not Next.js, SPA, or CMS

Not because they are bad technologies — they are excellent for the right
problems. A full-stack framework would solve problems I don't have:
per-request rendering, API routes, global client state. A client-rendered
SPA would add JavaScript runtime and browser state to a site whose main
job is delivering ready-to-read content. A CMS would trade versioned files for an external dependency with logins,
backups, and a bill. None of that complexity was justified by the current
problem, so none of it got in.

Bilingualism as an architecture requirement

Bilingualism is not a plugin bolted on afterward — it shaped the
architecture. Portuguese lives at /, English at /en/, and every
published item is required to exist in both languages: the publication
model rejects at build time anything published without its reviewed
counterpart. Every page carries its own canonical and three alternates
(current language, opposite language, and x-default), and the artifact
validator checks the pairs across every generated page. Incomplete content
simply never reaches production: drafts are excluded from the production
build and included only in the preview build. The projects page
and the about page exist in both languages because the
system would accept nothing less.

Quality as part of the product

Here, quality is not a promise — it is a pipeline. Every change goes
through strict typechecking (zero errors), linting, verified formatting,
unit tests for the publication model (4/4), thirteen browser tests
(reciprocal PT/EN navigation, keyboard menu with Escape, no-JavaScript
navigation, localized 404, explicit external links, 320px reflow with
automated WCAG checks), and
artifact validation in both build modes. If something breaks, the build
says so — before any human needs to check.

CI/CD pipeline with GitHub Actions covering type checking, linting, unit
tests, Playwright E2E tests, artifact validation, and production builds.
Delivery follows a Continuous Delivery model: the same CI-approved
artifact is published after explicit manual approval, without rebuilding,
to a dedicated production branch and then deployed to Hostinger.

Measured performance

What follows are laboratory measurements, never real-user data. I ran
Lighthouse 13.4.1 in headless Chromium, mobile simulation (412×823
viewport, simulated network and CPU throttling), three runs per page,
serving the local static build — methodology recorded in
scripts/lighthouse.ts, results in reports/lighthouse-summary.json. On
the home page: performance 100 and accessibility 100 across all three
runs, LCP between roughly 1.5 s and 1.7 s, CLS around 0.0006, zero TBT,
and about 93.5 KB of initial transfer. The asset budget
(reports/assets.json) shows where the lightness comes from: roughly
6 KB of gzipped CSS, about 0.2 KB of inline JavaScript, and fonts totaling
around 48 KB. Lab numbers inform decisions; they prove nothing about real
user experience, and I would never present them as such.

A bug worth finding

The artifact validator flagged a broken link on the Portuguese 404 page:
/404/. The cause was in the header's language switcher, which used
Astro.url.pathname as the current-language link — and with trailing
slashes always on, that pathname renders as /404/. But the actual
artifact Astro generates for 404.astro is 404.html, a special 404
document for static hosting; the /404/ route never existed. The fix went
into the routing model (the 404 page now links its canonical
/404.html and /en/404/ paths), not into the test. Weakening the
validator would have hidden the symptom and destroyed its value. The
episode became a working rule: fix the publication model, never work
around the validator.

Trade-offs

Every choice has a cost, and these are mine, accepted: no backend, no
database, no CMS — any new content requires a commit, a build, and a
deploy. No React at launch — richer interactivity in the future will
require revisiting the decision. No analytics — there is no real-usage
telemetry, so the lab metrics above are the ceiling of what I can claim
today. And bilingualism costs double review: every piece must exist, make
sense, and be approved in both languages. A deliberate cost, because
international reach and consistency across languages are part of the
product I want to build.

Conclusion

Mature engineering is not about choosing the most sophisticated stack —
it is about choosing complexity proportional to the problem. This
portfolio could have been a full-stack monolith, an SPA, or a CMS
instance; those approaches could work, but they would introduce
capabilities and operational costs this project's requirements never asked
for. Technology starts from the human
problem, not from the code: understand what needs to change, choose
deliberately, and examine the outcome with evidence. FROM REAL PROBLEMS TO
INTELLIGENT PRODUCTS — including when building the showcase itself.

⚡ C# Tip: ToCharArray() vs ToArray() — Why It Matters More Than You Think

marcelotaparelli — Fri, 04 Jul 2025 18:29:21 +0000

As developers, we often write code that “just works” — but sometimes, small choices can have a big impact on performance.

One example? The subtle but meaningful difference between ToCharArray() and ToArray() when working with strings in C#. Let’s break it down 👇

🧩 What Each One Does

✅ ToCharArray()

Native method on the string class.
Returns a char[].
Fast and direct.

🚧 ToArray()

Extension method from LINQ (System.Linq).
Treats the string as IEnumerable.
Adds abstraction and intermediate steps.

🔍 The Overhead Problem

Calling ToArray() on a string might seem harmless, but it invokes LINQ under the hood. That means:

Extra allocations
Added abstraction
Unnecessary iteration

Meanwhile, ToCharArray() skips all that and copies the characters directly — with less memory and better performance. In tight loops, I've seen it run 2–3x faster. That adds up.

🧪 When Should You Use Each?

Let’s keep it simple:

👉 Use ToCharArray() when you're working directly with strings — parsing, looping, checking characters, etc.
👉 Use ToArray() when you're working with LINQ and want to convert the result of a query to an array.

Example:

// Using LINQ to filter and convert to array
var expensive = products
 .Where(p => p.Price > 100)
 .ToArray(); // 👍 This is a perfect case for ToArray()

💡 Final Thoughts

Is this a micro-optimization? Maybe.
But understanding how your tools work under the hood can make you a better developer — and sometimes, a faster one too.

If this helped or surprised you, give it a share so others don’t fall into the same trap 😄

I’d love to hear your thoughts — have you encountered other subtle performance gotchas in C#?