DEV Community: marcelotaparelli

DefectRisk

marcelotaparelli — Sat, 10 Oct 2026 09:01:01 +0000

Review ~30% of modules → capture 71.26% of known defects.

In the historical once-only evaluation of the frozen raw Random Forest,
652 of 2,177 modules were flagged and 300 of the 421 modules with recorded
defects entered that queue. This measures prioritization; it does not show
that reviewers found every bug or validate the later calibrated system.

I built the evaluation process, compared models at equal review capacity,
and delivered a frozen artifact with a real inference CLI. DefectRisk
estimates risk to help a team decide where to start reviewing.

Problem

A team cannot inspect every module with equal attention. The useful question
is: if we can review only part of the codebase, can ML concentrate more
defects inside that queue?

Defective modules are the minority class. On JM1, predicting everything clean
would yield about 80.65% accuracy while capturing no defects. Recall,
precision, and review effort therefore guide the product decision.

Evaluation integrity came first

The project uses JM1 / OpenML 1053, version 1: 10,885 modules, 21 static
code metrics, and approximately 19.35% defective modules.

The initial audit found 299 complete feature vectors shared between
training and test in the random split. The data included duplicates and
conflicting-label groups: identical metrics with different defect records.
Evaluation could partly measure recognition of repeated examples rather
than generalization.

I grouped original vectors before splitting. Labels stay outside group
identity; identical vectors remain on the same side of every training/test,
validation, and cross-validation boundary. I retained the rows, including
conflicting ones, and added overlap audits before fitting. Imputation,
transformations, and class weights learn only from fitting partitions.

Model development

Development progressed through Logistic Regression baseline → class
weighting → threshold and review-workload analysis → Random Forest → HGB →
XGBoost, with limited nested tuning, feature engineering, and an ensemble
comparison.

The main decision was to compare models at equal review capacity,
especially around 30%, instead of relying on the default 0.50 threshold.
Five outer folds with groups estimated results; three inner folds selected
tree configurations without consulting the outer fold's outcomes.

In the training-only selection study, RF captured 953 of 1,685 defective
modules, versus 906 for balanced Logistic Regression, at similar workload.
HGB, XGBoost, derived features, and a small ensemble did not produce a
material gain sufficient to replace RF. The selected weighted forest has
200 trees, depth 8, and a minimum of three examples per leaf. These CV
comparisons are separate from the historical result below.

Historical raw RF result

Reviewing roughly one third of the highest-risk modules concentrated about
seven out of ten modules with recorded defects inside the queue.

The model and policy were frozen before the single held-out evaluation.
The policy selected its cutoff from scores and capacity without consulting
test labels, keeping tied scores together.

Measure	Historical held-out
Modules evaluated	2,177
Modules with recorded defects	421
Flagged for review	652 (29.95%)
Defective modules captured	300 of 421
Recall	71.26%
Precision	46.01%
F1	0.5592
Average Precision (AP)	0.6553
Defective modules missed	121

The queue also contained 352 modules recorded as clean, the false
positives. This is human review prioritization, not automatic bug detection.
71.26% is recall on this test, not accuracy, probability correctness, or
expected production performance.

This result belongs to the frozen raw RF. It is not final validation of
the calibrated artifact or abstention policies developed afterward.

Calibration and uncertainty

Later, I evaluated sigmoid and isotonic calibration with nested CV and
groups, using training data only. For the fixed RF configuration,
sigmoid improved Brier score from 0.18692 to 0.13863, while AP remained
approximately stable (0.40703 → 0.40619). The score measures probability
quality; interpretation also considers reliability curves and the support
within each range.

Calibration improved probability interpretation without creating new
predictive information. The attempt at automatic HIGH classification with
≥90% precision found no sufficiently supported tail. The conservative
policy left 98.44% UNCERTAIN: 136 LOW, zero HIGH, and 8,572 UNCERTAIN.
Even among LOW modules, seven had recorded defects.

The product remains risk ranking + human review prioritization.
DefectRisk predicts risk and ranking, not certainty. The historical test
was not reused for calibration or policy design; the later system still
requires a genuinely independent external holdout.

Executable artifact

The delivery includes the frozen calibrated artifact defectrisk-rf-sigmoid-v1,
an ordered feature schema, dependency versions, training fingerprints,
and checksums. The CLI validates the CSV and loads the verified artifact,
without training:

python -m pip install -e .
defectrisk rank examples/modules.csv

Run from the DefectRisk repository root with the recorded dependencies.
The flow is CSV with static metrics → verified frozen artifact →
calibrated probability → descending risk ranking. Module identifiers stay
outside the model; table and JSON output are available.

Schema and checksum checks verify consistency and integrity, not authenticity.
Joblib loading must use only trusted local artifacts. Reproducing saved CV
evidence neither reopens the historical test nor independently validates
the final artifact.

Limits and next investment

JM1 is historical; features contain only static metrics.
Label and snapshot provenance is imperfect; ambiguous or conflicting labels exist.
There is no guarantee of generalization across projects, languages, or versions.
Calibration and policies have training/CV evidence only and require a new independent external holdout with frozen settings.
Probabilities do not establish individual certainty or causality; a low score does not waive standard testing or review.

The available evidence suggested that the current features had become the
main bottleneck, without proving a mathematical information ceiling. The
recommended next investment is better prediction-time data: code churn,
defect history, ownership, test coverage, commit/change history, and review
history. These signals were not implemented; they would require clear
snapshots and label horizons.

The article When Better Models Are Not Enough
explains how I decided to stop model exploration.

Evidence

OpsPilot AI: Building RAG and Agents Without Giving the LLM Authority

marcelotaparelli — Wed, 07 Oct 2026 20:09:27 +0000

An operator finds context in runbooks and documentation, interprets what is happening, and turns some of that information into tracked work. Finding an instruction and opening an issue seem closely related. The second task, however, crosses a boundary: it produces an effect in another system.

That was my starting point for OpsPilot AI. I wanted to search tenant-owned operational knowledge, answer with references, and prepare a GitLab issue for another person to approve. The interesting question emerged when I stopped treating all of this as “the agent's response.”

Did the AI find enough evidence? Should it have authority to execute an external action? Those are different questions. A proposal can satisfy the schema, have a convincing title, and still fall outside the user's permissions.

I chose to keep probabilistic interpretation inside deterministic boundaries. A correctly generated action is still a proposal. The system authenticates, authorizes, records approval, and controls execution.

Define the system before adding the model

I modeled the authenticated principal, tenant knowledge, proposal, approval, and execution separately. Combining them into one “agent state” object would make it easy to confuse generated information with granted authority.

The principal carries tenant, subject, and roles derived from configured credentials. The proposal holds suggested work. Approval records who accepted which action. Execution records attempts and external outcomes. PostgreSQL persists this lifecycle; it does not depend solely on the process remaining alive.

I separated domain contracts from the database, model, and GitLab adapters. That lets me exercise a planner that proposes forbidden actions without paying for a model call to discover whether policy enforcement works.

The domain also constrains the product: each run can prepare at most one issue proposal. I did not give the agent a shell, arbitrary SQL, or permission to call any URL. Restricting the action space reduces what I need to authorize, observe, and recover. Adding tools would require revisiting the authority model, rather than just adding names to a prompt.

Retrieval must respect the access boundary

Before generating an answer, I needed to decide which documents could enter its context. Tenant identity is not an argument the model chooses. It comes from the authenticated principal and follows the transaction into the database.

Ingestion normalizes text, splits it into overlapping chunks, and stores text and embeddings atomically after external calls complete. Retrieval combines exact cosine vector search in pgvector with PostgreSQL full-text search. Reciprocal rank fusion combines ranking positions instead of assuming that scores from different retrieval methods are directly comparable. Embedding-space identifiers prevent comparisons between incompatible vectors.

Tenant filters apply before ranking and limits. Searching globally and filtering only the top-k would be a different architecture: other tenants' documents could consume the result budget, as well as expanding the opportunity for leakage.

I also configured transaction-local tenant context and forced row-level security. Tests against real PostgreSQL exercise missing predicates and connection-pool reuse. The limit matters: the runtime role can set tenant context, so RLS does not protect against arbitrary SQL executed as that role. It also does not provide document ACLs within a tenant.

RAG does not end when chunks are found

I separate the path into boundaries that can fail independently:

query → retriever → ranked evidence
                         ↓
                     generation
                         ↓
                 citation validation
                         ↓
                 answer or abstention

The adapter requests structured output. The application checks it again: fabricated or duplicate IDs are rejected, citations must belong to authorized retrieved chunks, and metadata comes from the database. Uncited output becomes a fixed abstention. When retrieval returns nothing, there is no reason to call generation.

This controls ID provenance, not the truth of every statement. A chunk can be authorized while an answer misinterprets its content. Citation membership does not establish entailment: citing an existing document does not prove that the conclusion follows from it.

That distinction changes diagnosis. Retrieval may miss the right document; generation may fail despite receiving the right context; or a formally valid citation may not support the claim. Measuring only the final answer would hide which part needs improvement.

I do not extend this control into a universal guarantee about issues either. An authorized proposal does not become semantically correct because it exists in the database. Human review still needs to assess its content.

Measure retrieval without mistaking ranking for an answer

The first retrieval dataset was too easy to distinguish strategies. Retrieval-v2 introduced synthetic documents and questions covering paraphrases, identifiers, ambiguity, multiple relevant documents, and hard negatives. I separated 24 development queries from 36 held-out queries, with labels defined before running the retrievers.

MRR@5 measures how early the first relevant document appears. A first-place result receives 1 for that query; second place receives 1/2; no relevant document within the five considered positions receives 0. The mean summarizes that behavior. The protocol deduplicates documents from retrieved chunks; several chunks from one document can still consume part of the retrieval budget.

Held-out strategy	MRR@5
Lexical	0.7532
Vector with fake embedder	0.4375
Hybrid with fake embedder	0.6306

Lexical retrieval outperformed hybrid retrieval in this experiment. The vector branch used deterministic word hashing, not a semantic model. RRF does not automatically repair a weak ranking: documents appearing in both branches can outrank a relevant result found only by the lexical branch.

The retrieval-v2 report attributes the result to commit 7ba3378. The held-out set was run once and is now consumed; CI uses development queries. Subsequent readiness changes altered the fingerprint, and the guard refuses to rerun the held-out on current code.

This does not show that hybrid search is generally worse. It does not measure real embeddings, groundedness, or answer quality. Evaluating another configuration would require a new frozen protocol and a fresh held-out set. That is the same concern about measurement scope I discussed in evals: stop guessing, start measuring.

The planner proposes; application code decides what is allowed

LangGraph orchestrates the workflow. It does not define the authority model.

The planner can select search_knowledge, prepare_gitlab_issue, or final_answer. create_gitlab_issue is not a model tool. The graph has no path from planning nodes to execution; entering execution depends on persisted state after approval.

I bounded steps, deadlines, and invalid-output attempts. These limits contain loops and protocol failures, but do not make a decision useful. A planner can finish within its budget and still propose work nobody needs.

During preparation, policy resolves the project alias within the tenant's configuration, checks labels, and maps permitted assignees to configured IDs. The model cannot supply an arbitrary project ID or choose its own permissions.

Imagine a client submitting this as supposed proof of authority:

{ "allowed_project": "secret-admin" }

This is an antipattern, not an OpsPilot contract. Valid JSON or a client assertion that something is “allowed” does not grant access. Authority must come from authenticated identity and application policy. The project's contracts reject extra fields; planner schemas contain no tenant, approval, or authorization flag.

Approve the exact action, not just the intention

“A human approved it” is incomplete. What was approved? Is that still the object about to be executed? Has policy changed in the meantime?

I persist the canonical action with tenant, requester, resolved project, title, description, labels, and assignees before requesting approval. A SHA-256 hash of canonical JSON binds the decision to that content. Normalization and ordering prevent equivalent representations of labels or assignees from producing different identities. Execution uses the stored action; it does not ask the model to generate it again.

The approver must hold the appropriate role, belong to the same tenant, and be a different subject from the requester. The transaction checks the supplied hash and recomputes the stored action's hash. The runtime role has no UPDATE permission on proposals or approvals.

Before producing the external effect, execution checks approval, hash, and current policy again. If the project was removed from the allowlist, the action fails closed. If the action changed outside the normal workflow, the approval no longer applies. A changed action requires a new run and approval.

probabilistic interpretation
            ↓
structured proposal
            ↓
deterministic policy
            ↓
human approval (action hash)
            ↓
approved immutable action
            ↓
revalidation → execution
            ↓
reconciliation / audit

The hash protects the link between content and decision. It does not evaluate whether the issue makes sense and does not replace authorization. Human review adds operational work. I accepted that cost to control a consequential external action.

GitLab's response was lost. Can I send the POST again?

This failure mode connected the agent most clearly to familiar distributed-systems problems: GitLab receives the create request, stores the issue, and loses the response. The client observes a timeout. The issue may already exist.

Immediately repeating the POST can duplicate work. I distinguish failure before sending, terminal rejection, and an ambiguous outcome. A timeout after possible transmission, a 5xx, or a malformed 2xx response can conceal a completed write. None necessarily means “nothing happened.”

Each approved action gets a stable key derived from tenant, run, and hash. The adapter includes a marker in the issue description. When an outcome is ambiguous, an explicit /resume request searches for that marker before considering another send.

If the issue is found, the system records success without creating another. If lookup fails, ambiguity remains. If nothing is found, a grace period must elapse before a resend is allowed, within the attempt budget. There is no automatic recovery worker.

PostgreSQL uses a lock, owner, and lease to control execution claims and prevent a late owner from overwriting a newer owner's records. A lease cannot cancel an HTTP request already in flight. Delayed search visibility or removal of the marker can also lead to duplicates. This is not an exactly-once guarantee.

Tests with real PostgreSQL and fake GitLab exercise lost responses, database failure after the external effect, and process death after creation. They establish recovery in those controlled scenarios. The real GitLab smoke validated creation, lookup, and resume of a successful run; it did not measure real lost-response recovery or indexing delays.

Security belongs outside the prompt

The most useful adversarial test assumes that the planner has already been compromised. Documents tell it to skip approval and use a forbidden project. The scripted planner obeys, requests direct creation, tries the forbidden project, and emits an approval decision.

In the exercised cases, application code refuses forbidden tools and projects. The only permitted proposal still waits for a human, with zero requests to GitLab. This tests the authority boundary even when interpretation fails.

Prompts that label evidence “untrusted data” and strict schemas help organize the interface. They do not replace authentication or authorization. They also do not eliminate answer poisoning within a tenant: the model can receive malicious information it is authorized to read.

The controls protect specific boundaries: tenant isolation, permitted tools, human decisions, runtime validation, and an audit trail. Adversarial regression verifies defined behaviors; it does not establish resistance to every prompt injection.

Observability follows the boundaries

A log saying “agent failed” would be insufficient. I need to distinguish missing evidence, model timeout, policy denial, time spent awaiting approval, and an ambiguous external effect.

I instrumented boundaries with OpenTelemetry: HTTP requests, retrieval and its branches, AI calls, planning, authorization, approval, execution, GitLab, and reconciliation. Request IDs, run IDs, and trace IDs relate logs, spans, and audit events.

Approval and resume are new requests. I do not claim that the entire human wait lives inside one continuous trace. The run and trace IDs recorded in the audit connect those entries. This lets me reconstruct the lifecycle without recording full prompts or documents.

Allowlists restrict attributes, and request, tenant, or document IDs do not become metric labels. That reduces content exposure and uncontrolled cardinality. Configured and served models are distinct facts; missing usage or cost remains unknown, rather than zero.

Telemetry export can lose signals and fails open so that it does not govern product availability. Authorization and approval remain fail-closed. Persisted state and the audit trail are authoritative; a successful span does not decide whether execution completed. The observability record documents these signals and local collector-failure tests.

Different evals answer different questions

The agent evaluation passed 16/16 cases with scripted/offline planners, real PostgreSQL, and fake GitLab. It measures controls under defined sequences, including adversarial behavior. It does not measure a real LLM's ability to select tools.

Retrieval-v2: exercises synthetic held-out ranking with fake embeddings. It does not establish real semantic retrieval or correct answers.
Agent eval: exercises offline policy, approval, states, and recovery. It does not establish real LLM decision quality.
OpenAI smoke: exercises the real API, schemas, embeddings, and small database-backed RAG. It does not establish quality, entailment, or measured cost.
GitLab smoke: exercises sandbox approval, creation, confirmation, and resume. It does not establish recovery from real faults or exactly-once.

Generation quality asks a different question: is the answer correct, and do its claims follow from the evidence? The project does not yet evaluate that behavior with a real model. Ranking, workflow, and integration checks can pass without answering that question.

The OpenAI record from October 4, 2026 confirms real 256-dimensional embeddings, answer and planner schemas, citations belonging to context, usage, served-model records, and a bounded timeout. The GitLab record from October 5 confirms no request before approval, creation and GET, a second approval returning 409, resume with one matching issue and one create POST, and confirmed closure.

These were separate runs, and the GitLab smoke uses an offline planner. There was no single live OpenAI + GitLab E2E. OpenAI pricing was unconfigured: passing a conditional cost check does not establish measured cost.

Release v0.1.0 is published on GitHub. Its local record documents 241 unit and 72 integration tests, plus clean-room reproduction. The current-state document records the owner's subsequent confirmation that hosted CI passed. Those counts describe that validation, not performance under production traffic. The live records retain each run's scope.

What remains unproven

The held-out is small, synthetic, English-only, and authored by one person. The fake embedder does not test semantic understanding. There is no real-model quality evaluation, measured cost, or validation of the combined system against real operational traffic.

Static tokens provide neither SSO nor identity lifecycle management. There are no within-tenant ACLs. Exact vector search scales linearly; a small corpus does not establish scale. Recovery requires explicit resume, and search-based reconciliation retains its visibility and in-flight-request limits.

Terraform describes an AWS blueprint validated with offline plans. No AWS deployment was performed; cloud runtime, restore, and the AWS telemetry path remain unvalidated. Passing configuration validation does not establish operation in that environment.

The release-image scan retained 44 HIGH findings without reported fixes, with zero fixable HIGH/CRITICAL findings. The gate blocks fixable findings; passing it does not mean zero vulnerabilities. Four classified IaC risks also remain documented. These are historical results, not a new scan performed for this article.

What I take from this project

My main lesson was to make the contract between interpretation and effect explicit. The model can retrieve context, interpret a request, and propose work. The software system must determine authority, verify approval and external outcomes, and retain recoverable states when those outcomes are uncertain.

The next evaluations need to address the remaining gaps: real embeddings under a new protocol, answer quality, and live recovery. A passing smoke test does not answer those questions. Separating these responsibilities taught me more than simply getting an agent to call a tool.

Evidence and code

I inspected the public OpsPilot AI repository at snapshot cacf611. Links pin that state; the retrieval-v2 benchmark belongs to the historical freeze identified in its report.

Current state and validation provenance .
Workflow, approval, recovery, and adversarial tests .
Deterministic policy in code .
Isolation, untrusted evidence, and RLS limits .
Image scan and gate scope .
OpsPilot AI case for the project overview.

Laya vs Jev in Practice: Zero-Shot, Fine-Tuning, and the Real Cost of an Open-Source Decision Model

marcelotaparelli — Fri, 02 Oct 2026 23:43:01 +0000

TypeSafe announced Jev on September 15, 2026, introducing its first “System One Model.” Laya's official Hugging Face history dates the commit marked “Initial release” to September 18, 2026. Those public records are three days apart — an interval between an announcement and a release record, not a measure of development time. Sources: TypeSafe's announcement and Laya's initial release record.

By the time I completed the Jev 1.13 evaluation, Laya was already an open-weight alternative worth investigating. Learning a tool was just the first step. The more durable work was evaluating a new approach with a sound method and measuring whether it improves the system.

On the same Ops Triage AI held-out set, Laya went from 8/70 correct tuples zero-shot to 60/70 after domain adaptation. Jev had reached 66/70 without domain-specific training. The interesting question is what changes when we can modify a model's weights, and what we take responsibility for in exchange.

First, what is a decision model?

Jev is a non-generative AI model specialized in structured decisions. It takes state or context expressed in natural language and returns typed decisions with probability distributions, rather than generating an open-ended token sequence like a conventional generative LLM.

It is still AI. TypeSafe calls the category System One Models; that is the company's terminology, not a universally established academic taxonomy. I also use the more neutral terms “probabilistic decision model” and “structured decision model.”

Generative LLM
text → token-by-token generation → text

Decision model
state + set of choices → distribution → typed decision

This changes both the interface and the task. In ticket triage, category, priority, and risk can be closed choices. The software receives values it already understands, without asking for a paragraph and extracting an enum afterward.

LLMs can produce structured outputs too. The distinction goes beyond model size or JSON formatting: a decision model is designed to score a decision space instead of generating free-form text. A bounded answer space reduces parsing problems and out-of-schema values. A valid choice can still be semantically wrong.

Where Laya fits

Laya addresses a similar role with code and weights released under Apache 2.0, non-autoregressive inference, self-hosting, and fine-tuning. The general English checkpoint combines ModernBERT-large with a decision head for 421 million parameters, according to the model card at the revision we used.

We pinned convaiinnovations/laya at revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851, using laya==0.3.23. The runtime release was published on October 1, 2026. Library version and weight revision are separate identities, and both were frozen.

This base checkpoint is distinct from laya-typed-decisions, which has already been fine-tuned on other workflows. We did not use the multilingual checkpoint either. Here, “zero-shot” means no training specific to our domain, not a model with no prior training.

A decision model inside a real architecture

A voice-driven website-building demonstration I watched made the role easier to understand. It described a library of more than 1,000 assets/components, with Jev selecting actions and assets. The conceptual flow was:

voice → real-time transcription → context adaptation/correction
      → decision model → asset/action selection → UI

This is an account of the observed demonstration, without a verifiable URL in the project context or benchmark value.

Jev does not have to generate the website. A speech model can transcribe, another layer can normalize context, and a decision model can choose among permitted actions. Speech models, LLMs, and decision models can serve different roles in the same architecture. Code remains responsible for executing and validating the actions.

The benchmark we already had

Our most useful asset was a frozen held-out set of 70 synthetic English tickets, previously used for the deterministic baseline, Ollama, and Jev 1.13. We kept the tickets, labels, and metric definitions. Exact-tuple accuracy counts a ticket as correct only when category, priority, and risk all match.

We split the investigation into two questions:

Experiment	Comparison	Question
A — out of the box	Historical Jev 1.13 × base Laya zero-shot	What happens without domain-specific training?
B — domain adaptation	The same, unchanged Jev result × adapted Laya	How much can specialization close the observed gap?

We did not rerun Jev. We retained its frozen historical result through OpenRouter: typesafe/jev-1.13, resolved to typesafe/jev-1.13-20260917.

Experiment B is asymmetric. Jev received no domain-specific training examples; Laya received 1,120 TRAIN tickets. The taxonomy is shared, but prompts and inference paths differ. Laya's questions were adapted to its token budget before the freeze and held constant between A and B. We are evaluating generalization without adaptation and the capacity to specialize, with different exposure to the domain.

Experiment A: base Laya zero-shot

For this general checkpoint, zero-shot, on this taxonomy and frozen held-out set:

Metric	Base Laya zero-shot
Category accuracy	84.29% — 59/70
Priority accuracy	31.43% — 22/70
Risk accuracy	62.86% — 44/70
Exact tuple	11.43% — 8/70

The clearest pattern was 40/40 expected LOW priorities predicted as MEDIUM. Before attributing that to the model, the audit examined options, order, indices, mapping, serialization, and normalization. It found no integration bug that invalidated the result: the archived probabilities placed MEDIUM first, matching the published choice.

The audit used parsed outputs with copied choices and probabilities; untouched native responses were not retained byte for byte. That evidence supports checking the mapping: MEDIUM was already the model's choice, with no LOW-to-MEDIUM conversion in the adapter.

Why this does not contradict Laya's published benchmark

The material that prompted the investigation showed much stronger results. The laya-typed-decisions model card explains the key difference: that checkpoint was fine-tuned on the corresponding benchmark's workflows. The card distinguishes the specialized checkpoint from the base model and notes that its Jev figures were published by third parties, not measured in the same run.

The published metrics belong to a different dataset, protocol, and checkpoint, and measure individual decisions rather than our per-ticket tuple. That distinction raised a second question: was weak zero-shot quality structural, or was this model missing adaptation to our domain?

Experiment B: keep learning separate from evaluation

We created 1,120 TRAIN tickets and 280 VALIDATION tickets, also synthetic, from the operational taxonomy. A deterministic generator used authored scenarios, without an LLM and without reading the held-out file or its errors to construct examples.

TRAIN — 1,120 tickets → changes weights
VALIDATION — 280 tickets → selects checkpoint
HELD-OUT — the same 70 tickets → final evaluation after the freeze

The held-out set stayed outside dataset generation, training, validation, checkpoint selection, thresholds, calibration, and hyperparameter tuning. This was a follow-up on a benchmark whose aggregate result we already knew; its file and errors were isolated from construction of the new data.

VALIDATION shares scenario families with TRAIN, although the wording is distinct. It supports checkpoint selection, but is not an independent real-world population.

Fine-tuning started from the same base checkpoint as A. Our deterministic labels became one-hot targets, unlike the soft distributions from a probabilistic teacher in Laya's official example. We did not fit temperatures afterward: candidate checkpoints used temperature 1.0. This is not an exact replication of the external benchmark recipe, and the difference may affect confidence.

The selection rule favored the highest VALIDATION exact-tuple accuracy, with predeclared tie-breakers based on macro-F1, mean Brier, and earliest epoch. Epoch 4 was selected. Freeze 2, commit 00a9a64, fixed data, checkpoint, code, inference, and thresholds before the single official adapted evaluation: 70/70 valid responses and zero failures.

The infrastructure needed to adapt the model

The local host offered four vCPUs, roughly 4 GB of RAM, and no GPU. Full fine-tuning there was not a reasonable option. We used a temporary NVIDIA RTX A5000 with 24 GB VRAM, Python 3.11, and PyTorch 2.5.1 with CUDA 12.1.

Training ran for four epochs with mixed precision — fp16 autocast and fp32 master weights — and gradient accumulation to manage memory use.

This supplied concrete evidence of my direction in ML Systems / AI Infrastructure. The task included checking whether training accounting matched the updates that actually happened.

In the first run, GradScaler skipped an optimizer update while the scheduler advanced anyway. We detected the anomaly, changed the scheduler to advance only after applied updates, preserved the initial run's artifacts, and repeated training before opening the adapted held-out evaluation.

The corrected official run recorded 420 update attempts, 413 applied optimizer updates, seven GradScaler skips, and 413 scheduler steps. It became the candidate on methodological validity, not because its accuracy was more convenient. Selection, freeze, and final evaluation followed that correction.

The jump: 11.43% → 85.71%

On the same held-out set:

Metric	Laya zero-shot	Adapted Laya
Category accuracy	84.29% — 59/70	97.14% — 68/70
Priority accuracy	31.43% — 22/70	94.29% — 66/70
Risk accuracy	62.86% — 44/70	92.86% — 65/70
Exact tuple	11.43% — 8/70	85.71% — 60/70

LOW → MEDIUM errors fell from 40/40 to 0/40. Adaptation clearly changed behavior in this domain, as documented in the public final report.

Fine-tuning did not fix everything. Two categories, four priorities, and five risk labels remained wrong. Errors overlapped across fields, leaving ten incorrect tuples. HIGH/CRITICAL priority recall decreased from 13/14 zero-shot to 12/14 after adaptation. Improving aggregate accuracy does not guarantee improvement in every severity error.

Jev and adapted Laya: read the asymmetry carefully

Metric	Jev 1.13 — none of our TRAIN tickets	Adapted Laya — 1,120 TRAIN tickets
Category accuracy	100%	97.14%
Priority accuracy	98.57%	94.29%
Risk accuracy	95.71%	92.86%
HIGH/CRITICAL priority recall	100% — 14/14	85.71% — 12/14
HIGH risk recall	85.71% — 6/7	100% — 7/7
Exact tuple	94.29% — 66/70	85.71% — 60/70

Jev showed strong out-of-the-box generalization on this benchmark. Laya showed how much control open source offers when we can adapt a model to the domain. Jev's 94.29% came without our TRAIN tickets; Laya went from 11.43% zero-shot to 85.71% with adaptation. Adapted Laya had higher HIGH-risk recall, but the denominator is just seven examples.

Confidence remained a problem

Adapted Laya's mean answer_confidence — its rounded top-option probability — was nearly saturated:

Field	Mean answer_confidence	Multiclass Brier	Top-label ECE
Category	0.99986	0.05715	0.02844
Priority	0.99997	0.11429	0.05711
Risk	0.99884	0.13841	0.07027

Ten of the 70 tuples were still wrong. Brier and ECE describe this small sample; they do not certify calibration on future traffic.

The exploratory gate required all three answer_confidence values to meet the threshold. From 0 through 0.90, it retained 70/70 tickets at 85.71% exact-tuple accuracy. At 0.95 and 0.99, it retained 69/70 at 86.96%. It barely filtered errors. No threshold was selected or changed using these results.

This returns to the caution in the Jev article: high confidence does not automatically mean a reliable probability of correctness. Jev reports its own confidence; Laya distinguishes answer_confidence from an entropy-based confidence. Those different semantics prevent a direct comparison of the values.

The gap between learning and generalization

Split, selected checkpoint	Tickets	Exact tuple
TRAIN	1,120	100%
VALIDATION	280	92.50%
HELD-OUT	70	85.71%

The sequence 100% → 92.50% → 85.71%, TRAIN loss recorded as zero in the final epoch, and saturated confidence show a very strong fit to training data. Since VALIDATION still shares scenario families with TRAIN, the held-out drop deserves attention.

I interpret the perfect fit and split differences as signs consistent with memorization and overconfidence. The sample is too small to determine the severity of that effect or estimate a production generalization gap.

Local did not automatically mean faster

Model	Measured path	Mean latency per ticket
Base Laya	Local CPU, four AMD EPYC vCPUs, fp32	1,414 ms
Adapted Laya	Same CPU context, fp32	1,619 ms
Jev 1.13	Historical API/network call	569 ms

Laya timings include tokenization, inference over three questions, parsing, and validation; the per-request mean excludes model loading. Adapted held-out inference ran on CPU, not the training A5000.

On this specific CPU, self-hosted Laya was not automatically faster than calling Jev remotely. Since infrastructure and measurement dates differ, these timings describe the observed serving paths. Historical Ollama hybrid latency also measures a broader path than standalone inference.

Open source does not mean zero cost

Jev's historical run reported US$ 0.003789618 for 70 requests. That is API usage cost for that run, not a current pricing projection or a complete operating-cost estimate.

Laya inference incurred US$ 0 in external per-token/request charges. CPU/GPU, RAM/VRAM, electricity, cloud resources, and operations still matter. CPU cost was not monetized because the necessary rate and measurements were unavailable.

For training, the recorded Pod rate was approximately US$ 0.27 per GPU-hour, or US$ 0.28/hour with container disk. Measured tasks — base VALIDATION, the first anomalous run, and corrected training/evaluation — totaled 1,523.51 seconds, about 25m24s.

The report estimates US$ 0.1185 for those tasks at the rate including disk. That is not the Pod's total bill: it excludes setup, provider startup, and idle time; total billed time and the invoice were unavailable. The official CPU evaluation added no GPU time.

Laya removes per-token charges while moving cost and responsibility into infrastructure and operations. Preparing data, investigating an anomaly, and managing artifacts also belong in the decision, even without a monetary measurement of engineering time.

The real trade-off: convenience and control

Jev demonstrated the value of a decision model that works very well without training specific to this domain. Laya demonstrated a different advantage: open weights let us change a model when zero-shot is insufficient.

Open source did not automatically give us better accuracy, lower latency, or better calibration. It gave us control. That meant taking responsibility for the dataset, training, GPU, checkpoint, calibration, versioning, deployment, observability, and operating costs. This experiment exercised data, training, and evaluation; deploying and operating the adapted service remain future work.

That is the engineering decision: how much is the ability to adapt behavior and host the model under our constraints worth, given the work required to sustain it?

Limitations

A synthetic, English-only HELD-OUT set of just 70 tickets from one domain; no real traffic or production validation.
One official run per protocol; Jev is a historical result, without repeated runs to measure variance.
Synthetic TRAIN and VALIDATION data; VALIDATION shares scenario families with TRAIN.
Different domain exposure, questions, and inference paths for Jev and Laya.
Different hardware and measurement dates; latency paths are not equivalent.
Different confidence semantics; calibration and coverage analyses are descriptive and based on small samples.
Untouched native responses were not preserved byte for byte for the zero-shot audit.
The roughly 843 MB adapted checkpoint is outside public Git. Its hash, size, metadata, and results are public, but the weights are not available for download there.
Total billed Pod cost was unavailable; the estimate covers measured tasks only.
Jev and Laya remain evaluation-only in Ops Triage. Neither was integrated into HybridPolicy because of these experiments.

What I would do in a real system

I would start with labeled real tickets permitted for that use, separating scenarios, sources, and time periods so VALIDATION does not merely rephrase TRAIN. I would set acceptance criteria by severity and error cost before opening a new final evaluation.

Calibration and thresholds would need separate, representative data, with human review and fallback evaluated as part of the policy. I would repeat protocols to measure variance and compare latency, throughput, loading, memory, and cost on the serving paths I actually intend to operate.

Before integration, I would version weights and configuration alongside data and code, verify the license and context limits, and prepare monitoring for drift and errors, plus rollback. Typed decisions simplify integration; reliability still depends on the surrounding system.

Evidence and reproducibility

All figures come from the public reports and artifacts in Ops Triage AI, inspected at commit e8c75f0. The links below pin that snapshot. Its README still summarizes only zero-shot; domain-adaptation evidence lives in the final report and artifacts.

Zero-shot: Experiment A report and result JSON.
Adaptation: protocol and Freeze 2.
Experiment B results: held-out JSON and attempt journal.
Training and selection: corrected run, VALIDATION selection, and weight metadata.
Audit: mapping and archived outputs and preserved initial run.
Historical reference: Jev 1.13 report and Jev artifact.
Upstream project: official Laya GitHub repository and license at the runtime release.

Jev 1.13 in Practice: I Tested a Decision Model Against Rules and a Local LLM

marcelotaparelli — Wed, 23 Sep 2026 02:33:26 +0000

Jev 1.13 got every category classification right in my held-out set: 1.0000 category accuracy. The same experiment measured 0.9857 priority accuracy and 0.9571 risk accuracy. Those are results on 70 synthetic tickets; they do not prove that Jev is superior in production.

I decided to test Jev because I already had two reference points in ops-triage-ai: a deterministic rules baseline and a classifier using local Ollama. The question was whether a third paradigm — a typed probabilistic decision model — would produce different results on the same taxonomy.

What changes compared with a chat LLM

I did not use Jev as a free-form conversation and then interpret prose afterward. A separate adapter sent decisions through the OpenRouter Decisions API using typesafe/jev-1.13, with typed responses and probability distributions. All 73 API responses during the experiment resolved to typesafe/jev-1.13-20260917.

The adapter implements the TriageClassifier interface, but remained separate from the Ollama classifier and HybridPolicy. I used Bun native fetch and added no dependencies. Jev was not integrated into the application flow and did not replace the local model.

How I froze the experiment

I used the same taxonomy and the same 70 synthetic labels from the historical held-out benchmark. Development calls happened before the freeze. In commit 4c41e0b, I froze the configuration and evaluation, then ran the held-out set once for the official result.

That separation matters: the final set was not used to tune the model and then measure that tuning on the same set. It is still a small synthetic sample, but the protocol makes clear what was frozen and which run produced the numbers.

Results

Metric	Deterministic	Ollama	Jev 1.13
Category accuracy	0.8286	0.9571	1.0000
Category macro-F1	0.8512	0.9550	1.0000
Priority accuracy	0.9000	0.9143	0.9857
Risk accuracy	0.9571	0.9143	0.9571
HIGH/CRITICAL priority recall	0.7857	1.0000	1.0000
HIGH risk recall	0.5714	0.7143	0.8571

Jev got the full category, priority, and risk tuple right in 66 of 70 cases (0.9429). The historical benchmark did not emit standalone exact-tuple accuracy for the deterministic classifier or Ollama. I therefore do not compare 66/70 with per-field metrics or the hybrid path's exact tuple: they measure different things.

Measured cost and latency

Across Jev's 70 standalone calls, mean latency was 569.4 ms, with p50 at 545.5 ms, p95 at 712.2 ms, and a maximum of 1,142.2 ms. The report recorded 90,229 input tokens, a total reported cost of US$ 0.003789618, and zero API or schema failures.

That cost is the reported amount for this OpenRouter run. The historical Ollama benchmark did not emit cost or standalone latency. Its published 6–7 seconds describe the full hybrid path; that is not an equivalent comparison with a standalone Jev call.

Confidence is not semantic correctness

Jev preserved probability distributions, continuous confidence, and top-1/top-2 margins. That gives more information for evaluating a decision, but it does not guarantee the decision is right: one incorrect HIGH risk classification had 0.86 confidence.

For this observed dataset, exploratory calibration analysis reported Brier scores of 0.0005 for category, 0.0386 for priority, and 0.0800 for risk; top-1 ECE was 0.0054, 0.0329, and 0.0226, respectively. These are statistics over 70 synthetic tickets, with few examples in each range. They do not show that probabilities are calibrated for real tickets.

Selective automation: an exploratory signal

When requiring all three confidence values to reach a threshold of 0.80, 54 of 70 tickets remained covered, and all covered cases had the observed tuple correct. At 0.90, 51 of 70 remained, also with 100% observed exact accuracy.

This does not define a production threshold. The denominator is small, the examples are synthetic, and the result does not establish calibration or performance beyond this set. It is a hypothesis to test with representative data and separately defined review criteria.

What this benchmark does not prove

It does not estimate performance in production or on real tickets.
One official run does not measure run-to-run variance.
70 synthetic examples do not represent the full domain distribution.
The result does not establish that Jev is generally superior to Ollama or rules.
Exploratory calibration metrics do not validate confidence for real decisions.
I did not compare Ollama standalone cost or latency because the historical benchmark did not emit them.

What I would do architecturally

I would keep the current decisions: the deterministic baseline and Ollama remain in the existing evaluation and hybrid policy; Jev stays benchmark/evaluation only. Before any integration, I would expand and diversify the labeled data, repeat the frozen evaluation to measure variance, and define how human review and severity errors factor into acceptance criteria.

The useful result is not “Jev won.” A third paradigm produced different measurable signals in a reproducible run with explicit limits — material for guiding the next evaluation, not for justifying a production replacement.

Evidence

Ops Triage AI case
Repository: ops-triage-ai on GitHub
Jev 1.13 held-out report at the freeze commit: jev-1.13-held-out-4c41e0b.md
Run JSON artifact: jev-1.13-held-out-4c41e0b.json

A Valid JWT Does Not Mean Authorized Access

marcelotaparelli — Sat, 19 Sep 2026 21:12:04 +0000

A security review of Salus brought me back to a question that remains after authentication:

Can this user access this specific resource?

Consider this scenario:

User A creates patient X.
User B signs in and receives a valid JWT.
B calls GET /patients/X.

If the application looks up the patient by patientId alone, B may receive a patient belonging to A. B is authenticated, but is not authorized to access that object. This is Broken Object Level Authorization (BOLA).

Authorization must reach the query

A valid JWT establishes who made the request. The access rule still needs to use that identity to scope the resource lookup:

patientId + currentUser.id → authorized resource

Instead of:

findById(patientId)

the operation could conceptually use:

findByIdAndOwner(patientId, currentUser.id)

The method name is incidental. The access boundary belongs in the rule and the lookup, including reads, updates, and deletes.

What should the API return for someone else's patient?

A 403 Forbidden response can tell B that patient X exists. When even that fact is sensitive, 404 Not Found may be preferable: X is outside the set of resources B is allowed to see.

The choice depends on the API's policy and should be applied consistently. It makes HTTP responses less useful for probing which IDs exist.

Security as testable behavior

The path does not end at login → JWT → middleware. It continues through the operation on each object:

identity → authentication → authorization → ownership → resource access

This is the property that became a Salus security regression test:

User A creates patient X.
User B tries to access X.
Expected result: 404.

The hardening work reproduced this scenario as a failing test (RED): User A creates X, User B receives a valid JWT and tries to access it. Ownership was then introduced, operations were scoped by ownerId, and cross-user access began returning 404 in this context. The regression test now passes (GREEN), alongside JWT verification on patient routes.

TDD and security regression tests protect this behavior as the system evolves.

Security should be verifiable behavior, not just configuration.

Project: Salus on GitHub

Evals: I Stopped Asking Whether the LLM “Looks Good” and Started Measuring

marcelotaparelli — Wed, 16 Sep 2026 16:03:28 +0000

When I started working with LLMs, one of the hardest questions looked deceptively simple:

how do I know the model is actually getting better?

Running a few examples by hand and thinking "that answer looks good" works at first.

But it does not scale.

And, more importantly, it produces no evidence.

That is when I started to understand the role of evals.

What is an eval?

An eval is a structured way of testing the behavior of an AI system.

The idea is fairly simple:

input
+
produced answer
+
expected answer
+
metric
=
evaluation

Instead of subjectively asking whether an answer turned out well, you define up front what you expect and measure the distance between actual behavior and expected behavior.

An article by Martin Fowler on GenAI patterns describes this kind of mechanism as scoring and judging: the model's output goes through a scorer that produces metrics or feedback about the result.

That is exactly the principle I applied in my operational triage project.

Before the LLM, I wrote code

In ops-triage-ai, the system receives tickets and has to determine things like:

category;
priority;
risk;
suggested team.

Before putting an LLM on the problem, I implemented a deterministic classifier with hand-written rules in TypeScript.

Simplified:

ticket
   ↓
deterministic rules
   ↓
classification

That created something extremely valuable:

a baseline.

I now had a concrete implementation to compare any AI-based solution against.

Then I wrote down expected answers

I set aside a collection of tickets and defined in advance what the correct classification for each one should be.

Conceptually something like:

{
  "ticket": {
    "title": "Production API unavailable",
    "description": "Users cannot access the service"
  },
  "expected": {
    "category": "INCIDENT",
    "priority": "CRITICAL",
    "risk": "HIGH"
  }
}

The classifier receives the ticket.

Its output is compared against expected.

And then we compute metrics.

               ┌──────────────────┐
ticket ───────►│    classifier    │
               └────────┬─────────┘
                        │
                        ▼
                 predicted output
                        │
expected output ────────┤
                        ▼
                    scorer
                        │
                        ▼
                    metrics

That scorer can be plain code.

It does not need to be another LLM.

My code became part of the experiment

That is where I found the most interesting idea.

The deterministic code I would normally write to solve the problem also became an experimental reference.

I could run:

dataset
   ├── deterministic classifier
   └── LLM classifier

and compare both on exactly the same examples.

On the project's final held-out set of 70 synthetic tickets, for instance:

Category accuracy

Deterministic: 82.9%
LLM:           95.7%

For HIGH/CRITICAL priority recall:

Deterministic: 78.6%
LLM:           100%

But something even more important happened.

The model did not win everywhere.

On overall risk classification:

Deterministic: 95.7%
LLM:            91.4%

Without an eval, it would have been easy to look at a few good LLM answers and conclude the model was simply better.

The metrics told a more interesting story.

And then the eval started shaping the architecture

At that point the question stopped being:

How do I make the LLM replace my rules?

and became:

Where does each approach work best?

That pushed the project toward a hybrid architecture.

                 ticket
                   │
          ┌────────┴────────┐
          ▼                 ▼
 deterministic           LLM
 classifier           classifier
          │                 │
          └────────┬────────┘
                   ▼
              hybrid policy
                   │
             ┌─────┴─────┐
             ▼           ▼
         decision    human review

The baseline stopped being just an old version of the system.

It started serving as:

a reference;
a divergence signal;
a fallback;
a component of the human-review policy.

The eval did not just measure the architecture.

It helped determine the architecture.

It also changes how you develop with LLMs

Without structured evaluation, the loop tends to look like this:

change the prompt
↓
run a few examples
↓
looks better
↓
deploy

With evals:

change prompt/model/policy
↓
run the dataset
↓
measure results
↓
compare against baseline
↓
analyze regressions
↓
decide

That difference looks small.

But it is the difference between experimenting and simply trusting an impression.

Not every metric needs to come from an LLM

There is a lot of discussion about LLM-as-a-judge, where another model grades the produced answer.

That is useful when the criteria are subjective, such as:

clarity;
relevance;
coherence;
quality of an open-ended answer.

But when there is a verifiable answer, traditional code is usually simpler.

In my case:

predicted.category === expected.category;

already answers an important question.

The evaluation tool should be proportional to the problem.

The main takeaway

I used to think of evals as something that happens after building an AI system.

I see it differently now.

The eval is part of development itself.

It helps answer:

did the model improve?
where did it get worse?
by how much?
in which cases?
compared to what?

And perhaps the most important question:

does this improvement actually justify putting the LLM in this part of the system?

Without that, it is easy to build a convincing demo.

With it, engineering starts to show up.

Keep exploring

The full details — official metrics, trade-offs, and limitations — are in the project case.

Project: ops-triage-ai on GitHub

The LLM Didn't Win Everywhere — and That's What Made the Project Interesting

marcelotaparelli — Tue, 15 Sep 2026 09:33:21 +0000

I closed an important stage of ops-triage-ai, an operational triage
system that combines a deterministic baseline, a local LLM, and a hybrid
policy with human review.

The most interesting result was not simply "the LLM was better."

What the benchmark measured

Evaluation on a frozen held-out set of 70 synthetic tickets, in a single
run. Deterministic baseline versus the local LLM:

Category accuracy: 82.9% → 95.7%
HIGH/CRITICAL priority recall: 78.6% → 100%
HIGH risk recall: 57.1% → 71.4%

And the point that matters: the deterministic baseline still won on
overall risk accuracy — 95.7% against the LLM's 91.4%.

The LLM did not win everywhere. The regression is published alongside the
gains, with nothing smoothed over.

The question changed

Instead of:

"How do we replace rules with AI?"

the architecture answers:

"How do we combine different behaviors in a way that is safe, observable,
and auditable?"

The final architecture

Deterministic baseline as reference and fallback;
Local LLM for semantic interpretation;
Hybrid policy to decide when human review is required;
Audit trail preserving ticket, run, decision, and feedback.

In the final benchmark:

51.4% of cases were routed to human review;
the classifiers disagreed on 32.9% of tickets;
there was no fallback and no LLM failure during the official run.

Review rate describes operations here — how many cases asked for review —
not review precision/recall. The held-out set has no independent ground
truth for that.

Accepted limits

Synthetic dataset, in English, with 70 examples. A single official run,
with no measurement of LLM variance. The system has not been validated
under real production traffic. Hybrid latency sits around 6 to 7 seconds:
fine for asynchronous triage, not for a critical synchronous path. The
full details — official metrics, trade-offs, and the 13 published
limitations — are in the project case.

The takeaway

A useful AI system does not need to trust the model blindly.

Sometimes the best architecture comes precisely from understanding where
the model is better — and where it is not.

Project: https://github.com/marcelotaparelli/ops-triage-ai

Why I Built My Portfolio with Bun + Astro + MDX Instead of a More Complex Stack

marcelotaparelli — Sun, 13 Sep 2026 13:05:47 +0000

Choosing a stack is a product decision before it is a technical one. This
article documents why this portfolio runs on Bun, Astro, and MDX — and,
more importantly, the reasoning that led me to turn down more sophisticated
alternatives. Nothing here is theory: every claim points to something
verifiable in this repository.

The problem

I needed a portfolio that could carry professional authority as a Software
Engineer going deeper into Applied AI Engineering. The requirements were
concrete: bilingual content (Portuguese and English), project cases and
articles with their own identity, solid SEO, high performance, genuine
accessibility, and maintenance simple enough that I would never have to
think about it twice. No backend, no platform team, no infrastructure
budget — just me, the code, and the content.

The question behind the architecture

The guiding question was simple: what is the simplest solution capable of
solving this problem? Not "which stack is trending," and not "which one
shows off the most skill." Technology chosen for hype often charges
interest in complexity. Every layer I added had to justify itself against this specific
problem — a mostly static content site maintained by one person.

Why Bun

Bun is this project's toolchain: package manager, runner for the
development scripts, and runtime for the validation tooling. The dev
server, build, artifact validation, asset measurement, and release-check
scripts are TypeScript executed directly by Bun, with no separate
transpilation layer for tooling. I make no comparative claims I haven't
measured here — the decision was about structural simplicity: one tool
where there would otherwise be several. What is observable is objective
but contextual: in this implementation, on the local environment, the full
build renders every page in about two seconds.

Why Astro

This site's content is predominantly static, so HTML is generated at build
time and served as files — no application server, no database, no
per-request runtime. Astro was chosen for exactly that model: it lets me
write composable editorial content while shipping almost no JavaScript to
the browser. The script embedded in the home page is 277 bytes (198
gzipped) and exists for exactly one job: the mobile menu. Zero bytes of
client-side framework — measured, not estimated. React was never installed
because no problem required React. This model also simplifies SEO: every
page arrives as complete HTML, with its own canonical and alternates.

Why MDX

Articles and cases live versioned alongside the code, in the same
repository, under the same review process. There is no CMS or backend to
maintain, update, or pay for — and no content outside version control. MDX
gives the content structure (schema-validated frontmatter requiring a
bilingual pair, slug, category, and review status), which makes every piece
of writing an artifact as reviewable as any other code. Publishing means
merging and building; rolling back means reverting a commit.

Why not Next.js, SPA, or CMS

Not because they are bad technologies — they are excellent for the right
problems. A full-stack framework would solve problems I don't have:
per-request rendering, API routes, global client state. A client-rendered
SPA would add JavaScript runtime and browser state to a site whose main
job is delivering ready-to-read content. A CMS would trade versioned files for an external dependency with logins,
backups, and a bill. None of that complexity was justified by the current
problem, so none of it got in.

Bilingualism as an architecture requirement

Bilingualism is not a plugin bolted on afterward — it shaped the
architecture. Portuguese lives at /, English at /en/, and every
published item is required to exist in both languages: the publication
model rejects at build time anything published without its reviewed
counterpart. Every page carries its own canonical and three alternates
(current language, opposite language, and x-default), and the artifact
validator checks the pairs across every generated page. Incomplete content
simply never reaches production: drafts are excluded from the production
build and included only in the preview build. The projects page
and the about page exist in both languages because the
system would accept nothing less.

Quality as part of the product

Here, quality is not a promise — it is a pipeline. Every change goes
through strict typechecking (zero errors), linting, verified formatting,
unit tests for the publication model (4/4), thirteen browser tests
(reciprocal PT/EN navigation, keyboard menu with Escape, no-JavaScript
navigation, localized 404, explicit external links, 320px reflow with
automated WCAG checks), and
artifact validation in both build modes. If something breaks, the build
says so — before any human needs to check.

CI/CD pipeline with GitHub Actions covering type checking, linting, unit
tests, Playwright E2E tests, artifact validation, and production builds.
Delivery follows a Continuous Delivery model: the same CI-approved
artifact is published after explicit manual approval, without rebuilding,
to a dedicated production branch and then deployed to Hostinger.

Measured performance

What follows are laboratory measurements, never real-user data. I ran
Lighthouse 13.4.1 in headless Chromium, mobile simulation (412×823
viewport, simulated network and CPU throttling), three runs per page,
serving the local static build — methodology recorded in
scripts/lighthouse.ts, results in reports/lighthouse-summary.json. On
the home page: performance 100 and accessibility 100 across all three
runs, LCP between roughly 1.5 s and 1.7 s, CLS around 0.0006, zero TBT,
and about 93.5 KB of initial transfer. The asset budget
(reports/assets.json) shows where the lightness comes from: roughly
6 KB of gzipped CSS, about 0.2 KB of inline JavaScript, and fonts totaling
around 48 KB. Lab numbers inform decisions; they prove nothing about real
user experience, and I would never present them as such.

A bug worth finding

The artifact validator flagged a broken link on the Portuguese 404 page:
/404/. The cause was in the header's language switcher, which used
Astro.url.pathname as the current-language link — and with trailing
slashes always on, that pathname renders as /404/. But the actual
artifact Astro generates for 404.astro is 404.html, a special 404
document for static hosting; the /404/ route never existed. The fix went
into the routing model (the 404 page now links its canonical
/404.html and /en/404/ paths), not into the test. Weakening the
validator would have hidden the symptom and destroyed its value. The
episode became a working rule: fix the publication model, never work
around the validator.

Trade-offs

Every choice has a cost, and these are mine, accepted: no backend, no
database, no CMS — any new content requires a commit, a build, and a
deploy. No React at launch — richer interactivity in the future will
require revisiting the decision. No analytics — there is no real-usage
telemetry, so the lab metrics above are the ceiling of what I can claim
today. And bilingualism costs double review: every piece must exist, make
sense, and be approved in both languages. A deliberate cost, because
international reach and consistency across languages are part of the
product I want to build.

Conclusion

Mature engineering is not about choosing the most sophisticated stack —
it is about choosing complexity proportional to the problem. This
portfolio could have been a full-stack monolith, an SPA, or a CMS
instance; those approaches could work, but they would introduce
capabilities and operational costs this project's requirements never asked
for. Technology starts from the human
problem, not from the code: understand what needs to change, choose
deliberately, and examine the outcome with evidence. FROM REAL PROBLEMS TO
INTELLIGENT PRODUCTS — including when building the showcase itself.

⚡ C# Tip: ToCharArray() vs ToArray() — Why It Matters More Than You Think

marcelotaparelli — Fri, 04 Jul 2025 18:29:21 +0000

As developers, we often write code that “just works” — but sometimes, small choices can have a big impact on performance.

One example? The subtle but meaningful difference between ToCharArray() and ToArray() when working with strings in C#. Let’s break it down 👇

🧩 What Each One Does

✅ ToCharArray()

Native method on the string class.
Returns a char[].
Fast and direct.

🚧 ToArray()

Extension method from LINQ (System.Linq).
Treats the string as IEnumerable.
Adds abstraction and intermediate steps.

🔍 The Overhead Problem

Calling ToArray() on a string might seem harmless, but it invokes LINQ under the hood. That means:

Extra allocations
Added abstraction
Unnecessary iteration

Meanwhile, ToCharArray() skips all that and copies the characters directly — with less memory and better performance. In tight loops, I've seen it run 2–3x faster. That adds up.

🧪 When Should You Use Each?

Let’s keep it simple:

👉 Use ToCharArray() when you're working directly with strings — parsing, looping, checking characters, etc.
👉 Use ToArray() when you're working with LINQ and want to convert the result of a query to an array.

Example:

// Using LINQ to filter and convert to array
var expensive = products
 .Where(p => p.Price > 100)
 .ToArray(); // 👍 This is a perfect case for ToArray()

💡 Final Thoughts

Is this a micro-optimization? Maybe.
But understanding how your tools work under the hood can make you a better developer — and sometimes, a faster one too.

If this helped or surprised you, give it a share so others don’t fall into the same trap 😄

I’d love to hear your thoughts — have you encountered other subtle performance gotchas in C#?