DEV Community: adjia mvoa ndji gabriel monfils

MailGuard Digest: Engineering a Local-First DMARC Analysis Pipeline with Python

adjia mvoa ndji gabriel monfils — Thu, 29 Oct 2026 15:18:58 +0000

What if analyzing email authentication failures didn't require uploading your organization's email infrastructure data to a cloud AI service?

That question became the starting point for MailGuard Digest, an open-source Python application that transforms DMARC aggregate reports into structured security findings and readable briefings.

🔗 Source code: https://github.com/kiddsaut07/mailguard-digest

This project sits at the intersection of email security, defensive engineering, data validation, local persistence, and optional local AI.

But the interesting part isn't simply parsing an XML file or adding a language model to a Flask application. It's designing a pipeline that can process untrusted input, produce traceable results, preserve the distinction between evidence and interpretation, and remain useful even when AI is unavailable.

That's the engineering problem I wanted to tackle for Hacktoberfest 2026.

1. The engineering problem: turning DMARC XML into actionable evidence

DMARC aggregate reports provide visibility into how receiving systems evaluate messages claiming to originate from a domain.

Depending on the report, they can contain:

Reporting organization and report-period metadata.
Published SPF, DKIM, and DMARC policy information.
Source IP addresses and message counts.
SPF and DKIM authentication results.
DMARC disposition and alignment-related information.

These reports are valuable for identifying misconfigured sending services, unexpected sources, and authentication failures. However, they are commonly delivered as XML documents, often compressed or bundled in archives.

Manually inspecting multiple reports quickly becomes tedious.

A useful analysis tool therefore needs to do more than deserialize XML. It needs to validate the input, normalize the relevant fields, calculate meaningful statistics, identify noteworthy conditions, preserve the results, and communicate the findings without overstating what the evidence proves.

The core challenge: build a reliable analysis pipeline in which security-relevant facts are produced by explicit application logic—not invented by a language model.

2. Architecture: deterministic analysis first, AI second

MailGuard Digest follows a deliberately lightweight architecture built around Flask, Python, SQLite, Pydantic, and defusedxml.

At a conceptual level, the processing pipeline is:

DMARC report upload
        |
        v
File type and size checks
        |
        v
Safe XML / archive processing
        |
        v
Schema validation and normalization
        |
        v
Deterministic analysis engine
        |
        +----------------------+
        |                      |
        v                      v
Structured findings     Optional AI narration
        |                      |
        +-----------+----------+
                    |
                    v
          Readable security briefing
                    |
                    v
        SQLite / JSON / CSV / print

The diagram represents the application's intended processing flow rather than a claim that every operation runs in a separate service.

The main architectural decision is the separation between analysis and explanation.

The deterministic layer calculates counts, evaluates explicit conditions, and produces structured findings. Optional AI narration can help explain those findings in natural language, but it is not the authority for the underlying statistics or security conclusions.

This separation makes the application easier to test, reason about, and debug.

3. Secure ingestion: treat uploaded files as untrusted input

A report parser should not assume that every uploaded file is well-formed, harmless, or even the format its filename claims it is.

MailGuard Digest supports:

.xml
.xml.gz
.zip

Supporting these formats introduces several input-handling concerns: malformed XML, unexpected archive contents, oversized inputs, excessive record counts, and resource consumption.

XML parsing with `defusedxml`

XML parsing is a security boundary, not just a data extraction step.

MailGuard Digest uses defusedxml to reduce exposure to dangerous XML constructs associated with entity expansion and related parser attacks.

This is one layer of defense, not a guarantee that every malicious input is harmless. File validation, processing limits, error handling, and dependency maintenance still matter.

Bounded processing

The application also applies limits to archive and record processing.

This matters because a compressed file can expand dramatically, and a report containing a large number of records can consume significant CPU and memory even when the compressed upload appears small.

The objective is to make resource consumption more predictable and reject inputs that exceed the application's configured boundaries.

Validation with Pydantic

Parsing and validation are different operations.

A document can be syntactically valid XML but still contain missing, malformed, or inconsistent values. Pydantic models provide a structured validation layer for the data the application expects to work with.

Together, these controls establish a more disciplined ingestion boundary:

Accept supported input, reject invalid data where appropriate, normalize what can be processed safely, and handle failures explicitly.

4. Deterministic security analysis: facts before interpretation

Once a report has been parsed and validated, the application can calculate its statistics and identify relevant conditions.

The analysis focuses on information such as:

Total reported message counts.
SPF and DKIM authentication outcomes.
DMARC evaluation and disposition information.
Sources associated with authentication failures.
Conditions that may warrant configuration review or further investigation.

The important distinction is between a measurement, a finding, and a conclusion.

For example:

Measurement: a set of messages has an SPF failure result.
Finding: the affected source deserves investigation, especially if it is expected to send mail for the domain.
Conclusion: the source is malicious.

The first can be extracted from the report. The second depends on context and explicit analysis rules. The third cannot be established from an authentication failure alone.

A legitimate sender can fail authentication because of configuration errors, forwarding behavior, or other delivery-path issues. Conversely, passing authentication does not prove that a message is benign.

MailGuard Digest is therefore designed to support investigation, not to act as a definitive email threat-detection engine.

This distinction is essential in defensive security: an authentication result is evidence, not a complete verdict.

5. Local-first storage and export

A local security tool should make its results useful without requiring a separate infrastructure stack.

MailGuard Digest uses SQLite to retain analyzed report results and their history. It also supports exporting results to JSON and CSV, with browser-based printing for PDF output.

These choices keep the MVP relatively lightweight:

Component	Purpose
Flask	Application routes and user-facing workflow
`defusedxml`	Safer XML parsing
Pydantic	Structured data validation
SQLite	Local persistence
JSON / CSV	Portable analysis exports
pytest	Automated verification
Ollama	Optional local AI narration

A deliberate privacy choice is that the application does not retain the original uploaded XML reports by default. This reduces unnecessary raw-report retention, although the structured analysis stored in the database can still contain sensitive infrastructure information.

Local storage should not be confused with automatic security: database files, exports, backups, and the machine itself still need appropriate protection.

6. Optional local AI with Ollama

The next question was how to make technical findings easier to understand without making a paid cloud AI API a requirement.

MailGuard Digest includes an optional integration with Ollama for local language-model narration.

The intended workflow is straightforward:

The deterministic analysis produces structured findings.
The application prepares the information needed for an explanation.
A compatible local model can generate a natural-language briefing.
The briefing complements the structured results rather than replacing them.

The key design constraint is that AI narration is optional. The application should not depend on a model to calculate authentication statistics or determine whether the underlying report was parsed correctly.

Running a model locally can avoid sending its input to a remote AI provider, provided the workflow is genuinely local and no other component transmits that data. It also shifts the resource burden to the user's machine.

Local inference is not free of trade-offs: model availability, RAM, CPU performance, latency, and output quality all affect the experience.

I also distinguish between implementing an integration and validating it in a real environment. Compatibility with a particular model and successful end-to-end inference should be reported only when verified with that model and configuration.

7. Web application security is part of the implementation

Even a local-first tool needs a threat model.

MailGuard Digest includes CSRF protection and security-related response headers, alongside its input-processing controls.

These mechanisms address different risks:

CSRF protection: helps defend state-changing requests against cross-site request forgery.
Security headers: provide browser-side protections depending on the specific headers and deployment configuration.
Input validation and processing limits: reduce exposure to malformed files and resource-exhaustion conditions.
Loopback-oriented deployment: limits the intended exposure of an MVP that does not yet implement authentication and authorization.

That final point is especially important.

MailGuard Digest is intended for local use and development, not as a ready-made multi-user security service. It should not be exposed directly to an untrusted network without a proper deployment review and additional access controls.

Security is not a checklist that makes an application invulnerable. It is a set of controls whose effectiveness depends on the threat model, implementation, configuration, and testing.

8. Testing the application, not just the happy path

A parser that works for one sample report is not enough.

The project's final acceptance audit reported 417 passing tests. The reported test coverage spans areas including parsing, analysis rules, persistence, web routes, security protections, exports, and AI-related safeguards.

The audit also included a fresh virtual-environment installation and Python compilation checks.

These checks provide evidence that the implementation passed the project's automated acceptance suite in the tested environment. They do not prove the absence of defects, establish universal compatibility, or replace dedicated penetration testing and broader deployment testing.

Further useful work includes additional resource-exhaustion testing, accessibility checks, and verification across different supported runtime environments.

I consider this distinction important: a passing test suite is a meaningful engineering result, but it is not a certification of security.

9. Why keep the stack lightweight?

I deliberately chose a conventional Python stack instead of introducing a frontend framework, a separate API service, a distributed database, and container orchestration into the MVP.

The reasoning was practical:

Fewer moving parts to configure and maintain.
A smaller operational footprint for local execution.
A straightforward path from uploaded data to analysis results.
A testable separation between parsing, validation, analysis, persistence, and narration.
No mandatory paid AI API.

The goal was not to avoid complexity at all costs. It was to spend complexity where it matters: input safety, data correctness, explainable rules, failure handling, and tests.

10. Current limitations and next steps

MailGuard Digest is an open-source project for local DMARC analysis. It is not yet presented as a production-ready, multi-user service.

Current limitations include:

No authentication or authorization system.
A loopback-oriented deployment model.
The need for additional adversarial resource-exhaustion testing.
Further browser accessibility and deployment hardening work.
Dependence on local hardware and configuration for optional AI inference.

These limitations are part of the project's current scope, not details to hide behind a polished interface.

Possible future work includes broader adversarial test cases, clearer analysis provenance, additional report-format edge cases, and improved accessibility.

11. What this project taught me

MailGuard Digest was an opportunity to work on several connected engineering problems rather than just build a user interface.

It brought together:

Python application design with Flask.
Parsing untrusted XML and compressed archives.
Data modeling and validation with Pydantic.
Explicit, deterministic analysis rules.
Local persistence with SQLite.
Automated testing with pytest.
Web security controls.
Optional local language-model integration.

The biggest lesson is architectural: adding AI to a security tool should not weaken the reliability of its underlying analysis.

A model can help communicate complex findings. It should not be allowed to quietly replace measured values with plausible-sounding guesses.

My guiding principle for MailGuard Digest is simple: use deterministic code for facts, and use AI to make those facts easier to understand.

Explore the repository

🔗 GitHub: https://github.com/kiddsaut07/mailguard-digest

If you work with Python, email authentication, defensive security, XML processing, or local AI, I would appreciate feedback on the architecture, threat model, testing strategy, and user experience.

Issues, suggestions, and contributions are welcome.

MailGuard Digest is intended to support DMARC monitoring and investigation. It is not a definitive verdict on whether a message is malicious, and its current local-first design should not be interpreted as a production security guarantee.

Are LLMs Actually Useful for Security Operations? A Benchmark-Driven Analysis

adjia mvoa ndji gabriel monfils — Thu, 29 Oct 2026 15:05:59 +0000

Two numbers, published months apart, describe the same job.

One says a frontier model triages alerts with 98% accuracy. The other says the best of five off-the-shelf reasoning pipelines in a realistic replay misses 40.4% of attacks. Both are honest measurements. They are also, taken at face value, irreconcilable — and the gap between them is the most useful thing I found in this review.

I screened 378 unique records, extracted 32 quantitative comparisons, and graded every one by how fairly it was run. This is an attempt to answer a narrower question: for which security-operations tasks does the measured evidence support using an LLM, and under what conditions does it not?

The short answer: LLMs are genuinely useful as interfaces to structured reasoning and as extractors of unstructured context. They are not competitive as drop-in classifiers, they are dangerous as autonomous actors, and almost every headline number about them is measured under conditions that do not exist in a SOC.

How I evaluated the evidence

I queried the arXiv API across 18 topic clusters on 29 October 2026: 431 entries, 378 unique after de-duplication. I screened titles and abstracts, retrieved 44 full texts, and kept 47 papers. I added nine foundational and standards documents — Arp et al.'s USENIX Security critique (S48), Sommer and Paxson on the closed-world assumption (S49), Axelsson on base rates (S50), NIST AI 600-1 (S51), the OWASP LLM Top 10 (S52), MITRE ATLAS (S53) and ATT&CK (S54), NIST SP 800-61r3 (S55), and the Loghub dataset paper (S56) — plus three practitioner reports from DEV (S57, S58, S59).

Then I graded every comparison:

Class A — same data, same harness, same metric, both arms run by the same authors. Differences are attributable to method.
Class B — shared dataset and metric, but one arm tuned elsewhere or a baseline cited rather than rerun. Directionally usable; magnitudes soft.
Class C — single system measured, or a benchmark adjacent to operations. Capability ceiling only.
Class D — interviews, surveys, standards, ROI models. Constraints, never performance.

I did not pool results across classes, and I did not average anything. Where a number came from a class B comparison, it is reported as a class B number.

One structural caveat up front: 34 of the 59 sources are preprints or under review. The direction of the findings is stable across independent groups. The specific magnitudes are not yet.

First, separate four different tasks

Most public arguments about "LLMs in the SOC" fail because the participants are describing different jobs.

Detection — is this event malicious? High volume, binary, latency-sensitive, evaluated with precision/recall at extreme imbalance.
Triage — does this alert deserve a page, a ticket, or nothing? Low volume, policy-shaped, judged on operational cost.
Investigation — what happened, and what else is connected? Open-ended, multi-source, judged on completeness and correctness of reconstruction.
Response — what do we do, and may the system do it? Consequential, gated, judged on restraint as much as on action.

A result in one does not transfer to another: the 98% headline is task 2 on a balanced sample, the 40% headline is task 3 on realistic telemetry.

A fifth category dominates search results and should be ignored operationally: security knowledge quizzes. CSEBenchmark reports a best score of 85.42% on 11,050 multiple-choice questions (S05); CyberMetric has GPT-4o at 96.25% on its 80-question set against a human expert mean of 72.24% (S07); CS-Eval spans 38.08–87.57 across 19 models (S08). SECURE shows the same pattern on ICS material, with GPT-4 at 87.6% on key-vulnerability questions but GPT-3.5 at 8.4% on unseen ones (S06); CyberSOCEval's malware questions top out at 23–34% (S09). These are recall tests. They show a model has read the documentation, not that it can read your logs.

Detection: a tie, then two inversions

The cleanest experiment stress-tests XGBoost against a LoRA-tuned RoBERTa on network intrusion detection, with a locked 29-feature schema and 50,000 flows per class (S01).

On the same dataset, they tie. XGBoost scores 0.9966 macro F1 on NF-UNSW-NB15-v2, the transformer 0.9938; on NF-CSE-CIC-IDS2018-v2 it is 0.9776 versus 0.9745. The LLM buys nothing here except roughly 5,806× lower throughput.

Change one variable and the ranking inverts. Train on UNSW, test on CIC, and XGBoost holds 0.8116 F1 while RoBERTa collapses to 0.6582, with a false-positive rate of 0.7756 and balanced accuracy of 0.5477. Train on CIC, test on UNSW, and both fail near chance (0.0730 and 0.0599). The transformer is not more general. It is more brittle to distribution shift.

Change the variable again — this time to adversarial evasion, drifting malicious flows toward benign — and the transformer wins: 0.9552 versus 0.6917 F1 at ε=0.15, about 26 points. Both keep false positives under 1% in that regime.

So "which is better" has no answer without naming the stress axis. Same-distribution: tie. Cross-environment: classical. Adversarial perturbation: LLM.

The same pattern shows up in log anomaly detection, where method papers (S21, S22) report gains largely on the same benchmark family. A benchmark spanning HDFS, BGL, Thunderbird and Spirit puts fine-tuned DeBERTa at 95.3%–98.9% F1, Drain plus a random forest at 86.4%–95.1%, and GPT-4 zero-shot at 81.2%–88.3% (S03). Prompted GPT-4 also costs 890 ms per prediction and $8.40 per 1,000 predictions, against 0.3 ms for the parser-plus-classifier. Few-shot prompting closes some of the gap (85.6%–91.7%) but never overtakes a fine-tuned encoder.

Two results sharpen the point.

First, model advantage is not monotone in size or prompting effort. An adaptation study finds hard prompts at 57.56–60.93 F1 while LoRA reaches 98.89 and frozen-layer tuning 99.08 (S04). Quantization is nearly free: 16-, 8- and 4-bit versions score 98.89 / 99.04 / 98.98. But scale gains are dataset-dependent — GPT-2 Medium, Large and XL score 57.04 / 91.90 / 94.77, and on Thunderbird the Medium model collapses to 10.26. "Bigger model, better detection" is not a law.

Second, when an LLM is used as the classifier rather than as the interface to one, it loses badly. In explainable NIDS, zero-shot GPT-4 reaches macro precision/recall of 50.85 / 53.47 and Llama3-8B 48.56 / 40.56, against a cited random forest at 92.17 weighted F1 (S15). On 83,446 phishing emails, a random forest hits 0.9801 accuracy and 0.9789 F1 while the best local quantized LLM, Qwen2.5-32B at 8-bit, reaches 81%, and the 7B and 14B variants sit at 49% and 55% — near chance (S16). The 32B needed about 127% more VRAM than the 14B for two points. On 10,000 balanced phishing URLs, Grok-3 few-shot reaches 0.9399 F1 (S17) — but the balance is again doing much of the work.

The honest summary for detection: use the LLM to build features, parse logs, or explain a detector — not to be the detector. The cost section shows where that division of labour actually pays.

Triage: the base-rate trap

Triage is where the marketing lives: it has a written policy and an obvious cost per alert.

The strongest positive result I found is also the easiest to misread. SIABENCH reports GPT-5 at 98% accuracy on 100 balanced TII-SRC-23 alerts and 97.1% on a 35-alert CIC subset, while GPT-4o-mini drops to 51.4% on that CIC subset (S10). But that CIC subset contains five true positives and thirty false ones: a detector that called everything benign would already score 85.7%. On 68 live, post-cutoff challenges, Claude-3.5-Sonnet scores 60.3% against GPT-4o's 36.8%, and on a separate 20-question post-September-2024 subset GPT-5 scores 85% against GPT-4o's 75%.

Note the construction: 50/50 attack/benign, and 35 alerts. Now do the arithmetic the abstract does not.

Assume 90% sensitivity and 0.1% attack prevalence — ten true incidents in 10,000 alerts. At a 1% false-positive rate, the model raises about 100 false alerts against 9 true positives. Positive predictive value is 8.3%. At a 0.008% FPR it is 91.8%; at 1% prevalence and 1% FPR it is 47.6%. Same model, three completely different operational realities, determined entirely by prevalence and FPR. (This is my arithmetic, not a reproduction of any study; Axelsson formalised the underlying problem in 2000.)

That is why the more informative triage study is the less quotable one. A multi-agent pipeline evaluated on 1,022 benign and 225 attack alerts replayed through a live SIEM — realistic imbalance — reaches F1 0.958, recall 0.969, and a false-negative rate of 3.1%, escalating 18.4% of cases (S11). The five reasoning strategies it compares against land between 0.371 and 0.744 F1, with the best at 40.4% FNR. The winner used 5,075 tokens per case; a weaker pipeline burned 33,438 and did worse.

That is a class B result: the authors built both the benchmark and the winning system. Treat 0.958 as an existence proof — agentic evidence retrieval can work — not as an expected production number.

The practitioner side is more encouraging and less rigorous. A DEV experiment gave six models 200 generated alerts against a written page/ticket/ignore policy, 600 calls each (S57). Two models scored 600/600; the cheap one cost about $0.16 per 1,000 alerts at p50 4.4 s, with one 60-second outlier. But 200 synthetic alerts cannot bound production error, and even their optimistic 95% upper bound is 1.5% — a lot of wrong pages at SOC volume. The deliberately misleading severity-label baseline scored 102/200, which is the useful part: cheap triage is mostly about reading policy, not about reasoning.

Investigation and response: the failure is restraint, not detection

Investigation evidence is thinner.

Mapping 73 NIDS rules to MITRE ATT&CK, the best LLM reaches technique F1 0.62 and tactic F1 0.69; an SVM trained on the LLM's own labels reaches 0.87 and 0.92 (S14). Rare techniques sit at 0.20–0.23. The lesson is not that LLMs map badly; the LLM is a good labeller and a poor reasoner about labels, and the hybrid beats either alone.

On unguided threat hunting over 106 real attack procedures and 86 sub-techniques, the best frontier model reaches 44% mean recall and clears the authors' 50%-per-tactic bar on 7 of 13 tactics, with near-zero recall on Credential Access and Initial Access (S12).

Response is different. Given execution authority, a frontier agent achieved 100% containment in an adversarial incident-response environment — while taking unnecessary actions in 82.5% of episodes, acting by step 4 (S32). A second model contained 62.5% with a 45% false-positive action rate. Detection was largely right; restraint was not — the correct thing to measure, and almost nobody does.

Response-planning research attacks this directly: constraint-checked planning with abstention (S33) and lightweight fine-tuned planners reporting up to 22% shorter recovery times (S34). Report quality is now measured by checklist-based frameworks (S46), which is progress, though report quality is not incident outcome. The most useful response result is architectural. Over 200 real incidents from a financial-sector SOC, a deterministic policy verifier removed 466 non-compliant, approval-gated actions without reducing task recall, across three reruns with a run-level violation rate of 0.8700–0.8750 (S36). The same inline policy text moved two providers in opposite directions. Which means: do not trust the model to hold the policy. Put the policy in code and let the model propose.

Hallucination is an adversarial property, not a prompt-hygiene problem

The framing most teams use — "add grounding, add citations, add a system prompt" — assumes hallucination is a model defect that better prompting fixes. The injection literature says otherwise.

Telemetry is attacker-controlled input. If an LLM reads log fields, an adversary can write to the model's context. Across 48 strategy/defense/task combinations, average injection success was 26.6% with naive prompting and 11.8% under the strongest defense — but the average hides the structure (S28). Direct overrides never worked (0%). Persona hijacks suppressed 68% of malicious logs. Summarisation tasks jumped from 96% with no defense to 38% under constrained output. The attack that works is not "ignore previous instructions"; it is "here is a plausible log line that reframes the incident".

A larger benchmark of 12,847 log entries, 2,569 of them adversarial, reports attack success up to 88.2%, averaging 83.4%, against undefended pipelines; layered defenses cut 90.4% but left 8.4% residual, and a technique called Context Stitching still succeeded 76.4% of the time (S29). A third study of 15 enriched injection samples found success rates of 66.7% to 86.7% across models, with at least one model/string pair fully evading every attack case (S30). A fourth measured verbatim command execution from cloud logs ranging from 0% on Claude Sonnet 4.6 to 86.2% on Llama 3.3 70B (S31).

Two conclusions follow. First, injection resistance is a property of the model and the output constraint, not of the prompt template — capability and safety are different axes, and the most capable model is not automatically the safest. Second, the decisive control is architectural: constrain what the model can emit, and verify every emitted action against a deterministic layer, as S36 does.

Hallucination in analysis has a parallel failure mode. In a phishing explainability study, Llama-2-7B and Llama-3-8B produced explanations with CC-SHAP alignment above 0.95 while classifying phishing correctly only 40% and 30% of the time; Wizard-7B hit 80% accuracy with alignment around 0.12 (S18). Explanation confidence and explanation correctness are uncorrelated. A fluent rationale is not evidence. A survey of vulnerability-assessment hallucination argues the root cause is a gap between deductive security reasoning and autoregressive generation, and that chain-of-thought and tool use narrow but do not close it (S35).

Cost, latency, and the local-deployment question

Three numbers decide feasibility.

Latency per decision. In S03: 0.3 ms (parser + random forest), 12.4 ms (fine-tuned DeBERTa), 156 ms (local LLaMA-3), 340 ms (GPT-3.5), 890 ms (GPT-4). At 10,000 alerts a day, 890 ms is tolerable — but not inside a synchronous detection path, and it costs nearly 3,000× the classifier it replaces.

Cost per unit. GPT-4 at $8.40 per 1,000 log predictions (S03) is not catastrophic for triage and is absurd for detection at flow volume. A distilled 0.5B detector reports 0.3–0.5 s per session at under $50/day (S19). The DEV triage experiment landed at about $0.16 per 1,000 alerts for a cheap model (S57).

Return on investment. A Monte Carlo model of SOC adoption finds LLM deployment unprofitable for most configurations at current prices (S37). It is profitable only for high-accuracy API deployments with low capital cost and low uncertainty. Full autonomy raises the supportable capital expenditure from about 10% to 80–90% of baseline; removing the rework penalty alone only gets to about 20%. Self-hosted GPU costs outweigh savings at typical SOC volumes. That is the strongest argument against "just run it locally": the hardware math is worse than the API bill, unless your volume is low or your data cannot leave.

Token economics reinforce the point. In 200 vulnerability-work agent traces, reasoning and trigger design consumed 60.4% of tokens and 68.7% of attributed cost, and only 24.4% of matched comparisons preserved success at lower cost (S38). Prompt engineering cannot fix a reasoning-bound workload. Nor is offensive capability a proxy for defensive reliability: on CyberGym's 1,507 real vulnerabilities, top agent configurations solve about 20%, and GPT-5 moves from 7.7% to 22.0% with extended thinking (S44).

There is one genuinely optimistic systems result. Small models under 10B, used for log parsing with a targeted parser, improved average accuracy by 12.5%, grouping F1 by 23.8% and template F1 by 50.6% over a prior small-model parser, with lower latency than semantic baselines (S41). Cross-model parsing studies find CodeLlama-7B extracting 10% more templates than GPT-3.5 (S42), while a review of 29 parsing methods finds reproducibility problems severe enough that re-benchmarking is often impossible (S43). Long-context local inference remains memory-bound: one system needs about 47 GB, of which 33.5 GB is KV cache, for long-context work (S39). Split edge-cloud inference cuts per-token latency by up to 46.1%, though that is a generic systems result rather than a security one (S40). Local is viable for narrow, structured sub-tasks. It is not viable as a general SOC brain on commodity hardware.

What the evidence does not prove

The holes are worth stating plainly.

No end-to-end field trial measures MTTD or MTTR with LLMs against a controlled baseline in a production SOC.
No independent replication of the strongest positive result (S11). It is a class B existence proof, not a confirmed effect.
No cost-per-resolved-incident study exists; S37 is a model, not a measurement.
No benchmark spans collect → triage → investigate → contain with realistic prevalence at every stage; newer resources such as SecLog-Bench add multi-stage, multi-source labels but still simulate the enterprise (S45).
No evidence that benchmark accuracy transfers across SIEMs, schemas or organisations. S02 and S13 suggest the opposite.
Benchmark scores are unstable. Across 8 cybersecurity benchmarks and 10 models, a single pipeline choice shifted scores by more than 80 percentage points, and 9 of 10 models moved at least 3 ranks on some benchmark (S13). Any leaderboard you see is one pipeline's opinion.
Label leakage is unresolved. HDFS and BGL, used throughout the log-anomaly literature, have known artefacts. Several 98–99% F1 results are probably optimistic, which is why I graded them B despite internal consistency.

And a practical warning: 93% of real analyst queries in one 10-month, 45-analyst study mapped to standard NICE tasks, but analysts used LLMs as aides for sensemaking, not as decision-makers (S25). Twenty interviews (S24), a 25-interview study (S26), a Reddit analysis (S27) and a survey (S23) converge on the same picture: about 55% exploring GenAI, roughly 10% in production, and hallucination plus verification burden as the top complaints. The gap between demo and deployment is not a skills gap. It is an evidence gap.

What I would actually do

Do not put an LLM in the detection path. Use it to parse, normalise, enrich, and explain. Keep the classifier classical or a fine-tuned encoder. S01, S03 and S15 all point the same way.
Always evaluate at your real prevalence. Balanced test sets are for debugging, not for decisions. Report precision, FPR and MCC alongside accuracy, and compute the expected false-alert load per day. If a vendor shows you 98%, ask for the confusion matrix at 0.1% prevalence.
Put policy in code, not in the prompt. S36's deterministic verifier removed 466 bad actions with no recall loss; the same prompt text moved two providers in opposite directions.
Constrain outputs and verify actions. Injection success falls when the model can only emit a typed verdict or a whitelisted action, and rises when it can write prose or commands (S28, S31).
Measure restraint, not just detection. Track unnecessary actions and escalation rate. A system that contains 100% of incidents while acting unnecessarily 82.5% of the time is not safe; it is fast.
Copy the architecture, not the demo. Practitioner builds converge on deterministic evidence collection, a constrained inference layer, and gated containment (S58, S59).
Budget for rework. The ROI model is blunt: most deployments lose money unless accuracy is high, the API is cheap, and the verification loop is short (S37). The hidden cost is the analyst time spent checking the model, which is exactly the cost vendors exclude.

🌐 Community Wisdom

🧾 Cloudflare's Agentic Security Ops: Separating Evidence from Inference @mech_app_ai — The architecture that holds up is deterministic evidence collection first, inference constrained to cite it, and a validation agent before any action. 🔗 Read Full Discussion on DEV
💸 Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage? @devopsdaily — A cheap model can match a written on-call policy on 200 synthetic alerts for about $0.16 per 1,000 calls, but the authors are honest that this cannot bound production error. 🔗 Read Full Discussion on DEV
🔗 Stop Piping Raw Alerts into an LLM: Building a SOC Triage Agent That Correlates First and Escalates Last @adam_lewandowski_59674796 — Single-alert triage cannot see a campaign, so the practical pattern is normalise, correlate, tier the escalation, and gate containment behind deterministic checks. 🔗 Read Full Discussion on DEV

The verdict

LLMs are useful for security operations in a narrower band than the marketing claims and a wider band than the sceptics allow.

They win where the task is unstructured context reduction: parsing heterogeneous logs, mapping rules to tactics, summarising an investigation, drafting a response plan that a human then verifies. They lose where the task is high-volume calibrated classification and where the system is allowed to act without a deterministic gate. They are least trustworthy exactly where the stakes are highest, because the failure mode is not a wrong answer — it is a confident, fluent, well-formatted wrong answer that survives review.

The single most important number in this review is not 98% or 40%. It is 82.5% — the share of episodes where a containment-capable agent acted unnecessarily while scoring 100% on containment (S32). If you take one thing from the benchmarks, take this: the metric you optimise will be the failure you ship.

References

S01. Atif, M. E., & Ali, M. H. (2026). A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection. arXiv:2609.13511.

S02. HIDBench: Benchmarking LLMs for Host Intrusion Detection (2026). arXiv:2605.21773.

S03. LLM-Enhanced Log Anomaly Detection: A Comparative Benchmark (2026). arXiv:2604.12218.

S04. Log Anomaly Detection with LLMs: Adaptation, Scale and Quantization (2026). arXiv:2609.31371.

S05. CSEBenchmark: A Comprehensive Benchmark for Cybersecurity Knowledge (2025). arXiv:2504.11783.

S06. SECURE: Benchmarking LLMs for ICS Security (2024). arXiv:2405.20441.

S07. Tihanyi, N., et al. (2024). CyberMetric: A Benchmark Dataset for Evaluating LLMs' Cybersecurity Knowledge. arXiv:2402.07688.

S08. CS-Eval: A Comprehensive Benchmark for Cybersecurity (2024). arXiv:2411.16239.

S09. CyberSOCEval: Benchmarking LLMs for Malware Analysis and Threat Intelligence Reasoning (2025). arXiv:2509.20166.

S10. SIABENCH / Before You Hand Over the Wheel (2026). arXiv:2603.06422.

S11. Investigation Failures in LLM Alert Triage / AIDA (2026). arXiv:2610.10608.

S12. Cyber Defense Benchmark: Unguided Threat Hunting in Raw Windows Event Logs (2026). arXiv:2604.19533.

S13. Benchmark Scores Are Pipeline-Dependent (2026). arXiv:2609.08765.

S14. Mapping NIDS Rules to MITRE ATT&CK with LLMs (2024). arXiv:2412.10978.

S15. Explainable Network Intrusion Detection with LLMs (2024). IEEE BDCAT. doi:10.1109/BDCAT63179.2024.00021.

S16. Quantized Local LLMs vs Classical ML for Phishing Email Detection (2025). IEEE SNPD.

S17. Phishing URL Classification with Frontier LLMs (2025). NeurIPS LAW Workshop. arXiv:2602.02641.

S18. Phishing Detection and Explanation Faithfulness (2025). arXiv:2506.13746.

S19. AMAN / LogAtlas: Distilled Anomaly Detection for Heterogeneous Logs (2026). arXiv:2602.06777.

S20. Parameter-Efficient Log Anomaly Detection (2025). PAKDD. arXiv:2503.08045.

S21. LogLLM (2024). arXiv:2411.08561.

S22. LogTinyLLM (2025). arXiv:2507.11071.

S23. LLMs in Security Operations: A Survey (2026). J. Electrical and Computer Engineering. arXiv:2509.10858.

S24. Beyond the Hype: Practitioner Perspectives on LLMs in SOCs (2026). arXiv:2608.17154.

S25. LLMs in the SOC: A Longitudinal Study of 3,090 Analyst Queries (2025). arXiv:2508.18947.

S26. From Chasing Ghosts to Missed Attacks (2026). arXiv:2608.00672.

S27. Practitioner LLM Use in Cybersecurity: A Reddit Analysis (2026). SOUPS. arXiv:2604.09998.

S28. Poisoning the Watchtower: Prompt Injection Against SOC Copilots (2026). arXiv:2605.24421.

S29. Context Contamination / LogInject-1.0 (2026). arXiv:2607.14493.

S30. Just Testing, Move Along: Evading LLM Log Interpretation (2026). arXiv:2607.24174.

S31. LogJack: Indirect Prompt Injection Through Cloud Logs (2026). arXiv:2604.15368.

S32. OpenSec: Measuring Restraint in Incident-Response Agents (2026). arXiv:2601.21083.

S33. Hallucination-Resistant Security Planning (2026). IEEE/IFIP NOMS. arXiv:2602.05279.

S34. Lightweight Incident-Response Planning (2025). arXiv:2508.05188.

S35. AI Slop in Vulnerability Assessment (2026). SiMLA. arXiv:2608.25667.

S36. SOCpilot: Policy Compliance of LLM-Drafted Incident Response Plans (2026). arXiv:2605.05501.

S37. From Hype to ROI: The Economics of LLM Adoption in SOCs (2026). arXiv:2610.33352.

S38. Where Do the Tokens Go? Cost Attribution for Security Agents (2026). arXiv:2610.11602.

S39. LeoAM: Long-Context Inference on Commodity GPUs (2025). arXiv:2506.20187.

S40. Privacy-Aware Edge-Cloud Collaborative Inference (2026). arXiv:2607.13093.

S41. Small Is Beautiful: EFParser for Log Parsing (2026). FSE. arXiv:2601.22590.

S42. A Comparative Study of LLMs for Log Parsing (2024). ESEM. arXiv:2409.02474.

S43. System Log Parsing with LLMs: A Review (2025). arXiv:2504.04877.

S44. CyberGym: Evaluating AI Agents on Real-World Vulnerabilities (2026). ICLR. arXiv:2506.02548.

S45. SecLog-Bench (2026). arXiv:2610.31000.

S46. MESSALA: Evaluating SOC Report Quality (2026). ESORICS. arXiv:2601.03013.

S48. Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and Don'ts of Machine Learning in Computer Security. 31st USENIX Security Symposium. https://www.usenix.org/conference/usenixsecurity22/presentation/arp

S49. Sommer, R., & Paxson, V. (2010). Outside the Closed World: On Using Machine Learning for Network Intrusion Detection. IEEE S&P. doi:10.1109/SP.2010.25

S50. Axelsson, S. (2000). The Base-Rate Fallacy and the Difficulty of Intrusion Detection. ACM TISSEC. doi:10.1145/357830.357849

S51. NIST (2024). AI Risk Management Framework: Generative AI Profile (NIST AI 600-1). doi:10.6028/NIST.AI.600-1

S52. OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications. https://genai.owasp.org/llm-top-10/

S53. MITRE. ATLAS. https://atlas.mitre.org/

S54. MITRE. ATT&CK. https://attack.mitre.org/

S55. NIST (2025). SP 800-61 Rev. 3: Incident Response Recommendations. https://csrc.nist.gov/pubs/sp/800/61/r3/final

S56. Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2020). Loghub: A Large Collection of System Log Datasets. arXiv:2008.06448.

S57. DevOps Daily (2026). Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage? https://dev.to/devopsdaily/page-ticket-or-ignore-is-the-cheap-model-good-enough-for-alert-triage-4iak

S58. mech.app (2026). Cloudflare's Agentic Security Ops: Separating Evidence from Inference. https://dev.to/mech_app_ai/cloudflares-agentic-security-ops-separating-evidence-from-inference-lm2

S59. Lewandowski, A. (2026). Stop Piping Raw Alerts into an LLM. https://dev.to/adam_lewandowski_59674796/stop-piping-raw-alerts-into-an-llm-building-a-soc-triage-agent-that-correlates-first-and-escalates-20pp

Method note: 378 unique records screened from 18 arXiv queries on 29 Oct 2026; 44 full texts extracted; 47 papers plus 9 standards/datasets and 3 practitioner reports included. Quantitative claims are traced in sources.csv and benchmark_matrix.csv; comparisons are graded A–D by methodological comparability. This review reports published measurements and its own arithmetic only — no experiments were run for it. Drafted with AI assistance and checked against the primary sources.

One YAML rule, any SIEM: introducing DetectForge

adjia mvoa ndji gabriel monfils — Thu, 29 Oct 2026 14:07:37 +0000

Every detection team I have talked to ends up in the same place: the same detection logic written four times. Once for Splunk, once for Elastic, once for Sentinel, once for the in-house thing nobody wants to touch. The rule that says "whoami.exe just ran" is identical everywhere. The query is not.

DetectForge is my attempt at the boring-but-important layer underneath that problem. It is a Python toolkit that parses portable YAML detection rules into typed objects, validates them, runs them through a processing pipeline, and hands them to a backend that emits the query for the platform you actually run.

Repo: https://github.com/kiddsaut07/detectforge — LGPL-2.1, Python 3.11+.

What's actually in the box

A typed rule model: rules, log sources, detections, detection items, tags, levels, statuses, MITRE ATT&CK references.
Collections with global / repeat / reset actions, load_ruleset, and merging, so shared metadata lives in one place.
Correlation rules: event_count, value_count, temporal, temporal_ordered, value_sum, value_avg, value_percentile, value_median, with dependency-aware ordering.
Filters that rewrite matching rules by appending a negated condition before conversion.
Processing pipelines: field mapping, prefix/suffix, value and string transformations, placeholders, custom state, external sources, per-rule conditions.
Conversion backends built on a Backend / TextQueryBackend framework with per-format finalisers and error collection.
Validation with pluggable validators, exclusions and configuration.
Plugin discovery for backends, pipelines and validators.

Quick start, output pasted from a real run

pip install "git+https://github.com/kiddsaut07/detectforge.git"

from detectforge.backends.test import TextQueryTestBackend
from detectforge.collection import DetectForgeCollection

rules = DetectForgeCollection.from_yaml(
    """
title: Suspicious Whoami Execution
id: 11111111-1111-1111-1111-111111111111
status: test
description: Detects execution of whoami
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\\whoami.exe'
    condition: selection
level: low
"""
)

print(TextQueryTestBackend().convert(rules))

['Image endswith "\\whoami.exe"']

TextQueryTestBackend is a reference backend that ships with the project (it exists so the test suite has a target). Real conversions come from plugin backends.

Collections and correlations

The interesting part starts when rules reference each other. This one document defines shared log source metadata, repeats it across rules, and defines a correlation on top:

action: global
logsource:
    product: windows
---
action: repeat
title: "Failed Logon"
name: failed-logon
id: 22222222-2222-2222-2222-222222222222
status: test
detection:
    selection:
        EventID: 4625
    condition: selection
---
title: Multiple Failed Logons
name: corr-multiple-failed-logons
status: test
correlation:
    type: event_count
    rules:
        - failed-logon
    group-by:
        - TargetUserName
    timespan: 5m
    condition:
        gte: 10

References are resolved by id or name, and the collection orders rules so dependencies are converted first. Output from the same run:

EventID=4625
| aggregate window=5min count() as event_count by TargetUserName
| where event_count >= 10

The part I care about most: treating rules as untrusted input

Detection content is usually community content. You download a ruleset, you point a converter at it, and suddenly you are parsing ten thousand YAML files written by strangers into a query your SIEM will execute. A rule parser is an attack surface, so DetectForge is defensive by default:

Duplicate keys are rejected, not silently merged. Most YAML loaders keep the last value, which is exactly how you get a rule that does not do what it says it does:

YAMLError -> Duplicate key 'level'

Alias expansion is capped. YAML anchors are shared references when parsed, but rule processing walks the structure as a tree, so a chain of aliases expands exponentially. MAX_ALIAS_EXPANSION stops the billion-laughs pattern dead instead of exhausting memory.

RE2 is the default regex engine. SafePolicy compiles with Google's RE2 — no backtracking, so a pathological pattern in a third-party rule cannot pin a CPU. TrustedPolicy (plain Python re) is opt-in for the cases RE2 cannot express, rather than the other way around.

Template variables and external data sources are opt-in. Placeholders do not read files or pull data from disk unless you explicitly allow it.

There is also a validation framework, which is what catches the boring problems before conversion:

from detectforge.validation import DetectForgeValidator
from detectforge.validators.core import validators

validator = DetectForgeValidator(list(validators.values()))
for issue in validator.validate_rules(iter(rules.rules)):
    print(issue)

issue=IdentifierExistenceIssue severity=medium description="Rule has no identifier (UUID)" rules=[None]

How the pipeline fits together

Rule YAML → typed objects → collection (globals applied, references resolved) → processing pipeline (field mapping, transformations, per-rule conditions) → backend → query text.

The backend layer is where most of the work lives: TextQueryBackend handles the expression tree, operator mapping, escaping, wildcards and per-format finalisers. Adding a platform means subclassing it and filling in the pieces your query language needs, not reimplementing rule handling.

Where it stands

Version 0.1.0. 18k lines of library code, a 1,500-test suite, Sphinx docs and Read the Docs config, and a single reference backend — the production backends are meant to arrive as plugins. It is early, and the API is still allowed to move.

What I would genuinely like feedback on:

the shape of the Backend / TextQueryBackend contract for new platforms,
the correlation types that are missing (what do you actually write in production?),
the plugin registry and its compatibility checks.

If that is useful to you, a star on the repo helps other detection engineers find it: https://github.com/kiddsaut07/detectforge. Issues and rule-format edge cases are even more welcome than stars.

PALABRE: An AI-Powered Cultural Treasure Hunt That Gets You Off Your Phone

adjia mvoa ndji gabriel monfils — Fri, 09 Oct 2026 12:46:38 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 1 — Touch Grass.

PALABRE: What If AI's Job Was to Get You Off Your Phone?

We are getting very good at building AI that keeps us on screens.

More answers.
More recommendations.
More generated content.
More reasons to keep scrolling.

So when I saw this week's Touch Grass challenge, I wanted to try something deliberately backwards:

What if AI's job was not to keep you online, but to send you outside?

That question became PALABRE.

Your phone knows the internet. Your neighborhood knows the stories.

PALABRE is a real-world cultural treasure hunt powered by AI.

It gives you a mission, sends you into your neighborhood to talk to a real person, lets you record what you hear, analyzes that testimony, preserves the resulting knowledge, and generates the next mission from the story you just collected.

The screen is only the beginning.

The real experience happens outside it.

What I Built

PALABRE is a real-world cultural treasure hunt powered by AI.

Instead of asking the user to spend more time talking to an AI, PALABRE gives them a mission that must be completed in the real world.

The loop is simple:

MISSION
   ↓
GO OUTSIDE
   ↓
TALK TO A REAL PERSON
   ↓
RECORD THE STORY
   ↓
AI ANALYSIS
   ↓
PRESERVE THE KNOWLEDGE
   ↓
NEXT MISSION
   ↓
GO OUTSIDE AGAIN

The project starts with the neighborhoods of Yaoundé, Cameroon, where much knowledge is still passed through conversation rather than databases, websites, or formal archives.

A proverb.

A craft lesson.

A neighborhood memory.

A piece of advice from an elder.

A story about work, family, migration, food, language or everyday life.

PALABRE is designed to turn those conversations into a growing living oral archive.

And there is one important twist:

the next mission is generated from the story that was just collected.

So the experience doesn't simply repeat the same AI prompt.

It evolves.

The Problem I Wanted to Solve

A lot of our digital lives revolve around consuming information that already exists online.

But some of the most valuable knowledge around us is not online at all.

It lives in conversations.

A proverb repeated by an elder.

A lesson learned through years of work.

A neighborhood memory.

A craft technique.

A family story.

An expression that only makes sense when someone explains where it came from.

That knowledge can disappear without ever becoming searchable.

And there is an interesting contradiction here:

AI can generate enormous amounts of new information, while millions of real human stories remain uncaptured.

PALABRE explores a different use of AI:

not generating another story, but helping preserve a story that already exists.

The PALABRE Loop

The entire product is built around one physical loop:

MISSION
   ↓
GO OUTSIDE
   ↓
TALK TO A REAL PERSON
   ↓
RECORD THE STORY
   ↓
AI ANALYSIS
   ↓
CULTURAL MEMORY
   ↓
NEXT MISSION
   ↓
GO OUTSIDE AGAIN

The important part is the last step.

PALABRE does not simply analyze a conversation and end the experience.

The next mission is generated from the testimony that was just collected.

That means the quest can evolve through actual human encounters.

A conversation creates a new question.

A new question creates another conversation.

The archive grows through people, not through endless AI chat.

What the Explorer Actually Does

A PALABRE mission might ask the explorer to return to their neighborhood and ask a merchant, neighbor, craftsperson or elder a meaningful question.

For example:

What is a piece of advice you were given by your elders that still guides your work today?

The application then gets out of the way.

You leave the screen.

You find someone.

You ask.

You listen.

You record.

You come back.

That is intentional product design.

For PALABRE, a successful session should end with less screen time, not more.

Building a Real Voice-Recording Experience

The audio layer is implemented directly in the browser using the Web Audio API and MediaRecorder.

I wanted recording to feel like a proper part of the product rather than a file-upload form.

PALABRE handles:

microphone permissions
supported recording formats
real-time audio-level feedback
recording duration limits
invalid or empty recordings
file-size validation
re-recording
server-side upload validation

The current maximum recording duration is three minutes, with a 25 MB upload limit.

The flow is:

Browser microphone
      ↓
Audio recording
      ↓
FormData upload
      ↓
Express API
      ↓
Speech-to-text / AI

AI That Listens Instead of Talking

This is where PALABRE becomes more than a recording app.

The AI pipeline separates speech recognition from cultural analysis.

Conceptually:

AUDIO
  ↓
SPEECH-TO-TEXT
  ↓
TRANSCRIPT
  ↓
AI ORCHESTRATOR
  ↓
STRUCTURED CULTURAL ANALYSIS

Instead of asking an LLM for an arbitrary paragraph, PALABRE expects a structured result:

{
  "transcript": "...",
  "theme": "...",
  "extracted": "...",
  "summary": "...",
  "culturalSignificance": "...",
  "nextMission": "..."
}

That output becomes the bridge between the human conversation and the next physical action.

I Was Careful About Cultural Hallucinations

This part mattered more than I expected.

If an AI is analyzing a cultural testimony, inventing a plausible-sounding proverb is not a harmless mistake.

It becomes misinformation about someone's culture.

So PALABRE's analysis instructions explicitly tell the model to:

stay grounded in the testimony
preserve what was actually said
avoid inventing names, dates or traditions
distinguish direct information from inference
avoid unsupported anthropological claims
mark unclear speech instead of pretending to understand it

The objective is not to make the result sound impressive.

The objective is to make it faithful.

That distinction matters when dealing with oral history.

From One Human Voice to a Preserved Memory

The result turns the recording into a structured cultural artifact.

PALABRE can surface:

What was shared

The core content of the testimony.

The AI heard

The main insight extracted from the story.

Why it matters

A cautious explanation of its significance for community memory.

Preserved voice recording

The original recording can remain attached to the story.

That last part is important.

An oral archive containing only transcripts loses something fundamental.

You lose the voice.

The rhythm.

The pauses.

The personality.

PALABRE therefore treats the original recording as part of the memory itself.

The Next Mission Is Generated From the Story

This is the part I care about most.

Imagine that someone tells you how an older relative taught them a craft.

PALABRE can use that testimony to create a new mission that asks you to find another person related to that theme and ask another meaningful question.

The product therefore forms a chain:

HUMAN STORY
     ↓
AI UNDERSTANDING
     ↓
CULTURAL MEMORY
     ↓
NEXT PHYSICAL QUESTION
     ↓
ANOTHER HUMAN STORY

The AI is not the destination.

It is the mechanism that keeps the human experience moving.

The next mission comes from the story you just heard.

That's what makes PALABRE different from simply putting an LLM behind a form.

Field Proof: Did the Digital Mission Actually Lead Outside?

I wanted the project to take the challenge literally.

So PALABRE includes Field Proof.

At the beginning of a mission, the app records the starting time and can request a best-effort location.

At submission, it can calculate an approximate straight-line distance using the Haversine formula.

Only derived values are sent to the server:

time outside
distance

Raw coordinates are not stored.

And the mechanism is deliberately non-blocking.

No location?

The mission still works.

No GPS fix?

The mission still works.

The feature is there to acknowledge the physical world, not to turn PALABRE into a tracking application.

Under the Hood: An AI Orchestrator Instead of a Single Vendor

I also did not want the application permanently tied to one AI provider.

So the server contains a provider abstraction and an AI Orchestrator.

The current architecture supports:

Ollama
Qwen
DeepSeek
GLM
Gemini

The first four provide paths toward open-weight or local inference, while Gemini can remain available as a fallback depending on deployment configuration.

The application can inspect provider configuration and health, route requests according to capabilities, and fall back between configured providers.

The high-level architecture is:

                         ┌── Ollama
                         ├── Qwen
VOICE → STT → TRANSCRIPT → AI ORCHESTRATOR ──┼── DeepSeek
                         ├── GLM
                         └── Gemini
                                  ↓
                       STRUCTURED ANALYSIS
                                  ↓
                         CULTURAL MEMORY
                                  ↓
                           NEXT MISSION

This means the rest of PALABRE does not need to know which model is doing the reasoning.

The intelligence layer can evolve independently.

Why Open Innovation Matters Here

This challenge could have been solved with a simple:

Browser → one closed AI API → response

That would work.

But it would also make the intelligence layer much harder to replace.

PALABRE instead keeps the AI layer modular.

That creates room to:

swap models
test different providers
run local inference
reduce vendor lock-in
choose different inference strategies
move more processing closer to the user over time

This matters especially for a project starting in Cameroon.

A product designed for real communities cannot assume unlimited bandwidth, expensive hardware, permanent connectivity, or identical access to cloud services.

The longer-term architecture I want for PALABRE looks like:

AUDIO
  ↓
LOCAL SPEECH-TO-TEXT
  ↓
LOCAL OPEN-WEIGHT MODEL
  ↓
CULTURAL ANALYSIS
  ↓
ARCHIVE

That is a direction rather than a claim that every deployed component is already fully local today.

The important thing is that the architecture does not make that future impossible.

The Technology Behind the Experience

The user sees a relatively simple product.

Underneath, there is quite a bit going on.

Frontend

React 19
TypeScript
Vite
Tailwind CSS
Web Audio API
MediaRecorder
browser geolocation

Backend

Node.js
Express
multipart audio handling with Multer
rate limiting
server-side AI orchestration
structured output validation

Data

persistent story metadata
optional Firestore synchronization
original voice recordings when available
public collective oral archive

AI Layer

provider abstraction
capability detection
provider health checks
automatic fallback
speech-to-text abstraction
strict JSON validation

The result is a fairly simple user experience sitting on top of an intentionally decoupled architecture.

Testing the Failure Paths

A demo that works once is not enough.

I added tests around:

provider configuration
AI routing
fallback behavior
JSON sanitization
provider health
capability detection
speech-to-text routing
health monitoring
debug tooling

The repository currently reports:

48 tests passing.

The goal was to make failure predictable rather than invisible.

For example, if a configured provider becomes unavailable, the orchestrator can move to the next available provider instead of forcing the entire explorer experience to stop immediately.

Design Was Part of the Experiment

I deliberately avoided the usual "AI startup" visual language.

No:

glassmorphism
purple-blue gradients
floating AI brains
giant dashboards
excessive cards
futuristic holograms

PALABRE uses a much more editorial direction:

#090909  → background
#141414  → surfaces
#242220  → borders
#FF6B35  → signal / accent
#F2EEE8  → primary text
#77736D  → muted text

The typography combines:

Syne for display headlines
Fraunces for editorial and oral-story moments
Plus Jakarta Sans for interface text
a monospace style for technical metadata

The design is intentionally restrained.

The irony is important:

an application about leaving the screen should not try to maximize your attention on the screen.

Built in Yaoundé, Designed to Travel

PALABRE starts in Yaoundé, Cameroon.

That starting point matters to me because oral knowledge is not some abstract historical concept.

It exists in ordinary conversations.

In neighborhoods.

At workshops.

In markets.

In families.

On the street.

The initial use case is local, but the model is broader.

The same interaction could preserve community memory in another city, another country, or another language.

The system is designed around a universal action:

go meet someone and listen.

What I Learned

The biggest lesson was that the hard part of AI product development is often not the model call.

It is everything surrounding the model.

How do you capture audio reliably?

What happens when the microphone fails?

What happens when the model returns malformed JSON?

What happens when an AI provider is unavailable?

How do you keep the AI grounded in what a person actually said?

How do you prevent a cultural archive from turning hallucinations into "facts"?

And perhaps the most important question:

What should the human do after the AI finishes?

For PALABRE, the answer is intentionally not:

"Keep chatting."

It is:

"Go find someone else to talk to."

What's Next?

PALABRE is an MVP, not a finished cultural infrastructure.

The next iterations I want to explore are:

More local inference

A complete local voice → open-weight model pipeline for environments with limited connectivity.

Better multilingual support

Especially for the linguistic reality of Cameroon, including French, English, Pidgin, Camfranglais and eventually more local languages where reliable speech data and models permit it.

A stronger oral archive

Better browsing and discovery of collected stories while preserving the context in which they were recorded.

Better field interactions

More meaningful ways to connect missions to real locations and communities without turning the product into a surveillance system.

The larger idea is simple:

use open AI to strengthen human memory instead of replacing human interaction.

Why PALABRE Fits "Touch Grass"

The challenge asks us to build with open-source AI/open-weight AI in a way that gets people off the screen and into the world.

PALABRE takes that literally.

The best version of the product is the one where:

SCREEN TIME
     ↓
MISSION
     ↓
SCREEN OFF
     ↓
REAL CONVERSATION
     ↓
VOICE
     ↓
AI
     ↓
MEMORY
     ↓
NEXT MISSION
     ↓
SCREEN OFF AGAIN

The technology is important.

But the technology is not the point.

The point is the person standing outside the application.

Try PALABRE

🌍 Live demo: palabre.onrender.com

💻 GitHub repository: github.com/kiddsaut07/palabre

PALABRE is open source under the Apache-2.0 license.

Try the mission.

Then do the part the application cannot do for you:

put your phone down.

Go talk to someone.

Listen.

Come back.

Bring a story with you.

Final Thought

We are entering an era where AI can generate almost anything on demand.

Maybe that makes the things AI cannot generate for us even more valuable.

A person's memory.

A voice.

A neighborhood story.

A lesson passed from one human being to another.

PALABRE is my small experiment in using AI not to replace that interaction, but to send us back toward it.

Your phone knows the internet.
Your neighborhood knows the stories.

Links

🌍 Live Demo: https://palabre.onrender.com

💻 GitHub: https://github.com/kiddsaut07/palabre

AI Assistance Disclosure

I used AI tools during development and while drafting and refining this article. I reviewed the final version and kept the technical claims grounded in the actual PALABRE repository and implementation.

devchallenge #hf26challenge #Hacktoberfest #OpenSource #AI #TouchGrass #Render