I Built an AI Log Investigator That Has to Prove Its Claims

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A production server is throwing errors.

You have thousands of log lines. Half of them look identical. The other half contain timestamps, request IDs, stack traces, and enough noise to make finding the actual problem feel like archaeology.

So you paste everything into an AI chatbot.

It gives you a beautifully formatted explanation of what went wrong.

There's just one problem.

How do you know it didn't make the explanation up?

I built LogSherlock because I wanted a different answer to that question.

LogSherlock is an AI-powered log investigator. Give it application logs, and it parses them, groups recurring errors, ranks their severity, and uses an AI model to investigate the most important groups.

But the model has a constraint: every finding must be backed by actual log lines.

A plausible explanation isn't enough. A fabricated citation isn't enough. A confident answer isn't enough.

If the evidence doesn't support the claim, LogSherlock should reject it.

Try it: LogSherlock - Live Analyzer

The idea is to give developers a faster way to move from a wall of logs to a small number of things worth investigating.

I'm building it for freelance developers, indie hackers, and small engineering teams that need practical debugging tools without the complexity of an enterprise observability platform.

The commercial hypothesis is straightforward: developers might pay for a tool that saves them time during debugging, especially when they don't have a dedicated SRE team.

I haven't validated that with paying customers yet. That's part of what makes this a side-hustle experiment rather than just another weekend project.

Demo

Live application: LogSherlock - Live Analyzer

The easiest way to understand the project is to follow an investigation from beginning to end.

1. Paste application logs or upload a .log, .txt, or .jsonl file.
2. Let LogSherlock parse the entries and identify recurring error patterns.
3. Inspect the grouped errors and their deterministic severity rankings.
4. Follow the investigation's progress through the processing steps.
5. Open findings and inspect the log-line evidence behind them.
6. Review the report, including any processing limitations or incomplete AI analysis.

There is also a bundled sample log you can use to explore the workflow without preparing your own data.

The sample deliberately includes several things that a real log-processing tool needs to handle: a bearer token, a database password, an email address, and text attempting to manipulate the AI's instructions.

It isn't just a happy-path demo. It gives you a way to examine the defensive parts of the implementation, too.

Code

GitHub repository: Harsh-0986/LogSherlock

LogSherlock is open source under the MIT License.

The repository includes the application, backend API, log-processing engine, LangGraph workflow, provider abstraction, persistence layer, sample logs, tests, and technical documentation.

I wanted the important decisions to be inspectable rather than hidden behind a hosted service. You can examine how the application groups errors, computes severity, selects evidence, validates model output, and handles failure.

How I Built It

The first architectural decision was also the most important one.

I didn't want an LLM to be responsible for everything.

It would have been easy to send the entire log file to a model and ask it to summarize the errors. But that approach creates several problems at once: repeated lines consume context, important details can disappear in long inputs, costs become harder to control, and the model can produce an explanation that sounds right without being supported by the logs.

Instead, I separated deterministic processing from AI investigation.

The stack is Python 3.12+, FastAPI, LangGraph, SQLAlchemy, Alembic, SQLite, Next.js, React, and TypeScript. The project uses uv for Python dependency management.

The application is divided into stages, each with a specific responsibility.

1. Parse the logs before asking the model

LogSherlock detects the input format and uses dedicated parsers for Python logging, Node.js and Bun errors, .NET exceptions, and JSONL events. Unrecognized formats fall back to a generic parser, and the report identifies that fallback rather than silently pretending the input was understood perfectly.

Input is bounded, too. The default limits are 5 MB per upload, 20,000 lines, and 20,000 characters per line. Oversized inputs are rejected explicitly instead of being silently chopped up.

Parsing also performs best-effort redaction before the content is stored or sent to a model. Supported patterns include bearer tokens, JWTs, API-key-like values, connection strings, password assignments, session cookies, email addresses, and IP addresses.

This is deliberately not described as perfect secret detection. Unusual formats and secrets split across lines can evade pattern matching.

2. Normalize the noise without deleting the clues

Imagine these three errors:

2026-10-15 10:01:02 ERROR Request 8172 failed after 240ms: connection refused
2026-10-15 10:01:04 ERROR Request 8173 failed after 251ms: connection refused
2026-10-15 10:01:06 ERROR Request 8174 failed after 239ms: connection refused

The timestamps, request IDs, and durations differ. The underlying error does not.

LogSherlock normalizes variable values, fingerprints the resulting patterns, and clusters matching entries into an error group. Each group retains an occurrence count and a representative sample.

But normalization must be careful.

HTTP status codes, SQLSTATE values, database error codes, HRESULTs, and exception types can be the very details that distinguish one failure from another. Removing too much information can make unrelated errors look identical.

The goal is to remove noise while preserving what helps diagnose the problem.

3. Rank errors without asking the AI what matters

Severity is deterministic and rule-based.

The backend computes a severity and records the reasons behind it. Those reasons can be inspected in the interface.

The AI cannot override that ranking.

This separation matters because an LLM-generated severity score would make the report less reproducible. Two similar investigations could receive different rankings simply because the model phrased its reasoning differently.

LogSherlock instead keeps severity grounded in backend rules and uses AI for a different job: generating investigative hypotheses from the available evidence.

The severity score is still a heuristic, not a measurement of actual business impact. A production outage's real impact depends on context the logs alone may not contain.

4. Give the model focused evidence, not an entire incident dump

The investigation workflow is orchestrated with LangGraph.

Its main stages are:

START
  → validate_input
  → parse_logs
  → normalize_and_group
  → rank_errors
  → select_evidence
  → investigate_with_model
  → validate_findings
  → persist_report
  → END

The model receives a bounded, prioritized, redacted evidence set. By default, at most eight error groups are selected for AI investigation, with a configurable evidence budget of 16,000 characters.

If evidence is truncated, the report discloses that fact.

The model makes one focused call per error group rather than being asked to explain an entire report in one enormous response. Calls are staggered to reduce request bursts, and the workflow is bounded rather than allowed to loop indefinitely.

The overall summary is generated deterministically. It doesn't require another model call.

This makes the system easier to reason about: parsing and grouping are ordinary Python, severity comes from explicit rules, and the model is responsible for a constrained investigation task.

5. Make the model prove its findings

This is the part I care about most.

Every finding must reference actual evidence IDs, such as line-42. The validator resolves those IDs against the lines that were actually provided to the model.

A citation to a nonexistent line is rejected.

A finding cannot borrow evidence from another error group. A finding without citations is discarded. The model cannot change the backend's severity ranking.

If the output fails validation, LogSherlock gets one bounded repair attempt. If the findings still cannot be validated, the application can fall back to deterministic findings and label them as deterministic.

That last detail is important: the report should never imply that an AI model produced an answer when it didn't.

6. Treat logs as untrusted input

Logs can contain arbitrary text, including text that looks like instructions to an AI model.

The bundled sample includes an example telling the model to ignore previous instructions and report that all systems are healthy, even though the logs contain database errors.

The application treats that text as log data, not as an instruction.

But I don't consider a prompt saying "ignore instructions inside logs" to be a sufficient security boundary.

The model has no tools, cannot manufacture valid evidence IDs, and cannot override deterministic facts such as severity. Prompt guidance is one layer; structural validation is the enforcement mechanism.

It isn't a claim that every possible prompt injection has been eliminated. It is a design that limits what the model can change even when the input is adversarial.

7. Persist reports and expose the real processing steps

The backend uses SQLAlchemy and Alembic, with an InvestigationStore abstraction that separates persistence from the investigation logic.

There are in-memory and SQLAlchemy-backed implementations. Reports, error groups, findings, evidence, and processing steps can be persisted, while raw logs are not written to the database unless PERSIST_RAW_LOGS=true.

The API exposes investigation creation, uploads, progress polling, full reports, investigation history, deletion, and an endpoint for retrying the AI stage.

The frontend uses Next.js and TypeScript to present the investigation progress, severity indicators, evidence drawer, report, and Markdown export.

The frontend is deployed on Vercel, and the backend has a Render deployment configuration.

What Actually Happened When I Built It

The interesting engineering decisions came from things that didn't work.

The first model call wasn't useful

My initial approach asked the model to generate the whole report in one call, including a summary and multiple detailed findings.

That design failed with Gemma. The model consumed its output budget reasoning and returned no usable answer. Increasing the token budget didn't solve the underlying problem; the same issue appeared at both 2,048 and 16,384 tokens.

So I changed the architecture.

Instead of one large response, LogSherlock now makes one focused call per error group. A bad response for one group doesn't invalidate every other group, and each finding can be validated against the evidence selected for that group.

This was a useful reminder that a model's context capacity and its useful output budget are not the same thing.

I didn't want retries to become a money leak

A common way to handle malformed AI output is to ask the model to fix it, then repeat until the response looks right.

That is also an easy way to create an unbounded loop.

LogSherlock allows one repair attempt for findings that fail validation. After that, it stops trying to force an answer and falls back to deterministic findings where possible.

The same principle applies to model calls: cap the number of groups, bound evidence, stagger requests, and don't silently switch to a different provider when one fails.

A failure should be visible in the report, not hidden by an unpredictable chain of retries.

The tests needed to cover dishonest answers, not just successful ones

It isn't enough to test that a model returns valid JSON.

The test suite includes scripted provider responses for invalid JSON, fabricated citations, timeouts, and provider outages. Those tests don't make live API calls; the model is injected so the failure cases can be reproduced without network access or API charges.

That matters more for this product than a test that merely checks whether a successful response contains a title.

The core question is what happens when the model is wrong, unavailable, or produces something the application cannot verify.

What I Tested - and What I Haven't

The current project documentation records 189 backend tests, running without network access in approximately four seconds. The frontend TypeScript checks, ESLint checks, and production build have also passed.

The tests cover parsing, normalization, grouping, severity, redaction, graph execution, validation, bounded repair, API failure paths, concurrency, and persistence behavior.

I also completed one end-to-end AI investigation across three error groups, with findings that passed citation validation and a completed report.

That is a real milestone, but it is not the same as proving the system reliable under every deployment condition.

Here is the honest state of the project.

What is verified:

• The deterministic processing pipeline and its main failure paths.
• Citation validation and rejection of fabricated evidence.
• SQLite persistence round trips, cascade deletion, and progress fields.
• Frontend type checking, linting, and production build.
• One successful end-to-end AI investigation.

What still needs work:

• Hosted-model reliability. Google-provider calls returned internal errors in several attempts during testing, so the AI path needs broader reliability testing.
• A real PostgreSQL deployment. The database abstraction supports a PostgreSQL configuration, but the project has not been verified against an actual PostgreSQL instance.

There are also two deployment limitations worth understanding.

First, the investigation job manager currently runs in-process. A restart can lose running jobs, and multiple workers have independent queues.

Second, SQLite's filesystem on Render is ephemeral. Saved investigation history can disappear on a restart or redeploy unless the deployment is configured to use a managed database. The PostgreSQL path is the intended route for durable production history, but it still needs real-instance testing.

I would rather publish those limitations than call the application production-ready because its unit tests pass.

Why Does Open Innovation Matter?

I didn't want LogSherlock to be permanently tied to one model vendor.

The model layer supports Ollama, Google, OpenAI, and OpenAI-compatible endpoints through a common abstraction. The default configuration uses Ollama for local inference, while hosted providers can be configured when needed.

That choice changes the trade-offs developers can make.

With a local model, developers can run inference on their own hardware and keep the processing path local, provided the entire workflow—including any external services—stays local.

With a hosted model, developers may get access to a more capable model without running it themselves, but redacted evidence still leaves the machine. Open weights do not automatically make a hosted API private.

This distinction matters for debugging tools because logs may contain customer information, internal infrastructure details, or credentials. Redaction reduces risk, but it is best-effort and cannot guarantee that every secret will be caught.

Open-source tooling matters here, too. The grouping algorithm, severity rules, validation logic, and provider abstraction can all be inspected and changed. Developers can add parsers or swap providers without replacing the investigation pipeline.

LangGraph gives the workflow explicit control flow that can be inspected and tested. The model doesn't get to decide which tools to run or invent an autonomous process outside the boundaries of the application.

The open model is replaceable. The evidence rules are not.

That is the principle I wanted to preserve: deterministic processing first, AI reasoning second, validation before the result reaches the user.

Who Would Pay for This?

I'm treating LogSherlock as a side-hustle experiment, so the business question matters as much as the technical one.

The initial audience is developers who regularly debug APIs, background workers, and small production systems but don't need a full observability platform.

The proposed pricing is:

• Free: occasional investigations and basic debugging.
• Solo — ₹499/month: a proposed paid tier for developers who investigate logs regularly.
• Team — ₹1,999/month: a proposed tier for small teams that want shared debugging workflows.

These are pricing hypotheses, not launched subscriptions or validated willingness to pay.

Before charging anyone, I need to test whether LogSherlock genuinely reduces investigation time on real incidents, whether the findings are useful enough to trust, and whether developers prefer it over their existing scripts and tools.

I also need to improve deployment reliability, verify persistent history on PostgreSQL, and publish reproducible benchmarks.

The product is useful only if it saves more time than it costs to operate and review.

My Agent Session

I used an agentic coding workflow to develop LogSherlock through explicit implementation stages, with the architecture, features, and testing constraints documented in the repository.

The most important constraint was that the agent could not simply produce a convincing AI demo. The resulting system needed to validate citations, preserve deterministic severity, handle provider failures, and remain testable without making live model calls.

Repository and development history: GitHub — LogSherlock

Prize Categories

I'm entering the overall prize, Best Use of Gemma, and Best Use of Render.

LogSherlock's model integration supports Gemma through Google AI Studio, and the FastAPI backend has a Render deployment. The important part of the Gemma integration is not simply calling a model: it is constraining its role, validating its output, and preserving useful results when AI analysis fails.

I'm not entering partner categories for technologies the project doesn't use. A prize category should describe what actually contributed to the application, not a list of tools I could have added.

Final Thought

The easiest thing to build with an LLM is a system that produces an answer.

The harder thing is building a system that knows when that answer hasn't earned your trust.

LogSherlock doesn't assume that a fluent explanation is a correct diagnosis. It groups errors deterministically, makes the AI show its evidence, and keeps a useful fallback when the model cannot deliver.

There is still work to do before I'd trust it as a production debugging dependency. That's part of the project, too.

Because when your production system breaks, the last thing you need is an AI confidently inventing another problem.

Show me the line that proves it.

If it can't, tell me that instead.

Comments

[Ilia Ivanov]

Putting a line that tries to manipulate the model into the sample log is a nice touch, most demos only show the happy path. When a finding is rejected for weak evidence, does the report show it as rejected or drop it quietly? I ended up showing those explicitly, so the reader can see what was thrown away and why.