A client sends you an AWR report under NDA. I built a second opinion that never leaves your laptop

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Every Oracle DBA who freelances knows this email. "The system was slow yesterday between 2 and 4. AWR attached. Can you take a look?" Attached is a 200-page HTML report covering every wait event, every statement and every segment the database touched. The client is paying by the hour, and the report is covered by an NDA. It holds their SQL text, table names and host names.

The first hour on a report like that is always the same triage. Is the problem parsing, I/O, locking, redo, or one bad statement? Load Profile, Top Events, SQL ordered by gets, back to the events, then the instance activity stats. It's skilled work, but it's routine. And the obvious shortcut is off the table: you can't paste a client's AWR report into a hosted chatbot.

So I built Second Opinion. It's a first-pass AWR and Statspack reviewer that runs entirely on your own machine, with an open model. Once the model is loaded, it takes under a second on my desktop GPU to rank the likely root causes, show the numbers behind each one, and draft a short note for the client. Every number in that note is checked against the report before you see it.

In short: on 18 held-out lab reports with faults I injected myself, Second Opinion named the cause in 17 of 18. Oracle's own ADDM got 7/9 of the AWR ones, and Standard Edition can't run ADDM at all. Gemma 4 given the raw report got 10/18. Neither model ever quoted a number that wasn't in its input. I also fine-tuned Qwen3-8B on Tinker to rank the causes with no rules at all. Untuned it got 11/18; tuned, 16/18, with the top cause right on every run, and it runs locally in Ollama.

What I Built

Second Opinion is a command-line tool and a small Python library:

$ second-opinion awrrpt_1_17_18.html          # AWR, as awrrpt.sql writes it
$ second-opinion sp_10_11.lst                 # Statspack, any release
$ second-opinion pasted.txt                   # the sections a colleague pasted into a ticket

It reads the report and runs the checks a DBA runs in their head. Gemma 4 then writes up what those checks found:

Statspack report · ORCLCDB 12.2.0.1.0 · 13-Oct-26 16:58:47 to 17:05:47 (7.0 min, DB time 27.71 min)

**High Hard Parse Rate Indicates Inefficient SQL Usage**
Likely causes, most important first: Hard parsing (literal SQL, no binds)

## Note for the client
The Statspack report indicates a significant performance issue primarily driven by hard parsing.
The evidence shows that 100.0% of parses are hard parses, with 8811.8 hard parses per second. This
high rate is contributing to the high wait time on library cache: mutex X, which accounts for 39.5%
of DB time. To address this, first, review the application code to ensure all SQL statements
utilize bind variables instead of literals. ...

## Findings (computed from the report, not by the model)
- Hard parsing (literal SQL, no binds) (score 1.0): Most parses are hard parses...
  hard_parses_per_sec=8811.8, parses_per_sec=8812.8, hard_parse_pct=100.0,
  shared_pool_event=library cache: mutex X, shared_pool_event_pct_db_time=39.5

_Numbers checked: 3; every number in the note was found in the report. Model: gemma4:e4b. 0.6 s, all local._

That's a real run on one of the held-out reports. It isn't perfect: the note's second suggestion, about indexing, has nothing to do with hard parsing. That's exactly why the output is a draft for a DBA, not something to send unread.

Who pays for this:

• Freelance and contract DBAs. They bill roughly $90–180 an hour, and on Fiverr an "AWR/Statspack performance review" package sells for around $220. Each report takes 30–60 minutes of triage before the real work starts. A tool that does the first pass, can't leak the report, and gives a draft client note gets that hour back on every engagement. Even one recovered billable hour a month would make $15–20 a month an easy sale. I'd price it as a one-time license per consultant, because "runs on your laptop" and "subscription" sit awkwardly together.
• Shops on Standard Edition. SE and SE2 can't license the Diagnostics Pack, so they have no AWR and no ADDM. Statspack is what they have, and often nobody reads it. Second Opinion treats Statspack as a first-class input, and on Standard Edition it is the only automated reviewer there is.
• Managed-service teams that get a "slow yesterday" ticket at 2 a.m. and want a first read before they wake the senior DBA.

Demo

A two-minute narrated walkthrough: a real report, the review, the Sentry traces, and the Tinker before/after.

Code

pyaroslav / second-opinion

A first-pass Oracle AWR/Statspack review on your own machine, with a local open model (Gemma 4) and a check that rejects invented numbers.

Second Opinion

Second Opinion gives a first-pass review of an Oracle AWR or Statspack report. It runs on your own machine with a local open model, Gemma 4 via Ollama, so the report never leaves the box.

This is a real run on one of the held-out lab reports (reports/test/hard_parse-statspack-*):

$ second-opinion statspack.txt
# Second Opinion: statspack.txt
Statspack report · ORCLCDB 12.2.0.1.0 · 13-Oct-26 16:58:47 to 13-Oct-26 17:05:47 (7.0 min, DB time 27.71 min)

**High Hard Parse Rate Indicates Inefficient SQL Usage**

Likely causes, most important first: Hard parsing (literal SQL, no binds)

## Note for the client

The Statspack report indicates a significant performance issue related to SQL execution. The primary
finding is a high rate of hard parses, with 100.0% of parses being hard parses. This is evidenced by
the high hard_parses_per_sec of 8811.8 and the associated wait event library cache: mutex X
consuming 39.5% of

…

View on GitHub

How I Built It

Don't hand the model the whole report

My first attempt was the obvious one: give Gemma 4 the AWR report and ask what's wrong. An AWR report for an 18-second window is about 18,000 tokens of text, and a real one-hour report is far bigger. On my held-out set, Gemma reading the raw report found the cause in 10 of 18. It reads alarming-looking wait events as problems on healthy systems, and it misses every full table scan, because the evidence for a full scan is spread across three sections (SQL ordered by gets, instance activity, segments by table scans).

So the model never reads the report. The work is split three ways:

1. Parsers (awr.py, statspack.py, text.py) turn AWR HTML, AWR text, Statspack from any release, or a few pasted sections into one small dict: load profile, top events, time model, a few instance activity stats, top SQL per category, and segments. Statspack and AWR text are fixed-width, so column names are read from the header above each dashed rule. Pasted text often has its whitespace mangled, so rows are read token by token: a name, then numbers. On all 32 lab runs, the text parser gives exactly the same findings as the HTML parser.
2. Rules (rules.py, about 150 lines) are the checks I'd run by hand: hard parsing, full scans (from logical reads, or from db file scattered read when the scans come from disk), row locks, commit storms, NOCACHE sequences, one CPU-heavy statement, and Resource Manager CPU caps. Each fires with the numbers that made it fire. A materiality gate stops a large percentage of an idle database from becoming a finding: "74% of DB time is log file sync" means nothing at 0.1 active sessions.
3. Gemma 4 (e4b, local in Ollama, about 160 tokens per second on my GPU) only sees the findings and a little context. It ranks the causes, writes a headline and drafts the client note.

A model that can't invent numbers

A DBA can live with a model that writes clumsy prose. A model that invents a statistic, in a note going to a client, is a disaster. So every number in Gemma's note is checked against the numbers it was given, allowing for rounding and thousands separators. If any number isn't there, the note goes back once with the offending numbers listed. If a number still can't be found after the retry, it is printed in a warning, not quietly dropped.

On the held-out set, neither Gemma reviewer quoted a number that wasn't in its input, and Second Opinion's retry never fired.

Testing it against faults I injected myself

You can't evaluate a diagnosis tool on reports where nobody knows the answer, so I built a fault lab. There are two disposable Oracle databases in Docker on my workstation:

Database	What it produces
Oracle AI Database 26ai Free 23.26.3	AWR + ADDM (Diagnostics Pack included in Free)
Oracle Database 12.2 Standard Edition 2, control_management_pack_access=NONE	Statspack only

lab/faults.py runs a workload with a known fault for 5–7 minutes: literal SQL, a missing index, sessions fighting over the same rows, commit-per-row, a NOCACHE ORDER sequence, a cartesian join, or a healthy light load. It takes a snapshot before and after, then saves the AWR, ADDM or Statspack report a DBA would get.

The rules were written while looking at a 14-run dev set. Then I wrote a held-out set: the same causes, but different SQL, different tables and different session counts, plus two runs with two faults at once. I scored it once.

Held-out, 18 runs. A reviewer passes if the injected cause is in its top two. With two faults at once, both have to be in its top three.

Reviewer	AWR (9)	Statspack (9)
Oracle ADDM	7/9	not available on SE2
Gemma 4, raw report	5/9	5/9
Rules alone	9/9	8/9
Second Opinion	9/9	8/9

What the misses taught me:

• ADDM found the right SQL statement in both full-scan runs, but its advice was "Run SQL Tuning Advisor", which needs the Tuning Pack. It never said that the query full-scans a 400,000-row table with no index on its where column.
  • My first ADDM scoring used keyword matching and gave it 5/9. When I read the ADDM reports by hand, two "misses" were findings titled "Hard Parse" and "Sequence Usage", mislabelled because the text nearby mentioned CPU. I switched to mapping ADDM's finding titles, and 7/9 is the corrected figure.
• Our one miss is the NOCACHE sequence on 12.2 Statspack. That report shows the commit pressure but no sequence wait in its top events. I left the rules as they are rather than tune them on the test set.
• Lab caveats. In the healthy AWR run, the light update workload hit two genuine ORA-00060 deadlocks. Second Opinion still called it healthy and listed row locks below the significance line. 26ai Free also caps its own CPU through Resource Manager, so busy runs show resmgr:cpu quantum; that's real for that database, but an artefact of Free.

Real reports from the wild

Lab faults are tidy. To see how it copes with what people actually paste, I collected ten AWR and Statspack reports posted publicly on the OraFAQ forums and Jonathan Lewis's blog. Each has an expert's diagnosis in the thread, and all ten are text, from Oracle 9i to 11g, and often only a few sections.

Two things broke at first:

• I had no AWR-text parser, so I wrote one.
• My rules had never seen db file scattered read, because in the lab every table fit in memory. On real systems that's the textbook sign of full scans from disk. I added that check, plus one that says "this wait dominates and nothing explains it" instead of calling such a system healthy.

After that, the rules matched the expert's diagnosis on 7 of the 10. The other three were two excerpts that left out the SQL sections the diagnosis depended on, and one log-switch problem just under my threshold. I wrote those two checks after reading these cases, so this is a set of worked examples, not a test score.

Thread	Report	Expert's call (paraphrased)	Second Opinion	Match
Statspack report: suggestion	Statspack	Full scans of a big table (scattered reads)	full scans	✓
statspack report	Statspack	Scattered reads on top: the SQL is full-scanning	full scans	✓
Upgrade Argh	AWR text	Small tables' scans turned into direct path reads after an upgrade	full scans	✓
High System I/O wait class	AWR text	log file sync dominates (slow redo writes vs. too many commits debated)	commit storm, other	✓
Fast now, fast later	Statspack	Single-block reads grow as the data grows	other: db file sequential read dominates	✓
Network waits from AWR	AWR text	Nothing to fix: network waits are a tiny share	healthy	✓
Reading AWR	AWR text	Mostly CPU, no instance-level problem	healthy	✓
NOLOGGING in 11gR2	AWR text	Log switches too frequent (checkpoint incomplete, ~19%)	healthy	✗
Analysing Statspack 13	Statspack excerpt	CPU-bound on logical I/O from scans; needs the SQL sections	healthy	✗
Analysing Statspack 5	Statspack excerpt	Repeated single-row deletes each doing a short full scan	healthy	✗

The three misses are honest misses. Two are blog excerpts where the answer lives in SQL sections that weren't quoted, and a tool that only sees the Top 5 events can't find it there either. The third is a real configuration problem that sits below my 25% threshold.

Fine-tuning a small model on Tinker

The rules in Second Opinion are hand-written, and I wrote them knowing which faults I'd inject. So the fair question is whether a small open model could learn that triage itself from the parsed numbers, with no rules at all. Tinker let me try that for about the price of a coffee.

Data. I left two generators running overnight against both lab databases: 524 short runs with random faults (one at a time, or two at once a quarter of the time), random session counts and 90–300 s windows. They use their own tables, SQL and sequences, so nothing overlaps the held-out set. A label is a fault that actually reached the report. I dropped 52 runs where the injected fault never showed. In one, for example, a literal-SQL flood had aged the CPU-heavy statement out of the shared pool before the snapshot, so no report could show it. That left 425 runs to train on and 61 to validate on (47 generated plus my 14 dev runs). The 18 held-out runs stayed untouched.

Task. The input is the parsed report, trimmed to what triage needs. The output is JSON ranking the likely causes. The base model and the tuned one get the same prompt, with Qwen3's thinking turned off.

Training. LoRA rank 32 on Qwen3-8B, 3 epochs, 81 steps, with a checkpoint saved after each epoch. Training used 3.66M tokens, about $1.61 at list price. With the early test run and the sampling for evaluation, the whole experiment cost roughly $2 of a $10 credit. I picked the epoch on the validation set:

Qwen3-8B, validation (61 runs)	Pass	Top cause right	False causes
Base, untuned	40	31	136
Tuned, epoch 1	55	61	0
Tuned, epoch 2	51	57	4
Tuned, epoch 3	57	61	1

Then I scored the base model and epoch 3 once on the 18 held-out runs:

Qwen3-8B, held-out (18 runs)	AWR	Statspack	Top cause right	False causes
Base, untuned	5/9	6/9	11/18	38
Tuned	8/9	8/9	18/18	0

"Pass" is the same rule as the main table: the cause in the top 2, or both causes in the top 3 when two were injected. A false cause is a label that wasn't injected. I don't count cpu_cap as a false cause, because Free really does throttle itself, and the training labels include it when the report shows it.

The base model's problem is easy to see: it names three causes every time, so it is never sure and often wrong. After tuning it names one or two, and the first one is right on every held-out run. Its two misses are both the full-scan + commit-storm pair, where it names the commit storm and drops the full scan.

This doesn't replace the rules. The rules plus Gemma pipeline is still the tool, at 17/18 with a written note and every number checked. But a tuned 8B model with no rules reached 16/18 on the same runs. That tells me the triage isn't only in my head, and a model can learn it from lab data I can make as much of as I like.

So it went into the tool as a second, independent read. second-opinion report.html --triage-model second-opinion-qwen3 adds the tuned model's own ranking under the review, checks the numbers in its evidence the same way, and says whether it agrees. On the held-out set it catches the review's one miss: on the 12.2 Statspack sequence run, the rules say "committing too often", and the tuned model says "sequence contention", which is right. It also disagrees on the healthy AWR run with real deadlocks. A disagreement isn't a verdict; it's a reason to read both before writing to the client.

Running it locally. The tuned model only counts for this use case if it runs offline, so I exported the adapter to Ollama. That was harder than expected:

• Ollama 0.35 no longer takes a LoRA adapter in a Modelfile.
• Merging into the stock 4-bit model and saving it back at 4-bit lost the tuning.
• A base I converted myself from the Hugging Face weights gave repeating garbage once merged.

What works is merging the adapter into the exact qwen3:8b weights Ollama already has and stopping at 8-bit (lab/tune/export.py). The local copy passes 16/18 too, and it agrees with Tinker's top cause on 17 of 18 runs, at about 0.5–0.7 s per report. The adapter is on Hugging Face: pyaroslav/second-opinion-qwen3-8b-lora, in PEFT and GGUF form. Where it differs, it misreads the healthy AWR run as row locks (the run with the real deadlocks), and it gets one full-scan + commit-storm pair that the Tinker copy missed.

Watching the agent with Sentry

Tracing an agent whose whole selling point is "nothing leaves your laptop" is a bit of a contradiction. So the first design decision was what Sentry may see. With SENTRY_DSN set, each review becomes one gen_ai.invoke_agent transaction with these child spans:

• gen_ai.execute_tool for parsing;
• gen_ai.execute_tool for the rules;
• one gen_ai.chat per model call, including the retry.

What leaves the machine: span names, durations, the model name, token counts, which causes the rules found, and how many numbers failed the check. There are no prompt or response attributes, send_default_pii=False, and tracing is off unless you turn it on. Sentry's own agent view confirms it:

Setup. I wrapped both reviewers in the benchmark as agents, "second-opinion" and "raw-baseline", and ran all 18 held-out reports through each.

Performance (tokens, latency, cost). The agents dashboard made the cost of the naive approach very concrete:

Agent	Tokens per review	Median latency (local GPU)	Found the cause
raw-baseline (whole report to Gemma)	16K–24K	2.3 s	10/18
second-opinion (parse + rules + Gemma)	1.2K–1.4K	0.7 s	17/18

The dollar cost is $0 either way, because it runs locally. But about 15 times fewer tokens is what lets a 4B-parameter model run on a consultant's laptop at all, instead of needing a 32K context and a bigger GPU.

What I found, and how I debugged it. The first version of that table said 5.0 s for Second Opinion and 6.7 s for the raw baseline. Then I compared it with the traces from running the CLI on the same reports, and the CLI reviews took 0.65–0.92 s:

Same model, same prompt, but a 7× difference in latency. The waterfall put almost all the time in the gen_ai.chat span, so the problem wasn't my code. In the benchmark, the two reviewers alternate, and they asked Ollama for different context sizes (16K for mine, 32K for the raw report). I checked Ollama's load_duration directly. With the same context size, a warm call takes about 0.1 s. Alternating 16K and 32K forces a ~4.2 s model reload on every single call. The benchmark had been timing model reloads, not reviews.

The fix is one line: a single context size for every call. After the fix, the same benchmark in Sentry alternates between 1.9–2.8 s for the raw baseline and 0.64–0.72 s for Second Opinion:

Accuracy didn't change, but every latency figure in this post did, and without the traces I would have published the wrong ones.

Why Does Open Innovation Matter?

For this job, "open" is the product, not a nice-to-have:

• The report can't leave the machine. An AWR report is a map of a client's system: SQL text, schema and table names, host names, sometimes bind values. A DBA under NDA can't paste it into a hosted model, however good that model is. An open-weight model running in local Ollama is the only way an LLM gets to touch this work at all.
• Statspack shops are the ones without money. The people with no Diagnostics Pack, and so no ADDM, are exactly the people who won't buy another subscription. A free open model on their own hardware is the only version of this they'll ever use.
• I could change the model, not just the prompt. With open weights I could fine-tune a small model for this exact task on Tinker, merge the adapter into the model Ollama already had, and run it offline. It agrees with the Tinker-hosted copy on 17 of 18 held-out runs. A closed API would have given me a prompt and nothing else.
• Every step is inspectable. The parser, the rules and the number check are about 900 lines of readable Python. A DBA can read why the tool said what it said, which is what they'd demand before putting their name on a client note.

My First User

I didn't want to put a client's report anywhere near this, so the first "user" was a blind reviewer. A separate Claude instance, with no access to my code, labels or results, was told to act as a senior DBA. It was given six lab reports it had never seen, renamed report-1 to report-6, and asked for a by-hand triage: ranked causes, the numbers that convinced it, and what it would check next. I also renamed one sequence whose name gave the answer away. Then I scored its read and Second Opinion's against the injected faults, with the same pass rule as everywhere else:

Report	Injected	Blind reviewer	Second Opinion
AWR	full scan	full scan, CPU cap ✓	full scan, CPU cap, CPU SQL ✓
Statspack	NOCACHE sequence	commits, sequence ✓	commits, sequence ✓
Statspack	healthy	healthy ✓	healthy ✓
AWR	commit storm	commits, small redo logs ✓	commits ✓
Statspack	row locks + literal SQL	hard parse, row locks ✓	hard parse, row locks ✓
AWR	CPU-heavy SQL + literal SQL	commits, CPU SQL ✗	hard parse, commits, CPU cap ✗

What I learned:

• 5 of 6 each, and both missed the same report, from opposite sides. In the last report, the reviewer found the CPU-heavy regex query but set the literal SQL aside, because hard parsing was only 3.5% of DB time (ADDM agreed). Second Opinion ranked the literal SQL first and never named the regex query, even though it takes a third of DB time. The CPU-SQL rule only fires when CPU is at least half of DB time. Here commit waits and the Resource Manager cap took so much of the time that CPU was only 36%, so the rule never fired. A single statement taking a third of DB time should count whatever else is going on. That gate is my next fix.
• My label was wrong, and both of them caught it. That run's literal-SQL workload commits after every row, so there really is a commit storm (log file sync at 42% of DB time). Both reviewers named it. My injected-fault label didn't. Lab labels are what I meant to inject, not always everything the database actually did.
• The reviewer added things the tool can't say yet: undersized redo logs (32 log switches in 3 minutes), and that the CPU-cap waits are a symptom of the full scan, not a separate problem. Both are going on the list.

So the "user" is a model playing a DBA, not a paying consultant, and six reports is a small sample. A real DBA's blind pass is the next step.

Prize Categories

• Best Use of Tinker: a LoRA fine-tune of Qwen3-8B on 425 lab reports I generated with injected faults, picked on a 61-run validation set and scored once on 18 held-out runs. It went from 11/18 to 16/18, with the top cause right on 18/18 instead of 11/18 and false causes down from 38 to 0, for about $2. Then I exported it to run offline in Ollama, since a model that has to phone home is useless for reports under NDA. The adapter is published on Hugging Face.
• Best Use of Sentry Agent Tracing: every review is traced as an agent run (parse → rules → gen_ai.chat), with tokens and latency per call and no report data leaving the machine. The traces showed that the raw-report approach costs about 15 times more tokens. They also caught a benchmark bug: Ollama reloading the model on every call, which had inflated my latencies 7×.