DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

"Please Produce the Logs from Three Years Ago" — Four Ways to Survive an Audit by Design

#audit #security #devops #logging

Two weeks to produce them.

You retained them, yes. But how many days does it take to get them back out?

Retention period and retrieval time are two different designs. Skip one and the other breaks.

Being asked for logs by an auditor runs on a different clock than incident response. An incident is about now. An audit is about three years ago, in a shape somebody else decided. Four scenes here — a request scoped by date, checking raw logs before ingestion, the economics of retention, and archives that outlive everything around them — each starting from what actually jams.

Up front: UwView Pro's sequence search returns only the places where terms appear in that order, so an audit trail like "request → approval → execution" becomes the query itself. Drill-down search stacks the narrowing and keeps the original line numbers all the way to the last stage, so an excerpt can carry the one thing a submission needs: which line of which original. The index and the compression are saved to a sidecar, so from the second open onward the file comes back with line numbers in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup, results vary). Everything runs on your own machine; the file is never sent anywhere (details at the end)

This article covers retrieving and reading a log you already hold. What you must retain, for how long, and whether a submission satisfies a requirement belong to your organisation's policy and the applicable law and regulators.

1. The retention period is met. The retrieval was never designed.

Situation

The request arrives: "Produce the operation logs for one account, for October 2023."

Policy says three years. You're inside it. The files exist. You go to open them, and stop.

That month's archive is 31 daily files, compressed, a few GB each. The names are sequential; the dates inside are unknown until you open them. And nothing here tells you which file that account appears in, or where.

So you decompress them one at a time and grep. Check free space, run one, read the output, delete, next. This does not finish today.

Why it happens

Because the policy only ever specified a period.

What's written down is "retain for N years." What isn't written down is "retrievable within N hours" — and what isn't written down doesn't get designed. So retention gets automated and retrieval stays manual.

There's a second mismatch. An audit request does not arrive in the units you stored things in. You stored by host, by date, by service. The request comes as "this account," "this transaction ID," "this window" — cuts that run across your storage layout. The way you filed it is not an index into the way you're asked for it. That's the jam.

Working with general-purpose tools, and where it stops

Narrow the file set first.

ls /archive/app-202310*.log.gz

for f in /archive/*.log.gz; do
  printf '%s\t' "$f"; gzip -dc "$f" | head -1
done
Enter fullscreen mode Exit fullscreen mode

Then search across them without unpacking to disk.

zgrep -h 'user_id=A1B2C3' /archive/app-202310*.log.gz > hits.txt
Enter fullscreen mode Exit fullscreen mode

zgrep saves you the unpacking step in the shell, but what it does internally is decompress-and-scan, sequentially. Run it over tens of gigabytes and it costs what that costs — the point of Part 8.

Three limits.

First, -h drops the filename and omitting it prefixes every line with one. A submission needs "which file, which line," and zgrep cannot give you the original line number at all (-n exists, but it counts within the decompressed stream).

Second, each extra condition is another pass. To confirm "that account made a configuration change somewhere in October," you filter by account, then look for the change within the result. zgrep A | grep B looks like one pass, but in practice every re-narrowing means reading the compressed source from the top again.

Third, and this one hurts most in practice: grep output has no context. Matching lines alone can't answer "what happened just before this action," so you end up wanting ±20 lines. Add -C 20 and the output inflates twentyfold — and you're back to a human reading 31 files' worth.

2. Look at the real thing before you write the ingest rule

Situation

Application logs are going into the log platform. You write the parse definition.

The spec documents the log format. You build the regex to match, and load. Ingestion runs. A few days later the dashboard shows about sixty percent of the volume you expected.

Lines that failed to parse were dropped. What kind of lines they were, you can't tell — they were dropped.

Why it happens

Because a spec documents one line, from the happy path.

Real logs contain lines no spec mentions.

  • Stack traces. One exception spans dozens of lines. From line two on there's no timestamp, so a line-oriented parser either discards all of it or counts each line as its own event.
  • Lines that differ by level. DEBUG carries an extra field — the kind of difference that never reaches a spec.
  • The day the format changed. A library upgrade moved a delimiter. During the transition, both shapes coexist in one file.
  • Mixed encodings. Lines that arrived from an external system still in Shift_JIS is a real situation, not a hypothetical (Part 3).
  • Broken lines. Cut at a rotation boundary, or half-written when the process died.

In an audit context this matters for a specific reason: "some lines didn't make it in" is itself a finding. When completeness is the question, "sixty percent is loaded" is not an answer.

Working with general-purpose tools, and where it stops

Count the shapes in the real file before you load it.

grep -cv '^[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}' app.log

awk -F'\t' '{print NF}' app.log | sort | uniq -c | sort -rn | head

awk '{print length}' app.log | sort -n | tail -5
Enter fullscreen mode Exit fullscreen mode

The second one earns its keep. If you expect eight fields and sevens and nines show up, the definition is already broken — before you load anything.

Three limits.

First, you can only count what you thought to count. Field counts, sure. "What do the seven-field lines actually look like?" needs eyes on the real lines. awk 'NF==7' will pull them, and that is one more full pass through the file, per idea you have. Tens of gigabytes, once per idea.

Second, the meaning lives at the tail of the distribution. If three lines out of a million have a different shape, those three are the ones worth reading. sort | uniq -c is excellent at ranking the common cases and no help at all at finding the rare one — you still have to go look at those three.

Third, sampling isn't enough in some of these cases. head -10000 tells you nothing about a malformed line sitting forty gigabytes in. As Part 13 put it, inspection sometimes presupposes being able to see all of it.

3. Compression ratio and retrievability draw on the same budget

Situation

You review storage. Better compression means more years on the same disk.

You set a policy: last month uncompressed, a year at normal compression, everything older at high compression in the archive. Capacity visibly improves.

Six months later, an audit asks for something two years old. That's the high-compression tier. Decompression takes a while. The scratch volume isn't big enough, so you do it in chunks. What you saved on storage has come back as retrieval labour.

Why it happens

Because the cost you cut and the cost you added land in different ledgers.

Storage is billed monthly and visible. Retrieval labour appears as one engineer's overtime in the month an audit happens. Both are costs; they never sit in the same table, so you end up optimising exactly one of them.

And the trade-off isn't binary. There are at least three axes.

  • Compression ratio (drives storage cost)
  • Decompression speed (drives retrieval time; higher ratios tend to be slower)
  • Whether random access is possible at all — the one that gets overlooked

That third one is the crux. Most compression formats are built as streams, so "decompress only the middle" isn't available. To read the byte at 1 GB you decompress the first 1 GB. While that holds, "pull out only the part you need" is not merely slow — it doesn't exist.

So the real trade-off isn't "compression versus searchability." It's whether the format carries an index. With one, you can raise the ratio and keep retrieval fast.

Working with general-purpose tools, and where it stops

Measure on your own data first.

for lv in 1 6 9; do
  time gzip -"$lv" -c app.log > "/tmp/t-$lv.gz"
  ls -l "/tmp/t-$lv.gz"
done

time gzip -dc /tmp/t-9.gz > /dev/null
Enter fullscreen mode Exit fullscreen mode

As a general tendency, raising the level makes compression much slower while decompression changes far less. Given that you compress once and retrieve many times, that asymmetry is on your side.

The numbers move a lot with format, implementation, and the data itself. Repetitive text like logs behaves nothing like already-compressed payloads. Building that table for your own data is the actual work of this section.

Two limits.

First, block-level indexing costs operations. Split, compress per block, and keep a separate catalogue of what's where, and random access works. Building, updating, and retaining that catalogue happens in a different budget line than the capacity you saved.

Second, the person deciding compression is not the person retrieving. Platform engineering watches capacity; audit response does the pulling. A policy set for one side's convenience is inherited by the other side six months later. The "delete, keep, or compress" call from Part 4 has to be made with that split in view.

4. Will it still be readable in seven years?

Situation

A financial system. Transaction logs, retained seven years.

You accumulate them, quietly. Five years pass. An audit asks for an older year. The files are there. You open one.

You can't read it.

The encoding is different — that system wrote Shift_JIS. The delimiter differs from today's format. The in-house tool that used to read these no longer runs; its OS left support. The person who wrote it has left.

Why it happens

Because the retention period outlives everything around the data.

List what changes in seven years and the size of the problem shows.

  • Encoding. Somewhere in there is the year the house standard moved from Shift_JIS to UTF-8. Logs with the same name differ on either side of it.
  • Format. Fields get added, delimiters move, timestamp formats change.
  • The tool that read it. Bespoke viewers stop running after an OS upgrade. When a runtime leaves support, keeping the tool alive becomes a question of keeping a whole environment alive.
  • People. Nobody guarantees that, seven years on, the organisation still contains someone who knows what field 4 meant.
  • Media. Optical and tape formats retire along with their readers.

And the last point is the essential one. What you need in seven years isn't "to read it with the tool of the day" — it's for a human to read what's inside. A log is text. As long as it's text, it is in principle readable. What makes it unreadable is almost always something around it.

Working with general-purpose tools, and where it stops

For long-term archives, store the instructions next to the data.

cat > /archive/2019/README.txt <<'EOF'
encoding : Shift_JIS (CP932)
delimiter: TAB
fields   : 1=timestamp(JST, yyyy/MM/dd HH:mm:ss) 2=account 3=action 4=result
note     : field 5 (session_id) was added on and after 2019-08-01
EOF

file -i old.log
head -c 3 old.log | od -An -tx1        # see a BOM with your own eyes

iconv -f CP932 -t UTF-8 old.log > /tmp/old-utf8.log
Enter fullscreen mode Exit fullscreen mode

That README.txt is unglamorous and has the best odds of anything here of still working in seven years. Plain text needs no tool in order to read itself.

Three limits.

First, iconv makes a copy. Leaving the original untouched is correct practice (Part 16), but it doubles the volume and adds "which of these is the original" to your bookkeeping. Hand a converted file to an auditor and you acquire the job of explaining that nothing was lost in conversion.

Second, a mixed file can't be converted in one pass. If Shift_JIS and UTF-8 coexist inside one file, iconv either stops somewhere or emits mojibake — the situation covered in Part 11.

Third, nobody has a motive to write that README at the time it must be written. The person who suffers is someone else, seven years out. Which is why this can't be an individual good habit — it has to be a step inside the procedure that creates an archive.

What the four had in common

Scene Shape of the request What jams With general-purpose tools What's left over
Dated production request "This subject, three years ago" Storage units ≠ request units zgrep across the set No original line numbers. Every re-narrowing rereads
Pre-ingest check "Is anything missing?" Real lines the spec omits Count field-count and length distributions You can only count what you thought to count
Retention economics "Cut the capacity" Saved cost and added labour, different ledgers Measure per compression level Without an index, random access is impossible
Long-term archive "From seven years ago" Retention outlives everything around it Ship the instructions; iconv to read only Conversion makes copies. Mixed files resist one pass

Four different departments, four different clocks. The right-hand column rhymes anyway, because in all four the assumptions present at storage time are absent at retrieval time.

Whoever filed it knew the split, the encoding, the meaning of each field. Whoever retrieves it knows none of that. A stranger, seven years later, standing in front of tens of compressed gigabytes with no context and no query. That is the shape of audit response.

Three conditions, then.

  • Readable while still compressed — as long as retrieval requires scratch space and a wait, it stays a labour cost.
  • The original coordinates survive narrowing — a submission needs a correspondence ("line N of the original"), not a loose fragment. If every stage renumbers, you can't build one.
  • It finishes on your own machine — whether audit-scope logs may go to an external service is a policy question, not a technical one. Where the answer is no, this is the first filter anything has to pass.

Back to those two weeks. That was never a retention problem. It was the time when nobody storing the logs thought about the person who would have to get them back out.

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls, and searches from the moment it opens. It doesn't load the whole file into memory, so it opens files larger than RAM. The index builds in the background; when it completes, line numbers appear.

Of the three conditions above, the free edition covers the third.

  • Everything runs on your own machine. The file isn't sent anywhere. Where taking audit-scope logs off-site is the contentious part, that's a precondition rather than a feature.
  • It never writes to the original. No splitting, no extracting, so a file pulled from an archive stays one file, unmodified.
  • Encoding switches while the file stays open (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). For reading, the second copy iconv makes in section 4 stops being necessary.

The first two conditions are UwView Pro territory.

  • Sequence search: only the places where w1 → w2 → w3 appear in that order. The audit-shaped sequences — "request → approval → execution," "login → privilege change → logout" — become the query directly (the implementation post). Honestly: each stage scans forward from the previous hit, so it costs roughly what a full-text search costs.
  • Drill-down search: narrow a result by another term. Section 1's "filter by account, then find the configuration change" becomes a second stage instead of a second read of the source. Tabs show term (count), and the original line numbers survive to the last stage (the drill-down post).
  • ±N is per stage: ±1 while narrowing, ±20 on the stage you actually read. You choose the width after you know what you need, so section 1's "-C 20 inflates the output twentyfold" doesn't happen (free edition is fixed at ±1; variable ±N is Pro).
  • Tally (frequency ranking): counts the values a regex captures, ranked, click a row to descend to it. Close to section 2's field-count distribution, without a pass per idea. It does not compute sums or averages — its lineage is grep -oE | sort | uniq -c, not awk.
  • The index and the compression are saved: from the second open onward the file comes back with line numbers, instantly (0.02–0.07 s measured on a 47.73 GB text file; one specific setup, results vary). Audit work is never read-once. Follow-up questions, clarifications, writing the submission — the same file, reopened across days.
  • ~1/9 storage that stays searchable: directly against section 3's "no index, no random access." Pro's sidecar (.uwvz) carries a per-block offset table, so you jump to the part you need while it stays compressed — no decompress-from-the-top scan the way zgrep does it (Part 8).

The sidecar also keeps an XxHash3 table per compressed block and verifies on every decompression, so bit rot during long storage, or a copy that was truncated in transit, won't be read straight past unnoticed. This is not tamper detection. XxHash3 is a non-cryptographic hash chosen for speed. What tamper resistance needs is sha256sum plus separated custody; this check exists to catch accidents (Part 16).

Where it honestly stops

UwView is a viewer. It is not a SIEM, not a log management platform, not an audit-trail system.

  • It doesn't write your ingest definitions (section 2). No parse validation, no loading, no automatic detection of what got dropped.
  • It doesn't automate retention policy (section 3). No generation management, no deletion, no inventory reports.
  • Writing the README.txt in section 4 is your job. There's no metadata management.
  • It records no access trail — who opened what, when. If your audit requires viewing logs, this tool alone cannot satisfy it.
  • No cross-log correlation, no alerting, no report generation, no threat-intel matching.
  • It handles text. Binary database dumps and disk images are out of scope.

One more constraint, stated plainly for people who touch archive storage. Pro's sidecar (.uwvz) is created as a new file beside the original. The original itself doesn't change by a single byte, but if your practice is that nothing new appears inside the archive area, copy the file to a working volume before opening it. (The sidecar keys on the original's length and mtime, so it invalidates itself automatically if the original changes.)

What this tool covers is the step before all of that: read the stored raw log, as it is, on your own machine, with the original coordinates intact, with your own eyes. If you're at the scale that needs correlation and alerting, that's a different product's job.

For reference, non-destructive differential editing (Edit Upgrade) exists as a separate licence, but every one of this article's four sections is read-only work. What you need here is the View side.

The same things, from the command line

v1.6.0 added a uvp command (and uvf for the free build). It uses the same .uwvz as the GUI, so an index built from the shell is already there when you open the file in the app. Mapped onto this article's four parts:

uvp /archive/app-20231015.log 'user_id=A1B2C3' 'config.change' -C 20 -out audit-202310.txt

uvp /archive/app-20231015.log -seq 'REQUEST,APPROVE,EXECUTE' -C 5

uvp app.log -uniq '^(.{0,10})' -head 20

uvp /archive/2023/app-20231015.log.uwvz 'user_id=A1B2C3'

uvp /archive/2019/old.log.uwvz -extract -out /tmp/restore/
Enter fullscreen mode Exit fullscreen mode

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. Section 1's "work through 31 files" becomes one uvp line inside for f in /archive/app-202310*.log; do ... done, with the exit code telling you which files actually held something. Stated plainly, though: uvp records no access trail either. An audit that requires a log of who viewed what, and when, is still not something this tool answers on its own.

Stated honestly: on the first question ripgrep is 15–20% faster, because uvp builds its index first. On a 3 GB file that fits in RAM, rg stays ahead on the second question too. uvp pays off past 10 GB, when you ask the same file more than one question (the measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Actual audit response, internal control, and retention policy must follow your organisation's rules and the applicable law and regulators. Nothing here is advice on statutory retention obligations or on what a submission must contain. Retention periods, dates, account names, filenames, and field layouts are illustrative and describe no specific regime, organisation, or case. The behaviour of gzip, zgrep, grep, awk, iconv, file, and od varies by implementation (GNU/BSD/busybox), version, and build options, as do option names and defaults — check your own man pages. Statements about compression ratio and decompression speed are general tendencies; results change with the data, the format, and the implementation. Measured figures come from one specific setup and are not a guarantee of the same result — disk type, filesystem, fragmentation, encryption, page-cache state, and concurrent processes all move them substantially. If you spot an error, a comment is welcome and I'll check and correct it.

Top comments (0)