DEV Community

y4u
y4u

Posted on Edited on Originally published at uvp.y42u.net

Find Only Where 'New York Central Park coffee_shop' Appear in That Order — Sequence Search on a 258 GB Local File

#showdev #security #logging #tools

Sequence Search in UwView Pro: New York 100,492 hits → Central Park 2,798 → coffee_shop 1,246, with the right-click History menu showing the sequence path

A follow-up to Drill-down Search: the same tabbed UI now has a second mode, Sequence Search.

What does searching by "order" mean?

Drill-down finds words that occur near each other. Sequence Search finds order: only the places where w1 → w2 → w3 appear in that sequence.

In log terms: you want only the spots where "login → timeout → restart" happened in that order. An and/or search can only say "all three words are present" — but an incident is a story, and stories have order. A different order is a different incident.

In the security world this is exactly Elastic EQL's sequence and Splunk's transaction — but both require the data to be ingested into a server first. Almost nothing lets you do this against a multi-gigabyte file sitting on your own machine.

Real example: 1,246 "New York → Central Park → coffee_shop" flows in 4.5 billion lines

The screenshot above is the entire US OpenStreetMap extract — 258.68 GB, 4,509,830,821 lines, searched directly from its compressed .uwvz archive:

New York (100,492) → Central Park (2,798) → coffee_shop (1,246)

"New York" appears 100,492 times. 2,798 of those are later followed by "Central Park" — and 1,246 flows continue on to "coffee_shop". Note how different this is from Drill-down on the same three words (100,492 → 226 → 3): Drill-down finds the words near each other; Sequence links occurrences in order, however far apart — the flow in the screenshot chains hits tens of thousands of lines apart (line 151,568,463 → 151,599,349 → 151,616,193). Usage is identical to Drill-down: switch the refine bar's mode to "Sequence" and keep adding words.

If an earlier word reappears mid-match, the automaton restarts from it (New York → Central Park → another New York? Matching resumes from the new New York). What survives is only the clean, uninterrupted flow.

The path itself is visible — the History menu

My favorite part of this implementation is the right-click menu on any result.

Open History (drill-down path) and you see the position of every word that formed that sequence — e.g. "line 151,568,463: New York → line 151,599,349: Central Park → line 151,616,193: coffee_shop" — and you can jump to the file at any of those lines.

No reverse-engineering "why did this line match?" in your head. You see the actual path of the matched flow, and you can walk it. Even EQL and Splunk don't show it this directly.

Speed

Each stage of Sequence Search scans the file body starting from the previous stage's result positions. The starting points are narrowed, but the processing itself is equivalent to a full-text search — so each stage takes roughly as long as a full-text search. This is the one way it differs from Drill-down, where stages after the first are instant (Drill-down only re-searches inside the previous stage's ±N context blocks). As with Drill-down, the ±N context lines are set independently per stage.

Get it

Drill-down and Sequence Search ship in UwView Pro — released and available now (introduced in V1.3, current release V1.4.0). The free edition keeps the classic single-stage search with ±1 context and result export.

UwView Pro is available on Polar (V1.4.0 added the Edit Upgrade for non-destructive editing). Every license comes with a 14-day free trial — get a trial key on Polar, no payment details required.

One more thing: the editing-capable version once announced as **UwEditor Pro* has shipped — as of V1.4.0 it is integrated into UwView Pro as the Edit Upgrade license, adding non-destructive editing (replace-all, pause, resume) on top of your View License.*

Related

Top comments (0)