Snook: The Protected Ingestion Plane

XogtaTech — Sun, 23 Aug 2026 02:11:42 +0000

Move operational data into open lakehouse tables without making raw data the
price of using cloud analytics.

Modern ingestion usually begins with a network diagram. A cloud service,
integration runtime, or distributed processing cluster needs access to each
database, file server, application host, stream, or legacy platform. That
brings credentials, inbound routes, vendor allowlists, staging areas, and a
period in which sensitive data has moved but has not yet been protected.

Snook starts from a different boundary: execute beside the source, protect each
sensitive field before the first landed file, and send protected analytical
data plus verifiable evidence onward.

Snook is a protected ingestion product that owns the path from source evidence
to an open table:

database, file, stream, or mainframe source
  -> schema and contract
  -> bounded extraction or decode
  -> quality and protection
  -> Parquet
  -> Iceberg
  -> manifest and lineage evidence

The scheduler and lakehouse may remain in the cloud. Raw-data access does not
have to. Snook supports relational databases, structured files, bounded stream
windows, COBOL-described datasets, and DB2 unloads through the same contract,
protection, Parquet, Iceberg, manifest, and lineage model.

Walk Into the Worst Case

A friendly product demonstration would start with CSV, Parquet, or a simple SQL
table. Those sources matter, but they avoid the difficult question: does the
architecture still hold when extraction is inseparable from binary framing,
schema interpretation, character conversion, and exact decimal decoding?

That is why this test walks into mainframe data. Mainframe is an example, not
the product boundary, and it gives us three increasingly demanding source
paths:

COBOL-described datasets. Fixed, variable, and blocked EBCDIC records are decoded from a copybook, including packed decimal, binary numerics, REDEFINES, OCCURS, and hierarchical record structures.
DB2 for z/OS UNLOAD datasets. Snook reads the binary UNLOAD data together with the SYSPUNCH LOAD control card. The LOAD card is the schema source: its positions, DB2 types, and null indicators drive the same decode, protection, Parquet, and evidence path without a database connection or copybook.
Live DB2 extraction. When direct access is appropriate, Snook can read DB2 through IBM's CLI driver, with bounded incremental windows and parallel numeric range extraction. Credentials remain references and are masked from manifests.

The same contract model covers PostgreSQL, MySQL and MariaDB, SQL Server,
Oracle, Sybase ASE, structured files, and bounded Kafka windows. Mainframe is
where we chose to stress the architecture, not the market category Snook is
limited to.

Prove the Cat, Then Carry the Kitten

There is a useful engineering test for an ingestion architecture: if a cat can
pass through the opening, a kitten can pass through it too.

EBCDIC is the cat. It is the path most likely to be excluded from a friendly
benchmark because every byte requires interpretation. Record boundaries may be
fixed, RDW-framed, or block-framed. Text needs a host code page. Decimal values
may be zoned or packed. Binary numbers, overlays, repeated groups, and
record-type hierarchies all have to be interpreted before a columnar writer can
begin its work.

A CSV copy, a Parquet relocation, or a straightforward SQL result would make a
better-looking demonstration. It would prove less. We deliberately tested the
larger animal: a wide EBCDIC dataset with protection in the hot path and an
Iceberg commit at the end.

That does not make every easier source identical. It demonstrates that the
bounded execution, protection, Parquet, catalog, and evidence architecture can
carry the least forgiving source shape without requiring a cluster.

The Product Boundary

Snook brings a small Go execution layer to the source environment. A versioned
contract controls:

source identity and schema source;
record format, code page, and decode behavior;
destination table and load mode;
schema evolution and quality rules;
field classification and protection mode;
batch, worker, memory, and Custody concurrency budgets; and
lineage and evidence emitted by the run.

The UI is the authoring and review surface for that contract. Operators can
sample through the production decoder, inspect discovered fields and types,
review classification evidence, assign protection, set execution limits, and
validate the exact artifact that the runner will execute.

For the measured EBCDIC run, the UI discovered all 100 fields and displayed
source and destination types together. Classification remained reviewable. The
classifier identified the card and phone fields cleanly; the synthetic Canadian
SIN also resembled a US routing number. Snook surfaced that disagreement rather
than silently turning a probabilistic guess into security policy.

Fast binaries are useful. A reproducible, inspectable decision surface is what
makes them an enterprise product.

Protection Before Landing

The execution order is deliberate:

source bytes -> decode/extract -> quality -> protection -> Parquet -> Iceberg

Cleartext exists only in the in-flight batch. Snook does not write raw Parquet
and clean it up in a second job. Data on the destination side of the boundary is
already protected.

The benchmark contract made three different decisions:

card_number used a Snook Custody vault-token;
sin used FF1 format-preserving encryption; and
phone used FF1 format-preserving encryption.

FF1 retains shape, which matters when downstream schemas cannot absorb a width
or type change. It remains reversible with the key and should not be presented
as automatic PCI descoping.

A Custody token is a random surrogate. It preserves no meaningful portion of
the card number and cannot be reversed without Custody. Snook Custody is the
licensed vault itself, not an adapter to a separate vault product.

Mode	Raw value leaves the source?	Preserves format?	Security boundary
Raw cloud pull	Yes	Yes	Protection occurs after movement
Local FF1	No	Yes	Reversible cryptographic protection
Snook Custody token	No	No	Random token with isolated recovery capability

Protection is fail-closed. If a contract requires Custody and Custody is
unavailable, the run stops. It does not land cleartext or silently substitute a
weaker mode.

Most of the platform is open source. Snook Custody is the licensed component
and is not planned for open-source release. An open-source-only deployment can
choose FF1 for a PCI field, but the contract records the choice and requires an
explicit acknowledgement of the different security boundary.

The Deliberately Difficult Test

The source was a 24.45 GB synthetic EBCDIC file containing:

50,000,000 fixed-length records;
489 bytes per record in code page 037;
100 COBOL fields per record;
one million distinct synthetic card numbers repeated across the file;
a synthetic SIN and phone number on every row;
30 text fields;
40 display-numeric fields; and
26 signed packed-decimal fields.

The generator and copybook shared one layout definition. Independent boundary
checks verified a file size of exactly 24,450,000,000 bytes, no remainder
against the 489-byte record length, and a card sequence repeating at exactly one
million unique values.

This was a COBOL-copybook EBCDIC test, not a DB2 UNLOAD benchmark. DB2 UNLOAD
uses its LOAD control card as the schema source and joins the same downstream
protection, Parquet, Iceberg, manifest, and lineage path. Its throughput should
be measured separately rather than inferred from this result.

The UI-authored contract selected:

append into raw.wide_customers_50m_auto through an Iceberg REST catalog;
first-run table creation and additive schema evolution;
eight balanced decode ranges and eight local protection workers;
batches of 8,192 rows;
four Custody requests globally in flight;
a maximum Custody request window of 10,000 values; and
fail-closed protection.

The CPU worker and Custody budgets are separate. Increasing local decode
parallelism cannot multiply load on the shared security service. With the
8,192-row batch inside the 10,000-value request window, the client used four
requests of roughly 2,048 values.

The contract passed snook-run validate across all 100 fields and produced a
stable SHA-256 fingerprint. The artifact reviewed in the UI was the artifact
executed by the runner.

50 Million Rows: Five Workload Profiles Side by Side

Five runs used the same 24.45 GB EBCDIC shape, 50 million rows, 100 fields,
eight decode workers, and eight local protection workers. The first measured
steady-state resolution with every card already enrolled. The second
concentrated 500,000 previously unseen cards at the tail. The third distributed
a fresh 500,000-card set randomly across the file so synchronous assignment and
DR could overlap the full scan. Those runs allowed four Custody requests in
flight. The fourth repeated the randomized shape with 100,000-row decode
batches, 100,000-value Custody windows, and one request in flight. The fifth
used that same bulk contract and randomized file after Custody had been reset
to an empty database and eight valid empty shards. Its one million baseline
cards and 500,000 randomized cards were therefore all new to Custody.

The host was a MacBook Pro with a 12-core Apple M4 Pro, 24 GB RAM, and SSD
storage. It was not a dedicated benchmark server. Snook Custody, API, UI,
Marquez, and destination storage ran on the same machine and shared its CPU,
memory, and SSD. No Spark cluster or JVM participated.

Runtime statistic	Warm	500K tail	500K random/default	500K random/bulk	Empty Custody/bulk
Source read	24.45 GB	24.45 GB	24.45 GB	24.45 GB	24.45 GB
Rows processed	50,000,000	50,000,000	50,000,000	50,000,000	50,000,000
Fields decoded	5.00 billion	5.00 billion	5.00 billion	5.00 billion	5.00 billion
Existing or repeated cards resolved	50,000,000	49,500,000	49,500,000	49,500,000	48,500,000
New mappings minted	0	500,000	500,000	500,000	1,500,000
Custody request shape	default	default	2,048 mostly	100,000 mostly	100,000 mostly
Custody requests	not recorded	not recorded	24,416	504	504
Wall-clock time	283.03 s	603.29 s	559.09 s	395.38 s	241.51 s
Row throughput	176,660/s	82,879/s	89,431/s	126,461/s	207,031/s
Field throughput	17.67M/s	8.29M/s	8.94M/s	12.65M/s	20.70M/s
Protection operations	150,000,000	150,000,000	150,000,000	150,000,000	150,000,000
Protection throughput	about 530,000/s	248,637/s	268,293/s	379,382/s	621,092/s
Source scan rate	82.38 MiB/s	38.65 MiB/s	41.71 MiB/s	58.97 MiB/s	96.55 MiB/s
Peak resident memory	3.4 GB	2.07 GB	1.70 GB	2.02 GB	2.79 GB
Protected output	30.22 GB	30.22 GB	30.22 GB	22.45 GB	22.45 GB
Iceberg result	8 files, 50M rows	8 files, 50M rows	8 files, 50M rows	8 files, 50M rows	8 files, 50M rows

The empty-Custody run took 241.51 seconds. It minted exactly 1,500,000
mappings, resolved the remaining 48,500,000 repeated card occurrences, and
used 504 Custody requests: 496 requests of 100,000 values and eight final
requests of 50,000 values. Primary and standby each held 1,500,000 index rows
and 1,500,000 vault rows after the run, with synchronous replication streaming
at zero lag.

This result does not mean that 50 million tokens were minted: the fixture has
1.5 million distinct card values. It also should not be read as proof that
minting is faster than lookup. The prior bulk run started with roughly 42
million mappings in Custody, while this run started with 64-byte empty shard
files and an empty database. The 153.87-second difference therefore includes a
substantially different lookup and database working set as well as a different
mint/resolve mix.

The steady-state run took 283.03 seconds wall-clock. Marquez measured the
pipeline at 281.002 seconds and recorded it as COMPLETED.

One machine decoded five billion COBOL field values, performed three protection
operations per row, wrote protected Parquet, and committed an Iceberg snapshot
in under five minutes.

The evidence agreed across the steady-state run:

extraction reported 50,000,000 rows and eight balanced outputs totaling 30,218,410,226 bytes;
protection reported 150,000,000 transformed values and no empty protected result;
50,000,000 card values passed through Custody;
100,000,000 SIN and phone values passed through FF1;
the Iceberg snapshot contained eight data files and exactly 50,000,000 rows; and
Marquez recorded the job as completed in 4 minutes 41 seconds.

Why Custody Did Not Become the Bottleneck

Custody separates high-volume token resolution from encrypted PAN recovery. A
keyed HMAC locates a mapping; the mapping returns a random token. The lookup
side does not hold the capability needed to recover card numbers, while the
vault side seals PAN storage with AES-GCM-SIV.

For steady-state ingestion, Custody hashes and deduplicates each batch,
partitions and sorts lookup keys, then merge-scans compact memory-mapped shards
together with an in-memory delta of newer assignments. An existing card does
not require a point query to PostgreSQL or a read from encrypted PAN storage.

Custody statistic	Measured result
Card values sent	50,000,000
Existing mappings resolved	50,000,000
New mappings minted	0
Integrated resolution rate	176,660 values/second
Concurrency budget	4 requests globally across all decode workers
Assignment DR policy	Built-in synchronous replication gate

This distinction matters. The one million unique cards had already been
enrolled, so the run measured integrated steady-state resolution, not new-card
enrollment and not an isolated Custody ceiling.

New assignment has different work and semantics. Custody generates a random
token, seals the PAN, enforces uniqueness, and checks its built-in DR gate. An
assignment is not released until synchronous PostgreSQL replication confirms
the mapping on a connected standby. If that guarantee is unavailable, Custody
refuses the assignment rather than weakening durability. Enrollment throughput
depends on new-card ratio, database working set, and storage latency and needs
its own controlled benchmark.

Testing the Enrollment Tail

To measure that different path, a second 50-million-row run kept 49.5 million
card values on the existing one-million-card cycle and replaced the final
500,000 with 500,000 distinct, previously unseen cards. The source remained a
24.45 GB, 100-field EBCDIC file. Decode and local protection retained eight
workers each, while Custody remained capped at four requests globally.

The complete mixed run took 603.29 seconds. It produced eight equal
6.25-million-row Parquet files, and the Iceberg snapshot independently recorded
eight data files, 30,218,394,136 bytes, and exactly 50,000,000 rows.

The counters reconcile across independent evidence. The extraction manifest
reported exactly 500,000 minted and 49,500,000 looked up. Custody's PAN index
and encrypted vault each grew by exactly 500,000 rows. Its standby remained in
synchronous streaming state, and the completed OpenLineage event named the
committed Iceberg table.

This is deliberately harsher than a normal incremental run. New cards were one
percent of all rows, arrived as a concentrated 500,000-card tail, and represented
half as many new identities as the one-million-card steady-state working set.
Production runs will not usually enroll 500,000 distinct cards at once. Their
throughput will sit closer to the steady-state result as the new-card ratio
falls, with the exact position determined by arrival shape and storage latency.

The test also made Custody share the same 24 GB laptop, CPU, SSD, and Docker
memory allowance as ingestion, PostgreSQL primary and standby, Iceberg,
Marquez, API, and UI services. A production deployment places Custody and its
database tier on larger standalone machines, isolating assignment work and
providing more memory, CPU, and storage headroom. This benchmark intentionally
does not convert that headroom into an unmeasured speedup claim; it records the
more constrained shared-host result.

Distributing Enrollment Across the Run

The third fixture used seed 20260822 to place exactly 500,000 fresh cards at
unique positions across all 50 million rows. Each of the eight decode shards
received between 62,217 and 62,813 new cards, close to the ideal 62,500. The
fresh range was disjoint from both previously enrolled ranges, and an exhaustive
scan verified 49.5 million base-cycle cards and 500,000 fresh cards before the
old tail fixture and its Iceberg data were removed.

The randomized run completed in 559.09 seconds, 44.20 seconds or 7.3 percent
faster than concentrated-tail enrollment. It sustained 89,431 rows per second
and 268,293 protection operations per second. The extraction manifest reported
exactly 500,000 minted and 49,500,000 looked up; both Custody tables grew by
500,000 rows; the synchronous standby finished with zero lag; and Iceberg
recorded eight files, 30,218,393,255 bytes, and 50 million rows.

Distribution helped, but not by eliminating assignment cost. It overlapped
enrollment with decode, FPE, lookup, and Parquet work instead of leaving all
500,000 durable assignments after a warm prefix. On this shared laptop that
also made Custody and PostgreSQL contend with ingestion throughout the scan.
The 7.3 percent gain is therefore more informative than the assumption of a
dramatic speedup: arrival shape matters, while synchronous durability and the
shared resource envelope still set the pace.

Using Custody's Bulk Transaction Window

The fourth run kept the same randomized one-percent enrollment shape but
changed the transaction geometry. The default run handed 8,192-row decode
batches to a four-way planner, producing 24,384 requests of 2,048 values plus
32 final requests of 1,924. The bulk run used 100,000-row decode batches, a
100,000-value Custody window, and concurrency one. Custody observed 496
requests of 100,000 values and eight shard-final requests of 50,000.

That reduced request and synchronous transaction count from 24,416 to 504, a
48.4-times reduction. Wall time fell from 559.09 to 395.38 seconds: 163.71
seconds saved, or a 29.3 percent reduction, for the same 49.5 million resolved
and 500,000 durably minted card values. Throughput increased from 89,431 to
126,461 rows per second.

The bulk run reconciled independently. Custody reported exactly 500,000
assignments and 49.5 million shard resolutions; its PAN index and encrypted
vault each grew by 500,000 rows; synchronous standby lag returned to zero; the
extraction manifest recorded 150 million protection operations; and Iceberg
committed eight files with exactly 50 million rows. The completed output was
22,446,990,196 bytes. That smaller Parquet result coincided with the larger
decode batches and resulting row-group layout; it is reported as an observed
pipeline effect, not attributed to the Custody transaction window alone.

This is the stronger production result for a planned bulk load. It does not
replace the default limit: smaller batches and four-way concurrency remain the
conservative posture for a shared Custody service handling live payment
traffic. A dedicated, larger production Custody deployment can select the bulk
budget deliberately rather than allowing an ingestion job to consume shared
capacity by accident.

Market Context Without Benchmark Theatre

Public, reproducible mainframe conversion benchmarks are rare. Cobrix publishes
a useful historical Spark 2.2.1 EBCDIC-to-Parquet test: 30 million fixed-length
records, 167 fields, 1,341 bytes per record, and results from one to 64
one-core executors.

Pipeline	Parallelism	Configured/observed memory	Shape	Time	Rows/second	Input MiB/second
Snook	8 decode workers on one 12-core host	3.4 GB observed peak RSS	50M x 100 fields	283.03 s	176,660	82.38
Cobrix published result	8 one-core Spark executors plus 4-core driver	36 GB configured JVM heap	30M x 167 fields	894 s	33,557	42.91
Cobrix published result	64 one-core Spark executors plus 4-core driver	260 GB configured JVM heap	30M x 167 fields	214 s	140,187	179.28

Snook produced 5.26 times the published eight-executor row rate and 1.92 times
its byte rate. It also exceeded the published 64-executor row rate by 26%, while
Cobrix processed more bytes per second because its records were 2.74 times
wider.

This is scale context, not a current product ranking. Cobrix used different
data, historical software, and unspecified executor hardware. Its benchmark
performed decode and Parquet output. Snook additionally enforced a contract and
schema, executed 150 million protection operations, created eight outputs, and
committed Iceberg on a shared host. A current same-data, same-hardware test is
required to compare decoder performance directly.

The Cobrix memory figures are configured Spark heap, not observed RSS: 4 GB for
the driver plus 4 GB for each executor. They therefore total 36 GB at eight
executors and 260 GB at 64 executors. The published setup does not report peak
resident memory, garbage-collection time, or the additional executor memory
overhead, off-heap buffers, JVM metadata, Python processes, container allowance,
and operating-system memory required by a Spark deployment. Snook's 3.4 GB is
an observed process peak, so the two columns describe different measurements;
the configured heap still makes the difference in runtime architecture and
resource envelope visible.

The defensible conclusion is narrower and more useful: protected ingestion can
operate in the performance range usually associated with a distributed
conversion job even when the source is wide EBCDIC, without requiring raw data
to traverse that distributed system.

What Snook Changes in the Architecture

Snook is not arguing against cloud compute. It changes what the cloud must be
trusted to receive.

The cloud control plane can schedule work and receive:

protected Parquet;
an Iceberg table commit;
manifests containing row counts, schema fingerprints, digests, and protection statistics; and
OpenLineage events describing source-to-table movement and protection modes.

It does not need an inbound route to every source database, application host,
file server, or mainframe dataset. It does not need a staging zone of raw PANs
just because analytics happen elsewhere.

That is the product position:

Snook is the protected ingestion plane: bounded execution beside the
source, open analytical tables at the destination, and evidence across the
boundary.

The strongest alternatives remain strong at their own layer. Spark-based
decoders offer mature distributed ecosystems and broad mainframe edge-case
coverage. Cloud migration suites offer managed operations and extensive
connectors. Enterprise tokenization vendors offer established certifications,
policy consoles, and production references.

Snook's advantage is the combination those categories usually leave the buyer
to assemble: source-aware extraction, protection before landing, random-token
custody, open table delivery, explicit resource budgets, and run evidence under
one executable contract. Mainframe decoding is the hardest example in this
article, not a qualification on that product position.

The Claim We Can Make

We should not claim that 176,660 rows per second is universal Custody throughput,
DB2 UNLOAD throughput, or proof that Snook beats every current Spark deployment.
It is one fully described integrated result.

We can make the more consequential claim:

Snook decoded 50 million wide EBCDIC records, protected three sensitive
values on every row, wrote Parquet, committed Iceberg, and emitted lineage in
under five minutes on one shared 24 GB machine with 3.4 GB peak process RSS.

Three enrollment stress runs then resolved 49.5 million existing card values
and durably minted 500,000 new mappings under synchronous DR. Concentrating the
new cards at the tail took 603.29 seconds; distributing a fresh set with default
requests took 559.09 seconds; and repeating that shape with 100,000-value bulk
transactions took 395.38 seconds. These are evidence of assignment cost,
arrival-shape effects, and transaction geometry, not a claim that a normal
production run receives 500,000 new cards or that Custody would share the
ingestion host in production.

EBCDIC was chosen because competitors and benchmark authors have every reason
to choose an easier source. It is the cat-sized proof. DB2 UNLOAD, live DB2,
other relational databases, and simpler files enter the same protected product
path, while retaining the measurements and conformance evidence appropriate to
their own source type.

The default architecture no longer has to be "move raw data first, govern it
later." Snook brings the governed execution boundary to the data and moves only
what the destination should have received in the first place.

DEV Community: XogtaTech