A row level data quality gate is good at exactly one thing: deciding whether a single value, looked at in isolation, is acceptable. Is the credit limit a non negative number? Is the new IC twelve digits? Is the operation code one of the values the spec allows? These checks are cheap, they run fast, and they catch the genuinely broken records that every ingestion pipeline produces. The problem is that data quality validation built only on row level rules has a blind spot you can drive a truck through: it cannot tell the difference between a number that is wrong and a number that is merely unusual. A contributor file where every total arrears figure is suddenly 90 percent lower than last month will sail through every null check, type check, and range check you have written, because each individual value is, on its own, perfectly valid.
We hit this on a platform modernization for a regulated credit bureau, where dozens of external contributors send monthly files that feed a medallion lakehouse. The records that hurt us were never the malformed ones. Those got rejected loudly at the door. The dangerous records were the well formed ones that quietly contradicted everything we already knew about that account.
Why well formed data is the real risk
Consider the kinds of corruption that a row level rule physically cannot see. A contributor swaps a column mapping during an export and the credit limit field now carries the product limit, so every limit drops by a plausible looking amount. A truncated extract delivers only half a contributor's accounts, so the aggregate arrears for that contributor collapse overnight. A bad join upstream doubles a credit limit. Each of these passes the obvious gates, because in every case the value sitting in the cell is a real number in a sane range.
What makes them detectable is history. The account had a balance last month. The contributor had a known arrears total last cycle. A credit limit that was stable for eleven statements does not jump 40 percent in the twelfth without something behind it. The signal is not in the row, it is in the delta between the row and its own past. So the only place to catch this class of error is a check that has the previous state in hand at the moment it evaluates the current one.
The Hybrid Baseline check
The mechanism we settled on is a Hybrid Baseline check, and the name describes the trick. Before validation runs, the job unions the incoming batch with the latest partition already committed to the Silver Iceberg table for the same contributor. It then applies a Spark LAG() window function over statement_date, partitioned by account, to populate a set of transient prev_ columns (prev_credit_limit, prev_month_in_arrears, and so on) for every row in the batch. After that step, each current row physically carries its own previous month state alongside it, and a trend rule becomes an ordinary row level comparison between two columns on the same row.
The "hybrid" part matters because contributors do not always send one tidy month at a time. A backfill might arrive as January through March in a single file. A naive lookup of "the last record in Silver" would baseline all three months against the same stale December figure. By unioning the batch with history first and then running LAG() over the ordered statement dates, January gets baselined against December, February against January, and March against February, all inside one pass. Multi month bulk loads validate correctly without special casing.
When a contributor is brand new and has no history at all, the join produces empty prev_ columns rather than crashing. The trend rules simply have nothing to compare against and are skipped for those rows, which is the correct behavior: you cannot validate a trend that does not exist yet.
What a trend rule actually looks like
The execution engine is AWS Glue Data Quality, and the rules are expressed in DQDL, the Data Quality Definition Language that GlueDQ evaluates row by row. We do not hand write the DQDL. The job loads rules from a control CSV and generates DQDL dynamically, which keeps the rules in version controlled configuration rather than buried in code.
Once the prev_ columns are attached, the temporal rules read like plain business logic. Limit stability says that credit_limit, product_limit, and instalment_amount must equal their previous month values unless the account status is flagged as an administrative change, which is the one legitimate reason a limit moves. Delinquency control says month_in_arrears cannot swing by more than two months in a single statement cycle, because an account does not realistically jump from current to four months overdue in one step. And a correlation rule catches internal contradictions that history exposes: if amount_in_arrears is greater than zero, then month_in_arrears cannot be zero.
These sit beside the conventional row level categories rather than replacing them. The same job still enforces completeness on mandatory fields, identity rules such as requiring at least one valid form of ID, domain checks that pin enums like the operation code to an allowed set, and financial consistency such as credit_limit equaling product_limit plus temporary_limit. The baseline rules are the layer on top that the other four categories cannot provide on their own.
Contributor specific rule sets
A single global ruleset would be wrong here, because the contributors do not run the same business. An installment lender that does buy now pay later has high frequency, small amount transactions and a delinquency pattern that looks nothing like a traditional bank's. Applying one set of arrears thresholds to both would either be too loose for one or too strict for the other.
So the rules are modular and selected by contributor type. The job takes a contributor_type parameter, loads the matching ruleset from the control CSV, and generates DQDL from that subset. Installment style contributors get rules tuned to smaller, more frequent movements, while the rest get the traditional credit limit and collateral logic. Adding a new contributor profile is a configuration change, not a code change, which is what keeps this maintainable as the number of contributors grows.
Handling a failed baseline: quarantine versus alert
A failed trend check is not automatically a reason to stop the pipeline, and treating every anomaly as fatal would make the system brittle. Rules carry a severity, and severity decides the response.
A critical failure means hard reject. The specific rows that failed are pulled out of the batch, written to a dq_error bucket as row level CSVs partitioned by date and contributor, and never reach Silver. The healthy rows from the same batch still flow through, so one bad account does not block a contributor's entire load. The quarantined records become a work item for the data operations team, who either correct the source and re ingest or apply a manual patch. This is the response you want for a 90 percent arrears collapse: you do not let it into the single source of truth, and you make it loud enough that a human looks at it.
A warning failure means soft log. The row is ingested into Silver, but the violation is recorded in the run summary for later review. This is the right call for movements that are suspicious but not impossible, where blocking the data would cause more harm than letting it through with a flag. Alongside the error CSVs, the job writes a JSON summary with total, passed, and failed counts per run, which feeds straight into pipeline observability so the trend in rejection rates is itself something you watch over time. A contributor whose warning count is creeping up month over month is telling you something before it ever becomes a critical failure.
One detail that keeps the whole thing clean: the prev_ columns and every other piece of validation metadata are transient. They exist only for the duration of the check and are stripped before the passed rows are committed. The Silver table never sees them, so it stays strictly conformant to the target schema while still having been validated against its own history.
Takeaway
Row level checks are necessary and they are not sufficient. They guarantee that each value is well formed, but well formed is not the same as correct, and the gap between those two is where the expensive errors live. A Hybrid Baseline check closes that gap by giving every row its own past to be measured against: union the batch with the historical Iceberg partition, lag the previous month onto each record, validate the trends in GlueDQ with contributor specific rules, and let severity decide between quarantine and alert. The result is a Silver layer that is not just clean but consistent with itself over time, which is the property regulators and downstream analysts actually care about.
If you are building data quality into the medallion lakehouse and want trend aware validation rather than just row level gates, reach out or take a look at how we approach this in our data engineering services.
Top comments (0)