DEV Community: Marcus

Reading Complex XML in SQL Server — Shredding a Thousand Attributes Generically Instead of Looping for Hours

Marcus — Sat, 31 Oct 2026 18:32:48 +0000

A catalog of around 3000 product configurations arrives as XML. Depending on the equipment, a configuration carries different and additional attributes, around a thousand across all variants. In an earlier project, the evaluation of such a catalog ran in application code: a loop over the object types, inside it a loop over the attributes, plus lists in memory that grew with every iteration. The program took several hours. Where exactly the time went can no longer be said in hindsight. What is certain is what the design demanded: code per type, a separate lookup per attribute, and a schema that had to be in your head beforehand. This article shows how to shred XML in SQL Server without knowing the attributes in advance: load it raw, shred it into a path/value table that holds one row per value with its path in the document, read the schema off the data, and let the database generate the views per type. In the measurement of this article that takes not hours but around 15 seconds for a catalog of the same size.

At a glance

Why loops over object types and attributes get so expensive with a thousand attributes, and what the database route does differently: the attributes become data rows instead of code.
How the catalog is loaded into SQL Server as a single XML value and then split into 3000 table rows, one per configuration. From these rows, a query in which no attribute name appears fetches every value together with its path in the document, for example /Drivetrain/Gears with the value 14. The result is 1.5 million such rows of path and value, and the whole route takes around 15 seconds in the measurement instead of hours.
How an inventory built from these rows makes required, optional and variant attributes per type readable, without anyone having known the attributes beforehand.
How the database generates views with more than 600 columns from the inventory itself, and where the limits of the method are: column limits, typing, very large files. What it requires is little: the element of the sub-documents, an upper bound for the depth, and lists that repeat uniformly in at least one configuration.

Prerequisites: Docker and SQL Server 2022 as a container (Developer Edition). The scripts run with sqlcmd, the command-line client that ships with the container. The call is at the start of the section "Load raw". The sample catalog, its generator and the scripts come in four downloads at the end of the article: the XML route and the JSON route, each for SQL Server and for Postgres. Postgres 18.6 serves for the bridge at the end.

The catalog: three types, one drivetrain with two faces

The example is a bicycle configurator with 3000 configurations. There are three object types: road bike, trekking bike and cargo bike. The three types share the groups General, Frame, Brakes and Wheels, and already here they resemble each other more than they differ. What the Drivetrain contains depends on its XML attribute kind. With kind="chain" it holds chainrings, sprockets, rear derailleur, front derailleur, gear ratios and chainline. With kind="hub" it holds gears, gear range, belt, hub type and coaster brake. The same group, but different attributes depending on the variant. The Brakes also carry their design as an XML attribute, type, and not as a child element. Lighting and Suspension are optional groups that a road bike rarely has and a cargo bike almost always. Two groups are lists. Accessories contains zero to four Part elements with name, weight and a standard-equipment flag. Wheels always contains two Wheel elements, one for the front and one for the rear, with rim, tyre and spoke count.

"Attribute" in this article means a business characteristic of a configuration, such as the number of gears or the material of the frame. In the XML such a characteristic is usually an element and only rarely an XML attribute. Where the XML construct is meant, the text says XML attribute explicitly.

To reach the order of a thousand attributes, every configuration additionally carries twelve of twenty parameter groups with 48 values each, every single one of which is present with a probability of eighty percent. These groups are simply called G01 to G20 and are filler. They are there so that the measurements have the right order of magnitude, not because they mean anything. Which type carries which groups is distributed so that the types overlap but do not coincide.

This is what one configuration looks like, shortened:

  1: <Configuration id="C000001" type="cargo_bike">
  2:   <General>
  3:     <Model>CargoBike-214</Model>
  4:     <ModelYear>2019</ModelYear>
  5:     <Weight>13.25</Weight>
  6:     <Price>4576.94</Price>
  7:   </General>
  8:   <Frame>
  9:     <Material>Aluminum</Material>
 10:     <Size>56</Size>
 11:   </Frame>
 12:   <Drivetrain kind="hub">
 13:     <Gears>14</Gears>
 14:     <GearRange>371</GearRange>
 15:     <Belt>yes</Belt>
 16:   </Drivetrain>
 17:   <Brakes type="Disc hydraulic">
 18:     <Rotor_front>160</Rotor_front>
 19:     <Pads>sintered</Pads>
 20:   </Brakes>
 21:   <Wheels>
 22:     <Wheel><Position>front</Position><Rim>584x23</Rim><Tyre>32-622</Tyre><Spokes>24</Spokes></Wheel>
 23:     <Wheel><Position>rear</Position><Rim>559x25</Rim><Tyre>47-622</Tyre><Spokes>36</Spokes></Wheel>
 24:   </Wheels>
 25:   <Lighting>
 26:     <Front_Lumen>70</Front_Lumen>
 27:     <PowerSource>HubDynamo</PowerSource>
 28:   </Lighting>
 29:   <Accessories>
 30:     <Part><Name>BottleCage</Name><Weight>275</Weight><Standard>yes</Standard></Part>
 31:     <Part><Name>Speedometer</Name><Weight>441</Weight><Standard>yes</Standard></Part>
 32:   </Accessories>
 33:   <Parameter>
 34:     <G09><P02>552.04</P02><P04>861.71</P04><P06>B</P06></G09>
 35:   </Parameter>
 36: </Configuration>

The catalog does not come from a real project but from a Python script, the generator in the download. The script rolls the dice for the type, the equipment and the values of every configuration, but starts the random generator with the same seed on every run. That is why every run produces exactly the same catalog, and all numbers in this article can be reproduced with it. The XML is 28 MB in size and contains a good 1.5 million values. The same data exists as JSON Lines, one line per configuration, in the notation produced by the Python package xmltodict: attributes as keys with @, repeated elements as arrays. Each of the two files comes with the generator in a download of its own, XML and JSON separately. The German version of this article uses the same catalog with German element names. The structure and all numbers are identical.

Why loops over variable attributes get expensive

The obvious design in application code looks like this: for every object type there is a class or a mapping table that knows its attributes. For every configuration, the code walks through the attributes of that type, looks up the node in the document, reads the value and stores it in a list. If a value turns up that the list does not know yet, the list grows. With three types and twenty attributes that is manageable. With a thousand attributes, some of which overlap and some of which exist in only one variant, it becomes code that rebuilds the structure of the catalog, line by line. Every new attribute is a code change. Every access to a node is a separate search through the document. And every list that grows with progress is searched a little more slowly with every iteration.

Where exactly the hours came from in that project can no longer be proven, and this article does not claim to know. It only claims that the design multiplies the costs: types times attributes times configurations, plus lists whose size depends on the order of processing. The general argument for why a loop per element loses against a query over the set is made in the article Set-based vs. row by row on another case study. Here it is about the special case where the loop does not even know what it is looping over, because the schema only exists in the data. The programming language is not the problem. A program that walks the tree once and emits every leaf with its path needs no schema and stays fast, and that is exactly what the twin of this article shows in Python. It only gets expensive through a design that searches the document again for every attribute.

The database route turns the task around. It does not know the attributes and does not need to. What it requires is little: the element under which the configurations sit, and an upper bound for the depth of the tree. Added to that is an assumption about lists: it recognises them only when an element name repeats within them in at least one configuration. It loads the document raw, shreds it with generic queries in which no attribute name appears into rows of path and value, and only afterwards does anyone look at which paths exist at all. The thousand attributes are then a thousand different values in one column, not a thousand lines of code. The schema is counted, not known. And the views that a consumer wants to see at the end are built by the database from this count. Architecturally this is ELT: load raw, then transform in SQL. What distinguishes that from classic ETL is explained in ETL vs. ELT.

The following diagram shows the whole route with the tables that come into being in the next sections, and with the numbers from the sample catalog.

Load raw: one row per configuration

All SQL Server scripts of this article run against a Docker container. The catalog and the scripts are copied with docker cp to /tmp in the container, which is where the paths in the scripts point. Every file is run with sqlcmd, which ships with the image: the switch -i names the file, -C accepts the self-signed certificate of the container, and -I turns on QUOTED_IDENTIFIER, which the XML methods require. For the remaining files only the file name changes.

  1: docker run -d --name 147-mssql-xml-en -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD='S3cret!Pass' `
  2:     mcr.microsoft.com/mssql/server:2022-latest
  3: docker exec 147-mssql-xml-en /opt/mssql-tools18/bin/sqlcmd -C -S localhost -U sa -P 'S3cret!Pass' `
  4:     -Q "CREATE DATABASE catalog"
  5: docker cp .\configs.xml 147-mssql-xml-en:/tmp/configs.xml
  6: docker cp .\147001-mssql-xml-load.sql 147-mssql-xml-en:/tmp/147001-mssql-xml-load.sql
  7: docker exec 147-mssql-xml-en /opt/mssql-tools18/bin/sqlcmd -C -I -S localhost -U sa -P 'S3cret!Pass' `
  8:     -d catalog -i /tmp/147001-mssql-xml-load.sql

The first step loads the whole file as a single value of type xml. OPENROWSET in SINGLE_BLOB mode returns the file as varbinary, and CONVERT to xml parses it once into SQL Server's internal representation. Right after that the document is split: nodes() returns one row per Configuration element, value() fetches the two XML attributes as columns, and query('.') stores the fragment as an xml value of its own.

  1: -- --------------------------------------------------------------------------------
  2: -- 147001: Load the catalog and split it into one row per configuration (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- The XML methods require QUOTED_IDENTIFIER ON; sqlcmd starts with OFF (switch -I).
  5: SET QUOTED_IDENTIFIER ON;
  6: GO
  7: IF SCHEMA_ID('stage') IS NULL EXEC ('CREATE SCHEMA [stage]');
  8: GO
  9: -- --------------------------------------------------------------------------------
 10: -- The whole document as a single xml value
 11: -- --------------------------------------------------------------------------------
 12: DROP TABLE IF EXISTS [stage].[raw_xml];
 13: CREATE TABLE [stage].[raw_xml]
 14: (
 15:     doc   xml NOT NULL
 16: );
 17: 
 18: INSERT INTO [stage].[raw_xml] (doc)
 19: SELECT
 20:    CONVERT(xml, T01.BulkColumn)
 21: FROM
 22:    OPENROWSET(BULK '/tmp/configs.xml', SINGLE_BLOB) AS T01;
 23: 
 24: -- --------------------------------------------------------------------------------
 25: -- One row per configuration: key, type and the fragment as xml
 26: -- --------------------------------------------------------------------------------
 27: DROP TABLE IF EXISTS [stage].[config];
 28: SELECT
 29:     T02.n.value('@id',  'varchar(20)') AS config_id
 30:    ,T02.n.value('@type', 'varchar(20)') AS config_type
 31:    ,T02.n.query('.')                   AS node
 32: INTO [stage].[config]
 33: FROM
 34:    [stage].[raw_xml] T01
 35:    CROSS APPLY T01.doc.nodes('/Catalog/Configuration') AS T02(n);
 36: 
 37: SELECT
 38:     config_type
 39:    ,COUNT(*) AS configs
 40: FROM
 41:    [stage].[config]
 42: GROUP BY
 43:    config_type;
 44: -- cargo_bike      1025
 45: -- road_bike        995
 46: -- trekking_bike    980

The split is the most important step of the whole method, for a reason that only becomes visible later. All subsequent XQuery expressions then run against fragments of around 9 KB instead of a document of 28 MB. What that is worth is shown more drastically by Postgres than by SQL Server: there, the same generic query over the whole document was still running after 36 minutes and could not even be cancelled. More on that in the Postgres bridge section below.

One trap right at the start: the XML methods nodes(), value() and query() require QUOTED_IDENTIFIER ON, alongside further session options such as ANSI_NULLS, ANSI_PADDING and ANSI_WARNINGS, which are already set in sqlcmd. The tool sqlcmd, however, starts with QUOTED_IDENTIFIER OFF, and the error message does name the option, but does not say which construct in the statement needs it. Instead it lists all the features that depend on it. The switch -I on the sqlcmd call takes care of that, alternatively the first line in the script.

Shred generically: every leaf with its path

The goal is a table with four columns: configuration, type, path and value. In the SQL Server world this breaking of a document into rows is called XML shredding, and the tools for it are the methods nodes() and value() of the xml type. The values sit in the leaves of the tree, that is, in elements without child elements, and additionally in XML attributes. An XML attribute can hang on any element, including one with children, like kind on the Drivetrain. The path is the chain of element names from the Configuration element down to the leaf. For an XML attribute it ends with @ and the attribute name. From the excerpt above this yields rows such as /Frame/Material with the value Aluminum, /Drivetrain/Gears with the value 14 or /Drivetrain/@kind with the value hub.

This happens in two passes over all 3000 fragments, called pass 1 and pass 2 in what follows. The reason for the split are the lists. A configuration with two accessory parts has two elements with the same path /Accessories/Part/Name. To tell them apart, every element would have to carry its position among its same-named siblings, that is, whether it is the first or the second part. This position is called ordinal in what follows and appears in square brackets in the path, for example /Accessories/Part[2]/Name. Computing it is expensive in SQL Server: for each of the 1.5 million leaves the XQuery engine would have to count the siblings before it, and that raises the runtime in the measurement from around two to around 40 seconds. Pass 1 therefore fetches all values with their path, but without positions. From its result, an ordinary grouping reveals at which positions elements repeat at all. In the catalog these are Part under Accessories and Wheel under Wheels. Pass 2 then computes the positions only for these positions, that is, for around 12,000 elements instead of 1.5 million. XML attributes are not elements and are skipped by both passes. They get a short pass of their own.

Pass 1: all leaves, no positions

The path expression /Configuration//*[not(*)] matches every element in the fragment that has no child element, at any depth. The names of its ancestors come from the parent axis: local-name(..) is the name of the parent element, local-name(../..) that of the grandparent, and so on. In XQuery, SQL Server supports the axes child, descendant, descendant-or-self, parent, attribute and self, but not ancestor. That is why the depth has to be chosen as a fixed number. The catalog has at most three levels below the configuration, so four name columns suffice with one column in reserve: the fourth always holds Configuration. That is exactly what can be checked. A leaf for which Configuration appears in none of the ancestor columns lies deeper than the reserve, and from the fifth level on its path would be cut off at the front without any error message pointing it out. Whoever has deeper documents adds columns.

  1: -- --------------------------------------------------------------------------------
  2: -- 147002: Pass 1 - every leaf element with the names of its ancestors (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- One pass over all configurations. The parent axis (..) yields the ancestor
  5: -- names; positions are deliberately NOT computed here.
  6: SET QUOTED_IDENTIFIER ON;
  7: GO
  8: DROP TABLE IF EXISTS [stage].[leaf_raw];
  9: SELECT
 10:     T01.config_id
 11:    ,T01.config_type
 12:    ,T02.n.value('local-name(.)',        'varchar(100)') AS name0   -- the leaf itself
 13:    ,T02.n.value('local-name(..)',       'varchar(100)') AS name1   -- parent
 14:    ,T02.n.value('local-name(../..)',    'varchar(100)') AS name2   -- grandparent
 15:    ,T02.n.value('local-name(../../..)', 'varchar(100)') AS name3   -- great-grandparent
 16:    ,T02.n.value('.',                    'varchar(200)') AS value
 17: INTO [stage].[leaf_raw]
 18: FROM
 19:    [stage].[config] T01
 20:    CROSS APPLY T01.node.nodes('/Configuration//*[not(*)]') AS T02(n);
 21: 
 22: SELECT
 23:    COUNT(*) AS leaves
 24: FROM
 25:    [stage].[leaf_raw];
 26: -- 1516476

This single pass delivers all 1,516,476 leaf elements of the 3000 configurations in around two seconds. Lines 12 to 15 fetch the four names, line 16 the value, and that is all. What is deliberately missing is the position of an element among its siblings. It would be needed to tell the two Part elements in the accessories apart. The obvious XQuery expression for it is let $i := . return count(../*[. << $i]) + 1: count the siblings that precede me in the document. With two such expressions, one for the leaf and one for its parent element, the same query takes twenty times as long instead of around two seconds, measured 1.8 against 40.2 seconds. For each of the 1.5 million leaves SQL Server then walks the sibling list and compares document positions, and exactly this evaluation is expensive in the XQuery engine. Computing the position across the board is therefore not an option. It is computed in the second pass only where it is needed.

XML attributes

XML attributes are not elements and do not appear in pass 1, regardless of whether they hang on a leaf or on an element with children. They get a short pass of their own. In SQL Server 2022 the method nodes() also returns attribute nodes, the path /Configuration//@* matches all XML attributes in the fragment, and the parent axis leads from the XML attribute to the element that carries it. The two XML attributes of the Configuration element itself are already columns and are excluded.

  1: -- --------------------------------------------------------------------------------
  2: -- 147003: Attributes - nodes() also returns attribute nodes (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- The attributes of the configuration itself (id, type) are already columns in
  5: -- [stage].[config] and are excluded here.
  6: SET QUOTED_IDENTIFIER ON;
  7: GO
  8: DROP TABLE IF EXISTS [stage].[leaf_attr];
  9: SELECT
 10:     T01.config_id
 11:    ,T01.config_type
 12:    ,T02.n.value('local-name(.)',        'varchar(100)') AS attr_name
 13:    ,T02.n.value('local-name(..)',       'varchar(100)') AS name1   -- element carrying the attribute
 14:    ,T02.n.value('local-name(../..)',    'varchar(100)') AS name2
 15:    ,T02.n.value('local-name(../../..)', 'varchar(100)') AS name3
 16:    ,T02.n.value('.',                    'varchar(200)') AS value
 17: INTO [stage].[leaf_attr]
 18: FROM
 19:    [stage].[config] T01
 20:    CROSS APPLY T01.node.nodes('/Configuration//@*') AS T02(n)
 21: WHERE
 22:    T02.n.value('local-name(..)', 'varchar(100)') <> 'Configuration';
 23: 
 24: SELECT
 25:     name1
 26:    ,attr_name
 27:    ,COUNT(*) AS n
 28: FROM
 29:    [stage].[leaf_attr]
 30: GROUP BY
 31:     name1
 32:    ,attr_name
 33: ORDER BY
 34:     name1
 35:    ,attr_name;
 36: -- Brakes       type   3000
 37: -- Drivetrain   kind   3000

The catalog has two further XML attributes, kind on the drivetrain and type on the brakes, and the pass takes half a second.

Pass 2: ordinals only where something repeats

Without positions, the two Part elements of a configuration fall onto the same path /Accessories/Part/Name. For the inventory in the next section that is actually fine, for a view with one column per path it is not. The solution has two steps. First, a set-based query reads off pass 1 which parent elements repeat at all: if, within one configuration, more than one leaf with the same name sits under the same parent path, then either the parent element repeats, or the leaf itself repeats within the parent element. Both need ordinals. Which of the two cases applies, the grouping does not reveal, because in pass 1 both look the same. The template below solves the first one. The second one is shown by the inventory after pass 2: the path then carries more than one value in a single configuration.

  1: -- --------------------------------------------------------------------------------
  2: -- 147004: Which parents repeat? Set-based, from pass 1 (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- A parent element repeats when more than one leaf of the same name sits under
  5: -- the same parent path within one configuration. The same query also finds
  6: -- leaves that repeat within a single parent.
  7: DROP TABLE IF EXISTS [stage].[repeated_parent];
  8: WITH
  9: CTE_leaf_count AS
 10: (
 11:    SELECT
 12:        config_id
 13:       ,name3
 14:       ,name2
 15:       ,name1
 16:       ,name0
 17:       ,COUNT(*) AS n
 18:    FROM
 19:       [stage].[leaf_raw]
 20:    GROUP BY
 21:        config_id
 22:       ,name3
 23:       ,name2
 24:       ,name1
 25:       ,name0
 26: )
 27: SELECT DISTINCT
 28:     CASE WHEN name2 = 'Configuration' THEN ''
 29:          WHEN name3 = 'Configuration' THEN '/' + name2
 30:          ELSE '/' + name3 + '/' + name2 END AS parent_path
 31:    ,name1 AS parent_name
 32: INTO [stage].[repeated_parent]
 33: FROM
 34:    CTE_leaf_count
 35: WHERE
 36:    n > 1;
 37: 
 38: SELECT
 39:     parent_path
 40:    ,parent_name
 41: FROM
 42:    [stage].[repeated_parent]
 43: ORDER BY
 44:    parent_path;
 45: -- /Accessories   Part
 46: -- /Wheels        Wheel

The query finds two positions, /Accessories with the child Part and /Wheels with the child Wheel, and takes under a second for that. Only now does the expensive position computation come into play, and only for the nodes that this detection has delivered. For every position found, a template yields a SELECT that fetches the affected parent elements with nodes(), computes their position among their same-named siblings and reads the leaves below them with their own position. At such a position every element gets its ordinal, even when a configuration has only a single element there. A configuration with three parts gets /Accessories/Part[1]/Name to /Accessories/Part[3]/Name, one with exactly one part gets /Accessories/Part[1]/Name. If the single part stayed without an ordinal, the first element of the same list would sit on two different paths depending on the configuration. The inventory would then count one list as two things, and the path without ordinal would end up as a column in the type's view, filled only for the configurations with exactly one part. The two wheels of every configuration are accordingly called /Wheels/Wheel[1]/Rim and /Wheels/Wheel[2]/Rim.

  1: -- --------------------------------------------------------------------------------
  2: -- 147005: Pass 2 - ordinals only for the parents that repeat (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- For every row of [stage].[repeated_parent] the template yields one SELECT.
  5: -- At such a position every element gets its ordinal, even when a configuration
  6: -- has only a single element there: a list stays a list, and its first element
  7: -- sits on the same path in every configuration.
  8: -- The expensive position expression runs only over the few affected nodes,
  9: -- not over all leaves.
 10: -- If leaves repeat directly within one parent element, the template gets the
 11: -- same expression once more on leaf level (T03). That costs a multiple of the
 12: -- runtime of pass 2 and is not needed here. Whether the case exists is shown
 13: -- by the inventory: n_values greater than n_configs.
 14: SET QUOTED_IDENTIFIER ON;
 15: GO
 16: DECLARE @template nvarchar(max) = N'
 17: SELECT
 18:     T01.config_id
 19:    ,T01.config_type
 20:    ,''{path}/{name}''
 21:     + T02.n.value(''let $p := .
 22:                     return concat("[", string(count(../*[local-name() = "{name}"][. << $p]) + 1), "]")'', ''varchar(10)'')
 23:     + ''/'' + T03.n.value(''local-name(.)'', ''varchar(100)'') AS path
 24:    ,T03.n.value(''.'', ''varchar(200)'') AS value
 25: FROM
 26:    [stage].[config] T01
 27:    CROSS APPLY T01.node.nodes(''/Configuration{path}/{name}'') AS T02(n)
 28:    CROSS APPLY T02.n.nodes(''./*[not(*)]'') AS T03(n)';
 29: 
 30: DECLARE @sql nvarchar(max);
 31: 
 32: SELECT
 33:    @sql = STRING_AGG(
 34:              CAST(REPLACE(REPLACE(@template, N'{path}', parent_path), N'{name}', parent_name) AS nvarchar(max))
 35:             ,N' UNION ALL ')
 36: FROM
 37:    [stage].[repeated_parent];
 38: 
 39: DROP TABLE IF EXISTS [stage].[leaf_pass2];
 40: CREATE TABLE [stage].[leaf_pass2]
 41: (
 42:     config_id     varchar(20)  NOT NULL
 43:    ,config_type   varchar(20)  NOT NULL
 44:    ,path          varchar(400) NOT NULL
 45:    ,value         varchar(200) NULL
 46: );
 47: 
 48: INSERT INTO [stage].[leaf_pass2]
 49: EXEC sp_executesql @sql;
 50: 
 51: SELECT
 52:     path
 53:    ,COUNT(*) AS n
 54: FROM
 55:    [stage].[leaf_pass2]
 56: WHERE
 57:       path LIKE '%/Name'
 58:    OR path LIKE '%/Rim'
 59: GROUP BY
 60:    path
 61: ORDER BY
 62:    path;
 63: -- /Accessories/Part[1]/Name   2406
 64: -- /Accessories/Part[2]/Name   1831
 65: -- /Accessories/Part[3]/Name   1243
 66: -- /Accessories/Part[4]/Name    615
 67: -- /Wheels/Wheel[1]/Rim        3000
 68: -- /Wheels/Wheel[2]/Rim        3000

Pass 2 runs over 6095 Part elements and 6000 Wheel elements instead of 1.5 million leaves and takes under a second. The template applies at every position the detection reports, no matter how deep the repeated element sits in the fragment. Below it, however, it reads only the direct leaf children, the way Name, Weight and Standard sit directly under Part. If leaves repeat directly within one parent element, the template gets the same expression once more on leaf level. That costs around 15 seconds instead of one in the measurement, because navigating to the parent element from every leaf is expensive, but it stays limited to the positions found. Two forms the template does not cover. If there is a further level between the repeated element and its leaves, say Part/Product/Name, then the detection attributes the repetition to the intermediate element Product, and pass 2 assigns every Product the ordinal 1, because every Part has only one Product. The two names then land on the same path /Accessories/Part/Product[1]/Name. The same applies to lists within lists. Both cases need a further CROSS APPLY stage with the same pattern, and whether they occur is revealed by the inventory in the next section through a single metric. A third form escapes the detection itself, because it is a heuristic over the leaf names: it finds a list only when at least one leaf name repeats within it. If of two Part elements one carries only a Name and the other only a Weight, no leaf name repeats, and the two parts look like a single one in the path/value table. The inventory's metric does not show this case. It stays hidden, however, only as long as not a single configuration carries the list uniformly. As soon as one does, the detection reports the position, and pass 2 assigns the ordinals in all configurations.

One shortcut looks tempting and is not one. ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) over the output of nodes() delivers the document order in practice, and in the measurement the ordinals derived from it matched the XQuery computation in all 12,095 cases. The documentation, however, does not guarantee this order, and the window function forces the optimizer into a plan that raises the runtime of pass 1 from around two to around 40 seconds. Both together are enough to leave the shortcut alone.

Merge

The path/value table comes from three sources: pass 1 without the leaves under the repeated parent elements, plus pass 2 and the XML attributes. The path is assembled from the name columns, up to the name Configuration, which marks the beginning. An index on type and path helps all the queries that now follow. At the end of the script stands a reconciliation: the number of leaves and XML attributes in the fragments must match the number of rows in the path/value table. This reconciliation only counts. Whether every value sits on the right path, it does not show.

  1: -- --------------------------------------------------------------------------------
  2: -- 147006: Merge - the path/value table (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- Pass 1 without the leaves under repeated parents, plus pass 2 and the
  5: -- attributes. Result: one row per leaf value with its path.
  6: -- The reconciliation at the end counts with XML methods and needs QUOTED_IDENTIFIER ON.
  7: SET QUOTED_IDENTIFIER ON;
  8: GO
  9: DROP TABLE IF EXISTS [stage].[leaf];
 10: SELECT
 11:     T01.config_id
 12:    ,T01.config_type
 13:    ,CASE WHEN T01.name1 = 'Configuration' THEN ''
 14:          WHEN T01.name2 = 'Configuration' THEN '/' + T01.name1
 15:          WHEN T01.name3 = 'Configuration' THEN '/' + T01.name2 + '/' + T01.name1
 16:          ELSE '/' + T01.name3 + '/' + T01.name2 + '/' + T01.name1 END
 17:     + '/' + T01.name0 AS path
 18:    ,T01.value
 19: INTO [stage].[leaf]
 20: FROM
 21:    [stage].[leaf_raw] T01
 22: WHERE
 23:    NOT EXISTS
 24:    (
 25:       SELECT
 26:          1
 27:       FROM
 28:          [stage].[repeated_parent] T02
 29:       WHERE
 30:              T02.parent_name = T01.name1
 31:          AND T02.parent_path = CASE WHEN T01.name2 = 'Configuration' THEN ''
 32:                                     WHEN T01.name3 = 'Configuration' THEN '/' + T01.name2
 33:                                     ELSE '/' + T01.name3 + '/' + T01.name2 END
 34:    )
 35: UNION ALL
 36: SELECT
 37:     config_id
 38:    ,config_type
 39:    ,path
 40:    ,value
 41: FROM
 42:    [stage].[leaf_pass2]
 43: UNION ALL
 44: SELECT
 45:     config_id
 46:    ,config_type
 47:    ,CASE WHEN name2 = 'Configuration' THEN ''
 48:          WHEN name3 = 'Configuration' THEN '/' + name2
 49:          ELSE '/' + name3 + '/' + name2 END
 50:     + '/' + name1 + '/@' + attr_name
 51:    ,value
 52: FROM
 53:    [stage].[leaf_attr];
 54: 
 55: CREATE INDEX ix_leaf_type_path ON [stage].[leaf] (config_type, path);
 56: 
 57: SELECT
 58:     COUNT(*)                  AS leaves
 59:    ,COUNT(DISTINCT config_id) AS configs
 60:    ,COUNT(DISTINCT path)      AS paths
 61: FROM
 62:    [stage].[leaf];
 63: -- 1522476   3000   1024
 64: 
 65: -- --------------------------------------------------------------------------------
 66: -- Reconciliation: leaves and attributes in the source against the rows of the table
 67: -- --------------------------------------------------------------------------------
 68: -- The attributes of the configuration itself (id, type) do not count. If the sum
 69: -- of the first two columns differs from the third, something was lost or
 70: -- duplicated while merging.
 71: SELECT
 72:     SUM(T01.node.value('count(/Configuration//*[not(*)])', 'int')) AS source_leaves
 73:    ,SUM(T01.node.value('count(/Configuration/*//@*)',      'int')) AS source_attributes
 74:    ,(SELECT COUNT(*) FROM [stage].[leaf])                           AS leaf_rows
 75: FROM
 76:    [stage].[config] T01;
 77: -- 1516476   6000   1522476
 78: 
 79: SELECT
 80:     path
 81:    ,value
 82: FROM
 83:    [stage].[leaf]
 84: WHERE
 85:        config_id = 'C000001'
 86:    AND path LIKE '/Drivetrain%'
 87: ORDER BY
 88:    path;
 89: -- /Drivetrain/@kind          hub
 90: -- /Drivetrain/Belt           yes
 91: -- /Drivetrain/CoasterBrake   no
 92: -- /Drivetrain/GearRange      371
 93: -- /Drivetrain/Gears          14
 94: -- /Drivetrain/HubType        N8

The result is 1,522,476 rows from 3000 configurations with 1024 distinct paths. The reconciliation adds up: 1,516,476 leaves and 6000 XML attributes in the source give the same number. The assignment of value to path was checked separately for this article, against a single pass that computes the position for each of the 1.5 million leaves. A comparison with EXCEPT over configuration, path and value yields zero differences in both directions. The ordinals of the parts turn three paths into twelve, those of the wheels four paths into eight, the rest are the 1004 paths of the catalog including the XML attributes /Drivetrain/@kind and /Brakes/@type. For the configuration from the excerpt, the table delivers under /Drivetrain exactly the six rows one would expect: the XML attribute and the five hub attributes.

In total, the XML route in SQL Server looks like this. The runtime per script was measured in a Docker container under Docker Desktop on a current desktop machine: Intel Core i7-14700KF with 28 logical cores, 62 GB RAM and an NVMe SSD under Ubuntu, of which 28 cores and 8 GB go to the Docker VM, the SQL Server container limited to 4 GB of memory, SQL Server 2022 build 16.0.4265.3 without further tuning, in the second run in the same container. The numbers are orders of magnitude on this one system, not benchmarks. For perspective: a three-year-old laptop with an Intel Core i7-1165G7 needed, in an earlier measurement with a smaller version of the catalog, around three times as long for the whole route and around ten times as long for the XQuery steps pass 1 and pass 2.

Step	Time
Load as `SINGLE_BLOB` and split into 3000 configurations	8 s
Pass 1: all leaves with ancestor names	1.8 s
XML attributes	0.5 s
Detection of repeated parent elements	0.7 s
Pass 2: ordinals for the repetitions	0.9 s
Merge, index and reconciliation	2.4 s
total	around 15 s

The same with JSON and OPENJSON

If the catalog arrives as JSON, there is no detour through pass 1 and pass 2, because an array brings its ordinals with it. The file is loaded as SINGLE_CLOB and split with STRING_SPLIT at the line breaks into one row per configuration. That STRING_SPLIT guarantees no order does not matter here, because every line is a self-contained configuration. After that, a recursive CTE opens every node with OPENJSON: the anchor is the document, the recursive part opens everything that is an object or an array and appends the key or the index to the path. Whatever is neither object nor array is a leaf.

  1: -- --------------------------------------------------------------------------------
  2: -- 147007: The same catalog as JSON Lines - recursively with OPENJSON (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- One row per configuration; attributes follow the xmltodict convention (@kind),
  5: -- repeated elements are arrays. Array indexes are counted from 1, as with XML.
  6: DROP TABLE IF EXISTS [stage].[raw_json];
  7: SELECT
  8:    CAST(T02.value AS nvarchar(max)) AS doc
  9: INTO [stage].[raw_json]
 10: FROM
 11:    OPENROWSET(BULK '/tmp/configs.jsonl', SINGLE_CLOB) AS T01
 12:    CROSS APPLY STRING_SPLIT(CAST(T01.BulkColumn AS nvarchar(max)), CHAR(10)) AS T02
 13: WHERE
 14:    LEN(T02.value) > 2;
 15: 
 16: -- --------------------------------------------------------------------------------
 17: -- Open recursively: objects (type 5) and arrays (type 4) are shredded further,
 18: -- everything else is a leaf. OPENJSON returns key/value in Latin1_General_BIN2,
 19: -- hence COLLATE DATABASE_DEFAULT on both sides of the recursion.
 20: -- --------------------------------------------------------------------------------
 21: DROP TABLE IF EXISTS [stage].[leaf_json];
 22: WITH
 23: CTE_node AS
 24: (
 25:    SELECT
 26:        CAST(JSON_VALUE(doc, '$."@id"')  AS nvarchar(20))  COLLATE DATABASE_DEFAULT AS config_id
 27:       ,CAST(JSON_VALUE(doc, '$."@type"') AS nvarchar(20))  COLLATE DATABASE_DEFAULT AS config_type
 28:       ,CAST(doc AS nvarchar(max))                         COLLATE DATABASE_DEFAULT AS node
 29:       ,CAST(N'' AS nvarchar(400))                         COLLATE DATABASE_DEFAULT AS path
 30:       ,CAST(5 AS int)                                                              AS json_type
 31:    FROM
 32:       [stage].[raw_json]
 33:    UNION ALL
 34:    SELECT
 35:        CAST(T01.config_id   AS nvarchar(20))  COLLATE DATABASE_DEFAULT
 36:       ,CAST(T01.config_type AS nvarchar(20))  COLLATE DATABASE_DEFAULT
 37:       ,CAST(T02.value       AS nvarchar(max)) COLLATE DATABASE_DEFAULT
 38:       ,CAST(T01.path
 39:             + CASE WHEN T01.json_type = 4
 40:                    THEN N'[' + CAST(CAST(T02.[key] AS int) + 1 AS nvarchar(10)) + N']'
 41:                    ELSE N'/' + T02.[key] END
 42:             AS nvarchar(400)) COLLATE DATABASE_DEFAULT
 43:       ,CAST(T02.type AS int)
 44:    FROM
 45:       CTE_node T01
 46:       CROSS APPLY OPENJSON(CASE WHEN T01.json_type IN (4, 5) THEN T01.node ELSE N'[]' END) AS T02
 47:    WHERE
 48:       T01.json_type IN (4, 5)
 49: )
 50: SELECT
 51:     config_id
 52:    ,config_type
 53:    ,CAST(path AS varchar(400)) AS path
 54:    ,CAST(node AS varchar(200)) AS value
 55: INTO [stage].[leaf_json]
 56: FROM
 57:    CTE_node
 58: WHERE
 59:    json_type NOT IN (4, 5);
 60: 
 61: SELECT
 62:     COUNT(*)             AS leaves
 63:    ,COUNT(DISTINCT path) AS paths
 64: FROM
 65:    [stage].[leaf_json];
 66: -- 1528476   1026

Two details make the difference between a query that runs and one that does not even compile. First, OPENJSON returns the columns key and value in the collation Latin1_General_BIN2, while the columns of the anchor carry the collation of the database. In a recursive CTE, anchor and recursive part must have identical types, collation included, otherwise SQL Server reports "Types don't match between the anchor and the recursive part". The COLLATE DATABASE_DEFAULT on both sides is therefore mandatory. Second, OPENJSON counts array indexes from zero, XML ordinals count from one. The + 1 in the path aligns the two, so that both routes deliver the same paths.

The JSON route needs around 25 seconds for loading and recursion together. It delivers 1,528,476 rows with 1026 paths, that is, 6000 rows and two paths more than the XML route. The difference are @id and @type, which JSON carries as paths of their own, while they are columns on the XML route. All 1,522,476 rows of the XML route reappear with configuration, path and value in the JSON result, the ordinals of the lists included. The code, a single script, is much shorter than the five scripts of the XML route from pass 1 to the merge, but at around 25 seconds the runtime is above the around 15 seconds of the XML route. Whoever has the choice, because the catalog is converted anyway, can skip the XML shredding. What the conversion looks like in Python is shown by the twin of this article, which follows shortly.

Inventory: reading the schema off the data

With the path/value table, the actual question of the project becomes a grouping: which type carries which paths, and how often? Per type and path, the values are counted, as are the configurations in which the path occurs. The share of all configurations of the type then says what one is dealing with.

  1: -- --------------------------------------------------------------------------------
  2: -- 147009: Inventory - which type carries which paths, and how often (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: DROP TABLE IF EXISTS [stage].[inventory];
  5: WITH
  6: CTE_configs AS
  7: (
  8:    SELECT
  9:        config_type
 10:       ,COUNT(DISTINCT config_id) AS configs
 11:    FROM
 12:       [stage].[leaf]
 13:    GROUP BY
 14:       config_type
 15: )
 16: SELECT
 17:     T01.config_type
 18:    ,T01.path
 19:    ,COUNT(*)                                                                         AS n_values
 20:    ,COUNT(DISTINCT T01.config_id)                                                    AS n_configs
 21:    ,CAST(100.0 * COUNT(DISTINCT T01.config_id) / MAX(T02.configs) AS decimal(5, 1)) AS share_pct
 22: INTO [stage].[inventory]
 23: FROM
 24:    [stage].[leaf] T01
 25:    INNER JOIN CTE_configs T02
 26:    ON
 27:      T02.config_type = T01.config_type
 28: GROUP BY
 29:     T01.config_type
 30:    ,T01.path;
 31: 
 32: -- --------------------------------------------------------------------------------
 33: -- Required or optional: from a share of 99 percent a path counts as required for
 34: -- the type, everything below is optional or belongs to a variant.
 35: -- The threshold is deliberately not 100 percent. In real data a required value is
 36: -- missing in a few configurations, and the path should stay required so that
 37: -- exactly these gaps show up. For strict counting use 100.0.
 38: -- --------------------------------------------------------------------------------
 39: SELECT
 40:     config_type
 41:    ,COUNT(*)                                           AS paths
 42:    ,SUM(CASE WHEN share_pct >= 99.0 THEN 1 ELSE 0 END) AS required_paths
 43:    ,SUM(CASE WHEN share_pct <  99.0 THEN 1 ELSE 0 END) AS optional_paths
 44: FROM
 45:    [stage].[inventory]
 46: GROUP BY
 47:    config_type
 48: ORDER BY
 49:    config_type;
 50: -- cargo_bike      640   34   606
 51: -- road_bike       634   29   605
 52: -- trekking_bike   640   29   611
 53: 
 54: SELECT
 55:     path
 56:    ,n_configs
 57:    ,share_pct
 58: FROM
 59:    [stage].[inventory]
 60: WHERE
 61:        config_type = 'trekking_bike'
 62:    AND path LIKE '/Drivetrain/%'
 63: ORDER BY
 64:    path;
 65: -- /Drivetrain/@kind             980   100.0
 66: -- /Drivetrain/Belt              367    37.4
 67: -- /Drivetrain/Chainline         613    62.6
 68: -- /Drivetrain/Chainrings        613    62.6
 69: -- /Drivetrain/CoasterBrake      367    37.4
 70: -- /Drivetrain/FrontDerailleur   613    62.6
 71: -- /Drivetrain/GearRange         367    37.4
 72: -- /Drivetrain/GearRatio_max     613    62.6
 73: -- /Drivetrain/GearRatio_min     613    62.6
 74: -- /Drivetrain/Gears             367    37.4
 75: -- /Drivetrain/HubType           367    37.4
 76: -- /Drivetrain/RearDerailleur    613    62.6
 77: -- /Drivetrain/Sprockets         613    62.6

A path with a share close to one hundred percent is a required attribute of the type. The query draws the line at 99 percent and not at 100, so that a required attribute missing in a few configurations stays required and exactly these gaps show up. The classification is a finding from the data and not a schema statement. Whether an attribute is required in the business sense is decided by the business side, and an element that is present but empty counts as present here. A path with a small share is optional. And a path that occurs in a fixed fraction of the configurations belongs to a variant. The drivetrain of the trekking bike shows this without any prior knowledge: /Drivetrain/@kind appears in all 980 configurations, the chain attributes appear in 62.6 percent of them, the hub attributes in the remaining 37.4 percent. The two variants of the drivetrain can be read off the count, and so can the fact that they exclude each other. The inventory takes under half a second.

The same count is, as a side effect, a data quality tool. A path that occurs in only three out of a thousand configurations is either a rare extra or a typo in the element name. A required attribute at 99.8 percent has two configurations in which it is missing. And a path that counts more values than configurations is either a repetition that pass 2 did not resolve, or a pair of element names in the same configuration that differ only in upper and lower case and collapse under the database's default collation. This metric, n_values greater than n_configs, is a plausibility check for paths that are assigned more than once within a configuration. In the catalog it applies to no path. Whether the shredding was complete, it does not answer. That is what the reconciliation of the row counts at the end of the merge is for. Whoever wants to derive check rules from such metadata finds the idea continued in Deriving data quality rules from the schema, there for relational schemas.

Generate views instead of writing them

Nobody writes a view with six hundred columns by hand, and nobody should have to. The inventory knows every path per type, and every path becomes a column: MAX(CASE WHEN path = '…' THEN value END) grouped by configuration. STRING_AGG assembles the column list, QUOTENAME turns the path into a valid column name and the type into a valid view name, and sp_executesql runs the DDL. Because paths and type come from the data, they pass through QUOTENAME as identifiers and through a REPLACE that doubles single quotes as literals. With XML names that is mere caution, because they cannot contain single quotes. With keys from JSON it is necessary. List paths with an ordinal stay out. They get views of their own with one row per list element, otherwise columns like accessories_part_3_name would arise that stay empty for anything but four parts.

  1: -- --------------------------------------------------------------------------------
  2: -- 147010: Generate the views from the inventory instead of writing them (SQL Server)
  3: -- --------------------------------------------------------------------------------
  4: -- One view per object type with one column per path. List paths (with [n]) stay
  5: -- out; they get views of their own with one row per list element.
  6: -- Run the script once per object type: first as it stands with 'trekking_bike',
  7: -- then with 'road_bike' and with 'cargo_bike' in @config_type. Every run creates
  8: -- the view of its type (vw_trekking_bike, vw_road_bike, vw_cargo_bike).
  9: -- CREATE VIEW must be alone in its batch, hence the DDL runs through
 10: -- sp_executesql. The typed view and the sample query at the end belong to the
 11: -- trekking bike and assume that its run was the first.
 12: -- Identifiers pass through QUOTENAME, literals through REPLACE('''') - paths from
 13: -- JSON, unlike XML names, may contain single quotes.
 14: DECLARE @config_type varchar(20) = 'trekking_bike';   -- then 'road_bike', then 'cargo_bike'
 15: DECLARE @sql         nvarchar(max);
 16: 
 17: SELECT
 18:    @sql = N'CREATE OR ALTER VIEW [dbo].' + QUOTENAME(N'vw_' + @config_type) + N' AS'
 19:         + N' SELECT config_id'
 20:         + STRING_AGG(
 21:              CAST(N', MAX(CASE WHEN path = ''' + REPLACE(path, '''', '''''') + N''' THEN value END) AS '
 22:                   + QUOTENAME(LOWER(REPLACE(REPLACE(SUBSTRING(path, 2, 400), '/', '_'), '@', 'attr_')))
 23:                   AS nvarchar(max))
 24:             ,N'') WITHIN GROUP (ORDER BY path)
 25:         + N' FROM [stage].[leaf] WHERE config_type = ''' + REPLACE(@config_type, '''', '''''') + N''' GROUP BY config_id;'
 26: FROM
 27:    [stage].[inventory]
 28: WHERE
 29:        config_type = @config_type
 30:    AND path NOT LIKE '%[[]%';
 31: 
 32: EXEC sp_executesql @sql;
 33: 
 34: SELECT
 35:    COUNT(*) AS columns_generated
 36: FROM
 37:    INFORMATION_SCHEMA.COLUMNS
 38: WHERE
 39:    TABLE_NAME = 'vw_' + @config_type;
 40: -- XML route:  trekking_bike 621, road_bike 615, cargo_bike 621
 41: -- JSON route: trekking_bike 623, road_bike 617, cargo_bike 623
 42: GO
 43: -- --------------------------------------------------------------------------------
 44: -- Second layer: typing where it is needed. All values of the generated view are
 45: -- text; only here do price, weight and lumen get their types.
 46: -- --------------------------------------------------------------------------------
 47: CREATE OR ALTER VIEW [dbo].[vw_trekking_bike_typed]
 48: AS
 49: SELECT
 50:     config_id
 51:    ,general_model                                        AS model
 52:    ,TRY_CONVERT(smallint,       general_modelyear)       AS model_year
 53:    ,TRY_CONVERT(decimal(10, 2), general_price)            AS price
 54:    ,TRY_CONVERT(decimal(6, 2),  general_weight)          AS weight_kg
 55:    ,frame_material                                          AS frame_material
 56:    ,drivetrain_attr_kind                                         AS drivetrain_kind
 57:    ,TRY_CONVERT(tinyint,        drivetrain_gears)              AS gears
 58:    ,TRY_CONVERT(smallint,       lighting_front_lumen)            AS lighting_front_lumen
 59: FROM
 60:    [dbo].[vw_trekking_bike];
 61: GO
 62: SELECT
 63:     frame_material
 64:    ,COUNT(*)                        AS n
 65:    ,CAST(AVG(price) AS decimal(10, 0)) AS price_avg
 66: FROM
 67:    [dbo].[vw_trekking_bike_typed]
 68: GROUP BY
 69:    frame_material
 70: ORDER BY
 71:    frame_material;
 72: -- Aluminum   246   3178
 73: -- Carbon     246   3443
 74: -- Steel      248   3426
 75: -- Titanium   240   3116

The view for the trekking bike has 621 columns and comes into being in under half a second. The script creates the view of one type per run. For the road bike and the cargo bike it runs once more with the respective type in @config_type and delivers views with 615 and 621 columns. A query over it, say the average price per frame material over 980 configurations, runs in around 40 milliseconds. This is the moment the approach pays off: a new attribute in the catalog requires no change to shredding, inventory and view generator. It is a new row in the inventory and, on the next run, a new column in the view. Manual work remains only for the typed view, if the attribute is to get a type there.

All values of the generated view are text, just as they stood in the XML. They get types in a second layer, and only the columns that someone needs typed. TRY_CONVERT is the right tool for that, because a single faulty value must not bring the whole view down. What to watch out for per data type is covered in Safe type conversion with T-SQL.

Three limits belong to this. A column name is an identifier of at most 128 characters. For longer input QUOTENAME returns NULL, and because STRING_AGG skips NULL values, the column silently disappears from the view. The column count at the end of the script is therefore more than a control output: 620 paths without ordinal plus config_id must give 621 columns. Furthermore, the mapping of path to column name is not unique. /A/B and /A_B both become a_b, and the view fails with the message that column names must be unique. The same happens with /A/@b and /A/attr_b, which both become a_attr_b, and with Gears and gears as soon as the path column carries a binary collation and the two spellings sit in the inventory as separate paths. Whoever has such names appends a running number from the inventory. SQL Server allows 1024 columns per view, Postgres 1600, where in Postgres a materialized table additionally requires the row to fit into an 8-KB block. With a thousand attributes and one view per type that still works out, with two thousand it no longer does. Then views are generated per assembly instead of per type, that is, one view for the frame, one for the drivetrain, and the consumers join through the configuration ID. And the path/value table itself is an intermediate format, not a target model. It has no types, no constraints, and every query has to pivot first. Whoever uses it as the data model for applications builds the entity-attribute-value pattern with all its well-known drawbacks. As a landing zone for a document with an unknown schema it is exactly right, and the generated views are the layer the consumers see. If a view is queried often, it can be materialized as a table.

Postgres bridge

The same route leads through JSONB in Postgres, and there it is even the fastest. COPY loads the JSON Lines file as one row per configuration, where CSV mode with impossible delimiter and quote characters makes sure that backslashes in the text are not read as escapes. The recursive CTE opens objects with jsonb_each and arrays with jsonb_array_elements including the ordinality, and both happen in one lateral join, so that one recursion serves both cases.

  1: -- --------------------------------------------------------------------------------
  2: -- Load JSON Lines: one row per configuration. CSV mode with impossible quote and
  3: -- delimiter characters keeps backslashes in the text from being read as escapes.
  4: -- --------------------------------------------------------------------------------
  5: DROP TABLE IF EXISTS stage.raw_json;
  6: CREATE TABLE stage.raw_json
  7: (
  8:     doc   jsonb NOT NULL
  9: );
 10: COPY stage.raw_json FROM '/tmp/configs.jsonl' WITH (FORMAT csv, QUOTE E'\x01', DELIMITER E'\x02');
 11: 
 12: -- --------------------------------------------------------------------------------
 13: -- Flatten recursively: objects via jsonb_each, arrays via jsonb_array_elements
 14: -- with ordinality; anything that is neither object nor array is a leaf.
 15: -- --------------------------------------------------------------------------------
 16: DROP TABLE IF EXISTS stage.leaf_json CASCADE;   -- CASCADE: the views of an earlier run depend on it
 17: CREATE TABLE stage.leaf_json AS
 18: WITH RECURSIVE
 19: CTE_node AS
 20: (
 21:    SELECT
 22:        doc ->> '@id'  AS config_id
 23:       ,doc ->> '@type' AS config_type
 24:       ,doc            AS node
 25:       ,''::text       AS path
 26:    FROM
 27:       stage.raw_json
 28:    UNION ALL
 29:    SELECT
 30:        T01.config_id
 31:       ,T01.config_type
 32:       ,T02.node
 33:       ,T01.path || T02.step
 34:    FROM
 35:       CTE_node T01
 36:       CROSS JOIN LATERAL
 37:       (
 38:          SELECT
 39:              '/' || key AS step
 40:             ,value      AS node
 41:          FROM
 42:             jsonb_each(CASE WHEN jsonb_typeof(T01.node) = 'object' THEN T01.node ELSE '{}'::jsonb END)
 43:          UNION ALL
 44:          SELECT
 45:              '[' || ordinality || ']'
 46:             ,value
 47:          FROM
 48:             jsonb_array_elements(CASE WHEN jsonb_typeof(T01.node) = 'array' THEN T01.node ELSE '[]'::jsonb END) WITH ORDINALITY
 49:       ) T02
 50: )
 51: SELECT
 52:     config_id
 53:    ,config_type
 54:    ,path
 55:    ,node #>> '{}' AS value
 56: FROM
 57:    CTE_node
 58: WHERE
 59:    jsonb_typeof(node) NOT IN ('object', 'array');

Loading takes under half a second, the recursion just under four, and the result is row-identical to the OPENJSON route: 1,528,476 rows, 1026 paths. More on COPY and its variants is in Transferring data.

XML directly works in Postgres as well, with xmltable instead of nodes(), and with a trap that is bigger than anything in SQL Server. The function xmltable over the whole 27-MB document with the row path /Catalog/Configuration//*[not(*)] was still running after 36 minutes in the measurement with Postgres 18.6. It reacted neither to pg_cancel_backend nor to pg_terminate_backend and only ended with the container. That was no outlier, as a follow-up measurement with partial catalogs shows: for 50, 100 and 200 configurations the same query takes 4, 17 and 99 seconds, that is, a good five times as long per doubling, and for 3000 configurations that would be hours. The obvious explanation is a call into the XML library libxml2 that does not check for interrupts. What the measurement proves is only the behaviour, not the cause. After the split into one row per configuration, the same xmltable runs as a lateral join over the fragments in around four seconds, and it does so even when it computes the positions right away, because count(preceding-sibling::*) is cheap on a 9-KB fragment. The ordinals only for repeated siblings then come set-based with dense_rank() over configuration, parent path and name. The script for it is in the XML download for Postgres. Postgres generates the views from the paths with format() and runs the DDL with \gexec, three views with around 620 columns each, each in around ten milliseconds.

  1: -- --------------------------------------------------------------------------------
  2: -- Generate the views from the paths: format() builds the DDL, \gexec runs it.
  3: -- List paths (with [n]) and the two attributes of the configuration stay out.
  4: -- --------------------------------------------------------------------------------
  5: SELECT format(
  6:           'CREATE OR REPLACE VIEW stage.vw_%s AS SELECT config_id%s FROM stage.leaf_json WHERE config_type = %L GROUP BY config_id;'
  7:          ,config_type
  8:          ,string_agg(
  9:              format(E'\n   ,max(value) FILTER (WHERE path = %L) AS %I'
 10:                    ,path
 11:                    ,regexp_replace(lower(trim(both '/' from path)), '[^a-z0-9]+', '_', 'g'))
 12:             ,'' ORDER BY path)
 13:          ,config_type)
 14: FROM
 15:    (SELECT DISTINCT config_type, path FROM stage.leaf_json WHERE path NOT LIKE '/@%' AND path NOT LIKE '%[%') T01
 16: GROUP BY
 17:    config_type \gexec
 18: 
 19: SELECT
 20:     table_name
 21:    ,count(*) AS columns_generated
 22: FROM
 23:    information_schema.columns
 24: WHERE
 25:    table_name LIKE 'vw\_%'
 26: GROUP BY
 27:    table_name
 28: ORDER BY
 29:    table_name;
 30: -- vw_cargo_bike      621
 31: -- vw_road_bike       615
 32: -- vw_trekking_bike   621

Limits and decisions

The method has a fixed frame, and that frame should be known before using it.

Depth: The ancestor names are columns, and their number is fixed. Four levels cover the catalog, a document with ten levels needs ten columns or a different way of building the path, and the check from pass 1 shows whether the reserve is enough. The recursive CTE of the JSON route does not know this limit, but in SQL Server it knows another one: after 100 recursion levels it stops with an error, until OPTION (MAXRECURSION 0) lifts the limit. Postgres recurses without a limit.
Namespaces: local-name() ignores namespaces, which is intended here. Two elements with the same local name from different namespaces thus fall onto one path. Whoever has to tell them apart adds namespace-uri() as a further column. A name() with prefix does not exist in SQL Server's XQuery, and the prefix would only be a spelling of the URI anyway. To address a namespace in the path on purpose, there is WITH XMLNAMESPACES.
Spelling and collation: XML names are case-sensitive, the text columns of the database with the default collation SQL_Latin1_General_CP1_CI_AS are not. Two elements Gears and gears thus collapse onto one path in the inventory, and the detection of repetitions wrongly reports their parent element as repeated. Whoever has such names creates the name and path columns with a binary collation such as Latin1_General_100_BIN2. If both spellings sit in the same configuration, the inventory shows the case through the same metric as a missing ordinal: more values than configurations. If they are spread over different configurations, the metric stays silent, and the case only shows up once the paths are counted under a binary collation.
Column widths and character set: Names with varchar(100), values with varchar(200) and paths with varchar(400) are tailored to the catalog, whose longest path has 32 and whose longest value has 19 characters. Longer strings are silently truncated by value() to the target width, without error and without warning. Just as silently, characters that the code page of varchar does not know are lost: Łódź becomes Lódz. Whoever expects such values takes nvarchar. For foreign documents the widths are adjusted to the data. A first pass with wide columns and MAX(LEN(…)) shows what is needed. A blanket varchar(max) is no option for the path: it is the key of the index on the path/value table, and a max type cannot be an index key.
Mixed content: Text alongside child elements falls through [not(*)]. Such documents need an additional pass over text() nodes.
Elements with XML attributes only: An element like <Warranty years="2"/> has neither text nor children. Pass 1 matches it nonetheless and delivers a row with an empty value, because [not(*)] only asks about child elements. The XML attribute is collected correctly by its own pass as /General/Warranty/@years. The empty row disappears if the predicate is extended by [not(@*) or text()]: leaves without XML attributes are kept even when empty, leaves with XML attributes only if they have text. A plain [text()], by contrast, would also discard deliberately empty elements such as <Belt/>. Whoever reaches for normalize-space() for this gets an error, because the function is missing in SQL Server's XQuery.
Types and localization: All values are text. Decimal point or comma, date formats and yes/no spellings are decided only in the typed view, and there with TRY_CONVERT, not with CONVERT.
File size: An xml value holds up to 2 GB, and SINGLE_BLOB reads the whole file at once. With documents in the gigabyte range the split is the bottleneck, and from a certain size a streaming parser outside the database is the better choice. That is the topic of the twin article on Python.
Known, small schema: Whoever has twenty attributes and knows them needs none of this. A nodes() with twenty value() columns is then shorter and faster. The generic route pays off when the schema is large, variable or unknown.

As a decision aid:

Starting point	Recommended route
XML, schema large or unknown, file fits into memory	Split, pass 1, detection, pass 2, inventory, generated views
JSON, schema large or unknown	One row per document, recursive CTE with `OPENJSON` or JSONB, inventory, generated views
XML, but conversion to JSON planned anyway	Convert, then the JSON route
Schema small and known	`nodes()` with fixed `value()` columns, no path/value table
File in the gigabyte range	Streaming parser outside the database, load as JSON Lines, rest in SQL

Summary

The hours in that project did not come from the programming language but from the design: code per type, access per attribute, schema in your head.
The database route does not know the attributes in advance. It loads raw, shreds generically into path and value, and afterwards counts what exists. What it requires are the element of the sub-documents, an upper bound for the depth, and lists in which an element name repeats in at least one configuration.
In SQL Server, the split into one row per configuration is the decisive step. After that, one pass over nodes() with the parent axis delivers all leaves in around two seconds.
Computing positions for all 1.5 million leaves raises the runtime from around two to around 40 seconds. They are therefore computed only for the elements that repeat according to pass 1.
The inventory makes required, optional and variant attributes per type visible, and from it the database generates the views with hundreds of columns itself.
JSON does without the second pass, in Postgres JSONB is the fastest route, and xmltable over a whole large document is a trap there.
The path/value table is a landing zone, not a data model. The generated and typed views are the layer for the consumers.

FAQ

How do I read XML with an unknown structure in SQL Server?

Load the document as an xml value, split it with nodes() into one row per sub-document, and then fetch every leaf element with nodes('//*[not(*)]'). The parent axis local-name(..) delivers the names of the ancestors, from which the path is built, and value('.') delivers the value. The result is a path/value table from which a grouping makes the schema readable. Required are the element of the sub-documents and an upper bound for the depth of the tree. The method recognises lists only when an element name repeats within them in at least one configuration.

How do repeated elements get their position in SQL Server?

The XQuery expression let $p := . return count(../*[local-name() = "Part"][. << $p]) + 1 counts the same-named siblings that precede the current element in the document, and thus delivers its position. For all 1.5 million leaves this expression is too expensive. It raises the runtime from around two to around 40 seconds. So pass 1 fetches all leaves without position, a grouping reads off at which positions elements repeat at all, and pass 2 computes the position only there. A ROW_NUMBER() over the output of nodes() looks simpler, but does not guarantee the document order.

Isn't the path/value table exactly the EAV model everyone warns about?

Yes, in form. No, in role. The entity-attribute-value pattern is a problem when it is the permanent data model of an application, because types, constraints and simple queries are missing. Here the table is an intermediate stage between the raw document and the generated views. It is filled once per load run, counted once, and the consumers see views with real columns. Whoever queries the views often materializes them and is back at an ordinary table model.

XML or JSON: what should I load if I have the choice?

JSON, if the ordinals of lists matter and the conversion happens anyway. An array brings its order with it, and the recursive CTE needs no second pass. XML directly, if the file arrives that way and should not be touched again. In SQL Server the XML route took around 15 and the JSON route around 25 seconds for 1.5 million values. In Postgres, JSONB was clearly ahead at around four seconds.

Why not OPENXML?

OPENXML with sp_xml_preparedocument delivers an edge table of the whole document in which every node is a row with an ID and a parent ID, in document order. That sounds like exactly the generic access this article is looking for. In the measurement, the edge table alone for 3 million nodes took around 55 seconds, the text column arrives as ntext and cannot be indexed, and the prepared document occupies memory until it is explicitly released. The route through nodes() was around thirty times faster in this measurement and needs no handles.

Why not just nodes() with fixed paths per attribute?

Because that is exactly the loop in the code, only in SQL: one value() call per attribute, a thousand calls per type, and a change for every new attribute. For a known schema with twenty attributes that is the right way. For a thousand attributes that nobody knows completely, the generic shredding with a subsequent inventory in SQL is the version that does not have to change with every catalog change.

Upstream:

Set-Based vs. Row by Row: PostgreSQL Refresh from 114 to 12 Seconds (the general argument against the loop per element)
ETL vs. ELT — How to Tell Which Pattern You Actually Built (load raw, then transform in SQL)
Design Pattern // The Architecture of an ETL Process (the layers into which the path/value table fits)

Downstream:

Deriving Data Quality Rules from the Schema (the inventory as a data quality tool)
Design Pattern // Safe Type Conversion with T-SQL (the typed second view layer)
Transferring Data: bcp, COPY, pgloader, ETL (loading on the Postgres side)

Downloads:

The Context Window Is Full Before the First Prompt — When Project Rules in Claude Code Become a Fixed Cost of Every Session

Marcus — Thu, 22 Oct 2026 20:23:31 +0000

A subagent in Claude Code is supposed to answer a one-line question about the tool's documentation. It aborts before taking its first step: Prompt is too long, 347,814 tokens against a limit of 200,000. The question itself had loaded nothing. What had loaded were the project instructions that every session in this repository receives at start. The subagent ran on a model with a context window of 200,000 tokens, and the request that was meant to start it was already almost twice the size of that window. The main session that had started it ran with a window of one million tokens. It did not abort, it was half full before anyone had typed anything.

This article describes how these project instructions in the private DI² project could grow to that size. It shows what a measurement with /context found and what three levers achieved that were applied afterwards. The mechanics and all numbers apply to Claude Code as of October 15, 2026. The measurement was taken with version 2.1.145, the subagent's failure comes from a session with version 2.1.292. The cause, however, is not limited to Claude Code. It affects every tool that hands an agent project rules at start. In such rule files, two kinds of text accumulate over time: instructions for the next step, say where a new test file belongs, and evidence of past steps, say which test file secures which bug. This article calls the first a rule and the second a record. Both belong in the repository. Into the context that every session loads at start belongs only the rule, because only the rule is needed for the next step.

The essentials up front:

Project rules are a fixed cost of every session: Everything in the project's CLAUDE.md, its imports and the rule files without a file scope loads in Claude Code at start, before the first prompt arrives. In the DI² repository that was a measured 454,900 tokens on October 15, 2026, just under half of a one-million-token context window and more than twice a 200,000-token window.
The dividing line is rule versus record, not long versus short: A rule says where a file belongs and which pattern it uses, a record proves what was checked back then and why. After a good six months of maintenance, the largest rule file consisted to 84 percent of an inventory table with one row per test file. Nobody needs it while writing a test, yet every session paid for it at start.
Three levers, measured before and after: Bind rules with a file scope to paths, move inventories into a neighboring file, and phrase instructions so that the agent looks things up instead of preloading everything. Afterwards the project instructions stood at 64,700 instead of 454,900 tokens, and the subagent that had aborted before started again.
Binding to paths has a limit: A bound rule arrives only with the result of the tool call, and for a file that is just being created, that is too late. Three safety nets help against that, and without a guardrail in CI the growth comes back.

Prerequisite: Claude Code basics are explained briefly where they are needed. All token numbers come from /context in a fresh session, with date and version. Where a number is only estimated or extrapolated, it says so.

What a Session Costs Before Anyone Types

At the start of a session, Claude Code reads three kinds of project instructions. The project's CLAUDE.md in the root directory of the repository. Everything this file pulls in via @ import, up to four levels deep. And every rule file under .claude/rules/ that carries no paths: frontmatter. How a repository ends up with 30 such rule files and what is in them is described in 27 Rule Files Instead of One CLAUDE.md. The point here is that these three items sit in the context before the first prompt is typed, and they do so in every session, regardless of whether the task takes one line or a whole day.

How much that is, the /context command shows. It lists every loaded instruction file individually with a token count that the tool determines itself, rounded to the hundred. The count depends on the session's model: A probe counted the same files noticeably lower under Haiku 4.5 than under Opus 4.7. The reason is the tokenizer: Models from Claude 4.7 onwards use a newer one that, according to the documentation, counts the same text at roughly 30 percent more tokens than the older one, somewhat more or less depending on the text. It also depends on the content, per file between 1.9 and 2.2 bytes sat on one token, and it is reproducible for the same state and the same model. For a repeatable measurement, a start without the interactive interface is enough, with /context as the only input:

claude -p "/context"

In the DI² repository, this gave the following on October 15, 2026, with the Opus 5.5 model, before any change:

Item	Tokens
`CLAUDE.md`, its imports and 27 rule files without `paths:`	454,900
Auto-memory (user-local memory file, not in the repository)	4,700
System prompt, tools, skills, agents	27,900
Occupied before the first prompt	487,500

The first row is the fixed cost this article talks about: the project instructions that load at every start and that the project itself controls. The two other items come on top but do not depend on the repository.

Which window this number fills depends on the model. The main session ran on Fable 5.1 and Opus 5. Both have a window of one million tokens in Claude Code, and both count text like Opus 5.5: A probe with the same two files gave the same values under all three models. There, 487,500 tokens meant 49 percent occupied before anyone had typed anything. For a model with a 200,000-token window, it is 244 percent. After deducting the reserve of 33,000 tokens that Claude Code, in the measured version 2.1.145, keeps free for automatic compaction, nothing is left there, and that is the kind of model the subagent from the opening of this article ran on. Its error message named 347,814 tokens, not 487,500, because the two do not count the same thing: The message counts the subagent's request with the Haiku 4.5 tokenizer, /context counts the main session's start with the Opus 5.5 tokenizer. This article therefore compares only within /context with the same model, the error message proves the yes or no. Three files carried just under half: the test rule with 124,000 tokens, the rule for the application's register layout with 70,700, and the brand rule with 28,600.

The request that led to this measurement came the evening before, with the test rule open in the editor. It comes from the session transcript, was dictated, and is smoothed here for language and translated from German:

I've noticed that the context window fills up pretty quickly when I
develop new features or fix a bug. I just opened the file testing.md
and saw that it is pretty long. It probably gets loaded into the
context window in full whenever a unit test runs, or whenever, and
uses up a lot of it that way. Can you please analyze where and during
which actions the context window fills up quickly, and make
suggestions for how we deal with it?

Requested was an analysis with suggestions, not a measurement. The agent answered with “I'll first measure what actually loads at session start”, and what came next was an estimate: the size of the loaded files in bytes divided by 3.7, a rule of thumb for bytes per token. The files totaled 902,576 bytes that day, the result was 243,939, rounded 244,000 tokens. Observed rather than estimated was only one part, namely which files were actually loaded. That the agent could look up in its own instructions. Exactly three rule files were missing there, the ones for backend, frontend and security, and those are not the three largest just named, but the three that were already bound to file paths at the time and therefore do not load at start. How that works is shown by the lever “Binding rules to files with paths:”. The measurement came only the day after, and it gave 454,900 tokens for 909,920 bytes, that is 2.0 bytes per token. The rule of thumb had underestimated the token count of German-language Markdown with tables, paths and code identifiers by almost half. All numbers in this article are therefore either measured or explicitly marked as an estimate or extrapolation. Whoever wants to know their own fixed cost measures it.

Grown With the Work

The 454,900 tokens were the result of a good six months of maintenance. The first rule file was created at the end of March 2026, and the repository's history makes it possible to trace how many tokens the rule files would have cost on the first of each month. These values are not measured, that would have taken a separate session in a separate worktree for every month. They are extrapolated from the file sizes with the ratio of 2.0 bytes per token measured on October 15. Whether the historical states had the same ratio is not checked, per file it lay between 1.9 and 2.2 on the day of measurement. As an order of magnitude that is enough, and the table claims no more.

State	Rule files total (tokens, extrapolated)	Of which the test rule (tokens, extrapolated)
April 1, 2026	8,600	does not exist yet
May 1, 2026	26,600	does not exist yet
June 1, 2026	117,300	does not exist yet
August 1, 2026	219,000	13,000
September 1, 2026	250,900	22,500
October 1, 2026	391,100	94,600
October 14, 2026	441,100	119,800

Since June the volume has almost quadrupled, and between September and October alone roughly 140,000 tokens came in, 72,000 of them in the test rule. Behind this jump is a maintenance duty. The test rule required that every new test file add a row to its inventory: which file checks what, why the test came about, and which counter-check proved that it catches the bug. Every single one of these rows makes sense. Whoever later wants to know whether a function already has a test finds the answer there. In sum, however, the duty turned the rule file into a record that every session pays for.

And every one of these rules came out of a real bug, and every one prevents the same bug from coming back. Their value is not in question. What makes them expensive is the moment they load, which is the same for all of them, namely always, and the fact that in the same file two kinds of text grow that are needed at different times.

Rule or Record: The 84 Percent Case

The DI² project's test rule was, at 124,000 tokens, the largest rule file, measured with /context. Of that, around 104,000 tokens went to the inventory table with one row per test file, 162 rows on the day of measurement. That is 84 percent of the file, calculated via the table's share of the bytes, because /context counts per file, not per section. The measurement after extracting the table confirmed the arithmetic: Without it, the file cost 20,300 tokens. The remaining 16 percent were the actual rules: where test files go, which test environment to choose, mock patterns, time budgets, load tests, and a list of things one does not do. The longest single table row had 15,129 characters, more than the entire rule file for functions or the one for triggers.

What the inventory rows answer is a record question: What was checked back then, why, and with which counter-check? What is needed when writing a new test is a rule question: Where does the file go, which environment, which mock pattern? The rows are not bad. They are in the wrong place, and in the wrong place they are paid for at every session start, including in the sessions in which nobody writes a test.

Kind	Answers	Example
Rule	the next decision: where, how, what not	“Test files sit next to the source file and end in `.test.ts`.”
Record	a past one: what was checked, why, with which counter-check	“This file checks the rounding because an amount was rounded wrongly. Negative control: test red without the fix.”

The same pattern showed up in four more rule files, there at harmless size. The rule for dropdown menus consisted to 35 percent of an inventory of its users, the one for naming toasts to 33 percent, the one for confirmation dialogs to 33 percent, the one for dialogs to 28 percent. In absolute terms that was only a few thousand tokens each, which is why it never stood out. The test rule differs from them not in pattern but only in scale: one row per test file, with growing test coverage.

The dividing line runs along the function of a text, not its form. The same test rule holds a list of known gaps that are deliberately not covered by tests. It looks like an inventory but is a rule, because it is meant to act at the next intervention: Whoever touches one of these places should know that no test will catch them. This list stayed in the rule file.

What Gets Added While Working

The fixed cost is only the beginning. As soon as a session works, reads come on top, and the largest of them is almost half the size of the fixed cost itself. The following values are extrapolated, not measured with /context, because the command shows only the start. They are based on the file sizes and the measured ratio of 2.0 bytes per token.

Action	Tokens (extrapolated)
Reading the bug register to find the next free number	around 210,000
Reading the application's largest component once	around 155,000
Loading the tool guide for the bug workflow	around 22,000

The first row is the interesting one. What is needed there is a single number, the highest number assigned. The table is sorted in descending order, so the first data row carries the answer. That the entire register wanders into the context instead, more than a full context window, is not the tool's fault. It comes from the instruction “read the register and determine the next number”. The agent does what it says.

Together with the fixed cost, this explains the observation that stood at the beginning: start, tool guide, one large file, and the session is full before the first line of code is written. For subagents it is more than an annoyance. A subagent usually receives the project instructions, only the built-in Explore and Plan agents start without them. The main session's context window it does not inherit: Its window follows its own model, and some of the built-in agents run on smaller models than the main session. The agent that answers questions about the Claude Code documentation ran, in version 2.1.292, on Haiku 4.5 with a 200,000-token window. With 454,900 tokens of instructions, it could not start in this repository, while an agent that inherits the main session's model still had room with one million tokens. The failure from the opening of this article was exactly that, here the message verbatim:

Prompt is too long · the request is ~347814 tokens (limit 200000) but this
conversation is only ~227482 tokens — the rest is system prompt, tool
definitions, and attachment content. A single-exchange conversation cannot
be compacted; reduce attached files/tools or start with less context.

The failure even prevented its own investigation: The attempt to clarify the tool's loading rules from the documentation via a subagent failed at the very context limit it was supposed to explain. Since version 2.1.293, the same agent runs on Haiku 5.5 with a window of one million tokens. Reproduced on October 20, 2026, with version 2.1.301 on the state before the rework: The agent started, its request had 485,056 tokens, counted with that model's newer tokenizer, and it answered the probe question. The fixed cost has not become smaller by that. Only the threshold at which it becomes visible has moved back.

The table's second row, the largest component, applies to the day of measurement. Since then it has resolved itself, for a reason that had nothing to do with the context window. The largest component had 7,103 lines. In September, a bug fix had failed on it because the interaction of two load runs was not visible in any diff, and since then the file was under a lock for any non-trivial change. On the same day as the rework of the rules, its decomposition therefore began, behavior-neutral and in cuts, each its own commit. On October 19, 2026, it consisted of an orchestrator with 1,607 lines and 23 units for load paths, locks, mutations and dialogs, none above 1,000 lines, with 21 test files of their own alongside. The code did not get smaller in the process but grew by a third, because every unit needs a header, an interface and imports. What dropped is what a single task has to load. Whoever changes a load path today reads a hook with 900 lines, extrapolated around 22,000 tokens, instead of the whole file at around 155,000. That is the same dividing line as with the rules, only in source code: Nothing was shortened, what changed is what enters the context and when. The cut was needed first by the human who wants to see an interaction in the diff. The agent that reads the file gets the same benefit.

Three Levers, and What They Cost

The same rule applies to all three: No rule is shortened in substance. Each came from a real bug. What changes is only when it loads and what sits next to it in the same file.

Binding Rules to Files With `paths:`

Claude Code knows a paths: frontmatter for rule files. A rule that carries it does not load at start but only when the agent reads, writes or edits a file that matches one of the patterns. For Write and Edit this has applied since version 2.1.288 of October 2, 2026, before that only a read loaded the rule. The measurement in this article with 2.1.145 is unaffected, it counts only the start. The patterns are glob patterns, relative to the repository:

---
paths:
  - "src/components/**"
  - "src/app/**"
  - "src/lib/**"
---

In the DI² repository, 3 of 30 rule files used this on the day of measurement, although most rules are tied to a file type. The SQL rules apply to files under db/, the rules for register layout, dialogs and toasts to components. After the rework, 24 of 30 files carried a paths: frontmatter. That was 264,300 tokens and thus a good two thirds of the total savings.

The lever has a limit. A bound rule arrives only with the result of the tool call, because the call on a matching file is what loads it. A probe on October 15, 2026, in Claude Code 2.1.292 showed it: In a session whose context did not yet hold the frontend rule, a read of a component loaded it only in the result of that read. When editing an existing file this is harmless, because Claude Code's Edit tool requires a prior read of the file and the rule is then already there. If the agent changes the file through the shell instead, it does not load the rule. When writing a new file, the content is written before the rule arrives. A second probe on October 20, 2026, showed this in two versions: A rule required a specific first line for every new file under src/. In 2.1.145 the Write call did not load the rule at all, in 2.1.301 it arrived after the call's result, and in both cases the file came into being without that line. Whether the agent notices that afterwards depends on the run: In a third probe with 2.1.301 it again wrote the file without the line, saw the rule with the result of the call, and added the line with a second call. That cannot be relied on. Precisely when a convention is meant to prevent something from being created wrongly, it is not there.

Three safety nets close this gap:

Creation rules stay global: The test rule and the rule for page scaffolds apply when creating files that do not exist yet. Both stay without paths: and carry a comment at the top of the file that explains why. Without that comment, the next cleanup session tidies them away as well.
Workflow skills read explicitly beforehand: The skill that creates backend code reads the SQL rules before it writes the first file. The skills for frontend and interface design got the same step for the interface rules.
A rule map in CLAUDE.md: One line per bound rule with file name and scope, without content. The map costs 1,800 tokens and deliberately sits in CLAUDE.md, because that file is read in again after an automatic compaction of the context. Without it the agent does not know that a rule exists, and therefore does not ask for it either. Then binding would not be a saving but a loss of substance.

The comment at the top of the test rule, translated from German:

<!-- Deliberately WITHOUT paths: frontmatter. A bound rule loads only with
     the result of the Read/Write/Edit on a matching file. For a file that
     does not exist yet, that is too late. Applies when CREATING a test
     file (location, suffix, mocks, budgets). Do not bind. -->

Two lines from the map:

- .claude/rules/sql.md — overarching SQL style guide (naming, alignment), db/**
- .claude/rules/tables.md — table DDL, constraints, comments, db/**

Two pitfalls come with it. The frontmatter is YAML, and Claude Code reads it at the start of every session to decide which rules are bound. A glob pattern that begins with an asterisk and stands without quotes, say - **/*.tsx, is read by YAML as a reference to an anchor, not as text. The parser aborts, and Claude Code then loads the rule as if it had no paths:, that is, at every start. In normal operation it does not report this. A probe in Claude Code 2.1.145 confirmed it: Of three rules with paths:, exactly the one with the unquoted asterisk appeared among the files loaded at start, and only claude --debug reported the parser error, without naming the file. The saving is then gone, and nobody notices. Patterns in the frontmatter therefore always belong in quotes.

The second pitfall is a binding that is too narrow. The brand rule, which among other things governs quotation marks and punctuation in user-facing text, was first bound to the dictionary files and one further single file. User-facing text, however, also lives in other files next to them, and from exactly one such file came the bug that had once triggered the rule. It is now bound to the whole directory. A binding that is too wide costs something only when the rule loads. One that is too narrow never loads it where it is needed.

Separating Rule and Record

The test rule's inventory table moved into a file of its own under docs/, with all 162 rows unchanged. The test rule itself kept its rules and two sentences about the inventory: where it lives and that every new test file adds a row there. Without this maintenance duty in the rule, the inventory becomes orphaned. The rule file thus shrank from 124,000 to 20,300 tokens. That is 103,700 tokens saved, a good quarter of the total.

This is what the separation looks like in concrete terms. Since then, only the reference remains in the test rule, here shortened and translated from German:

## Inventory

The inventory of test files (one row per file: what it checks,
triggering bug, negative control) lives outside this rule file in
docs/inventories/unit-tests.md. Do not read the whole inventory,
search for the file name instead.

The extracted file begins with a note on what it is and how to read it. Then follows one row per test file, here one of the shorter ones:

# Inventory of Unit Test Files

> Record, not rule. One row per test file: what it checks and why,
> often with triggering bug and negative control. The rules for unit
> tests live in .claude/rules/testing.md.
>
> Read selectively, not in full: The file name is at the start of
> every row, a search for it finds exactly that one row.
>
> Maintenance duty: add a row for every new test file.

| Test file                  | Project | Tests                                      |
|----------------------------|---------|--------------------------------------------|
| src/lib/topo-order.test.ts | unit    | computeProcessingOrder + refineWithinLevel |
|                            |         | (levels, cycles, tie-break, isStale)       |

What matters is how the extracted file is built. Moving alone would only have relocated the problem, because reading 104,000 tokens in one go as soon as the agent needs the inventory would hardly be better. The file therefore has one row per test file, and the file name is at the start of the row. A search for a file name finds the row without the whole file being read. The file's header says so explicitly: do not read, search.

In the DI² project this has become a convention, which itself costs 1,500 tokens because it sits in a globally loaded rule. A new inventory is created under docs/, never as a section of a rule file. The rule names the location and the maintenance duty in one sentence. And large files are not pulled into CLAUDE.md via @ import, because what it imports loads with it at every start. Splitting a large file across several imports therefore saves nothing.

Looking Up Instead of Preloading

The third lever changes no rule but instructions. The skill that creates a new bug read the bug register in full to find the next number. Now it reads at most the first 30 lines, because the register is sorted in descending order and the first data row carries the highest number. Two further places that searched the register for a specific entry now search with grep instead of reading. The same thought led to two changes to CLAUDE.md: The roadmap table of the product description, 89 percent of that file, moved into a file of its own, and the feature index is no longer pulled in via import. The agent reads both when it needs them, and the skills that need them say so explicitly. Together 25,500 tokens.

Not every full read is waste. The review of all skills found three that stay. The feature index is read in full by eight skills because the context there is actually needed. The product description is small after the rework. The test-case index is appended where it is read. The lever applies to instructions of the form “read the file to find a single value”. For a file whose entire content carries the decision, it does not apply.

Before and After: 454,900 to 64,700 Tokens

The rework ran in five steps, each its own commit and individually revertible. First came the purely relocating ones, then binding to paths as the riskiest step, last the guardrail. There was no written-out implementation request. The agent had proposed a spec from the analysis and asked questions about it, which the maintainer answered by selection. Questions and answers are in the transcript verbatim, shown here translated from German:

How far should I take the context rework now?
  -> Everything, including the scoping pass
One feature for all four levers, or split up?
  -> One feature, four building blocks
Scope the register and layout rules? That is the big item,
but the risky one.
  -> Delegate the decision to /architecture
Detach the PRD roadmap and the feature index from the always-includes
of CLAUDE.md?
  -> Yes, with a lean remainder
Should a guard against relapse be created? Without it the next rule
file grows into a record dump again without anyone noticing.
  -> Yes, as a CI guard with baseline

The four levers in this count are the three of this article and the guard, which this article treats separately as the guardrail. The implementation request after that consisted of two words: “sofort umsetzen”, implement now. During implementation, after the loading semantics had been clarified, one last question came: whether the register and layout rules should be bound as well, by the estimate at the time 115,000 tokens together, among them the register rule with a measured 70,700. The answer was yes, and that became the largest item of the savings.

The acceptance criteria, among them the measurement with /context in a fresh session and the subagent that has to start again, had been written into the spec by the agent. The maintainer accepted them with the spec rather than formulating them, writes no Node, and could not have written the check script. The rework of the rule files, the script, and the decision which rules stay global were worked out by the agent. From the maintainer came the problem description, the answers above, and at the end the measurement itself: /context and the subagent probe in a fresh session, started by hand, because a running session cannot re-measure its own start state. What mattered was that the measurement stood in the spec as acceptance from the beginning and was not set after the result.

The measurement was taken with the tool itself. Both states were available as separate Git worktrees, /context was the first and only input of a new session without the interactive interface, Claude Code 2.1.145 with the Opus 5.5 model, both runs on October 15, 2026, minutes apart on the same machine. A counter-check in an interactive session counted file by file the same.

Category	Before	After
Project instructions (`CLAUDE.md`, imports, rule files)	454,900	64,700
Auto-memory (user-local)	4,700	4,700
System prompt, tools, skills, agents	27,900	27,200
Occupied before the first prompt	487,500	96,600
Share of a one-million-token context window	49 %	10 %
Share of a 200,000-token context window	244 %	48 %
Free in a 200,000-token window after the 33,000-token reserve for automatic compaction	nothing	70,400

Broken down by lever:

Lever	What changed	Tokens
Binding rules to files	21 more rule files with `paths:`, 3 → 24 of 30	−264,300
Separating rule and record	Test rule 124,000 → 20,300, inventory alongside	−103,700
Looking up instead of preloading	Roadmap out of the product description, feature index no longer an import	−25,500
Cost of the implementation	Rule map (+1,800), convention “rule versus record” (+1,500)	+3,300
Total	files loaded at start 30 → 8: `CLAUDE.md`, imports 2 → 1, unbound rule files 27 → 6	−390,200 (−85.8 %)

The second proof is a yes or no. The same subagent that had aborted before started after the rework in a fresh session with 66,887 tokens of context and answered the probe question correctly.

Three things the numbers do not say.

Measured is the fixed cost, not the working cost: Bound rules are not gone. A session on the register interface still loads the register rule with its 70,700 tokens as soon as it touches the first component. A session on SQL or documentation no longer loads it. Measured is the start, not a typical working day.
The limit of binding is measurable: Of the 64,700 tokens, 31,300 belong to the two rules that deliberately stay global because they have to act before a file exists. That is just under half of the remainder, and it cannot be removed without shortening the rules.
Binding shifts who sees a rule: Before the rework, every skill had all rules in its context because all of them loaded at start. After the rework, a bound rule loads only when the agent reads or edits a matching file. The skill that reviews a diff does not do that. It reads the diff via git diff in the shell and touches not a single file in the process. Its context therefore holds none of the bound rules, for instance not the SQL conventions for files under db/. It would have checked SQL against conventions it does not know, and found nothing. This came up only in the quality assurance of the rework, not during the rework itself. The skill now matches the paths in the diff against the map and reads the corresponding rules explicitly before it reviews. Behind that is the same reflex as in The Agent Measures Where I Click: do not assume what the agent sees, go and look.

Without a Guardrail, the Growth Comes Back

The growth from the section “Grown With the Work” came from a maintenance duty without a cap. Without a cap, the savings would be a snapshot, and the next well-meant inventory would produce the same number again in half a year. With the rework, a check script therefore went into CI, without dependencies, as a blocking step. It calculates with bytes divided by 2.0, the measured ratio, and on the state of October 15 that put it about two percent off /context. That is an approximation with one day's ratio, not a measurement. Measuring continues with /context, the guard only draws the line, and the budget of 80,000 tokens leaves room against the measured 64,700 for the error of the approximation. It checks seven things:

Check	Reports
Fixed cost	`CLAUDE.md`, its imports and all unbound rules above the budget of 80,000 tokens. The total is printed on every run so that a new import does not add fixed cost unnoticed.
Size	a single rule file above the threshold, with separate thresholds for bound and unbound rules
Inventory share	more than 25 percent of a rule file's lines sit under an inventory heading. This check runs separately from size: A large file of pure rule prose is no finding, a small file that is half inventory is. The guard thus does not force rules to be shortened, but records to be moved.
Frontmatter	a `paths:` block that cannot be parsed, unquoted patterns included. Such a rule counts toward the fixed cost, because that is how the tool loads it.
Map	a bound rule without a line in the map, or a map line without a file
Orphaned inventory	number of test files against number of inventory rows
Baseline	a tolerated legacy that no longer occurs. The baseline may only shrink.

The guard checks files, not behavior. Whether a skill actually reads a bound rule before it reviews, it cannot see. The review skill without SQL rules from the section before was found by quality assurance, not by the guard.

The first run immediately produced a finding nobody had expected. The orphaned-inventory check found five test files without an inventory row, all from the time before the rework. The maintenance duty “one row for every new test file” had thus let the inventory grow to 104,000 tokens and still had not been kept. A duty that nobody checks therefore keeps neither the inventory complete nor the rule file small.

The decomposition of the largest component ended with a second guard on the same pattern, this time for source files. It checks every file under src/ for at most 1,000 lines, tests included, with a baseline for the legacy that may only shrink, and a margin of two percent so that a bug fix on an old file gets through and a feature does not. Its first run found 33 files above the limit, six were expected. The rule for it does not sit in a file that loads at every start but in the rule for components, pages and hooks, which is bound to exactly these paths. It therefore enters only the sessions in which such a file can grow. For all other files, the guard speaks up itself.

What the Solution Is Not

Shortening rules until they say nothing is not it. After the rework, 20,300 tokens of pure rule prose remained in the test rule, and the acceptance criterion had demanded “at most 12,000”. That number came from the rule of thumb of the day before, with the same error as all the estimates. The number was revised, not the rule. What a rule says, it has earned from a bug, and a token budget is no reason to forget that again.

A product critique is not it either. The binding mechanism existed the whole time, it was documented, and it demonstrably worked. That 3 of 30 files used it is a matter of maintenance discipline. That an agent has a rule in its context when it needs it is, moreover, the precondition for it to follow the rule at all.

And a check that calculates with the same wrong assumption as what it checks is least of all. The guard from the section “Without a Guardrail, the Growth Comes Back”, the check script in CI, was at first calibrated not with the 3.7 of the first estimate but with a second wrong number: The failed subagent's error message named, next to the 347,814 tokens of the whole request, around 227,000 tokens for the conversation, and the file size divided by that number gave 4.0 bytes per token. Only, the message did not count the same thing as /context. It came from a different model and referred to a part of the request. Measured, it is 2.0. Quality assurance and code review calculated with the same factor, confirmed “33,000 of 45,000 tokens” and considered the goal reached. Correct was 64,700 of 80,000. That came to light only when /context ran in a fresh session. Two checks that report green prove nothing if they take the same number from the same source. Why a green test can confirm, in exactly this way, the very bug it was supposed to find is shown in The Green Test That Proves Nothing.

One caveat belongs here. The analysis that triggered this article ran in the same session whose context window it examined. Measuring itself costs context.

Summary

Claude Code loads project instructions before the first prompt, in every session. In the DI² repository that was 454,900 tokens, just under half of a one-million window and more than twice a 200,000 window, grown over a good six months with the work on the project.
The dividing line is rule versus record. A rule governs the next decision, a record proves a past one. Both belong in the repository, only the rule in the context at start. 84 percent of the largest rule file was record.
Three levers, none of which shortens a rule: bind rules with a file scope to paths, move inventories into a neighboring file with the file name at the start of each row, write instructions to look up rather than preload. Result 64,700 tokens, and the subagent starts again.
Binding to paths comes too late for new files and shifts who sees a rule. Against that help creation rules that stay global, skills that read explicitly, and a map in CLAUDE.md. Against the growth helps a guard in CI, because a duty that nobody checks keeps neither the inventory complete nor the rule file small.

FAQ

How do you measure what a Claude Code session loads at start?

With the /context command as the first input of a new session. It lists every loaded instruction file individually with its token count. For a repeatable measurement, claude -p "/context" without the interactive interface is enough, checked with Claude Code 2.1.145. In Git Bash, path conversion has to be off for this (MSYS_NO_PATHCONV=1), otherwise /context reaches the model as a file path. The fresh session matters: A running session or a subagent carries the instruction state from its own start.

Isn't it enough to count characters and divide by four?

For English prose that is roughly right, for German-language Markdown with tables, paths and code identifiers it is not. Measured, the rule files sat at 2.0 bytes per token, per file between 1.9 and 2.2. An estimate with the factor 3.7 came in almost half too low, and a guard that calculated with 4.0 halved all results without anyone noticing. On top, the number depends on the model: The newer tokenizer from Claude 4.7 onwards counts the same text at roughly 30 percent more tokens than the older one, somewhat more or less depending on the text.

Does a rule with paths: load in time when the agent creates a new file?

No. The rule arrives at the earliest with the result of the tool call, for a new file the content is already written by then, and before version 2.1.288 a Write call did not load it at all. Whether the agent then fixes the file on its own depends on the run. When editing an existing file there is no gap, because the Edit tool requires a prior read. For rules that have to act at creation, three ways remain: leave them global without paths:, read them explicitly beforehand in the creating skill, and keep a map in CLAUDE.md so that the agent knows the rule exists. Checked in Claude Code 2.1.145, 2.1.292 and 2.1.301.

Why not simply shorten the rules?

Because every rule came from a bug that comes back without it. The DI² project's test rule kept 20,300 tokens of pure rule prose after the rework, although the target read “at most 12,000”: The target number came from a wrong estimate, and the number was revised. The savings come from when a rule loads and from no longer mixing it with record text, not from its content. Whoever still wants to shorten first checks whether a section proves a past decision. Then it is record and can move out.

Does the agent still see a bound rule when it touches no matching file?

Only if it knows the rule exists. A question about the SQL naming convention in conversation does not load the SQL rule, because no file under db/ is read. The same goes for a review that reads the diff via git diff rather than through a file. That is why CLAUDE.md holds a map with one line per bound rule, and skills that review without file access read the relevant rules explicitly. Otherwise binding would be a loss of conventions that goes unnoticed.

Parent articles:

Agentic Coding from a User's Perspective — the sub-hub of this branch: The work does not disappear, it shifts, among other places into the maintenance of rules.
AI-Assisted SQL Development with Claude Code — the entry point to rules, skills and agents.

Sibling articles:

The Agent Measures Where I Click — there the agent measures the product, here its own working material. The method is the same: measure first, then interpret.
The Control Loop — rules are the means by which code can be checked that one does not read oneself. This article shows what that means costs at session start.

Downstream:

27 Rule Files Instead of One CLAUDE.md — how the rule files are built whose load time is shifted here.
Deriving SQL Conventions with Claude Code — how the rules come about whose running cost is measured here.
The Green Test That Proves Nothing — why two checks with the same wrong assumption confirm the goal they miss.

The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent

Marcus — Tue, 20 Oct 2026 12:27:13 +0000

A bug gets reported the way it feels: the message does not appear, the page barely responds, a tab no longer loads after a restart. A user can only describe what the interface shows, and the interface is the end of a long chain. A coding agent with access to the database, the container runtime and a shell does not see the interface and starts debugging somewhere else. What it sees instead is what has been stored, what has been started and what has been swapped out.

This article describes four debugging sessions from the private DI² project in which exactly this difference decided the outcome. In three of them, a measurement settled early what a guess would have left open: which part of the chain is affected, whether the new code can be to blame at all, and whether a protection actually takes effect. In one case, the same method spent an hour ruling out explanations without hitting the cause. The same agent then built an explanation from the remaining signs that was internally consistent and still wrong, and presented it as a finding with more confidence than the evidence allowed. Both sides belong in the same article because they show the same way of working, once at its strength and once at its limit.

All four cases arose during development, not while running an application with users. Development here does include the maintainer setting up the project’s four environments and keeping them running, and a database developer does not come with a server background for that. The agent proposed the commands for it, and they were only executed after the maintainer had approved them.

The essentials up front:

The first question is not “why” but “is the suspicion even right”: Refuting a suspicion is easier than confirming a cause. Confirmation requires tracing the whole chain. Refutation is often a single query, and an agent with access to the database and the server should start exactly there. In one of the cases, a comparison across four environments disposed of the suspicion that the newly deployed version was to blame within a few minutes, without anyone reading a line of code.
The database tells you whether something was never written or just not displayed: Many applications store messages in the database first and only display them from there. If a message is missing from the interface, it may have been lost while being written to the database or only on its way to the display. A look at the record and its timestamp settles that immediately. If the message is in the database, everything before storage is ruled out, and the search starts behind it.
Tools have blind spots you need to know: A service whose memory has been swapped out to disk almost entirely shows up in the container statistics with a small value, because they only report the share in RAM. Anyone who knows the service’s normal footprint may pause at that, but a small value easily reads like a healthy one. How much has actually been swapped out only shows up in the machine’s memory overview.
A step that ran is not a step that worked: A brake on automated password guessing had sat correctly in the configuration for months. It counted requests on the old login path, however, which no longer existed after the switch to an external login service, and the actual login route ran unthrottled. A different fix removed a symptom without hitting the cause, so that everything was back the next day. In both cases the only thing that helped was measuring the effect directly instead of trusting the configuration file or the success message.
The flip side belongs here too: A language model effortlessly tells an explanation that sounds internally consistent even when the evidence does not support it. That is exactly what happened in one of the four cases. The most useful follow-up question to an agent is therefore not “why” but which concrete measurement supports this statement.

Prerequisite: Basic database knowledge and a rough idea of what a container is are enough. The cases are told so that the diagnostic path stays easy to follow, not the application behind it. Commands appear as patterns, not as instructions to copy onto someone else’s system.

What the Agent Sees and the Browser Does Not

An error report from the browser is an observation at the end of the chain. Between the user’s action and what they see lie a form, a server action, a database transaction, a notification path and a rendering step. Any of these stations can be the cause, and the report does not say which one it was. Anyone who only has the interface has to guess at which station to start looking.

An agent working in the same environment has different instruments. It can ask the database directly whether a record exists and when it was written. It can check which containers are running and which of their ports are actually reachable from outside. It can open a shell on the machine and look at how much memory is really available. This access does not make it smarter than the human. It makes certain questions cheap that used to be expensive, and cheap questions get asked earlier.

The difference shows most clearly in refuting. Confirming a cause is laborious because you have to trace the whole chain. Falsifying a hypothesis is often a single query. That is the lever in the four cases that follow. In none of them did the agent guess the cause. In three, a measurement decided early where to keep looking, and in one, the same method spent an hour only ruling things out without hitting the cause.

Case 1: Is the Message Missing, or Just Not Displayed?

The maintainer reported that an expected error message did not appear in the interface. The application first stores such messages as a record in the database, and the interface displays them from there. The observation from the browser therefore leaves two entirely different causes open. Either the message was never written to the database, in which case the fault lies in the write path. Or the record is there and just does not reach the interface, in which case the fault lies on the way from the database to the display, in the notification path. Both guesses are equally plausible, and they lead to completely different places in the source code.

Instead of following the bug description, the agent looked in the database first. The message was there. It had been written 22 seconds after the triggering change, in two write operations 348 milliseconds apart. That settled the question before anyone had read a line of application code. The write had worked, the display had not, and the search moved to the notification path. That is where the cause was.

The pattern behind it is simpler than it sounds. If a bug is reported as “something is missing from the display” and the application stores the affected data before displaying it, there is a measurement that immediately decides whether to look before or behind the storage step. For that, the query has to reach the same database the application reads from. Case 2 shows that even this can go wrong. As a rule of thumb, the measurement looks like this:

Ask the database whether the expected record exists,
and read its timestamp.

It does not exist  -> the fault lies in the write path.
It exists          -> the fault lies between storage and display.

The timestamps are more than a by-product, even if on their own they do not explain anything. What the agent had measured was only that the record was written twice, 348 milliseconds apart, and that this happened 22 seconds after the triggering change. The timestamps do not say why. Two write operations can stem from a second write path or from a retry, and a delay from a background job that only runs at intervals. Gaps like these, though, are often the cheapest way to reconstruct a sequence nobody remembers.

Case 2: Two Hours Down the Wrong Rabbit Hole

After the development machine was restarted, one of the application’s tabs no longer loaded. The error messages alternated between “relation does not exist” and “column does not exist”, and on top of that a test project was missing that had still been there the day before. Everything about it looked like damage to the database or a bug in the application.

The agent formed hypotheses one after another and checked each one with a measurement. Among other things, it compared the instance identifier (system_identifier) across both access paths, a number PostgreSQL generates when a database instance is set up and that the instance keeps from then on. It was identical on both paths. The database log showed not a single error since startup, the activity view showed no connection from the application, and the agent read the connection address directly from the environment of the running process rather than from a configuration file. Each of these checks was quick, and apart from the identifier comparison each one ruled out an obvious explanation. Even so, after an hour no cause had been found.

It was the human who found it, and only in passing. The maintainer threw in two observations that no tool shows: the test project had still been there the day before, and the database normally ran on its default port. That turned attention to the container’s port forwarding. A container is sealed off from the outside, and for the application to reach its database, the machine forwards one of its own ports to the container. This forwarding was configured but not active. The port was held by two forwarding processes that looked like leftovers from an earlier run. They were terminated, the application worked again, and the case was considered solved.

The Fix Worked and Still Missed the Cause

The next day everything was back. The same symptoms, the same hour of debugging, and this time the investigation went deeper. Two complete container worlds were running side by side on the machine, meaning two Docker installations, each with its own engine, the service that actually runs the containers. One was a system engine, the relic of a first installation whose removal had remained incomplete, and it was set up to start with every boot. The other was Docker Desktop, the installation that was actually intended. Both brought up a stack with the same name, including a database with the same container name.

That creates a race for the port. Only one process can hold a given port on the machine, and whoever starts first gets it. In this setup, the container of the other world still started, just without forwarding, and its database was then unreachable from outside. The application connects through the machine’s port and ends up in whichever world holds it. After a boot, that was the system engine, because it came up first.

The cross-checks the agent used to inspect the data ran along a different path. They were queries via docker exec, which run directly inside the database container and bypass the port. Which engine a docker command talks to is normally determined by the active Docker context, a setting of the command-line tool, and the active context was Docker Desktop’s. There was a container with exactly the same name there, and the cross-checks ran against its database without anything indicating it. The system engine was not hidden. It was simply not addressed by any docker command, and that is why its containers did not show up in any list. So the two data sets slowly drifted apart, because clicks and scripts hit one database or the other depending on which world held the port.

That turns the supposed fix from the day before into something other than what it seemed. Terminating the orphaned processes did work, because the application ran again afterwards. It did not hit the cause, because the second installation started again with the next boot. The fix even concealed the actual problem, because it removed the symptom reliably enough to end the search. Only the relapse showed that the supposedly orphaned processes belonged to the system engine and were therefore a symptom, not the culprit. The case was only resolved once the system engine had been shut down and uninstalled.

Spotting Two Docker Engines Earlier: The Rules From the Second Incident

Two rules made it into the project documentation. The first concerns the order: the question of which engine the active Docker context addresses belongs at the start of every local diagnosis, before any listing of containers. docker context ls shows all configured contexts with their endpoint and marks the active one with an asterisk. If the DOCKER_HOST environment variable is set, it overrides the context, and the list then says so explicitly. Several contexts are only a hint, because a context can also point to a remote machine or to the same endpoint as another. If the asterisk is on the wrong context, though, everything measured afterwards is worthless.

The second rule concerns the proof of identity: when the interface and a cross-check show contradictory data, the instance identifier can be used to check whether both paths reach the same database instance at all. An instance carries exactly one identifier. If it differs between the two paths, the paths demonstrably reach two different instances, and every further cross-check along the wrong path checks the wrong database. The reverse does not hold. A physical copy of the data directory, for example from a backup with pg_basebackup, carries the same identifier as its original, so an identical identifier does not rule out two instances. If a logical dump is restored instead, the target instance keeps its own identifier.

Query the instance identifier over both paths and compare:

  Path 1: docker exec directly in the container (engine of the active context)
  Path 2: through the machine's port, the way the application connects

  SELECT system_identifier FROM pg_control_system();

Different values  -> two different instances, proven.
Same values       -> no proof, a physical copy carries the same identifier.

One detail cost a lot of time and is worth mentioning. As long as the other engine held the port, restarting the container did not bring the forwarding back either. Anyone who tries that takes the container for broken, while the problem lies with the engine that occupies the port.

Case 3: “The Page Is Barely Responding”

Immediately after three deployments in a row, the maintainer reported a vague symptom. The prompt was a single line:

the page is barely responding. can we measure anything

One suspicion was obvious: new code had just gone out to three environments, so presumably the newly deployed version had broken something that used to work. The agent did not chase that suspicion in the code but compared the environments. It first called a static version route across all four environments and then the start page and the login page in each environment. The result disposed of the suspicion at once, because all environments were affected, including the production environment, which was not running the newly deployed version at all. That ruled out the newly deployed version as the cause, and the search moved to what all four environments share.

The decisive step lay outside the application, in a shell on the machine itself. The system load, which uptime reports as a measure of how much the machine currently has to do, was low. That made it unlikely that processing power was the bottleneck. At the same time, 2.9 gigabytes sat in swap space on disk, although 5.6 gigabytes of RAM were free. The identity provider, the service that handles login, was hit particularly hard. Of its 802 megabytes, the system had swapped 754 out to disk, and only 48 were still in RAM.

Swapped-out memory only comes back when a process needs it again. Linux does not bring it back on its own just because space has become free. That is why it stayed on disk even though there had long been room again. That fits the symptom: the first login request took 9.64 seconds, every further one only 0.19 seconds. On first access, the system has to fetch the swapped-out parts back from disk, and after that they are in RAM again. An ordinary cold start produces the same pattern, though. What pointed to swapping as the cause here was that almost all of this service’s memory was demonstrably on disk.

A tool trap is an inseparable part of this case. The container statistics that docker stats displays only report the share in RAM, not the swapped-out share. The affected service appeared there with 48 megabytes. Anyone who knows that it normally uses around 800 megabytes could have paused at that value, because it is not inconspicuous but suspiciously low. With a sluggish service, however, you look for a value that is too high, and a small one then reads more like a healthy one. Even then, pausing only gets you a suspicion. The gap to the normal footprint can have other causes too, such as a service that has just been restarted. How much has actually been swapped out on the machine only shows up in its memory overview, for example with free -h, because the docker stats table has no column for it.

For sluggish services despite low load:

  docker stats   -> shows only the share in RAM
  free -h        -> also shows the machine's swap space

The telltale sign is all three together:
  first request takes seconds, second is instant
  low system load
  swap space in use while RAM is free

The agent did not carry out the fix itself. Bringing the swap space back is a system-wide intervention, and the maintainer’s private data lives on the same machine. The agent handed over the command sequence and put a pre-check in front of it that calculates whether the swapped-out memory fits into the free RAM at all. If it does not, the kernel mechanism that kills processes when memory runs out kicks in, and a sluggish service becomes a dead one. The request looked roughly like this:

Give me the command sequence to bring the swap space back.
Put a calculation in front of it that checks whether the swap in use
fits into the free RAM, and that aborts if the reserve is too small.
Also tell me what prevents this state permanently.

(The second prompt is reconstructed from the task and its result, not taken from a log. The first one appears above as it was typed, translated from German.)

The maintainer then ran the command sequence, and only its result confirmed the explanation. Login dropped from 9.64 to 0.19 seconds and swap usage to zero, without a single line of application code. To make a repeat at the next memory spike less likely, a kernel parameter was adjusted as well. It weighs whether the system, under memory pressure, prefers to swap out process memory or to drop the file cache, and it was lowered from its default of 60 to 10. That makes the kernel more inclined to sacrifice the file cache instead of pushing a service out to disk. It is a weighting, not a guarantee. If RAM runs short during a spike, the kernel swaps anyway, and whether the same value helps on another machine depends on what occupies the memory there.

One detail of this case is uncomfortable and still belongs in the story. The agent had long since produced the first hint itself without following it up. Its own first measurement call had taken 5.16 seconds, the second 0.10. That is exactly the pattern it later found again on the machine. At first it dismissed it as a cold start. That was possible, but it never checked that interpretation.

Case 4: The Rate Limit That Never Kicked In

The reverse proxy configuration contained a limit on requests per minute for the login path. Anyone reading it concluded that login was protected against automated password guessing. The line was there, it was written correctly, and it had been active for months.

The limit sat on the endpoint of a login procedure the application used to handle itself. Since the switch to an external identity provider, that endpoint no longer exists. The throttled path was therefore never called. There was no effective throttling of login anywhere, and the dangerous part is not the missing throttle. It is the existing one, because it reassures every check that looks at the configuration.

The fix put the limit on the identity provider’s login path, and this time its effect was checked independently of the configuration. The endpoint was actually hammered with requests, and after only a few attempts it answered with the status for too many requests. A cross-check against a neighboring endpoint that must not be throttled kept returning regular responses. That established that the throttle takes effect on the login path and does not slow down the neighboring endpoint.

In addition to the throttle in the proxy, there is a second line of defense in the identity provider itself. It temporarily locks an account when too many logins for that account fail in a row. Since the fix, the setting for this is in the project’s setup script. Whether it is active was nevertheless not inferred from that script but queried directly from the running identity provider for all four environments. A script says what is supposed to be set, not what actually is set.

The two lines complement each other because they hit different attacks. The throttle in the proxy counts per source address and slows down the simple case in which someone tries many passwords from one place. If the attempts come from many addresses, it is not enough on its own. The lockout in the identity provider counts per account instead, and therefore also takes effect when the attempts against the same account come from many addresses. Either one alone would have looked like a solution and left one of the two cases open.

On the same day, the same class of error occurred a second time, this time while rolling out the fix. The rollout run printed two screens full of checkmarks, and afterwards the configuration appeared to be active in all four environments. In fact, the copy command came to nothing because it ran in the wrong directory, and the setup script ran in its old version. The effect was zero. The script prints its checkmark regardless of which status the call before it returned. The checkmark testifies that the line after it was reached, and nothing else.

It was only noticed because one specific expected line was missing from the output. It was not the overall impression that revealed the error but a targeted search for a single sentence that should have been there. The agent caused this third instance itself. It had handed over the rollout commands without pushing the corresponding commit first. The commands could not work on the server at all, because the code there did not yet contain the fix.

A Step That Ran Is Not a Step That Worked

At first glance, Case 2 and Case 4 do not look related. One plays out on a development machine and is about containers, the other on a server and is about security configuration. They still share the same structure, and that structure is the real yield of the four cases.

In both cases there was a signal that testified to the step, and nobody checked the effect. The orphaned processes were terminated, so the problem was solved. The throttle was in the configuration, so login was protected. The rollout script printed its checkmarks, so the fix was active everywhere. Each of these signals was true in itself. The process was terminated, the line was in the file, the script had run to completion. Between signal and effect, the chain had still broken, and none of the signals noticed.

What helped in every case was the same recipe. You look for a second, independent route to the same question, ideally one that turns out differently when the effect is missing than when it has occurred. For a configuration, that means hammering the endpoint and reading the setting back from the running system instead of reading the file. For a fix, it means restoring the condition under which the error occurred, and after a startup problem that includes another restart. For a rollout, it means searching for one specific expected line instead of judging the overall impression.

This kind of cross-check is easy to automate, which makes it one of the places where an agent can do more than a visual check. It has to be asked to, though. On its own, it does not replace the success signal with an independent measurement, and an agent that is only asked whether the change has been rolled out reads the same checkmarks as the human.

The Flip Side: When the Agent Tells a Story

Case 2 holds a second lesson, and it is less flattering. In the middle of the fruitless search, the agent built an explanation from the available signs in which everything fit together, and presented it as a finding. The explanation was internally consistent, it contradicted none of the measurements, and it was wrong. Only when the maintainer asked what exactly it meant did the agent take it apart and separate which part had been measured and which part interpreted.

That was not a slip but lies in how a language model is built. It produces fluent, plausible text, and an explanation of a cause is exactly that: fluent text. Forming a fourth statement from three observations that feels like a conclusion costs a language model practically nothing. Backing up the same statement costs it just as much as anyone else. The whole difference lies between these two steps, and it cannot be seen in the finished text.

For the user, that means splitting every finding into two parts. The first part is what was measured, and it can be named: a query, a return value, a timestamp, a status code. The second part is what was inferred from it, and that is a claim. The most useful follow-up question to an agent is therefore rarely “why”. It is which concrete measurement supports this statement, meaning which query, which return value, which timestamp. If the answer points to a concrete measurement, the finding holds. If it points to a synthesis of several clues, the finding is a hypothesis and should be treated as one.

How easily an estimate passes for a measurement is shown by a later case from the same project. The agent was asked to analyze where the context window fills up quickly and announced that it would first measure what loads at session start. What it then delivered was an extrapolation from file sizes and a rule of thumb. Its result came in almost half below the value the measurement gave the following day. The case is described in The Context Window Is Full Before the First Prompt.

A second limit concerns not interpreting but acting. The swap space from Case 3 would have been technically within the agent’s reach. It still did not carry out the intervention, because the same machine holds data that does not belong to the project. Whether a system-wide intervention runs on someone else’s data is not a decision an agent makes alone, and that holds even when the proposed command is correct. The agent delivered the proposal along with the safety calculation, and the human executed it. That is exactly how the work should be divided.

A related side finding belongs here too, because it came up in the same week. When two sessions share a working tree, a failed test run is at first only a hypothesis and not yet a bug in the product, because it may just as well stem from the other session’s work. What follows from that for parallel operation is covered in Two Agents, One Working Tree. And that, conversely, a green test run is also only a signal and not proof is shown in the article The Green Test That Proves Nothing. What follows from that for manual test cases is covered in How to Write Manual Test Cases.

What Remains

Together the four cases cost about four hours of debugging, and most of that went to the two incidents with the two Docker engines. Four things connect them.

The first measurement should narrow the search, not deliver the cause. Does the record exist? Are all environments affected? Do two access paths land in different database instances? Questions like these do not yet say what is broken, but they rule out whole areas. They cost almost nothing and therefore belong at the start of every search.
The most obvious suspicion gets a cross-check first and an investigation only afterwards. Whether the last change is to blame at all can be settled in minutes. Anyone who skips that question and goes straight to the code risks hours in the wrong place.
A success signal shows that something ran, not that it worked. Terminated processes, a line in the configuration and a screen full of checkmarks all testify only to a step. Whether the effect has occurred has to be shown by a second route that is independent of that step.
An agent’s finding contains measurement and interpretation, and both sound the same. They sit in the same paragraph and in the same tone. Separating them is up to the user, and it costs only one follow-up question: which concrete measurement supports this statement.

None of these cases was solved because the model understood the application particularly well. They were solved because questions that used to be too expensive suddenly cost only a minute, and in one case because a human contributed two observations that no tool shows. The first is what an agent brings when it can query the database and the server directly. The second stays with the human.

FAQ

Where do you start with a vague symptom like “the page is slow”?

Not with a hypothesis but with a comparison. If the same application runs in several environments, the quickest question is whether all of them are affected. If an environment that does not even have the suspect version is affected too, the suspect version is ruled out as the cause, and the search moves to what the environments share. That can be the machine, the network or a shared service.

Why doesn’t docker stats show a memory problem?

Because docker stats only reports the share in RAM, not the swapped-out share. A service whose memory has been swapped out to disk almost entirely shows up there with a very small value. Anyone who knows how much the service normally needs may pause at that, because the value is not inconspicuous but too low. How much has actually been swapped out, however, only becomes visible in the operating system’s memory overview, for example with free -h. The telltale sign is the combination of three things: pages that take seconds on the first request and respond instantly on the second, a low system load, and swap space in use while RAM is free. Each of these alone has other explanations too.

How do you notice that two Docker engines are running in parallel?

The most conspicuous sign is a port that answers although, according to docker ps, the container has no forwarding at all. Add to that data that does not match between the interface and a cross-check, and changes that have disappeared again after reloading the page. The check itself is short: first list the existing Docker contexts with docker context ls and see which one is active and where it points, then determine which process actually holds the port. Both belong before any further container diagnosis, because a wrong context invalidates every measurement that follows.

Should an agent be allowed to run system commands itself?

Within the project, there is little against it as long as the effect can be reversed. System-wide interventions are a different matter, especially when the same machine holds data that does not belong to the project. The division of labor that has proven itself here is that the agent delivers the command sequence together with a pre-check and the human executes it. That costs a minute and keeps the responsibility where it belongs.

How do you stop an agent from making up a cause?

Not completely, because producing plausible text is how it is built. What helps is asking for the measurement that led to the statement. Whoever asks for the concrete query and its return value gets either solid evidence or the information that it is a synthesis. Both are useful as long as they are kept apart. It also helps to have the agent actively look for the opposite, meaning the measurement that would refute its own explanation.

Parent articles:

Agentic Coding from a User’s Perspective — the sub-hub of this branch, where the control loop and the limits come together.
AI-Assisted SQL Development with Claude Code — the entry point to rules, skills and agents.

Sibling articles:

One VPS, Four Environments, No Cookie Banner — the setup in which the four environments share one machine and therefore also its RAM.
Two Agents, One Working Tree — why a red test run in a shared working tree is a hypothesis at first.
The Context Window Is Full Before the First Prompt — here the agent measures the product, there its own working material: what the project rules cost at the start of every session.

Downstream:

Set-Based vs. Row by Row: PostgreSQL Refresh from 114 to 12 Seconds — the fully measured case from the same project: a loop over 1,000 tables, its execution plan and what the rebuild cost.

Set-Based vs. Row by Row: PostgreSQL Refresh from 114 to 12 Seconds

Marcus — Mon, 28 Sep 2026 11:03:37 +0000

One click on "Refresh schema", and after 60 seconds the browser shows the error message 504 Gateway Timeout, because the web server in front of the application will not wait any longer for a response. On the server, the work continues undisturbed, only nobody can see it. So a second click follows, then a third, and shortly afterwards four runs are computing at the same time. A single run across 1,000 tables took a measured 114 seconds. The requirement was less than three seconds.

The cause was not a slow query but a loop that asked the source database for each table individually and stored the result individually. This article describes how that loop was measured and replaced with set-based statements, that is, statements that read and write all 1,000 tables at once, and which options were weighed against each other. It also describes what went wrong along the way. The first of those statements was slower than 1,000 single queries combined, and the finished result missed its targets despite a gain by a factor of 9 to 19.

The essentials up front:

A loop in application code costs more per iteration than the same loop inside the database: Whoever processes tables one at a time instead of with one statement over all of them pays a fixed overhead for every iteration, no matter how little data the iteration moves. If the loop lives in application code and opens its own database connection for each table, connection setup, authentication and a transaction are added per iteration. In the case described, one iteration cost around 114 milliseconds, which for all 1,000 tables adds up to the 114 seconds mentioned at the start. The same loop, rebuilt for this article as a procedure inside Postgres, cost around three milliseconds per table and thus around three seconds for the same 1,000 tables.
The loop was not carelessness, and its strengths had to be rebuilt: For each table it called a function that already existed for activating a single table and that worked correctly there. If one table failed, the others carried on, and even four concurrent runs left no contradictory data behind. With the 15 to 70 tables that the real databases of this application have, the runtime never stood out. The solution that processes all tables at once had to restore this error isolation explicitly.
One statement over all tables is not automatically faster than a thousand single ones: The first query that read the keys of all 1,000 tables at once from information_schema, the views in which Postgres exposes its own metadata, took around five seconds on Postgres 17. Issued once per table instead, the same query adds up to an extrapolated half a second. The execution plan showed why: Postgres 17 had chosen a plan for this query that re-evaluated one of the two views involved for every table, and so it had rebuilt the loop internally. A query directly on the system tables in pg_catalog behind those views took milliseconds, but it also showed every database user the keys of tables they are not allowed to read, and therefore needed a condition of its own for access privileges.
114 seconds became 12, and the targets were still missed: Those twelve seconds apply to a run that rewrites all 1,000 tables. The more common case is a run over an unchanged set of tables, and for that one the target was three seconds, which allows three milliseconds per table. The new logic takes around 3.7 milliseconds per table and is thus just under a quarter over budget. Across 1,000 tables that is 3.7 seconds. The run still takes up to 6.4 seconds. The target was missed somewhere else, because the missing 2.7 seconds are what every call on the shared server costs before it touches the first table, among other things for authentication, permission checks and reading the table selection. Next to 114 seconds of loop, this share had never stood out, and the estimate before the rebuild did not include it.

Prerequisite: Basic SQL and an idea of what a stored procedure is. The examples are PL/pgSQL and were run on Postgres 17 and 18 in throwaway containers. The case described comes from an application written in TypeScript, whose code is not shown here because the mechanics do not depend on the programming language.

The Starting Point: A Schema Snapshot Across 1,000 Tables

The private project DI² generates ETL pipelines from the metadata of a source database. To know what it is working with, it reads the columns, the primary and unique keys and the foreign keys of every activated table and stores them as a snapshot in its own database. A click on "Refresh schema" repeats this read for all tables of a connection.

This refresh was implemented as a loop. For each table, the application code called a function that already existed. It read the metadata of a single table from the source and replaced that table's snapshot in one transaction by deleting the old rows and inserting the new ones. The function had been written for a different occasion, activating a single table, and for that occasion it was correct. The loop on top of it was the obvious next step.

The specification had carried a target for the refresh for months: less than three seconds for 1,000 tables. Nobody had measured it, because the real databases in the project had between 15 and 70 tables. The measurement was made up for with a synthetic source database of 1,000 tables with 50 columns each. The tables were empty, because the refresh reads only the catalog, that is, the views and system tables in which Postgres describes which tables, columns and keys exist. The measurement ran on the project's development server, a shared machine with four virtual cores and 7.6 gigabytes of memory.

Measurement, code analysis and rebuild were done with a coding agent, which had also written the application code. The trade-offs and decisions this article tells about were made by the maintainer. What that division of labor looks like day to day is described in the article Agentic Coding from a User's Perspective.

The result of the measurement:

Metric	Value
Duration of the loop, single run without concurrent runs	114.4 s
Difference between two runs	6 ms
Duration per table	around 114 ms
Reading the table list, the only set-based step	22 to 43 ms
Target	under 3 s, missed by a factor of 38

None of this result reached the browser. The web server in front of the application ended the request after 60 seconds with a gateway timeout while the loop kept running on the server. Whoever clicked again started a second run next to the first. In the measurement session, up to four loops were running at the same time this way, and one of them delivered its response only after about seven minutes.

Measuring Runtime When the Web Server Times Out

Because the timeout cut off every measurement in the browser, the duration had to come from the data itself. The refresh replaces the snapshot of every table, and every newly written row receives a timestamp in created_on. After a run without concurrent runs, the span between the first and the last written row therefore gives the duration of the loop to within about one iteration, because the timestamp records the start of an iteration and not its end. The query for it needs no instrumentation at all:

  1: SELECT
  2:     count(DISTINCT table_name)        AS table_count
  3:    ,min(created_on)                   AS first_row
  4:    ,max(created_on)                   AS last_row
  5:    ,max(created_on) - min(created_on) AS duration
  6: FROM
  7:    meta.column_snapshot
  8: WHERE
  9:    schema_name = 'perf';

This calculation has a prerequisite that is easy to overlook. A default such as now() returns in Postgres the moment the current transaction began, not the moment of the insert. In the case study, every table ran in its own transaction, so every table got its own timestamp. If a procedure instead writes all tables in a single transaction, every row carries the same value, and the span is zero. A rebuild of the loop as a Postgres procedure, shown in the section "The Same Loop Inside the Database", confirmed exactly that. With one COMMIT per table, the query returned the duration of the loop, 5 milliseconds less than the stopwatch on the call, which is the last iteration, whose end no timestamp records anymore. Without COMMIT it returned 00:00:00. Whoever needs the moment of every single row sets clock_timestamp() as the default. statement_timestamp() does not help here, because it returns the start of the statement the client sent, and for the whole procedure that is the CALL.

Two independent runs came to 114.439 and 114.445 seconds and were thus practically identical. A statement about the distribution would have needed more runs, but for finding the cause, the order of magnitude was enough. Since the rebuild, the application measures its own duration and writes it to the audit log, so the detour via the timestamps is no longer needed. That instrumentation was itself one of the consequences of the analysis. Why a query on the stored data often leads to a finding faster than clicking through the user interface is shown in the article The Agent Measures Where I Click.

Where the 114 Seconds Went

Per table, the following happened in sequence:

Three connections to the source database: The code opened one connection each for columns, keys and foreign keys. Before each of these connections it loaded the connection profile from its own database and decrypted the password. Then it established a fresh TCP connection, authenticated, issued a single catalog query for exactly this table and closed the connection again. For 1,000 tables, that is 3,000 connection setups per refresh.
Up to four procedure calls in its own database: One after the other, the column snapshot was replaced, the derived validation rules were reconciled, the key snapshot was replaced and the foreign-key snapshot was replaced.

The source database ran on the same machine, and the network between the containers costs less than a millisecond. The 114 milliseconds per table therefore consisted almost entirely of costs that every iteration pays regardless of the amount of data. How exactly they split between connection, profile and procedure calls, the measurement did not record. A separate measurement with pgbench between two containers gives the order of magnitude. There, one connection setup with password authentication cost 17 milliseconds on average, while a simple query over an existing connection cost 0.36 milliseconds. Postgres starts a separate server process for every new connection, and exactly this overhead was incurred three times per table in the loop. Three connection setups thus explain around 50 of the 114 milliseconds, and the rest is spread over loading and decrypting the profile and the four procedure calls.

The counter-evidence comes from the only step of the refresh that was already set-based, meaning it asked its question in a single statement for all tables instead of once per table. The list of all 1,000 tables came back in 22 to 43 milliseconds, which is less than 0.04 percent of the total time.

The target also yields a budget, and this calculation settles the question earlier than any optimization attempt. Three seconds for 1,000 tables come to three milliseconds per table. A single connection setup alone costs a multiple of that. As long as every iteration opens a connection, the target is therefore out of reach, no matter how fast the queries inside it are.

The Same Loop Inside the Database

The case study plays out in application code, but the pattern is the same as with a cursor in a procedure that works through a result set row by row. Among SQL developers it goes by the mocking name RBAR, short for "Row By Agonizing Row". In the world of ORMs it is known as the N+1 problem: first one query for the list, then another one for every element. To see which part of the cost is tied to the loop itself and which to the connections, the snapshot was rebuilt inside Postgres for this article. It reads from information_schema, the standardized set of views through which most SQL databases describe their tables and columns. The target table takes one row per column:

  1: CREATE SCHEMA IF NOT EXISTS meta;
  2: 
  3: CREATE TABLE meta.column_snapshot
  4: (
  5:     schema_name               text          NOT NULL
  6:    ,table_name                text          NOT NULL
  7:    ,column_name               text          NOT NULL
  8:    ,data_type                 text          NOT NULL
  9:    ,ordinal_position          int           NOT NULL
 10:    ,created_on                timestamptz   NOT NULL DEFAULT now()
 11:    ,created_by                text          NOT NULL DEFAULT current_user
 12:    ,PRIMARY KEY (schema_name, table_name, column_name)
 13: );

The sequential version reads the tables of a schema and writes each table's snapshot in its own transaction, the way the case study did. The COMMIT inside the loop requires that the CALL is not inside an open transaction block:

  1: -- --------------------------------------------------------------------------------
  2: -- Parameters
  3: -- --------------------------------------------------------------------------------
  4: --    p_schema_name        text
  5: --       Schema whose tables are taken into the snapshot
  6: -- --------------------------------------------------------------------------------
  7: CREATE OR REPLACE PROCEDURE meta.sp_load_column_snapshot_loop
  8: (
  9:     IN    p_schema_name        text
 10: )
 11: LANGUAGE plpgsql
 12: AS $procedure$
 13: DECLARE
 14:    l_table_name              text;
 15: BEGIN
 16: 
 17:    FOR l_table_name IN
 18:       SELECT
 19:          table_name
 20:       FROM
 21:          information_schema.tables
 22:       WHERE
 23:          table_schema = p_schema_name
 24:       ORDER BY
 25:          table_name
 26:    LOOP
 27:       -- --------------------------------------------------------------------------------
 28:       -- One table: delete the old snapshot, write the new one
 29:       -- --------------------------------------------------------------------------------
 30:       DELETE FROM
 31:          meta.column_snapshot
 32:       WHERE
 33:              schema_name = p_schema_name
 34:          AND table_name  = l_table_name;
 35: 
 36:       INSERT INTO meta.column_snapshot
 37:       (
 38:           schema_name
 39:          ,table_name
 40:          ,column_name
 41:          ,data_type
 42:          ,ordinal_position
 43:       )
 44:       SELECT
 45:           table_schema
 46:          ,table_name
 47:          ,column_name
 48:          ,data_type
 49:          ,ordinal_position
 50:       FROM
 51:          information_schema.columns
 52:       WHERE
 53:              table_schema = p_schema_name
 54:          AND table_name   = l_table_name;
 55: 
 56:       -- Each table is written in its own transaction
 57:       COMMIT;
 58:    END LOOP;
 59: END;
 60: $procedure$;

The lines in detail:

Lines 17 to 54: The loop runs over the table list from information_schema.tables, and for each table a DELETE and an INSERT follow. That is the N+1 pattern: one query for the list, N more for the work.
Line 57: The COMMIT ends the transaction after every table and thus rebuilds per-table atomicity. Without this line, the whole loop would run in a single transaction, and an error at table 500 would also roll back the 499 finished tables. The timestamp measurement from the previous section would then give zero, because now() returns the start of the transaction and stays constant within it. Procedures may COMMIT since Postgres 11, functions may not, and even a procedure may do so only as long as the caller holds no transaction open. If the caller opens one, for instance through a library that wraps every call in BEGIN and COMMIT, the procedure fails at the first COMMIT with invalid transaction termination. What the COMMIT does not provide is the error isolation of the case study. This procedure has no error handling, and an error at table 500 ends the call. The 499 finished tables stay written, the remaining 500 stay unprocessed.

The set-based version does the job with two statements over the whole schema and therefore needs no loop. The difference sits in the WHERE clause: only the schema remains there, no longer the single table.

  1: -- --------------------------------------------------------------------------------
  2: -- Parameters
  3: -- --------------------------------------------------------------------------------
  4: --    p_schema_name        text
  5: --       Schema whose tables are taken into the snapshot
  6: -- --------------------------------------------------------------------------------
  7: CREATE OR REPLACE PROCEDURE meta.sp_load_column_snapshot
  8: (
  9:     IN    p_schema_name        text
 10: )
 11: LANGUAGE plpgsql
 12: AS $procedure$
 13: BEGIN
 14: 
 15:    DELETE FROM
 16:       meta.column_snapshot
 17:    WHERE
 18:       schema_name = p_schema_name;
 19: 
 20:    INSERT INTO meta.column_snapshot
 21:    (
 22:        schema_name
 23:       ,table_name
 24:       ,column_name
 25:       ,data_type
 26:       ,ordinal_position
 27:    )
 28:    SELECT
 29:        table_schema
 30:       ,table_name
 31:       ,column_name
 32:       ,data_type
 33:       ,ordinal_position
 34:    FROM
 35:       information_schema.columns
 36:    WHERE
 37:       table_schema = p_schema_name;
 38: END;
 39: $procedure$;

This also removes the COMMIT per table, because both statements run in the same transaction. In return, it gives up per-table atomicity: an error rolls back the whole run and not just the table being processed. The two versions have a second difference: the set-based one also deletes rows for tables that no longer exist, while the loop leaves them in place, because it only runs over the tables that are present.

On Postgres 18.6 in a Docker container on a laptop, the set-based version took around 0.6 seconds in three runs on a freshly created table, the loop around 3 seconds. Both versions wrote exactly the same 50,000 rows, and a comparison with EXCEPT ALL over the five business columns, that is, without the timestamps, showed no difference in either direction. On Postgres 17 the times varied more, and the ratio was similar. The laptop numbers serve as an order of magnitude, not as a benchmark, because with repeated overwriting without VACUUM both versions became noticeably slower.

A factor of five (3 seconds ÷ 0.6 seconds) is significant. It is still a long way from what the case study experienced, because there one iteration cost 114 milliseconds instead of three. The difference is made by the overhead that every iteration there paid on top of the actual work: three connection setups, three authentications, loading and decrypting the profile and four separate procedure calls. So a loop is not expensive simply because it is a loop. It becomes expensive in proportion to what an iteration costs beyond the work, and that share grows with every connection, every process and every transaction that sits between the loop and the data.

What Spoke for the Loop

Before the rebuild, the loop deserves a fair assessment, because it was not carelessness. Its advantages were real, and some of them the new solution had to rebuild explicitly.

Reuse: The function for a single table already existed and was tested. A loop on top of it brought neither a new code path nor a new class of errors.
Per-table error isolation: If table 738 fails because of a missing permission or a concurrent DROP TABLE, the application code catches the error, increments an error counter, and the remaining 999 tables carry on. This property comes from the loop's error handling and not from the transaction per table that the next point describes.
Per-table atomicity: Delete and re-insert ran in a transaction of their own for every table, so that even under concurrency no half snapshot of a single table ever came about. It does not guarantee a common state of all 1,000 tables from one and the same run, but the snapshot does not need that either, because every table is read on its own and used on its own. In the measurement session this proved itself: of seven runs, five overlapped, not one reported an error, and in the end exactly 50,000 rows for 1,000 tables were in place.
Constant memory footprint: The application never held more than one table in memory.
Readability: The control flow was an ordinary loop that anyone understands on first reading.
Inconspicuous at small scale: A database with 48 tables would have taken an extrapolated five to six seconds. That is a noticeable wait, but not an alarming one.

The problem came with scale, not with the original decision.

What Spoke Against the Loop and What the Switch Costs

Four points spoke against the loop:

Fixed cost times count: Connection, authentication, profile and procedure call are incurred in every iteration, while the actual amount of data per table is tiny. More than 99.9 percent of the runtime sat in the loop over the tables.
The catalog answers set questions at almost the price of a single question: In the rebuild, information_schema.columns delivered all 50,000 columns in around 0.3 seconds. The columns of a single table took 6 to 15 milliseconds as a query of their own, and repeated a thousand times that is 6 to 15 seconds. The largest part of that is planning the view query, on Postgres 18.6 around 5 milliseconds against around one millisecond of execution. In a PL/pgSQL procedure this planning does not happen on every iteration: the procedure creates a prepared statement on the first iteration and switches after a few iterations to a cached plan that does not depend on the concrete values, provided that plan is not much worse. That is why the loop in the rebuild gets by with around three milliseconds per table. With the setting plan_cache_mode = force_custom_plan, which re-plans on every iteration, the same loop took around two and a half to three seconds longer.
Set-based writing spreads the cost over many rows: In the same project, a single INSERT into a different table wrote around 91,000 rows in about one second, as an order of magnitude and not as a comparison under equal conditions. The loop moved less data with its 50,000 rows and took 114 seconds for it.
Collision with timeouts: A synchronous operation that takes longer than the timeout of the web server in front of it produces an error message behind which the work continues. Users click again, and the runs pile up.

The switch to set-based processing, however, has a price of its own:

The batch size becomes a decision in its own right: A batch is a single call of a procedure that accepts the data of many tables at once, in the case described as a JSON document with all columns, keys and foreign keys of those tables. How many tables belong in it, someone has to decide, and before the switch the question did not arise at all. The amount of data per call has to fit the memory limits of the application and the parameter limits of the database drivers.
Errors hit larger units: If a batch fails, that affects all tables in it. The per-table count has to be restored actively.
Transactions take longer: A batch holds its locks longer than many small transactions do.
More code has to be right at once: Each of the three supported engines, PostgreSQL, SQL Server and MySQL, needs its own set-based queries against its catalog. The new batch procedures are new objects with tests of their own.
Equivalence has to be proven: The rebuild must not change the stored result, so a before-and-after comparison of the stored data becomes mandatory. Every difference has to be explainable: unexplained differences are errors of the rebuild, explained ones may be errors of the old code.

The Decision: Four Options and a Budget Calculation

With the budget of three milliseconds per table, the options were quick to sort:

Option	Change	Expectation
A: Set-based queries and batch writes	three to four catalog queries for all tables over one connection, batch procedures instead of single calls	2.5 to 5 seconds, the only way towards the target
B: Reuse the connection	one connection and one profile per run or a connection pool, the loop stays	factor 2 to 4, that is 30 to 60 seconds, three queries and four procedure calls per table remain
C: Parallelize	several workers work through the loop	linear gain at higher load on the source, no target without A
D: Drop the target	the refresh is rare and triggered by hand	does not solve the five to six seconds on real databases

Option A was chosen, supplemented by detection of unchanged tables. A table whose columns, keys and foreign keys have not changed since the last refresh is not rewritten at all.

Two rulings were part of the decision. First, the loop was replaced and not left standing as a second path next to the new solution. Activating a single table has since run through the set-based path as well, with a set of exactly one element. The function from whose reuse the loop once arose no longer exists. Second, the blanket target was replaced with three separate targets after the server's hardware had been checked: under three seconds when nothing has changed, under ten seconds when all 1,000 tables are rewritten, and under two seconds for a realistic 50 tables.

How a Refresh Runs Today

This is what the whole process looks like today in one piece, described for a click on "Refresh schema" across all 1,000 tables. For a single table, exactly the same path runs, then with a set of one element.

Read the catalog: The application opens a single connection to the source database and asks four questions, each over all 1,000 tables at once: the columns, the type facts it needs to classify unknown data types, the unique and primary keys, and the foreign keys. After that, the complete catalog sits in the application's memory. Tables for which the catalog returns not a single column are sorted out here and reported as failed, instead of producing an empty snapshot.
Read the stored checksums: A single SELECT on the application's own database fetches the last stored checksum for all of these tables. That is a different server than in step 1. There the foreign source was queried, here the application's own data.
Compare: For each table, the application computes a SHA-256 checksum over exactly the data it would store and compares it with the stored one. If no stored checksum exists, the table counts as changed. The result is two groups.
- Unchanged: Nothing is written for these tables. They only receive the stamp "last verified against the source", through a single procedure call for the whole group.
- Changed: These tables are cut into batches of 100. The number is a constant in the application code and not a setting, and if it does not divide evenly, the last batch is smaller. For each batch, one call goes to the database, with the complete data of the 100 tables as a JSON document in the parameter. The procedure behind it contains no loop of its own. It unpacks the document and writes columns, keys and foreign keys of all tables in the batch with one DELETE and one INSERT each. Every batch is a transaction of its own. For 1,000 changed tables, that is ten calls.
Catch errors: If a batch fails, the others carry on. The tables of the failed batch are then re-run one by one, through the same path with a set of one element. Only if the error would hit every further call anyway, for instance because the data model has meanwhile been frozen or the connection has been deleted, does the run abort instead of trying again for every single table.
Update the validation rules: Finally, the validation rules derived from keys are updated, and only for the tables in the "changed" group.

From 3,000 connection setups and 4,000 single calls, this leaves one connection, four catalog queries and roughly ten to eleven procedure calls.

How the New Solution Brings Back the Loop's Strengths

Of the six strengths from the section "What Spoke for the Loop", the new solution had to rebuild two explicitly. For the others, the balance is mixed.

Per-table atomicity comes back through the batches. Every batch runs in a transaction of its own, and because the boundary between two batches always lies between two tables and never in the middle of one, no half snapshot of a table comes about even under concurrency. A batch does, however, hold its locks longer than the many small transactions of the loop. So that two overlapping runs do not end up in a deadlock, with each waiting for rows the other holds, the procedure locks the table entries of a batch at the start in a fixed order, with SELECT … ORDER BY id FOR UPDATE. The second run still has to wait until the first one's batch is complete, and it then works on that state.

Per-table error isolation comes back as a fallback level, as step 4 of the process shows: if a batch fails, the others carry on, and its tables are re-run one by one, so that in the end the count is per table again. The first version did not distinguish two cases here. It treated a single faulty table the same as a cause that hits all tables, such as a connection that has meanwhile been locked. In that case all ten batches failed, and the re-runs produced another 1,000 calls that failed just the same. The finished version recognizes such errors by their message and aborts the re-run.

The constant memory footprint is changed, not preserved. After step 1, the catalog of all tables sits completely in the application's memory, whereas the loop never held more than one table. What is bounded instead is the size of a single write call: the 100 tables per batch are the result of a trade-off between the size of the document passed in, the duration of the transaction and the time a batch holds locks. A particularly wide table merely makes its batch larger.

Reuse and readability the new solution has given up. The function for a single table no longer exists, and the set-based path is more code, spread across three engines. The inconspicuousness at small scale is preserved but turns out weaker than hoped: 48 tables now take 2.9 to 4.4 seconds instead of an extrapolated five to six. The largest part of that is a fixed base cost, which the results section breaks down.

Skipping Unchanged Tables

For each table, the application computes a checksum over exactly the data it would store and compares it with the checksum of the last run. If both match, the table is not rewritten. It only receives a stamp with the time of the check, so that it stays visible that it was verified against the source.

Set-Based Is Not Automatically Fast: Why a Query Over All Tables Can Be Slower

The first set-based version read the keys of all tables through the two standard views information_schema.table_constraints and information_schema.key_column_usage. That is the portable version over the standard views, and filtered to a single table it had run in the project for years without attracting attention:

  1: SELECT
  2:     T01.table_name
  3:    ,T01.constraint_name
  4:    ,T01.constraint_type
  5:    ,T02.column_name
  6:    ,T02.ordinal_position
  7: FROM
  8:    information_schema.table_constraints T01
  9:    INNER JOIN information_schema.key_column_usage T02
 10:    ON
 11:          T02.constraint_schema = T01.constraint_schema
 12:      AND T02.constraint_name   = T01.constraint_name
 13:      AND T02.table_name        = T01.table_name
 14: WHERE
 15:        T01.table_schema    = 'perf'
 16:    AND T01.constraint_type IN ('PRIMARY KEY', 'UNIQUE')
 17: ORDER BY
 18:     T01.table_name
 19:    ,T01.constraint_name
 20:    ,T02.ordinal_position;

The join runs over schema and constraint name and additionally over the table name in line 13. That condition is necessary because in Postgres a foreign key may carry the same name as the unique constraint of another table in the same schema. Without it, key_column_usage would attribute the columns of that foreign key to the key. Between primary and unique keys the case cannot occur, because the index behind them claims the name within the schema.

Before deployment, the query was measured on Postgres 17. For 1,000 tables it took 4.7 to 5.1 seconds. The same query, issued for each table individually, came to 0.35 to 0.51 seconds extrapolated from a sample of 25 tables. The set-based query was thus around ten times slower than the single queries it was meant to replace, and it would have used up the gain of the entire catalog read.

EXPLAIN ANALYZE showed why. Both views are themselves queries over several system tables, with privilege checks and type conversions. The planner, that is, the part of Postgres that chooses the execution path for every query, had chosen a Nested Loop for this query, a join that walks through the other side again for every row of one side. In this plan, the join condition sat on the Nested Loop itself and not inside the subplan of the view key_column_usage, and so it executed that subplan 1,000 times, once per constraint, which for 1,000 tables with one primary key each means once per table, and each time it read the entire system table pg_constraint. In the plan, that shows up as loops=1000 on exactly this subplan. The timings on such a node are averages per iteration and have to be multiplied by the loops= value, otherwise the most expensive node in the plan looks harmless. So the loop had not disappeared from this plan, it had moved from the application code into the execution plan. As a cross-check, SET enable_nestloop = off brought the same query down from 4.7 seconds to 48 milliseconds. That is a diagnosis and not a solution for production. It does not prove that a Nested Loop would be wrong in principle, only that the chosen plan was expensive for these data volumes.

The rebuild for this article confirms the behavior and at the same time shows that it depends on the version. On Postgres 17.10 the query took 3.9 to 4.8 seconds, on Postgres 18.6 only 29 to 34 milliseconds, because the planner chooses a Hash Join there. That is no blank check for information_schema, though, because a join of the columns view with information_schema.tables ran into the same pattern on 18.6 as well and took over 99 seconds. That is why the procedure in the previous section does without that join.

The solution in the project was a query directly on pg_catalog, the Postgres system tables from which the information_schema views themselves read. It connects pg_constraint through the column array conkey with pg_attribute and, like the view version, restricts itself to primary and unique keys:

  1: SELECT
  2:     T02.relname                       AS table_name
  3:    ,T01.conname                       AS constraint_name
  4:    ,T01.contype = 'p'                 AS is_primary_key
  5:    ,T04.attname                       AS column_name
  6:    ,T03.ordinal_position::int         AS ordinal_position
  7: FROM
  8:    pg_catalog.pg_constraint T01
  9:    INNER JOIN pg_catalog.pg_class T02
 10:    ON
 11:      T02.oid = T01.conrelid
 12:    CROSS JOIN LATERAL unnest(T01.conkey) WITH ORDINALITY AS T03 (attnum, ordinal_position)
 13:    INNER JOIN pg_catalog.pg_attribute T04
 14:    ON
 15:          T04.attrelid = T01.conrelid
 16:      AND T04.attnum   = T03.attnum
 17: WHERE
 18:        T02.relnamespace = 'perf'::regnamespace
 19:    AND T01.contype      IN ('p', 'u')
 20:    -- visibility as in information_schema.columns
 21:    AND (
 22:           pg_has_role(T02.relowner, 'USAGE')
 23:        OR has_column_privilege(T02.oid, T04.attnum, 'SELECT, INSERT, UPDATE, REFERENCES')
 24:        )
 25: ORDER BY
 26:     T02.relname
 27:    ,T01.conname
 28:    ,T03.ordinal_position;

The condition in lines 20 to 24 belongs to the next section. The query returned the same rows, the comparison with EXCEPT ALL showed no difference, and the runtime dropped in a direct comparison run from 4.0 seconds to 13.6 milliseconds. In the rebuild it took around 30 to 60 milliseconds on both Postgres versions. Whoever joins catalog views should therefore look at the execution plan before taking the set-based version for the faster one. The telltale sign is a subplan that reads an entire system table and carries a loops= value equal to the number of rows on the outer side, here the number of tables. The high loops= value alone is not yet a finding, because a Nested Loop that hits an index on the inner side is often the fastest plan. The query on pg_catalog applies only to Postgres. For SQL Server and MySQL, the project has its own catalog queries anyway, there through the sys views and through information_schema respectively, and the problem did not occur there: on both engines, the set-based version was faster than the single queries from the start.

The Privilege Trap When Switching to pg_catalog

The switch to pg_catalog has a side effect worth knowing. information_schema.table_constraints shows only tables that the current role owns or on which it has some privilege other than SELECT. A pure read-only role sees not a single key there, while information_schema.columns shows it all columns. The source connections in the project work with exactly such read-only roles, and that is why the key snapshots of all Postgres sample connections had been empty from the start. Nobody had noticed, because an empty list does not look like an error. The rebuild fixed this defect as a side effect.

pg_catalog has the opposite problem: the system tables pg_constraint and pg_attribute do not filter by the role's privileges. A role therefore sees the constraint definitions there even for tables whose data it is not allowed to read. That is why the new query carries, in lines 20 to 24, a condition of its own that reproduces the visibility rule of information_schema.columns, and it did not become measurably slower for it. It takes only the privilege check from that view. The view's remaining filters, such as those against dropped columns or temporary schemas of other sessions, the key query does not need, because a key cannot contain a dropped column and the schema is fixed in advance. So the condition is not a general replacement for the visibility logic of information_schema, and whoever rebuilds a different catalog view on pg_catalog has to carry over its conditions one by one. Whoever derives validation rules from keys, as the article Deriving Data Quality Rules from the Schema describes, should know which role the query runs under.

A second finding belongs to the set-based read itself. The catalog reader creates an entry for every requested table. If the query returns no row for a table, because it no longer exists or the role is not allowed to see its columns, that entry stays empty, without any error message. The code would have turned that into a snapshot without columns and, during reconciliation, deleted the derived validation rules of that table. The code analysis before deployment found this. In a set, a missing element does not stand out, and whoever reads a set therefore has to check for themselves whether something is missing.

The Result: Factor 9 to 19, Targets Still Missed

The measurement after deployment ran on the same server and against the same source database with 1,000 tables:

Scenario	Before	After	Target
1,000 tables, all rewritten	114.4 s, timeout after 60 s	12.1 s	under 10 s
1,000 tables, unchanged	–	5.8 to 6.4 s	under 3 s
48 tables, all rewritten	extrapolated 5 to 6 s	4.4 s	under 2 s
48 tables, unchanged	–	2.9 to 3.1 s	under 2 s
Idle, no stored tables	–	2.7 s	–

Instead of 3,000 connections and 4,000 single calls, the refresh now makes three to four catalog queries over a single connection and about ten batch calls. The run with a full rewrite is around nine times faster than before, the run over an unchanged set of tables around 19 times, where this factor measures both changes together, the set-based processing and the skipping of unchanged tables. No run reaches the timeout anymore, and in all nine measurement runs not a single snapshot failed.

All three targets were still missed. The budget estimate before the rebuild had predicted 0.5 to 1.5 seconds for the unchanged run and 2.5 to 5 seconds for the full rewrite. The explanation is in the last row of the table. A refresh over a connection without stored tables reads only the table list and writes nothing, and it still takes 2.7 seconds. That is the fixed base cost of every call to the application server. It covers, among other things, authentication and permission checks, reading the table selection twice, before and after writing, and preparing the response, on a machine that shares its memory with other services. Only the sum was measured, the shares were not recorded individually. On top of this base, the new logic adds around 3.7 milliseconds per table in the unchanged run, across 1,000 tables therefore 3.7 seconds. Both parts together give the upper end of the measured 5.8 to 6.4 seconds (2.7 + 3.7). The base is a lower bound taken from the idle run, because reading the table selection grows with the number of tables. A run that rewrites all 1,000 tables takes more per table than these 3.7 milliseconds, because it additionally deletes and re-inserts 50,000 rows.

Before this explanation was allowed to stand, an obvious alternative was ruled out. Two gigabytes were sitting in swap on the machine. Swap was cleared and the measurement repeated, with practically the same result. That largely ruled out swap as the cause, while other influences of the shared machine remained open. That swap on this same server had once before been the cause of a sluggish application is described in the article The Agent Measures Where I Click. The shared server itself is introduced in the article One VPS, Four Environments, No Cookie Banner.

As long as the loop took 114 seconds, 2.7 seconds of base cost made up a good two percent of the runtime, and nobody noticed them. After the rebuild, in the unchanged run, they are a good 40 percent. Whoever removes the cost per element exposes the fixed costs, and a budget calculation should therefore include them from the start. Lowering the base further would have meant work outside the snapshot code. The decision was to document the deviation and not to optimize further, because the actual problem was gone: there are no more runs lasting minutes, no timeout and no piled-up clicks.

When an Optimization Changes What a Display Means

Skipping unchanged tables has two consequences that have nothing to do with speed.

The first concerns the timestamp. Every snapshot row carries the time at which it was written, and the table register derives a table's "as of" date from it. As long as every refresh rewrote every table, that moment was also the moment of the last check against the source, because both happened at the same instant. With skipping, the two come apart. An example: a table was last rewritten three weeks ago and has not changed in the source since. Today's refresh checks it, finds it unchanged and skips it. Its snapshot rows therefore keep the timestamp from three weeks ago. If the register kept showing that timestamp as the "as of" date, a table that was checked a minute ago would display a three-week-old date, and the user would have to assume the refresh had passed it over. That is why there have been two points in time since the rebuild. The snapshot timestamp now moves only when something has actually changed at the source, and so it becomes the change date. The time of the last check is written by the refresh as a stamp of its own, also for every skipped table.

The second concerns the success message. Before, it reported how many snapshots had been updated, and with 1,000 tables it always said 1,000. After the rebuild, "nothing changed" is the normal case, the number of rewritten tables is then zero, and the existing message would have shown "no changes". The most expensive kind of check, in which 1,000 tables are verified live against the source, would thus have produced exactly the same message as a run over an empty connection. The message therefore states both numbers, that is, how many tables stayed unchanged and how many were rewritten.

Whoever replaces a loop with set-based statements and skips work in the process should also check the displays and messages that rely on the old behavior. Otherwise "a lot was checked" silently turns into "nothing was found", and the user loses exactly the confirmation they clicked for.

What Remains

An iteration costs its work plus everything it additionally crosses. Inside the database, an iteration cost around three milliseconds, across connection and application 114. The useful question is therefore not whether a loop is allowed, but what every iteration pays beyond the actual work and how often it runs.
The budget per element comes before any optimization. Three milliseconds per table settled connection reuse as the sole solution before anyone built it. The budget has to include the fixed costs, otherwise it misses the target in a place where nobody is looking.
After the rebuild, the execution plan goes on the table. The first set-based query was slower than the single queries because the planner executed it as a Nested Loop over an entire system table. A loops= value equal to the number of outer rows is the cue to look at the affected subplan, and it gets expensive when that subplan reads an entire system table on every iteration.
An equivalence proof also finds errors in the old code. The comparison before and after the rebuild uncovered a privilege gap through which empty key snapshots had been stored for years.
Skipped work changes what displays mean. Timestamp and success message had to be redefined, although nothing in their code had changed.

FAQ

Is a loop in SQL always slower than a set-based statement?

No. Set-based means the same as set-oriented: one statement over all rows instead of processing them row by row. It is often faster, because the database can plan the work over the whole set together and fewer fixed costs are incurred per element, but that is no guarantee. How large the gap is depends on what an iteration costs. Inside the database, the loop in the rebuild was around five times slower, and with a few dozen elements that hardly matters. It gets considerably more expensive when every iteration brings its own connection, transaction or network call. If the planner chooses a plan for the set-based statement that internally executes it element by element again, it is even slower than the loop, as the case study shows.

What does the N+1 problem have to do with cursor loops?

Both process a set element by element and differ in how many system boundaries each element crosses along the way. With the N+1 problem, an ORM first loads a list and then issues a separate query for every element, each of them a separate trip to the database. A cursor loop in a procedure, by contrast, repeats the work inside the database, without additional queries from outside. The case study shows a third variant, in which application code even opens a new connection per element. Every boundary between the loop and the data, that is, a connection, a process or a transaction, raises the fixed cost of an iteration.

Why is a join over information_schema views sometimes so slow?

The views in information_schema are themselves queries over several system tables. If you join two of them, it can happen that the Postgres query planner cannot push the join condition into the views and evaluates one view completely for every row of the other. In the execution plan you recognize this by a Nested Loop whose inner subplan reads an entire system table and carries a loops= value equal to the number of outer rows, here the number of tables. Whether it happens depends on the query and the version: the key query was affected on Postgres 17 and not on Postgres 18, the join of columns and tables was affected on both. The remedy is a query directly on pg_catalog or a single view without a join.

Why does a read-only role see no keys in information_schema.table_constraints?

This view shows only tables that the role owns or on which it has some privilege other than SELECT. information_schema.columns, by contrast, shows the columns already with SELECT. A role that may only SELECT on a table therefore sees all of its columns but none of its primary or unique keys. Whoever needs the keys for such a role reads them from pg_constraint and checks visibility themselves. The query in the case study reproduces the rule of information_schema.columns and not that of table_constraints: it shows a role the keys if the role owns the table, checked with pg_has_role, or has a privilege on the column, checked with has_column_privilege.

How does a loop keep running after an error instead of aborting?

The COMMIT alone is not enough for that. It secures what has already been written, but it still ends the call at the first unhandled error. The loop can only keep running if every iteration gets its own BEGIN … EXCEPTION block that catches the error and increments a counter. When the block catches an error, Postgres has already rolled back the changes of that iteration, because a block with error handling runs as a subtransaction. The COMMIT after it then has nothing left to commit for that table, and no half snapshot can come about. There is a trap here: the COMMIT has to go after this block, not inside it. Inside a block with error handling it fails with cannot commit while a subtransaction is active, and a WHEN OTHERS handler swallows exactly this error silently. The procedure then runs through without error and writes nothing.

How do you measure the duration of a loop when the browser gives up first?

The duration can be read from the written data. If every iteration writes its rows in a transaction of its own and stamps them with now(), the span between the smallest and the largest timestamp after an undisturbed run gives the duration of the loop up to the last iteration. If everything runs in a single transaction, now() returns the same value everywhere, and the column then needs clock_timestamp() as its default. In the long run this is no substitute for instrumentation, but it works after the fact and without any code change.

Downstream:

Design Pattern // The Architecture of an ETL Process — at which stage of an ETL pipeline catalog reads like this one have their place.
Deriving Data Quality Rules from the Schema — how mandatory fields, keys and type limits from information_schema turn into validation rules.
Reading Complex XML in SQL Server — Shredding a Thousand Attributes Generically Instead of Looping for Hours — the same idea applied to an XML document: shred generically instead of looping per object type and attribute.

From the same project:

The Agent Measures Where I Click — four debugging sessions in which a measurement replaced the guess.
One VPS, Four Environments, No Cookie Banner — the shared machine on which these measurements ran as well.
Agentic Coding from a User's Perspective — how working with a coding agent changes the daily routine.

Two Agents, One Working Tree — What Broke When I Ran Parallel Claude Code Sessions in One Git Repository

Marcus — Sat, 12 Sep 2026 06:46:26 +0000

A Claude Code session opens a Git repository with no pending changes, works for half an hour and commits whatever has changed at the end. Run alone, that is correct. As soon as a second session or a human works in the same repository, the very same routine can pull someone else’s work into the commit, and Git does not warn about it.

This article tells five real cases from a little over a week in which parallel Claude Code sessions shared the same repository, sometimes with each other, sometimes with the maintainer. Three of the five cases were the agent’s mistakes. The agent noticed and reported two of them itself. The third was only discovered by the neighbouring session whose work had vanished. The cases yield countermeasures, and the most obvious one is precisely the one that does not hold.

The essentials up front:

The common root: A session sees a repository with no pending changes when it starts. If a new or modified file appears during its run because a parallel session or the human has just created or changed it, that looks to the session exactly like a file its own tooling produced. So it treats the foreign file like its own and either commits it or cleans it away. The assumption „everything new comes from me“ is never contradicted, because Git records no author for changes in the working directory.
The index is shared state: Git collects changes for the next commit in a staging area called the index, and git add puts them there. When several sessions work in the same working directory, they share its index too. Adding only your own file with git add <file> instead of git add -A does not protect you, because git commit without a path commits the entire index, including whatever another session has already put there.
Destructive operations are never cleanup in parallel operation: Whoever resets a file to its previous version, discards a state or deletes a file hits everything under the given path, not just their own work. The judgement „that was me“ is an assumption that can be checked, not an observation.
Shared counters collide: If two sessions assign the next sequential number at the same time, say for a new bug entry, both read the same highest number and both assign the same new one. If both then politely step aside to the next number, they collide again. Checking beforehand alone does not prevent this, because the window lies between reading and writing.
The machine is shared too: Parallel sessions share not only the repository but the computer the human is working on at the same time. An agent that starts 220 endless loops for a load test sees only exit codes afterwards, not the desktop that stutters for twelve minutes on the human’s side. These costs have no feedback loop to the agent, and with several sessions running the human cannot even tell which one is causing them.
What holds: Against foreign index entries, git commit --only <paths>, a look at the index as a separate step before the commit and a git show --stat afterwards hold up. Shared counters need a tie-break rule agreed in advance, a right-of-way rule for the tie that decides without negotiation which side steps aside, plus a check after writing that compares the number list against the files on disk instead of just looking for duplicate numbers. None of these commands protects against simultaneous changes to the same file. Where the workflow allows it, each session therefore gets its own Git worktree.
The rule came after the damage: The countermeasures have been in a rule file of the blog repository since the second case. They are a consequence of the cases, not a precondition.

Prerequisite: Basic Git knowledge is assumed, prior Claude Code experience is not. The two Git terms that carry the article, working tree and index, are briefly explained in the first section. The cases come from two repositories, the private app project DI² and the repository of this blog, all between 28 August and 4 September 2026.

Why Parallel Claude Code Sessions Produce Errors That Don’t Exist Alone

A coding agent session has a clear view of its task and a blind spot for its surroundings. Two Git terms come up again and again in this article and deserve a short introduction. Git calls the files that sit on disk and that it compares against the last commit the working tree. The index is the staging area for the commit: with git add you put the changes there that the next commit is supposed to contain. The session sees the working tree when it starts, it sees the output of its commands, and at the end it sees which files have changed. What it does not see is whether someone else has used the same working tree in the meantime. Git does not record which process or which session caused a change in the working tree. In normal operation every working tree has its own index, and whoever works in the same working tree shares that index with everyone else there. A second session or a human with an open editor leaves traces in it that cannot be told apart from your own.

From this follows an assumption that is correct when a session runs alone and wrong when it does not: „The working tree was clean when I started, so everything new comes from me.“ Whether it turns into a problem depends on timing, not on how careful the session is, and none of it feels any different to the session than the normal case.

All cases in this article go back to the same cause. With every command, a session only thinks about the files it touched itself. Git and the shell do not know that boundary: git commit takes the whole index, a checkout or a delete command acts on everything under the given path no matter who changed it, and a load test occupies the whole machine. As long as a session works alone, this goes unnoticed, because everything that changed came from it anyway. As soon as a second session or a human works in the same repository, that is no longer true, and the session does to foreign work exactly what it would do to its own. That is where the errors come from that could not occur at all without a second actor in the same working tree or on the same machine.

The five cases below occurred within a little over a week in the everyday work of two repositories, without anyone looking for them. Sometimes two sessions worked side by side on different topics, sometimes one session worked while the maintainer touched files in the same repository. Both are perfectly normal modes of operation once you work with more than one agent. The parent article Agentic Coding from a User’s Perspective places parallel operation within the overall experience. This article deals with it alone.

Case 1: Foreign Work Swept Into a Commit

In the DI² project, the maintainer and an agent session were working on the same repository, at times simultaneously. The session routinely committed with git add -A and wrote a detailed commit message that stayed close to its topic. Between two of its commits, the maintainer had been working somewhere else entirely, on a test case convention, a skill and a CI guard. The session’s next git add -A commit swept up those four files. The commit message is about a scroll bugfix. Of the roughly one hundred lines of someone else’s convention work, it says not a word.

This was noticed not at commit time but hours later, and only by chance. While material was being incorporated into an article collection, it contained quotations from repo files with the note that they were not yet committed and that the follow-up commit should be located during verification. The quoted passages were found, but in a commit whose message is about a completely different topic. Without that cross-check of the evidence, the case would have gone undetected.

This is not data loss, the work is fully in the repository. The damage lies in traceability. Anyone looking for the origin of the convention change no longer finds it via the commit message, only via a content search with git log -S. On top of that, the commit silently claims authorship of someone else’s work, complete with the agent’s Co-Authored-By line. The obvious lesson is to stop staging wholesale and to name your own files individually. The second case shows why that lesson is incomplete.

Case 2: The Obvious Countermeasure Doesn’t Hold

One day later, in the repository of this blog, two parallel Claude Code sessions were working in the same working tree, this time with no human involved. Session A had staged five renames of code files with git mv but had not committed them yet. At that moment, session B created its own commit, and did so exactly according to the lesson from case 1, with git add on a single, explicitly named file. The commit swept up the five foreign renames anyway. Its message is exclusively about the material collection session B had been working on.

The reason is not carelessness but the way Git works. git add <path> only controls what additionally goes into the index. git commit without a path then commits the entire index, including everything someone else has already put there. In a working tree used by several actors at once, the index is thus shared state. Whoever took „name files individually“ as the lesson from case 1 is not protected against this case and feels safe anyway.

Session B had even built in a check, a git status in the same command. But it hung behind the commit via &&:

git add material.md && git commit -m "…" && git status --short

So the output appeared only after the fact had been created. A check that runs after the action is no longer a check, it is a log entry. The reliable order separates the look at the index from the commit and names the paths explicitly on the commit:

git diff --cached --name-status     # separate step before the commit: What is in the index?
git commit --only material.md -m "…"
git show --stat --oneline HEAD      # afterwards: Does the commit contain exactly the expected files?

What matters here is not the --only keyword but the path argument on the commit. As soon as paths appear on the command line, Git, according to its documentation, commits only their current content and leaves changes already staged for other paths untouched. --only is the default mode for this and merely makes the intent explicit. The rule file of this blog spells it out anyway, because then the command shows its intent. The behaviour was reproduced for this article in a throwaway repository. With a foreign rename in the index, git add b && git commit produces a commit with two files, git commit --only b produces a commit with a single file, and the foreign rename stays in the index unchanged.

The path argument does have a limit, though, and it matters for this topic. It protects against foreign entries at other paths, not against foreign changes to the same file. Git commits the current content of the named paths from the working tree, not the state the session put there with git add. If another session has edited the same file shortly before, its change lands in the commit as well, and no command warns about it. This too was reproduced: the index held one line of the session’s own, the working tree an additional foreign one, and git commit --only committed both. git show --stat then showed two insertions instead of one, a hint only noticed by someone who expects the number. So git commit --only solves the problem of the shared index, not the problem of the shared file.

In this case too, the damage was traceability, not loss. Not a single line of the renames themselves changed, but the commit message does not describe them. What went well in this case is at the same time the contrast to the next one. Session B noticed the mistake itself, did not repair it on its own authority and informed the other session. A repair would have damaged the history further, because session A’s follow-up commit was already sitting on top of the affected commit. A git reset --soft to the state before would have taken that commit out of the branch. Its changes would have remained in the working tree, but the next commit would have absorbed them under session B’s message, and the original commit would no longer have been reachable via the branch, typically only via the reflog. This too was reproduced for this article in the throwaway repository. The decision to leave the history as it was rested with the maintainer. Session A did not simply take the report on trust but cross-checked it with git show --stat and then left a pointer to the actual location in the revision log of the affected article. That pointer closes the gap the commit message left behind.

Case 3: Foreign Work Deleted

Case 3 is the mirror image of case 1. There, a session swept up foreign work, here it deleted foreign work. The cause is the same, the damage is the more expensive of the two.

In the DI² project, two sessions were working on different topics at the same time. Session A was fixing a bug, session B was drafting a UI concept for another feature in parallel. Session A had seen a working tree with no pending changes at the start. During a longer tool run, two foreign files appeared there, a modified concept file and a new mockup. Session A attributed both to its own run and justified this by claiming its tooling had worked beyond the assignment. Then it discarded both files. It reverted the modified file to its previous version and deleted the new one.

In the summary to the maintainer, the mistake got worse. Session A not only reported the rollback but also claimed the discarded text had contained a fabricated user approval. That approval had in fact been given, just in the other session. A deletion thus turned into a false accusation against the session’s own tool chain as well.

It was not session A that found the case. Session B restored its work and sent a message into the other session. It is rendered here in substance, not verbatim:

Two files from my working state just disappeared: the modified
concept file and the new mockup. Both are mine, not products of
your tooling. I have restored them.
From now on, please run no restore or cleanup operations
(checkout, restore, delete) on paths you have not touched
yourself in this session. If you notice files you cannot
attribute: report them, do not remove them.

The message did not undo the deletion, session B had already done that itself. But it formulated the rule that later went into the rule file, and it gave the session that caused the damage a piece of information it could never have read from the working tree: that it was not alone.

Why the misattribution is understandable was described above. Why it is dangerous nonetheless is shown by the difference between a commit and a change Git does not yet know about at all. A commit can usually be reconstructed or taken back with a git revert. A discarded change that was neither committed nor put into the index with git add, by contrast, cannot be recovered by Git. Here it only came back because the second session still had its state in memory. Two lessons follow. Destructive operations such as reverting, resetting and deleting act on everything under the given path, not just on your own work. That is why, in parallel operation, they are never mere cleanup. And your own attribution is an assumption, not an observation. „That was me“ can be checked, via timestamps, content and change history. Whoever does not check it should not claim it either, least of all in a summary to the human.

Self-control did not work in this case. Without the second session, the work would have vanished silently, and the false claim would have stood as a finding. That makes it the most uncomfortable and therefore the most valuable of the five cases.

Case 4: Duplicate Numbers, Because Politeness Is Symmetric

The fourth case has its point not in the conflict itself but in the cooperative reaction to it. Both sides behaved decently, and precisely because of that the conflict arose a second time.

Two sessions in the DI² project each created a new bug independently of one another. Number assignment follows a documented rule: the highest number assigned in the index file plus one gives the next. Both sessions read the same highest number, and both assigned the same new one. Both noticed the collision, and both stepped aside to the next number. The second collision was identical to the first.

Checking beforehand would not have helped, because both sides had calculated correctly. The critical window lies between reading and writing. A check run before would have given both sides a green light and lulled them into false security. The first reflex, „just check first“, treats the problem like a slip. In fact it is a race: whoever gets overtaken by the other side between their read and their write has lost despite a correct check. Such collisions on sequential numbers therefore cannot be ruled out in parallel operation, only detected and resolved.

In the end, two things helped. The first was detection after writing, in both directions. The project has an automatic check for this, called a guard there, which compares the index file against the files on disk. It reports not only „number assigned twice“ but also „file without index row“ and „index row without file“, three different intermediate states of the same process. A check that only knew duplicates would have let the later stages through. The origin of this guard is remarkable. It had been built two days earlier for an entirely different symptom, counters in the header of an index file that had drifted away from the content below. What it then caught was a duplicate assignment by parallel sessions, which nobody had thought of when building it. The reason is transferable. The guard does not check the symptom but the condition behind it that must always hold: index and files on disk must match in both directions. A check that had only looked at whether the counter was right would have stayed silent at the collision.

The second was an asymmetric resolution rule. The other session proposed sorting alphabetically by identifier, and the smaller identifier gets the smaller number. What matters is not that the rule is particularly clever but that both sides can compute it to the same result without negotiation. On the first collision, stepping aside is the decent thing to do. But if both step aside, the same conflict arises again, just like two people in a corridor each making way for the other. Two cooperative parties cannot get past each other without an asymmetric rule. Whoever runs parallel agents therefore needs not only detection but a tie-break rule that is fixed in advance, a right-of-way rule for the tie. Otherwise the agents negotiate, and negotiating costs rounds.

Two side findings from the resolution belong here as well. The first concerns the countermeasure from case 2. git commit --only keeps its promise, but it refuses to work on a new file as long as Git does not know the file yet. The error message then reads pathspec did not match any file(s) known to git. So the file first has to be added with git add, and exactly that step puts your own entry next to the foreign entries already sitting in the index. The safeguard held, because the commit took only the one file. But the procedure forces you through precisely the state you wanted to avoid. The second side finding is an ownership criterion for shared files. A file that cannot be split and carries entries from both sides belongs in the commit of the side whose substantive change it carries, not in the commit of the side that happens to finish first. One of the two sessions had initially proposed the wrong direction and was corrected by the other. Its commit would otherwise have contained half a process without the file move that belonged to it.

Case 5: The Machine Is Shared Too

The first four cases are about collateral damage in the repository. The fifth hits the human’s machine. Parallel sessions share not only the repository but also the computer the human is working on at the same time. What is shared here is no longer Git state but computing power, and different tools apply, not git but taskset, timeout and trap. The machine’s load is the one resource whose consumption the agent causes and cannot perceive itself.

The assignment in the DI² project was a test that failed only sporadically. The diagnosis found not a slow test but two time budgets that did not fit together. The test library gives each individual wait 1000 ms, the test runner gives the whole test body 5000 ms, and the densest tests chain seven waits in a row. Without load, the file runs through in just over a second. So the defect can only be observed under load at all. That makes the verification expensive for the machine, and it makes it legitimate at the same time.

The agent generated the load itself, with a single line. It started 220 CPU endless loops in the background, then three to five full runs of the test suite followed, and the whole thing was repeated three times, over twelve minutes in total. On the maintainer’s 28-core workstation, that produced a load average of 234, roughly eight times the core count, and every core sat at one hundred percent. The agent did not notice. The maintainer noticed his machine no longer responding, opened the process monitor and asked in a third session, with a screenshot: „Are you still doing something? When you run, CPU usage goes up considerably.“

The actual finding is not a wrong idea but a lost one. An earlier subagent on the same bug track had answered the same question with six endless loops, pinned to two cores via taskset. That produces the same contention for the test suite, but the rest of the machine stays free. The later subagent went unpinned to 56 loops, then to 200 and finally to 220. The frugal method had already been tried on the same track and was not picked up again when scaling up. Knowledge one session has is not automatically available to the next.

Cleanup did happen, but in the wrong place. Every command ended in kill $(jobs -p), and that worked: no processes were left behind, and the load average dropped back into the single digits. But the cleanup hung on the last line of the command instead of on a trap. A timeout, a crash or an abort by the human would have left 220 endless loops running until the next reboot, and that precisely at the moment the human steps in because the machine has frozen. Transparency was not lacking either, it just came in the wrong tense. The final report named the method correctly and unprompted, with load average and core count. That disclosure was complete and ineffective at the same time, because it came only once the damage was over. An announcement beforehand would have cost two sentences.

Why does the agent not notice? It could well measure the load, uptime or top would put the load average in its output. What it lacks is the feedback about what that number means for the human. It sees exit codes and no stuttering desktop, and no tool tells it which load the owner of the machine is willing to accept right now. The costs fall entirely on a human it cannot observe. Unlike with tokens or run time, there is no feedback loop here. Resource-intensive verification remains the right thing to do, because load and repetition are often the only way to demonstrate a sporadic problem at all. Three conditions make it tolerable. The load runs pinned instead of on the whole machine, provided the test does not require system-wide contention. The cleanup hangs on a trap instead of on the last line. And the agent announces beforehand what it is about to do instead of reporting afterwards, because whether a machine may be unusable for twelve minutes is the owner’s decision. The following minimal pattern shows the first two conditions in code. It is not a runbook, and not a cleanup guarantee either:

trap 'kill $(jobs -p) 2>/dev/null' EXIT INT TERM
for i in $(seq 1 6); do taskset -c 0,1 timeout 900 sh -c 'while :; do :; done' & done
taskset -c 0,1 <test run>

Line 2 creates the contention on exactly the two cores on which the test also runs in line 3, and gives each loop its own timeout after which it terminates itself. The rest of the machine stays with the human. Line 1 cleans up the started loops when the shell exits. This trap has two limits, and both were reproduced for this article. If an abort signal arrives while the test run is still going, Bash executes the trap only once the test run returns. And against a kill -9 of the shell itself, it does not help at all. That is exactly what the timeout in line 2 is for. The third condition belongs in the assignment, and there it costs two sentences. The following assignment is a pattern, not a quotation from the session:

Reproduce the sporadic failure under load. Conditions:
load only on two pinned cores (taskset), cleanup via trap,
and tell me beforehand how long the machine will be loaded.
If you need more than two cores, ask first.

One side finding concerns attribution. Three sessions were running in parallel, and the maintainer could not tell which of them was causing the load. In the end the evidence came from the session transcripts, where the command stood with a timestamp, and from the decay of the load average, which matched the time of the last logged run. Whoever runs several agents in parallel needs a method to attribute load to a session, and needs it before it becomes necessary.

What Holds and What Doesn’t

Countermeasures emerged from the five cases, and they emerged after the fact. After case 2 they went into the Git rule file in the repository of this blog, as a section of its own called „Parallel sessions“, and since case 4 they also carry the side finding about new files. How such a rule file is built and why the countermeasure lives there and not in the maintainer’s head is described in I Gave Claude Code 27 Rule Files Instead of One CLAUDE.md. The balance after a little over a week of parallel Claude Code sessions looks like this.

What holds:

git commit --only <paths>: The path argument on the commit limits it to the given paths, which can also be directories or patterns, and leaves staged changes outside them alone, --only makes that explicit. Against foreign index entries this is the simplest and most robust variant, because it does not depend on the session’s attention. Against foreign changes to the same file it does not protect.
The look at the index as a separate step before the commit: A git diff --cached --name-status or a git status --short runs as its own command before the commit and not behind it via &&. Whatever sits there and does not belong to your own change gets named or excluded. This check is a plausibility check and not a lock, because between the look and the commit another session can change the index again.
git show --stat --oneline HEAD after the commit: This look checks whether the commit contains the expected files and only those. It finds the mistake the two steps before let through. Whether the content of the files is right as well is only shown by git show HEAD -- <path>.
Report instead of repair: If foreign work sits in your own commit on a branch someone else is still working on, the session informs the affected side and leaves the decision about the history to the maintainer. If the history stays as it is, the session leaves a pointer to the location where someone would look for it.
One worktree per session, where the workflow allows it: With git worktree add, every session gets its own working tree with its own index, and cases 1 to 3 disappear in the form described here. Shared remain the repository objects and the branches, among other things HEAD and the index are specific to each worktree. Everything outside the working tree stays shared as well, the machine and the shared counters included, more on that in the FAQ.
Fix the tie-break rule in advance: For every shared counter there is an asymmetric rule that both sides can compute to the same result without negotiation.
Check the condition, not the symptom: The guard compares index and files on disk in both directions. A check like that also catches errors nobody thought of when it was built.
Cross-check instead of believing: Both sessions in case 4 re-measured each other’s status reports instead of taking them over. That found an omission in the report that the reporting side could not have noticed itself.

What doesn’t hold:

„Just check first“: In a race between two writers, the window lies between reading and writing. The check beforehand gives a green light and lulls into security.
The lesson from case 1 as protection against case 2: git add <file> instead of git add -A closes one gap and leaves the neighbouring one open. A lesson that closes one gap creates a false sense of security for the next.
Repair via git reset <commit>: Resetting the branch acts on everything that has been added since that commit. If a commit by another session is already sitting on top, it disappears from the branch. The three modes differ only in what gets reset besides the branch: --soft leaves index and working tree alone, the default mode resets the index, and --hard additionally discards unsaved changes to tracked files in the working tree, including another session’s, and overwrites untracked files that stand in the way of a tracked file of the target state. The damage of the repair is then greater than that of the mistake.
Cleanup on the last line: It does not survive precisely the abort that needs it most.
Transparency after the fact: A complete report after the damage is honest, but it no longer changes anything.

One side finding belongs here because it concerns the figures with which agents demonstrate their diligence. A shared working tree distorts not only files but measurements. A full test run, started at the very moment the other session was renaming a file, reported three failed test files with an error that had nothing to do with them in substance. On the next run, everything was green. Whoever writes such numbers into a report documents a state that never existed. In parallel operation, a single red run is therefore not a finding but a hypothesis, and it only counts once it can be repeated on a quiet tree. How measurements can be separated from assumptions when the agent takes over the diagnosis will be the subject of a separate article on this blog about debugging with a coding agent.

Summary

To a session, its own and foreign changes in the working tree look the same, and Git never contradicts the assumption „everything new comes from me“.
In a shared working tree, the Git index is shared state. git add <file> does not protect, because git commit without a path commits the entire index. Against foreign entries at other paths, git commit --only <paths> is the most robust of the variants considered here, supplemented by a look at the index before the commit and a git show --stat afterwards. Against simultaneous changes to the same file, a separate worktree helps most reliably.
Destructive operations are never mere cleanup in parallel operation. Your own attribution is an assumption, and whoever cannot check it reports instead of deleting.
Shared counters can collide at any time in parallel operation. What is needed is a check after writing that verifies the intended state, and an asymmetric tie-break rule fixed in advance.
Parallel sessions also share the human’s machine, and its load is a resource with no feedback loop to the agent. Resource-intensive verification therefore runs pinned, time-limited, with cleanup via trap and with an announcement beforehand.
The rule came after the damage. Three of the five cases were the agent’s mistakes. The agent reported two of them itself, the third was found by the neighbouring session whose work had been lost.

FAQ

Is it enough to stage files individually instead of using git add -A?

No. git add <file> only controls what additionally goes into the index. If something from another session or from the human is already sitting there, git commit without a path takes it along. Protection only comes from git commit --only <paths>, which ignores the rest of the index, or from a look at the index as a separate step before the commit. A git status hanging behind the commit via && shows the mistake only once it has already happened.

Can git commit --only prevent foreign changes?

Only one kind of them. The path argument keeps staged changes at other paths out of the commit, that is the shared-index problem from case 2. It does not keep out what another session has changed in the same file, because Git commits the current content of the named paths from the working tree. Whoever wants to be sure what is in the commit checks the content afterwards with git show HEAD -- <path> or gives every session its own worktree from the start.

What happens when two sessions edit the same file?

In a shared working tree the file exists only once, and Git reports nothing. If both change individual spots, both changes end up in the file afterwards, and whoever commits first takes the other’s along. If one session instead rewrites the whole file from an older read, the other session’s change has silently vanished. If it was staged, its state still sits as a blob in the index. After another git add it is only a dangling object that git fsck --lost-found may still find, but that is no guarantee. If it was never staged, it is gone. The situation can only be recognised by clues: an MM in git status on a file you did not stage yourself, or an editing tool that refuses to write because the file has changed since it was last read. Claude Code’s Edit tool behaved exactly like that in the sessions behind this article, as of September 2026, a sed or a Python script does not. Git only detects the situation reliably once both sessions work in their own worktrees on their own branches. Then there are two versions, and merging them produces a merge conflict as soon as both have changed the same lines. Different lines Git merges automatically.

What do you do when a commit has swept up foreign work?

On a branch someone else is still working on, the first reflex, repairing the commit on your own authority, is the wrong one. Whoever works alone on an unpublished branch may reset it. A git reset to an earlier commit and other interventions in the history act on everything that has been added since that commit. If a commit by another session is already sitting on top, the correction takes it out of the branch, and with --hard it also discards that session’s unsaved changes to tracked files. Better to inform the affected side and let the maintainer decide whether the history gets touched. If it stays as it is, a pointer to the actual location belongs where someone would look for it, for instance in the revision log of the affected article or in the spec of the affected task. The content is not lost. Only its origin can no longer be found via the commit message.

How does a session recognise that it is not alone in the repository?

With Git alone, not reliably at all, and that is precisely the core of the problem. There are clues: an index that is not empty at the start, files that newly appear or change during a tool run although your own tooling did not touch them, or a commit in the log that did not come from your own session. None of them is proof. That is why a session treats its attribution as an assumption and checks it via timestamps and content before discarding anything. The most reliable information comes from the human. Whoever starts two sessions tells both that they are not alone.

Are Git worktrees the solution for parallel Claude Code sessions?

For cases 1 to 3 they largely solve the problem, because they remove its common cause, the shared working tree. A worktree gives every session its own working tree with its own index, only the repository, its objects and its branches stay shared. In practice a worktree also means a branch of its own per session or a detached HEAD, because a branch cannot be checked out in two worktrees at the same time. So the merge back is part of the workflow, and that is where Git then reports the conflicts on the same file as well. The difference to the shared working tree lies exactly there: own files on disk, own index, own HEAD, but the same commits and branches underneath. Foreign files in the working tree and foreign entries in the index thereby disappear. The remaining cases persist, though. The machine is still shared, and shared counters then collide when the branches are merged instead of at write time. Besides, in practice the human often works in the main working tree, and a session running there has all five problems again. Worktrees separate the working trees, not the actors.

How do you commission load tests without freezing your own machine?

Best with three conditions in the assignment. The load runs pinned to a few cores, via taskset for instance, provided the test does not require system-wide contention, and the test runs on the same cores so that the contention arises where it is needed. The cleanup hangs on a trap so that it takes effect when the script ends, and every load loop gets its own timeout so that it terminates even if the shell gets killed the hard way. And the agent announces beforehand how long the machine will be loaded, so that the human can decide whether to allow that right now. An agent cannot derive these conditions on its own, because it can measure the load but cannot see its cost to the human.

Parent article:

Agentic Coding from a User’s Perspective — Experience: The Work Doesn’t Disappear, It Shifts — the overall experience of which parallel operation is one section.

Hub:

AI-Assisted SQL Development with Claude Code — Rules, Skills and Agents That Enforce Conventions — the enforcement system the rule file from this article belongs to.

Sibling article:

The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent — why a finding only holds once a measurement stands behind it, and why a red test run in a shared working tree is only a hypothesis at first.

Rule files:

I Gave Claude Code 27 Rule Files Instead of One CLAUDE.md — how a rule file is built into which a countermeasure like git commit --only goes.

Next door:

SSIS vs. SQL: Source Code Management — Why SP Diffs Are Readable and .dtsx Diffs Are Not — source code management from the era before agents, with the same core concern: a diff has to stay readable.

An EU Region Is Not an EU Provider — Vercel, Supabase and the GDPR Third-Country Question

Marcus — Thu, 10 Sep 2026 13:25:48 +0000

"We host in Frankfurt, so we're GDPR-compliant." The sentence comes up in almost every conversation about Vercel and Supabase, and it is half the answer to a question that has two halves. The region answers where the data lives. It does not answer whose law applies to the companies that process that data.

This article takes the second half apart. It explains what an EU region delivers legally and what it does not, why the European Commission has twice declared transfers to the United States permissible by decision and the Court of Justice of the European Union has struck down both decisions while the third is under review right now, and what the CLOUD Act, a US law from 2018, has to do with a data center in Frankfurt. At the end there is a checklist for the third-country question that lets you examine and document your own use of Vercel or Supabase, plus a decision aid for the alternative. The occasion is a concrete one: In the DI² project this question was asked before the first deploy, and the answer shaped the infrastructure.

The essentials up front:

Not compliant by default: Vercel and Supabase offer EU regions and a data processing agreement, the contract that lays down what the provider may do with its customers' data. The conditions for a compliant deployment still have to be established and documented by the user. The default setting is not the compliant one, and at Vercel functions run in the United States unless you choose otherwise.
The chain counts, not the seat: Vercel is a US company. Supabase's contracting party sits in Singapore, the US company Supabase, Inc. appears as a subcontractor for support in its own list, and hosting runs on Amazon Web Services (AWS). Whose law applies is not stated by the provider's legal notice but by its list of subcontractors that touch the data. The GDPR calls them sub-processors, and the providers publish them as a "Subprocessor List".
Three attempts at a transfer mechanism: The European Commission has three times found by decision that the United States offers an adequate level of protection, each time on the basis of an arrangement with the US government. The first two, Safe Harbor of 2000 and Privacy Shield of 2016, were struck down by the Court of Justice in 2015 and 2020, both times for the same reason. Against the third, the Data Privacy Framework of 2023, the General Court of the European Union dismissed the first action in September 2025, and the appeal is pending before the Court of Justice. As of 9 September 2026 it stands.
The CLOUD Act does not ask where the data is stored: A provider under US jurisdiction must in principle hand over data in its possession or under its control, no matter which country it is stored in. There is no agreement with the EU that would open the law's special objection procedure. Microsoft France confirmed as much before the French Senate in June 2025.
A contract does not cure that: The data processing agreement governs the relationship between user and provider. It does not override a US law. Both risks land on the user in the end, because the user is responsible for the processing, even though the provider has obligations of its own. The checklist at the end makes them visible instead of removing them.
The alternative is a provider question: Anyone who wants to rule the two risks out rather than carry them has only one route, no provider under US jurisdiction in the chain. That means an EU provider or a server of your own in the EU, and both come at a price that the sibling article on infrastructure puts into numbers.

Prerequisite: The article is aimed at developers and small teams in the EU who run a web application with a login on Vercel, Supabase or comparable services, or plan to. No legal background is required, and terms such as adequacy decision, standard contractual clauses and data processing agreement are explained where they first appear. All provider statements are taken from the providers' own documentation, quotations verbatim, retrieved on 9 September 2026. The article is not legal advice, and it does not replace an assessment of your own case.

The Short Answer: Not by Default

Can you use Vercel and Supabase in a GDPR-compliant way? Yes, but not by creating an account and picking Frankfurt as the region. Both providers say so themselves, just not on the landing page. Vercel writes in its compliance documentation that it supports its customers' GDPR compliance and lists five commitments to that end, from the security level to the standard contractual clauses. Supabase writes that it supports GDPR-compliant deployments. In both cases the subject of the sentence is the customer. The provider supplies the tools, and the user has to apply them, document them and answer for them.

What that involves fits in one paragraph. The user signs the data processing agreement, which the GDPR calls a contract under Article 28 and practice usually calls a DPA. The user chooses the region instead of accepting the default. The user documents the basis on which data may leave the territory of the EU, because with both providers it can do so even with an EU region, at Vercel by the provider's own reservation, at Supabase through support and subcontractors. The user extends the privacy policy to name the recipient, the third country and the basis. And the user consciously decides whether to carry two risks that no contract takes away. Those two risks are the core of this article. The rest is the groundwork needed to understand them.

What an EU Region Delivers — and Who Is in the Chain

An EU region is a promise about storage location. At Supabase that means: Whoever creates a project in an AWS region gets the database, the auth service and storage hosted in that region, as the security page states. At Vercel the region is a choice for the part of the application that computes on the server, the so-called serverless functions, and the default is not Europe. The compliance documentation says it in one sentence: "The default location for Vercel functions is the U.S." In front of those functions sits Vercel's worldwide delivery network, a CDN (content delivery network), which accepts visitors' requests at the nearest of twenty locations and serves the pages from there. The chosen region applies to the functions, not to that network.

The question of whether a service can be used in a GDPR-compliant way has two parts. The first is: Where does the data live? The region answers it, and for Supabase and Vercel alike that part is settled. The second is: Which companies process the data, and whose law are those companies subject to? No region answers that. The answer lies in the list of subcontractors the provider brings in to run the service and who touch the data in the process. Both providers publish that list, and at both it reads differently from what the marketing suggests.

Vercel: Vercel, Inc. is a US company. It is certified under the EU-US Data Privacy Framework, and it additionally relies on standard contractual clauses for transfers. Both statements are in the compliance documentation. Directly below them stands the reservation that puts the region into perspective: "Vercel may transfer data to and in the United States and anywhere else in the world where Vercel or its service providers maintain data processing operations." Vercel may therefore move data to the United States and to any other location where it or its service providers process data. Backups, according to the same page, are replicated globally and are not accessible to customers.

Supabase: Here the research for this article produced a different picture from the expected one. According to the terms of service and the data processing agreement, the contracting party is not a US company but Supabase Pte. Ltd., based in Singapore. The terms place the contract under California law, and the data processing agreement relies on standard contractual clauses for transfers, a contract text prescribed by the European Commission in which the recipient commits to European data protection rules, and does not mention the Data Privacy Framework. Supabase's own subprocessor list, in the version dated 1 June 2026 and retrieved on 9 September 2026, then names the companies that actually touch the data: Amazon Web Services for hosting, Cloudflare, Google and Fly.io for further hosting services, Vercel for hosting, and at the top of the list Supabase, Inc. for support. Supabase's privacy policy names the United States and Singapore as transfer destinations, each on the basis of standard contractual clauses, and describes its own services as "primarily hosted in and provided from the United States". The contracting party sits in Singapore, the data sits with a US host, and the US company has access as a subcontractor.

For the GDPR the seat in Singapore changes nothing about the classification. An adequacy decision is the European Commission's finding that a country outside the EU protects personal data as well as the EU itself does. Where one exists, data may go there as if it stayed in the EU. For Singapore no such decision existed at the time of writing, and for the United States it applies only to companies with an active certification under the Data Privacy Framework, and only to the data types that certification covers. Every transfer to Singapore is therefore a third-country transfer that needs a mechanism of its own, in Supabase's case the standard contractual clauses. What the case shows is something else: The provider's legal notice is the wrong place to answer the provider question. The right place is the list of subcontractors. As soon as a service provider under US jurisdiction appears there that processes personal data or can access it, and with services on AWS, Google Cloud or Azure one regularly does, two things have to be examined separately for that data. First, whether a third-country transfer takes place and on what basis, which is the next section. Second, what state access risk that link brings with it, which is the section after that. How large the second risk actually is depends on which data the link really has in its possession or under its control. The EU region has settled the storage location. The chain has settled which legal systems have a say.

Three things come on top with every region, because they do not hang on the storage location. Support access, which the provider needs to operate the service, happens from wherever its staff sit. Metadata, meaning account data, invoices and access logs, runs in the provider's systems and not in the chosen region. And backups sit wherever the provider replicates them, at Vercel explicitly worldwide.

The Transfer Mechanism: Three Attempts

The first of the two risks concerns the basis on which data may leave the EU at all. The GDPR allows transfers to a country outside the EU only under the conditions of its Chapter V. The yardstick behind it is that the data keeps a level of protection there that is essentially equivalent to the European one, and there are three routes to that. The first is an adequacy decision by the European Commission under Article 45, with which the Commission finds across the board that a country offers that level. The second is appropriate safeguards under Article 46, in practice almost always the Commission's standard contractual clauses, with which the recipient contractually commits to European rules, more rarely binding corporate rules within a group of companies. The third is a set of narrowly drawn exceptions under Article 49, such as explicit consent for a single transfer, which are no use for a running service.

For the United States the Commission has taken the first route three times, and twice the Court of Justice has struck the decision down.

Safe Harbor, the 2000 decision: US companies could self-certify that they observed certain principles. After the revelations about US surveillance programs in 2013, Max Schrems, then a law student in Vienna, lodged a complaint against Facebook with the Irish data protection authority because his data was going to the United States. The complaint reached the Court of Justice via the Irish High Court, and the Court declared the decision invalid on 6 October 2015. The reasoning was that US authorities had general access to the content of communications and EU citizens had no legal remedy against it.

Privacy Shield, the 2016 decision: The successor came with an ombudsperson in the US State Department and further commitments. Again it was a complaint by Schrems, who by then was running data protection cases full time with the organization noyb, and both judgments therefore carry his name, Schrems I and Schrems II. The Court of Justice struck the decision down on 16 July 2020, on the same grounds in updated form. The surveillance powers under Section 702 of the Foreign Intelligence Surveillance Act (FISA), the US law on foreign intelligence, and Executive Order 12333, a presidential order issued without Congress, were not limited to what is necessary, EU citizens could not challenge them before any US court, and the ombudsperson was neither independent nor able to bind the intelligence services. In the same judgment the Court made clear that the standard contractual clauses remain valid, but that whoever exports data must assess for the specific transfer whether the law and practice of the receiving country guarantee the level of protection, and must provide additional measures where needed. That assessment is usually called a transfer impact assessment, and it applies then as now. Between 2020 and 2023 it affected almost every transfer to the United States, and since the third attempt only those that do not rely on it, as in the Supabase case.

Data Privacy Framework, the 2023 decision: The third attempt rests on an executive order of the US President from October 2022 that lays down proportionality principles for the intelligence services and sets up a review court, the Data Protection Review Court, before which EU citizens can bring complaints against surveillance measures. On that basis the Commission adopted the adequacy decision on 10 July 2023. US companies self-certify again, and the US Department of Commerce keeps the list. Vercel says it is on it. Supabase does not mention the framework in its contract, its security page or its privacy policy and relies on standard contractual clauses.

The decision is under attack. The French member of parliament Philippe Latombe challenged it directly before the General Court of the European Union, and the General Court dismissed the action on 3 September 2025. It found that the United States ensured an adequate level of protection at the time of the decision, and it expressly tied that finding to that point in time. On 31 October 2025 Latombe lodged an appeal with the Court of Justice, case number C-703/25 P, on four grounds. On 4 June 2026 the Court admitted Microsoft as an intervener in support of the Commission. As of 9 September 2026 no hearing date is known, and the decision stands.

Two things are worth keeping in view. First, the framework rests on an executive order, not on a statute, and an executive order can be changed by the next president. Second, the US oversight body that is supposed to monitor compliance with the commitments, the Privacy and Civil Liberties Oversight Board, has been without a quorum since three of its members were dismissed in January 2025. Its own members page lists a single serving member on 9 September 2026. The dismissals are themselves the subject of litigation: A federal court in Washington declared them unlawful in May 2025, and the appeals court stayed that ruling in July 2025 for the duration of the proceedings. Whether and how that enters the judicial review of the framework is open. For the user both mean that the basis of their transfer depends on political decisions in another country.

If this decision is struck down too, what happened in 2020 happens again. Whoever relied on the framework alone is left without a basis and has to switch to standard contractual clauses, including their own assessment of US law. Whoever documented both from the start, the provider's list entry and the standard contractual clauses in the data processing agreement together with the assessment of US law, has less to do on that day. That is exactly why the checklist below carries them as two separate items.

The CLOUD Act: Possession or Control, Not Storage Location

The second risk, the CLOUD Act, is independent of the first, and it survives every adequacy decision. Its origin is a dispute over emails. In 2013 US investigators demanded from Microsoft, by warrant, a mailbox that sat in a data center in Dublin. Microsoft refused, arguing that a US warrant did not reach as far as Ireland. An appeals court sided with Microsoft in 2016, the Supreme Court took the case, and before it could decide, Congress passed a law that answered the question. The Clarifying Lawful Overseas Use of Data Act, CLOUD Act for short, was signed on 23 March 2018.

The core rule is in 18 U.S.C. § 2713. A provider must in principle hand over content and subscriber records that are "within such provider's possession, custody, or control", and that "regardless of whether such communication, record, or other information is located within or outside of the United States". In one sentence: What counts is whether the provider possesses or controls the data, not where it is stored. A data center in Frankfurt is no argument against this rule.

The law provides a special objection procedure for the case that disclosure would violate the law of another state, but only under two conditions at once. The customer concerned must not be a US person, meaning neither a US citizen nor resident in the United States, and the United States must have concluded an agreement under this law with the country where the data is stored. Such agreements exist so far with the United Kingdom, in force since October 2022, and with Australia, in force since January 2024. With the EU or any of its member states there is none. For data in Frankfurt the second condition is missing, and with it this route. What remains is the general possibility of challenging a demand as unlawful, as Microsoft did in the Ireland case. That presupposes a provider that does so.

On the European side stands Article 48 of the GDPR. It recognizes decisions by authorities of a third country only if they rest on an international agreement, such as a mutual legal assistance treaty in which states promise each other support in investigations. A US provider with EU customers thus stands between two legal systems that demand opposite things. The data processing agreement with the customer does not resolve that conflict, because a contract between two companies does not override a US law. What the contract can deliver is a commitment to inform the customer and to contest demands for disclosure as far as the law allows.

What that sounds like in practice was shown by Microsoft France before a committee of inquiry of the French Senate in June 2025. Anton Carniaux, in charge of legal and public affairs there, was asked whether he could guarantee that data of French citizens would never be handed to US authorities without France's consent. His answer is in the minutes of the session of 10 June 2025: "Non, je ne peux pas le garantir, mais, encore une fois, cela ne s'est encore jamais produit." No, he could not guarantee it, but it had never happened so far either. And on the question of the obligation: "Lorsque nous sommes obligés de les donner, nous les donnons." When Microsoft is obliged to hand data over, it hands it over. The two sentences belong together. The first names the risk and how often it has materialized so far, the second the rule behind it: disclosure after the provider's own check of whether the demand is lawful, not before. That is not a statement about other providers, but it is one about the limit of what any provider can promise.

That leads to proportionality. The probability that US authorities take an interest in the user table of an ETL tool with a handful of users is low. The risk is structural, not practical. It consists in the user writing a promise into their privacy policy that they cannot keep, because their provider cannot keep it. Whoever understands that and accepts it consciously acts differently from someone who does not know it. That difference is what the last item of the checklist is about.

A question that comes up regularly here concerns the European subsidiaries of US groups. If the contracting party is called AWS Europe in Luxembourg or Microsoft Ireland, does that help? The seat of the contracting party alone does not answer the question. The rule turns on possession, custody or control, and whether a company within a group controls the data in that sense is a matter of actual access, not of the org chart. The Microsoft Ireland case was the trigger for the law because it concerned data held by a European subsidiary, but no blanket rule follows from it that every EU subsidiary automatically holds data under the control of its US parent. What does follow is the duty to check which company actually gets to the data. That is exactly what makes the Supabase case above so instructive. There it is the other way around: The contracting party sits outside the United States, and the US company is in the chain regardless, with support access.

The Checklist

What a user has to examine and record for the third-country question at Vercel, Supabase or a comparable service can be worked through in seven items. The list does not create compliance and does not replace an assessment of the processing as a whole. It does not remove the two risks, it makes them visible and documents the decision to carry them. For an operator who gets asked, that is the difference between an answer and a shrug.

Read the subprocessor list: Mark every company subject to US jurisdiction (host, parent company, support), and record for each link which data it processes or can view. Only for that data does the CLOUD Act question arise.
Sign the data processing agreement: Conclude the contract under Art. 28 GDPR (DPA) and file it. Check whether it contains standard contractual clauses.
Choose the region: Do not accept the default. At Vercel set the function region, at Supabase the project region. Record what the region covers and what it does not (CDN, backups, support, metadata).
Document the transfer mechanism, both separately: The provider's active entry under the Data Privacy Framework (look it up yourself on dataprivacyframework.gov, including its scope) and the standard contractual clauses from the contract together with your own assessment of US law (transfer impact assessment). If the framework is struck down, the second basis carries on, but only with that assessment.
Extend the privacy policy: Recipient with company name and seat, third country, transfer mechanism, and a note on possible government access under the law of the third country.
Subscribe to changes of the subprocessor list: Both providers announce changes, and the list changes several times a year.
Decide the residual risk consciously and record it with a date: Which data sits with the provider, how serious would access be, and why is the operator prepared to carry that.

The provider information in this article is current as of 9 September 2026. The lists change, and what counts is always the current version at the provider. Item 1 comes first because it determines the others. Whoever reads the list knows for whom to document item 4 and what to write in item 5. Item 7 is the one most people leave out. It is the only one that demands a decision rather than a document, and it records the actual risk decision that stands behind the other six items.

The Alternative: EU Provider or Self-Hosting

Anyone who wants to rule the two risks out rather than carry them has only one route, no provider under US jurisdiction in the chain. Anyone who can carry them has the checklist. In practice the one route means either an EU provider that runs its own data center, or a server of your own with such a provider.

What an EU provider solves: With a provider based in the EU, with its own infrastructure and with no third-country subcontractors in the chain, examples for the first two conditions being Hetzner in Germany, IONOS in Germany and OVHcloud in France, without rating the products, the third-country transfer can drop away for the infrastructure layer, provided support access and the remaining data flows also stay in the EU. Then there is no adequacy decision to check, no standard contractual clauses for the host and no CLOUD Act for the data center. The data processing agreement remains mandatory, and so does the look at the subprocessor list, because an EU provider can use US service providers. Reading the list is item 1 here too.

What self-hosting costs: The price is not primarily money but operations. Whoever runs an application with a login on their own server takes on the reverse proxy, which passes requests from the internet on to the application, the certificates, the identity provider, which manages login and users, the database, the backups and the monthly updates across every layer. In the DI² project that was the decision, and the sibling article One VPS, Four Environments, No Cookie Banner describes what it cost, from the RAM budget through secrets in two places to the monthly bill. On your own server in the EU the third-country question does not arise for the infrastructure. That is the real difference between the two routes, not the price.

Decision aid: Three questions suffice for most cases. First, which data sits with the provider, only account data or also what users enter into the application? The closer to health, financial or employee data, the heavier item 7 of the checklist weighs. Second, who will ask, a customer with a data protection department of their own, a supervisory authority, nobody? Whoever has business customers in the EU will get the question. Third, who carries operations, and what happens when that person is out for two weeks? A managed service answers the third question better than a server of your own, and that is a legitimate reason for Vercel or Supabase. It should just be taken together with the answers to the first two questions and written down.

Summary

An EU region answers where the data lives. It does not answer whose law applies to the companies that process it. That answer is in the subprocessor list.
Vercel is a US company with a certification under the Data Privacy Framework, a US default for functions and an explicit reservation to transfer data to the United States and to any other processing location. Supabase's contracting party sits in Singapore, the US company Supabase, Inc. is a subcontractor, and hosting sits with AWS. In both chains there are companies under US jurisdiction.
The transfer mechanism for the United States, a decision of the European Commission, has been struck down twice by the Court of Justice, and the third is under review right now. As of 9 September 2026 the Data Privacy Framework stands, and appeal C-703/25 P is pending. Whoever documents the list entry and the standard contractual clauses separately has less to do on the day of a judgment.
The CLOUD Act in principle obliges providers under US jurisdiction to hand over data in their possession or under their control, regardless of storage location. The law's special objection procedure presupposes an agreement that does not exist with the EU. A data processing agreement changes none of that.
The checklist makes both risks visible and documents the decision to carry them. It does not remove them. Whoever does not want to carry them needs an EU provider or a server of their own in the EU, and both cost operations rather than money.

FAQ

Is Vercel or Supabase GDPR-compliant if I pick the Frankfurt region?

Not by default, the region only settles the storage location. At Vercel the documented reservation remains, to transfer data to the United States and to any other processing location, and at Supabase the US companies remain in the subprocessor list, above all the host AWS and Supabase, Inc. for support. Both are third-country transfers that need a mechanism, a data processing agreement and a note in the privacy policy. The region is item 3 of the checklist, not the checklist.

Is the Data Privacy Framework still valid in September 2026?

Yes. The European Commission's adequacy decision of 10 July 2023 stands, and the General Court of the European Union dismissed the first action against it on 3 September 2025. Appeal C-703/25 P has been pending before the Court of Justice since 31 October 2025, Microsoft has been admitted as an intervener in support of the Commission since 4 June 2026, and as of 9 September 2026 no hearing date is known. To check the current state, search for the case number on curia.europa.eu.

What happens if the Court of Justice strikes down the Data Privacy Framework too?

The same as in July 2020 with Privacy Shield. Transfers that rest on the framework alone lose their basis from one day to the next. Whoever has additionally agreed standard contractual clauses in the data processing agreement relies on them from that day on and has to carry out the assessment of US law themselves, as every company did between 2020 and 2023. That is why the checklist documents both mechanisms separately.

Does the CLOUD Act apply to AWS, Azure or Google Cloud data in Germany?

Storage in Germany changes nothing if the provider is subject to US jurisdiction. The rule turns on possession, custody or control, so the question is which company within the group actually gets to the data. A contracting party based in Luxembourg or Dublin does not answer that by itself, but a US parent in the org chart does not automatically answer it the other way either. Whether and how a group contests a disclosure in a given case is a matter of its contract and its practice, not of the subsidiary's seat.

Do I need to care about this for a hobby project with five users?

The obligations apply regardless of size as soon as personal data is processed, and a login table with email addresses is enough for that. The risk of government access is structural, not practical, with five users. What remains is the duty not to promise anything in the privacy policy that the provider cannot keep. The checklist takes an afternoon for a small project, and item 7 is then an honest line instead of an empty promise.

Sibling article:

One VPS, Four Environments, No Cookie Banner — How I Host a Next.js App with Self-Hosted Keycloak — the route without a US provider in the chain, with the setup, the decisions and the cost side.

Hub of this branch:

Agentic Coding from a User's Perspective — Experience: The Work Doesn't Disappear, It Shifts — the experience level that the infrastructure decisions belong to.

Cluster hub:

AI-Assisted SQL Development with Claude Code — Rules, Skills and Agents That Enforce Conventions — the enforcement system behind the project.

Agentic Coding from a User's Perspective — Experience: The Work Doesn't Disappear, It Shifts

Marcus — Wed, 09 Sep 2026 15:46:16 +0000

Since spring 2026, two projects have been taking shape with the same coding agent: an ETL generator on Next.js and PostgreSQL, and this blog. The agent does not complete lines, it works through assignments in the repository. This article collects my agentic coding experience from a user's perspective. I build databases, ETL processes and the framework behind them, and I use the agent as a tool without doing AI research or building AI tools myself.

Is it worth it? Yes. I spent years building client-server applications in VB.NET and C#, and I am now building a TypeScript application without knowing TypeScript. That is the visible gain. The less visible part is that the work has shifted: from programming to structuring. Rules and skills for the agent, feature specs, bug files, test cases, documentation. In my working life, other people used to deliver that work, a project lead for example, and I consumed it as a developer. I underestimated how much of it there is.

The essentials up front:

Is it worth it? Yes: A TypeScript application with frontend, tests and CI, built by someone who does not know TypeScript but knows databases, ETL and .NET.
The work is not gone, it has shifted: Rules, skills, specs, bug files, test cases, documentation. Project leads and teams used to deliver that.
The guidance lives in the repo, not in the prompt: The prompts are short because rule files and specs carry the work. That includes intermediate states.
The infrastructure was the biggest hurdle: one server with four environments, self-hosted identity, containers, TLS, and no cookie banner because the architecture does not need one.
Control is a loop: An agent extends the radius of what I can build, not the radius of what I can judge. Whatever I cannot judge therefore goes through recurring checks that the agent runs. A green signal is not proof.
This article is itself an example: The draft comes from the agent. Selection, corrections and publication are my work.

Prerequisite: none. Anyone who wants to go deeper will find the way into this blog's technical articles at the end.

What Agentic Coding Means Here

A coding agent is a language model that, with tool access, executes several steps in a row on its own inside a repository: reading files, running commands, changing code, running tests, committing. The difference to autocompletion is not the quality of individual lines but the mode of work. I formulate an assignment, the agent works through it and reports a result.

In both projects the tool is Claude Code, in the terminal and in VS Code. The observations do not depend on the product, though. They depend on an agent having access to repository, shell and database, and on a human deciding what happens with it. The mechanics behind rule files, skills and sub-agents are covered by the hub article AI-Assisted SQL Development with Claude Code. This article is about what they changed in my daily work.

Case Study 1: A TypeScript Application Without TypeScript Skills

The app project is called DI² and is an ETL generator on Next.js and PostgreSQL. The database layer consists of more than 150 DDL files, the repository is private, and I work on it alone. On the ETL side, the app's tools build on a framework of my own that I am publishing on GitHub step by step.

From .NET to TypeScript

What I bring is data models, SQL, ETL processes and years of client-server applications in VB.NET and C#. I know how an application with database, business logic and user interface is cut, where state lives and where errors originate. What I do not bring is the language. TypeScript, the React ecosystem and the test tooling of the web world were new to me, and to this day I cannot read a file of this application the way I read a stored procedure.

The frontend, the CI pipeline and the test infrastructure came into being anyway. The clearest example is the 241-line Node script that checks the project's SQL conventions and that I could not have written. My role was a different one: I knew the conventions, I could judge a finding as right or wrong, and I made the framing decisions. Acceptance ran through runs, not through reading the code, as the article about the guard shows. An agent extends the radius of what I can build. It does not extend the radius of what I can judge. Where the two diverge, acceptance needs a criterion that can be checked without understanding the code.

Conventions in the Repo Instead of in My Head

In a classic solo project, the conventions live in the developer's head. With an agent that starts every session without the history of the previous one, that does not work. In the DI² project this turned into 27 rule files, 28 by now, one per convention, each with its reasoning. Features and bugs get their own files with a running ID before a single line of code exists. How such a rule set is derived from an existing codebase is described in the generate-refine-derive loop. What it costs when rules alone are not enough is shown by 799 hardcoded font sizes despite an unambiguous design rule. What these rule files cost at the start of every session, and how that can be measured and brought down, is described in The Context Window Is Full Before the First Prompt.

The second half of this rule I only noticed when it failed. I accidentally ended a session in the middle of a bug fix and asked a new one to pick up where the old one had left off. The agent found no remains, neither in the working tree nor in the stash. That getting back in still took only minutes was down to the bug file in the repository: error description, database evidence, reproduction steps and three named hypotheses with a checking order. The intermediate state lives in the repo, or it does not live at all.

The Infrastructure Was the Biggest Hurdle

One part of the project that cost me a lot of time is not a line of application code but the infrastructure underneath: a single VPS at Hetzner with four environments, a self-hosted Keycloak as identity provider, an nginx in front of it, a good dozen containers, and a VPS provider that blocks outbound mail ports. I did not write a single line of it. What I decided was that there would be one server instead of four, a self-hosted identity instead of a cloud service, and a location in the EU, and behind these decisions is a design goal I cared about in particular: the application should get by without a cookie banner. What remains are four cookies that are permitted without consent: the login's session token and CSRF token, a language cookie for the public pages, and the state of the navigation. The setup, the decisions behind it and the cost side are described in One VPS, Four Environments, No Cookie Banner.

Reading Is No Longer a Control

Two things together changed the way I work. The first is the language: Whoever cannot read the language of their code cannot control it by reading. The second is the volume. The agent writes so many lines in so little time that reading and understanding all of it would not be possible even if I knew the language. Reading my own code in full, the way I knew it from my .NET projects, therefore drops out as a control. That is not an admission of failure but an unavoidable change in the way of working. If reading drops out as a control, something else has to take its place.

The Control Loop

The control I cannot provide myself has to happen somewhere else: in a loop of checks that the agent runs, but whose yardstick is not its own judgment but rules, tests and audit reports. It does not run once but repeatedly. After every piece of feedback, a rule is sharpened. Recurring checks have skills, that is, versioned prompts in the repository, among them one for security audits and one for update checks. I had unit tests pushed under existing code after the fact. The foundation for that only came in July 2026, and by now there are more than 700 tests. Since the end of August there is a catalog of manual test cases with acceptance criteria. All in all, 14 skills and 28 rule files carry this loop. How that loop runs in detail, what evidence there is for it and where it has failed is described in The Control Loop.

It has found things I would not have found by reading. A rate-limiting rule in the web server had been pointing at an endpoint that no longer existed since the identity provider was switched, and for months it looked like protection. Another case started with a vague sentence from me that the page was barely responding. The agent compared all four environments, found production affected as well although it did not even have the new version, and discovered 2.9 GB in swap. The identity service was sitting on disk, hence 9.6 seconds for the first login and 0.19 seconds for every subsequent one. No application code was involved. Both cases are told in detail in The Agent Measures Where I Click.

One lesson matters more to me than any number: A success signal is not proof. A test suite was green for four weeks while three fix attempts went right past a scroll bug, because the test had stubbed exactly the value whose wrongness was the bug. An audit report said "passed", and only my follow-up question revealed that the decisive click path had never been executed. Both signals attest that a process ran, not that an effect occurred. Since then, verifying means finding a second, independent route to the same question. How to write test cases that deliver that is covered in the article How to Write Manual Test Cases.

Case Study 2: Writing a Blog with an Agent

This blog has been run editorially with the same tool since May 2026. Back then the articles were moved from WordPress into a repository, one folder per topic with a German and an English version. Today there are 65 topics, 55 of them live in both languages. What the agent does here differs less from the app project than I would have expected.

Spec, Rounds, Revision Log

Before every article comes a spec: role in the topic cluster, target audience, search intent, outline, boundary to neighboring articles. Then the article runs through fixed rounds, from the critique of the existing text through structure and style to the English version. Every round gets an entry in a revision log and its own commit. Nine rule files carry the conventions, among them style heuristics against typical markers of AI-written text: no semicolon in the prose, at most one dash per paragraph, no first-person plural. Every one of these heuristics came out of a piece of feedback from me. The agent writes drafts, I correct, and the correction becomes the rule for the next article.

The way onto the live site remains manual. I copy the text into the WordPress block editor, set images and SEO fields, and publish. Afterwards the agent verifies remotely via live HTML and REST API. For the article on the functional aesthetics of SQL it found four defects across the two language versions that way, defects invisible in the repository because they only arise in the editor: an image that had been pointing at another article's file for months, an SEO title that lost its last letter on paste, a jump link that led nowhere because of an old HTML anchor, and, in the English version, a link to the article on SSMS editor options that pointed at the German instead of the English version. I corrected that link twice in the block editor, and twice it stayed wrong. The agent read the post's modification timestamp via the REST API: It had not moved on either attempt, so my save clicks had not been saves at all. The cause was the editor's link dialog. Whoever types in the new URL and then clicks outside the dialog instead of confirming it with Enter or "Apply" discards the change, even if they press "Update" afterwards. On the third attempt the link held. The finding has been a note in the rule file for live verification ever since.

The Review Comes from a Fresh Session

A language model can also evaluate a text, and I have used that from the start. The methodology, however, I had to develop myself. While reviewing an article I noticed that a model anchors on its earlier verdicts within the same chat session: first rating about 8 out of 10, after several revisions in the same session 9.7, in a fresh session clearly more critical again. The high scores were not quality but history. Since then every review runs in a fresh session of a model from a different vendor, with a frozen standard prompt. The score is a diagnosis, not a goal.

I hand the result back to the agent as a raw paste, and it sorts it into three categories: adopt, matter of taste, already addressed. Experience from more than twenty reviews: 30 to 50 percent of the points are valid. The review is deliberately context-free, and precisely for that reason it needs as its counterpart the agent with repository context, which can look up in the revision log that a suggestion was already rejected months ago. A review is a finder, not proof. When one flagged a statement as "too absolute", the agent's cross-check showed it was not too absolute but wrong. What settled it in the end was a test against running database containers.

Fresh, though, means more than having no history. A new session of the same model that wrote the text would have no prior context, but it would bring the same blind spots: the patterns I read as markers of AI-written prose are more likely caught by a model from a different vendor than by one from the same family. And a session inside my development environment sees the repository, loads the rules automatically and can look up in the log what the template deliberately leaves out. So the counted review stays with the outside model and the same template for every article, otherwise the scores cannot be compared. A fresh session of my own model has a different place: as a rule check that holds the text against the blog’s style heuristics and conventions. That is a different question from “is the text good”, and it needs exactly the context the reviewer must not have. These review experiences will get an article of their own.

What Generalizes

The agentic coding experience in both projects produced the same patterns. Four of them carry my daily work.

The guidance lives in the repo, not in the prompt. Rule files, specs, bug files and revision logs are the actual work. The prompts only name the entry point and the assignment. The intermediate state belongs there too: What an agent has worked out is recorded promptly as a commit or a note in the feature or bug file. Language models live on text. What is not written down does not exist for the next session.

The structuring work is the price of the gain in radius. I get an application I would not have built alone and a blog I would not sustain alone. In return I take on roles that are spread across a team: writing requirements, formulating acceptance criteria, declaring test cases, maintaining documentation.

The repo learns, not the agent. Every session starts without the history of the previous one. What was learned from a mistake exists for the next session only if it is written down: as a rule or a checkpoint in the repository. That also works backwards: Many defects I correct in old articles today were not violations when they were written, because the rule only came afterwards. The existing body ages relative to the rules that were learned from it.

Learning remains mandatory. This blog holds the position that formatting is learning: Whoever indents a statement by hand builds the mental model of the data. For agentic coding that applies with more force. Whoever does nothing but sign off loses, over time, the model they need for signing off.

Where It Gets Rough: Limits from a User's Perspective

The limits are the part of my agentic coding experience that has occupied me the most.

Code Drift Despite Rules

A rule with a stated reason improves the hit rate but does not guarantee compliance, and at high volume every residual rate turns into measurable drift. The frontend case with 799 hardcoded font sizes is the evidence, and the answer was not a better rule but a check that does not get tired. The blog's editorial work knows the same effect: Despite the style heuristics, every first draft from the agent contains semicolons or compressed notation, and I recognize AI-written text by exactly these patterns. What is in the rules still has to be checked, only the checking moves from reading to counting.

Collateral Damage in the Working Tree and on My Machine

As soon as more than one session works in the same repository, or one session works alongside me, errors arise that do not exist in single operation: someone else's work in my own commit, deleted files of a parallel session, numbers assigned twice, and once 220 infinite loops that froze my computer for twelve minutes. Against the shared index, only git commit --only with explicit paths turned out to be reliable, and the lesson from the first case would not have prevented the second. The five cases and what holds against them are in Two Agents, One Working Tree.

You Can Only Sign Off on What You Can Judge

Review effort is the new bottleneck. The answer is not to read faster but to shift acceptance: to runs instead of reading, to criteria instead of impression, to external reviews instead of my own goodwill. Each of these shifts presupposes that I can formulate the criteria and recognize a result as right or wrong. With the Node guard I could, because I knew the SQL conventions. With a script whose domain logic I could not judge, the same division of labor would be a risk.

Added to that is a weakness that belongs to the tool's strength. An agent usually tests hypotheses systematically with measurements, and precisely because of that it hardly shows on the occasions when it builds a coherent story from incomplete clues instead. During one troubleshooting session the agent formulated a plausible cause as if it had been measured. It had not been. Since then I ask of every finding which part of it was measured.

Working Prompts as Examples

The prompts in both projects are short because the guidance lives in the repository. Three examples show what a prompt still has to do then.

Starting a revision series. Verbatim apart from the translation from German, only the internal identifiers are replaced by their meaning:

let's start the triage of the legacy articles. all relevant changes in the
articles and the lessons from revising them go into the article about agentic
coding

Two lines are enough because the spec, the work list and the rule files exist in versioned form. The agent read them, ran the first quality pass and established the material collection for this article as a side duty of the series.

Having an article reviewed externally. This prompt goes to ChatGPT in a fresh session, that is, to a model from a different vendor. It is versioned and has been frozen since August 2026. The original is German, shown here in translation and shortened:

You are a critical, independent technical editor for a technical blog about
SQL Server, Postgres, data quality and ETL. You know neither the author nor
earlier versions of the text, and you have no reason for leniency.

Rate the article in four dimensions with a score from 1 to 10 each:
technical correctness, structure and readability, SEO fitness, language and style.

Duties: Every point of criticism quotes its location verbatim. For every point
a concrete suggestion for improvement. No courtesy points. At the end, an overall
score and the five most important points, prioritized by impact.

--- ARTICLE ---

The most important sentence is the second one. Without it, the model rates the history instead of the text. The duty to quote the location verbatim makes the result sortable, because a point without a location can neither be implemented nor rejected with reasons.

A recurring check as a skill. A skill is a versioned prompt that I call with a slash command. This is how the description of the skill that checks for updates begins, verbatim apart from the translation and the replaced file paths:

Checks updates for dev dependencies (npm, Docker image declarations, GitHub
Actions) AND for the components that actually run on the VPS (OS packages,
Docker Engine, nginx, running container tags, Certbot). Writes findings to the
update documentation and to a dated security report. Keeps a history of
applied/deferred updates. Modifies nothing on its own.

The skill defines what is checked, where the result goes and what the agent is not allowed to do. For the server side it produces a checklist of read-only commands that I run myself. The acceptance criterion is the history, which shows me at the next run what was left undone.

How This Article Came About

This article is about transparency, so it has to deliver it itself. The text is a draft by the agent, produced from a spec and a material file that agent sessions kept over six weeks. Into this file, whichever session was working wrote down, on my instruction, what happened during the work, its own mistakes included. The first version was twice as long and written in the third person. That it became shorter, stands in the first person and mentions the infrastructure at all is the result of a discussion in which I asked the agent to listen first before rewriting.

My role is the same as in both case studies. I decided that this article exists and what it carries. I read the draft, discard, cut, correct and approve. I have it rated in a fresh session of another model, transfer it into WordPress by hand and check the links. What I do not do is write the text myself. And what I cannot do is in the text: write a Node script, see an effect in swap, retrieve a data point from a lost session. I had no prior experience of what this way of working would bring. What I underestimated was not the tool but the share of rules, specs, test cases and documentation that I had previously received from others.

FAQ

Can you tell that a text was produced with AI assistance?

Often, yes. Such texts give themselves away through recurring patterns. In German, in my experience, these are above all semicolons in the prose, verbless afterthoughts behind a dash, parentheses into which a whole piece of information is packed, and abstract terms that appear as the subject of a sentence and act there like a person. I have recorded these patterns in a rule file, and every draft is read against it before publication. Whether you can tell how a text came about therefore depends less on the model than on the correction pass that a human makes with a concrete list.

Can you build a TypeScript application without TypeScript skills?

Yes, under conditions. You need a model of how an application is cut, where state lives and where errors originate. In my case that came from years with .NET and databases. You need acceptance criteria that can be checked without reading the code: runs, tests, audits. And the infrastructure underneath remains a hurdle of its own that no programming language makes smaller.

How do you control what you cannot judge yourself?

Not by reading, but through a loop of checks that work without understanding the code: rules that are sharpened after every piece of feedback, skills for recurring checks, tests that have to be red on the old version, and audits with a report. The loop does not replace understanding, but it replaces reading. Every verdict has to state its open points as a condition, otherwise a "passed" travels further than its caveat.

Where do you start as a user?

With a single convention that lives in the repository and one assignment that has an acceptance criterion. Not with a large rule set, because that grows out of feedback. Anyone who wants to set up a database project with Claude Code will find a concrete entry point in the article on the open starter kit, and the mechanics behind it are explained by the cluster's hub article.

Is the effort worth it for a solo project?

Going by my agentic coding experience in both projects, yes. The time saved on programming is largely eaten up by the structuring work. What remains is a gain in radius: a TypeScript application including infrastructure that I would not have built alone, and a blog with spec, review protocol and revision log that a single person would not sustain without an agent. The condition is that you take on the roles that project leads and teams otherwise carry.

Hub:

AI-Assisted SQL Development with Claude Code — Rules, Skills and Agents That Enforce Conventions — the enforcement system whose user experience this article describes.

Spokes of this article:

One VPS, Four Environments, No Cookie Banner — How I Host a Next.js App with Self-Hosted Keycloak — the setup, the decisions behind it and the cost side that this article only touches on.
Two Agents, One Working Tree — What Broke When I Ran Parallel Claude Code Sessions in One Git Repository — five cases from parallel operation and the countermeasures that hold.
The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent — four debugging sessions in which measuring replaced guessing, along with the limit of that way of working.
The Control Loop — How I Get AI-Generated Code Checked That I Can't Read Myself — the five stations of the loop, their evidence and their blind spots.
An EU Region Is Not an EU Provider — Vercel, Supabase and the GDPR Third-Country Question — the second infrastructure spoke: why an EU region settles the storage location but not the provider question, with a checklist.
The Context Window Is Full Before the First Prompt — When Project Rules in Claude Code Become a Fixed Cost of Every Session — the tooling spoke: what the rule files cost at the start of every session, measured before and after, and why the dividing line is rule versus record.

Infrastructure and deploy:

Database CI/CD with PostgreSQL — the lifecycle from object file to deploy.
GitHub Actions for Postgres Deploys — a Throwaway Database as Quality Gate — the deploy gate behind the infrastructure.
Deploying a SQL Schema Without a Migration Tool — the directory convention of the database layer.

Rules, drift, guard:

I Gave Claude Code 27 Rule Files Instead of One CLAUDE.md — structure and upkeep of the rule files.
AI-Assisted Coding Gave Me 799 Hardcoded Font Sizes — the measured drift finding.
Why sqlfluff Can't Lint Our SQL Conventions — and a 240-Line Script Can — the guard I could not have written myself.
Deriving SQL Conventions with Claude Code — the Generate-Refine-Derive Loop — how the rule files come about.

Getting started:

Setting Up a Claude Code Project with a Development Workflow and Database — the Open Starter Kit Explained — the practical beginning.

Next door:

The Functional Aesthetics of SQL — Why Structured Code Is Faster to Edit — the thesis "formatting is learning".
Is Claude a Woman or a Man? — and Why We Ask in the First Place — the other non-technical question about the tool.

Design Pattern // Logging an ETL Process with T-SQL — How to Capture Run, Component and Action in Evaluable Log Tables

Marcus — Sat, 05 Sep 2026 10:10:41 +0000

An ETL process finishes without an exception — but was everything really loaded that should have been? The mere fact that a process did not abort says nothing about whether it actually did what was expected of it. A readable, evaluable log is what turns a gut feeling into a defensible statement.

This design pattern logs an ETL run on three levels and answers the questions that success or failure hinge on:

How long does the ETL process take overall?
How long does a single component take — a stored procedure, an SSIS package or another building block?
How long does a specific SQL statement take?
How many rows did a SQL statement actually affect?
And above all: did the process as a whole, a component or a single statement complete successfully?

TL;DR — what this article covers:

Three-tier logging — the tables [LL].[Execution], [LL].[Component] and [LL].[Trace] capture run, component and action at increasing granularity.
Stored-procedure toolkit — procedures such as [LL].[spInsertTrace] and [LL].[spUpdateTrace] write and update the log records.
Exception handling with [LL].[Error] — a TRY/CATCH pattern ends the run in an orderly way and records every error in an evaluable form.
Continuation of the ETL architecture — the example uses schema T2 from the architecture article and adds the logging layer.

Prerequisites. SQL Server and a basic understanding of stored procedures and TRY/CATCH. This article is part of the ETL design-pattern cluster. As a lead-in, see Data Quality in an ETL Process and The Architecture of an ETL Process.

The Result

At its core the approach is straightforward: at the start of every action a log record is written, and once the action finishes it is updated with the outcome success or failure — and, where useful, with further information. In principle, that is all there is to it. In practice, a little more is needed after all.

Even though creating a log should not be a big deal, there is quite a bit to say about it. This article starts with the result and works backwards from there. The process logging presented here is three-tier and uses the following log tables:

[LL].[Execution]
[LL].[Component]
[LL].[Trace]

These tables log an ETL process and its associated components and steps at increasing granularity — read from top to bottom. The ETL process, the components and the individual steps are each logged with exactly one record.

Closely tied to logging the process is logging errors. The approach presented here uses the following table for that:

[LL].[Error]

The following figures show the result of three-tier logging for a simple, compact, but complete ETL process.

In the table [LL].[Trace] every action is logged with, among other things, the name of the procedure, the target entity the procedure processes, a short description of what was actually done, and further information such as the number of rows affected and the execution time.

Table [LL].[Trace]

In the table [LL].[Component] the calls of procedures and SSIS packages — or, more generally, components — are logged. Here, too, a short description of the component’s task, the target entity and the execution time are recorded.

Table [LL].[Component]

At the top level, every execution of the ETL process is logged with exactly one record in the table [LL].[Execution].

Table [LL].[Execution]

The three process-logging tables — [LL].[Execution], [LL].[Component] and [LL].[Trace] — contain two columns, [State] and [Success], in which the success or failure of an action, a component or the ETL process is stored.

Now the Derivation…

After this brief preview of the result, the following aspects of the approach need to be clarified:

Three-Tier Logging
Table Description, Declaration and Data Model
ETL Process Execution Status
Logging Procedures
Exception Handling
Example: Logging and Exception Handling in an ETL Process

Three-Tier Logging

The approach logs an ETL process in the three tables [LL].[Execution], [LL].[Component] and [LL].[Trace]. Each table is intended for logging specific artifacts. This section introduces how each table is used, its columns and the code to create it.

[LL].[Execution]

This table logs the execution of an ETL process with exactly one record. An ETL process needs a clearly identifiable entry point. That can be a stored procedure, an SSIS package, a Talend job or a SQL Server Agent job. At the start of the entry point’s execution, a log record is inserted into this table. After all tasks have been processed successfully, this record is updated to status success, or to error in the failure case. The table thus provides an overview of all executions of the ETL process, along with information such as the status and the duration of the run.

[LL].[Component]

This table logs the execution of a component with exactly one record. A component can be a stored procedure, an SSIS package or a Talend job. A component is characterized by the fact that it controls and performs one or more data manipulations. As with the [LL].[Execution] table, a log record is inserted at the start of execution and updated to status success or error on completion. The table therefore holds, per ETL run, a list of the executed components, along with information such as the status and the duration of the run.

[LL].[Trace]

The name of this table already hints that it is intended for detailed logging of the ETL process’s actions — it creates a trace. Which actions are logged is a design decision for the developer. It is advisable, however, to log at least every INSERT, UPDATE and DELETE statement with its own record. In production systems, this granularity should be weighed deliberately against log volume and write load — a single INSERT … SELECT can move millions of rows and still remains exactly one trace record. Besides the status fields already mentioned, the table also stores the number of rows affected, which lets the developer judge whether the statements did exactly what was expected.

[LL].[Error]

Errors detected in an ETL process belong in the log. A distinction has to be made between logging exceptions and logging data errors. The structure of this table is designed for logging data errors and contains columns in which every data error can be recorded completely and in an evaluable form. This article focuses on logging an ETL process rather than logging data errors. Logging the process is closely tied to explicit exception handling, and a separate section is dedicated to exception handling and the logging of exceptions.

The LL Schema Name

The schema name LL stands for Logging Layer. This schema holds all log tables and the stored procedures used for logging.

Table Description, Declaration and Data Model

The following sections describe the tables for logging the process as well as the table for logging errors. Finally, a diagram shows the data model of these tables.

Three notes up front: The duration of an execution derives, at the execution level, from the columns [Start] and [End], and at the component and trace levels from [CreatedOn] (written when the action starts) and [ModifiedOn] (set by the closing update via trigger). The timestamps use datetime with GETUTCDATE() for historical reasons — for a new design, datetime2 with SYSUTCDATETIME() would be the natural choice today, and its higher precision pays off precisely for trace records written in quick succession. And the audit columns [ModifiedOn]/[ModifiedBy] are maintained by an AFTER UPDATE trigger per table: that keeps the logging procedures lean, but costs an additional UPDATE on every status change and is a deliberate trade-off at very high log volumes. With the default database setting RECURSIVE_TRIGGERS OFF, the trigger's self-update does not recurse (measured on SQL Server 2022).

[LL].[Execution]

Columns

The table has the following columns:

Column	Data type	Null?	Description
`[Id]`	`bigint` (IDENTITY)	NOT NULL	Primary key, sequential run id.
`[Process]`	`nvarchar(max)`	NOT NULL	Name of the ETL process.
`[Start]`	`datetime`	NOT NULL	Start time of the run (UTC).
`[End]`	`datetime`	NULL	End time of the run.
`[DeltaStart]`	`datetime`	NULL	Start of the delta window (for incremental loads).
`[DeltaEnd]`	`datetime`	NULL	End of the delta window.
`[User]`	`nvarchar(128)`	NULL	Executing DB login (`SUSER_SNAME()`).
`[Machine]`	`nvarchar(128)`	NULL	Host the run was started from.
`[Version]`	`int`	NULL	Version number of the ETL process.
`[State]`	`nvarchar(128)`	NOT NULL	Status: processing / warning / success / error.
`[Success]`	`bit`	NOT NULL	0 = not (successfully) completed, 1 = successful.
`[CreatedOn]`	`datetime`	NOT NULL	Creation time, default `GETUTCDATE()`.
`[CreatedBy]`	`nvarchar(100)`	NOT NULL	Creating login, default `SUSER_SNAME()`.
`[ModifiedOn]`	`datetime`	NULL	Last modification, set by the update trigger.
`[ModifiedBy]`	`nvarchar(100)`	NULL	Login of the last modification, set by the update trigger.

Declaration

The table is created with the following statement:

-- ----------------------------------------------------------------------------
-- [LL].[Execution] - logging table (top level): exactly one record per ETL
-- run. Inserted at the start of the entry procedure with [State] = 'processing'
-- and updated to 'success' or 'error' at the end.
-- ----------------------------------------------------------------------------
CREATE TABLE [LL].[Execution]
(
    [Id]           bigint        IDENTITY (1, 1) NOT NULL
   ,[Process]      nvarchar(max)                 NOT NULL
   ,[Start]        datetime                      NOT NULL
   ,[End]          datetime                          NULL
   ,[DeltaStart]   datetime                          NULL
   ,[DeltaEnd]     datetime                          NULL
   ,[User]         nvarchar(128)                     NULL
   ,[Machine]      nvarchar(128)                     NULL
   ,[Version]      int                               NULL
   ,[State]        nvarchar(128)                 NOT NULL
   ,[Success]      bit                           NOT NULL
   ,[CreatedOn]    datetime
       CONSTRAINT [DF_LL_Execution_CreatedOn]
       DEFAULT (GETUTCDATE())                    NOT NULL
   ,[CreatedBy]    nvarchar(100)
       CONSTRAINT [DF_LL_Execution_CreatedBy]
       DEFAULT (SUSER_SNAME())                   NOT NULL
   ,[ModifiedOn]   datetime                          NULL
   ,[ModifiedBy]   nvarchar(100)                     NULL
   ,CONSTRAINT [PK_LL_Execution]
       PRIMARY KEY CLUSTERED ([Id] ASC)
   ,CONSTRAINT [CK_LL_Execution_StateSuccess]
       CHECK (   ([State] = N'processing' AND [Success] = 0)
              OR ([State] = N'warning'    AND [Success] IN (0, 1))
              OR ([State] = N'success'    AND [Success] = 1)
              OR ([State] = N'error'      AND [Success] = 0))
);
GO

CREATE TRIGGER [LL].[TR_LL_Execution_Update]
ON [LL].[Execution]
FOR UPDATE
AS
BEGIN
    SET NOCOUNT ON;

    UPDATE [LL].[Execution]
       SET
           [ModifiedOn] = GETUTCDATE()
          ,[ModifiedBy] = SUSER_SNAME()
       FROM
          [LL].[Execution]
          INNER JOIN inserted
            ON inserted.[Id] = [LL].[Execution].[Id];
END;
GO

[LL].[Component]

Columns

Column	Data type	Null?	Description
`[Id]`	`bigint` (IDENTITY)	NOT NULL	Primary key of the component.
`[ExecutionId]`	`bigint`	NOT NULL	Foreign key to `[LL].[Execution]`.
`[Source]`	`nvarchar(5)`	NOT NULL	Type of source (e.g. SSIS, T-SQL).
`[Component]`	`nvarchar(128)`	NOT NULL	Name of the component (procedure, package, job).
`[Version]`	`int`	NULL	Version number of the component.
`[Entity]`	`nvarchar(128)`	NOT NULL	Target entity the component processes.
`[Step]`	`nvarchar(max)`	NOT NULL	Description of the step.
`[Description]`	`nvarchar(max)`	NULL	Additional description.
`[FileId]`	`bigint`	NULL	Reference to a processed file.
`[State]`	`nvarchar(128)`	NOT NULL	Status: processing / warning / success / error.
`[Success]`	`bit`	NOT NULL	0 = not (successfully) completed, 1 = successful.
`[CreatedOn]`	`datetime`	NOT NULL	Creation time, default `GETUTCDATE()`.
`[CreatedBy]`	`nvarchar(100)`	NOT NULL	Creating login, default `SUSER_SNAME()`.
`[ModifiedOn]`	`datetime`	NULL	Last modification, set by the update trigger.
`[ModifiedBy]`	`nvarchar(100)`	NULL	Login of the last modification, set by the update trigger.

Declaration

-- ----------------------------------------------------------------------------
-- [LL].[Component] - logging table (middle level): exactly one record per
-- component call (stored procedure, SSIS package, Talend job).
-- Foreign key to [LL].[Execution].
-- ----------------------------------------------------------------------------
CREATE TABLE [LL].[Component]
(
    [Id]           bigint        IDENTITY (1, 1) NOT NULL
   ,[ExecutionId]  bigint                        NOT NULL
   ,[Source]       nvarchar(5)                   NOT NULL
   ,[Component]    nvarchar(128)                 NOT NULL
   ,[Version]      int                               NULL
   ,[Entity]       nvarchar(128)                 NOT NULL
   ,[Step]         nvarchar(max)                 NOT NULL
   ,[Description]  nvarchar(max)                     NULL
   ,[FileId]       bigint                            NULL
   ,[State]        nvarchar(128)                 NOT NULL
   ,[Success]      bit                           NOT NULL
   ,[CreatedOn]    datetime
       CONSTRAINT [DF_LL_Component_CreatedOn]
       DEFAULT (GETUTCDATE())                    NOT NULL
   ,[CreatedBy]    nvarchar(100)
       CONSTRAINT [DF_LL_Component_CreatedBy]
       DEFAULT (SUSER_SNAME())                   NOT NULL
   ,[ModifiedOn]   datetime                          NULL
   ,[ModifiedBy]   nvarchar(100)                     NULL
   ,CONSTRAINT [PK_LL_Component]
       PRIMARY KEY CLUSTERED ([Id] ASC)
   ,CONSTRAINT [FK_LL_Component_ExecutionId]
       FOREIGN KEY ([ExecutionId])
       REFERENCES [LL].[Execution] ([Id])
   ,CONSTRAINT [CK_LL_Component_StateSuccess]
       CHECK (   ([State] = N'processing' AND [Success] = 0)
              OR ([State] = N'warning'    AND [Success] IN (0, 1))
              OR ([State] = N'success'    AND [Success] = 1)
              OR ([State] = N'error'      AND [Success] = 0))
);
GO

CREATE TRIGGER [LL].[TR_LL_Component_Update]
ON [LL].[Component]
FOR UPDATE
AS
BEGIN
    SET NOCOUNT ON;

    UPDATE [LL].[Component]
       SET
           [ModifiedOn] = GETUTCDATE()
          ,[ModifiedBy] = SUSER_SNAME()
       FROM
          [LL].[Component]
          INNER JOIN inserted
            ON inserted.[Id] = [LL].[Component].[Id];
END;
GO

[LL].[Trace]

Columns

Column	Data type	Null?	Description
`[Id]`	`bigint` (IDENTITY)	NOT NULL	Primary key of the trace record.
`[ExecutionId]`	`bigint`	NOT NULL	Foreign key to `[LL].[Execution]`.
`[ComponentId]`	`bigint`	NOT NULL	Foreign key to `[LL].[Component]`.
`[Source]`	`nvarchar(5)`	NOT NULL	Type of source (e.g. SSIS, T-SQL).
`[Component]`	`nvarchar(128)`	NOT NULL	Name of the calling component.
`[Task]`	`nvarchar(128)`	NULL	Task name (e.g. SSIS task).
`[Entity]`	`nvarchar(128)`	NULL	Target entity of the action.
`[Step]`	`nvarchar(max)`	NOT NULL	Description of the step.
`[Description]`	`nvarchar(max)`	NULL	Additional description.
`[FileId]`	`bigint`	NULL	Reference to a processed file.
`[Action]`	`nvarchar(100)`	NULL	Type of action (insert / update / delete / …).
`[AffectedRows]`	`bigint`	NULL	Number of rows affected by the action.
`[State]`	`nvarchar(100)`	NOT NULL	Status: processing / warning / success / error.
`[Success]`	`bit`	NOT NULL	0 = not (successfully) completed, 1 = successful.
`[CreatedOn]`	`datetime`	NOT NULL	Creation time, default `GETUTCDATE()`.
`[CreatedBy]`	`nvarchar(100)`	NOT NULL	Creating login, default `SUSER_SNAME()`.
`[ModifiedOn]`	`datetime`	NULL	Last modification, set by the update trigger.
`[ModifiedBy]`	`nvarchar(128)`	NULL	Login of the last modification, set by the update trigger.

Declaration

-- ----------------------------------------------------------------------------
-- [LL].[Trace] - logging table (finest level): exactly one record per single
-- action (INSERT/UPDATE/DELETE, single SQL step, single task). Foreign keys
-- to [LL].[Execution] and [LL].[Component].
-- ----------------------------------------------------------------------------
CREATE TABLE [LL].[Trace]
(
    [Id]           bigint        IDENTITY (1, 1) NOT NULL
   ,[ExecutionId]  bigint                        NOT NULL
   ,[ComponentId]  bigint                        NOT NULL
   ,[Source]       nvarchar(5)                   NOT NULL
   ,[Component]    nvarchar(128)                 NOT NULL
   ,[Task]         nvarchar(128)                     NULL
   ,[Entity]       nvarchar(128)                     NULL
   ,[Step]         nvarchar(max)                 NOT NULL
   ,[Description]  nvarchar(max)                     NULL
   ,[FileId]       bigint                            NULL
   ,[Action]       nvarchar(100)                     NULL
   ,[AffectedRows] bigint                            NULL
   ,[State]        nvarchar(100)                 NOT NULL
   ,[Success]      bit                           NOT NULL
   ,[CreatedOn]    datetime
       CONSTRAINT [DF_LL_Trace_CreatedOn]
       DEFAULT (GETUTCDATE())                    NOT NULL
   ,[CreatedBy]    nvarchar(100)
       CONSTRAINT [DF_LL_Trace_CreatedBy]
       DEFAULT (SUSER_SNAME())                   NOT NULL
   ,[ModifiedOn]   datetime                          NULL
   ,[ModifiedBy]   nvarchar(128)                     NULL
   ,CONSTRAINT [PK_LL_Trace]
       PRIMARY KEY CLUSTERED ([Id] ASC)
   ,CONSTRAINT [FK_LL_Trace_ComponentId]
       FOREIGN KEY ([ComponentId])
       REFERENCES [LL].[Component] ([Id])
   ,CONSTRAINT [FK_LL_Trace_ExecutionId]
       FOREIGN KEY ([ExecutionId])
       REFERENCES [LL].[Execution] ([Id])
   ,CONSTRAINT [CK_LL_Trace_StateSuccess]
       CHECK (   ([State] = N'processing' AND [Success] = 0)
              OR ([State] = N'warning'    AND [Success] IN (0, 1))
              OR ([State] = N'success'    AND [Success] = 1)
              OR ([State] = N'error'      AND [Success] = 0))
);
GO

CREATE TRIGGER [LL].[TR_LL_Trace_Update]
ON [LL].[Trace]
FOR UPDATE
AS
BEGIN
    SET NOCOUNT ON;

    UPDATE [LL].[Trace]
       SET
           [ModifiedOn] = GETUTCDATE()
          ,[ModifiedBy] = SUSER_SNAME()
       FROM
          [LL].[Trace]
          INNER JOIN inserted
            ON inserted.[Id] = [LL].[Trace].[Id];
END;
GO

[LL].[Error]

As mentioned above, the structure of this table is designed for logging data errors, which are not the subject of this article. The table is, however, also used for logging exceptions. The following description covers only the columns required for logging an exception.

Columns

Column	Data type	Null?	Description
`[Id]`	`bigint` (IDENTITY)	NOT NULL	Primary key of the error record.
`[ExecutionId]`	`bigint`	NOT NULL	Foreign key to `[LL].[Execution]`.
`[ComponentId]`	`bigint`	NULL	Foreign key to `[LL].[Component]` (if known).
`[TraceId]`	`bigint`	NULL	Foreign key to `[LL].[Trace]` (if known).
`[ErrorType]`	`char(1)`	NOT NULL	Type of error (exception vs. data error).
`[Source]`	`nvarchar(5)`	NOT NULL	Type of source (e.g. SSIS, T-SQL).
`[Component]`	`nvarchar(128)`	NOT NULL	Component in which the error occurred.
`[TaskName]`	`nvarchar(128)`	NULL	Task name.
`[Entity]`	`nvarchar(128)`	NULL	Affected target entity.
`[Step]`	`nvarchar(max)`	NULL	Step in which the error occurred.
`[Description]`	`nvarchar(max)`	NULL	Error text (`ERROR_MESSAGE()`).
`[Number]`	`int`	NULL	Error number (`ERROR_NUMBER()`).
`[Line]`	`int`	NULL	Error line (`ERROR_LINE()`).
`[State]`	`nvarchar(max)`	NULL	Error state (`ERROR_STATE()`).
`[CreatedOn]`	`datetime`	NOT NULL	Creation time, default `GETUTCDATE()`.
`[CreatedBy]`	`nvarchar(100)`	NOT NULL	Creating login, default `SUSER_SNAME()`.

Declaration

-- ----------------------------------------------------------------------------
-- [LL].[Error] - logging table for exceptions and data errors.
-- In the context of this article we only use the exception columns;
-- the data-error columns (ID1Value/ID2Value/ID3Value/ErrorValue,
-- ID*ColumnName, FileName, FileId) are relevant for a follow-up article.
-- ----------------------------------------------------------------------------
CREATE TABLE [LL].[Error]
(
    [Id]              bigint        IDENTITY (1, 1) NOT NULL
   ,[ExecutionId]     bigint                        NOT NULL
   ,[ComponentId]     bigint                            NULL
   ,[TraceId]         bigint                            NULL
   ,[ErrorType]       char(1)                       NOT NULL
   ,[Source]          nvarchar(5)                   NOT NULL
   ,[Component]       nvarchar(128)                 NOT NULL
   ,[TaskName]        nvarchar(128)                     NULL
   ,[Entity]          nvarchar(128)                     NULL
   ,[Step]            nvarchar(max)                     NULL
   ,[SchemaName]      nvarchar(128)                     NULL
   ,[TableName]       nvarchar(128)                     NULL
   ,[FileId]          bigint                            NULL
   ,[ID1Value]        nvarchar(max)                     NULL
   ,[ID1ColumnName]   nvarchar(128)                     NULL
   ,[ID2Value]        nvarchar(max)                     NULL
   ,[ID2ColumnName]   nvarchar(128)                     NULL
   ,[ID3Value]        nvarchar(max)                     NULL
   ,[ID3ColumnName]   nvarchar(128)                     NULL
   ,[ErrorValue]      nvarchar(max)                     NULL
   ,[ErrorColumnName] nvarchar(128)                     NULL
   ,[FileName]        nvarchar(128)                     NULL
   ,[Description]     nvarchar(max)                     NULL
   ,[Number]          int                               NULL
   ,[Line]            int                               NULL
   ,[State]           nvarchar(max)                     NULL
   ,[CreatedOn]       datetime
       CONSTRAINT [DF_LL_Error_CreatedOn]
       DEFAULT (GETUTCDATE())                     NOT NULL
   ,[CreatedBy]       nvarchar(100)
       CONSTRAINT [DF_LL_Error_CreatedBy]
       DEFAULT (SUSER_SNAME())                    NOT NULL
   ,CONSTRAINT [PK_LL_Error]
       PRIMARY KEY CLUSTERED ([Id] ASC)
   ,CONSTRAINT [FK_LL_Error_ExecutionId]
       FOREIGN KEY ([ExecutionId])
       REFERENCES [LL].[Execution] ([Id])
   ,CONSTRAINT [FK_LL_Error_ComponentId]
       FOREIGN KEY ([ComponentId])
       REFERENCES [LL].[Component] ([Id])
   ,CONSTRAINT [FK_LL_Error_TraceId]
       FOREIGN KEY ([TraceId])
       REFERENCES [LL].[Trace] ([Id])
);
GO

Data Model

One deliberate redundancy in [LL].[Trace] stands out: [ExecutionId] sits there in addition to [ComponentId], so that run-level evaluations can skip the join through [LL].[Component]. The database does not enforce the consistency of that pair — a trace record could in theory point to a component of a different run. Anyone who wants to enforce it adds a composite foreign key on [Component]([Id], [ExecutionId]).

ETL Process Execution Status

The three process-log tables have two columns, [State] and [Success], which store the current status of the ETL process, of a component’s execution or of a specific action — for example an INSERT, UPDATE or DELETE. The current status is held in the [State] column with the values processing, warning, success and error. Success is stored in the [Success] column as 1. Only the combination of both values reveals the current status reliably. The following combinations are valid:

`[State]`	`[Success]`	Meaning
processing	0	Action, component or run has started and is still running.
warning	0	Finished, but with a warning — not counted as success.
warning	1	Finished with a warning, yet counted as success.
success	1	Completed successfully.
error	0	Aborted with an error.

Valid combinations of the [State] and [Success] columns

Other combinations of status values are not allowed. This is enforced twice: the process-logging procedures raise an exception if an invalid combination is passed in, and a CHECK constraint per table additionally rejects invalid combinations at the database level (error 547, measured on SQL Server 2022).

Logging Procedures

The following procedures, among others, are available for inserting and updating log records in the tables above:

Procedure	Schema	Purpose
`spInsertExecution`	`LL`	Opens the execution log (top level) at the start of the run.
`spUpdateExecution`	`LL`	Closes the execution log with status success or error.
`spInsertComponent`	`LL`	Opens a component log for a component.
`spUpdateComponentSuccess`	`LL`	Closes a component log successfully.
`spUpdateComponentError`	`LL`	Closes a component log in the error case.
`spInsertTrace`	`LL`	Writes a trace record for a single action.
`spUpdateTrace`	`LL`	Updates a trace record (general).
`spUpdateTraceSuccess`	`LL`	Updates a trace record to success / 1.
`spUpdateTraceError`	`LL`	Updates a trace record to error / 0.
`spInsertErrorException`	`LL`	Writes a caught exception to `[LL].[Error]`.

Procedures for process logging

In essence these procedures perform an INSERT or UPDATE on the log tables. The values to be logged are passed as parameters. The procedures validate the parameters passed in and raise an exception in case of invalid parameters. For parameter validation and for raising errors consistently, they use the two helpers [dbo].[spRaiseError] and [dbo].[fnIsNullOrEmpty].

The following code examples show the procedures [LL].[spInsertTrace], [LL].[spUpdateTrace] and [LL].[spUpdateTraceSuccess]. In addition, [LL].[spInsertExecution] stands in for all the remaining procedures, which follow the same pattern: validate the parameters, run the INSERT or UPDATE, return the generated id.

[LL].[spInsertTrace]

-- ----------------------------------------------------------------------------
-- [LL].[spInsertTrace] - inserts a trace record into [LL].[Trace]
-- and returns the generated id via @p_traceId.
-- ----------------------------------------------------------------------------
-- Parameters:
--    @p_executionId    bigint        Execution id of the current ETL run
--    @p_componentId    bigint        Component id of the calling component log
--    @p_traceId        bigint OUTPUT Id of the newly inserted trace record
--    @p_source          nvarchar(5)   Source system (SSIS, T-SQL, ...)
--    @p_component       nvarchar(128) Name of the calling component
--    @p_task            nvarchar(128) Task name (e.g. SSIS task), optional
--    @p_entity          nvarchar(128) Target entity, optional
--    @p_step            nvarchar(max) Description of the step
--    @p_description     nvarchar(max) Additional description, optional
--    @p_fileId         bigint        Reference to [LL].[FileList].[Id], optional
--    @p_action          nvarchar(100) Action label (Insert/Update/Delete/...)
--    @p_affectedRows   bigint        Number of rows affected
--    @p_state           nvarchar(100) processing / warning / success / error
--    @p_success         bit           0 = processing/warning/error, 1 = success
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [LL].[spInsertTrace]
    @p_executionId    AS bigint
   ,@p_componentId    AS bigint
   ,@p_traceId        AS bigint OUTPUT
   ,@p_source         AS nvarchar(5)
   ,@p_component      AS nvarchar(128)
   ,@p_task           AS nvarchar(128)   = NULL
   ,@p_entity         AS nvarchar(128)   = NULL
   ,@p_step           AS nvarchar(max)
   ,@p_description    AS nvarchar(max)   = NULL
   ,@p_fileId         AS bigint          = NULL
   ,@p_action         AS nvarchar(100)   = NULL
   ,@p_affectedRows   AS bigint          = NULL
   ,@p_state          AS nvarchar(100)
   ,@p_success        AS bit
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @component AS nvarchar(128);
    DECLARE @table     AS TABLE ([Id] bigint);
    DECLARE @message   AS nvarchar(max);

    SET @component = OBJECT_SCHEMA_NAME(@@PROCID) + N'.' + OBJECT_NAME(@@PROCID);

    BEGIN TRY
        -- Parameter checks
        IF (@p_executionId IS NULL)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_executionId'' is NULL.', @component;
            RETURN 0;
        END;

        IF (@p_componentId IS NULL)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_componentId'' is NULL.', @component;
            RETURN 0;
        END;

        IF ([dbo].[fnIsNullOrEmpty](@p_source, 1) <> 0)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_source'' is either NULL or an empty string.', @component;
            RETURN 1;
        END;

        IF ([dbo].[fnIsNullOrEmpty](@p_component, 1) <> 0)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_component'' is either NULL or an empty string.', @component;
            RETURN 1;
        END;

        IF ([dbo].[fnIsNullOrEmpty](@p_step, 1) <> 0)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_step'' is either NULL or an empty string.', @component;
            RETURN 1;
        END;

        IF ([dbo].[fnIsNullOrEmpty](@p_state, 1) <> 0)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_state'' is either NULL or an empty string.', @component;
            RETURN 1;
        END;

        IF (@p_success IS NULL)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_success'' is NULL.', @component;
            RETURN 1;
        END;

        -- Validate the State/Success combination (whitelist)
        IF NOT (   (@p_state = N'processing' AND @p_success = 0)
                OR (@p_state = N'warning'    AND @p_success IN (0, 1))
                OR (@p_state = N'success'    AND @p_success = 1)
                OR (@p_state = N'error'      AND @p_success = 0))
        BEGIN
            SET @message = CONCAT(N'Invalid state ''', @p_state, N''' for p_success = ''', CAST(@p_success AS nvarchar(100)), N'''.');
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN 1;
        END;

        -- Write the trace record
        INSERT INTO [LL].[Trace]
        (
             [ExecutionId]
            ,[ComponentId]
            ,[Source]
            ,[Component]
            ,[Task]
            ,[Entity]
            ,[Step]
            ,[Description]
            ,[FileId]
            ,[Action]
            ,[AffectedRows]
            ,[State]
            ,[Success]
        )
        OUTPUT Inserted.[Id] INTO @table
        VALUES
        (
             @p_executionId
            ,@p_componentId
            ,@p_source
            ,@p_component
            ,@p_task
            ,@p_entity
            ,@p_step
            ,CASE WHEN @p_description IS NULL OR DATALENGTH(@p_description) = 0
                  THEN NULL ELSE @p_description END
            ,@p_fileId
            ,@p_action
            ,@p_affectedRows
            ,@p_state
            ,@p_success
        );

        SELECT @p_traceId = [Id] FROM @table;

        RETURN 0;
    END TRY
    BEGIN CATCH
        THROW;
    END CATCH;
END;
GO

[LL].[spUpdateTrace]

-- ----------------------------------------------------------------------------
-- [LL].[spUpdateTrace] - updates an existing trace record
-- with Description, Action, AffectedRows, State and Success.
-- ----------------------------------------------------------------------------
-- Parameters:
--    @p_traceId        bigint        Id of the trace record to update
--    @p_description     nvarchar(max) Description of the result
--    @p_action          nvarchar(100) Action label, optional
--    @p_affectedRows   bigint        Number of rows affected
--    @p_state           nvarchar(100) processing / warning / success / error
--    @p_success         bit           0 = processing/warning/error, 1 = success
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [LL].[spUpdateTrace]
    @p_traceId         AS bigint
   ,@p_description     AS nvarchar(max)
   ,@p_action          AS nvarchar(100) = NULL
   ,@p_affectedRows    AS bigint
   ,@p_state           AS nvarchar(100)
   ,@p_success         AS bit
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @component AS nvarchar(128);
    DECLARE @tempId    AS bigint;
    DECLARE @message   AS nvarchar(max);

    SET @component = OBJECT_SCHEMA_NAME(@@PROCID) + N'.' + OBJECT_NAME(@@PROCID);

    BEGIN TRY
        -- Parameter checks
        IF (@p_traceId IS NULL)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_traceId'' is NULL.', @component;
            RETURN 1;
        END;

        IF (@p_success IS NULL)
        BEGIN
            SET @message = N'The parameter ''p_success'' is NULL.';
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN 1;
        END;

        IF ([dbo].[fnIsNullOrEmpty](@p_state, 1) <> 0)
        BEGIN
            SET @message = N'The parameter ''p_state'' is either NULL or an empty string.';
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN 1;
        END;

        -- Validate the State/Success combination (whitelist)
        IF NOT (   (@p_state = N'processing' AND @p_success = 0)
                OR (@p_state = N'warning'    AND @p_success IN (0, 1))
                OR (@p_state = N'success'    AND @p_success = 1)
                OR (@p_state = N'error'      AND @p_success = 0))
        BEGIN
            SET @message = CONCAT(N'Invalid state ''', @p_state, N''' for p_success = ''', CAST(@p_success AS nvarchar(100)), N'''.');
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN 1;
        END;

        -- Check that the trace record exists
        SELECT @tempId = [Id]
        FROM   [LL].[Trace]
        WHERE  [Id] = @p_traceId;

        IF (@tempId IS NULL)
        BEGIN
            SET @message = N'A record with [Id] = ''' + CAST(@p_traceId AS nvarchar(max)) + N''' could not be found.';
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN 1;
        END;

        -- Update the trace record
        UPDATE [LL].[Trace]
           SET
               [Description]  = CASE
                                  WHEN (@p_description IS NULL OR DATALENGTH(@p_description) = 0)
                                   AND ([Description]   IS NULL OR DATALENGTH([Description])   = 0)
                                  THEN NULL ELSE @p_description END
              ,[Action]       = @p_action
              ,[AffectedRows] = @p_affectedRows
              ,[State]        = @p_state
              ,[Success]      = @p_success
         WHERE [Id] = @p_traceId;

        RETURN 0;
    END TRY
    BEGIN CATCH
        THROW;
    END CATCH;
END;
GO

[LL].[spUpdateTraceSuccess]

-- ----------------------------------------------------------------------------
-- [LL].[spUpdateTraceSuccess] - convenience wrapper around spUpdateTrace for
-- the common case "action finished successfully" (State = 'success',
-- Success = 1). Saves setting both fields on every call.
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [LL].[spUpdateTraceSuccess]
    @p_traceId         AS bigint
   ,@p_description     AS nvarchar(max)
   ,@p_action          AS nvarchar(100) = NULL
   ,@p_affectedRows    AS bigint
AS
BEGIN
    SET NOCOUNT ON;

    EXEC [LL].[spUpdateTrace]
         @p_traceId
        ,@p_description
        ,@p_action
        ,@p_affectedRows
        ,N'success'
        ,1;
END;
GO

[LL].[spInsertExecution]

The procedure below was not shown in detail in the original example, but it stands in for all the remaining logging procedures: once you understand how it is built, you can derive spInsertComponent, spUpdateComponent*, spUpdateTraceError and spInsertErrorException analogously — the only difference is the column assignment.

-- ----------------------------------------------------------------------------
-- [LL].[spInsertExecution] - reference for the remaining logging procedures.
--
-- Structurally identical to [LL].[spInsertTrace]: parameter checks -> INSERT
-- -> id returned via OUTPUT variable. Once you understand the pattern, you can
-- spInsertComponent, spUpdateComponent*, spUpdateTraceError,
-- derive spInsertErrorException, spUpdateExecution analogously - only the
-- column assignment differs.
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [LL].[spInsertExecution]
    @p_executionId    AS bigint OUTPUT
   ,@p_process        AS nvarchar(max)
   ,@p_version        AS int = NULL
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @component AS nvarchar(128);
    DECLARE @table     AS TABLE ([Id] bigint);

    SET @component = OBJECT_SCHEMA_NAME(@@PROCID) + N'.' + OBJECT_NAME(@@PROCID);

    BEGIN TRY
        IF ([dbo].[fnIsNullOrEmpty](@p_process, 1) <> 0)
        BEGIN
            EXEC [dbo].[spRaiseError] N'The parameter ''p_process'' is either NULL or an empty string.', @component;
            RETURN 1;
        END;

        INSERT INTO [LL].[Execution]
        (
             [Process]
            ,[Start]
            ,[User]
            ,[Machine]
            ,[Version]
            ,[State]
            ,[Success]
        )
        OUTPUT Inserted.[Id] INTO @table
        VALUES
        (
             @p_process
            ,GETUTCDATE()
            ,SUSER_SNAME()
            ,HOST_NAME()
            ,@p_version
            ,N'processing'
            ,0
        );

        SELECT @p_executionId = [Id] FROM @table;

        RETURN 0;
    END TRY
    BEGIN CATCH
        THROW;
    END CATCH;
END;
GO

Exception Handling

To ensure that an ETL process ends in an orderly way rather than aborting hard without proper logging, explicit exception handling is required. Orderly means in this case that the process catches a raised exception, performs an UPDATE on the relevant log record in the tables [LL].[Execution], [LL].[Component] and [LL].[Trace] — wherever applicable — setting [State] = error and [Success] = 0, and logs the exception in the table [LL].[Error]. The following diagram shows the basic mechanics of exception handling:

Exception handling

Only after logging has completed may the process re-raise the exception to the caller, which would then abort the calling process hard. Re-raising the exception to the caller is not strictly necessary, however, once the error has been logged and the relevant log records have been updated to [State] = error and [Success] = 0.

Exception handling is required at least at the top level of an ETL process’s execution, which is logged in the table [LL].[Execution].

One point deserves particular attention here: the interplay of logging and transactions. If the ETL work runs inside an open transaction and the logging procedures write in the same session, a ROLLBACK in the CATCH block also rolls back the log records already written — of all cases, it is the failure case that loses its log. If an error additionally puts the transaction into the uncommittable state (XACT_STATE() = -1, the usual case for runtime errors in a TRY block under SET XACT_ABORT ON), an INSERT into the log tables fails in the CATCH block with error 3930 until the ROLLBACK has been executed (measured on SQL Server 2022). The order in the CATCH block is therefore: roll back the transaction first, then log. The example in this article works without an explicit transaction and deliberately sidesteps the problem.

And the error logging itself is not guaranteed to succeed. If one of the logging calls in the CATCH block fails — say, on missing permissions or a full log —, its exception masks the original error. Anyone who wants to guard against that wraps the logging calls in the CATCH block in a TRY/CATCH of their own: the final THROW then re-raises the original exception in every case.

The code example below shows exception handling including the logging into the tables [LL].[Execution] and [LL].[Error] in line with the diagram above:

-- ----------------------------------------------------------------------------
-- Exception-handling skeleton: shows the complete pattern of
-- TRY/CATCH + insert error + update execution + THROW. The order in the
-- CATCH block is crucial - log first, then THROW.
-- ----------------------------------------------------------------------------
-- Execution logging variables
DECLARE @executionId     AS bigint;
DECLARE @processName     AS nvarchar(max);
DECLARE @version         AS int;

-- Error logging variables
DECLARE @componentId     AS bigint;
DECLARE @traceId         AS bigint;
DECLARE @source          AS nvarchar(5);
DECLARE @component       AS nvarchar(128);
DECLARE @task            AS nvarchar(128);
DECLARE @entity          AS nvarchar(128);
DECLARE @step            AS nvarchar(128);
DECLARE @state           AS nvarchar(128);
DECLARE @success         AS bit;

-- Exception variables (lowercase by convention)
DECLARE @error_message   AS nvarchar(max);
DECLARE @error_number    AS int;
DECLARE @error_line      AS int;
DECLARE @error_state     AS nvarchar(max);

BEGIN TRY
    -- Initialize the execution logging variables
    SET @processName  = N'Name of ETL-Process';
    SET @version      = 123;

    -- Initialize the error logging variables
    SET @componentId  = NULL;
    SET @traceId      = NULL;
    SET @source       = N'sql';
    SET @component    = N'Procedure Name';
    SET @task         = NULL;
    SET @entity       = N'Any Entity';
    SET @step         = N'Do something';

    EXEC [LL].[spInsertExecution] @executionId OUTPUT, @processName, @version;

    -- Workload (the actual ETL work happens here)
    THROW 50001, N'Any Exception', 1;

    SET @state   = N'success';
    SET @success = 1;
    EXEC [LL].[spUpdateExecution] @executionId, @state, @success;
END TRY
BEGIN CATCH
    SET @error_message = ERROR_MESSAGE();
    SET @error_number  = ERROR_NUMBER();
    SET @error_line    = ERROR_LINE();
    SET @error_state   = ERROR_STATE();

    SET @state   = N'error';
    SET @success = 0;

    IF @executionId IS NOT NULL
    BEGIN
        EXEC [LL].[spInsertErrorException]
             @executionId
            ,@componentId
            ,@traceId
            ,@source
            ,@component
            ,@task
            ,@entity
            ,@step
            ,@error_number
            ,@error_message
            ,@error_line
            ,@error_state;

        EXEC [LL].[spUpdateExecution] @executionId, @state, @success;
    END;

    THROW;
END CATCH;

Example: Logging and Exception Handling in an ETL Process

After these explanations of the basics of exception handling, the following diagram shows the exception handling of a simple, compact, but complete ETL process:

Example of exception handling and logging for an example ETL process

This example shows the logging of an ETL process built entirely from stored procedures, consisting of five stored procedures. The entry procedure [T2].[spETLProcess] calls the two procedures [T2].[spDoSomething_1] and [T2].[spDoSomething_2]. The first one logs an INSERT, an UPDATE and a DELETE step each, while the second procedure executes two further procedures, [T2].[spDoSomething_2_1] and [T2].[spDoSomething_2_2]. These procedures, too, log an INSERT, an UPDATE and a DELETE step each — the DML statements themselves are deliberately only sketched as placeholder comments in the demo procedures. During the execution of the DELETE statement in the procedure [T2].[spDoSomething_2_2], an exception is raised that leads to an orderly termination of the ETL process.

Procedure calls of the example ETL process

Since showing all procedures would lead to repetition, only the entry procedure [T2].[spETLProcess] and the procedure [T2].[spDoSomething_2_2] are shown here. All remaining worker procedures use the same TRY/CATCH pattern as [T2].[spDoSomething_2_2].

Two decisions in the example are deliberate. The worker procedures re-raise a caught exception via THROW, so that every level can close its own log record. The entry procedure [T2].[spETLProcess], by contrast, consumes the exception after logging is complete — the run ends in an orderly way with [State] = error, without aborting the caller hard. If an orchestrator such as SQL Server Agent is supposed to see the failure, an additional THROW belongs at the end of its CATCH block, otherwise the job ends as spuriously successful from the scheduler's point of view.

And when passing on the row count, order matters: @@ROWCOUNT is captured into the variable @affectedRows immediately after the respective DML statement, and only that variable is passed to [LL].[spUpdateTraceSuccess]. After an EXEC, @@ROWCOUNT does not carry the business row count of the called component but the value of its last executed statement, and even a simple variable assignment sets the value to 1 (measured on SQL Server 2022). The orchestration steps in [T2].[spETLProcess] therefore log no row count of their own — the workers deliver it in their trace records. In the demo workers, SET @affectedRows = @@ROWCOUNT; deliberately remains part of the placeholder comment, and the placeholder steps pass NULL, because without real DML there would be no business row count to capture. For statements beyond 2 billion rows, ROWCOUNT_BIG() takes the place of @@ROWCOUNT. The column [AffectedRows] is sized as bigint for that.

[T2].[spETLProcess]

-- ----------------------------------------------------------------------------
-- [T2].[spETLProcess] - example entry procedure of the ETL process.
-- Calls two worker procedures spDoSomething_1 and spDoSomething_2
-- and logs both calls in [LL].[Trace].
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [T2].[spETLProcess]
AS
BEGIN
    SET NOCOUNT ON;

    -- Error variables
    DECLARE @error_message  AS nvarchar(max);
    DECLARE @error_number   AS int;
    DECLARE @error_line     AS int;
    DECLARE @error_state    AS nvarchar(max);

    -- Logging variables
    DECLARE @component      AS nvarchar(128);
    DECLARE @task           AS nvarchar(128);
    DECLARE @source         AS nvarchar(5);
    DECLARE @step           AS nvarchar(max);
    DECLARE @entity         AS nvarchar(max);
    DECLARE @message        AS nvarchar(max);

    DECLARE @traceId       AS bigint;
    DECLARE @componentId   AS bigint;
    DECLARE @executionId   AS bigint;

    DECLARE @description    AS nvarchar(max);
    DECLARE @affectedRows   AS bigint;
    DECLARE @action         AS nvarchar(100);
    DECLARE @state          AS nvarchar(100);
    DECLARE @success        AS bit;

    SET @message       = NULL;
    SET @description   = NULL;
    SET @affectedRows  = 0;

    SET @component = OBJECT_SCHEMA_NAME(@@PROCID) + N'.' + OBJECT_NAME(@@PROCID);
    SET @source    = N'T-SQL';
    SET @entity    = N'[].[]';

    BEGIN TRY
        -- Open the execution log
        EXEC [LL].[spInsertExecution] @executionId OUTPUT, N'ETL process', 123;

        -- Open the component log
        SET @step        = N'Orchestrate ETL process';
        SET @description = N'';
        EXEC [LL].[spInsertComponent]
             @executionId, @componentId OUTPUT, @source, @component, NULL, @entity, @step, @description;

        -- ----------------------------------------------------------------
        -- Call [T2].[spDoSomething_1]
        -- ----------------------------------------------------------------
        SET @task        = NULL;
        SET @step        = N'Execute [T2].[spDoSomething_1]';
        SET @action      = N'execute';
        SET @description = NULL;
        SET @state       = N'processing';
        SET @success     = 0;
        EXEC [LL].[spInsertTrace]
             @executionId, @componentId, @traceId OUTPUT
            ,@source, @component, @task, @entity, @step, @description
            ,NULL, @action, NULL, @state, @success;

        EXEC [T2].[spDoSomething_1] @executionId;

        -- Orchestration step without a row count of its own - the DML rows
        -- are logged by the workers in their own trace records
        EXEC [LL].[spUpdateTraceSuccess] @traceId, @description, @action, NULL;

        -- ----------------------------------------------------------------
        -- Call [T2].[spDoSomething_2]
        -- ----------------------------------------------------------------
        SET @task        = NULL;
        SET @step        = N'Execute [T2].[spDoSomething_2]';
        SET @action      = N'execute';
        SET @description = NULL;
        SET @state       = N'processing';
        SET @success     = 0;
        EXEC [LL].[spInsertTrace]
             @executionId, @componentId, @traceId OUTPUT
            ,@source, @component, @task, @entity, @step, @description
            ,NULL, @action, NULL, @state, @success;

        EXEC [T2].[spDoSomething_2] @executionId;

        EXEC [LL].[spUpdateTraceSuccess] @traceId, @description, @action, NULL;

        -- Close the component log
        EXEC [LL].[spUpdateComponentSuccess] @componentId, @description;

        -- Close the execution log
        SET @state   = N'success';
        SET @success = 1;
        EXEC [LL].[spUpdateExecution] @executionId, @state, @success;
    END TRY
    BEGIN CATCH
        SET @error_message = ERROR_MESSAGE();
        SET @error_number  = ERROR_NUMBER();
        SET @error_line    = ERROR_LINE();
        SET @error_state   = ERROR_STATE();

        IF @executionId IS NOT NULL
        BEGIN
            EXEC [LL].[spInsertErrorException]
                 @executionId, @componentId, @traceId, @source, @component
                ,NULL, NULL, @step
                ,@error_number, @error_message, @error_line, @error_state;

            IF @traceId     IS NOT NULL EXEC [LL].[spUpdateTraceError]     @traceId,     @description;
            IF @componentId IS NOT NULL EXEC [LL].[spUpdateComponentError] @componentId, @description;

            SET @state   = N'error';
            SET @success = 0;
            EXEC [LL].[spUpdateExecution] @executionId, @state, @success;
        END;
    END CATCH;
END;
GO

[T2].[spDoSomething_2_2]

-- ----------------------------------------------------------------------------
-- [T2].[spDoSomething_2_2] - deepest worker procedure in the example tree.
-- Called by [T2].[spDoSomething_2] (see diagram 041007.png).
-- Simulates an INSERT, an UPDATE and a DELETE - the DELETE raises an
-- exception to demonstrate the exception-handling pattern.
-- ----------------------------------------------------------------------------
CREATE OR ALTER PROCEDURE [T2].[spDoSomething_2_2]
(
    @p_executionId  AS bigint
)
AS
BEGIN
    SET NOCOUNT ON;

    -- Error variables
    DECLARE @error_message  AS nvarchar(max);
    DECLARE @error_number   AS int;
    DECLARE @error_line     AS int;
    DECLARE @error_state    AS nvarchar(max);

    -- Logging variables
    DECLARE @component      AS nvarchar(128);
    DECLARE @task           AS nvarchar(128);
    DECLARE @source         AS nvarchar(5);
    DECLARE @step           AS nvarchar(max);
    DECLARE @entity         AS nvarchar(max);
    DECLARE @message        AS nvarchar(max);

    DECLARE @traceId       AS bigint;
    DECLARE @componentId   AS bigint;

    DECLARE @description    AS nvarchar(max);
    DECLARE @affectedRows   AS bigint;
    DECLARE @action         AS nvarchar(100);
    DECLARE @state          AS nvarchar(100);
    DECLARE @success        AS bit;

    SET @message       = NULL;
    SET @description   = NULL;
    SET @affectedRows  = 0;

    SET @component = OBJECT_SCHEMA_NAME(@@PROCID) + N'.' + OBJECT_NAME(@@PROCID);
    SET @source    = N'T-SQL';
    SET @entity    = N'[].[]';

    BEGIN TRY
        IF (@p_executionId IS NULL)
        BEGIN
            SET @message = N'The parameter ''p_executionId'' is NULL.';
            EXEC [dbo].[spRaiseError] @message, @component;
            RETURN -1;
        END;

        -- Open the component log
        SET @step        = N'Do something';
        SET @description = N'';
        EXEC [LL].[spInsertComponent]
             @p_executionId, @componentId OUTPUT, @source, @component, NULL, @entity, @step, @description;

        -- ----------------------------------------------------------------
        -- INSERT (succeeds)
        -- ----------------------------------------------------------------
        SET @task        = NULL;
        SET @step        = N'Insert data';
        SET @action      = N'insert';
        SET @description = NULL;
        SET @state       = N'processing';
        SET @success     = 0;
        EXEC [LL].[spInsertTrace]
             @p_executionId, @componentId, @traceId OUTPUT
            ,@source, @component, @task, @entity, @step, @description
            ,NULL, @action, NULL, @state, @success;

        -- An INSERT statement would go here, immediately followed by:
        -- SET @affectedRows = @@ROWCOUNT;

        EXEC [LL].[spUpdateTraceSuccess] @traceId, @description, @action, NULL;

        -- ----------------------------------------------------------------
        -- UPDATE (succeeds)
        -- ----------------------------------------------------------------
        SET @task        = NULL;
        SET @step        = N'Update data';
        SET @action      = N'update';
        SET @description = NULL;
        SET @state       = N'processing';
        SET @success     = 0;
        EXEC [LL].[spInsertTrace]
             @p_executionId, @componentId, @traceId OUTPUT
            ,@source, @component, @task, @entity, @step, @description
            ,NULL, @action, NULL, @state, @success;

        -- An UPDATE statement would go here, immediately followed by:
        -- SET @affectedRows = @@ROWCOUNT;

        EXEC [LL].[spUpdateTraceSuccess] @traceId, @description, @action, NULL;

        -- ----------------------------------------------------------------
        -- DELETE (deliberately raises an exception)
        -- ----------------------------------------------------------------
        SET @task        = NULL;
        SET @step        = N'Delete data';
        SET @action      = N'delete';
        SET @description = NULL;
        SET @state       = N'processing';
        SET @success     = 0;
        EXEC [LL].[spInsertTrace]
             @p_executionId, @componentId, @traceId OUTPUT
            ,@source, @component, @task, @entity, @step, @description
            ,NULL, @action, NULL, @state, @success;

        -- A DELETE statement would go here - artificially raised exception:
        THROW 50001, N'Exception in [T2].[spDoSomething_2_2]', 1;

        EXEC [LL].[spUpdateTraceSuccess] @traceId, @description, @action, NULL;

        -- Close the component log
        EXEC [LL].[spUpdateComponentSuccess] @componentId, @description;
    END TRY
    BEGIN CATCH
        SET @error_message = ERROR_MESSAGE();
        SET @error_number  = ERROR_NUMBER();
        SET @error_line    = ERROR_LINE();
        SET @error_state   = ERROR_STATE();

        IF @p_executionId IS NOT NULL
        BEGIN
            EXEC [LL].[spInsertErrorException]
                 @p_executionId, @componentId, @traceId, @source, @component
                ,NULL, NULL, @step
                ,@error_number, @error_message, @error_line, @error_state;

            IF @traceId     IS NOT NULL EXEC [LL].[spUpdateTraceError]     @traceId,     @description;
            IF @componentId IS NOT NULL EXEC [LL].[spUpdateComponentError] @componentId, @description;
        END;

        THROW;
    END CATCH;
END;
GO

The following script contains the statement to execute the ETL process. To obtain a compact process log, the log tables are cleared beforehand. This clearing, including DBCC CHECKIDENT, is a pure demo reset for reproducible ids — in live operation, log tables are not emptied. The closing SELECT statements produce the result shown at the beginning.

-- ----------------------------------------------------------------------------
-- Example run: execute the ETL process and evaluate the log.
--
-- Block 1 clears the log tables (TRUNCATE / DELETE + RESEED),
-- block 2 starts the example ETL process, block 3 shows the result.
-- The SELECT statements return exactly the three result sets shown at the
-- start of the article (tables 041001, 041002, 041003).
-- ----------------------------------------------------------------------------

-- 01: Clear the log tables
TRUNCATE TABLE [LL].[Error];
TRUNCATE TABLE [LL].[Trace];
DELETE FROM    [LL].[Component];
DELETE FROM    [LL].[Execution];
DBCC CHECKIDENT (N'[LL].[Component]', RESEED, 0);
DBCC CHECKIDENT (N'[LL].[Execution]', RESEED, 0);

-- 02: Start the ETL process
EXEC [T2].[spETLProcess];

-- 03: Evaluate the log
SELECT * FROM [LL].[Execution];
SELECT * FROM [LL].[Component];
SELECT * FROM [LL].[Trace];

Conclusion

ETL processes are data-driven processes. If data arrives that was not expected, there is a high probability that either not all data — or even wrong data — ends up in the target system. Something this article mentions only in passing: at the lowest logging level, the number of rows affected can also be logged. Knowing the number of expected versus actually processed rows is a good indicator of the success or failure of an ETL process.

This approach thus supports the development of robust ETL processes and at the same time makes the data quality of a run assessable.

What goes without saying in software development is often neglected when building ETL processes: explicit exception handling. The exception handling presented here first ensures that a process ends in an orderly way in the event of an error. But when is there actually an error? And does an error always have to abort the process? Combined with the knowledge of the expected versus actually processed number of rows, genuine error handling can be implemented.

The approach presented here is, in a way, an invitation to engage more with the data and the expected result. The log — and in particular the knowledge of the rows processed — helps the developer judge the correctness of the work already during the development of the ETL process.

The procedures presented here merely provide a toolkit for logging. Used correctly, they produce a readable, evaluable log.

FAQ

What is the difference between [LL].[Component] and [LL].[Trace]?

Both log steps of a run, but at different levels of granularity. [LL].[Component] holds exactly one record per called component (stored procedure, SSIS package, Talend job) — that is, "which building block ran when and with what status?". [LL].[Trace] goes one level deeper and logs the individual actions within a component (typically every INSERT, UPDATE and DELETE) together with the number of rows affected. A component therefore usually has several trace entries.

Why a dedicated [LL] schema instead of putting the logging tables in the default schema?

The LL schema ("Logging Layer") cleanly separates the logging infrastructure from the business data. This has practical advantages: permissions can be granted precisely (for example, write access to LL only for the ETL procedures), retention and backup of the logs can be controlled separately, and in scripts it is immediately clear that an object belongs to logging. It also prevents the logging tables from colliding with business tables of the same name.

Can the pattern be implemented with Postgres instead of SQL Server?

Yes — the concept is database-neutral: the logging tables and the three-tier structure stay the same. The concrete implementation, however, has to be adapted to PL/pgSQL semantics. Instead of THROW, you re-raise the exception in PL/pgSQL with RAISE, TRY/CATCH becomes a BEGIN … EXCEPTION WHEN OTHERS THEN … END block, and @@ROWCOUNT corresponds to GET DIAGNOSTICS <var> = ROW_COUNT. One important difference remains: an EXCEPTION block forms an implicit subtransaction in PL/pgSQL. If it catches an error, all changes of the block are rolled back — including log records already written (measured on PostgreSQL 16). The error logging therefore belongs in the EXCEPTION branch — what the handler itself writes is kept. If you want to measure the execution time of individual statements without your own trace table, you can additionally draw on pg_stat_statements, as an aggregated statement statistic without the run-level correlation via execution, component and trace ids.

How does this pattern differ from SQL Server Audit or pgAudit?

SQL Server Audit and pgAudit are database-side auditing mechanisms: they log database and server events — logins, DDL, also object access such as SELECT or INSERT — at the infrastructure level, without the business correlation of an ETL run via execution, component and trace ids. The pattern presented here, by contrast, is application logging — it records the business view of the ETL process: which run, which component, which action, how many rows, success or failure. The two complement each other. For the question "did the ETL run do the right thing functionally?" application logging is the appropriate tool.

Why are log entries missing after an error with a rollback?

Because the log INSERTs ran in the same transaction as the ETL work. A ROLLBACK in the CATCH block then also rolls back the log records already written. If the error puts the transaction into the uncommittable state (XACT_STATE() = -1, the usual case for runtime errors in a TRY block under SET XACT_ABORT ON), an INSERT in the CATCH block even fails with error 3930 as long as the ROLLBACK is outstanding. In the CATCH block, the rule is therefore: roll back the transaction first, then log — or keep the logging outside the surrounding transaction in the first place (see the Exception Handling section).

How do you log the number of rows affected with @@ROWCOUNT?

Capture the value of @@ROWCOUNT into a variable immediately after the INSERT, UPDATE or DELETE statement and pass that variable to the logging procedure — every further statement in between overwrites the value. In this pattern, the number ends up in the [AffectedRows] column of the trace record via [LL].[spUpdateTraceSuccess]. For statements with more than 2 billion rows, ROWCOUNT_BIG() takes the place of @@ROWCOUNT. The column [AffectedRows] is sized as bigint accordingly.

How do I integrate an SSIS package into the pattern?

An SSIS package is treated like any other component: at the start it calls [LL].[spInsertComponent] (for example from an Execute SQL Task or a Script Task), performs its tasks and finishes with [LL].[spUpdateComponentSuccess], or with [LL].[spUpdateComponentError] in the error case. The returned ComponentId is passed along through an SSIS variable so that downstream trace calls can reference it. This way the package appears seamlessly alongside the T-SQL components in the same log.

This article is part of the ETL design-pattern cluster:

Data Quality in an ETL Process — the root of the article series.
Design Pattern // The Architecture of an ETL Process — provides the schema T2 used in the example of this article.
ETL vs. ELT — How to Tell Which Pattern You Actually Built — classifies the pattern logged here as ETL vs. ELT.
Checking Data Quality with SQL — a Configurable Framework — uses the same error-table idea for generic data quality checks.
Design Pattern // Safe Type Conversion with T-SQL
Data Quality // Fundamentals of Type Conversion with T-SQL

Deriving Data Quality Rules from the Schema — What the Metadata Already Knows

Marcus — Wed, 02 Sep 2026 11:13:56 +0000

The rule "country_code is mandatory" lives in your database twice: once as NOT NULL in the target table's schema, and once as a hand-typed row in the check configuration. On the next ALTER TABLE, only one of the two places changes, and the check silently goes wrong. With derived data quality rules you no longer type that repetition: the metadata already knows which columns are mandatory, which keys must be unique and which value ranges the types allow. And by pinning the derived state at deployment time, you detect schema drift instead of suffering it.

The essentials up front:

NOT NULL, key uniqueness and type bounds can be projected mechanically from information_schema — including the message text and the business key.
Projected at deployment time, not on every run: an ETL process is a contract between the source and the target system. The derived rules are generated and persisted at deployment, schema changes are not adopted automatically.
The live projection stays on as a drift detector: a diff against the deployed state shows when the contract has to be renegotiated.
Read-only is not a convenience: derived rules are neither edited nor deactivated, the is_active flag belongs to the manual rules. Whoever wants a change changes the schema and redeploys.
Handwritten rules remain for the business logic the schema does not contain. Exactly those deserve the review time.

Prerequisite: Postgres as the example engine and the framework from the sub-hub Data Quality Checks with SQL with its shared error table and the rule configuration dq.check_rule. The metadata principle needs only information_schema and thus carries over to any engine that offers it. The code shown uses Postgres syntax and has to be adapted to the dialect of any other engine.

The rule that already exists

The framework from the sub-hub works off a configuration table: one row is one rule. That is deliberate, because a new check is a data row instead of a code change, and the business side can read along. Such a row is still rolled out through a deployment — why it has to be that way is shown further down in this article. The price of the configuration only shows over time. A large share of the rows you type into such a configuration merely repeats what the target table enforces anyway:

  1: CREATE SCHEMA IF NOT EXISTS core;
  2: 
  3: CREATE TABLE core.customer
  4: (
  5:     customer_id    int          NOT NULL
  6:    ,source_system  varchar(10)  NOT NULL
  7:    ,source_id      varchar(20)  NOT NULL
  8:    ,country_code   varchar(2)   NOT NULL
  9:    ,email          varchar(100)
 10:    ,age            numeric(3,0)
 11:    ,CONSTRAINT pk_customer         PRIMARY KEY (customer_id)
 12:    ,CONSTRAINT uq_customer_source  UNIQUE (source_system, source_id)
 13: );

This one table already dictates seven check rules for the staging layer: four mandatory fields (lines 5 through 8), two unique keys (lines 11 and 12), plus the length and value-range bounds of the types. Whoever enters them into dq.check_rule by hand creates seven copies of a truth that already lives in the schema.

Copies drift. When a column phone varchar(30) NOT NULL is added later, someone has to remember to extend the configuration as well. If they forget, the check stays silent and the load fails at the target constraint — exactly the scenario the pre-filter was supposed to prevent. If, the other way around, a NOT NULL is dropped, the old rule keeps checking against a requirement that no longer exists and produces findings without a basis. Both failures are quiet. Nobody gets a message that configuration and schema have drifted apart.

Which data quality rules can be derived

There is a simple way out of the drift problem: derive these rules instead of typing them. Everything the target schema enforces as a constraint or as a declared type bound translates mechanically into a rule row. The mapping onto the framework's rule types is one to one:

Metadata source	Derived check	Rule type in the framework	Dimension
`is_nullable = 'NO'`	mandatory-field check	`constraint` (WHERE clause)	Completeness
`PRIMARY KEY` / `UNIQUE`	uniqueness, cardinality 1 (all key columns `NOT NULL`)	`unique`	Uniqueness
`FOREIGN KEY`	reference check against the master table (see the outlook below)	`lookup`	Consistency / Integrity
`character_maximum_length`, `numeric_precision`/`_scale`	length and value-range bound	`constraint` (WHERE clause)	Validity

Which quality dimensions these checks pay into is laid out by the concept article Data Quality: Dimensions and Error Classes. The pattern itself comes from a metadata-driven ETL project: there, the projection ran over a dedicated metadata model with business-side extras such as business and alternate keys and a per-column null handling. For this article it has been transferred to information_schema so that it can be followed without a model of your own. The principle stays the same: the rules live in the metadata, and the projection merely reads them out.

NOT NULL from is_nullable

The simplest projection. Every mandatory column of the target table becomes a mandatory-field check on the staging layer — as a finished row in the configuration table's format, including where_clause and message text:

  1: SELECT
  2:     'constraint'                            AS check_type
  3:    ,'staging'                               AS schema_name
  4:    ,T01.table_name
  5:    ,T01.column_name                         AS check_column
  6:    ,format('%I IS NULL', T01.column_name)   AS where_clause
  7:    ,'E'                                     AS severity
  8:    ,format('%s is mandatory in the target schema', T01.column_name) AS message
  9: FROM
 10:    information_schema.columns T01
 11: WHERE
 12:        T01.table_schema = 'core'
 13:    AND T01.table_name   = 'customer'
 14:    AND T01.is_nullable  = 'NO'
 15: ORDER BY
 16:    T01.ordinal_position;

For the demo table this yields four rows:

check_column	where_clause	severity	message
`customer_id`	`customer_id IS NULL`	E	customer_id is mandatory in the target schema
`source_system`	`source_system IS NULL`	E	source_system is mandatory in the target schema
`source_id`	`source_id IS NULL`	E	source_id is mandatory in the target schema
`country_code`	`country_code IS NULL`	E	country_code is mandatory in the target schema

Two details are worth a look. The format() with %I on line 6 quotes the column name as an identifier — a column name with special characters yields a correct where_clause, not a broken one. And the projection targets staging (line 3) although it reads from core: checking happens at the source, enforcing happens at the target. That is the division of labour of the whole framework. What would fail as a constraint at the target, the pre-filter finds beforehand.

Uniqueness from PRIMARY KEY and UNIQUE

Keys live in two catalog views: information_schema.table_constraints knows the constraints, information_schema.key_column_usage their columns. With multi-column keys, the columns must be sorted by position, that is by ordinal_position within the key. That field does not count the column's position in the table but its position in the key, so it reproduces exactly the order from the DDL. Without an ORDER BY, array_agg emits the columns in whatever order the join happens to deliver them, and that order is neither guaranteed nor stable over time in Postgres. A different execution plan is enough to turn source_system,source_id into source_id,source_system on the next run.

For the check result this would be harmless, because a GROUP BY over both columns finds the same duplicates in any order. The damage happens one level up, because check_column carries the key columns as a comma-separated list, and that list identifies the rule. At runtime the runner splits it back up, builds the multi-column GROUP BY from it, and writes the value belonging to each listed column into error_value in the error log. If the order flips, the same rule looks like a different one on the next run, the string comparison against the target constraint fails, and the column-value pairs in the log come out in a different order. The message suffers too, because anyone reading "Key (source_id, source_system) not unique" has to re-sort the columns against the DDL in their head. The projection therefore sorts exactly once, when collecting into the array:

  1: WITH
  2: CTE_key_column AS
  3: (
  4:    SELECT
  5:        T01.table_name
  6:       ,T01.constraint_name
  7:       ,array_agg(T02.column_name ORDER BY T02.ordinal_position) AS key_column
  8:    FROM
  9:       information_schema.table_constraints T01
 10:       INNER JOIN information_schema.key_column_usage T02
 11:       ON
 12:             T02.constraint_schema = T01.constraint_schema
 13:         AND T02.constraint_name   = T01.constraint_name
 14:       INNER JOIN information_schema.columns T03
 15:       ON
 16:             T03.table_schema = T02.table_schema
 17:         AND T03.table_name   = T02.table_name
 18:         AND T03.column_name  = T02.column_name
 19:    WHERE
 20:           T01.table_schema    = 'core'
 21:       AND T01.table_name      = 'customer'
 22:       AND T01.constraint_type IN ('PRIMARY KEY', 'UNIQUE')
 23:    GROUP BY
 24:        T01.table_name
 25:       ,T01.constraint_name
 26:    HAVING
 27:       bool_and(T03.is_nullable = 'NO')
 28: )
 29: SELECT
 30:     'unique'                                  AS check_type
 31:    ,'staging'                                 AS schema_name
 32:    ,table_name
 33:    ,array_to_string(key_column, ',')          AS check_column
 34:    ,1                                         AS max_occurrence
 35:    ,'E'                                       AS severity
 36:    ,format('Key (%s) not unique'
 37:           ,array_to_string(key_column, ', ')) AS message
 38: FROM
 39:    CTE_key_column;

The result: one rule per key, not one per column.

check_column	max_occurrence	message
`customer_id`	1	Key (customer_id) not unique
`source_system,source_id`	1	Key (source_system, source_id) not unique

The composite key from line 12 of the table DDL lands position-correct as the comma list source_system,source_id in check_column, which is exactly the form the runner expects and splits back up.

The HAVING on line 27 additionally draws a semantic line. Only keys whose columns are all NOT NULL get projected. The additional join on information_schema.columns takes care of that. The reason lies in the NULL semantics of UNIQUE: under the default NULLS DISTINCT, Postgres allows any number of missing values in a UNIQUE column, while the check's GROUP BY collapses several NULL into one group and would report them as duplicates. A rule projected without this filter onto a nullable key would be stricter than the constraint it claims to mirror — it would flag rows the target accepts without complaint. For a PRIMARY KEY the condition always holds. A UNIQUE over nullable columns deliberately stays out and belongs in a user-defined rule instead, with whatever NULL treatment the business actually means. How the uniqueness check deals with composite keys and the NULL semantics of UNIQUE in detail is covered by the spoke Finding Duplicates with SQL. The cardinality of derived key rules is always 1: a UNIQUE constraint knows no "at most three times".

The target table's foreign keys can be projected into lookup rules as well. This is deliberately an outlook rather than a recipe, because that projection needs more metadata than the two above. information_schema.referential_constraints names only the two constraints involved, that is the foreign key and the referenced primary or unique key. The columns themselves live in key_column_usage again, on both sides: the referencing columns come from a lookup by the foreign key's name in their ordinal_position, the referenced columns from a second lookup into the same view by the name of the target constraint. The two sides are joined through position_in_unique_constraint, which states for each foreign key column where its counterpart sits within the referenced key. Only that mapping yields the column pairs the check builds its LEFT JOIN … IS NULL from. For this, the lookup routine accepts the child and master columns each as a comma-separated list, and both lists have to be position-aligned: the first child field belongs to the first master field, pair by pair. The projection therefore has to produce both lists in exactly the same key order, or the generated check joins the wrong pairs. On NULL semantics, catalog and routine do agree: under MATCH SIMPLE, the Postgres default, a composite foreign key accepts every row in which even one of the participating columns is NULL, and the lookup check skips exactly those rows too, because a missing value is a completeness finding and not an integrity one. Anyone looking that match type up in the catalog finds it there as match_option = 'NONE', not as SIMPLE. A foreign key declared as MATCH FULL would show up as FULL and would need a stricter check, because it rejects partially filled keys — one more reason this projection stays an outlook. How the check for orphaned records itself works is covered by the corresponding spoke.

Type bounds from numeric_precision and character_maximum_length

The data types themselves are check rules too. A varchar(2) says "at most two characters", a numeric(3,0) says "absolute value below 1000". In the staging layer, which still holds such values as unchecked text or generous types, this becomes a value-range rule:

  1: SELECT
  2:     'constraint'      AS check_type
  3:    ,'staging'         AS schema_name
  4:    ,T01.table_name
  5:    ,T01.column_name   AS check_column
  6:    ,CASE
  7:        WHEN T01.character_maximum_length IS NOT NULL
  8:        THEN format('length(%I) > %s'
  9:                   ,T01.column_name
 10:                   ,T01.character_maximum_length)
 11:        ELSE format('abs(%I) >= %s'
 12:                   ,T01.column_name
 13:                   ,trim_scale(10::numeric ^ (T01.numeric_precision - T01.numeric_scale))::text)
 14:     END               AS where_clause
 15:    ,'E'               AS severity
 16:    ,CASE
 17:        WHEN T01.character_maximum_length IS NOT NULL
 18:        THEN format('%s longer than %s characters'
 19:                   ,T01.column_name
 20:                   ,T01.character_maximum_length)
 21:        ELSE format('%s outside numeric(%s,%s)'
 22:                   ,T01.column_name
 23:                   ,T01.numeric_precision
 24:                   ,T01.numeric_scale)
 25:     END               AS message
 26: FROM
 27:    information_schema.columns T01
 28: WHERE
 29:        T01.table_schema = 'core'
 30:    AND T01.table_name   = 'customer'
 31:    AND (   T01.character_maximum_length IS NOT NULL
 32:         OR T01.numeric_precision_radix = 10)
 33: ORDER BY
 34:    T01.ordinal_position;

check_column	where_clause	message
`source_system`	`length(source_system) > 10`	source_system longer than 10 characters
`source_id`	`length(source_id) > 20`	source_id longer than 20 characters
`country_code`	`length(country_code) > 2`	country_code longer than 2 characters
`email`	`length(email) > 100`	email longer than 100 characters
`age`	`abs(age) >= 1000`	age outside numeric(3,0)

The filter on line 32 is this projection's fine print. For int and bigint columns, numeric_precision counts bits, not decimal digits — information_schema.columns reveals this via numeric_precision_radix = 2. Without the radix filter, the projection would claim a bound of 10 to the power of 32 for customer_id, which neither matches the type semantics nor makes for a meaningful check. Only for explicitly declared numeric(p,s) types (radix 10) does the precision carry a business statement.

Project at deployment time, not on every run

What do you do with the three queries? Two obvious answers each come with a catch. The first: take the results into the configuration by hand via INSERT INTO dq.check_rule. That only automates the typing effort. The inserted rows would be editable copies, stale again after the next ALTER TABLE. The second: turn the projection into the runtime rule set directly, as a view, and every rule is fresh at every moment. Exactly that freshness is the second answer's catch.

An ETL process is a contract between the source and the target system, and that contract is valid at development time. When either side changes, the contract is renegotiated, not adjusted automatically. A live view would do exactly that, in both directions. If the target gets stricter, say through a new NOT NULL column, the run that was green yesterday fails today without any deployment on the ETL side. If the target gets laxer because a constraint is dropped, the view silently drops the corresponding check, and the pre-filter softens without anyone having decided it. Add the audit angle: which rules were in force for a given run is a question a view can, by principle, not answer.

For perspective: within a single application that validates its own schema, live derivation is the right model. There is no second contracting party there, the schema is the only truth. The contract logic begins as soon as two systems are involved.

The conclusion that holds therefore separates two points in time. The projection itself stays a view — the tool that can compute the derived rules fresh from the metadata at any moment:

  1: CREATE OR REPLACE VIEW dq.derived_rule AS
  2: WITH
  3: CTE_pk_column AS
  4: (
  5:    -- primary key of the target table = business key of the error table
  6:    SELECT
  7:        T01.table_name
  8:       ,array_agg(T02.column_name ORDER BY T02.ordinal_position) AS pk_column
  9:    FROM
 10:       information_schema.table_constraints T01
 11:       INNER JOIN information_schema.key_column_usage T02
 12:       ON
 13:             T02.constraint_schema = T01.constraint_schema
 14:         AND T02.constraint_name   = T01.constraint_name
 15:    WHERE
 16:           T01.table_schema    = 'core'
 17:       AND T01.constraint_type = 'PRIMARY KEY'
 18:    GROUP BY
 19:       T01.table_name
 20:    HAVING
 21:       count(*) <= 3
 22: )
 23: ,CTE_key_column AS
 24: (
 25:    -- every key as an ordered column array; only keys whose columns are
 26:    -- all NOT NULL (otherwise the rule would check more strictly than
 27:    -- the constraint guarantees - Postgres default NULLS DISTINCT)
 28:    SELECT
 29:        T01.table_name
 30:       ,T01.constraint_name
 31:       ,array_agg(T02.column_name ORDER BY T02.ordinal_position) AS key_column
 32:    FROM
 33:       information_schema.table_constraints T01
 34:       INNER JOIN information_schema.key_column_usage T02
 35:       ON
 36:             T02.constraint_schema = T01.constraint_schema
 37:         AND T02.constraint_name   = T01.constraint_name
 38:       INNER JOIN information_schema.columns T03
 39:       ON
 40:             T03.table_schema = T02.table_schema
 41:         AND T03.table_name   = T02.table_name
 42:         AND T03.column_name  = T02.column_name
 43:    WHERE
 44:           T01.table_schema    = 'core'
 45:       AND T01.constraint_type IN ('PRIMARY KEY', 'UNIQUE')
 46:    GROUP BY
 47:        T01.table_name
 48:       ,T01.constraint_name
 49:    HAVING
 50:       bool_and(T03.is_nullable = 'NO')
 51: )
 52: -- NOT NULL from is_nullable
 53: SELECT
 54:     'constraint'                            AS check_type
 55:    ,'staging'                               AS schema_name
 56:    ,T01.table_name
 57:    ,T02.pk_column[1]                        AS id1_column
 58:    ,T02.pk_column[2]                        AS id2_column
 59:    ,T02.pk_column[3]                        AS id3_column
 60:    ,T01.column_name                         AS check_column
 61:    ,format('%I IS NULL', T01.column_name)   AS where_clause
 62:    ,1                                       AS max_occurrence
 63:    ,'E'                                     AS severity
 64:    ,format('%s is mandatory in the target schema', T01.column_name) AS message
 65: FROM
 66:    information_schema.columns T01
 67:    INNER JOIN CTE_pk_column T02
 68:    ON
 69:      T02.table_name = T01.table_name
 70: WHERE
 71:        T01.table_schema = 'core'
 72:    AND T01.is_nullable  = 'NO'
 73: UNION ALL
 74: -- uniqueness from PRIMARY KEY / UNIQUE
 75: SELECT
 76:     'unique'
 77:    ,'staging'
 78:    ,T01.table_name
 79:    ,T02.pk_column[1]
 80:    ,T02.pk_column[2]
 81:    ,T02.pk_column[3]
 82:    ,array_to_string(T01.key_column, ',')
 83:    ,NULL
 84:    ,1
 85:    ,'E'
 86:    ,format('Key (%s) not unique'
 87:           ,array_to_string(T01.key_column, ', '))
 88: FROM
 89:    CTE_key_column T01
 90:    INNER JOIN CTE_pk_column T02
 91:    ON
 92:      T02.table_name = T01.table_name;

The CTE on lines 3 through 22 pulls one more thing out of the metadata that the configuration table would otherwise demand by hand: the business key, through which the error table later maps a finding back to the source record. That, too, lives in the schema — it is the target table's primary key. As an ordered array it spreads across id1_column through id3_column, and the HAVING count(*) <= 3 on line 21 draws the line of the error table's contract: it carries no more than three mapping columns.

The projection becomes the rule set only through the deployment step: it reads the view exactly once and writes the result into a table of its own. That table is the signed contract. It records which derived rules were valid at deployment time, and the runner reads it exclusively:

  1: CREATE TABLE dq.deployed_rule
  2: (
  3:     check_type      text        NOT NULL
  4:    ,schema_name     text        NOT NULL
  5:    ,table_name      text        NOT NULL
  6:    ,id1_column      text
  7:    ,id2_column      text
  8:    ,id3_column      text
  9:    ,check_column    text        NOT NULL
 10:    ,where_clause    text
 11:    ,max_occurrence  int         NOT NULL DEFAULT 1
 12:    ,severity        char(1)     NOT NULL
 13:    ,message         text        NOT NULL
 14:    ,deployed_on     timestamptz NOT NULL DEFAULT now()
 15: );
 16: 
 17: -- deployment step: sign the contract. Runs at deployment time against
 18: -- the target system, not on every run of the runner.
 19: DELETE FROM dq.deployed_rule;
 20: 
 21: INSERT INTO dq.deployed_rule
 22: (
 23:     check_type
 24:    ,schema_name
 25:    ,table_name
 26:    ,id1_column
 27:    ,id2_column
 28:    ,id3_column
 29:    ,check_column
 30:    ,where_clause
 31:    ,max_occurrence
 32:    ,severity
 33:    ,message
 34: )
 35: SELECT
 36:     check_type
 37:    ,schema_name
 38:    ,table_name
 39:    ,id1_column
 40:    ,id2_column
 41:    ,id3_column
 42:    ,check_column
 43:    ,where_clause
 44:    ,max_occurrence
 45:    ,severity
 46:    ,message
 47: FROM
 48:    dq.derived_rule;

The table deliberately carries no active flag. The configuration table's is_active belongs exclusively to the manual rules, because a derived rule is neither edited nor deactivated. The deployed_on field documents, in passing, when the contract was signed.

The contract idea does not stop at the derived rules. The manual rules in the configuration table change only as part of a deployment as well — inserted, edited or toggled via is_active in a versioned way, not live. The reason is the same as with the schema: a rule list that can be rearranged by hand between two deployments no longer tells you, at a glance, what was in force during the last run. The execution log does record that, but the list is supposed to predict what runs, not merely document what ran. is_active is thus a versioned switch, not a live dial.

The check scope of a table is now — within the limits the projection itself draws — the union of both worlds: the deployed derived rules plus the active handwritten ones from the configuration table. Add a single user-defined rule as an example — an age limit that no constraint contains:

  1: INSERT INTO dq.check_rule
  2:     (check_type,   schema_name, table_name, id1_column,    check_column, where_clause, severity, message)
  3: VALUES
  4:     ('constraint', 'staging',   'customer', 'customer_id', 'age',        'age < 18',   'W', 'Customer under 18 - review with business');
  5: 
  6: SELECT
  7:     'derived'          AS origin
  8:    ,check_type
  9:    ,check_column
 10:    ,where_clause
 11:    ,severity
 12:    ,message
 13: FROM
 14:    dq.deployed_rule
 15: UNION ALL
 16: SELECT
 17:     'user-defined'
 18:    ,check_type
 19:    ,check_column
 20:    ,where_clause
 21:    ,severity
 22:    ,message
 23: FROM
 24:    dq.check_rule
 25: WHERE
 26:    active
 27: ORDER BY
 28:     origin
 29:    ,check_type
 30:    ,check_column;

origin	check_type	check_column	where_clause	severity
derived	constraint	`country_code`	`country_code IS NULL`	E
derived	constraint	`customer_id`	`customer_id IS NULL`	E
derived	constraint	`source_id`	`source_id IS NULL`	E
derived	constraint	`source_system`	`source_system IS NULL`	E
derived	unique	`customer_id`		E
derived	unique	`source_system`		E
user-defined	constraint	`age`	`age < 18`	W

Six out of this table's seven rules come from the schema. Exactly one was typed — the one that actually contains business knowledge. How large that share turns out does depend on the table, though: a target table without mandatory fields and without key constraints gives the projection simply nothing.

The projection as drift detector

What happens when the target schema changes after the deployment? To the running process, at first: nothing. The runner reads the deployed state, and that is exactly the point of the model. The change does not stay invisible, though, because the view keeps computing the live state — and the difference between the two can be queried mechanically:

  1: ALTER TABLE core.customer ALTER COLUMN email SET NOT NULL;
  2: ALTER TABLE core.customer DROP CONSTRAINT uq_customer_source;
  3: 
  4: SELECT
  5:     'added'       AS drift
  6:    ,check_type
  7:    ,check_column
  8:    ,message
  9: FROM
 10:    (
 11:       SELECT check_type, schema_name, table_name, id1_column, id2_column,
 12:              id3_column, check_column, where_clause, max_occurrence, severity, message
 13:       FROM
 14:          dq.derived_rule
 15:       EXCEPT
 16:       SELECT check_type, schema_name, table_name, id1_column, id2_column,
 17:              id3_column, check_column, where_clause, max_occurrence, severity, message
 18:       FROM
 19:          dq.deployed_rule
 20:    ) T01
 21: UNION ALL
 22: SELECT
 23:     'removed'     AS drift
 24:    ,check_type
 25:    ,check_column
 26:    ,message
 27: FROM
 28:    (
 29:       SELECT check_type, schema_name, table_name, id1_column, id2_column,
 30:              id3_column, check_column, where_clause, max_occurrence, severity, message
 31:       FROM
 32:          dq.deployed_rule
 33:       EXCEPT
 34:       SELECT check_type, schema_name, table_name, id1_column, id2_column,
 35:              id3_column, check_column, where_clause, max_occurrence, severity, message
 36:       FROM
 37:          dq.derived_rule
 38:    ) T02
 39: ORDER BY
 40:     drift
 41:    ,check_column;

drift	check_type	check_column	message
added	constraint	`email`	email is mandatory in the target schema
removed	unique	`source_system,source_id`	Key (source_system, source_id) not unique

The two directions of the diff tell different stories. An added row means: the target has become stricter. Without a redeploy, the pre-filter does not know the new guarantee, and the load would fail at the target constraint instead of at the pre-filter — the detector shows this before it happens. A removed row means: the target has become laxer. The pre-filter checks more strictly than necessary, which endangers no run but is worth a negotiation. In both cases the answer is the same: review the contract and sign it again deliberately, that is, run the deployment step once more. The diff works as a check in the deployment gate or as a warning before each run. It is a signal, never an automatic adoption.

Why read-only is not a convenience

For the deployed derived rules, a hard convention applies: no UPDATE, no deactivating, no deleting. That is why the table carries no is_active either — this flag belongs exclusively to the manual rules in the configuration table. What looks like a limitation is the core of the pattern, not its side effect.

If the derived rule were editable, it would be a copy with an expiry date. Any change to it would decouple it from its source, and from that moment on there would again be two truths: the one in the schema and the one in the rule. Worse still: a well-meant "let's switch this one check off for a bit" would bypass a guarantee that the target schema keeps enforcing regardless. The load would then fail at a constraint for which there apparently was no active rule. The check would no longer be the target's honest pre-filter but its own drifting opinion of it.

For the same reason, the criticality of derived rules follows the source semantics rather than an editor's judgement. A NOT NULL and a unique key are hard constraints at the target. Their derived rules therefore carry E as in error, because a violation would make the load fail there. That is not an opinion to be debated per rule but a property of the schema.

Whoever really wants to get rid of a derived rule has exactly one path: change the schema and redeploy. That sounds inconvenient, and it is intended. The discussion "does this column really have to be mandatory?" belongs at the table, not at a configuration row that merely mirrors the table.

The boundary you don't build

As soon as two rule sources exist, one question suggests itself: what happens when both check the same thing? If someone manually creates a uniqueness rule on customer_id that already exists as a derived one — do you need conflict detection, a precedence scheme, a merge?

The better answer is: you never let the situation arise. Instead of detecting and resolving conflicts between derived and user-defined rules, the user-defined territory is cut so that it never enters the derived one. Concretely: the keys the schema already secures are not offered again in the rule editor — no "check primary key" quick pick, no pre-filled uniqueness form for the constraint columns. Those checks already exist, automatically and non-negotiably. The free uniqueness type is there for the cases the schema does not know: a composite business key plus a validity date, or an "at most three times per region", in other words different column combinations and different cardinalities.

The conflict you don't build does not have to be resolved. There is no merge logic, no precedence table and no special case in the runner — not because the problem was solved elegantly, but because by construction it does not exist. That is a design decision, not an algorithm, and it is cheaper and more robust than any conflict detection you would have to write instead.

Duplicates among your own rules

Entirely without checks, though, the user-defined territory does not get by. Within the handwritten rules, the same check can accidentally be created twice, and then two identical findings for the same violation would sit in the error table. What gets blocked is therefore the exact content duplicate: same rule type, same columns, same condition.

What deliberately does not get blocked is worth noting. Several rules of the same type per column are explicitly allowed: two different value-range checks on the same column are two separate, legitimate checks with their own messages. A "one rule per column and type" constraint would forbid exactly the cases a freely configurable rule table is built for.

The second subtlety: the criticality is not part of a rule's identity. Two rules with an identical condition but different severity count as the same duplicate. Otherwise the same check could exist once as an error and once as a warning, and the same violation would be reported twice — once blocking, once not. Whoever wants to change a rule's severity edits the existing rule instead of placing a second one next to it.

What remains handwritten

After all the deriving, a remainder is left, and it is the most valuable part of the configuration. The schema only holds what the database can enforce. Everything else is business knowledge:

Cross-field conditions — a discount requires an active status, an end date lies after the start date.
Plausibility and temporal logic — an order date does not lie in the future, an age under 18 is possible but worth reviewing (the example rule from above).
Value lists owned by the business — allowed status values or product codes deliberately not cemented as constraints, because the business side maintains them.

The derived rules need no review time, because they cannot be wrong — they claim nothing the target schema does not enforce anyway. Every minute a review spends nodding off "customer_id is mandatory" is missing from the question of whether the age limit is right. The projection shifts the attention to where mistakes are actually possible: into the handwritten rules.

The honest limit

The projection is exactly as good as the schema it reads from. That is its strength and its limit at once, and the limit deserves the same clear look as the pattern itself.

Where mandatory fields exist only in the application, because the column allows NULL and only the input form enforces a value, information_schema sees nothing, and the projection yields nothing. The same goes for business keys that were never created as constraints, and for length bounds on text columns without a declared length. The projection must not silently paper over these gaps: it delivers exactly the guarantees the schema states, and not a single one more. If you want more derived, you have to make the schema more honest — which keys and constraints a target table should carry in the first place is covered by the article on Postgres table conventions.

This pattern deserves to be called by its name: it is a deficit of application development, and not a rare one. Many applications treat the database as mere storage. Mandatory fields, value ranges and keys are checked by application code, while the schema allows almost anything. As long as only the application itself writes, this goes unnoticed. But anyone who wants to load data into such an application past the input form, say during a migration or through an interface, stands there without any of the guarantees this article projects. The consequence is uncomfortable but clear: exactly the rules that an honest schema would make derivable have to be created and maintained manually as user-defined rules for safe loading — with the full drift risk this article started with.

On top of that comes a Postgres-specific subtlety: information_schema is portable but not complete. A CREATE UNIQUE INDEX without an accompanying constraint does not appear in table_constraints, and neither do partial unique indexes or the NULLS NOT DISTINCT behaviour of newer Postgres versions. If you use such constructs, project from the system catalogs pg_constraint and pg_index instead — with more detail, but without portability. A partial unique index translated into a full uniqueness rule would simply be wrong, so its index predicate belongs in the rule, or the index deliberately stays out of the projection.

Deriving does not replace thinking about data quality. It moves the thinking to where it belongs: into the schema.

FAQ

Which data quality rules can be derived from the schema?
Everything the target table declares as a constraint or type: mandatory-field checks from is_nullable, uniqueness checks from PRIMARY KEY and UNIQUE (provided the key columns are NOT NULL), reference checks from foreign keys, plus length and value-range bounds from character_maximum_length and numeric_precision. Not derivable is business logic without a schema trace — cross-field conditions, temporal plausibility, value lists owned by the business.

Why should derived data quality rules not be editable?
Because an editable derived rule would be a copy with an expiry date. After the first change there would be two truths: the one in the schema and the one in the rule. A switched-off check would additionally hide a guarantee the target still enforces. That is why the deployed state deliberately carries no is_active — the flag belongs to the manual rules. Whoever wants to change a derived rule changes the schema and redeploys.

Is information_schema enough, or do I need my own metadata model?
For NOT NULL, keys and type bounds, information_schema is enough, and it is portable — the same views exist in SQL Server. The Postgres system catalogs (pg_constraint, pg_index) additionally see unique indexes without constraints and partial indexes. A metadata model of your own pays off only once business metadata should join in that the database does not know — say, business keys without a database constraint or a per-column null-handling semantic.

What if schema and business requirement contradict each other?
Then one of the two sides is wrong, and that belongs resolved at the schema rather than overridden in the configuration. If the business says "email is mandatory" while the column allows NULL, the schema is too lax: add the constraint, and the derived rule follows automatically. Until then, a user-defined rule with severity warning can make the gap visible without faking a hard guarantee.

How do I keep derived and user-defined rules apart?
Through the origin of the data itself: derived rules live only in the deployment table, which only the deployment step fills, user-defined ones only in the configuration table. A combined view with an origin column (via UNION ALL) shows the complete check scope. Only one rule of discipline matters: never copy projected rules into the configuration table by hand — there they would be editable copies, and the drift starts all over again.

Framework and routines:

Data Quality Checks with SQL — the sub-hub: the configurable framework with error table, rule configuration and runner, whose configuration this article fills automatically.
Validating Data with SQL — the routine behind the projected WHERE rules: value ranges, mandatory fields and the NULL trap.
Finding Duplicates with SQL — the uniqueness routine: cardinality, composite keys and the NULL semantics of UNIQUE.
Finding Orphaned Records with SQL — the lookup routine that derived foreign-key rules build on.
Three Severity Levels, Not Pass/Fail — the severity spoke of the series: why E/W/I can do more than passed/failed, and how severity decides whether a rule becomes a constraint or stays in the pipeline.

Theory:

Data Quality: Dimensions and Error Classes — the concept frame: which dimensions the derived rules pay into.

Target schema:

Postgres Table Conventions — which keys and constraints a target table should carry so the projection has something to read.

Formatting SQL Statements (Part 2) — Statement Structure: SELECT, WHERE, FROM, JOIN

Marcus — Mon, 31 Aug 2026 09:04:54 +0000

If you can't tell at a glance where the WHERE clause of a 200-line SELECT statement starts and ends, you have a structure problem — not a content problem. This article shows how to format SQL statements so that SELECT, FROM, WHERE and JOIN stay immediately recognizable even in long queries.

→ Part of a series. This is part 2 and covers statement structure (SELECT, WHERE, FROM, JOIN). The basics for identifiers, delimiters, commas, and aliases are in Part 1 — Identifiers, Delimiters, Commas, Aliases.

TL;DR — what this article delivers:

Main elements (SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY) belong on separate lines with consistent indentation.
WHERE clause — align operands and operators column-style, indent equivalent conditions equally. The parenthesis structure becomes visually readable this way.
FROM clause — table directly after the JOIN operator, ON keyword on its own line, JOIN conditions formatted like a mini WHERE clause.
Postgres bridge + auto-formatters at the end — [brackets] vs. "quotes", DISTINCT ON, LATERAL. sqlfluff and pgFormatter complement manual discipline, they don't replace it.

Prerequisite: SSMS or any SQL editor with configurable tab width is enough. A live AdventureWorks database is not required — the examples illustrate patterns, not runnable pipelines.

General Thoughts on Indentation

A useful analogy for SQL structure is the outline of a table of contents. The indented version below is much faster to scan than the flat variant further down:

1. Chapter Level 1
   1.1. Chapter Level 2
      1.1.1. Chapter Level 3
      1.1.2. Chapter Level 3
   1.2. Chapter Level 2
2. Chapter Level 1
   2.1. Chapter Level 2
      2.1.1. Chapter Level 3

For comparison, the same TOC flat:

1. Chapter Level 1
1.1. Chapter Level 2
1.1.1. Chapter Level 3
1.1.2. Chapter Level 3
1.2. Chapter Level 2
2. Chapter Level 1
2.1. Chapter Level 2
2.1.1. Chapter Level 3

Flat TOCs work too, but only with additional formatting options like upper/lower case, bold, or italic to distinguish levels. In a SQL editor, those options are typically not available (SSMS renders plain text with syntax highlighting, no bold for identifiers). So indentation remains the structural tool.

One thing up front: All layout rules in this article are formatting conventions. SQL enforces neither the line breaks nor the indentation or the position of commas and keywords. The conventions aim to make the structure of a statement visible before you read it.

Main Language Elements

This article looks at the six central clauses of a SELECT statement:

SELECT
FROM
WHERE
GROUP BY
HAVING
ORDER BY

A complete SELECT statement can contain further clauses (WITH, TOP or LIMIT, OFFSET/FETCH, window definitions) — the same indentation principles apply to them. Treating the six clauses as the first level, their contents sit one indentation level deeper. The resulting basic structure of a SQL statement looks like this:

  1: SELECT
  2:    field list
  3: FROM
  4:    data sources
  5: WHERE
  6:    conditions on data source
  7: GROUP BY
  8:    grouping fields
  9: HAVING
 10:    conditions on aggregations
 11: ORDER BY
 12:    sort fields

The ground rule of this style guide: The main elements of a SQL statement sit on separate lines.

As counterexamples, here are two commonly seen formatting styles that ignore this rule. In both, you have to read at least parts of the statement to recognize where a main element starts and ends.

Left-Aligned Main and Sub-Elements

You occasionally find both top-level elements and the next-level elements left-aligned in the same column. You see this especially often in the FROM clause: Data sources (tables, views, CTEs) are indented the same as the introducing keyword FROM:

  1: SELECT
  2: field list
  3: FROM
  4: table1
  5: JOIN table2 ON [...]
  6: JOIN table3 ON [...]
  7: WHERE
  8: conditions on data source
  9: GROUP BY
 10: grouping fields
 11: HAVING
 12: conditions on aggregations
 13: ORDER BY
 14: sort fields

Right-Aligned Keywords

In this example, the main clauses of the SELECT statement (without regard for the BY keyword) are right-aligned. This indentation style makes alignment extra work because you have to deal with varying indent widths:

  1: SELECT field1, field2, field3
  2:   FROM table1
  3:   LEFT JOIN table2 ON [...]
  4:   LEFT JOIN table3 ON [...]
  5:  WHERE condition1
  6:     OR condition2
  7:     OR condition3
  8:  GROUP BY grouping fields
  9: HAVING condition1
 10:     OR condition2
 11:     OR condition3
 12:  ORDER BY sort fields

For simple statements, however, right-aligned keywords can read quite well — CREATE INDEX is a typical example:

  1: CREATE UNIQUE NONCLUSTERED
  2:  INDEX [IndexName]
  3:     ON [dbo].[FactInternetSalesReason]([SalesOrderNumber] ASC);

For full SELECT statements with multiple JOINs, group-by columns and complex conditions, this style breaks down.

SELECT Field List

The natural reading direction of a SQL statement is left to right and top to bottom. With keyboard and mouse, vertical navigation is easier than horizontal navigation. The scroll wheel and the Page Up/Page Down keys make fast vertical navigation possible even within long complex statements — provided that field lists are written vertically. The more important effect is editor-independent: One line per expression makes long field lists scannable and easier to compare, move, or extend.

Field names should be written as a vertical list with leading commas — the detailed reasoning (comma readability, box-selection pattern) is in Part 1, section “The Comma”. One field per line. Because the field list is logically subordinate to the SELECT keyword, field names are indented by the agreed indent width:

  1: SELECT
  2:     field1
  3:    ,field2
  4:    ,field3
  5: FROM [...]
  6: WHERE [...]
  7: GROUP BY
  8:     field1
  9:    ,field2
 10:    ,field3
 11: HAVING [...]
 12: ORDER BY
 13:     field1
 14:    ,field2
 15:    ,field3

WHERE Clause

The order here deliberately departs from SQL syntax: The WHERE clause introduces the central condition pattern first, which is then reused for the FROM and HAVING clauses. A WHERE clause contains one or more conditions (predicates) connected by logical operators. Two points deserve attention when formatting these conditions:

alignment of operands
indentation of equivalent conditions

Alignment of Operands

A simple condition consists of two operands and an operator (=, !=, <>, IN, NOT IN, etc.). In a compound expression built from multiple single conditions, operands and operators should be aligned column-style. In the following example, field names have different lengths and different operators are applied:

  1: [...]
  2: WHERE
  3:        T01.[field___1]    =  'something'
  4:    AND T01.[field__2]     <> 1
  5:    AND T01.[field_____3]  NOT IN (1, 2, 3)
  6:    AND T01.[field4]       = T02.[field5]

This produces a table-like notation that allows fast visual navigation within the condition components.

Indentation of Equivalent Conditions

If the WHERE clause contains more than one condition, they are connected by logical operators like AND or OR. Parentheses are required when the intended logical grouping differs from operator precedence (AND binds more tightly than OR). Beyond that, they can improve readability in complex expressions. Depending on complexity, you quickly end up with deeply nested structures.

To keep complex nested expressions readable, give particular attention to structure and indentation of the WHERE clause: equivalent conditions are written underneath each other with the same indent, and a logical connection of equivalent conditions gets an indent that matches the parenthesis hierarchy:

  1: [...]
  2: WHERE    (
  3:           (
  4:                 [operand01] = [operand02]
  5:              OR [operand03] = [operand05]
  6:              OR [operand05] = [operand06]
  7:           )
  8:       AND (
  9:                 [operand07] = [operand08]
 10:              OR [operand09] = [operand10]
 11:           )
 12:       AND [operand11] = [operand12]
 13:    )
 14: OR (
 15:       [operand13] = [operand14]
 16:    )

The logical connections become visually readable through this indentation. A screenshot of the same WHERE clause in Notepad++ makes the effect even more obvious, because the editor's vertical guide lines at the tab stops emphasize the parenthesis hierarchy:

FROM Clause

As with the other main elements, the subordinate elements of the FROM clause are written indented. In most cases these are data sources — tables, views, and Common Table Expressions (CTEs).

Sub-SELECTs in the FROM clause can be replaced by CTEs when the subquery forms a self-contained logical step, is needed more than once, or makes the statement hard to follow: A CTE structures it top-down. For small subqueries used only once, a derived table is often the more compact choice. Converting is a structuring decision — how the CTE is executed is up to the engine's optimizer. What Postgres does differently (MATERIALIZED / NOT MATERIALIZED) is covered in the FAQ at the end of the article.

For formatting a JOIN clause there are four building blocks:

table (or view / CTE)
JOIN operator
ON keyword
JOIN conditions

In the following code example, there is no visual anchor to identify the elements of the FROM clause:

  1: FROM
  2: table1 T01
  3: JOIN table2 T02
  4: ON T01.[FK] = T02.[ID]
  5: JOIN table3 T03 ON
  6: T02.[FK] = T03.[ID]
  7: JOIN table4 T04
  8: ON T03.[FK] = T04.[ID]
  9: WHERE
 10: [...]

The elements of the FROM clause belong on separate lines for readability. Exception: the joined table sits directly after the JOIN operator. In this style guide, the ON keyword sits left-aligned with the JOIN operator on the line below — that keeps JOIN and join condition visible as two separate structural levels. The same rules as for the WHERE clause apply to JOIN conditions:

  1: SELECT
  2:    [...]
  3: FROM
  4:    table1 T01
  5:    INNER JOIN table2 T02
  6:    ON
  7:      T01.[FK] = T02.[ID]
  8:    INNER JOIN table3 T03
  9:    ON
 10:          T02.[FK1]   = T03.[FK1]
 11:      AND T02.[field2] = T03.[field2]
 12:    INNER JOIN table4 T04
 13:    ON
 14:      T03.[FK] = T04.[ID]
 15: WHERE
 16:   [...]

GROUP BY, HAVING, ORDER BY

The remaining main elements follow the same principles as the SELECT field list and the WHERE clause — in short:

GROUP BY contains a list of grouping expressions — for simple queries formatted like a scaled-down SELECT: one expression per line with a leading comma, indented by convention.
HAVING is a condition list like WHERE — operands aligned column-style, equivalent conditions indented equally. The difference is semantic (post-GROUP BY filter), not typographic.
ORDER BY contains a list of sort expressions — formatted analogous to the GROUP BY clause, with optional ASC / DESC per expression (held in a separate column when both sort directions appear in the same statement).

  1: SELECT
  2:     T01.[Region]
  3:    ,T01.[Year]
  4:    ,SUM(T01.[Sales])   AS [Total]
  5: FROM
  6:     [dbo].[FactSales] T01
  7: GROUP BY
  8:     T01.[Region]
  9:    ,T01.[Year]
 10: HAVING
 11:        SUM(T01.[Sales]) >= 1000
 12:    AND COUNT(*)         >= 10
 13: ORDER BY
 14:     T01.[Region] ASC
 15:    ,T01.[Year]   DESC

Auto-Formatters and “Formatting Is Learning”

Tools like sqlfluff (multi-dialect — T-SQL, Postgres, MySQL, BigQuery, …) and pgFormatter (Postgres-focused) generate the layouts shown here automatically and are useful as a pre-commit hook or CI step. They are not a replacement for manual formatting.

The act of indenting, aligning aliases and placing parentheses forces the writer to read the statement fully and mentally model the table relationships. Auto-formatters produce the layout — they don't produce the mental model that emerges while writing. In the age of Copilot and Cursor, this matters twice: Generated SQL without understanding is a risk, because it produces technically correct queries that still don't answer the business question.

Pragmatic recommendation: format manually first, then let the formatter run as a final consistency pass (e.g. sqlfluff fix).

Postgres Bridge

The examples in this article use T-SQL notation ([brackets], positional T01 aliases). The layout rules themselves are engine-neutral — they apply 1:1 to Postgres as well. There are only a handful of places where Postgres behaves differently, and none of them change the format pattern:

Identifier quoting: The SQL standard uses double quotes for delimited identifiers ("name") — Postgres follows it. T-SQL typically uses [brackets] but also understands double quotes when QUOTED_IDENTIFIER is active. With case-sensitive identifiers, quoting becomes semantically relevant in Postgres (a separate article on case sensitivity in SQL Server vs. Postgres is planned).
DISTINCT ON: the Postgres idiom for “first row per group”. Which row comes first is determined only by an ORDER BY that fixes the order within each group unambiguously — its leftmost expressions must be the DISTINCT ON expressions. In T-SQL, emulate this with ROW_NUMBER() OVER (PARTITION BY … ORDER BY …) and a filter on rn = 1. The formatting follows the SELECT field list.
LATERAL: lets a subquery in the FROM clause reference columns of preceding FROM items. T-SQL uses CROSS APPLY / OUTER APPLY for this. A LATERAL join is formatted like any other JOIN.
RETURNING: Postgres returns the affected rows on INSERT/UPDATE/DELETE and, since version 17, on MERGE as well. SQL Server offers OUTPUT, a functionally related clause with its own syntax and forms (OUTPUT INTO). Both are formatted as their own clause line with a field list.
CTEs: WITH … is standardized and structured essentially the same in both engines, the extensions differ in detail. Postgres has the options MATERIALIZED / NOT MATERIALIZED (see FAQ).
JOIN indentation, WHERE parenthesis pattern, ORDER BY lists: There is no engine difference for these layout patterns.

So the format discipline carries on both engines. A separate follow-up article on the Postgres identifier specifics is planned.

Conclusion

The main elements of a SELECT statement belong on separate lines, their subordinate elements one indentation level deeper. Every pattern in this article derives from that ground rule: the field list as a vertical list with leading commas, the WHERE clause with column-aligned operands and equally indented equivalent conditions, the FROM clause with the table directly after the JOIN operator and the ON keyword on its own line.

Applied consistently, these patterns make the structure of a statement visible before its content is read: The parenthesis hierarchy of a nested WHERE clause and the relationships between the tables are already in the layout. The pattern is engine-neutral and carries in SQL Server and Postgres alike. The vocabulary level (identifiers, delimiters, commas, aliases) is covered in Part 1. More important than any single rule is that the same hierarchy stays consistently visible across the whole project — consistency beats purity.

FAQ

How do you format complex SQL queries with multiple JOINs and nested WHERE conditions?

Following the rules in this article: each clause on its own line, subordinate elements one level deeper. Each JOIN gets its table right next to the operator and its ON on the following line. In the WHERE clause, equivalent conditions sit underneath each other and the parenthesis hierarchy sets the indent depth. That keeps even a statement with five joins navigable.

When should a sub-SELECT be replaced by a CTE?

As soon as the subquery makes the statement hard to follow, is needed more than once, or forms a self-contained logical step. CTEs read top-down instead of nested and can be referenced multiple times in the same query. SQL Server, however, may execute the defining query again for each reference, because a CTE is not a materialized intermediate result there. For testing, temporarily replace the outer query with a SELECT * FROM cte_name — a CTE definition cannot be run standalone. Small subqueries used only once may stay.

Should the JOIN operator always be explicit (INNER, LEFT, RIGHT, FULL)?

Syntactically, the INNER is optional — a bare JOIN means INNER JOIN. This style guide still always writes out the JOIN type (INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN): the default is not obvious to every reader, and when skimming, the difference between JOIN and LEFT JOIN is easy to miss.

Does the format pattern apply to Postgres as well?

Yes — the layout rules transfer unchanged, see the Postgres Bridge section above. The only practical adjustment is identifier quoting ("…" instead of […]). The layout rules for SELECT field lists, WHERE clauses, FROM clauses and JOIN indentation are engine-neutral.

Is an auto-formatter like sqlfluff or pgFormatter enough — or do you still need to format manually?

Auto-formatters deliver layout, but not the learning effect. Anyone who only pipes a 200-line statement through a formatter hasn't read the statement. Anyone who structures it manually builds the mental models of table relationships — and often spots logical errors along the way. Pragmatic workflow: manual first, then the formatter as a finishing pass for consistency.

What about MATERIALIZED / NOT MATERIALIZED for CTEs in Postgres?

Since version 12, Postgres folds a non-recursive, side-effect-free CTE into the parent query by default when it is referenced exactly once — predicate pushdown is then possible. With multiple references, Postgres does not fold such a CTE by default and treats it as a separate computation instead. MATERIALIZED forces this separate computation, NOT MATERIALIZED allows folding even with multiple references. For performance-critical queries with expensive CTEs, the Postgres docs on WITH queries are worth a read. SQL Server has no corresponding syntax: a CTE there is not an independently materialized object, it is optimized as part of the overall statement.

Where do I find Part 1 (identifiers, delimiters, commas, aliases)?

Part 1 — Formatting SQL Statements. It covers the smaller building blocks: regular vs. delimited identifiers, leading vs. trailing comma, systematic T01/T02 aliases, qualified column names. Part 1 + Part 2 together form a complete style guide for SELECT statements.

Part of the series:

Formatting SQL Statements (Part 1) — Identifiers, Delimiters, Commas, Aliases — the vocabulary layer: regular vs. delimited identifiers, comma position, systematic aliases.
Structuring and Formatting SQL Statements — The Cluster Path — the hub article with the reading path through all formatting topics on this blog.

Upstream:

The Functional Aesthetics of SQL — the “why” of SQL formatting: Structured code is faster to read, review, and change.

Downstream:

Inline Comments in Complex SQL Statements — the meta layer: inline and block comments that document the structure shown here without destroying it.
Editor Options in SSMS — the editor basis for this layout: tab width, spaces instead of tabs, block selection.
SQL Conventions // PL/pgSQL Procedures You Can Still Read in Two Years — the continuation of the statement layout for Postgres procedures.

Formatting SQL Statements (Part 1) — Identifiers, Delimiters, Commas, Aliases

Marcus — Thu, 27 Aug 2026 12:10:05 +0000

Anyone who has ever had to debug a badly or completely unformatted SELECT with 30 columns and half a dozen joins knows the feeling: it isn't the SQL that eats up your day — it's hunting down what the statement is actually trying to do. SQL formatting isn't a matter of taste; it's a maintenance tool — and it starts with a naming convention.

→ Part of a series. This is part 1 and covers identifiers, delimiters, commas, and aliases. For the layout of longer statements, continue with Part 2 — Structure and Formatting.

What this article covers:

Identifiers and delimiters — when to use square brackets, when quotation marks, and what counts as a regular identifier
Comma, semicolon, spacer — the small separators with a big impact on readability
Table aliases and qualified column names — why systematic T01-style aliases scale better in wide statements than mnemonic abbreviations
Box selection as the killer argument for leading commas

SSMS serves as the example editor. The principles apply equally to DataGrip, VS Code, and DBeaver. The SQL examples reference AdventureWorksDW2017.

Why a Naming Convention at All?

Capital letters are an established device in most written languages to emphasise individual words. In programming, that translates into notations such as CamelCase (each compound word starts with a capital), camelCase (the same, except the first word is lower case), or snake_case (everything lower case with underscores between words). Conventions like these are bundled into a naming convention — a deliberate decision about how identifiers, function names, and data types should be written. In practice, however, you'll often find that developers don't even hold to their own preferred convention.

Sticking to a naming convention improves readability of any text, and of code in particular. tHE SAME Sentence Written Once Again With DEVIATIONS from the generally KNOWN convention that nouns are CAPITALISED and verbs and adjectives are NOT — and the text Turns Unreadable: "adherence to a NAMING convention IMPROVES The readability of text In General And of code In Particular."

The following statement comes from the view vTimeSeries in the database AdventureWorksDW2017, slightly reworked. It does not follow any naming convention:

  1: SELECT 
  2: case [Model] 
  3: WHEN 'Mountain-100' THEN 'M200' when 'Road-150' THEN 'R250' when 'Road-650' THEN 'R750' 
  4: WHEN 'Touring-1000' THEN 'T1000' ELSE LEFT(Model, 1) + Right([Model], 3) 
  5: END + ' ' + [Region] AS [ModelRegion] ,(convert(Integer, CalendarYear) * 100) + CONVERT(int, Month) AS [TimeIndex] 
  6: ,SUM(Quantity) AS [Quantity] 
  7: ,sum(Amount) AS Amount, calendaryear,[Month]
  8: ,[dbo].[udfbuildiso8601date] ([CALENDARYEAR], [Month], 25)
  9: as reportingdate
 10: FROM [dbo].[vDMPrep] 
 11: where [Model] IN ('Mountain-100', 'Mountain-200', 'Road-150', 'Road-250', 
 12: 'Road-650', 'Road-750', 'Touring-1000') 
 13: GROUP BY CASE [Model] 
 14: WHEN 'Mountain-100' THEN 'M200' WHEN 'Road-150' THEN 'R250' WHEN 'Road-650' THEN 'R750' 
 15: WHEN 'Touring-1000' THEN 'T1000' ELSE Left(Model,1) + Right(Model,3) 
 16: end+' '+[Region] ,(Convert(Integer, [CalendarYear]) * 100) + Convert(Integer, [Month])
 17: ,CalendarYear,[Month],[dbo].[udfBuildISO8601Date] ([CalendarYear], [Month], 25);

In the version below, function names are upper case, every field name uses delimiters, and data types are lower case. Every element is laid out consistently:

  1: SELECT 
  2:     CASE [Model] 
  3:        WHEN 'Mountain-100' THEN 'M200' 
  4:        WHEN 'Road-150'     THEN 'R250' 
  5:        WHEN 'Road-650'     THEN 'R750' 
  6:        WHEN 'Touring-1000' THEN 'T1000' 
  7:        ELSE LEFT([Model], 1) + RIGHT([Model], 3) 
  8:     END + ' ' + [Region] AS [ModelRegion] 
  9:    ,(CONVERT(int, [CalendarYear]) * 100) + CONVERT(int, [Month]) AS [TimeIndex] 
 10:    ,SUM([Quantity])  AS [Quantity] 
 11:    ,SUM([Amount])    AS [Amount]
 12:    ,[CalendarYear]
 13:    ,[Month]
 14:    ,[dbo].[udfBuildISO8601Date]([CalendarYear], [Month], 25) AS [ReportingDate]
 15: FROM 
 16:    [dbo].[vDMPrep] 
 17: WHERE 
 18:    [Model] IN ('Mountain-100', 'Mountain-200', 'Road-150', 'Road-250', 'Road-650', 'Road-750', 'Touring-1000') 
 19: GROUP BY 
 20:     CASE [Model] 
 21:        WHEN 'Mountain-100' THEN 'M200' 
 22:        WHEN 'Road-150'     THEN 'R250' 
 23:        WHEN 'Road-650'     THEN 'R750' 
 24:        WHEN 'Touring-1000' THEN 'T1000' 
 25:        ELSE LEFT([Model], 1) + RIGHT([Model], 3) 
 26:     END + ' ' + [Region]
 27:    ,(CONVERT(int, [CalendarYear]) * 100) + CONVERT(int, [Month])
 28:    ,[CalendarYear]
 29:    ,[Month]
 30:    ,[dbo].[udfBuildISO8601Date]([CalendarYear], [Month], 25);

Whether you like the formatting or not is a personal call. Either way, the second statement is, at a glance, tidier and easier to grasp.

The sections that follow walk through the most important parts of a naming convention. The list is not exhaustive; treat it as a starting point. One thing is worth saying up front: some of it is technical fact (how QUOTED_IDENTIFIER behaves, say, or how Postgres folds case), but most of it is reasoned convention — a choice a team can just as well make differently, as long as it makes that choice consistently. A second part picks up with best practices for the structure of SQL statements.

Regular Identifiers

Every database object has a name — its identifier. Each database vendor defines rules for what a valid identifier looks like. In SQL Server, identifiers are typically capped at 128 characters and must not contain spaces. The exact definition of a regular identifier is in the online documentation:

learn.microsoft.com/en-us/sql/relational-databases/databases/database-identifiers

For practical purposes, though, that definition is framed too generously. A convention worth having draws tighter boundaries than the database system demands.

Before settling on a spelling style, take a look at how your database actually treats identifiers. SQL Server with a case-insensitive collation does not distinguish between upper and lower case: FactInternetSales and factinternetsales point to the same table. That is the usual situation, since the installation defaults are case-insensitive, but the collation is in fact selectable per instance, per database, and even per column. Case-insensitivity gives you the freedom to choose an emphatic notation such as CamelCase without breaking your statement. Postgres, on the other hand, folds unquoted identifiers to lower case: FactInternetSales becomes factinternetsales internally. You can still write CamelCase without quotes, but it loses its capitalisation along the way. The only way to keep it is "double quotes", which becomes a burden on every statement you write. That is why snake_case is the common convention in the Postgres world.

Behind this blog are ten years of SQL Server practice with CamelCase as a settled convention. Since the switch to Postgres three years ago, snake_case has taken over there. The full depth of case-sensitivity differences between the two engines deserves its own article, and one is on the way. The examples here stay with the CamelCase notation that is typical for SQL Server and comes with AdventureWorksDW2017 anyway.

The underscore _ is a widely used word separator and a legal part of a regular identifier. In the SQL Server world it can nonetheless be avoided for the most part: CamelCase serves the same purpose and keeps identifiers more compact.

Special characters such as @, #, and $ are permitted by the definition, but under this blog's convention they have no place in an identifier. Technically they are allowed even without delimiters inside a regular identifier; they only carry a special meaning in leading position, where @ marks a variable and # a temporary table. That double role is precisely the problem. Using the characters in column or table identifiers creates visual noise and invites confusion with those special meanings:

  1: SELECT 
  2:     [EnglishDayNameOfWeek] AS [English@DayNameOfWeek]
  3:    ,[SpanishDayNameOfWeek] AS [Spanish#DayNameOfWeek]
  4:    ,[FrenchDayNameOfWeek]  AS [French$DayNameOfWeek]
  5: FROM 
  6:    [dbo].[DimDate];

By the definition, regular identifiers may even contain Unicode letters from any language. This blog's recommendation nonetheless limits the choice to the letters of the Latin alphabet [a-zA-Z] plus the digits [0-9] where digits are unavoidable. That reduces the rules for a good identifier to just two:

Letters of the Latin alphabet only
Digits as a fallback

A disciplined approach to regular identifiers always goes hand in hand with the development of a naming convention for the objects themselves.

Delimiters

Once an identifier contains a space, it stops being a regular identifier. Using non-regular identifiers is bad style. They are nonetheless permitted as long as they are wrapped either in quotation marks or in square brackets.

Double quotation marks are the form the SQL standard provides for delimiting identifiers. Microsoft additionally allows square brackets, diverging from the standard. This blog prefers the brackets as the proprietary form of delimiter.

Delimiters set identifiers apart clearly from the other language elements of a SQL statement and so contribute a great deal to readability. This blog's convention therefore applies delimiters throughout, regardless of whether an identifier is regular or not:

Schemas
Tables
Views
Column names
Aliases
All programmable objects (functions, stored procedures etc.)

This is a deliberate house convention, not a general best practice. For regular identifiers the delimiters are optional according to the Microsoft documentation, and plenty of teams quote only where they have to. Whoever opts for quoting throughout gains the visual separation, at the price of more characters per identifier.

The statement below is identical to the second one in the overview, but written without any delimiters:

  1: SELECT 
  2:     CASE Model 
  3:        WHEN 'Mountain-100' THEN 'M200' 
  4:        WHEN 'Road-150'     THEN 'R250' 
  5:        WHEN 'Road-650'     THEN 'R750' 
  6:        WHEN 'Touring-1000' THEN 'T1000' 
  7:        ELSE LEFT(Model, 1) + RIGHT(Model, 3) 
  8:     END + ' ' + Region AS ModelRegion 
  9:    ,(CONVERT(int, CalendarYear) * 100) + CONVERT(int, Month) AS TimeIndex 
 10:    ,SUM(Quantity)  AS Quantity 
 11:    ,SUM(Amount)    AS Amount
 12:    ,CalendarYear
 13:    ,Month
 14:    ,dbo.udfBuildISO8601Date(CalendarYear, Month, 25) AS ReportingDate
 15: FROM 
 16:    dbo.vDMPrep 
 17: WHERE 
 18:    Model IN ('Mountain-100', 'Mountain-200', 'Road-150', 'Road-250', 'Road-650', 'Road-750', 'Touring-1000') 
 19: GROUP BY 
 20:     CASE Model 
 21:        WHEN 'Mountain-100' THEN 'M200' 
 22:        WHEN 'Road-150'     THEN 'R250' 
 23:        WHEN 'Road-650'     THEN 'R750' 
 24:        WHEN 'Touring-1000' THEN 'T1000' 
 25:        ELSE LEFT(Model, 1) + RIGHT(Model, 3) 
 26:     END + ' ' + Region
 27:    ,(CONVERT(int, CalendarYear) * 100) + CONVERT(int, Month)
 28:    ,CalendarYear
 29:    ,Month
 30:    ,dbo.udfBuildISO8601Date(CalendarYear, Month, 25);

Plenty of auto-generated scripts from Microsoft tooling use square brackets as delimiters. That is not a documented recommendation, but it is an observable pattern: SQL Server Management Studio (SSMS) emits brackets in the SELECT and DDL statements generated through the context menu on a table. Microsoft is not entirely consistent here, though: when you create a view through the wizard, or look at the SQL panel of the Edit feature, delimiters are dropped wherever they can be. The two screenshots below — both showing auto-generated SQL — make a fine case study in unmaintainable code:

View

Edit Feature

One restriction applies to quotation marks as delimiters: they only work when the SQL Server setting QUOTED_IDENTIFIER is set to ON (the default in SSMS and in the common client libraries):

SET QUOTED_IDENTIFIER ON;

With the setting OFF, SQL Server reads double quotes as string literals, and non-regular identifiers can only be written with square brackets. The brackets work regardless of this setting — a practical advantage of the proprietary form. ON is more than a matter of style, too: several SQL Server features require it, among them indexed views, indexes on computed columns, and filtered indexes.

Details in the online documentation:

learn.microsoft.com/en-us/sql/t-sql/statements/set-quoted-identifier-transact-sql

The Spacer

What is meant here is the vertical spacer, that is, the blank line. The natural reading direction of a SQL statement is left to right and top to bottom.

While general-purpose code consists of many short statements, SQL is designed to do a lot of work in a single statement. An SQL statement can easily span a hundred lines or more. That makes writing a good SQL statement a particular challenge. A key criterion for grasping the structure and intent of a statement quickly isn't only a clear layout and consistent formatting, but also compactness. On a typical monitor at a sensible resolution, SSMS shows around forty lines of SQL when only a query window is open and no result pane. In day-to-day use, twenty-five to thirty lines is the realistic maximum.

There are colleagues in our trade who insert a blank line after every line

of code. Excessive blank lines force the reader to lean on the navigation

keys or the mouse wheel just to get from one end of the statement to the

other. Worse, the overall context of the statement becomes much harder

to take in.

Blank lines can be a useful stylistic device to separate logical blocks. Overusing them, however, makes the statement harder to read.

The Semicolon

The SQL standard provides for the semicolon as the terminator of a statement. SQL Server, at least, is forgiving here and doesn't force you to use one. There are only a handful of cases where the semicolon is mandatory.

One example: when using a Common Table Expression, the statement preceding the keyword WITH must end with a semicolon. To sidestep the problem, many developers write the leading WITH as ;WITH. Formally, the semicolon belongs to the preceding statement — ;WITH is a compatibility idiom, not a syntax of its own.

Other engines are stricter, or more precisely their tooling and procedural languages are. A single statement handed to Postgres directly by a driver works without a trailing semicolon. In psql and in multi-statement scripts such as pg_dump output, the semicolon marks the end of a statement, and inside PL/pgSQL function bodies it is part of the syntax as a statement terminator. Oracle follows the same pattern: in PL/SQL blocks (BEGIN … END;) the semicolon is required, while in clients such as SQL*Plus its role depends on the execution context.

Using the semicolon consistently is a sign of care either way, and it helps readability. What it does not do is make a statement portable — SQL dialects differ in entirely different places, from data types through functions to the procedural language. What it does remove is an unnecessary syntactic dependency on the client and the execution context.

The Comma

Field lists in SQL are separated by commas. Some developers put the comma before the field name, others put it after. The pro/con discussion usually centres on how easy it is to add or remove a field. The real reason the comma belongs at the front, though, is readability and the option to format the statement with box selection (called "column editor" or "Spaltenauswahl" in SSMS, "Box Selection" in VS Code and DataGrip). There is more on box selection in the article The Functional Aesthetics of SQL.

Leading Comma

  1: SELECT 
  2:     [EnglishDayNameOfWeek]
  3:    ,[SpanishDayNameOfWeek]
  4: FROM 
  5:    [dbo].[DimDate];

If you want to add another field, say FrenchDayNameOfWeek, it's easier to add it at the end — you only insert the text ,[FrenchDayNameOfWeek] after [SpanishDayNameOfWeek]:

  1: SELECT 
  2:     [EnglishDayNameOfWeek]
  3:    ,[SpanishDayNameOfWeek]
  4:    ,[FrenchDayNameOfWeek]
  5: FROM 
  6:    [dbo].[DimDate];

If you want the new field at position one, you need two edits: insert the line [FrenchDayNameOfWeek] before [EnglishDayNameOfWeek], and prepend a comma to [EnglishDayNameOfWeek]:

  1: SELECT 
  2:     [FrenchDayNameOfWeek]
  3:    ,[EnglishDayNameOfWeek]
  4:    ,[SpanishDayNameOfWeek]
  5: FROM 
  6:    [dbo].[DimDate];

Trailing Comma

When the comma is written after each field, the situation reverses: it's easier to add a new field at position one than at the end.

Readability

The comma is a separator. It marks the transition from one field to the next. When it sits at the end of each field, it loses its separating character because field names have different lengths.

In the statement below, it isn't immediately obvious whether the identifier [ProductName] is a column name or has some other role. The reader has to look at the line above to realise that [ProductName] is the alias for [EnglishProductName]:

  1: SELECT 
  2:     [EnglishProductName] AS
  3:     [ProductName],
  4:     [Size],
  5:     [Color],
  6:     [ListPrice],
  7:     [DealerPrice]
  8: FROM 
  9:    [dbo].[DimProduct];

When the comma sits in front of each field, the ambiguity goes away. It is immediately clear that [ProductName] is not a separate field — there is no comma in front of it, so it must belong to the line above:

  1: SELECT 
  2:     [EnglishProductName] AS
  3:     [ProductName]
  4:    ,[Size]
  5:    ,[Color]
  6:    ,[ListPrice]
  7:    ,[DealerPrice]
  8: FROM 
  9:    [dbo].[DimProduct];

The Comma and Box Selection

Adding aliases to every field of the following statement is a little awkward:

  1: SELECT 
  2:     [EnglishProductName]
  3:    ,[Size]
  4:    ,[Color]
  5:    ,[ListPrice]
  6:    ,[DealerPrice]
  7: FROM 
  8:    [dbo].[DimProduct];

Every comma has to move at least one position to the right, and the keyword AS and the alias have to be inserted in front of it. That applies to every row of the field list, one at a time, by hand:

  1: SELECT 
  2:     [EnglishProductName] AS [AliasEnglishProductName]
  3:    ,[Size] AS [AliasSize]
  4:    ,[Color] AS [AliasColor]
  5:    ,[ListPrice] AS [AliasListPrice]
  6:    ,[DealerPrice] AS [AliasDealerPrice]
  7: FROM 
  8:    [dbo].[DimProduct];

The same end result can be achieved with far less work if the SQL statement is laid out so that box selection works on it:

No commas need to move. With box selection, AS is typed once, the field names are used as the basis for the alias, copied as a block after AS, and the prefix Alias is prepended — again with box selection. The best part: the effort is essentially independent of the number of rows you have to process.

Box selection becomes truly effective only with leading commas. That is the killer argument for putting the comma at the front.

A note on the SSMS bias. This argument is largely about SSMS and its classic box selection. DataGrip and other modern editors offer more powerful refactoring tools — multi-cursor at arbitrary positions, automatic alias generation via refactor commands, semantic search-and-replace across the whole codebase. In that world you save the same effort even without leading commas. The box-selection argument carries less weight there. The readability arguments (see the "Readability" sub-section) hold regardless of the editor.

Function Names

SSMS highlights function names in pink, which makes them easy enough to spot. Since not every editor does syntax highlighting, function names should also be visually distinguishable by consistent upper- or lower-case spelling.

In other words: function names should be either fully upper case or fully lower case.

Microsoft itself, like most database vendors, writes function names in upper case in the online documentation. That documentation spelling is not a binding standard, but it is a convenient orientation — this blog's convention follows the upper-case form.

Table Aliases

A widely used practice is to derive a table alias as a "speaking" abbreviation from the table name. For the table FactInternetSales one might use the alias IS, taking the starting letters of the compound words in the table name (ignoring the Fact prefix). Since IS is a reserved word, that alias must be wrapped in delimiters. The example also shows a weakness of mnemonic abbreviations: they can collide with reserved words and then force additional quoting. For other tables, you might end up with aliases like these:

Table	Alias
DimCustomer	CUST
DimProduct	P
DimProductCategory	PC
DimProductSubcategory	PSC

A SELECT statement built on those tables might look like this:

  1: SELECT
  2:     CUST.[LastName]
  3:    ,CUST.[FirstName]
  4:    ,P.[EnglishProductName]
  5:    ,PC.[EnglishProductCategoryName]
  6:    ,PSC.[EnglishProductSubCategoryName]
  7:    ,[IS].[OrderDate]
  8: FROM
  9:    [dbo].[FactInternetSales] [IS]
 10:    LEFT JOIN [dbo].[DimCustomer] CUST
 11:    ON
 12:      [IS].[CustomerKey] = CUST.[CustomerKey]
 13:    LEFT JOIN [dbo].[DimProduct] P
 14:    ON
 15:      [IS].[ProductKey] = P.[ProductKey]
 16:    LEFT JOIN [dbo].[DimProductSubcategory] PSC
 17:    ON
 18:      P.[ProductSubcategoryKey] = PSC.[ProductSubcategoryKey]
 19:    LEFT JOIN [dbo].[DimProductCategory] PC
 20:    ON
 21:      PC.[ProductCategoryKey] = PSC.[ProductCategoryKey];

The uneven indentation of the field names is a side-effect of the different lengths of the aliases — somewhere between one and four characters. The field list looks "restless", and as soon as more language elements join the party (functions, CASE expressions etc.) it can become hard to read quickly.

Now imagine a SELECT over twenty tables or more. At some point it becomes hard to come up with a meaningful alias for every table. From roughly the fifth alias onward — that is a rule of thumb, not a hard boundary — the derivation from table names rarely improves readability any further, because the aliases are simply too cryptic.

Wouldn't systematic — possibly even indexed — aliases be easier to identify in a complex statement?

A systematic alias scheme could be defined like this:

Aliases must be a fixed number of characters
Aliases are indexed (with one or more leading letters)

If, for instance, you use the letter T for Table (or F for Fact table, D for Dimension, etc.) followed by a two-digit 1-based index, you end up with aliases such as T01, T02, D01, F01, ….

Using those aliases, the statement above reads more cleanly:

  1: SELECT
  2:     T02.[LastName]
  3:    ,T02.[FirstName]
  4:    ,T03.[EnglishProductName]
  5:    ,T05.[EnglishProductCategoryName]
  6:    ,T04.[EnglishProductSubCategoryName]
  7:    ,T01.[OrderDate]
  8: FROM
  9:    [dbo].[FactInternetSales] T01
 10:    LEFT JOIN [dbo].[DimCustomer] T02
 11:    ON
 12:      T01.[CustomerKey] = T02.[CustomerKey]
 13:    LEFT JOIN [dbo].[DimProduct] T03
 14:    ON
 15:      T01.[ProductKey] = T03.[ProductKey]
 16:    LEFT JOIN [dbo].[DimProductSubcategory] T04
 17:    ON
 18:      T03.[ProductSubcategoryKey] = T04.[ProductSubcategoryKey]
 19:    LEFT JOIN [dbo].[DimProductCategory] T05
 20:    ON
 21:      T05.[ProductCategoryKey] = T04.[ProductCategoryKey];

The field names in the SELECT list line up cleanly.

The real killer argument is the same here, too: aliases of equal length are what makes box selection useful in the first place.

There's another important reason to use aliases at all, and it applies to mnemonic and systematic ones alike: Intellisense (in SSMS) only really works once aliases are in place. When the developer types an alias followed by a dot, a context menu opens listing only the columns of the corresponding table.

When the cursor sits after the dot, the context menu can also be invoked manually with the shortcut Ctrl+Space. It works without a preceding alias as well. In that case, however, SSMS effectively offers the entire T-SQL vocabulary at once.

Aliases make SQL code easier to read and therefore more maintainable. In statements spanning several tables they are practically indispensable. This blog's convention applies them to simple single-table statements as well, for the sake of uniformity and Intellisense comfort.

Qualified Column Names

As soon as a statement references multiple tables, sooner or later the same column name appears in more than one of them and stops being unambiguous. In the example below we want to return name and phone number for both the employee and the reseller of a FactResellerSales fact:

The statement is not executable because the column Phone exists in both dimensions DimEmployee and DimReseller. SQL Server aborts with two errors:

  1: Msg 209, Level 16, State 1, Line 4
  2: Ambiguous column name 'Phone'.
  3: Msg 209, Level 16, State 1, Line 6
  4: Ambiguous column name 'Phone'.

Specifying a table alias in front of the column name is mandatory here:

T01.[Phone] AS [Employee_Phone]

or — if no table aliases are used — the table name itself has to precede the column:

[DimEmployee].[Phone] AS [Employee_Phone]

The latter, however, does nothing for readability.

A fully qualified object name in SQL Server consists of up to four parts: server, database, schema, and object. A column reference adds the column as a fifth element:

[Server].[Database].[Schema].[Table].[Column]

An example with fully qualified table names that include the database name:

Any qualification beyond the schema prevents the application from being deployable in a database that doesn't share the name AdventureWorksDW2017. Cases like that do occur in practice: ETL pipelines whose tables were fully qualified down to the database name ([Database].[Schema].[Table].[Column]) could not be deployed to production once they were finished.

Engine note: Postgres. The five-part scheme Server.Database.Schema.Table.Column only applies in SQL Server. Postgres by default does not allow cross-database queries: a cluster does hold several databases, but a connection is bound to exactly one of them, so the Database. element doesn't exist in practice. If you really need to read or write across database boundaries, you'll need the extensions postgres_fdw (Foreign Data Wrapper) or dblink. Both are extra setup, not a default. In practical terms this means that within a Postgres database, Schema.Table.Column (three parts) is the relevant qualification, while in SQL Server it is one option among several. The recommendation stays the same in both worlds: never qualify beyond the schema level.

Conclusion

A naming convention is the entry point to maintainable SQL code: regular identifiers built from letters and, where needed, digits; delimiters applied consistently; function names in upper case. The small separators have the largest effect of all. The leading comma keeps field lists readable and ready for box selection, the semicolon makes statements unambiguous towards the client and the execution context, and sparing blank lines keep the whole picture on a single screen.

Systematic aliases such as T01 play to their strength in wide statements and line the field list up cleanly. Qualified column names belong up to the schema level and no further. Which individual rule you pick matters less than it seems: SQL becomes maintainable through a consistent convention that makes structure visible and changes easy. Part 2 — Structure and Formatting builds on this and covers the layout of longer statements.

FAQ

Leading comma — isn't that unusual?

When reading, the comma loses its separating role if it sits at the end of a line (see the section "Readability"). When writing, the leading comma saves the most effort the moment box selection comes into play. Modern auto-formatters make the comma position configurable. sqlfluff (T-SQL and Postgres) places commas at the end of the line by default and only switches to leading commas through its layout configuration (line_position = leading). pgFormatter (Postgres) likewise defaults to the trailing comma and offers --comma-start for leading ones.

Do you need square brackets in Postgres too?

No. Postgres does not support [brackets]. Its delimiters are double quotes: "EnglishProductName". There is no QUOTED_IDENTIFIER setting either. Identifier quoting in Postgres is always on. Postgres folds identifiers without quotes to lower case (MyColumn becomes mycolumn). Identifiers in quotes keep their spelling and have to be quoted exactly that way on every reference. If your codebase is multi-engine in the long run, "double quotes" are ANSI-SQL-compliant and work in both worlds — the [brackets] are SQL Server-specific.

Systematic T01-style aliases vs. speaking aliases — which is better?

It's a trade-off. Speaking aliases (CUST, P, PC) are mnemonic and work fine as long as the statement only has three or four tables. As soon as five or more tables come into play — typical for ETL pipelines or wide reporting queries — the mnemonic edge fades and the alignment suffers from the varying lengths. Systematic T01-style aliases scale, stay the same length, and make box selection effortless. The price is semantic: T05.[CustomerKey] doesn't tell you the table. Anyone who wants to know jumps to the FROM clause. Systematic aliases optimise for visual structure and mechanical editing, then, while speaking aliases optimise for orientation in the content. As a rule of thumb, this blog uses speaking aliases up to four tables and systematic ones from five onward.

What if my team uses a different convention?

Consistency beats purity. If the team has settled on trailing commas or speaking aliases, applying that convention consistently is more important than picking the "better" variant.

In practice this is one of the hardest points overall. In every larger team, each developer has a very personal sense of what "looks good" and what doesn't. That individual matter of taste is exactly what stands in the way of team-wide uniformity, and experience across many projects bears this out again and again. All the more reason for the convention that the team has agreed on to be applied consistently across every team member. Only then do the SQL statements stay readable and maintainable for everyone involved.

Multi-cursor — does it replace box selection?

Modern editors like VS Code, DataGrip, and DBeaver provide true multi-cursor support (Ctrl+Alt+Down Arrow or Alt+Click) that goes beyond the rectangular box selection: cursors can sit at arbitrary positions rather than only along a rectangular column. SSMS only has classic box selection (Alt+Shift+Arrow). Azure Data Studio, once part of this line-up, was retired by Microsoft on 28 February 2026; the recommended successor is VS Code with the mssql extension. Conceptually, multi-cursor covers the same need and goes beyond it. Box-selection-friendly formatting remains the prerequisite either way. If you work across several editors, format once cleanly rather than once per tool.

Three Severity Levels, Not Pass/Fail — and Why Severity Decides Where the Rule Lives

Marcus — Sun, 23 Aug 2026 21:39:01 +0000

Most data quality systems don't die of bad checks. They die of a missing field in the rule model: the severity. A missing country code in three out of 80,000 rows blocks the nightly load, someone switches the check off "temporarily", and from that moment everything runs unchecked. Steering data quality with severity levels instead of binary pass/fail escapes that trap. And severity can do more: it decides whether a rule belongs in the target schema as a constraint or in the pipeline.

Key takeaways:

Binary pass/fail ends in one of two dead ends: either a triviality blocks the whole load, or everything passes and nobody reads the findings.
Three levels are enough: an error blocks, a warning is visible without blocking, an information merely documents.
Severity is a routing decision: only an error rule is a candidate for a CHECK or UNIQUE constraint in the target schema — warnings and information stay in the pipeline.
Promoting has a price: when a warning becomes an error, the existing data suddenly has to be clean. NOT VALID and VALIDATE CONSTRAINT make the transition controllable.
The limits can be named precisely: cross-row, cross-table and time-based rules are beyond what a constraint can do. Those belong in the pipeline, permanently.

Prerequisite: Postgres as the example engine and the framework from the sub-hub Data Quality Checks with SQL with the shared error table, the rule configuration dq.check_rule and the quality gate. The principle itself is engine-neutral and applies just as well to SQL Server or any other system with constraints.

Why binary fails

A binary check system knows exactly one question per record: pass or fail. That sounds clean and leads, in practice, into one of two dead ends.

Dead end one: everything that fails blocks. Then a missing email address stops the same load as a duplicate customer number. The process stands still, the business day waits, and the person on call makes a decision at three in the morning that actually belongs in a rule review: they switch the check off. Not out of carelessness, but because the system offers them no other gradation.

Dead end two: nothing blocks, everything is merely logged. Then a findings table fills up that somebody still reads in the beginning. A few weeks later thousands of entries sit there, important ones next to trivial ones, and because the system itself doesn't say which of them demand action, at some point nobody looks anymore. The check keeps running and is dead anyway.

Both reflexes are understandable. The problem is not that people handle the system badly. The problem is that a binary model withholds the one piece of information that should steer the handling: how bad is this finding?

The three levels

Three severity levels answer exactly that question, and you rarely need more than three. The pattern is not new: syslog levels and linters have known it for decades. What makes it interesting is what each level concretely triggers in the ETL process:

Error (E) blocks. A record with an error finding does not pass the quality gate and does not reach the target layer. Error is the right level for everything the load would fail on at the target anyway: missing mandatory values, duplicate keys, unknown references.
Warning (W) is visible but does not block. The record flows on, the finding stays logged. That covers the largest share of real-world data problems: things that are odd, unclean or worth a look, but must not hold up the process.
Information (I) merely documents. No pressure to act, no blocking, pure awareness, for instance a missing phone number whose frequency you want to observe.

In the framework's rule model, the severity (some rule models call the field "criticality") is a column of the configuration table, with a strict default and a CHECK on itself:

,severity   char(1) NOT NULL DEFAULT 'E'
,CONSTRAINT ck_check_rule_severity  CHECK (severity IN ('E', 'W', 'I'))

The default E is a deliberate decision: whoever writes a rule means it seriously at first. Downgrading to a warning is an active, documented step. The opposite default would silently turn new rules into mere observers. Five rules show the three levels in the configuration table's format:

INSERT INTO dq.check_rule
    (check_type,   schema_name, table_name, id1_column,    check_column,   where_clause,           severity, message)
VALUES
    ('constraint', 'staging',   'customer', 'customer_id', 'country_code', 'country_code IS NULL', 'E', 'country code is mandatory in the target schema')
   ,('constraint', 'staging',   'customer', 'customer_id', 'age',          'age IS NULL',          'E', 'age is mandatory in the target schema')
   ,('constraint', 'staging',   'customer', 'customer_id', 'age',          'age < 18',             'W', 'customer under 18 - needs business review')
   ,('constraint', 'staging',   'customer', 'customer_id', 'email',        'email IS NULL',        'W', 'email missing - needs business review')
   ,('constraint', 'staging',   'customer', 'customer_id', 'phone',        'phone IS NULL',        'I', 'phone number not provided - for information only');

The information rule deliberately targets a field that is optional from a business point of view: a missing phone number is not a defect, but a value whose rate you want to know. On age, by contrast, two rules with different levels sit side by side: if the age is missing entirely, that is an error, because the target treats the column as mandatory. If it is merely conspicuously low, that is a warning. The choice of level is a business statement, not a technical one.

What the levels mean in practice is shown by a run against seven demo rows (one clean, one per severity, and three combinations). The framework's staging table is extended by the optional column phone for this:

INSERT INTO staging.customer (customer_id, country_code, email, age, phone) VALUES
    (1, 'DE', 'a@example.com', 44,   '+49 30 1234567')   -- clean
   ,(2, NULL, 'b@example.com', 33,   '+49 40 2345678')   -- E: country code missing
   ,(3, 'AT', 'c@example.com', 17,   '+43 1 3456789')    -- W: under 18
   ,(4, 'CH', 'd@example.com', 52,   NULL)               -- I: phone number missing
   ,(5, NULL, 'e@example.com', NULL, '+41 44 5678901')   -- E+E: country code and age missing
   ,(6, NULL, 'f@example.com', 16,   '+49 89 6789012')   -- E+W: country code missing, under 18
   ,(7, NULL, NULL,            NULL, NULL);              -- E+E+W+I: everything at once

SELECT dq.fn_run_checks('staging', 'customer') AS total_findings;  -- 11

The runner writes the severity counters back into the source table, one per record and level:

customer_id	country_code	email	age	phone	sys_error	sys_warning	sys_info
1	DE	a@example.com	44	+49 30 1234567	0	0	0
2		b@example.com	33	+49 40 2345678	1	0	0
3	AT	c@example.com	17	+43 1 3456789	0	1	0
4	CH	d@example.com	52		0	0	1
5		e@example.com		+41 44 5678901	2	0	0
6		f@example.com	16	+49 89 6789012	1	1	0
7					2	1	1

The quality gate queries only one of the three columns:

SELECT
    customer_id
   ,country_code
   ,email
   ,age
   ,phone
FROM
   staging.customer
WHERE
   sys_error = 0;

Three of the seven rows pass the gate. The warning row and the information row flow along, the four rows with at least one error stay behind — logged, with a plain-text message, findable for follow-up. The combination rows show in passing that sys_error is a counter, not a flag, and that the three counters run independently of each other: row 5 carries two errors, row 6 one error and one warning, row 7 fills all three counters at once. The gate still asks only for sys_error = 0. No on-call engineer has to decide at night whether a missing phone number matters more than the daily close. That decision was made long ago, in the rule review, when someone wrote W into the row.

Severity is a routing decision

Up to here, severity looks like a label on the finding. It can do more: it answers the question of where a rule may be enforced.

The reasoning behind it is simple. A constraint in the target schema is the hardest form a rule can take: on its own, it knows no exception, no log line and no "flows on anyway". Only the calling application can soften that, the constraint itself knows no gradation. Only a rule whose violation must truly never reach the target deserves that hardness. That is exactly the definition of the error level. A warning, on the other hand, is supposed to be allowed through, otherwise it would be an error. A constraint that rejects warning violations would contradict the rule's own definition. The routing follows from that:

E is a candidate for a constraint: NOT NULL, CHECK, UNIQUE, foreign keys. The rule is enforced twice: up front as a check in the source, hard as a guarantee at the target.
W and I stay in the pipeline: they exist only as check rules, produce findings and never block. A target constraint for them is not merely unnecessary, it is wrong.

Severity is therefore not a report field but the switch that decides about constraint derivation. How the error rules of the configuration can conversely be derived from the target schema is shown by this series' sibling article. The coupling works in both directions, and derived rules always carry E there, because their source is a hard constraint.

Concretely, the routing looks like this. The target layer materializes the two error rules as NOT NULL on country_code and age, while the warning rules and the information rule deliberately have no counterpart there:

CREATE TABLE core.customer
(
    customer_id   int   NOT NULL
   ,country_code  text  NOT NULL
   ,email         text
   ,age           int   NOT NULL
   ,phone         text
   ,CONSTRAINT pk_customer  PRIMARY KEY (customer_id)
);

INSERT INTO core.customer (customer_id, country_code, email, age, phone)
SELECT
    customer_id
   ,country_code
   ,email
   ,age
   ,phone
FROM
   staging.customer
WHERE
   sys_error = 0;
-- INSERT 0 3

The 17-year-old row and the row without a phone number arrive as well. That is exactly the intent: the warning has served its purpose (the finding sits in the log), and the target accepts the record. What would happen if someone materialized the warning rule as a constraint after all is shown by the direct attempt:

ALTER TABLE core.customer
   ADD CONSTRAINT ck_customer_adult CHECK (age >= 18);
-- ERROR: check constraint ck_customer_adult of relation customer
--        is violated by some row

The ALTER TABLE fails on its own existing data, on precisely the row the warning was explicitly supposed to let pass. The error is not an accident but the database pointing out the contradiction: this rule was classified as W, and a CHECK is the enforcement form of E. The same holds for the information rule with the opposite sign: its hard counterpart would be a NOT NULL on phone — technically perfectly expressible, but wrong, because an information must never block. The target allows the missing phone number deliberately, the pipeline merely keeps count.

Four places, one decision

Zooming out of the framework, there are four places where a data quality rule can live. Each can do something the others cannot, and each has a price:

Place	What it can do	What it costs
Source (pre-filter before the load)	finds all bad records up front, classifies by severity, the load carries on with the good ones	its own infrastructure: error table, rule configuration, runner
Pipeline (check steps in the process)	the most expressive option — every rule type, every severity, trends and reports	the rule lives next to the data, and every path around the pipeline bypasses it
Target schema (constraints)	the only guarantee that every write path respects, including the manual hotfix	knows only two outcomes, can only express error semantics, blocks the whole run of a set-based load when in doubt
Reporting (dashboards, analyses)	makes warnings and information visible over time, shows trends	enforces nothing, depends on the findings of the other places

Today's industry default clearly sits on the second place: tools like Great Expectations, Soda or dbt tests formulate checks in the pipeline, and most of them know severity gradations as well. dbt, for instance, distinguishes error and warn per test. That is a workable model, not a mistake. You should just be able to name what you are buying into: a rule that lives exclusively in the pipeline is valid only there. The colleague with the direct INSERT, the second load script, the weekend migration — none of them run through the pipeline, and none of them hit its checks. A constraint in the target schema has no such gap, because it sits in the data itself.

The four places are therefore not competitors you pick one of. The decision is made per rule, and severity is its first criterion: error rules deserve the double floor of pre-filter plus constraint, warnings and information belong in pipeline and reporting.

Why the source still checks first

If only error rules become constraints, why check in the source at all? You could simply let the CHECK do what it is there for.

The answer is already in the framework article, and it remains fully valid under the routing view: a constraint knows only two outcomes. The row fits, or the whole load breaks. When loading thousands of rows, "breaks" is the worst of all options, because a single bad row stops the complete process, and the error message names at best that one row, not the other nine that were still waiting behind it.

One qualification belongs here: this all-or-nothing applies to the set-based load, that is, a single INSERT … SELECT moving all rows in one transaction. That is exactly how an ETL process developed in SQL works, and dbt belongs on this side as well, because its models compile to set-based SQL statements. Row-based ETL tools like SSIS or Talend process the rows individually instead: a failing row can be routed out through an error output there, the remaining rows carry on, and the constraint stops only that one row rather than the whole load. You pay for that convenience with the row-by-row processing itself, which is considerably slower than a set-based load, and with error handling that lives inside the tool instead of in a queryable table. The pre-filter is the set-based answer to the same need: it does what a row-based tool's error output does — just up front, in set logic and with severity levels.

The error output also has a second, less obvious weakness: the checks in the data flow run sequentially. The first hit routes the row out, and the remaining checks never see it — unless you explicitly wire the error path through all further check steps, which quickly clutters the data flow. In practice that means: you fix the first error found, run the load again, find the second one, fix it, find the third. The pre-filter knows no such iterating, because every rule runs set-based across all rows. That is why row 7 of the demo sits in the log completely after a single run, with two errors, one warning and one information.

The pre-filter in the source and the constraint at the target are therefore not alternatives but two halves of the same error rule. The check in the source finds all records that would fail at the target, classifies them and lets the load carry on with the clean ones. The constraint at the target guarantees the rule even for everything that goes past the pre-filter: the manual hotfix, the forgotten second script. If the pre-filter fails, the load breaks loudly instead of silently accepting bad data. If the constraint is dropped, the pre-filter still checks. Only together do the two produce the property neither half has alone: complete findings and a hard guarantee.

So the question is not whether schema or pipeline. The question is which rule deserves both — and that is answered by the severity.

What happens when you promote a rule

Rules are not static. The business decides that customers under 18 must no longer be created: the warning from above is to become an error. In the configuration that is one statement:

UPDATE dq.check_rule
SET
   severity = 'E'
WHERE
       schema_name  = 'staging'
   AND table_name   = 'customer'
   AND where_clause = 'age < 18';

SELECT dq.fn_run_checks('staging', 'customer') AS total_findings;  -- 11

From the next run on, the 17-year-old row blocks at the gate (sys_error = 1). That is the easy part. The demanding part follows from the routing: an error rule is a constraint candidate, so the target schema should guarantee the new hardness as well. And right here waits the price of promotion — the existing data suddenly has to be clean. The 17-year-old row arrived at the target long ago, perfectly legitimately, under the old rule. The direct ADD CONSTRAINT fails on it, as seen above.

Postgres offers a controlled path for this transition. NOT VALID accepts the constraint immediately without checking the existing data. For new rows it applies from the first second anyway:

ALTER TABLE core.customer
   ADD CONSTRAINT ck_customer_adult CHECK (age >= 18) NOT VALID;

INSERT INTO core.customer (customer_id, country_code, email, age)
VALUES (9, 'DE', 'x@example.com', 16);
-- ERROR: new row violates check constraint ck_customer_adult

DELETE FROM core.customer WHERE age < 18;

ALTER TABLE core.customer VALIDATE CONSTRAINT ck_customer_adult;

The order is the point. First NOT VALID seals the future, then the existing data is cleaned up calmly (in the demo via DELETE, in practice more likely: routed out for business review), and only the final VALIDATE CONSTRAINT re-checks the existing rows and completes the guarantee. Between the two steps the constraint sits in a documented intermediate state: it applies to new rows while the existing data is still unchecked. Postgres shows this state in the catalog as convalidated = false. The VALIDATE does not lock the table against writes, it runs with a weak lock alongside normal operation. One subtlety in passing: a CHECK does not fire on NULL, so a missing age would pass ck_customer_adult without complaint. Here that case is already caught by the NOT NULL that the error rule age IS NULL materializes at the target anyway. The same pattern carries a retrofitted NOT NULL on an existing table. How to roll that out without downtime is shown in a dedicated article.

For SQL Server the principle holds with different mechanics: WITH NOCHECK likewise adds a constraint without checking the existing data. The difference sits in the aftermath. The constraint stays permanently marked as not trusted until a WITH CHECK CHECK CONSTRAINT re-validates the existing rows, and an untrusted constraint is ignored by the optimizer in plan decisions. Whoever does only the first step has the guarantee but gives away performance.

Demoting, the reverse path, is just as much a routing decision, by the way: E becomes W, so the corresponding constraint at the target has to go, otherwise it keeps blocking a rule that is only supposed to observe. Promoting pulls constraints in, demoting clears them away — configuration and schema move together.

Taken together, that is the real payoff of the three levels: a rule has a controlled lifecycle. It starts as an observation, proves itself, gets promoted and moves into the schema as a guarantee via NOT VALID and VALIDATE CONSTRAINT — and, when needed, walks the same path back. Severity is the control knob of that lifecycle, not just a label on the finding.

The honest limits

The routing "E becomes a constraint" has one restriction that hides in the word candidate: not every error rule can become a constraint. A CHECK sees exactly one row. Within that row it may compare several columns, a CHECK (end_date > start_date) is perfectly legitimate. But three rule families lie beyond its reach:

Cross-row rules. "A key may occur at most three times" is one row in the configuration as a check rule (max_occurrence = 3), but not expressible with the declarative constraints (UNIQUE, CHECK, foreign keys), because UNIQUE knows only cardinality 1. Sum, share and distribution rules belong here as well.
Cross-table rules. A CHECK may not contain a subquery, Postgres rejects that with a clear error message. The only cross-table guarantee the schema knows is the foreign key. Everything beyond it stays pipeline work, say "the discount code must match the customer group".
Time-based rules. Postgres silently assumes that a CHECK expression always yields the same result, but does not enforce that immutability: a CHECK (order_date <= current_date) is accepted, and that is precisely what makes it a trap. What is valid today is no longer valid tomorrow, and at the latest when restoring a dump or running a VALIDATE, existing rows fail that were correct when they were created. Rules with a time reference belong in the pipeline, where every run checks against the current reference date.

The direction of view matters: these rules do not land in the pipeline because sadly nothing better exists. The pipeline is the right place for them: it checks set-based, knows the run context and can still rate a violation as an error and block it at the gate. A cross-row error rule is an error without a constraint counterpart, and that is not a contradiction to the routing but its honest limit.

Decision matrix

The per-rule decision, condensed:

Rule type	Typical severity	Place of enforcement
Mandatory field the target enforces	`E`	pre-filter in the source + `NOT NULL` at the target
Hard value range (type bound, country-code format)	`E`	pre-filter + `CHECK` at the target
Key uniqueness	`E`	pre-filter + `PRIMARY KEY`/`UNIQUE` at the target
Reference to a master table	`E`	pre-filter + `FOREIGN KEY` at the target
Business anomaly (worth a look, not blocking)	`W`	pipeline, finding in the log
Pure observation (frequencies, missing optional values)	`I`	pipeline + reporting
Cross-row (cardinality > 1, sums, shares)	`E` or `W`	pipeline only — `UNIQUE` cannot do cardinality > 1
Cross-table beyond the foreign key	`E` or `W`	pipeline only — `CHECK` cannot hold a subquery
Time-based (date against a reference day)	mostly `W`	pipeline only — a constraint ages badly

Two readings sit in this table. Top to bottom: the harder the guarantee, the higher the rule stands, and only the E rows reach the schema. And across: the place is never "either source or target". Every constraint row carries both, because the pre-filter delivers the findings and the constraint the guarantee.

FAQ

Why is binary pass/fail not enough?
Because it withholds the urgency of a finding. Either every triviality then blocks the load, or nothing blocks and the findings list goes stale. The three severity levels error, warning and information separate "must not proceed" from "worth a look" and "for information only", and they make the reaction configurable per rule instead of negotiable per night shift.

Which check rules belong in a CHECK constraint?
Only error rules whose violation must never reach the target, and of those only the ones a constraint can express: checks on a single row without a time reference and without looking into other tables. Warnings never belong in a constraint, because a constraint that rejects non-blocking findings contradicts the rule's own definition.

What happens when I promote a warning to an error?
Two things. From the next run on, affected records block at the quality gate. And the target schema should follow, because an error rule deserves a constraint counterpart. For that, the existing data has to be clean: in Postgres, ADD CONSTRAINT … NOT VALID takes the guarantee on immediately for new rows, then the existing data is cleaned up and re-checked via VALIDATE CONSTRAINT.

Does a warning block the load?
No, and that is its purpose. A record with warning findings passes the quality gate, and the finding stays in the log with a plain-text message. Blocking is reserved for the severity error alone. A warning that would need to hold up the process is misclassified and should be promoted.

Does data quality belong in dbt and Great Expectations or in the database?
Both have their place. Pipeline tools like dbt tests, Great Expectations or Soda are expressive, well maintained and know severity gradations themselves. Their checks, however, only apply to paths through the pipeline. A direct INSERT past it hits none of them. Hard error rules therefore deserve an additional constraint in the target schema that every write path respects.

Framework and routines:

Data Quality Checks with SQL — the series' sub-hub: error table, rule configuration, runner and the quality gate whose severity mechanics this article deepens.
Validate Data with SQL — the WHERE routine: value ranges, mandatory fields and the NULL trap of three-valued logic.
Find Duplicates with SQL — the uniqueness routine: cardinality, composite keys and the NULL semantics of UNIQUE.
Find Orphaned Records with SQL — the lookup routine: checking referential integrity without foreign keys.
Derive Data Quality Rules from the Schema — the routing's opposite direction: target constraints become error rules of the configuration, projected mechanically.

Theory:

Data Quality: Dimensions and Error Classes — the conceptual frame: which dimensions the checks cover and which they don't.

Target schema and deployment:

Postgres Table Conventions — which keys and constraints a target table should carry so hard counterparts exist at all.
Adding a NOT NULL Column to an Existing Table — the same promotion pattern for mandatory fields: expand/contract without downtime.
Design Pattern // The Architecture of an ETL Process — the staging architecture in which the pre-filter and the gate have their place.

DEV Community: Marcus

Reading Complex XML in SQL Server — Shredding a Thousand Attributes Generically Instead of Looping for Hours

The catalog: three types, one drivetrain with two faces

Why loops over variable attributes get expensive

Load raw: one row per configuration

Shred generically: every leaf with its path

Pass 1: all leaves, no positions

XML attributes

Pass 2: ordinals only where something repeats

Merge

The same with JSON and OPENJSON

Inventory: reading the schema off the data

Generate views instead of writing them

Postgres bridge

Limits and decisions

Summary

FAQ

Related articles

The Context Window Is Full Before the First Prompt — When Project Rules in Claude Code Become a Fixed Cost of Every Session

What a Session Costs Before Anyone Types

Grown With the Work

Rule or Record: The 84 Percent Case

What Gets Added While Working

Three Levers, and What They Cost

Binding Rules to Files With paths:

Separating Rule and Record

Looking Up Instead of Preloading

Before and After: 454,900 to 64,700 Tokens

Without a Guardrail, the Growth Comes Back

What the Solution Is Not

Summary

FAQ

Related Articles

The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent

What the Agent Sees and the Browser Does Not

Case 1: Is the Message Missing, or Just Not Displayed?

Case 2: Two Hours Down the Wrong Rabbit Hole

The Fix Worked and Still Missed the Cause

Spotting Two Docker Engines Earlier: The Rules From the Second Incident

Case 3: “The Page Is Barely Responding”

Case 4: The Rate Limit That Never Kicked In

A Step That Ran Is Not a Step That Worked

The Flip Side: When the Agent Tells a Story

What Remains

FAQ

Related Articles

Set-Based vs. Row by Row: PostgreSQL Refresh from 114 to 12 Seconds

The Starting Point: A Schema Snapshot Across 1,000 Tables

Measuring Runtime When the Web Server Times Out

Where the 114 Seconds Went

The Same Loop Inside the Database

What Spoke for the Loop

What Spoke Against the Loop and What the Switch Costs

The Decision: Four Options and a Budget Calculation

How a Refresh Runs Today

How the New Solution Brings Back the Loop's Strengths

Skipping Unchanged Tables

Set-Based Is Not Automatically Fast: Why a Query Over All Tables Can Be Slower

The Privilege Trap When Switching to pg_catalog

The Result: Factor 9 to 19, Targets Still Missed

When an Optimization Changes What a Display Means

What Remains

FAQ

Related Articles

Two Agents, One Working Tree — What Broke When I Ran Parallel Claude Code Sessions in One Git Repository

Why Parallel Claude Code Sessions Produce Errors That Don’t Exist Alone

Case 1: Foreign Work Swept Into a Commit

Case 2: The Obvious Countermeasure Doesn’t Hold

Case 3: Foreign Work Deleted

Case 4: Duplicate Numbers, Because Politeness Is Symmetric

Case 5: The Machine Is Shared Too

What Holds and What Doesn’t

Summary

FAQ

Related Articles

An EU Region Is Not an EU Provider — Vercel, Supabase and the GDPR Third-Country Question

The Short Answer: Not by Default

What an EU Region Delivers — and Who Is in the Chain

The Transfer Mechanism: Three Attempts

The CLOUD Act: Possession or Control, Not Storage Location

Binding Rules to Files With `paths:`