<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Judy</title>
    <description>The latest articles on DEV Community by Judy (@esproc_spl).</description>
    <link>https://dev.to/esproc_spl</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1191782%2Fdac3272f-c56a-41d9-914e-8f8fba86506b.jpg</url>
      <title>DEV Community: Judy</title>
      <link>https://dev.to/esproc_spl</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/esproc_spl"/>
    <language>en</language>
    <item>
      <title>Compilable and Executable Pseudocode (spec) Solves AI Coding Hallucinations</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Fri, 24 Jul 2026 03:56:21 +0000</pubDate>
      <link>https://dev.to/esproc_spl/compilable-and-executable-pseudocode-spec-solves-ai-coding-hallucinations-160f</link>
      <guid>https://dev.to/esproc_spl/compilable-and-executable-pseudocode-spec-solves-ai-coding-hallucinations-160f</guid>
      <description>&lt;p&gt;The AI-generated SQL – no one dares to say they fully “understand” it&lt;br&gt;
First, look at the following SQL query, which aims to find credit card customers whose transaction amount has increased for three consecutive months:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH ranked AS (
  SELECT customer_id, month, amount,
    LAG(amount,1) OVER(PARTITION BY customer_id ORDER BY month) as prev_1,
    LAG(amount,2) OVER(PARTITION BY customer_id ORDER BY month) as prev_2
  FROM transactions
)
SELECT customer_id, COUNT(*) FROM ranked
WHERE amount &amp;gt; prev_1 AND prev_1 &amp;gt; prev_2
AND prev_1 IS NOT NULL AND prev_2 IS NOT NULL
GROUP BY customer_id HAVING COUNT(*) &amp;gt;= 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The syntax is correct, and it runs. But is the logic sound? No one can say for sure right away.&lt;/p&gt;

&lt;p&gt;This is the daily routine of data analysts and developers today: they give a task to an AI, get dozens of lines of SQL in seconds, copy and paste the code, and it runs. But can they really trust the results? Modern AI can generate runnable SQL, but it is often not fully reliable.&lt;/p&gt;

&lt;p&gt;AI-generated SQL is a black box. The code may look correct, run without errors, and return a clean result, but what it actually does may have nothing to do with what you intended. Worse, you only realize it’s wrong after it has already executed. During the review phase, you have to mentally execute the SQL yourself, effectively redoing the work that the AI was supposed to handle.&lt;/p&gt;

&lt;p&gt;Even worse, even if the accuracy of AI-generated SQL reaches 90%, how can you know that this time they aren’t falling into the remaining 10%? Data-driven decisions should not become a game of chance. You get a SQL query that executes successfully and returns a clean result, but what it actually does is not what the user intended. This type of error cannot be caught by syntax checking. And the verification cost during the code review phase is extremely high: to confirm whether the logic is correct, you have to break down CTEs layer by layer, run them step by step, and compare the intermediate results.&lt;/p&gt;

&lt;p&gt;What is the real problem?&lt;br&gt;
In AI-assisted programming, there is a basic workflow: the user writes a clear specification and AI generates runnable code.&lt;/p&gt;

&lt;p&gt;This is not a question of whether AI is “capable” or not, but a hard constraint under current technical conditions. If the user only sketches out a couple of lines of specification, no AI can generate runnable engineering-grade code out of thin air – unless the task is a common, well-defined one such as bubble sort. The actual AI-assisted programming workflow is: the user writes a spec → the AI generates code → the user reviews it → spec the refined → the AI regenerates the code. The spec is both the starting point and a constraint throughout the entire process.&lt;/p&gt;

&lt;p&gt;But even with a perfectly clear specification, AI can still make mistakes – it’s a matter of probability. A poorly defined specification increases the chance that the AI will guess wrong, while a well-defined specification reduces that probability but can never eliminate it. This is because, at its core, AI is making probabilistic predictions. From input to output, it searches the space of possible answers for the most likely one, rather than logically deriving the only correct result.&lt;/p&gt;

&lt;p&gt;Then what can we do? Is there any way to turn a “likely answer” to a “definitive one”?&lt;/p&gt;

&lt;p&gt;The answer is: make the spec compliant enough to compile. Once the spec can be parsed and executed by a compiler, AI no longer needs to “guess” the code. The compiler is deterministic – the same input always produces the same output. No more “works this time, breaks the next”. This isn’t about “reducing the hallucination rate”; it’s about bypassing the stage where hallucinations can occur.&lt;/p&gt;

&lt;p&gt;So, the overall logic chain is clear: AI-assisted programming requires a spec → a well-defined spec can reduce hallucination rate → but it cannot eliminate hallucinations → make the spec compliant enough to compile → use a compiler instead of AI to generate code →hallucinations disappear.&lt;/p&gt;

&lt;p&gt;Compilable pseudocode: an advanced form of the specification&lt;br&gt;
Traditionally, the requirements document that programmers create is intended for humans. It contains natural language, charts and diagrams, and pseudocode. In the era of AI, the spec can evolve into a new form with two characteristics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Human-readable: No need to be familiar with the target language (SQL, Java, Python), you can grasp the logic at a glance.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Machine-compilable: The specification has a fixed syntactic structure and can be deterministically transformed into final code, without relying on AI guessing.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is “compilable pseudocode”, a more compliant spec. The difference between it and traditional pseudocode is this: traditional pseudocode won’t be compiled – it is intended only for humans to read; compilable pseudocode is both human-readable and machine-readable, so it can be compiled into target code.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fovssrmmdfdgxaqdofo41.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fovssrmmdfdgxaqdofo41.png" alt="an advanced form of the specification" width="799" height="197"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The value of this kind of spec is that it squeezes AI’s “creative space” down to a minimum. AI doesn't need to “understand” the business logic, “design” an implementation plan, and “guess” the correct code – it only needs to translate the uncompliant spec into a compliant one. Translation leaves a far narrower margin for error than “generating code directly from natural language”.&lt;/p&gt;

&lt;p&gt;More importantly, you can verify the correctness of the spec before it is compiled and executed. Each step of the logic can be verified independently, so errors are caught before execution rather than being debugged afterward.&lt;/p&gt;

&lt;p&gt;SQLazy’s workflow: The compilable pseudocode&lt;br&gt;
SQLazy brings this concept to the field of complex SQL development. Its workflow is essentially compilable pseudocode: developers describe data querying logic step by step in natural language, previewing the intermediate result at each step. Once the logic is confirmed correct, the compiler deterministically generates the SQL.&lt;/p&gt;

&lt;p&gt;Still on the same requirement as above: finding the credit card customers whose transaction amount has increased for three consecutive months. Let’s implement this using SQLazy’s workflow:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwlfjvr23vcd4mf9g8hqn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwlfjvr23vcd4mf9g8hqn.png" alt="SQLazy’s code" width="800" height="213"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6evz7frxkbsdocud4aq1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6evz7frxkbsdocud4aq1.png" alt="workflow" width="800" height="427"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;You don’t need to know SQL – the four steps alone make the logic clear. At every step you can preview the intermediate result:&lt;/p&gt;

&lt;p&gt;sort: Sort rows by customer and month.&lt;/p&gt;

&lt;p&gt;segment: Mark records where the transaction amounts fall, and generate a number for each consecutively rising segment.&lt;/p&gt;

&lt;p&gt;summarize: Count the number of months covered by each segment.&lt;/p&gt;

&lt;p&gt;filter: Select customers whose transaction amounts increase for at least consecutive 3 months (&amp;gt;=3).&lt;/p&gt;

&lt;p&gt;Here is another real-world example. A table stores time intervals for multiple accounts. Each account corresponds to multiple records, and the intervals may overlap. The goal is to merge all overlapping intervals within each account. With SQLazy, this can be implemented in just four steps:&lt;/p&gt;

&lt;p&gt;sort account_id, start_date: Sort rows by account and start date.&lt;/p&gt;

&lt;p&gt;compute end_date[:-1] max as prev_max: Calculate the maximum end date among all rows prior to the current row.&lt;/p&gt;

&lt;p&gt;segment condition start_date &amp;gt; prev_max as gid: Segment rows based on the condition, and assign a group number to each segment.&lt;/p&gt;

&lt;p&gt;summarize start_date min as start_date, end_date max as end_date: Aggregate by account and group number.&lt;/p&gt;

&lt;p&gt;Once the above logic is confirmed correct, the SQLazy compiler automatically generates native SQL – 100% accurate, with zero hallucination risk. If you switch the target database from MySQL to Oracle, you only need to change one option and the compiler automatically adapts to the target SQL dialect.&lt;/p&gt;

&lt;p&gt;SQLazy’s workflow is the spec, and the compiler is the last gate guaranteeing the code quality.&lt;/p&gt;

&lt;p&gt;In this process, the role of AI is also redefined. Conventionally, AI is responsible for generating the final SQL directly from the natural language – an extremely difficult task with a high hallucination rate. With SQLazy, AI is charged with only one thing: translating users’ colloquial, step-by-step descriptions into formal workflow syntax. The former is “decision-making” while the latter is “translation”. Even if the AI’s translation is slightly off, you can spot it at a glance at the workflow level, and the cost of correction is nearly zero.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsygvcmqw8z67gwjbr4i1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsygvcmqw8z67gwjbr4i1.png" alt="spec" width="800" height="428"&gt;&lt;/a&gt;&lt;br&gt;
Use LLM to convert the natural language into SQLazy syntax&lt;/p&gt;

&lt;p&gt;AI hallucinations in programming are a problem not because AI occasionally makes mistakes, but because of something more fundamental: under the conventional model, AI’s mistakes can only be discovered after execution. By the time you find the result is wrong, the code has already been generated – even executed – making the cost of correction extremely high.&lt;/p&gt;

&lt;p&gt;SQLazy’s compilable pseudocode shifts the moment of discovering errors from after execution to before execution. Humans confirm the logic at the spec level, and the compiler generates the final code. AI’s room for error is squeezed down to the “translation” step alone, where the result can be verified instantly by humans – solving the hallucination problem in AI programming.&lt;/p&gt;

&lt;p&gt;This does not mean programmers are going back to the era of “writing documentation”. Rather, it redivides the labor between humans and AI: humans handle design (describing “what to do” using structured spec); AI handles translation (converting colloquial spec into compliant spec); and the compiler handles execution (guaranteeing a deterministic output).&lt;/p&gt;

&lt;p&gt;The next phase of AI programming is probably not about making AI generate longer code, but about enabling humans to write clearer spec.&lt;/p&gt;

</description>
      <category>code</category>
      <category>sql</category>
      <category>devops</category>
      <category>software</category>
    </item>
    <item>
      <title>SQLazy: Querying the Start Timestamp of the Next Group from the Event Table</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Wed, 22 Jul 2026 07:24:00 +0000</pubDate>
      <link>https://dev.to/esproc_spl/sqlazy-querying-the-start-timestamp-of-the-next-group-from-the-event-table-4ja4</link>
      <guid>https://dev.to/esproc_spl/sqlazy-querying-the-start-timestamp-of-the-next-group-from-the-event-table-4ja4</guid>
      <description>&lt;p&gt;Problem description&lt;br&gt;
A table stores status values of an object at different timestamps, with one record per timestamp. Segment rows based on changes in the status value and output the effective start timestamp and the effective end timestamp for each status segment – that is, merge consecutive records with the same status into a single time interval.&lt;/p&gt;

&lt;p&gt;Source data:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvzybe3lhpudzmr4cz2g5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvzybe3lhpudzmr4cz2g5.png" alt="Source data" width="800" height="415"&gt;&lt;/a&gt;&lt;br&gt;
Expected output (intervals where each value remains constant):&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6q956aiw47ozk7kce3g6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6q956aiw47ozk7kce3g6.png" alt="Expected output " width="800" height="228"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Take object 1 as an example:&lt;/p&gt;

&lt;p&gt;The first three records contain the same value A, so they can be merged into a single interval: 2024-01-01 – 2024-01-04 (the timestamp when the next record begins).&lt;/p&gt;

&lt;p&gt;Value B remains for record 4 and record 5 and they can be merged into a single interval: 2024-01-04 – 2024-01-06 (the timestamp when the next record begins).&lt;/p&gt;

&lt;p&gt;Record 6 and record 7 have value A again, the same as before. However, since value B intervenes between them, a new group starts: 2024-01-06 – infinity (the end date of the last interval is 9999-12-31).&lt;/p&gt;

&lt;p&gt;Step-by-step implementation with SQLazy&lt;br&gt;
Key approach: sort rows by timestamp, and check whether the value in the current row has changed relative to the previous row. If it has changed, start a new group; otherwise, put it in the current group. Finally, take the minimum timestamp in each group as the start timestamp and the minimum timestamp in the next group as its end timestamp.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcx6y5eqfj8kxhxalnetb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcx6y5eqfj8kxhxalnetb.png" alt="Step-by-step implementation" width="799" height="291"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;[Run this example online]&lt;/p&gt;

&lt;p&gt;Let me walk through each step below:&lt;/p&gt;

&lt;p&gt;Step 1: Sort rows in chronological order in ascending order&lt;/p&gt;

&lt;p&gt;sort timestamp&lt;/p&gt;

&lt;p&gt;Sort rows by timestamp in ascending order, ensuring intervals in each group are processed in chronological order.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F26ciniiwswek1402vxup.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F26ciniiwswek1402vxup.png" alt="order in ascending order" width="800" height="429"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 2: Start a new partition when value changes&lt;/p&gt;

&lt;p&gt;segment value; change; as gid&lt;/p&gt;

&lt;p&gt;This is the core step. The “segment” statement segment rows by the “value” field: each time the value changes, a new group is started and assigned a group number (gid). Within each group, the value remains unchanged.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmda6ht6bna3bkefxr0a5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmda6ht6bna3bkefxr0a5.png" alt="Start a new partition " width="800" height="424"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 3: Aggregate by group number and value, and take the start timestamp&lt;/p&gt;

&lt;p&gt;summarize timestamp first as effective_from, id first as id; group gid, value&lt;/p&gt;

&lt;p&gt;Group rows by gid and value:&lt;/p&gt;

&lt;p&gt;Take the timestamp of the first record in each group and use it as the effective start timestamp for that status.&lt;/p&gt;

&lt;p&gt;id: Take the first value in each group.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe5dkl88t2qa5mtpi944t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe5dkl88t2qa5mtpi944t.png" alt="Group rows by gid and value" width="799" height="295"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 4: Calculate the effective end timestamp&lt;/p&gt;

&lt;p&gt;&lt;em&gt;compute nvl(effective_from[1], datetime("9999-12-31 00:00:00")) as effective_to&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;effective_from[1] takes the effective_from value of the next row after the current group as the end timestamp for the current group. If the current group is the last group (i.e., effective_from[1] is empty), use nvl to set it to a maximum date (9999-12-31), which represents “to present”.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgzd0pvowmlji7iy2pbuy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgzd0pvowmlji7iy2pbuy.png" alt="Calculate the effective end timestamp" width="800" height="296"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Finally, remove the helper column gid and output the final result.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9v9zg6gfohu4cuizw4am.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9v9zg6gfohu4cuizw4am.png" alt="remove the helper column gid " width="799" height="284"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Compile the steps into SQL&lt;/p&gt;

&lt;p&gt;Once the above steps are complete and verified, SQLazy’s compiler can automatically generate the equivalent native SQL (The Oracle syntax for this example):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH Value AS (
    SELECT
        id,
        value,
        timestamp
    FROM
        events
),
Value2 AS (
    SELECT
        gid,
        id AS id,
        value AS value,
        timestamp AS effective_from
    FROM
        (
            SELECT
                id,
                value,
                timestamp,
                SUM(
                    CASE
                        WHEN value &amp;lt;&amp;gt; col__5 THEN 1
                        ELSE 0
                    END
                ) OVER (
                    ORDER BY
                        CASE
                            WHEN timestamp IS NULL THEN 1
                            ELSE 0
                        END,
                        timestamp ASC
                ) + 1 AS gid
            FROM
                (
                    SELECT
                        Value.*,
                        LAG(value) OVER (
                            ORDER BY
                                CASE
                                    WHEN timestamp IS NULL THEN 1
                                    ELSE 0
                                END,
                                timestamp ASC
                        ) AS col__5
                    FROM
                        Value
                ) sub__6
        ) Value1
    GROUP BY
        gid
)
SELECT
    gid,
    id,
    value,
    effective_from,
    LEAD(
        effective_from,
        1,
        TO_DATE('9999-12-31 00:00:00', 'YYYY-MM-DD HH24:MI:SS')
    ) OVER (
        ORDER BY
            gid
    ) AS effective_to
FROM
    Value2
ORDER BY
gid;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You only need to verify the logic of each of the above four steps – no need to manually debug the SQL – and the compiler generates the production-ready code.&lt;/p&gt;

&lt;p&gt;SQLazy lets you describe logic using business language, rather than writing nested queries in SQL syntax. This example of “dividing time intervals based on value change” has one core statement: segment value; change, which directly corresponds to the business requirement of “starting a new group whenever the value changes”. To write SQL manually, you need to understand the window boundaries of LAG/LEAD, manually handle NULLs, and adjust for syntax differences across different databases. SQLazy, however, compresses this complex logic into 4 intuitive steps – sort, segment, aggregate, and take the the next row’s timestamp as the end timestamp. The intermediate result of each step can be verified independently, and the compiler generates the final runnable SQL. You just need to verify the business meaning of each step, and leave the rest to the compiler.&lt;/p&gt;

&lt;p&gt;Try SQLazy online: sqlazy.com (Free to use, signup not required)&lt;br&gt;
SQLazy project repository: github.com/SPLWare/SQLazy&lt;/p&gt;

</description>
      <category>devops</category>
      <category>sql</category>
      <category>programming</category>
      <category>sqlazy</category>
    </item>
    <item>
      <title>SQLazy: Merging Overlapping Time Intervals</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Thu, 16 Jul 2026 02:23:06 +0000</pubDate>
      <link>https://dev.to/esproc_spl/sqlazy-merging-overlapping-time-intervals-4fd0</link>
      <guid>https://dev.to/esproc_spl/sqlazy-merging-overlapping-time-intervals-4fd0</guid>
      <description>&lt;h2&gt;
  
  
  Problem description
&lt;/h2&gt;

&lt;p&gt;A table stores multiple time intervals for each account, and the intervals may overlap.&lt;/p&gt;

&lt;p&gt;Source data:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8m2kzxbh6kvefw4yiesu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8m2kzxbh6kvefw4yiesu.png" alt="Source data" width="799" height="495"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Expected output (merge overlapping intervals for each account):&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8ag38451p1lfionpe05n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8ag38451p1lfionpe05n.png" alt="output" width="800" height="276"&gt;&lt;/a&gt;&lt;br&gt;
Take account A as an example:&lt;/p&gt;

&lt;p&gt;The first three intervals (6/20-6/29, 6/25-7/25, 7/20-8/26) overlap and can be merged into 6/20 - 8/26;&lt;/p&gt;

&lt;p&gt;The nonoverlapping interval 12/25 - 1/25;&lt;/p&gt;

&lt;p&gt;The overlapping intervals (4/27-7/27, 6/25-7/14, 7/10-8/14) can be merged into 4/27-8/14;&lt;/p&gt;

&lt;p&gt;The nonoverlapping interval 9/10-11/12.&lt;/p&gt;

&lt;p&gt;Apply the same merging operations to account B: merge any intervals that overlap.&lt;/p&gt;
&lt;h2&gt;
  
  
  Step-by-step implementation with SQLazy
&lt;/h2&gt;

&lt;p&gt;Key approach: Check whether the current interval’s start date is later than the maximum end date among all previous intervals. If it is, the current interval does not overlap with any of the previous intervals and a new group should be started; otherwise, merge it into the current group.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fslp41rtwbw38bksp8w2u.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fslp41rtwbw38bksp8w2u.png" alt="Step-by-step implementation" width="800" height="329"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.sqlazy.com/?4Bs" rel="noopener noreferrer"&gt;[Run this example online]&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 1: Sort rows by account and the start date&lt;/p&gt;

&lt;p&gt;sort account_id, start_date&lt;/p&gt;

&lt;p&gt;Sort rows by account_id and start_date in ascending order, ensuring intervals within each account are processed in chronological order.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fge0yyqk6f7l9x297gtxu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fge0yyqk6f7l9x297gtxu.png" alt="Sort rows by account and the start date" width="800" height="438"&gt;&lt;/a&gt;&lt;br&gt;
Step 2: Calculate the maximum end date among all rows before the current row&lt;/p&gt;

&lt;p&gt;compute end_date[:-1] max as prev_max; partition account_id&lt;/p&gt;

&lt;p&gt;Within each account partition, calculate the maximum end_date value among all rows from the first row up to, but excluding, the current row, and record it as prev_max.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb1za7kgvv857kl1t5t44.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb1za7kgvv857kl1t5t44.png" alt="compute end_date" width="800" height="442"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 3: Segment rows based on the condition and assign group numbers&lt;/p&gt;

&lt;p&gt;segment condition start_date &amp;gt; prev_max as gid; partition account_id&lt;/p&gt;

&lt;p&gt;Check each row in sequence: if start_date &amp;gt; prev_max, the current interval does not overlap with any of the previous intervals and a new group is started (gid+1); otherwise, the row is assigned to the current group.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3vg9hrcgt00m29piegu8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3vg9hrcgt00m29piegu8.png" alt="segment condition start_date " width="800" height="440"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step 4: Aggregate by account and group number&lt;/p&gt;

&lt;p&gt;summarize start_date min as start_date, end_date max as end_date; group account_id, gid&lt;/p&gt;

&lt;p&gt;For each group, extract the earliest start date and the latest end date to form a merged interval.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7qu6hau5az99xna4zt5s.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7qu6hau5az99xna4zt5s.png" alt="summarize start_date" width="799" height="281"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Finally, remove the helper column gid.&lt;/p&gt;

&lt;p&gt;Compile the steps into SQL&lt;/p&gt;

&lt;p&gt;Once the above steps are complete and verified, SQLazy’s compiler can automatically generate the equivalent native SQL (using MySQL as an example):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH t2 AS (
    SELECT
        account_id,
        start_date,
        end_date,
        MAX(end_date) OVER (
            PARTITION BY account_id
            ORDER BY
                CASE WHEN account_id IS NULL THEN 1 ELSE 0 END,
                account_id ASC,
                CASE WHEN start_date IS NULL THEN 1 ELSE 0 END,
                start_date ASC
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS prev_max
    FROM
        acc
),
t3 AS (
    SELECT
        account_id,
        start_date,
        end_date,
        prev_max,
        1 + SUM(
            CASE
                WHEN start_date &amp;gt; prev_max THEN 1
                ELSE 0
            END
        ) OVER (
            PARTITION BY account_id
            ORDER BY
                CASE WHEN account_id IS NULL THEN 1 ELSE 0 END,
                account_id ASC,
                CASE WHEN start_date IS NULL THEN 1 ELSE 0 END,
                start_date ASC
        ) AS gid
    FROM
        t2
)
SELECT
    account_id,
    MIN(start_date) AS start_date,
    MAX(end_date) AS end_date
FROM
    t3
GROUP BY
    account_id,
    gid
ORDER BY
    account_id,
    start_date;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You only need to verify the logic of each of the four steps – no need to understand or debug the SQL – and the compiler will generate the production-ready code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why SQLazy is more efficient
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjkxznksxs1dtj89lx7xh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjkxznksxs1dtj89lx7xh.png" alt="Why SQLazy is more efficient" width="800" height="548"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;SQLazy lets you describe logic using business language, rather than writing nested queries in SQL syntax. This example of “merge overlapping intervals” is expressed clearly in just 4 steps – sort, calculate the maximum end date among the preceding rows, segment rows by condition, and aggregate. The compiler generates the final SQL for you, and you only need to verify the business meaning of each step.&lt;/p&gt;

&lt;p&gt;Try SQLazy online: &lt;a href="https://sqlazy.com/" rel="noopener noreferrer"&gt;sqlazy.com&lt;/a&gt; (Free to use, signup not required)&lt;br&gt;
SQLazy project repository: &lt;a href="https://github.com/SPLWare/SQLazy" rel="noopener noreferrer"&gt;github.com/SPLWare/SQLazy&lt;/a&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>programming</category>
      <category>discuss</category>
      <category>sqlazy</category>
    </item>
    <item>
      <title>SQLazy: Account-based Grouping with Sequence Number Reset on Gaps Exceeding 1 Hour</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Tue, 14 Jul 2026 02:50:39 +0000</pubDate>
      <link>https://dev.to/esproc_spl/sqlazy-account-based-grouping-with-sequence-number-reset-on-gaps-exceeding-1-hour-4jip</link>
      <guid>https://dev.to/esproc_spl/sqlazy-account-based-grouping-with-sequence-number-reset-on-gaps-exceeding-1-hour-4jip</guid>
      <description>&lt;p&gt;Problem description&lt;br&gt;
Group by account and reset the sequence number when the time interval between events exceeds 1 hour&lt;/p&gt;

&lt;p&gt;A table consists of two fields: account_number and dt. Records within each account are ordered by time, and sequence numbers (Seq) need to be generated according to the following rules:&lt;/p&gt;

&lt;p&gt;Group records by account and order them by datetime in ascending order.&lt;/p&gt;

&lt;p&gt;If the interval between the current event time and the previous one exceeds 1 hour, reset the sequence number to 1.&lt;/p&gt;

&lt;p&gt;Otherwise (the interval ≤ 1 hour), add 1 to the sequence number.&lt;/p&gt;

&lt;p&gt;Source data:&lt;/p&gt;

&lt;p&gt;account_number dt&lt;/p&gt;

&lt;p&gt;19 2024-04-03 07:02:02&lt;/p&gt;

&lt;p&gt;19 2024-04-03 07:02:41&lt;/p&gt;

&lt;p&gt;19 2024-04-03 14:58:49&lt;/p&gt;

&lt;p&gt;19 2024-04-03 19:58:49&lt;/p&gt;

&lt;p&gt;19 2024-04-05 14:58:49&lt;/p&gt;

&lt;p&gt;19 2024-04-05 14:59:31&lt;/p&gt;

&lt;p&gt;19 2024-04-17 23:56:13&lt;/p&gt;

&lt;p&gt;20 2024-04-17 23:59:13&lt;/p&gt;

&lt;p&gt;19 2024-04-18 00:15:13&lt;/p&gt;

&lt;p&gt;19 2024-04-18 14:56:13&lt;/p&gt;

&lt;p&gt;20 2024-04-18 07:41:55&lt;/p&gt;

&lt;p&gt;20 2024-04-18 19:41:55&lt;/p&gt;

&lt;p&gt;20 2024-04-18 19:56:55&lt;/p&gt;

&lt;p&gt;19 2024-04-19 07:41:55&lt;/p&gt;

&lt;p&gt;19 2024-04-19 07:42:20&lt;/p&gt;

&lt;p&gt;19 2024-04-19 08:41:20&lt;/p&gt;

&lt;p&gt;Expected output:&lt;/p&gt;

&lt;p&gt;19 2024-04-03 07:02:02 1&lt;/p&gt;

&lt;p&gt;19 2024-04-03 07:02:41 2&lt;/p&gt;

&lt;p&gt;19 2024-04-03 14:58:49 1&lt;/p&gt;

&lt;p&gt;19 2024-04-03 19:58:49 1&lt;/p&gt;

&lt;p&gt;19 2024-04-05 14:58:49 1&lt;/p&gt;

&lt;p&gt;19 2024-04-05 14:59:31 2&lt;/p&gt;

&lt;p&gt;19 2024-04-17 23:56:13 1&lt;/p&gt;

&lt;p&gt;19 2024-04-18 00:15:13 2&lt;/p&gt;

&lt;p&gt;19 2024-04-18 14:56:13 1&lt;/p&gt;

&lt;p&gt;19 2024-04-19 07:41:55 1&lt;/p&gt;

&lt;p&gt;19 2024-04-19 07:42:20 2&lt;/p&gt;

&lt;p&gt;19 2024-04-19 08:41:20 3&lt;/p&gt;

&lt;p&gt;20 2024-04-17 23:59:13 1&lt;/p&gt;

&lt;p&gt;20 2024-04-18 07:41:55 1&lt;/p&gt;

&lt;p&gt;20 2024-04-18 19:41:55 1&lt;/p&gt;

&lt;p&gt;20 2024-04-18 19:56:55 2&lt;/p&gt;

&lt;p&gt;Step-by-step implementation with SQLazy&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faoit9qn176r6s30d5rp4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faoit9qn176r6s30d5rp4.png" alt="SQLazy" width="800" height="174"&gt;&lt;/a&gt;&lt;br&gt;
[Run this example online]&lt;/p&gt;

&lt;p&gt;Let me walk through each step below:&lt;/p&gt;

&lt;p&gt;Step 1: Sort rows by account and datetime in ascending order&lt;/p&gt;

&lt;p&gt;sort account_number asc, dt asc&lt;/p&gt;

&lt;p&gt;Sort the original data by account_number and dt in ascending order. This step ensures that records within each account are processed in chronological order.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvlid3clewkzslnbzltwb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvlid3clewkzslnbzltwb.png" alt="ascending order." width="800" height="390"&gt;&lt;/a&gt;&lt;br&gt;
Step 2: Partition records, check whether the time interval exceeds 1 hour, and generate the group number&lt;/p&gt;

&lt;p&gt;segment condition ((dt[-1] elapse 3600 second)&amp;lt;= dt) partition account_number as grp&lt;/p&gt;

&lt;p&gt;Within each partition of account, check whether the interval between the current row’s dt and the previous row’s dt exceeds 1 hour (3,600 seconds). The condition (dt[-1] elapse 3600 second)&amp;lt;= dt means that after adding 3,600 seconds to the previous row’s timestamp, it is less than or equal to the current row’s timestamp. If so, the interval is greater than or equal to 1 hour, and a new group begins; otherwise, the current row remains in the same group as the previous row. In the end, a group number (grp) is assigned to each record.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fewrcnagg2i38m3tklgbd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fewrcnagg2i38m3tklgbd.png" alt="generate the group number" width="799" height="394"&gt;&lt;/a&gt;&lt;br&gt;
Step 3: Generate sequence numbers within each subgroup, partitioned by account_number, grp&lt;/p&gt;

&lt;p&gt;compute # as seq partition account_number, grp&lt;/p&gt;

&lt;p&gt;“#” represents a sequence number automatically generated in order within each subgroup, starting from 1. Records are partitioned by account number and group number, and within each subgroup, sequence numbers (seq) are assigned in the existing order of the records (which have already been sorted).&lt;/p&gt;

&lt;p&gt;Running this step produces the following output:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6mp0x5mraq1paf2wgp6e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6mp0x5mraq1paf2wgp6e.png" alt=" account_number, grp" width="800" height="394"&gt;&lt;/a&gt;&lt;br&gt;
Finally, remove the helper column grp, or simply export the required columns, to obtain the final output.&lt;/p&gt;

&lt;p&gt;Compile the steps into SQL&lt;/p&gt;

&lt;p&gt;Once the above steps are complete, SQLazy’s compiler can generate the equivalent native SQL, without the need to write it manually.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fjxbuy9inowjrgqrrhm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fjxbuy9inowjrgqrrhm.png" alt="Compile the steps into SQL" width="800" height="421"&gt;&lt;/a&gt;&lt;br&gt;
Below is the generated SQL statement written in the target database dialect (MySQL in this example):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;t2&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt;
    &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;dt&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt;
    &lt;span class="n"&gt;numEvents&lt;/span&gt;
&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="n"&gt;t3&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt;
    &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="k"&gt;CASE&lt;/span&gt;
        &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;col__5&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;INTERVAL&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt; &lt;span class="k"&gt;SECOND&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
      &lt;span class="k"&gt;END&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="k"&gt;PARTITION&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt;
      &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt;
        &lt;span class="k"&gt;CASE&lt;/span&gt;
          &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
          &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="k"&gt;CASE&lt;/span&gt;
          &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
          &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;grp&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="k"&gt;SELECT&lt;/span&gt;
        &lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;LAG&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
          &lt;span class="k"&gt;PARTITION&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt;
          &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt;
            &lt;span class="k"&gt;CASE&lt;/span&gt;
              &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
              &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
            &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="k"&gt;CASE&lt;/span&gt;
              &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
              &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
            &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;col__5&lt;/span&gt;
      &lt;span class="k"&gt;FROM&lt;/span&gt;
        &lt;span class="n"&gt;t2&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;sub__6&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt;
  &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;grp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;ROW_NUMBER&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;PARTITION&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;grp&lt;/span&gt;
    &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt;
      &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;CASE&lt;/span&gt;
        &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
      &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;CASE&lt;/span&gt;
        &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
      &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;seq&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;
  &lt;span class="n"&gt;t3&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt;
  &lt;span class="n"&gt;account_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;CASE&lt;/span&gt;
    &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;account_number&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;CASE&lt;/span&gt;
    &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;dt&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You only need to verify the logic of each of the three steps – no need to understand or debug the SQL – and the compiler will generate the production-ready code.&lt;/p&gt;

&lt;p&gt;Let’s compare SQL and SQLazy in a table:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjp1xhdgybwa7elch82j2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjp1xhdgybwa7elch82j2.png" alt="compare SQL and SQLazy " width="800" height="485"&gt;&lt;/a&gt;&lt;br&gt;
This example demonstrates how concisely SQLazy handles “time-interval-based session segmentation” problem – using a segment condition to directly express the rule “start a new group when the gap exceeds 1 hour”, combined with row numbers within each sub-partition. The method precisely maps to the business requirement.&lt;/p&gt;

&lt;p&gt;Try SQLazy online: sqlazy.com (Free to use, signup not required)&lt;br&gt;
SQLazy project repository: github.com/SPLWare/SQLazy&lt;/p&gt;

</description>
      <category>sql</category>
      <category>sqlazy</category>
      <category>programming</category>
      <category>dataanalytics</category>
    </item>
    <item>
      <title>Compilable and Executable Pseudocode (spec) Solves AI Coding Hallucinations</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Thu, 09 Jul 2026 03:08:21 +0000</pubDate>
      <link>https://dev.to/esproc_spl/compilable-and-executable-pseudocode-spec-solves-ai-coding-hallucinations-5ghk</link>
      <guid>https://dev.to/esproc_spl/compilable-and-executable-pseudocode-spec-solves-ai-coding-hallucinations-5ghk</guid>
      <description>&lt;p&gt;&lt;strong&gt;The AI-generated SQL – no one dares to say they fully “understand” it&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;First, look at the following SQL query, which aims to find credit card customers whose transaction amount has increased for three consecutive months:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH ranked AS (
  SELECT customer_id, month, amount,
    LAG(amount,1) OVER(PARTITION BY customer_id ORDER BY month) as prev_1,
    LAG(amount,2) OVER(PARTITION BY customer_id ORDER BY month) as prev_2
  FROM transactions
)
SELECT customer_id, COUNT(*) FROM ranked
WHERE amount &amp;gt; prev_1 AND prev_1 &amp;gt; prev_2
AND prev_1 IS NOT NULL AND prev_2 IS NOT NULL
GROUP BY customer_id HAVING COUNT(*) &amp;gt;= 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The syntax is correct, and it runs. But is the logic sound? No one can say for sure right away.&lt;/p&gt;

&lt;p&gt;This is the daily routine of data analysts and developers today: they give a task to an AI, get dozens of lines of SQL in seconds, copy and paste the code, and it runs. But can they really trust the results? Modern AI can generate runnable SQL, but it is often not fully reliable.&lt;/p&gt;

&lt;p&gt;AI-generated SQL is a black box. The code may look correct, run without errors, and return a clean result, but what it actually does may have nothing to do with what you intended. Worse, you only realize it’s wrong after it has already executed. During the review phase, you have to mentally execute the SQL yourself, effectively redoing the work that the AI was supposed to handle.&lt;/p&gt;

&lt;p&gt;Even worse, even if the accuracy of AI-generated SQL reaches 90%, how can you know that this time they aren’t falling into the remaining 10%? Data-driven decisions should not become a game of chance. You get a SQL query that executes successfully and returns a clean result, but what it actually does is not what the user intended. This type of error cannot be caught by syntax checking. And the verification cost during the code review phase is extremely high: to confirm whether the logic is correct, you have to break down CTEs layer by layer, run them step by step, and compare the intermediate results.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the real problem?&lt;/strong&gt;&lt;br&gt;
In AI-assisted programming, there is a basic workflow: the user writes a clear specification and AI generates runnable code.&lt;/p&gt;

&lt;p&gt;This is not a question of whether AI is “capable” or not, but a hard constraint under current technical conditions. If the user only sketches out a couple of lines of specification, no AI can generate runnable engineering-grade code out of thin air – unless the task is a common, well-defined one such as bubble sort. The actual AI-assisted programming workflow is: the user writes a spec → the AI generates code → the user reviews it → spec the refined → the AI regenerates the code. The spec is both the starting point and a constraint throughout the entire process.&lt;/p&gt;

&lt;p&gt;But even with a perfectly clear specification, AI can still make mistakes – it’s a matter of probability. A poorly defined specification increases the chance that the AI will guess wrong, while a well-defined specification reduces that probability but can never eliminate it. This is because, at its core, AI is making probabilistic predictions. From input to output, it searches the space of possible answers for the most likely one, rather than logically deriving the only correct result.&lt;/p&gt;

&lt;p&gt;Then what can we do? Is there any way to turn a “likely answer” to a “definitive one”?&lt;/p&gt;

&lt;p&gt;The answer is: make the spec compliant enough to compile. Once the spec can be parsed and executed by a compiler, AI no longer needs to “guess” the code. The compiler is deterministic – the same input always produces the same output. No more “works this time, breaks the next”. This isn’t about “reducing the hallucination rate”; it’s about bypassing the stage where hallucinations can occur.&lt;/p&gt;

&lt;p&gt;So, the overall logic chain is clear: AI-assisted programming requires a spec → a well-defined spec can reduce hallucination rate → but it cannot eliminate hallucinations → make the spec compliant enough to compile → use a compiler instead of AI to generate code →hallucinations disappear.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Compilable pseudocode: an advanced form of the specification&lt;/strong&gt;&lt;br&gt;
Traditionally, the requirements document that programmers create is intended for humans. It contains natural language, charts and diagrams, and pseudocode. In the era of AI, the spec can evolve into a new form with two characteristics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Human-readable: No need to be familiar with the target language (SQL, Java, Python), you can grasp the logic at a glance.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Machine-compilable: The specification has a fixed syntactic structure and can be deterministically transformed into final code, without relying on AI guessing.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is “compilable pseudocode”, a more compliant spec. The difference between it and traditional pseudocode is this: traditional pseudocode won’t be compiled – it is intended only for humans to read; compilable pseudocode is both human-readable and machine-readable, so it can be compiled into target code.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9e489e69bw27efpsipl0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9e489e69bw27efpsipl0.png" alt=" " width="799" height="197"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The value of this kind of spec is that it squeezes AI’s “creative space” down to a minimum. AI doesn't need to “understand” the business logic, “design” an implementation plan, and “guess” the correct code – it only needs to translate the uncompliant spec into a compliant one. Translation leaves a far narrower margin for error than “generating code directly from natural language”.&lt;/p&gt;

&lt;p&gt;More importantly, you can verify the correctness of the spec before it is compiled and executed. Each step of the logic can be verified independently, so errors are caught before execution rather than being debugged afterward.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SQLazy’s workflow: The compilable pseudocode&lt;/strong&gt;&lt;br&gt;
SQLazy brings this concept to the field of complex SQL development. Its workflow is essentially compilable pseudocode: developers describe data querying logic step by step in natural language, previewing the intermediate result at each step. Once the logic is confirmed correct, the compiler deterministically generates the SQL.&lt;/p&gt;

&lt;p&gt;Still on the same requirement as above: finding the credit card customers whose transaction amount has increased for three consecutive months. Let’s implement this using SQLazy’s workflow:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fynq2xg00ut01aznwdnh9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fynq2xg00ut01aznwdnh9.png" alt=" " width="798" height="214"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flhzswgnx004rkfx5oneo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flhzswgnx004rkfx5oneo.png" alt=" " width="800" height="427"&gt;&lt;/a&gt;&lt;br&gt;
You don’t need to know SQL – the four steps alone make the logic clear. At every step you can preview the intermediate result:&lt;/p&gt;

&lt;p&gt;sort: Sort rows by customer and month.&lt;/p&gt;

&lt;p&gt;segment: Mark records where the transaction amounts fall, and generate a number for each consecutively rising segment.&lt;/p&gt;

&lt;p&gt;summarize: Count the number of months covered by each segment.&lt;/p&gt;

&lt;p&gt;filter: Select customers whose transaction amounts increase for at least consecutive 3 months (&amp;gt;=3).&lt;/p&gt;

&lt;p&gt;Here is another real-world example. A table stores time intervals for multiple accounts. Each account corresponds to multiple records, and the intervals may overlap. The goal is to merge all overlapping intervals within each account. With SQLazy, this can be implemented in just four steps:&lt;/p&gt;

&lt;p&gt;sort account_id, start_date: Sort rows by account and start date.&lt;/p&gt;

&lt;p&gt;compute end_date[:-1] max as prev_max: Calculate the maximum end date among all rows prior to the current row.&lt;/p&gt;

&lt;p&gt;segment condition start_date &amp;gt; prev_max as gid: Segment rows based on the condition, and assign a group number to each segment.&lt;/p&gt;

&lt;p&gt;summarize start_date min as start_date, end_date max as end_date: Aggregate by account and group number.&lt;/p&gt;

&lt;p&gt;Once the above logic is confirmed correct, the SQLazy compiler automatically generates native SQL – 100% accurate, with zero hallucination risk. If you switch the target database from MySQL to Oracle, you only need to change one option and the compiler automatically adapts to the target SQL dialect.&lt;/p&gt;

&lt;p&gt;SQLazy’s workflow is the spec, and the compiler is the last gate guaranteeing the code quality.&lt;/p&gt;

&lt;p&gt;In this process, the role of AI is also redefined. Conventionally, AI is responsible for generating the final SQL directly from the natural language – an extremely difficult task with a high hallucination rate. With SQLazy, AI is charged with only one thing: translating users’ colloquial, step-by-step descriptions into formal workflow syntax. The former is “decision-making” while the latter is “translation”. Even if the AI’s translation is slightly off, you can spot it at a glance at the workflow level, and the cost of correction is nearly zero.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F86dvr0uerg2wpc46lsya.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F86dvr0uerg2wpc46lsya.png" alt=" " width="800" height="428"&gt;&lt;/a&gt;&lt;br&gt;
Use LLM to convert the natural language into SQLazy syntax&lt;/p&gt;

&lt;p&gt;AI hallucinations in programming are a problem not because AI occasionally makes mistakes, but because of something more fundamental: under the conventional model, AI’s mistakes can only be discovered after execution. By the time you find the result is wrong, the code has already been generated – even executed – making the cost of correction extremely high.&lt;/p&gt;

&lt;p&gt;SQLazy’s compilable pseudocode shifts the moment of discovering errors from after execution to before execution. Humans confirm the logic at the spec level, and the compiler generates the final code. AI’s room for error is squeezed down to the “translation” step alone, where the result can be verified instantly by humans – solving the hallucination problem in AI programming.&lt;/p&gt;

&lt;p&gt;This does not mean programmers are going back to the era of “writing documentation”. Rather, it redivides the labor between humans and AI: humans handle design (describing “what to do” using structured spec); AI handles translation (converting colloquial spec into compliant spec); and the compiler handles execution (guaranteeing a deterministic output).&lt;/p&gt;

&lt;p&gt;The next phase of AI programming is probably not about making AI generate longer code, but about enabling humans to write clearer spec.&lt;/p&gt;

</description>
      <category>sql</category>
      <category>programming</category>
      <category>productivity</category>
      <category>sqlazy</category>
    </item>
    <item>
      <title>SQLazy: Group-based Cumulative Sums</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Fri, 03 Jul 2026 03:35:38 +0000</pubDate>
      <link>https://dev.to/esproc_spl/sqlazy-group-based-cumulative-sums-2el9</link>
      <guid>https://dev.to/esproc_spl/sqlazy-group-based-cumulative-sums-2el9</guid>
      <description>&lt;p&gt;**&lt;/p&gt;

&lt;h2&gt;
  
  
  Problem description
&lt;/h2&gt;

&lt;p&gt;**&lt;br&gt;
Only retain invoiced rows; reset the cumulative amount at each invoiced row&lt;/p&gt;

&lt;p&gt;A business transaction table consists of four fields: ID, Date, Invoiced, and Amount. Each record represents one month, and Invoiced=1 indicates that an invoice was issued for that month.&lt;/p&gt;

&lt;p&gt;The query goal: Return every invoiced month for each ID, where the invoice amount equals the sum of all amounts from the previous invoiced month (exclusive), or the beginning of the table, to the current invoiced month (inclusive).&lt;/p&gt;

&lt;p&gt;Source data:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhigy5cbpuw61ycrvy7lj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhigy5cbpuw61ycrvy7lj.png" alt=" " width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
Expected output: Only retain invoiced rows, where each Amount is the cumulative sum since the previous invoiced row (exclusive).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftoukyfamcnd3368bsvpo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftoukyfamcnd3368bsvpo.png" alt=" " width="799" height="194"&gt;&lt;/a&gt;&lt;br&gt;
Take ID: AAA as an example:&lt;/p&gt;

&lt;p&gt;The first invoice - 2023-03: earlier January (10) + earlier February (15) + current (15) = 40&lt;/p&gt;

&lt;p&gt;The second invoice - 2023-06: Since the previous invoice: April (10) + May (10) + current (10) = 30&lt;/p&gt;

&lt;p&gt;Same for ID: BBB.&lt;/p&gt;

&lt;p&gt;**&lt;/p&gt;
&lt;h2&gt;
  
  
  Step-by-step implementation with SQLazy
&lt;/h2&gt;

&lt;p&gt;**&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyvpa7pxcxzkwfatzwmvu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyvpa7pxcxzkwfatzwmvu.png" alt=" " width="800" height="305"&gt;&lt;/a&gt;&lt;br&gt;
[&lt;a href="https://www.sqlazy.com/?4yy" rel="noopener noreferrer"&gt;Run this example online&lt;/a&gt;]&lt;/p&gt;

&lt;p&gt;Let me walk through each step below:&lt;/p&gt;

&lt;p&gt;Step 1: Sort rows by ID and Date (descending)&lt;/p&gt;

&lt;p&gt;sort id,dt desc&lt;/p&gt;

&lt;p&gt;This step prepares for the subsequent grouping. Once sorted in descending order, each invoiced row and the earlier non-invoiced rows after it are grouped together for cumulative summation.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyno8fvvt4nx3mabdiq2a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyno8fvvt4nx3mabdiq2a.png" alt=" " width="800" height="395"&gt;&lt;/a&gt;&lt;br&gt;
Step 2: Perform a cumulative sum on the Invoiced field to generate group numbers&lt;/p&gt;

&lt;p&gt;compute invoiced cum as grp partition id&lt;/p&gt;

&lt;p&gt;Within each ID partition, perform a cumulative sum on the Invoiced field in the current (descending) order, including the current row. The result is as follows:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchfsden7qeh5i79oka2t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchfsden7qeh5i79oka2t.png" alt=" " width="800" height="396"&gt;&lt;/a&gt;&lt;br&gt;
Step 3: Group and aggregate by ID and grp&lt;/p&gt;

&lt;p&gt;summarize dt max as dt invoiced max as invoiced amount sum as amount group id grp&lt;/p&gt;

&lt;p&gt;dt max: Since rows are sorted in descending order, the largest date within each group corresponds to the invoiced month (i.e., the date of the invoiced row).&lt;/p&gt;

&lt;p&gt;invoiced max: Each group contains at least one row where Invoiced = 1, so the maximum Invoiced value within each group is 1.&lt;/p&gt;

&lt;p&gt;amount sum: Sum all amounts within each group, i.e., the total amount since the invoiced row (inclusive).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzq00o8mmdurxqb53ubet.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzq00o8mmdurxqb53ubet.png" alt=" " width="799" height="228"&gt;&lt;/a&gt;&lt;br&gt;
Step 4: Select the required columns&lt;/p&gt;

&lt;p&gt;derive id dt invoiced amount&lt;/p&gt;

&lt;p&gt;This step only cleans up the output by removing the helper column grp.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F27lg6ftqk9vx8vg20asx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F27lg6ftqk9vx8vg20asx.png" alt=" " width="798" height="228"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Compile the steps into SQL&lt;/p&gt;

&lt;p&gt;Once the above steps are complete, SQLazy’s compiler can generate the equivalent native SQL, without the need to write it manually.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyky3fv2zypqffpqv9bpt.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyky3fv2zypqffpqv9bpt.png" alt=" " width="800" height="453"&gt;&lt;/a&gt;&lt;br&gt;
To generate MySQL SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH t3 AS (
  SELECT
    id,
    grp,
    MAX(dt) AS dt,
    MAX(invoiced) AS invoiced,
    SUM(amount) AS amount
  FROM
    (
      SELECT
        id,
        dt,
        invoiced,
        amount,
        SUM(invoiced) OVER (
          PARTITION BY id
          ORDER BY
            CASE
              WHEN id IS NULL THEN 1
              ELSE 0
            END,
            id ASC,
            CASE
              WHEN dt IS NULL THEN 1
              ELSE 0
            END,
            dt DESC ROWS UNBOUNDED PRECEDING
        ) AS grp
      FROM
        invoice
    ) t2
  GROUP BY
    id,
    grp
)
SELECT
  id,
  dt,
  invoiced,
  amount
FROM
  t3
ORDER BY
  id,
  grp

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You only need to verify the logic of each of the four steps – no need to understand or debug the SQL – and the compiler will generate the production-ready code.&lt;/p&gt;

&lt;p&gt;Let’s compare SQL and SQLazy in a table:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feogb2gbanukiipd6apkn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feogb2gbanukiipd6apkn.png" alt=" " width="799" height="401"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This example demonstrates how naturally SQLazy handles the “event-based cumulative reset” problem – breaking down complex grouping logic into clear steps with a simple sorting trick and a cumulative sum.&lt;/p&gt;

&lt;p&gt;Try SQLazy online: &lt;a href="https://www.sqlazy.com/" rel="noopener noreferrer"&gt;sqlazy.com&lt;/a&gt; (Free to use, signup not required)&lt;br&gt;
SQLazy project repository: &lt;a href="https://github.com/SPLWare/SQLazy" rel="noopener noreferrer"&gt;github.com/SPLWare/SQLazy&lt;/a&gt;&lt;/p&gt;

</description>
      <category>sql</category>
      <category>ai</category>
      <category>sqlazy</category>
      <category>programming</category>
    </item>
    <item>
      <title>Stop trusting AI-generated SQL blindly: Build queries step-by-step with SQLazy</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Fri, 19 Jun 2026 13:29:17 +0000</pubDate>
      <link>https://dev.to/esproc_spl/stop-trusting-ai-generated-sql-blindly-build-queries-step-by-step-with-sqlazy-2k</link>
      <guid>https://dev.to/esproc_spl/stop-trusting-ai-generated-sql-blindly-build-queries-step-by-step-with-sqlazy-2k</guid>
      <description>&lt;p&gt;AI can write runnable SQL statements, but it is often unreliable. SQLazyturns SQL development into a step-by-step, verifiable, and auditable workflow, with a compiler to ensure that the final output is correct.&lt;/p&gt;

&lt;p&gt;Problem: The AI-generated SQL is a black box&lt;br&gt;
We’ve all been in such a situation: throw a complex analytical query in ChatGPT or Claude and get a monster – dozens of lines of SQL code crammed into one lump. And you think: “This might run… but can I trust it?”&lt;/p&gt;

&lt;p&gt;In reality, the AI-generated SQL often fails in these ways:&lt;/p&gt;

&lt;p&gt;Incorrect join logic— Use the incorrect target table, or miss a necessary join condition.&lt;/p&gt;

&lt;p&gt;Aggregation mistake — GROUP BY does not align with the analysis goal, or miss non-aggregated columns.&lt;/p&gt;

&lt;p&gt;Missed filter condition— Miss a subtle business constraint (such as “only count active users”).&lt;/p&gt;

&lt;p&gt;Semantic deviation— What you mean by “revenue” may not be the same as what the model considers the “total amount”.&lt;/p&gt;

&lt;p&gt;Ignored boundary condition—Usually the NULL values, empty sets, and extreme values are elegantly ignored.&lt;/p&gt;

&lt;p&gt;Today, AI can generate runnable SQL. But you never know whether you can trust it. Once the query involves deeply nested window functions and subqueries, it becomes difficult to review, debug, maintain, and migrate.&lt;/p&gt;

&lt;p&gt;Worse still, when the results look off, how do you even begin to fix it?&lt;/p&gt;

&lt;p&gt;You can only manually run it CTE by CTE, inserting SELECT * FROM … statements everywhere just to trace the logic;&lt;/p&gt;

&lt;p&gt;Or tweak the prompt and ask AI to regenerate – only to end up with code that is even harder to read.&lt;/p&gt;

&lt;p&gt;In the end, it takes longer than just writing it yourself.&lt;/p&gt;

&lt;p&gt;This is the true cost of “black-box SQL generation”.&lt;/p&gt;

&lt;p&gt;**&lt;/p&gt;
&lt;h2&gt;
  
  
  How SQLazy works?
&lt;/h2&gt;

&lt;p&gt;**&lt;br&gt;
SQLazy does not generate a huge blob of SQL in one go. Instead, it turns the SQL development into a step-by-step, traceable workflow:&lt;/p&gt;

&lt;p&gt;Use semi-natural language syntax to describe what each step does.&lt;/p&gt;

&lt;p&gt;Verify the logic at each step – Intermediate results are inspectable.&lt;/p&gt;

&lt;p&gt;Let the compiler generate the final SQL.&lt;/p&gt;

&lt;p&gt;The final SQL is generated by the compiler, not the LLM. This means:&lt;/p&gt;

&lt;p&gt;Zero SQL errors from AI hallucination, with 100% correct results.&lt;/p&gt;

&lt;p&gt;The logic is fully auditable.&lt;/p&gt;

&lt;p&gt;The output is production-ready.&lt;/p&gt;

&lt;p&gt;Example: Count the longest consecutive rising days for a stock&lt;/p&gt;

&lt;p&gt;This is a typical analytical query, and difficult to write in pure SQL – some companies use it as the interview question, with fewer than 20% of the candidates getting it right.&lt;/p&gt;

&lt;p&gt;Let’s see how SQLazy tackles this step by step.&lt;/p&gt;

&lt;p&gt;First, describe the logic step by step&lt;/p&gt;

&lt;p&gt;Rather than wrestling with the nested subqueries, SQLazy lets you express the logic as simple, sequential actions:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn8wy00hx8t24r6rm2xzf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn8wy00hx8t24r6rm2xzf.png" alt=" " width="800" height="320"&gt;&lt;/a&gt;&lt;br&gt;
That’s it. One step, one simple action. Here’s the line-by-line explanation:&lt;/p&gt;

&lt;p&gt;Load data from a file, a database, or an in-memory table built into SQLazy. In the IDE or WEB, you can instantly view the result at each step.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjztnkkor2q4t1q8pwf4s.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjztnkkor2q4t1q8pwf4s.png" alt=" " width="800" height="379"&gt;&lt;/a&gt;&lt;br&gt;
Filter records for stock 110838:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8799fpcpj0g2k25ppi1e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8799fpcpj0g2k25ppi1e.png" alt=" " width="799" height="378"&gt;&lt;/a&gt;&lt;br&gt;
Sort records in ascending order by date:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdlr84dx3gr5gbd0aoyqa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdlr84dx3gr5gbd0aoyqa.png" alt=" " width="800" height="376"&gt;&lt;/a&gt;&lt;br&gt;
Mark breaks in the rising trend to separate consecutively rising streaks:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjq9vi9i0tatf1n13y8i.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjq9vi9i0tatf1n13y8i.png" alt=" " width="800" height="377"&gt;&lt;/a&gt;&lt;br&gt;
Count the days in each consecutively rising streak:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc0p5ei9m80ppfjmxj76c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc0p5ei9m80ppfjmxj76c.png" alt=" " width="800" height="378"&gt;&lt;/a&gt;&lt;br&gt;
Get the longest streak:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx5u8yl9pxjd3szo8coc7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx5u8yl9pxjd3szo8coc7.png" alt=" " width="798" height="133"&gt;&lt;/a&gt;&lt;br&gt;
The logic is crystal clear. Anyone can understand what the query does without being a SQL expert.&lt;/p&gt;

&lt;p&gt;What’s more, you can execute each step and inspect the intermediate result. For example, after the records are segmented in step 3, a new column called NoRisingDays is generated, showing the group number for each consecutive streak. If something looks wrong, you can fix them immediately – no need to wait the entire query finish and guess where things went wrong.&lt;/p&gt;

&lt;p&gt;Then generate SQL with the compiler&lt;/p&gt;

&lt;p&gt;SQLazy automatically compiles the above steps into the native SQL dialect for your target database. It currently supports MySQL, PostgreSQL, and Oracle, with Snowflake and BigQuery support on the roadmap.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;s2&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;CODE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CL&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;CODE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CL&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;t_3&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;CODE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;110838&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ContinuousDays&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;max_ContinuousDays&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;NoRisingDays&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;ContinuousDays&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;CODE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;CL&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;col__4&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;NoRisingDays&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;s2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;LAG&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DT&lt;/span&gt; &lt;span class="k"&gt;ASC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;col__4&lt;/span&gt;
      &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;s2&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;sub__5&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;s3&lt;/span&gt;
  &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;NoRisingDays&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;s4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The resulting SQL is difficult to understand, review, debug, and modify. But the SQLazy workflow is clear, easy to review and audit. As long as you follow the workflow correctly, the final SQL is guaranteed to be reliable.&lt;/p&gt;

&lt;p&gt;Here are the fundamental differences between SQLazy and ordinary AI SQL generators:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcdvoyda22kr3y35lua44.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcdvoyda22kr3y35lua44.png" alt=" " width="800" height="484"&gt;&lt;/a&gt;&lt;br&gt;
Here are my honest impressions from using SQLazy to run complex queries:&lt;/p&gt;

&lt;p&gt;Advantages:&lt;/p&gt;

&lt;p&gt;The result at each step is inspectable. Before, trying to write complex SQL meant imagining the intermediate results – they only exist in your head. Now you can inspect the actual data table after each step is executed, and catch and fix errors immediately. At last, no more black boxes, and what a relief!&lt;/p&gt;

&lt;p&gt;Logic is broken down into multiple steps, which naturally serve as document. Once a workflow is finished, if requirements change three months later, there is no need to re-analyze dozens of lines – just find the relevant step and modify it. And if someone else takes over, reading through steps is so much faster than deciphering the SQL.&lt;/p&gt;

&lt;p&gt;Debugging becomes significantly faster. Once, I wrote the wrong grouping condition at the 4th step, executed the code, spotted an extra row in the intermediate table that shouldn’t have been there, and pinpoint the error immediately. Before, I would have to run the entire SQL, add debug fields everywhere, then run it again… back and forth, over and over.&lt;/p&gt;

&lt;p&gt;Zero cross-database hassle. Write your logic once and generate SQL for both MySQL and Oracle – no manual dialect translation needed.&lt;/p&gt;

&lt;p&gt;Notes:&lt;/p&gt;

&lt;p&gt;Learning cost exists. You need to adapt to the “stepwise way of thinking” – resisting the urge to reach for window functions right away. The first couple of uses may feel slower, but the thinking becomes clearer once you settle into it.&lt;/p&gt;

&lt;p&gt;Not necessary for simple scenarios. It is faster to write a 3-line SELECT in SQL. SQLazy is better suited for the complex scenarios where your brain starts to struggle to keep up.&lt;/p&gt;

&lt;p&gt;SQLazy is not an almighty tool. Note the unsupported features and scenarios.&lt;/p&gt;

&lt;p&gt;**&lt;/p&gt;

&lt;h2&gt;
  
  
  Try SQLazy
&lt;/h2&gt;

&lt;p&gt;**&lt;br&gt;
Web Version (signup not required) &lt;a href="https://sqlazy.com" rel="noopener noreferrer"&gt;Free to use&lt;/a&gt; , ideal for quick trials.&lt;/p&gt;

&lt;p&gt;Desktop IDE: Best suited for daily work and large dataset processing, with unlimited local debugging.&lt;/p&gt;

&lt;p&gt;Repository address：&lt;a href="https://github.com/SPLWare/SQLazy" rel="noopener noreferrer"&gt;https://github.com/SPLWare/SQLazy&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The examples directory in the project contains step-by-step solutions to multiple real-world SQL problems, including the “Count longest consecutive days of stock price gains” problem demonstrated above, as well as scenarios such as session analysis and financial indicator calculations.&lt;/p&gt;

&lt;p&gt;Shortcomings of SQLazy&lt;/p&gt;

&lt;p&gt;Recursive queries are not supported (already on the roadmap).&lt;/p&gt;

&lt;p&gt;Very old databases (such as MySQL 5.5) are not supported.&lt;/p&gt;

&lt;p&gt;SQLazy itself is not open-source, but all example workflows and documentation are on GitHub under the MIT license.&lt;/p&gt;

&lt;p&gt;Expected feedback&lt;/p&gt;

&lt;p&gt;If you have ever encountered an analysis scenario where writing logic in pure SQL is overly convoluted, let us know and we’ll try to rewrite it with SQLazy – so you can see whether the workflow is genuinely easier to read than plain SQL.&lt;/p&gt;

&lt;p&gt;When you use AI to write SQL, which is your biggest pain point – accuracy, maintainability, or trustworthiness?&lt;/p&gt;

&lt;p&gt;When it comes to the “step-by-step” approach to SQL development, what do you think is the biggest weakness?&lt;/p&gt;

</description>
    </item>
    <item>
      <title>SPL practice: solve space-time collision problem of trillion-scale calculations in only three minutes</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Mon, 30 Mar 2026 08:37:07 +0000</pubDate>
      <link>https://dev.to/esproc_spl/spl-practice-solve-space-time-collision-problem-of-trillion-scale-calculations-in-only-three-1ko5</link>
      <guid>https://dev.to/esproc_spl/spl-practice-solve-space-time-collision-problem-of-trillion-scale-calculations-in-only-three-1ko5</guid>
      <description>&lt;h2&gt;
  
  
  Problem description
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Definition of space-time collision&lt;/strong&gt;&lt;br&gt;
Dataset A contains the time and space information of n source objects A1, …, An, and each piece of information includes three attributes: ID (iA), location (lA) and time (tA). It can be assumed that the same Ai will not appear twice in A at the same time, or in other words, no two pieces of information have the same iA and tA. Dataset B, which has the same structure as A, contains m target objects B1, …, Bm (with the similar attributes iB, lB, tB) that are to be confirmed whether they collided with A. Likewise, it can be assumed that Bi will not appear twice in B at the same time.&lt;/p&gt;

&lt;p&gt;This article involves many set-oriented operations. Instead of using the term “record” to refer to information of data set, we use the set-related term “member”.&lt;/p&gt;

&lt;p&gt;Group the dataset A by iA to get n subsets, and still name these subsets A1…, An. And correspondingly, split the dataset B into m subsets B1…, Bm. If ‘a’ belongs to the subset Ai and ‘b’ belongs to the subset Bj, and a.lA=b.lB and |a.tA-b.tB|&amp;lt;=1 minute (this time length can be changed), then we consider that ‘a’ collides with ‘b’, and that object Ai and object Bj collided once.&lt;/p&gt;

&lt;p&gt;Rule 1: The number of collisions of each Ai member is counted as once at most, which means that if a collides with b1 and b2, we consider that only one collision occurs between Ai and Bj.&lt;/p&gt;

&lt;p&gt;Rule 2: The member of Bj that once collided will no longer be identified as having had a collision. For example, if b collides with both a1 and a2, and assume a1.tA&amp;lt;a2.tA, then only the collision between a1 and b is identified as collision, and the collision between a2 and b is no longer identified as collision.&lt;/p&gt;

&lt;p&gt;Objective: find the top 10 objects Bj with the highest similarity for each Ai.&lt;/p&gt;

&lt;p&gt;The formula for calculating the similarity ‘r’ is: r(Ai, Bj)=E/U.&lt;br&gt;
where, the molecule E refers to the total number of collisions between Ai and Bj calculated based on the above rules;&lt;/p&gt;

&lt;p&gt;The denominator U refers to the total number of members of Ai and Bj after deduplication, which can be calculated using |Ai|+|Bj|-E’, where E’ refers to the number of Bj members that collide with a certain Ai member.&lt;/p&gt;
&lt;h2&gt;
  
  
  Data structure and data scale
&lt;/h2&gt;

&lt;p&gt;Dataset A&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fzt5ehzlwz8v691auyg4a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fzt5ehzlwz8v691auyg4a.png" alt=" "&gt;&lt;/a&gt;&lt;br&gt;
Dataset B&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fnh80lfg2efxc7q0yh349.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fnh80lfg2efxc7q0yh349.png" alt=" "&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;tA and tB are accurate to the second, and the time span is 30 days.&lt;/p&gt;

&lt;p&gt;The total number of records of Dataset A is 21 million rows, with daily addition of approximately 700,000 rows.&lt;/p&gt;

&lt;p&gt;The total number of records of Dataset B is 15 million rows, with daily addition of approximately 500,000 rows.&lt;/p&gt;

&lt;p&gt;The scale of n (the number of Ai) is 2.5 million, and that of m (the number of Bj) is 1.5 million.&lt;/p&gt;

&lt;p&gt;The number of locations is 10, which means the possibility of values of lA and lB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hardware environment and expectation&lt;/strong&gt;&lt;br&gt;
We hope to obtain the result within 15 minutes on a 40C128G server.&lt;/p&gt;

&lt;p&gt;The amount of data is not large and the data can be fully loaded into in-memory database. However, the calculation process is complicated. Since it is difficult to work out in SQL alone, external program (Java or Python) is needed. As a result, the overall performance is very low, and it took more than two hours to perform this task.&lt;/p&gt;
&lt;h2&gt;
  
  
  Problem analysis
&lt;/h2&gt;

&lt;p&gt;n is 2.5 million and m is 1.5 million. If we calculate the similarity of each pair of Ai and Bj according to the above-mentioned definition, we have to calculate 2.5 million * 1.5 million = 3.75 trillion pairs. Even if each CPU can calculate the similarity of one pair of members in only one microsecond (in fact, such complex set-oriented calculations cannot be worked out quickly), it would take several hours on the current multi-CPU environment. Obviously, this hard traversal method is not feasible.&lt;/p&gt;

&lt;p&gt;For a pair of Ai and Bj, according to the similarity calculation formula, we know that if there is no collision between the members of Ai and Bj, then E=0, and the similarity is also 0, and hence there is no need to perform the subsequent TopN calculation.&lt;/p&gt;

&lt;p&gt;Assume that the data in A are evenly distributed and the average number of members in each Ai is less than 10 (21 million/2.5 million), and that the Bj that collides with Ai must satisfy the condition |tA-tB|&amp;lt;=1 minute, if the data in B are evenly distributed, then there are approximately 350 (500,000/1440) members per minute on average. The 10 members of Ai will collide with a maximum of 350210=7000 B members (between one minute before and after each A member). The average number of members of Bj is also 10 (15 million/1.5 million), and the average number of B members in Bj is only 7000/10=700 after distributing 7000 B members into Bj. In other words, there are only 700 Bj that have the similarity not equal to 0 with Ai on average, which is much smaller than the total number of Bj (1.5 million, a difference of over 2,000 times). Considering the condition lA=lB, if all objects are also evenly distributed (which is unlikely), then the average number of Bj that have the similarity not equal to 0 with Ai can be further reduced by 10 times (10 locations).&lt;/p&gt;

&lt;p&gt;Based on the above information, we design the following algorithms:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;For each Ai, find the set of members of B that may collide with each member of Ai based on the time and location conditions, and denote the set as B’:
&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;B’=Ai.conj(B.select(tB&amp;gt;=tA-60 &amp;amp;&amp;amp; tB&amp;lt;=tA+60 &amp;amp;&amp;amp; lA==lB))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Note that during the calculation of B’, the corresponding B members will be filtered for each Ai member, so the members of both Ai and B may appear repeatedly in B’.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;For each Ai, we need to filter B with tB to obtain B’. If B is sorted by tB in advance, the binary search can be used to speed up. Moreover, we need to add the tA attribute of Ai to facilitate subsequent calculations (since lA and lB are always the same and iA is a fixed value for Ai, there is no need to add these two attributes). The calculation of B’ can be changed as follows:
&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;B’=Ai.conj(B.select@b(tB&amp;gt;=tA-60 &amp;amp;&amp;amp; tB&amp;lt;=tA+60).select(lA==lB).derive(Ai.tA))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;to determine the set composed of member pairs consisting of Ai members and B’j members that have collided (it is also the set of the member pairs consisting of Ai members and Bj members that have collided). In this equation, the member of Ai is represented as the field tA, and the member of Bj is represented as the field tB.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;If A is sorted by iA and tA in advance, and the members of Ai after grouping are also in order by tA, then likewise, the members of B’ will be ordered by tA, and the members of B’j will also in order by tA. In this way, the member with the minimum tA in each grouped subset of B’j.group(tB) will definitely be the first member. Therefore, the calculation of A’j can be simplified as:
&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A’j=B’j.group@1(tB)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;ol&gt;
&lt;li&gt;According to rule 1 that the number of collisions of each Ai member is counted as once at most, and based on the assumption mentioned at the beginning of this article that the same Ai will not appear twice in A at the same time, we just need to deduplicate tA, that is, the numerator can be calculated as follows:
&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;E=A’.icount(tA)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;ol&gt;
&lt;li&gt;In the formula U=|Ai|+|Bj|-E’ for calculating the denominator,&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A’j is the set composed of member pairs consisting of A members and B members, and these member pairs have been identified as having had a collision. Moreover, since the Bj members are already deduplicated (group@1(tB) means taking only one member from each grouped subset), the member of Bj will not appear repeatedly in A’j. Therefore, the members of A’j can correspond to the collided members of Bj one to one, and the equation E’=|A’j| holds.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;|Ai| and |Bj| can be calculated and saved in advance by grouping A and B by iA and iB respectively. Since the amount of data is not large, the results can all be stored in memory.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Finally, according to the formula E/U, we can get the similarity ‘r’ between Ai and Bj. What remains is the common task of calculating TopN for the similarity results.&lt;/p&gt;

&lt;p&gt;**Further optimization&lt;br&gt;
**8. In the step 1 above, to search the full data of dataset B for each a, there is still a certain amount of computation with binary search (2*log 15 million means about 50 comparisons). Considering both the number of minutes and the number of locations are not large (30-day time span, 1440 minutes per day and 10 locations mean only around 400,000), which can be fully held in memory, we can use aligned sequence to directly locate.&lt;/p&gt;

&lt;p&gt;When computing, let&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;G=B.align@a(30*1440,tB\60+1).(~.align@a(10,lB))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because there are two dimensions: time and location, we also use a two-layer aligned sequence. First, divide B into 43200 (30*1440) groups by the minute sequence number tB\60+1, and then divide the sub-group members into 10 groups by the location sequence number lB. For the tA of a certain member in Ai, we can use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;G’=G.m(tA\60)(lA) | G(tA\60+1)(lA) | G.m(tA\60+2)(lA)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to quickly and roughly screen out a small superset of B’. The time difference between tB and tA of the member in the superset is no more than 2 minutes (the difference between the minute sequence numbers at which tA and tB are located is not greater than 1), thereby filtering out a large number of B members that are impossible to collide on the time dimension. In the small superset, since there may be a small number of members whose difference between tB and tA is greater than 1 minute, we need to use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;G’.select@b(tB&amp;gt;=tA-60 &amp;amp;&amp;amp; tB&amp;lt;=tA+60)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to screen out an exact B’, which can significantly reduce the computing amount.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;In addition, when calculating U, we need to search for the corresponding |Ai| by iA. If iA is continuous integer, we can also find it directly by location to avoid search action, that is:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nA=A.groups@n(iA;count(1)).(#2)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now, |Ai|=nA(iA). B can be processed in the same way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Practice process
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Prepare the test data&lt;/strong&gt;&lt;br&gt;
We directly prepare the data that are already converted to sequence number. Assume that the time span is 30 days and the enumeration number of locations is 10, the simulated data script is as follows:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdv4pjbgjow0a6tmeluam.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdv4pjbgjow0a6tmeluam.png" alt=" "&gt;&lt;/a&gt;&lt;br&gt;
In A1, K represents the number of days, nA represents the total data amount of dataset A, nB represents the total data amount of dataset B, t represents the number of seconds of 30 days, and LN represents the enumeration number of locations.&lt;/p&gt;

&lt;p&gt;In A2 and A3, the composite tables A.ctx and B.ctx are created respectively, and the data of randomly generated data sets A and B are exported to the two composite tables respectively.&lt;/p&gt;

&lt;p&gt;tA (tB) refers to the number of seconds elapsed from the starting time point. For example, if the starting time point is 2023-08-23 00:00:00, then the value corresponding to the time point 2023-08-23 00:01:01 is 61.&lt;/p&gt;

&lt;p&gt;In A2, the @p option is used to create the composite table, indicating that the first field ‘iA’ is used as the segmentation key. During parallel computing, the composite table needs to be segmented. Since the records of the same ‘iA’ cannot be assigned to two segments, we use the @p option to ensure this during the segmentation of composite table.&lt;/p&gt;

&lt;p&gt;Special attention should be paid to different sort orders when saving A and B. A is sorted by iA and tA (the step 4 of ‘Problem analysis’), while B is sorted by tB and iB (the step 2 of ‘Problem analysis’). In this way, we can read the ordered data directly in subsequent operations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Computing script&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fw4lzfamwlm6qh9d035lf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fw4lzfamwlm6qh9d035lf.png" alt=" "&gt;&lt;/a&gt;&lt;br&gt;
A1: K=30 represents the number of days of the time span to be counted; LN=10 represents the enumeration number of locations;&lt;/p&gt;

&lt;p&gt;A4, A5: correspond to the step 7 of ‘Problem analysis’. Group the members by the ID of datasets A and B respectively, and count the number of members of each Ai and Bj, and store the result as sequence;&lt;/p&gt;

&lt;p&gt;A6: correspond to the step 8 of ‘Problem analysis’. Align and group the members of dataset B by minute, and then align and group the sub-group members by location to calculate the aligned sequence mentioned above;&lt;/p&gt;

&lt;p&gt;A8: correspond to the steps 1, 2 and 8 of ‘Problem analysis’. Divide the dataset A into several groups of Ai by iA, and loop through each group of Ai to obtain the corresponding B’; here we use ‘news’ instead of ‘conj’, which eliminates the derive action, and can obtain the same result. In addition, we also add iA to facilitate subsequent search for |Ai|;&lt;/p&gt;

&lt;p&gt;A9: correspond to the steps 3 and 4 of ‘Problem analysis’. The method of calculating A’j:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;B’.group(iB).group@1(tB)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;can be simplified as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;B’.group@1(iB,tB)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A10: correspond to the steps 6 and 7 of ‘Problem analysis’. The result of deduplicating and counting tA by Aj’ is the molecule E, which is equivalent to the step 5 of ‘Problem analysis’. By adding the previously calculated results |Ai| and |Bi| and then subtracting the number of records in the current group (i.e. E’), we can get the denominator U. Finally, calculate the top 10 records based on the similarity result ‘r’.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Convert to sequence number and restore&lt;/strong&gt;&lt;br&gt;
Convert the ID, location and time to sequence number (the sequence-numberization of time is to calculate the number of seconds from the start time to the current time). The data structure after conversion is as follows:&lt;/p&gt;

&lt;p&gt;Dataset A&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0pkbpiatfzf8by703jem.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0pkbpiatfzf8by703jem.png" alt=" "&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Dataset B&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2tqiq1mypc9qqzsohkgq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2tqiq1mypc9qqzsohkgq.png" alt=" "&gt;&lt;/a&gt;&lt;br&gt;
The creation of data with the above code is based on the premise that the members of all fields are already converted to the above data structure. Therefore, in practice, we need to perform data conversion and organization first, and then restore the data after calculation. For details, refer to the method described in SPL Practice: integerization during data dump . Since the said method is not the focus of this article, we won’t describe it again here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Actual effect
&lt;/h2&gt;

&lt;p&gt;When the total time span is 30 days (the data volume of data set A is 21 million rows, and that of data set B is 15 million rows), computing in SPL on a single machine (8C64G) takes 161 seconds including exporting all results to CSV file.&lt;/p&gt;

&lt;p&gt;In fact, achieving this performance requires using a small number of column-wise computing options of SPL Enterprise Edition. Since the use of such options doesn’t involve principle analysis, we do not describe it in detail in this article.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Postscript&lt;/strong&gt;&lt;br&gt;
This article discusses a typical object counting problem, which generally has the following characteristics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Count the number of objects that satisfy a certain condition.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The number of objects is very large, but the amount of data involved in each object is not large.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The condition is very complex, usually also related to the order, and requires some steps to determine.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Normally, solving such problem needs to sort the data by object. However, since the amount of data involved in this task is very small, the optimization of storage becomes unimportant. The key to solving this task is to provide powerful set-oriented computing ability, especially the ability to compute the ordered set. For example, the data type should be able to support the set of sets so that the grouped subsets can be retained without having to aggregate like SQL. Moreover, the two-layer aligned sequence should be supported, allowing us to access the members of set by location and, the ordered grouping functionality should be provided.&lt;/p&gt;

</description>
      <category>productivity</category>
      <category>devops</category>
      <category>opensource</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Ditch 10,000 Intermediate Tables—Compute Outside the Database with Open-Source SPL</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Fri, 13 Feb 2026 08:02:44 +0000</pubDate>
      <link>https://dev.to/esproc_spl/ditch-10000-intermediate-tables-compute-outside-the-database-with-open-source-spl-4j1k</link>
      <guid>https://dev.to/esproc_spl/ditch-10000-intermediate-tables-compute-outside-the-database-with-open-source-spl-4j1k</guid>
      <description>&lt;p&gt;Intermediate tables are data tables in databases specifically used to store intermediate results generated from processing the original data – which is why they are so named. They are summary tables usually created for speeding up or facilitating the front-end queries and analysis. For some large organizations, years of accumulation results in tens of thousands of intermediate tables, which is an incredible number, in their databases, bringing great trouble to database operation and usage.&lt;/p&gt;

&lt;p&gt;The large number of intermediate tables occupies too much database storage space, putting enormous pressure on storage capacity and increasing demand for capacity expansion. But database space is expensive and capacity expansion is exceedingly costly. Moreover, often there are restrictions on the expansion. It is not a good choice to cost you an arm and a leg with storing intermediate tables also because too many of them reduce database performance. Intermediate tables are not created out of thin air. Rather, they are generated from the original data through a series of computations that consume database computing resources. Sometimes, a lot of intermediate tables are produced during a computation. This consumes a large number of resources, and in serious cases, can slow down queries and transactions.&lt;/p&gt;

&lt;p&gt;Why are there so many intermediate tables? Below are main reasons:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. More than one step is needed to get the final result&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The original data table needs to undergo complicated computations before being displayed in a report. It is hard to accomplish this with one SQL statement but with multiple, continuous SQL statements. One statement generates an intermediate result that will be used by the next statement.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Long wait time in real-time computations&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For data-intensive and compute-intensive tasks, the wait time will be extremely long. So, report developers choose to run batch tasks at night and store results in intermediate tables. It is much faster to perform queries based on the intermediate tables.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Diverse data sources in a computation&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Files, NoSQL and Web service almost do not have computing abilities. Data originated from them needs to be computed using the database’s computing ability. With a mixed computation between such data and data stored in the database particularly, the traditional approach is to load the external data into the database and store it as intermediate tables.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Intermediate tables are hard to get rid of&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;As databases uses flat structure to arrange tables, it is very likely that one intermediate table is shared by multiple queries after it is created. Deleting it for a finished query could affect other queries. Worse still, you cannot know exactly which applications are using this intermediate table. This makes deletion impossible, not because you do not want to get rid of it, but because you dare not do it. The consequence is that tens of thousands of intermediate tables are accumulated in the database over time.&lt;/p&gt;

&lt;p&gt;But why we use databases to store intermediate data? According to the above causes of intermediate tables, the direct aim for storing intermediate data in the database as intermediate tables is to employ the database’s computational ability. The intermediate data will be further computed for subsequent use, and sometimes the computation is rather complicated. Now only databases (which are SQL-driven) have relatively convenient computing ability. Other storage formats like files have their own merits (high I/O performance, compressible and easy to be parallelly processed) though, they do not have computing abilities. If you try to perform computations based on files, you need to hardcode them in applications. That is far less convenient than using SQL. So, to make use of databases’ computing abilities is the essential reason of the existence of intermediate tables.&lt;/p&gt;

&lt;p&gt;In some sense intermediate data is necessary. But consuming a huge amount of database resources in order to get only more computing ability is obviously not a good strategy. If we can enable files to have equal computing ability and store intermediate data in the outside-database file system, then problems related to intermediate tables will be solved and databases will be unburdened from or relieved of overload.&lt;/p&gt;

&lt;p&gt;The open-source SPL can help to turn it into reality.&lt;/p&gt;

&lt;p&gt;SPL is an open-source structured data computation engine. It can process data directly based on files, giving files the computing ability. It is database-independent, offers specialized structured data objects and a wealth of class libraries for handling them, possesses all-around computational capability, and supports procedural control that makes it convenient to implement complex computing logics. All these features qualify SPL to replace databases in handling intermediate data and subsequent data processing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2uyu0kl9g9kwewbq0vvr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2uyu0kl9g9kwewbq0vvr.png" alt=" " width="800" height="389"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  SPL file-based computations
&lt;/h2&gt;

&lt;p&gt;SPL can perform computations directly based on files like CSV and Excel and multilevel data JSON and XML. It is convenient to read and handle them in SPL. We can store intermediate data in one of those file formats and handle it in SPL. Below are some basic computations:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F61n0jtodog3k4tmlc2l2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F61n0jtodog3k4tmlc2l2.png" alt=" " width="800" height="335"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On top of native syntax, SPL even offers supports of SQL92 standard, allowing programmers familiar with SQL to query files directly in SQL.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$select * from d:/Orders.csv where Client in ('TAS','KBRO','PNS')
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Support of complicated WITH clause:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$select t.Client, t.s, ct.Name, ct.address from
(select Client ,sum(amount) s from d:/Orders.csv group by Client) t
left join ClientTable ct on t.Client=ct.Client
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SPL has the edge on handling multilevel data like JSON and XML. To perform computations based on orders data of JSON format, for instance:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fy3pohvqwg9bpkyjzav32.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fy3pohvqwg9bpkyjzav32.png" alt=" " width="800" height="341"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The SPL implementation is concise compared with that in other JSON libraries (like JSONPath).&lt;/p&gt;

&lt;p&gt;SPL also allow users to query the JSON data directly in SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$select * from {json(file("/data/EO.json").read())}
where Amount&amp;gt;=100 and Client like 'bro' or OrderDate is null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SPL is particularly suitable for handling complex computing logics with its agile syntax and procedural control ability. To count the longest continuous days when the price of a stock rises based on stock records of txt format, for instance, SPL has the following code:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0b4vreozgu23v2zpm4aw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0b4vreozgu23v2zpm4aw.png" alt=" " width="800" height="117"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One more instance. To list the latest login interval for each user according to user login records of CSV format:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fr0fw3aqpbd6hdl9r8s0f.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fr0fw3aqpbd6hdl9r8s0f.png" alt=" " width="800" height="174"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Such computing tasks are hard to code even in SQL in databases. Yet they become easy when handled in SPL.&lt;/p&gt;

&lt;p&gt;The outside-database computing ability SPL supplies is an effective solution to problems triggered by too many intermediate tables in databases. Storing intermediate data in files releases database space resources, reduces demand for database expansion and makes database management more convenient. The outside-database computations do not take up database computing resources, and the unburdened database will be able to better serve other transactions.&lt;/p&gt;

&lt;h2&gt;
  
  
  High-performance file formats
&lt;/h2&gt;

&lt;p&gt;Text files are commonly used data storage format. They are general-purpose and easy to read, but, at the same time, they have extremely bad performance. Traditionally, text-based computations are hard to have satisfactory performance.&lt;/p&gt;

&lt;p&gt;Text characters cannot be computed directly. They need to be transformed to in-memory data types like integers, real numbers, dates and strings to be able to be processed. Yet text parsing is extremely complicated and takes exceptional long CPU time. Generally, hard disk reading takes up most of the time in accessing data on external storage, and text files’ performance bottle usually happens in the phase of data handling by CPU. Because of too complicated parsing, it is probably that the CPU time is greater than the hard disk reading time (especially with the high-performance SSD). So, text files are usually not used to process big data when high performance is demanded.&lt;/p&gt;

&lt;p&gt;SPL provides two high-performance binary storage formats – bin file and composite table. A bin file uses the binary format, is compressed (to occupy less space and allow fast retrieval), stores data types (to enable faster retrieval without parsing), and supports the double increment segmentation technique to divide an append-able file, which facilitates parallel processing in an effort to further increase computing performance.&lt;/p&gt;

&lt;p&gt;The composite table is a file storage format SPL uses to provide column-wise storage and indexing mechanism. It displays great advantage in handling scenarios where only a very small number of columns (fields) is involved. A composite table is equipped with the min-max index and supports double increment segmentation technique, letting computations to both enjoy the advantages of column-wise storage and be more easily parallelly processed to have better performance.&lt;/p&gt;

&lt;p&gt;The two high-performance file formats are convenient to use, and have basically the same uses as text files. To read a bin file and compute it, for instance:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2kl96innunss4o0fshs.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2kl96innunss4o0fshs.png" alt=" " width="800" height="169"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;When the size of data to be processed is large, SPL can use cursor to perform batch retrieval and multi-CPU-based parallel processing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;=file("/data/scores.btx").cursor@bm()
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When using files to store data and no matter which format the original data uses, they need to be at least converted to the binary format (like bin file) to get more advantages in both space usage and computing performance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ease of management
&lt;/h2&gt;

&lt;p&gt;Moving intermediate data out of database to file system can not only reduce database workload but make the data extremely easy to manage. Files can be stored in operating system’s tree-structure directories. This makes them convenient to use and manage. It is neat and tidy to place intermediate tables used by different systems and modules in separate directories. This completely eliminates shared reference and thus the long-standing issue of tight coupling between systems and modules due to messy use of intermediate tables in the database. Now intermediate tables can be safely deleted without any harmful effects when corresponding modules are not used any more.&lt;/p&gt;

&lt;h2&gt;
  
  
  Support of diverse data source
&lt;/h2&gt;

&lt;p&gt;In addition to the file sources, SPL can connect to and retrieve data from dozens of other data sources as well as perform mixed computations between different sources.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F565k2pkzys0djyikeuof.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F565k2pkzys0djyikeuof.png" alt=" " width="800" height="348"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After intermediate data is stored in files, we face cross-data-source computations when trying to perform full-data queries between the file and the database holding the real-time data. It is convenient to implement these T+0 queries in SPL:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwvqhw9i6ua0ne2mwr55o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwvqhw9i6ua0ne2mwr55o.png" alt=" " width="800" height="326"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Ease of integration
&lt;/h2&gt;

&lt;p&gt;SPL provides standard JDBC driver and ODBC driver for invocation by an application. For a Java program, the SPL code can also be integrated into it as an embedded computing engine, enabling the latter to have the ability to handle intermediate data.&lt;/p&gt;

&lt;p&gt;Sample of invoking SPL code through JDBC:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;…
Class.forName("com.esproc.jdbc.InternalDriver");
Connection conn =DriverManager.getConnection("jdbc:esproc:local://");
Statement st = connection.();
CallableStatement st = conn.prepareCall("{call splscript(?, ?)}");
st.setObject(1, 3000);
st.setObject(2, 5000);
ResultSet result=st.execute();
…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SPL is interpreted execution and naturally supports hot-swap. Data computing logics written in SPL and their modification, operation and maintenance take effect in real-time without the need of restarting the application, making programs’ development, operation and maintenance convenient and efficient.&lt;/p&gt;

&lt;p&gt;With SPL that offers outside-database computational capability, we can transfer intermediate tables to files, getting rid of the numerous of them from databases. This helps to relieve databases of overload and make it faster, more flexible and more scalable.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>database</category>
      <category>dataengineering</category>
      <category>opensource</category>
    </item>
    <item>
      <title>The Game-Changer Breaking Data Lake's Impossible Triangle</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Wed, 04 Feb 2026 08:11:55 +0000</pubDate>
      <link>https://dev.to/esproc_spl/the-game-changer-breaking-data-lakes-impossible-triangle-c1c</link>
      <guid>https://dev.to/esproc_spl/the-game-changer-breaking-data-lakes-impossible-triangle-c1c</guid>
      <description>&lt;h2&gt;
  
  
  A brief introduction to data lake
&lt;/h2&gt;

&lt;p&gt;Let’s start with data warehouse. A data warehouse is a subject-oriented data management system that aggregates data from different business systems and is intended for data query and analysis. As data expands and the number of business systems increases, data warehousing becomes necessary. In order to meet the business requirements, the raw data needs to be cleansed, transformed and deeply prepared before being loaded into the warehouse. Answering the existing business questions is the data warehouse’s core task. Those questions must be already defined.&lt;/p&gt;

&lt;p&gt;But what if a business question is not defined (which is potential data value)? According to the data warehouse’s rule, a business question is asked first and then a model is built for it. The chain of identifying, raising and answering questions thus becomes very long. On the other hand, the data warehouse, as it stores highly prepared data, has to obtain desired data by processing the raw data when the new question requires fine data granularity. This is extremely cost-ineffective. If there are many such new questions, a query process will be overburdened.&lt;/p&gt;

&lt;p&gt;So, in the context of this background, the data lake was born. It is a technology (or strategy) intended to store and analyze massive amounts of raw data. It enables to load as much raw data as possible into the data lake while keeping the highest fidelity as possible in storing it, and, in theory, extracting any potential data value based on full data. Speaking of this, the data lake’s two roles are absolutely obvious. One is data storage because the data lake needs to keep all raw data. The other is data analysis, which, from the technical point of view, is data computing, or the value extraction process.&lt;/p&gt;

&lt;p&gt;Let’s look at the data lake’s performance in the two aspects.&lt;/p&gt;

&lt;p&gt;The data lake stores full raw data, including structure data, semi-structured data and unstructured data, in its original state. The capacity of storing massive and diverse data is thus the data lake’s essential feature, which is different from the data warehouse that often uses databases to store structured data. Besides, loading data into the lake as early as possible helps fully extract value from association of differently themed data and ensure data security and integrity.&lt;/p&gt;

&lt;p&gt;The good news is that the massive raw data storage needs can be fully met thanks to the great advance of storage and cloud technologies. Enterprises can choose self-built storage cluster or the storage service provided by a cloud vendor to deal with their business demands.&lt;/p&gt;

&lt;p&gt;But, the toughest nut to crack is data processing! The data lake stores various types of data and each needs to be processed differently. The central and the most complicated part is structured data processing. With both historical data and newly generated business data, data processing mainly focuses on structured data. On many occasions, computations of semi-structured data and unstructured data will eventually be transformed to structured data computations.&lt;/p&gt;

&lt;p&gt;At present, SQL-based databases and related technologies, which are also the abilities data warehouses have, dominate structured data processing field. In other words, the data lake depends on data warehouses (databases) to compute structured data. That is nearly all data lake products do. Building the data lake to store all raw data and then the data warehouse to add data processing capability catering to business needs of enterprises. As a result, data in the lake needs to be loaded to the data warehouse again through ETL. An advanced approach automates the process to some degree. The approach identifies data in the lake that needs to be loaded to the warehouse and performs the loading while the system is idle. This is the main functionality of the currently hot concept of Lakehouse. But, no matter how data is loaded to the warehouse (including the extremely inefficient method that lets the warehouse access data lake through the external table), today’s data lake is made up of three components – massive data storage, data warehouse and a specialized engine (for, like, unstructured data processing).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fovuu467hwrtowwef9i6c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fovuu467hwrtowwef9i6c.png" alt=" " width="800" height="490"&gt;&lt;/a&gt;&lt;br&gt;
There are problems about this type of data lake framework.&lt;/p&gt;

&lt;h2&gt;
  
  
  The impossible triangle
&lt;/h2&gt;

&lt;p&gt;Data lakes are expected to meet three key requirements – storing data in its original state (loading high fidelity data into the lake), sufficient computing capacity (extracting the maximum possible data value) and cost-effective development (which is obvious). The current technology stack, however, cannot achieve all the three demands at the same time.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkn7omgyef54uc86s3wx9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkn7omgyef54uc86s3wx9.png" alt=" " width="800" height="520"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Storing data as it is was the initial purpose of building the data lake because keeping the original data unaltered helps to extract the maximum value from it. The simplest way to achieve the purpose is that the data lake uses a completely same storage medium to store data loaded from the source. There will be, for instance, a MySQL to hold data originally stored in MySQL, a MongoDB to receive data initially stored in MongoDB, and so on. This helps load data into the lake in as hi-fi format as possible and make use of the source’s computing ability. Though achieving computations across data sources is still hard, it is enough to handle computations only involving the current source’s data, meeting the basic requirement of sufficient computing power (as part i in the above figure shows).&lt;/p&gt;

&lt;p&gt;But the disadvantage is noticeable – the development is too expensive. Users need to put same storage mediums in place and copy all data sources accumulated over years to them. The workload is ridiculously heavy. If a data source is stored with commercial software, purchasing the software further pushes up the development cost. A relief strategy is to use a storage medium of same type, like storing Oracle data in MySQL, but it brings a side effect while the costs still stay high – some computations that could have been handled could become impossible or hard to achieve.&lt;/p&gt;

&lt;p&gt;Now, let’s lower the bar. We don’t demand that data be duplicated at loading but just store data in the database. By doing this, we obtain the database’s computing ability and meet the requirement of cheap development (as part ii in the above figure shows) at the same time. But this is infeasible since it heavily depends on one relational database into which all data needs to be loaded.&lt;/p&gt;

&lt;p&gt;Information may be easily lost during the loading process, which will fall short of the first requirement of building the data lake (loading high-fidelity data into the lake). Storing MongoDB data in MySQL or Hive is hard, for instance. Many MongoDB data types and relationships between sets do not exist in MySQL, such as the set data type like nested data structure, array and hash, and the instances of many-to-many relationship. They cannot be simply duplicated in the course of data migration. But rather, certain data structure needs to be restructured before the migration. That requires a series of sophisticated data reorganization steps, which is not cost-effective but needs a lot of people and time to sort out the business target and design appropriate form of target data organization. Without doing this, information will be lost, and errors, in turn, appear during the subsequent analysis. Sometimes errors are too deeply hidden to be easily visible.&lt;/p&gt;

&lt;p&gt;A general approach is to load data unalterably into large files (or as large fields in the database). This way the information loss is within an acceptable range and data basically remains intact. File storage has many advantages. It is more flexible, more open, and has higher I/O efficiency. Particularly, storing data in files (or in a file system) is cheaper.&lt;/p&gt;

&lt;p&gt;Yet, the problem of file storage is that files/large fields do not have computing capacity, making it impossible to meet the requirement of convenient/sufficient computing power. It seems that the impossible triangle is too strong to break.&lt;/p&gt;

&lt;p&gt;No approach can resolve the conflict between the demand for storing data in its initial state and the convenient use of it. Under the requirement for cost-saving lake building (loading data to the lake fast), high fidelity data loading and convenient/sufficient computing power are mutually exclusive. This goes against the data lake’s goal of openness.&lt;/p&gt;

&lt;p&gt;The underlying cause of the conflict is the contradiction between the closed database and its strict constraints. The database requires that data be loaded into it for computations and data needs to meet certain database constraints before being able to be loaded. In order to conform to the rules, data needs to be cleansed and transformed. And information loss happens during the process. Abandoning databases and switching to other routes (like files) cannot satisfy the demand of sufficient computing power, except that you turn to hardcoding. But hardcoding is too complicated and not nearly as convenient as databases.&lt;/p&gt;

&lt;p&gt;Actually, an open computing engine can become the breaker of the impossible triangle. Such an engine possessing sufficient and convenient computing power can compute the raw data, including data stored in diverse data sources, in real time.&lt;/p&gt;

&lt;h2&gt;
  
  
  SPL – the open data lake computing engine
&lt;/h2&gt;

&lt;p&gt;The open-source SPL is a structured data computing engine that provides open computing power for data lakes. It has diverse-source mixed computing capability that enables to compute raw data stored in different sources directly and based on its original status. No matter which storage mediums the data lake uses – same types as data sources or files, SPL can compute data directly and perform the data transformation step by step, making the lake building easier.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fv27spstzlvqd2ogewv7b.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fv27spstzlvqd2ogewv7b.png" alt=" " width="800" height="430"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Open and all-around computing power
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Diverse-source mixed computing ability&lt;/strong&gt;&lt;br&gt;
SPL supports various data sources, including RDB, NoSQL, JSON/XML, CSV, Webservice, etc., and mixed computations between different sources. This enables direct use of any type of raw data stored in the data lake and extraction of its value without the “loading” step and preparation. And this flexible and efficient use of data is just one of the goals of data lakes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0dcfh22zz5a6o1j1yhx7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0dcfh22zz5a6o1j1yhx7.png" alt=" " width="800" height="348"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Being agile like this, the data lake will be able to provide data services to applications as soon as it is established rather than after the prolonged cycle of data preparation, loading and modeling. The more flexible data lake service enables real time response to business needs.&lt;/p&gt;

&lt;p&gt;Particularly, SPL’s good support for files gives powerful computing ability to them. Storing lake data in a file system can also obtain computing power nearly as good as, even greater than, the database capability. This introduces computing capacity on the basis of part iii and makes the originally impossible triangle feasible.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F77jc8t019aezj5806cal.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F77jc8t019aezj5806cal.png" alt=" " width="800" height="547"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Besides text files, SPL can also handle data of hierarchical format like JSON naturally. Data stored in NoSQL and RESTful can thus be used directly without transformation. It’s really convenient.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fnqmcx0heiuyugmgcsxtg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fnqmcx0heiuyugmgcsxtg.png" alt=" " width="800" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;All-around computing capacity&lt;/strong&gt;&lt;br&gt;
SPL has all-around computational capability. The discrete data set model (instead of relational algebra) it is based arms it with a complete set of computing abilities as SQL has. Moreover, with agile syntax and procedural programming ability, data processing in SPL is simpler and more convenient than in SQL.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuyzo1l8hae2lpktxxpty.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuyzo1l8hae2lpktxxpty.png" alt=" " width="800" height="362"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;SPL boasts a wealth of class libraries for computations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accessing source data directly&lt;/strong&gt;&lt;br&gt;
SPL’s open computing power extends beyond data lake. Generally, if the target data isn’t synchronized from the source to the lake but is needed right now, we have no choice but to wait for the completion of synchronization. Now with SPL, we can access the data source directly to perform computations, or perform mixed computations between the data source and the existing data in the lake. Logically, the data source can be treated as part of the data lake to engage in the computation so that higher flexibility can be achieved.&lt;/p&gt;

&lt;h2&gt;
  
  
  High-performance computations after data transformation
&lt;/h2&gt;

&lt;p&gt;SPL’s joining makes data warehouse optional. SPL has all-around, remarkable computing power and offers high-performance file storage strategies. ETLing raw data and storing it in SPL storage formats can achieve higher performance. What’s more, the file system has a series of advantages like flexible to use and easy to parallelly process.&lt;/p&gt;

&lt;p&gt;SPL provides two high-performance storage formats – bin file and composite table. A bin file is compressed (to occupy less space and allow fast retrieval), stores data types (to enable faster retrieval without parsing), and supports the double increment segmentation technique to divide an append-able file, which facilitates parallel processing in an effort to further increase computing performance. The composite table uses column-wise storage to have great advantage in handling scenarios where only a very small number of columns (fields) is involved. A composite table is also equipped with the minmax index and supports double increment segmentation technique, letting computations both enjoy the advantages of column-wise storage and be more easily parallelly processed to have better performance.&lt;/p&gt;

&lt;p&gt;It is easy to implement parallel processing in SPL and fully bring into play the advantage of multiple CPUs. Many SPL functions, like file retrieval, filtering and sorting, support parallel processing. It is simple and convenient for them to automatically implement the multithreaded processing only by adding the @m option. They support writing parallel program explicitly to enhance computing performance.&lt;/p&gt;

&lt;p&gt;In addition, SPL supports a variety of high-performance algorithms SQL cannot achieve, the commonly seen TopN operation, for example. It treats calculating TopN as a kind of aggregate operation, which successfully transforms the highly complex sorting to the low-complexity aggregate operation while extending the field of application.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fz5y7qxkvdn7rep7drn0j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fz5y7qxkvdn7rep7drn0j.png" alt=" " width="800" height="192"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The SPL statements do not involve any sort-related keywords and will not trigger a full sorting. The statement for getting top N from a whole set and that from grouped subsets are basically the same and both have high performance. SPL boasts many more such high-performance algorithms.&lt;/p&gt;

&lt;p&gt;Assisted by all these mechanisms, SPL can achieve performance orders of magnitude higher than that of the traditional data warehouses. The storage and computation issues after data are transformed are solved. Data warehouses won’t be a data lake necessity any longer.&lt;/p&gt;

&lt;p&gt;Furthermore, SPL can perform mixed computations directly on/between transformed data and raw data by making good use of values of different types of data sources rather than by preparing data in advance. This creates highly agile data lakes.&lt;/p&gt;

&lt;p&gt;SPL enables performing lake building phases side by side while, conventionally, they can only be performed one by one (loading, transformation and computation). Data preparation and computation can be carried out concurrently and any type of raw, irregular data can be computed directly. Dealing with the computation and the transformation at the same time rather than in serial order is the key to building an ideal data lake.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Lakehouse? More Like a Lake + Warehouse Parking Lot</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Thu, 22 Jan 2026 06:50:56 +0000</pubDate>
      <link>https://dev.to/esproc_spl/lakehouse-more-like-a-lake-warehouse-parking-lot-4hfg</link>
      <guid>https://dev.to/esproc_spl/lakehouse-more-like-a-lake-warehouse-parking-lot-4hfg</guid>
      <description>&lt;p&gt;From all-in-one machine, hyper-convergence, cloud computing to HTAP, we constantly try to combine multiple application scenarios together and attempt to solve this type of problem through one technology so as to achieve the goal of simple and efficient use. Lakehouse, which is very hot nowadays, is exactly such a technology; its goal is to integrate the data lake with the data warehouse to give play to their respective value at the same time.&lt;/p&gt;

&lt;p&gt;The data lake and data warehouse have always been related closely, yet there are significant differences between them. The data lake pays more attention to retaining the original information, and its primary goal is to store the raw data “as is”. However, there are a lot of junk data in the raw data. Does storing the raw data “as is” mean that all the junk data will be stored in data lake? Yes, the data lake is just like a junk data yard where all the data is stored, regardless of whether they are useful or not. Therefore, the first problem that the data lake faces is the storage of massive (junk) data.&lt;/p&gt;

&lt;p&gt;Benefiting from the considerableprogress of modern storage technology, the cost of storing massive data is reduced dramatically. For example, using the distributed file system can fully meet the storage needs of data lake. But, the data storing ability alone is not enough, the computing ability is also required to bring the value into play. Data lake stores various types of data and each is processed differently, and the structured data processing is of the highest degree of importance. Whether it is historical data or newly generated business data, data processing mainly focuses on structured data. On many occasions, computations of semi-structured data and unstructured data will eventually be transformed to structured data computation. Unfortunately, however, since the storage schema itself (file system) of data lake does not have the computing ability, it is impossible to process the data directly on the data lake. To process the data, you have to use other technologies (such as data warehouse). The main problem that data lake is facing is “capable of storing, but incapable of computing”.&lt;/p&gt;

&lt;p&gt;For the data warehouse, it is just the opposite. Data warehouse is based on SQL system, and often has powerful ability to calculate the structured data. However, only after the raw data are cleansed, transformed and deeply organized until they meet database’s constraints can they be loaded into the data warehouse. In this process, a large amount of original information will be lost, even the data granularity will become coarse, resulting in a failure of obtaining the value of data with lower granularity. Moreover, the data warehouse is highly subject-oriented, and services one or a few subjects only. Since the data outside the subjects is not the target of data warehouse, it will make the range of usable data relatively narrow, making it unable to explore the value of full and unknown data as data lake does, let alone store massive raw data like data lake. Compared with data lake, the data warehouse is “capable of computing, but incapable of storing”.&lt;/p&gt;

&lt;p&gt;From the point of view of data flow, the data of data warehouse can be organized based on data lake, so a natural idea is to integrate the data lake with the data warehouse to achieve the goal of “capable of storing and computing”, which is the so-called “Lakehouse”.&lt;/p&gt;

&lt;p&gt;So, what is current implementing situation?&lt;/p&gt;

&lt;p&gt;The current method is oversimplified and crude, that is, open the data access rights on the data lake to allow the data warehouse to access the data in real-time (the so-called real-time is relative to the original ETL process that needs to periodically move the data from data lake to data warehouse. Yet, there is still a certain delay in practice). Physically, the data are still stored in two places, and the data interaction is performed through high-speed network. Due to having a certain ability to “real time” process the data of data lake, the implementation result (mostly at the architecture level) is now called Lakehouse.&lt;/p&gt;

&lt;p&gt;That’s it? Is that a Lakehouse in the true sense?&lt;/p&gt;

&lt;p&gt;Well, I have to say - as long as the one (who claims it is Lakehouse) doesn’t feel embarrassed, embarrassing is as the one (who knows what Lakehouse should be like) feels embarrassed.&lt;/p&gt;

&lt;p&gt;Then, how does the data warehouse read the data of data lake? A common practice is to create an external table/schema in the data warehouse to map RDB’s table, or schema, or hive’s metastore. This process is the same as the method that a traditional RDB accesses the external data through external table. Although the metadata information is retained, the disadvantages are obvious. Specifically, it requires the data lake can be mapped as tables and schema under corresponding relational model, and it also needs to organize the data before computing them. Moreover, the types of available data sources decrease (for example, we cannot perform mapping directly based on NoSQL, text, and Webservice). Furthermore, even if there are other data sources (such as RDB) available for computation in the data lake, the data warehouse usually needs to move the data to its local position when computing (such as grouping and aggregating), resulting in a high data transmission cost, performance drop, and many problems.&lt;/p&gt;

&lt;p&gt;For the current Lakehouse, in addition to “real-time” data interaction, the original channel for periodically organizing the data in batches is still retained. In this way, the organized data of data lake can be stored into the data warehouse for local computing. Of course, this has little to do with the Lakehouse, because it was done the same way before the “integration”.&lt;/p&gt;

&lt;p&gt;Anyway, both the data lake and data warehouse change little (only the data transmission frequency is improved, but many conditions have to be met), whether the data is transmitted from lake to warehouse through traditional ETL or external real-time mapping. Physically, the data are still stored in two places. The data lake is still the original data lake, and the warehouse is still the original data warehouse, and they are not integrated essentially!Consequently, not only are the data diversity and efficiency problems not fundamentally solved (lack of flexibility), but it also needs to organize the “junk” data of data lake first, and then load them into the warehouse before computing (poor real time performance). If you want to build a real-time and efficient data processing ability on the data lake through the “Lakehouse” implemented in this way, I'm afraid it's a joke.&lt;/p&gt;

&lt;p&gt;Why?&lt;/p&gt;

&lt;p&gt;If we think a little, we will find that the problem is in the data warehouse. The database system is too closed and lacks openness, it needs to load the data into the database (including external data mapping) before computing. Moreover, due to the database constraints, the data must be deeply organized to conform to the norms before being loaded into the database, while the raw data itself of data lake has a lot of “junk” data. Although it is reasonable to organize these data, it is difficult to respond to the real-time computing needs of data lake. If the database is open enough, and has the ability to directly calculate the unorganized data of data lake, and even the ability to perform mixed computing based on a variety of different types of data sources, and provide a high-performance mechanism to ensure the computing efficiency at the same time, then it is easy to implement a real Lakehouse. However, it is a pity that the database is unable to achieve this goal.&lt;/p&gt;

&lt;p&gt;Fortunately, esProc SPL does.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SPL - an open computing engine - helps implement a real Lakehouse&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The open-source SPL is a structured data computing engine that provides open computing power for data lake, which can directly calculate the raw data of data lake, there are no constraints, even no database to store the data. Moreover, SPL boasts the mixed computing ability for diverse data sources. Whether the data lake is built on a unified file system, or based on diverse data sources (RDB, NoSQL, LocalFile, Webservice), a direct mixed computing can be accomplished in SPL, and the value of data lake can be produced quickly. Furthermore, SPL provides a high-performance file storage (the storage function of data warehouse).The data can be organized unhurriedly when calculations are going on in SPL, while loading the raw data into SPL’s storage can obtain higher performance. Particular attention should be paid that the data are still stored in the file system after they are organized in SPL storage, and theoretically, they can be stored in the same place with the data lake. In this way, a real Lakehouse can be implemented.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0wvwlud5aj37vdv58tcg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0wvwlud5aj37vdv58tcg.png" alt=" " width="800" height="470"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In the whole architecture, SPL can perform unified storage and calculation directly based on data lake, and can also connect to diverse data sources in the data lake, and even directly read the external production data source. With these abilities, a real-time calculation on the data lake can be implemented, and in some scenarios that require high data timeliness (it needs to use the data before they are stored into the data lake), SPL can connect to the real-time data source, so the data timeliness is higher.&lt;/p&gt;

&lt;p&gt;The original way that moves the data from the data lake to data warehouse can still be retained. ETLing the raw data to SPL’s high-performance storage can achieve a higher computing performance. Meanwhile, using the file system to store the data enables the data to be distributed on the SPL server (storage) or, alternatively, we can still use the unified file of data lake to store the data, that is, the work of original data warehouse is completely taken over by SPL. As a result, the Lakehouse is implemented in one system.&lt;/p&gt;

&lt;p&gt;Let's take a look at these abilities of SPL.&lt;/p&gt;

&lt;h2&gt;
  
  
  Open and all-around computing power
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Diverse-source mixed computing ability&lt;/strong&gt;&lt;br&gt;
SPL supports various data sources, including RDB, NoSQL, JSON/XML, CSV, Webservice, etc., and has the ability to perform mixed computation between different sources. This enables direct use of any type of raw data stored in the data lake and gives play to the value of data without transforming the data, and the action of “loading into the database” is omitted. Therefore, the flexible and efficient use of data is ensured, and a wider range of business requirements is covered.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbut3krbyg9tmc6gvyvzw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbut3krbyg9tmc6gvyvzw.png" alt=" " width="800" height="349"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With this ability, the data lake will be able to provide data service for applications as soon as it is established rather than having to complete a prolonged cycle of data preparation, loading and modeling. Moreover, the SPL-based data lake is more flexible, and can provide a real time response based on business needs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Supporting file computing&lt;/strong&gt;&lt;br&gt;
Particularly, SPL’s good support for files gives powerful computing ability to them. Storing lake data in a file system can also obtain computing power nearly as good as, even greater than, the database capability. Besides text files, SPL can also handle the data of hierarchical format like JSON, and thus the data stored in NoSQL and RESTful can be used directly without transformation. It’s really convenient.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fzntcgftagmd6c3opf6kr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fzntcgftagmd6c3opf6kr.png" alt=" " width="800" height="334"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;All-around computing capacity&lt;br&gt;
SPL provides all-around computational capability. The discrete data set model (instead of relational algebra) it is based arms it with a complete set of computing abilities as SQL has. Moreover, with agile syntax and procedural programming ability, data processing in SPL is simpler and more convenient than in SQL.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuqjh7anfgy5zzxhmmbh2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuqjh7anfgy5zzxhmmbh2.png" alt=" " width="800" height="362"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Rich computing library of SPL&lt;/p&gt;

&lt;p&gt;This enables the data lake to fully has the computing ability of data warehouse, achieving the first step of integrating data lake with data warehouse.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accessing source data directly&lt;/strong&gt;&lt;br&gt;
SPL’s open computing power extends beyond data lake. Generally, if the target data isn’t synchronized from the source into the lake but is needed right now, we have no choice but to wait for the completion of synchronization. Now with SPL, we can access the data source directly to perform computations, or perform mixed computations between the data source and the existing data in the lake. Logically, the data source can be treated as part of the data lake to engage in the computation so that higher flexibility can be achieved.&lt;/p&gt;

&lt;h2&gt;
  
  
  High-performance computations after data organization
&lt;/h2&gt;

&lt;p&gt;In addition to its own all-around and powerful computing abilities, SPL provides file-based high-performance storage. ETLing raw data and storing it in SPL storage can achieve higher performance. What’s more, the file system has a series of advantages like flexible to use and easy to parallelly process. Having the data storage ability is equivalent to achieving the second step of integrating the data lake with data warehouse, and a new open and flexible data warehouse is formed.&lt;/p&gt;

&lt;p&gt;Currently, SPL provides two high-performance file storage formats: bin file and composite table. The bin file adopts the compression technology (faster reading due to less space occupation,), stores the data types (faster reading due to no need to parse the data type), and supports the double increment segmentation mechanism that can append the data. Since it is easy to implement parallel computing by using the segmentation strategy, computing performance is ensured. The composite table supports columnar storage, this storage schema has great advantage in scenarios where only a very small number of columns (fields) are involved. In addition, the composite table implements the minmaxindex and supports double increment segmentation mechanism, therefore, it not only enjoys the advantages of columnar storage, but also makes it easier to perform the parallel computing to improve the performance.&lt;/p&gt;

&lt;p&gt;Furthermore, it is easy to implement parallel computing in SPL and fully bring into play the advantage of multiple CPUs. Many SPL functions, like file retrieval, filtering and sorting, support parallel processing. It is simple and convenient for them to automatically implement the multithreaded processing only by adding the @moption. They support writing parallel program explicitly to enhance computing performance.&lt;/p&gt;

&lt;p&gt;In particular, SPL supports a variety of high-performance algorithms SQL cannot achieve. For example, the common TopN operation is treated as an aggregation operation in SPL, as a result, a high-complexity sorting operation can be transformed to a low-complexity aggregation operation while extending the range of application.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fe06zzhwu5zp0d1vwr03d.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fe06zzhwu5zp0d1vwr03d.png" alt=" " width="800" height="176"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In these statements, there are no any sort-related keywords and will not trigger a full sorting action. The statements for getting top N from a whole set and from grouped subsets are basically the same and both can achieve a higher performance. SPL boasts many more such high-performance algorithms.&lt;/p&gt;

&lt;p&gt;Depending on these mechanisms, SPL can achieve a performance that surpasses that of traditional data warehouse, the degree of surpassing is measured in orders of magnitude, and the full implementation of Lakehouse in data lake is not done in words but effective mechanisms.&lt;/p&gt;

&lt;p&gt;Furthermore, SPL can perform mixed computations on transformed data and raw data to give full play to the value of various types of data, instead of preparing data in advance. In this way, not only is the flexibility of data lake fully expanded, but it also has the function of real-time data warehouse. This achieves the third step of integrating the data lake with data warehouse, which takes into account both the flexibility and high performance.&lt;/p&gt;

&lt;p&gt;Through the above three steps, the path to build the data lake is improved (the original path needs to load and transform the data before computing), and the data preparation and computation can be carried out at the same time, and the data lake is built step by step. Moreover, in the process of building the data lake, the data warehouse is perfected, making the data lake has powerful computing ability, implementing the real Lakehouse. This is the correct method for implementing a real Lakehouse.&lt;br&gt;
SPL is now open-source. You can obtain the source code from&lt;a href="https://github.com/SPLWare/esProc" rel="noopener noreferrer"&gt; GitHub .&lt;/a&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>data</category>
      <category>database</category>
      <category>dataengineering</category>
    </item>
    <item>
      <title>Are Wide Tables Fast or Slow?</title>
      <dc:creator>Judy</dc:creator>
      <pubDate>Mon, 19 Jan 2026 07:54:58 +0000</pubDate>
      <link>https://dev.to/esproc_spl/are-wide-tables-fast-or-slow-3ba1</link>
      <guid>https://dev.to/esproc_spl/are-wide-tables-fast-or-slow-3ba1</guid>
      <description>&lt;p&gt;Wide tables are usually a standard component of the BI system. Many BI projects will first prepare wide tables at the beginning of construction. A wide table is formed by joining up multiple tables that have a certain association relationship. The result set does not conform to the normal forms; and there is a large amount of redundant data. Moreover, as wide tables need to be pre-created, they are not so flexible to use.&lt;/p&gt;

&lt;p&gt;But why do people very much prefer wide tables even if they have many shortcomings?&lt;/p&gt;

&lt;p&gt;Because wide tables are &lt;strong&gt;FAST&lt;/strong&gt;!&lt;/p&gt;

&lt;p&gt;Querying data in the wide table is usually faster than performing the real-time multi-table join. So, building wide tables is to avoid joins. Join operations are a long-standing problem in SQL. They are difficult to write and has poor performance. Find detailed analysis about SQL joins &lt;a href="https://c.esproc.com/article/1653353923359" rel="noopener noreferrer"&gt;HERE&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;However, though wide tables help avoid joins, too much extra data may be read during the computation because there is data redundancy and this increases I/O time. For example, there is an Orders table where each order corresponds to 5 records in OrderDetails table. By stretching the two to a wide table, data in Order table will repeat 5 times. What’s more, the Orders table has dimension tables such as Customer and Employee, and Customer table has dimension tables including Region, and so on. When all these tables are extended to form a wide table, the entire data volume will be enlarged many times. To perform a query on that wide table, such as summing up order amounts by customer’s region, a large volume of data will be retrieved and I/O overhead is huge.&lt;/p&gt;

&lt;p&gt;According to the above analysis, wide tables should have been slower. Why they are faster in the real-world practice? This is because relational database joins are too slow. Even if the wide table IO cost increases several times, the query is still faster than the real-time joins.&lt;/p&gt;

&lt;p&gt;If we can do some optimization to make the join run faster, can we get satisfactory performance while avoiding a series of wide table problems including redundant data, error from result set that does not conform to normal forms and stiffness?&lt;/p&gt;

&lt;p&gt;The answer is yes. But it is a pity that &lt;strong&gt;SQL cannot do that&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In the above document link, there are already detailed analysis about the SQL join. In a nutshell, the JOIN defined by Cartesian product is indeed very simple and the simple connotation gets broader denotation to cover various JOIN scenarios. But a too general definition makes it impossible to perform targeted optimization on different join operations. People can only think of some temporary solutions in engineering, but cannot fundamentally solve the problem.&lt;/p&gt;

&lt;p&gt;Here’s another fact. Since the debut of the database, optimization methods for simple SQL used for BI analysis have been stretched to the limit by various vendors, but even so wide tables are needed to solve the performance problem. It can be seen that it is difficult to deal with – we can even say – impossible to solve the performance problem of join operations.&lt;/p&gt;

&lt;p&gt;Is there anything we can do?&lt;/p&gt;

&lt;p&gt;We can use SPL to tackle the problem.&lt;/p&gt;

&lt;p&gt;SPL (Structured Process Language) is an open-source computing engine intended for structured data computations. It offers powerful computing ability independent of databases. The performance of handling join operations in SPL is much higher than those of both the SQL join and the wide table-based join. The language addresses the root of join operation performance problem as well as avoiding problems brought by wide tables.&lt;/p&gt;

&lt;p&gt;The commonly seen equi-joins in BI analyses are categorized into two types in SPL – foreign key join and primary key join. Each is provided their own performance optimization methods, which is explained in the second half of the above-mentioned post about Join Simplification and Acceleration. SPL specifically offers dimension table preload method and numberization method for the common foreign key joins and order-based merge method for primary key joins that help to significantly reduce the join operation complexity.&lt;/p&gt;

&lt;p&gt;Once we speed up the join operation, wide tables become useless and the volume of data to be read is reduced. The result is that SPL greatly increases BI performance.&lt;/p&gt;

&lt;p&gt;That’s theoretical explanations. How best does SPL’s field performance?&lt;/p&gt;

&lt;p&gt;A comparison test was performed. The test includes a common aggregation by dimension in multidimensional analysis after a join between a fact table and multiple, multilayer dimension tables, and an aggregation based on the wide table.&lt;/p&gt;

&lt;p&gt;The test data is based on a data set of TPCH 100G and a computation involving a join between one large fact table and multiple dimension tables is designed:&lt;/p&gt;

&lt;p&gt;Two table join between one fact table and one dimension table;&lt;br&gt;
Seven-table join involving joins between a primary-sub fact table and four dimension tables, during which a dimension table is used twice;&lt;br&gt;
Convert the seven-table join result to a wide table and perform wide-table-based aggregation.&lt;br&gt;
The products used for performing the test are two specialized OLAP databases – StarRocks and Clickhouse, which are famous for high-performance BI analysis. Below is the test result:&lt;/p&gt;

&lt;p&gt;Time unit: Second&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fm6lsfbnclnz98cvz9rp2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fm6lsfbnclnz98cvz9rp2.png" alt=" " width="800" height="244"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Find detailed test report &lt;a href="https://c.esproc.com/article/1690170794600" rel="noopener noreferrer"&gt;HERE&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;According to the test result, the SQL’s wide table is faster than the join, which verifies the previous analysis. The performance of SPL’s wide table is actually not as fast as ClickHouse, but its real-time join performance is very high. It is higher than joins performed in the two SQL databases (3-9 times faster), and even &lt;strong&gt;surpasses the database’s wide table method by huge margins&lt;/strong&gt;. If we take wide table’s shortcomings into account (redundant data, data error and stiffness), the advantage of SPL’s real-time join becomes more obvious. It not only avoids wide table defects but increases performance by N times.&lt;/p&gt;

&lt;p&gt;The wide-table-based join is not necessarily faster than the real-time join! Because of SPL, costly wide tables created to obtain high performance in the BI system become useless.&lt;/p&gt;

&lt;p&gt;SPL is now open-source. You can obtain the source code from &lt;a href="https://github.com/SPLWare/esProc" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt; .&lt;/p&gt;

</description>
      <category>database</category>
      <category>dataengineering</category>
      <category>performance</category>
      <category>sql</category>
    </item>
  </channel>
</rss>
