<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ramkumar Jayakumar</title>
    <description>The latest articles on DEV Community by Ramkumar Jayakumar (@rkj180220).</description>
    <link>https://dev.to/rkj180220</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F583870%2F5a6365bf-7358-445f-9f10-6bada7a164c5.jpg</url>
      <title>DEV Community: Ramkumar Jayakumar</title>
      <link>https://dev.to/rkj180220</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rkj180220"/>
    <language>en</language>
    <item>
      <title>The agent said $62,421.60. The records said $50,075.93.</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:44:46 +0000</pubDate>
      <link>https://dev.to/rkj180220/the-agent-said-6242160-the-records-said-5007593-3e8b</link>
      <guid>https://dev.to/rkj180220/the-agent-said-6242160-the-records-said-5007593-3e8b</guid>
      <description>&lt;h2&gt;
  
  
  The moment I stopped trusting a plausible number
&lt;/h2&gt;

&lt;p&gt;On a synthetic account, six cash deposits had landed in one week. I wanted to see what my system would do with a believable lie, so I played a misbehaving agent and posted two claims to the case board:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;em&gt;"The account received 6 cash deposits in the window."&lt;/em&gt; True.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;"Those deposits total **62,421.60&lt;/em&gt;&lt;em&gt;."&lt;/em&gt; Invented. The real total was &lt;strong&gt;50,075.93&lt;/strong&gt;. I had added a made-up 12,345.67 to the real number.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Both claims read like something a language model would write. Neither raised any alarm for me, and I wrote the lie.&lt;/p&gt;

&lt;p&gt;The verifier did not ask another model whether the claim sounded right. It re-ran the query against the records and wrote this on the claim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CONTRADICTED  expected sum == 62421.60, actual 50075.93
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first claim came back &lt;strong&gt;corroborated&lt;/strong&gt;. The second came back &lt;strong&gt;contradicted&lt;/strong&gt;, with the real number next to it.&lt;/p&gt;

&lt;p&gt;I want to be straight about that scene: &lt;strong&gt;I staged that lie.&lt;/strong&gt; But the system also met real model slips. In the evaluation I describe below, the specialists posted 389 claims across 46 synthetic alerts. 366 were corroborated, 5 were contradicted and 18 were unsupported. (One run, synthetic data, small model: treat those as illustrations of the mechanism, not a measured error rate for any real system.) During development, one specialist even asserted a link between a customer and &lt;em&gt;themselves&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;That small scene is the whole project: &lt;strong&gt;agents propose, records verify, a human decides.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The business problem
&lt;/h2&gt;

&lt;p&gt;Fraud and anti-money-laundering (AML) monitoring systems raise a lot of alerts. Most turn out to be harmless, but an investigator still has to prove that, alert by alert: pull transactions, check the customer file, check watchlists, look for links to other accounts, and write it all up. When an alert is real, the bank files a Suspicious Activity Report (SAR) with the regulator.&lt;/p&gt;

&lt;p&gt;AI agents can do a lot of that legwork. But a bank cannot accept two things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Agents that state facts that are not in the data.&lt;/strong&gt; A confident wrong total in a regulatory report is worse than no report.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An agent that can freeze an account or file a report on its own.&lt;/strong&gt; Accountability has to sit with a person.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I am not going to quote an industry false-alarm percentage here, because I don't have a figure I can verify. What I can show is a design that treats those two non-negotiables as architecture, not as prompt instructions, and a measurement of what that design costs and buys on synthetic data.&lt;/p&gt;




&lt;h2&gt;
  
  
  What I built
&lt;/h2&gt;

&lt;p&gt;I built &lt;strong&gt;Case Board&lt;/strong&gt; solo for a Fraud &amp;amp; AML hackathon track.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every alert becomes a &lt;strong&gt;case on a shared, typed GraphQL board&lt;/strong&gt;, stored in Azure Database for PostgreSQL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three specialist agents&lt;/strong&gt; investigate in parallel: transaction pattern, KYC and watchlist, and link analysis. A fourth, non-voting agent explains which rule fired. They run on &lt;strong&gt;Azure AI Foundry&lt;/strong&gt; model deployments: a small model for the specialists, and a larger model &lt;strong&gt;only&lt;/strong&gt; to draft the SAR narrative.&lt;/li&gt;
&lt;li&gt;The specialists never talk to each other. They read the bank's records through read-only tools and post &lt;strong&gt;typed claims&lt;/strong&gt; to the board.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;verifier with no language model in it&lt;/strong&gt; re-executes every claim against the records.&lt;/li&gt;
&lt;li&gt;A deterministic aggregator turns verified claims into a &lt;em&gt;proposal&lt;/em&gt;: clear, hold or escalate.&lt;/li&gt;
&lt;li&gt;The board compiles an &lt;strong&gt;evidence pack from verified claims only&lt;/strong&gt; (it lists the excluded claims too), and a SAR draft from the same.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only a human can hold, file or clear.&lt;/strong&gt; Filing is simulated: nothing is transmitted anywhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All data is synthetic: 60 generated alerts, 20 of them truly suspicious, with the answer key hidden from the agents by database permissions.&lt;/p&gt;




&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Case flow:&lt;/strong&gt; Synthetic alert → specialist agents → shared GraphQL board → deterministic verification against source records → human review.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All alert and account data shown in this prototype is synthetic. Filing is simulated.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Three design choices matter more than the rest.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Typed claims.&lt;/strong&gt; An agent can't just say "this account looks suspicious". Each claim carries an assertion with a type, such as &lt;em&gt;the sum of cash deposits on this account in this window equals X&lt;/em&gt;, or &lt;em&gt;these two customers share a phone number&lt;/em&gt;. Each type has a handler that re-runs the query. That is what makes the opening scene possible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. A shared board, not one coordinating AI.&lt;/strong&gt; The common way to build multi-agent systems is a &lt;em&gt;supervisor design&lt;/em&gt;: one coordinating model calls the specialists one after another and relays everything between them. I built exactly that as my baseline, with the same models, tools and alerts, so the only difference is how the work is organised. On the board, specialists react to case events and write structured findings with their sources.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Governance in the data layer.&lt;/strong&gt; More on that next.&lt;/p&gt;

&lt;p&gt;I also built a Foundry Agent Service backend for the specialists. It worked on one case, but a later re-test gave weaker evidence references, so &lt;strong&gt;the demo and every measured run use direct chat completions&lt;/strong&gt;, not Agent Service.&lt;/p&gt;




&lt;h2&gt;
  
  
  What the verifier does, and what it doesn't
&lt;/h2&gt;

&lt;p&gt;My first verifier checked that claims were &lt;em&gt;true&lt;/em&gt;. A reviewer (a fresh, adversarial pass over my code) showed that wasn't enough: a claim like "this account has at least zero transactions" is true, tight-looking and meaningless, and it would have been counted as support for a hold.&lt;/p&gt;

&lt;p&gt;After three adversarial review rounds, a claim only counts as &lt;strong&gt;support&lt;/strong&gt; for a hold or escalation if it is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;corroborated when re-run against the records,&lt;/li&gt;
&lt;li&gt;about the alert's &lt;strong&gt;own&lt;/strong&gt; account or customer (or one tightly linked to it),&lt;/li&gt;
&lt;li&gt;inside the &lt;strong&gt;alert window&lt;/strong&gt;,&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;tight&lt;/strong&gt; (not "at most 999,999,999"), and&lt;/li&gt;
&lt;li&gt;matching a &lt;strong&gt;template for the rule that fired&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Anything else can still be shown as corroborated context, but it never drives a decision.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What that does not mean.&lt;/strong&gt; The verifier catches arithmetically false claims, and it keeps true-but-irrelevant ones from steering a decision. &lt;strong&gt;It is not general hallucination detection&lt;/strong&gt;, and it cannot judge whether a true, in-template claim is &lt;em&gt;persuasive&lt;/em&gt;. A model can still choose which true facts to emphasise.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where human review fits
&lt;/h2&gt;

&lt;p&gt;The prototype separates investigation from decision authority. The application restricts case decisions to a reviewer role and records decision attempts in an audit log. The database also checks that a case has an approval before it enters a terminal state.&lt;/p&gt;

&lt;p&gt;The demo uses shared credentials, so it demonstrates the approval flow at a prototype level. A production version would need named reviewers, narrowly scoped credentials, and a stronger separation between the application and audit storage. Filing is simulated.&lt;/p&gt;

&lt;h2&gt;
  
  
  The evaluation, with its caveats attached
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Setup.&lt;/strong&gt; 60 synthetic alerts from a seeded generator (20 truly suspicious). I excluded the 14 I had used for prompt tuning or live checks, which leaves &lt;strong&gt;46 scored alerts: 34 benign and 12 suspicious&lt;/strong&gt;. Each alert was run &lt;strong&gt;once&lt;/strong&gt;, with the same small model, the same tools and the same specialist prompts for both designs. The supervisor baseline is a design &lt;strong&gt;I built myself&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Default policy against the supervisor baseline
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Supervisor baseline&lt;/th&gt;
&lt;th&gt;Case Board (default policy)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Benign alerts wrongly flagged&lt;/td&gt;
&lt;td&gt;13/34 (38.2%)&lt;/td&gt;
&lt;td&gt;11/34 (32.4%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Suspicious alerts missed&lt;/td&gt;
&lt;td&gt;0/12&lt;/td&gt;
&lt;td&gt;0/12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per alert (in + out, mean)&lt;/td&gt;
&lt;td&gt;27,297&lt;/td&gt;
&lt;td&gt;20,982&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time per alert (mean)&lt;/td&gt;
&lt;td&gt;56.0 s&lt;/td&gt;
&lt;td&gt;52.7 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verified claims behind each decision (mean)&lt;/td&gt;
&lt;td&gt;0 (checked afterwards, as a diagnostic)&lt;/td&gt;
&lt;td&gt;8.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unsupported claim rate&lt;/td&gt;
&lt;td&gt;6.9%&lt;/td&gt;
&lt;td&gt;4.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost per alert, investigation only&lt;/td&gt;
&lt;td&gt;USD 0.0091&lt;/td&gt;
&lt;td&gt;USD 0.0072&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Read this with it.&lt;/strong&gt; Synthetic data, 46 alerts, one run each, so the intervals are wide. &lt;strong&gt;The false-positive gap (32.4% against 38.2%) is within noise: the 95% intervals overlap, and I am not claiming a false-positive improvement.&lt;/strong&gt; The time difference is also within what I would call noise. Both designs caught all 12 suspicious alerts, and both still flagged nearly every look-alike &lt;em&gt;rapid-movement&lt;/em&gt; benign alert (the board 11 of 12, the supervisor 12 of 12). Cost uses Azure's published retail prices for the two deployments, retrieved on 2026-10-18, with input priced at the full uncached rate, so it is an &lt;strong&gt;upper bound&lt;/strong&gt;, and it covers the investigation only (one SAR draft with the larger model cost about USD 0.019). The baseline's claims were only verified &lt;em&gt;after&lt;/em&gt; its decisions, as a diagnostic, so "0" means its decisions did not depend on verification.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I will say:&lt;/strong&gt; in this one-run synthetic evaluation, both designs caught all 12 suspicious alerts; the board used about 23% fewer tokens and cost about 21% less per alert, and each board decision had a mean of eight verifier-checked claims behind it. These are prototype measurements, not production outcomes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I won't say:&lt;/strong&gt; that the board reduces false positives, that this would hold on real data, or that these are production outcomes.&lt;/p&gt;

&lt;p&gt;Fairness caveats you should know about: my specialist prompts were hand-tuned on five alerts, and they encode the synthetic generator's typologies, which helps &lt;em&gt;both&lt;/em&gt; designs but means this isn't a blind test. The supervisor's own prompt is a first draft that I never iterated. The board's timing includes real database round trips; the baseline writes nothing. Both ran on one machine, one alert at a time.&lt;/p&gt;

&lt;h3&gt;
  
  
  The decision trade-off: a policy dial
&lt;/h3&gt;

&lt;p&gt;Both designs struggled with benign rapid-movement alerts, so I tried one change. In the &lt;strong&gt;strict policy&lt;/strong&gt;, a wire count or sum no longer counts as support for a rapid-movement alert. Only ring evidence does: a link to another alerted account, or a shared payee.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Default policy&lt;/th&gt;
&lt;th&gt;Strict policy (optional)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Benign alerts wrongly flagged&lt;/td&gt;
&lt;td&gt;11/34 (32.4%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0/34 (0%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Suspicious alerts missed&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0/12&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2/12 (16.7%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alerts sent to human review&lt;/td&gt;
&lt;td&gt;23/46 (50.0%)&lt;/td&gt;
&lt;td&gt;10/46 (21.7%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost per alert, investigation only&lt;/td&gt;
&lt;td&gt;USD 0.0072&lt;/td&gt;
&lt;td&gt;USD 0.0073&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Benign alerts wrongly flagged   (lower is better)
  default  █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░  32%  (11 of 34)
  strict   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0%  ( 0 of 34)

Suspicious alerts missed        (lower is better)
  default  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0%  ( 0 of 12)
  strict   ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  17%  ( 2 of 12)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Read this with it.&lt;/strong&gt; Same 46 synthetic alerts, one run each, so one extra miss or false alarm moves these percentages a lot. &lt;strong&gt;I designed the strict policy after seeing the first results on these same alerts, so its numbers are in-sample and optimistic. It is not a held-out test.&lt;/strong&gt; One of its two misses is a structuring alert that the rule change does not touch, which points to ordinary run-to-run model variation rather than the policy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why the default stays lenient.&lt;/strong&gt; In AML, missing a suspicious alert costs a bank far more than reviewing a false alarm. So I ship the safer setting as the default, expose the strict one as an option (&lt;code&gt;AGENTS_RAPID_POLICY=strict&lt;/code&gt;), and show the trade-off instead of picking the prettier number.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run hygiene.&lt;/strong&gt; In my first full run, two board runs hit defects that I then fixed: a concurrency bug (two agents finishing at the same instant could leave a case stuck; fixed, with a regression test) and a case frozen by a finding with no checkable claim. I re-ran those two once on the fixed system, and the default-policy numbers above include those re-runs. In the strict-policy run, five runs failed from database and network timeouts rather than from decisions, and I re-ran each once. The raw first run is kept alongside.&lt;/p&gt;




&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Make authority a property of the architecture, not the prompt.&lt;/strong&gt; A prompt that says "never file a report" is a request. A database trigger that refuses the state change is a fact. I would rather the system be boring and refuse than clever and polite.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. True is not the same as relevant.&lt;/strong&gt; My first verifier happily corroborated meaningless-but-true claims. The fix was not a smarter model: it was scoping claims to the alert's own entities, window and rule. A fresh adversarial reviewer found what I would not have.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Report the dial, not just the best setting.&lt;/strong&gt; Tightening one rule removed every false alarm &lt;em&gt;and&lt;/em&gt; cost me two suspicious alerts. Showing both settings, with the reason for the default, is more useful to a reader than whichever number looks best.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Boring engineering decides whether a demo survives.&lt;/strong&gt; The failures that cost me time were not model failures. They were a race between two agents finishing together, stale database connections after my laptop changed networks, and a UI that re-rendered every few seconds and swallowed clicks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Separate proposing, verifying and deciding.&lt;/strong&gt; Agents write typed claims, something deterministic checks them against the records, and a human-only gate sits outside the model.&lt;/p&gt;




&lt;h2&gt;
  
  
  What this is not
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Not a production system, and not production results.&lt;/strong&gt; All data is synthetic, the evaluation is 46 alerts and one run each, and the strict policy is in-sample.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not deployed.&lt;/strong&gt; It runs locally against Azure AI Foundry and Azure Database for PostgreSQL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not built:&lt;/strong&gt; a Power Automate approval flow, a Copilot Studio assistant, a Power BI report, and Entra sign-in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not general hallucination detection.&lt;/strong&gt; It re-runs claims against records and keeps irrelevant true claims from driving decisions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a real approval workflow.&lt;/strong&gt; Demo approval, shared token, free-text approver name.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  What I would do next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Evaluate on a &lt;strong&gt;held-out&lt;/strong&gt; set that I did not look at while tuning, and on more than one run per alert.&lt;/li&gt;
&lt;li&gt;Use named reviewer identities, isolate database credentials, and add Entra sign-in.&lt;/li&gt;
&lt;li&gt;Look harder at the one place both designs struggled (look-alike rapid-movement alerts) with ring evidence that doesn't cost recall.&lt;/li&gt;
&lt;li&gt;Connect it to a real &lt;strong&gt;case-management&lt;/strong&gt; system, with real approvers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Everything above uses synthetic data. No customer data, and no real alerts, were used.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
    </item>
    <item>
      <title>The Answer Looked Fine. The AI Bill Didn’t.</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Sun, 08 Nov 2026 14:11:56 +0000</pubDate>
      <link>https://dev.to/rkj180220/the-answer-looked-fine-the-ai-bill-didnt-1d47</link>
      <guid>https://dev.to/rkj180220/the-answer-looked-fine-the-ai-bill-didnt-1d47</guid>
      <description>&lt;p&gt;A Friday release adds a friendly touch to a support chatbot. Greet customers by loyalty tier. Give a Gold member a more personal welcome before answering their question.&lt;/p&gt;

&lt;p&gt;The answer still looks reasonable. The order status is correct. Nobody changed the model.&lt;/p&gt;

&lt;p&gt;But the request costs more.&lt;/p&gt;

&lt;p&gt;That was the fictional case at the centre of &lt;strong&gt;Beyond the Prompt: The Engineering of AI Cost&lt;/strong&gt;, the session I presented with &lt;strong&gt;Balaji Saravanan at Presidio India Exchange 2026 on 4 November&lt;/strong&gt;. We used an application called &lt;strong&gt;AI Cost Engineering Lab&lt;/strong&gt; to follow a support request, inspect its receipt, investigate a regression and evaluate a release gate.&lt;/p&gt;

&lt;p&gt;I wanted the engineering work to be visible. A good answer on a chat screen hides a lot of decisions about context, retrieval, tools, retries and pricing. So we gave the audience a chatbot to start with and a detective game to investigate what happened behind it.&lt;/p&gt;

&lt;p&gt;Presidio’s published schedule listed our 50-minute talk in the Day 1 Tech Talk programme. &lt;a href="https://www.linkedin.com/posts/presidioindia_indiaexchange2026-techtalk-presidioindia-activity-7522892899655499776-u5zz" rel="noopener noreferrer"&gt;Event announcement&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This article follows that investigation, then goes deeper into the part I would build around a production application: a CI/CD gate that can refuse a cost regression without rewarding a worse answer.&lt;/p&gt;

&lt;p&gt;The game uses &lt;strong&gt;synthetic execution and teaching rates&lt;/strong&gt;. The chatbot has a separate live-provider path. The production pipeline described later is a reference design that extends the demo’s ideas. The screenshots and numbers below show the teaching application, with their evidence source identified.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the request the customer actually made
&lt;/h2&gt;

&lt;p&gt;Our fictional customer, Asha, asks about an order. ShopBot has customer-scoped, read-only tools for orders, shipments, profiles, policies and support cases. The visible experience is a support conversation. Underneath it, the application records the model invocations, tool work and evidence needed to explain the answer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchlgur2lvczdvp4l4lmu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchlgur2lvczdvp4l4lmu.png" alt="ShopBot customer support conversation in AI Cost Engineering Lab." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The final deployed ShopBot interface, shown with viewer access. The customer and records are fictional. Chat supports a separately selected live or synthetic execution source.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A single customer message can lead to several model calls. A tool result can become input to the next call. The next call can carry previous messages, tool definitions and policy passages again. A failed attempt may have already consumed billable usage before the application retries.&lt;/p&gt;

&lt;p&gt;That is why the first useful question is more specific than “which model did we use?”&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What did the complete request do, and what did that work cost?&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Read the receipt before choosing an optimization
&lt;/h2&gt;

&lt;p&gt;For a provider that bills these categories separately, a normalized model charge can be written as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;model charge =
    fresh input tokens × fresh input rate
  + cache-write tokens × cache-write rate
  + cache-read tokens × cache-read rate
  + output tokens × output rate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apply the rate’s unit conversion, such as dividing by one million for a per-million-token price. Then sum across every charged invocation and attempt in the request.&lt;/p&gt;

&lt;p&gt;The normalization matters. Providers can expose different usage fields and token inclusion rules. An adapter must map those fields into mutually exclusive billing categories before applying a dated price table. Adding cached tokens to an input total that already includes them would charge for the same work twice.&lt;/p&gt;

&lt;p&gt;Amazon Bedrock’s Converse API exposes usage, stop reasons and latency metrics. Its cache-read and cache-write fields give an application a way to retain the provider’s observations alongside its own trace. &lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html" rel="noopener noreferrer"&gt;Bedrock Converse response documentation&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I keep four distinctions in the receipt:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Provider usage&lt;/strong&gt; records the counters returned by the inference API.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Estimated composition&lt;/strong&gt; explains how much context appears to come from instructions, retrieval or tools. A character-based token estimate belongs here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calculated charge&lt;/strong&gt; applies a versioned price table to usage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Invoice reconciliation&lt;/strong&gt; accounts for what the provider actually billed, including applicable commercial terms.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Our lab calculates execution charges. A production cost model also needs to define how it treats embeddings, retrieval infrastructure, tool APIs, orchestration and human escalation. Keep that boundary attached to every number.&lt;/p&gt;

&lt;p&gt;If a required usage counter or price is unavailable, record an unknown value. Replacing it with zero makes the next comparison look cheaper without establishing any saving.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cost per resolved conversation changes the question
&lt;/h3&gt;

&lt;p&gt;Suppose two versions each process 100 conversations. This is an illustrative calculation:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Version A&lt;/th&gt;
&lt;th&gt;Version B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Total measured execution charge&lt;/td&gt;
&lt;td&gt;$1.00&lt;/td&gt;
&lt;td&gt;$0.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Conversations resolved under the rubric&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Charge per attempted conversation&lt;/td&gt;
&lt;td&gt;$0.0100&lt;/td&gt;
&lt;td&gt;$0.0080&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Charge per resolved conversation&lt;/td&gt;
&lt;td&gt;$0.0111&lt;/td&gt;
&lt;td&gt;$0.0133&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Version B spends less in total and more per successful outcome.&lt;/p&gt;

&lt;p&gt;The calculation is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cost per resolved conversation =
    total in-scope cost of all attempted conversations
    / number of conversations successfully resolved
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The numerator includes work spent on unsuccessful attempts. The resolution definition needs a versioned rubric. If nothing was resolved, the ratio is undefined and the release decision must handle that explicitly.&lt;/p&gt;

&lt;p&gt;For monthly planning, multiply expected conversation volume by expected cost per conversation. For a resolution target, account for the resolution rate and follow-up work. Report traffic volume separately so a larger workload does not masquerade as a per-request regression.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five places to investigate
&lt;/h2&gt;

&lt;p&gt;We organised the investigation around five layers. They are useful places to look for a cause. Their effects can overlap, so adding five independent “savings” estimates is unreliable.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What I would inspect&lt;/th&gt;
&lt;th&gt;A possible experiment&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;Exact model identity, routing, input/output rates, answer quality&lt;/td&gt;
&lt;td&gt;Route a bounded task to a smaller model and score the same cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context&lt;/td&gt;
&lt;td&gt;Stable instructions, message history, dynamic prefixes, cache observations&lt;/td&gt;
&lt;td&gt;Move changing data after the cache boundary&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval / RAG&lt;/td&gt;
&lt;td&gt;Retrieved passages, relevance, duplication, grounding&lt;/td&gt;
&lt;td&gt;Reduce redundant passages while checking answer coverage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tools&lt;/td&gt;
&lt;td&gt;Fields returned, payload size, repeated reads, failure behaviour&lt;/td&gt;
&lt;td&gt;Return the fields needed for this task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent behaviour&lt;/td&gt;
&lt;td&gt;Calls, attempts, retries, termination and escalation&lt;/td&gt;
&lt;td&gt;Bound the loop and inspect failed cases&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi0e5tn2q79sbq79vbl4w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi0e5tn2q79sbq79vbl4w.png" alt="Request Journey showing the customer, agent loop, retrieval, prompt builder, tools and quality check." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The Journey view animates recorded events from a completed synthetic run. Playing the trace does not make another model call.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The journey connects those layers. A broad tool response can increase context on several subsequent calls. A retrieval change can reduce input tokens and also remove the evidence needed to answer correctly. A smaller model can require more attempts. The full workflow decides whether the change helped.&lt;/p&gt;

&lt;h2&gt;
  
  
  Case #1 — The Expensive Greeting
&lt;/h2&gt;

&lt;p&gt;The game takes the investigator through &lt;strong&gt;Case file → Crime scene → Line-up → Lab → Verdict&lt;/strong&gt;. Each location has a job in the investigation.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faz51sda2hjyqk0iqxqs8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faz51sda2hjyqk0iqxqs8.png" alt="The detective map with the five locations for The Expensive Greeting." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The map makes the investigation sequence visible. The clues come from stored run evidence.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The crime scene establishes what changed
&lt;/h3&gt;

&lt;p&gt;CR-2041 is the fictional festival-weekend release. It introduces loyalty-tier greetings. We reproduce the healthy and changed versions on the same questions, then compare cost, quality and test volume.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmtu122yickzo0u4o6d6f.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmtu122yickzo0u4o6d6f.png" alt="The Crime scene showing before-and-after evidence for Model, Context, RAG, Tools and Agent." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This two-question reproduction is a diagnostic sample. The displayed charges use synthetic usage and teaching rates. It cannot establish a production regression rate.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The investigator opens evidence records, reads the before-and-after summaries and labels each contact. Those labels are notes. An accusation in the Line-up still has to survive a one-change lab test.&lt;/p&gt;

&lt;p&gt;The final game gives the player three Lives for wrong accusations, followed by unlimited attempts in Free Play. Those mechanics keep the investigation moving. They do not alter the evidence required to close the case.&lt;/p&gt;

&lt;h3&gt;
  
  
  The first clue is where the changing text appears
&lt;/h3&gt;

&lt;p&gt;Prompt caching depends on the content before the cache boundary remaining eligible for reuse. In this case, customer-specific material changes that prefix.&lt;/p&gt;

&lt;p&gt;The pattern looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Before CR-2041
  Stable instructions and tool definitions
  Cache boundary
  Customer-specific context and current question

After CR-2041
  Changing personalization
  Stable instructions and tool definitions
  Cache boundary
  Current question
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The teaching trace changes from a cache write followed by reads to repeated writes. Moving the varying material after the stable prefix is the Context experiment. Claude’s caching documentation describes exact prefix matching and explains why placing a breakpoint on changing content can keep producing fresh writes. &lt;a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching" rel="noopener noreferrer"&gt;Claude prompt caching documentation&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A production test also needs the provider’s minimum cacheable length, TTL, supported model and request format. A stable hash is evidence about prompt construction. Cache counters establish what the provider observed. An unknown cache observation stays unknown.&lt;/p&gt;

&lt;h3&gt;
  
  
  The second clue is what the profile tool returns
&lt;/h3&gt;

&lt;p&gt;The greeting needs a small projection of customer information. Returning the full profile can send unrelated fields into the prompt and carry them through later calls.&lt;/p&gt;

&lt;p&gt;A minimal response might look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"display_name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Asha"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"loyalty_tier"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Gold"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual allowlist follows the task and policy. If answering the question requires an order record, fetch that scoped record through the appropriate tool.&lt;/p&gt;

&lt;p&gt;Our Tools experiment changes the profile projection and measures the result. The lab also keeps a Tools-only variant. That lets the investigator compare the effect of fixing the payload while the context problem is still present.&lt;/p&gt;

&lt;p&gt;The lesson is to measure the combined change. Fixing one contributor does not prove the remaining request is within budget.&lt;/p&gt;

&lt;h3&gt;
  
  
  An alibi needs a boundary
&lt;/h3&gt;

&lt;p&gt;The same model identity before and after rules out a model switch in that comparison. It does not establish that every generation will behave identically.&lt;/p&gt;

&lt;p&gt;Similarly, a familiar successful-call count does not explain rejected attempts or retries. The game can leave Agent as &lt;strong&gt;Alibi unclear&lt;/strong&gt; when the evidence does not justify clearing it. A production investigation needs that same ability to withhold a conclusion.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the gate actually demonstrated
&lt;/h2&gt;

&lt;p&gt;The game’s larger gate trial uses 60 matched synthetic questions per comparison. This is separate from the two-question crime-scene reproduction.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F20gu2sywuneog3dpe5w7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F20gu2sywuneog3dpe5w7.png" alt="The release gate showing CR-2041 blocked and the version with both fixes passing." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Teaching verdicts from the final app. The values are calculated using synthetic execution and invented fixture rates.&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Teaching gate metric&lt;/th&gt;
&lt;th&gt;Healthy reference&lt;/th&gt;
&lt;th&gt;CR-2041&lt;/th&gt;
&lt;th&gt;Both fixes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Calculated charge per resolved chat&lt;/td&gt;
&lt;td&gt;$0.007263&lt;/td&gt;
&lt;td&gt;$0.011300&lt;/td&gt;
&lt;td&gt;$0.007504&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Eligible-call cache hit rate&lt;/td&gt;
&lt;td&gt;66.7%&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;66.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Profile output, p95 estimated tokens&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;1,616&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent steps, p95 successful model calls&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resolution under the fixture rubric&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The as-shipped version is &lt;strong&gt;BLOCK&lt;/strong&gt;. The version with both fixes is &lt;strong&gt;PASS&lt;/strong&gt; under the teaching policy. Its charge remains slightly above the reference but within the allowed increase.&lt;/p&gt;

&lt;p&gt;The implementation also carries an evidence-admissibility flag. A passing synthetic teaching verdict does not authorize a production release. The repository’s hosted cost-quality workflow exercises synthetic evaluations and gate plumbing. Production model replay, deployment approval and canary automation need their own implementation.&lt;/p&gt;

&lt;p&gt;That separation is where I would begin the production work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building a production cost-and-quality gate
&lt;/h2&gt;

&lt;p&gt;A gate needs an evidence-producing pipeline, a policy evaluator and an enforcement point that prevents release. Printing a red or green report only implements the middle part.&lt;/p&gt;

&lt;p&gt;The delivery path I would build is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PR and candidate artifact
       ↓
Gate 0 — static contracts and budget checks
       ↓
Gate 1 — bounded, matched real-model evaluation
       ↓
Gate 2 — evidence validation and policy decision
       ↓
Required check for the evaluated artifact
       ↓
Canary → observe → promote or roll back
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  1. Decide what success means before running the comparison
&lt;/h3&gt;

&lt;p&gt;Use a versioned evaluation set with known expectations. Include common support paths, ambiguous requests, tool failures, policy boundaries, long histories and cases that must escalate. Add slices for material production segments such as language or request category.&lt;/p&gt;

&lt;p&gt;Define resolution in observable terms. For an order-status question, that might require the correct order, the recorded shipment state and an estimate presented with the proper qualification. For a refund question, check eligibility, policy grounding and the permitted next action. A fluent answer alone does not satisfy those checks.&lt;/p&gt;

&lt;p&gt;Use deterministic checks for facts that the application can verify. If a model judge is useful for a broader rubric, version its model and prompt, calibrate it against human labels, and track disagreement. The judge’s output is another measurement with an error rate.&lt;/p&gt;

&lt;p&gt;Keep evaluation cases held out from prompt tuning. When a production incident becomes a regression case, version the new dataset and retain the previous one for comparison history.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Freeze the inputs and identify both artifacts
&lt;/h3&gt;

&lt;p&gt;A meaningful comparison records more than a branch name:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"candidate_digest"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sha256:&amp;lt;candidate artifact digest&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baseline_digest"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sha256:&amp;lt;approved baseline digest&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dataset_sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;versioned evaluation set hash&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"policy_sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;approved policy hash&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rubric_version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"support-resolution-v3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"price_table_version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"provider-region-date-version"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"evidence_source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"live_provider_evaluation"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cache_protocol"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"cold-and-warm-measured-separately"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These are example field names, not a record of a production deployment.&lt;/p&gt;

&lt;p&gt;Freeze the customer question, conversation history, tool fixtures and policy corpus for both sides. Pin model identifiers and inference parameters where the provider supports it. If model routing is the variable being tested, record that difference explicitly.&lt;/p&gt;

&lt;p&gt;A baseline must come from an approved registry of release artifacts. Allowing a candidate to select an unusually expensive reference or edit the acceptance policy would undermine the test.&lt;/p&gt;

&lt;p&gt;A content hash detects changed bytes. Establishing who produced the evidence also requires a trusted evaluator and authenticated artifact storage or attestations.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Run cheap checks first
&lt;/h3&gt;

&lt;p&gt;Gate 0 can reject obvious contract problems without buying inference:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Render prompts for different customers and clocks, then compare the designated stable prefix.&lt;/li&gt;
&lt;li&gt;Check model-visible tool payloads against a schema, field allowlist and size budget.&lt;/li&gt;
&lt;li&gt;Validate retrieval limits, output caps, retry limits and termination conditions.&lt;/li&gt;
&lt;li&gt;Check that a proposed model or route is allowed for the data classification.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These checks would detect the two specific construction problems built into our case. They cannot predict all runtime cost or quality behaviour.&lt;/p&gt;

&lt;p&gt;For payload sizing, record the tokenizer or estimator version. A character estimate can be a useful local budget check, while provider usage remains the source for billed model-token accounting.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Bound the real-model replay
&lt;/h3&gt;

&lt;p&gt;Gate 1 runs the candidate and baseline on matched cases. It should have a maximum number of calls, a reservation for worst-case spend, output-token caps, a deadline and a cancellation path.&lt;/p&gt;

&lt;p&gt;Account for retries and evaluator calls in the evaluation budget. Reserve spend before dispatching concurrent work so several workers cannot each approve against the same remaining balance. Keep final usage and outstanding reservations consistent when a worker times out.&lt;/p&gt;

&lt;p&gt;Use recorded tool responses when the goal is isolating prompt or agent behaviour. Use controlled integration tests when a tool contract itself changes. A replay of a historical model answer does not measure a new prompt’s effect on the model. Record what was replayed and what actually executed.&lt;/p&gt;

&lt;p&gt;Test cold and warm cache conditions separately. Interleave or randomize baseline/candidate order, control cross-run cache reuse, and document warm-up charges. Otherwise the version that ran second may inherit an advantage unrelated to its change.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Preserve every scheduled case and attempt
&lt;/h3&gt;

&lt;p&gt;The evidence contract should account for the complete scheduled set. A useful per-attempt record includes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;case ID and repetition ID
artifact and configuration identities
attempt number and status
provider request reference
usage categories and pricing version
calculated charge or missing-cost reason
latency, tool payload size and model-call count
rubric results and critical-failure flags
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Retain failures, timeouts and cancellations. Define whether a retry belongs to the same customer attempt and make that aggregation rule consistent. Reject duplicate identifiers and mismatched cohorts. Never compare only the intersection of successful cases and silently lose the difficult cases.&lt;/p&gt;

&lt;p&gt;Store raw traces behind access controls. CI reports can expose redacted summaries, case IDs and evidence links. Prompts and tool responses can contain customer data, so collecting full content needs an explicit data-handling policy. OpenTelemetry’s GenAI guidance provides a useful tracing vocabulary and treats content capture as an opt-in concern. &lt;a href="https://opentelemetry.io/blog/2026/genai-observability/" rel="noopener noreferrer"&gt;OpenTelemetry GenAI observability&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Choose thresholds with their meaning attached
&lt;/h3&gt;

&lt;p&gt;The lab policy provides a concrete example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Teaching values from the case, requiring calibration for production.&lt;/span&gt;
&lt;span class="na"&gt;minimum_matched_cases&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt;
&lt;span class="na"&gt;cost_per_resolved_max_increase&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.10&lt;/span&gt;
&lt;span class="na"&gt;cache_hit_rate_max_drop_percentage_points&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;15&lt;/span&gt;
&lt;span class="na"&gt;profile_payload_p95_max_increase&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.25&lt;/span&gt;
&lt;span class="na"&gt;profile_payload_p95_budget_tokens&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1000&lt;/span&gt;
&lt;span class="na"&gt;agent_steps_p95_max&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
&lt;span class="na"&gt;resolution_rate_floor&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.90&lt;/span&gt;
&lt;span class="na"&gt;resolution_rate_max_drop_percentage_points&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="na"&gt;critical_failures_max&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An increase from 100% to 98% is a two-percentage-point drop. A 10% relative cost increase uses a different calculation. Encode units in the schema and the report.&lt;/p&gt;

&lt;p&gt;A production policy should also define absolute spend, latency, coverage by important segment, permitted evidence sources and what happens when a metric is inapplicable. Treat cache hit rate as a diagnostic guardrail with a defined denominator. A deliberate cache trade-off can require a reviewed policy change while still being evaluated on cost and quality.&lt;/p&gt;

&lt;p&gt;Sixty is the demo’s minimum, not a universal sample-size recommendation. Choose production sample size around tolerated regression, variance, business risk and the cases you need to cover. Zero observed safety failures in a small sample does not establish a zero failure rate.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Handle model variability before deciding
&lt;/h3&gt;

&lt;p&gt;Repeated runs help distinguish a change from ordinary variation. Pair baseline and candidate by case. For repeated observations on the same conversation, resample or analyse at the conversation level so correlated attempts do not become fictitiously independent evidence.&lt;/p&gt;

&lt;p&gt;For cost per resolved conversation, one practical analysis is a paired bootstrap over cases. In each resample, calculate each side’s total cost divided by its resolved count, then calculate the ratio. A resample with no resolutions needs an explicit undefined-result policy.&lt;/p&gt;

&lt;p&gt;Choose the statistical rule in advance. For example, require the upper confidence bound of the cost ratio to remain below the allowed increase, alongside the quality constraints. If the interval crosses the boundary, return &lt;strong&gt;INCONCLUSIVE&lt;/strong&gt; and use the predetermined additional-evidence procedure. Re-running until one batch turns green invalidates the rule.&lt;/p&gt;

&lt;p&gt;Use an equivalent uncertainty treatment for resolution and other noisy metrics. Review important slices and tail behaviour as well as the overall mean. A global saving can hide a serious regression in one support path.&lt;/p&gt;

&lt;h3&gt;
  
  
  8. Make the decision fail closed
&lt;/h3&gt;

&lt;p&gt;The decision model needs at least these outcomes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Outcome&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;th&gt;Release action&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;INVALID&lt;/td&gt;
&lt;td&gt;Schema, identity or evidence-integrity checks failed&lt;/td&gt;
&lt;td&gt;Refuse and repair the evidence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BLOCK&lt;/td&gt;
&lt;td&gt;A known policy or critical-safety constraint failed&lt;/td&gt;
&lt;td&gt;Refuse and fix the change&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;INCONCLUSIVE&lt;/td&gt;
&lt;td&gt;Required coverage, metrics or statistical confidence are missing&lt;/td&gt;
&lt;td&gt;Refuse automatic promotion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WARN&lt;/td&gt;
&lt;td&gt;An explicitly defined review band was reached&lt;/td&gt;
&lt;td&gt;Follow the documented review policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;td&gt;Every required condition is satisfied&lt;/td&gt;
&lt;td&gt;Permit the next release stage&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is the shape of the decision core after evidence validation and metric computation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;decide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validation&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;validation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;INVALID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="n"&gt;failures&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;evaluate_known_hard_limits&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;failures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BLOCK&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;complete&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;coverage_sufficient&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;INCONCLUSIVE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uncertainty_within_policy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;INCONCLUSIVE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;in_review_band&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;WARN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is decision pseudocode. The validator, metric calculations and uncertainty checks are explicit responsibilities of the evaluator. They must also reject an evidence source that the release policy does not admit.&lt;/p&gt;

&lt;p&gt;Only an explicitly admissible PASS should produce a successful release check. A missing report, parser error, cancelled evaluation or controller crash must leave promotion blocked. Upload a sanitized report even when evaluation fails, while preserving the failing exit status.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. Put credentials and authority in a trusted control plane
&lt;/h3&gt;

&lt;p&gt;I would separate the production gate into three roles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Candidate execution&lt;/strong&gt; runs the application artifact in an isolated sandbox, with fixture data and tightly bounded tool access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evaluation control&lt;/strong&gt; selects approved datasets and policy, meters inference through a constrained service, and writes authenticated results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Release enforcement&lt;/strong&gt; validates the results and publishes the required check for the exact candidate identity.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The candidate must not receive a credential that can change policy, overwrite the baseline, forge usage, or mark itself approved. Keep provider credentials in the metered inference service and reconcile its request records with the run evidence.&lt;/p&gt;

&lt;p&gt;Untrusted PR code also needs isolation from CI secrets and write tokens. GitHub documents how checking out and executing PR code in a privileged &lt;code&gt;pull_request_target&lt;/code&gt; workflow can expose the base repository’s credentials. A label or review on an earlier revision does not make later pushed code safe. &lt;a href="https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target" rel="noopener noreferrer"&gt;GitHub Actions security guidance&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For an initial implementation, a trusted controller can accept a reviewed candidate digest, start the bounded evaluation and await its attested result. The submission contract can be small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"repository"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"approved-repository"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"candidate_sha"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;full commit SHA&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"candidate_digest"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sha256:&amp;lt;reviewed image digest&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baseline_release"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"approved-baseline-v12"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"evaluation_suite"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"support-heldout-v3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"request_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;unique idempotency key&amp;gt;"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server resolves the dataset, price table and policy from trusted versions. It validates artifact provenance, authorizes the requester, reserves budget and records the job. Retries with the same idempotency key return the same evaluation identity. The candidate cannot supply a larger budget in this request.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Connect the verdict to the merge and deploy controls
&lt;/h3&gt;

&lt;p&gt;Create a required check such as &lt;code&gt;ai-cost-quality/release&lt;/code&gt; through the trusted controller’s GitHub App. Attach it to the evaluated commit and retain the artifact digest in the report. Configure the expected App as the source of that required check.&lt;/p&gt;

&lt;p&gt;A new commit needs fresh evidence. If the merge result differs from the evaluated artifact, evaluate that result or rebuild and verify its identity before deployment. Merge queues need checks for their merge-group commits. GitHub’s documentation covers latest-commit requirements, expected check sources and the &lt;code&gt;merge_group&lt;/code&gt; trigger. &lt;a href="https://docs.github.com/en/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks" rel="noopener noreferrer"&gt;Required status checks&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Protect changes to the evaluator, policy and workflow with designated reviewers. Make any bypass explicit, time-limited and attributable. Apply the repository’s restrictions to the relevant privileged roles as well. GitHub provides branch protection and ruleset controls for this enforcement. &lt;a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches" rel="noopener noreferrer"&gt;Protected branches&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The deployment job checks that the approved digest, policy version and evidence validity period still match. It promotes that exact image. It must not rebuild an unreviewed artifact after the gate has passed.&lt;/p&gt;

&lt;h3&gt;
  
  
  11. Keep the production observation loop
&lt;/h3&gt;

&lt;p&gt;An evaluation set cannot represent every future conversation. Roll out to a bounded canary cohort with a defined observation window, spend limit and rollback procedure.&lt;/p&gt;

&lt;p&gt;Track cost per attempted and resolved conversation, resolution and escalation, critical failures, latency, calls, retries, tool payloads and cache observations. Compare like traffic segments and cache conditions. Quality labels can arrive later than usage, so define how long promotion waits for them and how late failures trigger rollback.&lt;/p&gt;

&lt;p&gt;Tie the rollback to a known-good artifact and configuration. Test the mechanism before relying on it. Also define failure behaviour when telemetry disappears or the rollout controller stops responding.&lt;/p&gt;

&lt;p&gt;An exception should identify the failed criterion, the accountable approver, the permitted scope and the expiry. That keeps an accepted risk distinguishable from a silent green check.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away from building the lab
&lt;/h2&gt;

&lt;p&gt;The detective game made it possible to open a clue, make an accusation and test an explanation. The underlying method applies without the game UI:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Measure → Diagnose → Optimize → Validate → Monitor.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Measure the whole attempted workflow. Diagnose through the request trace. Change a bounded control. Validate outcome quality beside cost. Keep monitoring after release.&lt;/p&gt;

&lt;p&gt;The production gate is the delivery version of that method. It needs enough evidence to justify a decision and enough authority to enforce it.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you to Balaji Saravanan, the Presidio India Exchange organisers and everyone who joined the session.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The sandbox used for the talk has since been torn down. These screenshots preserve the application and its teaching examples. The next step for this work is applying the evaluation and enforcement design to a workload with an approved production dataset, cost boundary and release policy.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written with AI assistance. The application behaviour and teaching figures were checked against the project’s implementation and final release evidence.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>finops</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Getting the Actual Client IP When Using Application Load Balancer (ALB) in AWS Lambda</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Sat, 20 Jul 2024 12:37:40 +0000</pubDate>
      <link>https://dev.to/rkj180220/getting-the-actual-client-ip-when-using-application-load-balancer-alb-in-aws-lambda-4fcn</link>
      <guid>https://dev.to/rkj180220/getting-the-actual-client-ip-when-using-application-load-balancer-alb-in-aws-lambda-4fcn</guid>
      <description>&lt;p&gt;When I was new to AWS, I faced an interesting challenge while working on a task to digitally sign a document, which required the client's IP as part of the e-signature. Initially, I was thrilled when the implementation seemed to work perfectly the first time. However, my excitement was short-lived. During testing, I noticed that the same IP address was being returned, even when I accessed the application from different machines. It was then that I realized the IP address I was receiving was not the actual client IP but the IP of the load balancer.&lt;/p&gt;

&lt;p&gt;This discovery led me down a path of investigation and learning. I had to dig deeper to understand what was happening and how to retrieve the real client IP. In this blog, I will share my experience and provide a comprehensive guide on how to achieve this using AWS Lambda and Python, ensuring you can accurately capture the client’s IP address when using an Application Load Balancer (ALB).&lt;/p&gt;

&lt;h3&gt;
  
  
  Understanding the Challenge
&lt;/h3&gt;

&lt;p&gt;When a client makes a request to your application through an ALB, the load balancer acts as an intermediary. Consequently, the IP address your application sees is that of the ALB, not the client's. To address this, ALB includes the client's IP in the X-Forwarded-For HTTP header. This header can contain multiple IP addresses in case the request has passed through multiple proxies.&lt;/p&gt;

&lt;p&gt;Here's what we need to handle:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Extract the Client IP&lt;/strong&gt;: Retrieve and parse the &lt;code&gt;X-Forwarded-For&lt;/code&gt;  header.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Handle Multiple IPs&lt;/strong&gt;: Ensure that we get the correct client IP even when multiple proxies are involved.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Security Consideration
&lt;/h3&gt;

&lt;p&gt;The X-Forwarded-For header should be used with caution due to potential security risks. The entries can only be considered trustworthy if added by systems that are properly secured within the network. This ensures that the client IPs are not tampered with and are reliable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Choosing the Right Tools
&lt;/h3&gt;

&lt;h4&gt;
  
  
  AWS Lambda and Python
&lt;/h4&gt;

&lt;p&gt;AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Python, with its simplicity and readability, is an excellent choice for handling this task within a Lambda function.&lt;/p&gt;

&lt;h4&gt;
  
  
  Key Components
&lt;/h4&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;AWS Lambda Function&lt;/strong&gt;: The core function that processes incoming requests.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Application Load Balancer (ALB)&lt;/strong&gt;: The load balancer that forwards requests to the Lambda function.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Implementation Details
&lt;/h3&gt;

&lt;h4&gt;
  
  
  Setting Up AWS Lambda with ALB
&lt;/h4&gt;

&lt;p&gt;First, ensure your Lambda function is set up and integrated with an ALB. Follow AWS's official guide if needed: &lt;a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html" rel="noopener noreferrer"&gt;Using Lambda functions as targets for Application Load Balancer.&lt;/a&gt;&lt;/p&gt;

&lt;h4&gt;
  
  
  Lambda Function Code
&lt;/h4&gt;

&lt;p&gt;Let's dive into the Python code for the Lambda function. This function will extract the client's IP address from the X-Forwarded-For header.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import json

def lambda_handler(event, context):
    # Extract the 'X-Forwarded-For' header
    x_forwarded_for = event['headers'].get('x-forwarded-for')

    if x_forwarded_for:
        # The first IP in the list is the client's IP
        client_ip = x_forwarded_for.split(',')[0]
    else:
        # Fallback if header is not present
        client_ip = event['requestContext']['identity']['sourceIp']

    # Log the client IP
    print(f"Client IP: {client_ip}")

    # Respond with the client IP
    return {
        'statusCode': 200,
        'body': json.dumps({'client_ip': client_ip})
    }

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  Explanation
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Extract the Header: Retrieve the X-Forwarded-For header from the incoming request.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Parse the Header: Take the first IP, which represents the client's original IP.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Fallback Mechanism: Use the source IP from the request context if the header is not present.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Logging and Response: Log and return the client's IP for verification.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Example Request and Response
&lt;/h4&gt;

&lt;p&gt;Request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
    "headers": {
        "x-forwarded-for": "203.0.113.195, 70.41.3.18, 150.172.238.178"
    },
    "requestContext": {
        "identity": {
            "sourceIp": "70.41.3.18"
        }
    }
}

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Response:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
    "client_ip": "203.0.113.195"
}

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Conclusion
&lt;/h3&gt;

&lt;p&gt;Identifying the actual client IP in an AWS Lambda function behind an ALB requires careful handling of the &lt;code&gt;X-Forwarded-For&lt;/code&gt; header. This approach ensures accurate IP logging and enhances the application's ability to personalize and secure user interactions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Refrences
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html" rel="noopener noreferrer"&gt;AWS ALB Documentation:&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html" rel="noopener noreferrer"&gt;Python in AWS Lambda:&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For" rel="noopener noreferrer"&gt;HTTP Headers Explained&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>aws</category>
      <category>python</category>
      <category>serverless</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Coding Efficiency: Designing a High-Performance Data Import System with Excel Integration</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Wed, 20 Mar 2024 10:20:24 +0000</pubDate>
      <link>https://dev.to/rkj180220/coding-efficiency-designing-a-high-performance-data-import-system-with-excel-integration-3ng5</link>
      <guid>https://dev.to/rkj180220/coding-efficiency-designing-a-high-performance-data-import-system-with-excel-integration-3ng5</guid>
      <description>&lt;h2&gt;
  
  
  Introduction: Bridging the Data Gap - Efficiently Importing Data from Other Systems
&lt;/h2&gt;

&lt;p&gt;In today's digital world, our applications often need to work together. Imagine you have a customer relationship management (CRM) system that stores all your customer information, but you also have a separate system for tracking sales and orders. To get a complete picture of your business, you might need to import data (like customer details) from one system into the other. This process of bringing data into a system from an external source is called a data import.&lt;/p&gt;

&lt;p&gt;Data imports can be crucial for making informed decisions. For example, by combining your customer information with your sales data, you can analyze trends and identify your most valuable customers. But how do you actually move this data between systems?&lt;/p&gt;

&lt;p&gt;There are several ways to approach data migration, each with its own advantages and limitations.  This blog post will explore how you can design a data import system using a familiar tool: Microsoft Excel.&lt;/p&gt;

&lt;p&gt;We'll delve into strategies for handling large datasets efficiently, ensuring data accuracy, and creating a user-friendly experience for the import process. By leveraging the right techniques, you can unlock the power of Excel import and bridge the data gap between your systems seamlessly.&lt;/p&gt;

&lt;h3&gt;
  
  
  Navigating the Maze of Data Migration
&lt;/h3&gt;

&lt;p&gt;Moving data between different systems can feel like navigating a maze. There are several approaches you can take, each with its own pros and cons. Let's explore some common methods for data import:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Manual Creation&lt;/strong&gt;: Imagine having to type in all your customer information one by one from a spreadsheet into your CRM system. This is manual data creation, and while it works for very small datasets, it's time-consuming, error-prone, and not scalable for large amounts of data.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Custom Scripts&lt;/strong&gt;: For more complex data transfers, you might consider writing custom scripts or programs. This approach offers a lot of flexibility, but they lack the efficiency and can be time-consuming to develop and maintain.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;API Integration&lt;/strong&gt;: Many applications offer APIs (Application Programming Interfaces) that allow them to talk to each other.  Imagine an API as a special translator between your CRM system and your sales data system. By using the API, you can build a connection that lets you import data automatically.  However, using APIs often requires development effort for both sides and may not be available for all systems.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Excel Import&lt;/strong&gt;: Simple and ubiquitous, but raises concerns about scalability and data integrity, especially for high volumes.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Excel Import: Reimagined for Efficiency
&lt;/h2&gt;

&lt;p&gt;While Excel import is a familiar and accessible solution for bringing data into your system, it can face challenges when dealing with large datasets or complex data validation needs.  Imagine you have a massive customer list with thousands of entries in an Excel spreadsheet.  Traditionally, importing this data might take a long time or lead to errors if the data isn't formatted correctly.&lt;/p&gt;

&lt;p&gt;Here's where we can reimagine Excel import for efficiency:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;High-Volume Processing&lt;/strong&gt;:  By leveraging specialized techniques, we can optimize the import process to handle large datasets without sacrificing performance or system stability.  This might involve splitting the data into smaller chunks for processing or using efficient algorithms for data validation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Robust Validation and Error Handling&lt;/strong&gt;: Data accuracy is crucial.  Our system can perform thorough checks on your Excel data to ensure it meets the requirements of your target system.  For example, it can verify data types (like numbers or dates) and identify any inconsistencies or missing information.  This helps catch errors early on and prevents inaccurate data from entering your system.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Streamlined Workflow&lt;/strong&gt;: The data import process shouldn't be a hassle.  We can design a user-friendly interface that guides you through each step, from configuring the import to uploading your Excel file.  This makes it easy for anyone, even beginners, to import data efficiently.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By incorporating these features, we can transform Excel import from a simple data transfer tool into a robust and efficient system.  Imagine being able to import your large customer list with confidence, knowing the system will handle the process smoothly and ensure the accuracy of your data.&lt;/p&gt;

&lt;h2&gt;
  
  
  System Design
&lt;/h2&gt;

&lt;p&gt;Let's delve into the design section for an efficient data import system using Excel, addressing key concerns such as:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Dynamic Field Mapping: Empowering User Flexibility
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flo5kgqulvb93jmvf04c6.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flo5kgqulvb93jmvf04c6.jpeg" alt="Field Mapping" width="800" height="96"&gt;&lt;/a&gt;&lt;br&gt;
One of the key challenges in data import systems is accommodating the diverse structure of data from different sources. This is where dynamic field mapping plays a pivotal role. Here's a closer look at how it works:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;User-Driven Configuration&lt;/strong&gt;: Instead of relying on fixed mappings, users are empowered to configure the import process based on their specific needs. Upon initiating a data import task, the system prompts users to define mappings between the columns in the Excel file and corresponding fields in the destination system.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Flexible Mapping Interface&lt;/strong&gt;: To ensure user-friendly interaction, the system provides an intuitive interface for mapping configuration. Users are presented with a list of available fields in the destination system, allowing them to easily match them with the columns in the Excel file. Additionally, the interface supports dynamic suggestions and auto-completion to streamline the mapping process.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Customizable Mapping Profiles&lt;/strong&gt;: Recognizing that users may need to import data from multiple sources with varying structures, the system supports the creation of customizable mapping profiles. Users can define and save multiple mapping configurations, each tailored to a specific source or import scenario. This flexibility allows for efficient handling of diverse data formats without the need for repetitive mapping tasks.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Validation and Error Handling&lt;/strong&gt;: To ensure data accuracy and integrity, the system performs validation checks during the mapping configuration phase. This includes verifying the consistency of data types, detecting potential mismatches between source and destination fields, and flagging any inconsistencies or conflicts for user review. Comprehensive error handling mechanisms are in place to guide users in resolving mapping issues effectively.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;To ensure seamless configuration and mapping between Excel data and the database, a structured approach leveraging MySQL is adopted. Here's how the database schema facilitates the process:&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;1.&lt;strong&gt;Import Configuration Table&lt;/strong&gt;: This table stores various import configurations available in the system, enabling users to manage different import scenarios effectively.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CREATE TABLE `import_configuration` (
  `import_configuration_id` int UNSIGNED NOT NULL,
  `configuration_name` varchar(55) NOT NULL,
  `created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `is_deleted` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`import_configuration_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;2.&lt;strong&gt;Configuration Mapping Table&lt;/strong&gt;: This table establishes mappings between Excel columns and database fields for each import configuration. It defines column titles, mandatory fields, sort order, and default values, facilitating accurate data mapping.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CREATE TABLE `import_mapping_columns` (
  `import_mapping_column_id` int UNSIGNED NOT NULL,
  `import_configuration_id` int UNSIGNED NOT NULL,
  `import_column_name` varchar(55) DEFAULT NULL,
  `import_column_title` varchar(100) NOT NULL,
  `is_mandatory` enum('0','1') NOT NULL,
  `sort_order` int NOT NULL,
  `default_value` json DEFAULT NULL,
  `is_deleted` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`import_mapping_column_id`),
  FOREIGN KEY (`import_configuration_id`) REFERENCES `import_configuration`(`import_configuration_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2.File Upload Phase: Streamlining Data Entry
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fxu2eexsoxrypda0gck2o.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fxu2eexsoxrypda0gck2o.jpeg" alt="File Upload Phase" width="331" height="1781"&gt;&lt;/a&gt;&lt;br&gt;
The file upload phase initiates the import process, ensuring seamless integration of Excel data into the system. Here's a concise overview of this crucial phase:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Configuration Preparation&lt;/strong&gt;: Users begin by configuring the import process, selecting the appropriate import configuration from the predefined options. This step ensures that the system knows how to interpret the incoming Excel data.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Excel Upload&lt;/strong&gt;: Once configured, users upload the Excel file containing the data to be imported. The system validates the file format and size to ensure compatibility and prevent data loss.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Storage in S3&lt;/strong&gt;: The uploaded file is securely stored in an Amazon S3 bucket, providing scalability, reliability, and easy access for subsequent processing steps.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Background Task Creation&lt;/strong&gt;: Concurrently, the system creates a background task or job to manage the import process efficiently. This task orchestrates the various steps involved in reading, validating, and importing the Excel data into the system.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  3. Excel Reading and Temporary Record Creation: Ensuring Data Integrity
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffwz7yaa630639eqrnj8a.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffwz7yaa630639eqrnj8a.jpeg" alt="Excel Reading and Temporary Record Creation: Ensuring Data Integrity" width="800" height="2323"&gt;&lt;/a&gt;&lt;br&gt;
In the Excel Reading and Temporary Record Creation phase, the system processes the uploaded Excel file, validates its contents, and creates temporary records for further validation. Here's a detailed breakdown of this crucial step:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;CRON Picks up the Background Task&lt;/strong&gt;: The scheduled task responsible for managing the import process retrieves the Excel file from the designated Amazon S3 bucket. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Updating Task Status&lt;/strong&gt;: To prevent duplicate processing, the system updates the status of the background task to "in progress." This prevents multiple instances from picking up the same task simultaneously, maintaining data integrity.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Heading Validation&lt;/strong&gt;: The system verifies the headers of the Excel file to ensure they match the expected format defined in the import configuration. Any discrepancies are flagged for user attention, preventing data misinterpretation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Excel Reading&lt;/strong&gt;: Using specialized libraries or tools, the system reads the data from the Excel file, extracting each row as a separate dataset for further processing. This step ensures accurate data extraction while handling large volumes efficiently.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Row Validation&lt;/strong&gt;: Each row of data undergoes validation against predefined rules and constraints specified in the import configuration. Mandatory fields, data types, and format checks are performed to identify any inconsistencies or errors. Use hashmaps or some md5 hashing mechanisms for getting duplicate entries within the excel file validation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Splitting into Chunks&lt;/strong&gt;: To optimize performance and manage memory usage, the dataset is split into manageable chunks for bulk insertion into the database. This ensures efficient processing, especially with large datasets.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bulk Insertion&lt;/strong&gt;: Validated data is bulk-inserted into temporary tables within the database, segregating valid records from exceptions. This step lays the groundwork for further validation and refinement of the imported data.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Within System Validations&lt;/strong&gt;: Concurrently, the system performs additional validations within the database environment, such as duplicate checks and cross-referencing against existing records. This ensures data consistency and integrity at every stage of the import process.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Temporary Records Creation&lt;/strong&gt;: Validated records are stored as temporary entries within the database, providing a snapshot of the imported data for subsequent review and refinement. Exception records are also retained for further investigation and correction.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By meticulously executing the Excel Reading and Temporary Record Creation phase, the system establishes a solid foundation for data integrity, enabling accurate and reliable integration of external data into the system.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    public function map($row): array
    {
        $this-&amp;gt;trimValues($row);

        $row['failed_columns'] = array();
        $row['validation_message'] = array();
        $row['import_status'] = 1;

        $this-&amp;gt;setDefaultValues($row);
        $this-&amp;gt;validate($row)

        $row['import_status'] = count($row['failed_columns']) &amp;gt; 0 ? 0 : 1;
        $row['failed_columns'] = json_encode($row['failed_columns']);
        $row['validation_message'] = !empty($row['validation_message']) ? implode(", ", $row['validation_message']) : null;

        return $row;
    }

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;// remove white spaces and non-breaking spaces
    protected function trimValues(&amp;amp;$row): void
    {
        $row = array_map(function ($value) {
            // For numeric values accept zero.
            if ($value == 0) {
                return $value;
            }
            return !empty(trim($value)) ? trim(str_replace("\xc2\xa0", '', $value)) : NULL;
        }, $row);
    }

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;public function registerEvents(): array
    {
        return [
            AfterImport::class =&amp;gt; function () {
                $this-&amp;gt;updateImportDetails();
                $this-&amp;gt;updateTimesheetImportCount();
            },
        ];
    }

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;// Insert the chunked Data
public function array(array $array)
{
    ImportDetails::insert($array);
    return $array;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    public function chunkSize(): int
    {
        return $this-&amp;gt;chunk_size;
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Verification &amp;amp; Real Record Creation: Validating and Finalizing Imported Data
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F80ok77e0l7cpoohl6ko9.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F80ok77e0l7cpoohl6ko9.jpeg" alt="Verification &amp;amp; Real Record Creation: Validating and Finalizing Imported Data" width="332" height="1339"&gt;&lt;/a&gt;&lt;br&gt;
After the temporary records are created from the Excel data, the system proceeds to verify and finalize the imported data. This phase ensures that only accurate and validated records are permanently stored in the system. Here's a detailed explanation of this crucial step:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Validate Imported Records&lt;/strong&gt;: All temporary records generated from the Excel data undergo thorough validation checks. This includes verifying data consistency, ensuring adherence to business rules, and detecting any anomalies or discrepancies.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Fix Exception Records&lt;/strong&gt;: If any validation errors are identified during the verification process, the system prompts users to review and rectify the exception records. Users can correct data inaccuracies or provide missing information to ensure completeness and accuracy.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Background Task for Real Record Generation&lt;/strong&gt;: Once the imported data passes validation, a background task is initiated to generate real records from the validated entries. This task orchestrates the finalization process, ensuring data integrity before permanent storage.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;CRON Picks Up the Task&lt;/strong&gt;: The scheduled task manager periodically picks up the task for real record creation, ensuring timely processing and minimal delay in data integration.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Validate for Duplicate Records&lt;/strong&gt;: Before creating real records in the database, the system performs a final check to ensure that no duplicate entries exist. This prevents redundancy and maintains data consistency within the system.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Construct Insertion Query&lt;/strong&gt;: Based on the validated temporary records, the system constructs an optimized insertion query using efficient bulk-insert techniques. This minimizes database overhead and maximizes performance during the final data insertion step.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bulk Insertion of Real Records&lt;/strong&gt;: The validated and deduplicated records are bulk-inserted into the system's database tables, finalizing the data import process. This step ensures that only accurate and verified data is permanently stored for use within the application.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fggrh49q8uyqcc9att3yd.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fggrh49q8uyqcc9att3yd.jpeg" alt="Real Record Creation" width="800" height="1672"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;By meticulously verifying and finalizing imported data, the system ensures data accuracy, integrity, and consistency, laying the groundwork for effective data-driven decision-making and business operations. This phase marks the culmination of the data import process, ensuring that the system is equipped with reliable and trustworthy information for optimal performance and functionality.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;In the dynamic landscape of data management, the seamless integration of external data into our systems is crucial. Throughout this journey, we've delved into the intricacies of designing a high-performance data import system, focusing on the versatility of Excel and the efficiency of our approach.&lt;/p&gt;

&lt;p&gt;From configuring import settings to finalizing real records, each phase of the data import process is pivotal in ensuring accuracy, integrity, and efficiency. Our commitment to optimization extends beyond functionality to include minimizing database hits and connections, while also striving to maintain O(n) time complexity.&lt;/p&gt;

&lt;p&gt;Through dynamic field mapping and a structured database schema, users can tailor import configurations to their needs, fostering flexibility and customization. This empowers organizations to navigate the complexities of data migration with confidence.&lt;/p&gt;

&lt;p&gt;During file upload, we prioritize compatibility and security, laying the groundwork for efficient processing. Subsequent phases, from Excel reading to temporary record creation and verification, ensure meticulous validation with minimal database interactions.&lt;/p&gt;

&lt;p&gt;As we conclude, it's clear that a well-designed data import system enhances operational efficiency and underpins informed decision-making. By embracing innovative solutions and optimization strategies, we navigate the complexities of data migration, driving growth and innovation in the digital age while ensuring minimal database overhead and optimal performance. Together, let's continue harnessing the power of efficient data integration to unlock new possibilities and opportunities.&lt;br&gt;
Feel free to post your ideas as to how you would process the data imports.&lt;/p&gt;

</description>
      <category>programming</category>
      <category>systemdesign</category>
      <category>dataimports</category>
      <category>excel</category>
    </item>
    <item>
      <title>General Docker Troubleshooting, Best Practices &amp; Where to Go From Here</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Fri, 19 Jan 2024 13:00:00 +0000</pubDate>
      <link>https://dev.to/rkj180220/general-docker-troubleshooting-best-practices-where-to-go-from-here-1gj6</link>
      <guid>https://dev.to/rkj180220/general-docker-troubleshooting-best-practices-where-to-go-from-here-1gj6</guid>
      <description>&lt;h2&gt;
  
  
  Tackling Common Issues: No Space and Sluggish Containers
&lt;/h2&gt;

&lt;p&gt;Addressing common issues is crucial for maintaining a seamless Docker experience. Let's highlight the importance and break down each of the upcoming sections:&lt;/p&gt;

&lt;h3&gt;
  
  
  No Space Issue:
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Image Cleanup:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;To remove unused images:&lt;br&gt;
&lt;/p&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker rmi &amp;lt;img1-id&amp;gt; &amp;lt;img2-id&amp;gt; &amp;lt;img3-id&amp;gt; ..
&lt;/code&gt;&lt;/pre&gt;

&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Execute system prune:&lt;br&gt;
&lt;/p&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker system prune
&lt;/code&gt;&lt;/pre&gt;

&lt;/li&gt;
&lt;li&gt;&lt;p&gt;This command intelligently removes unnecessary data, including stopped containers and obsolete layers.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Container Performance:
&lt;/h3&gt;

&lt;p&gt;1.&lt;strong&gt;Monitoring Container Stats:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Use the &lt;code&gt;docker stats&lt;/code&gt; command:&lt;br&gt;
&lt;/p&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker stats &amp;lt;name/id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Provides real-time snapshots of your container's performance.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;2.&lt;strong&gt;Inspecting Container Details:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Explore detailed information in JSON format:&lt;br&gt;
&lt;/p&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker inspect &amp;lt;container name/id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;


&lt;ul&gt;
&lt;li&gt;For less detailed output:
&lt;/li&gt;
&lt;/ul&gt;

&lt;pre class="highlight shell"&gt;&lt;code&gt;  docker inspect &amp;lt;container name/id&amp;gt; | less
&lt;/code&gt;&lt;/pre&gt;


&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;3.&lt;strong&gt;Viewing Running Processes:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker top &amp;lt;container name/id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Reveals running processes inside the container.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Docker Best Practices
&lt;/h2&gt;

&lt;p&gt;Adhering to best practices is paramount for ensuring security and stability within your Docker environment:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Image Security:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Use verified images for enhanced security.&lt;/li&gt;
&lt;li&gt;Leverage Container Image Scanners like Clair, Trivy, and Dagda for unverified images.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Avoid Using Latest:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Specify image versions to maintain stability.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;User Privileges:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Utilize non-root users within containers for increased security.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Where to Go Next
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Docker Compose:
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Multi-Container Apps:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docker excels for single-application deployment.&lt;/li&gt;
&lt;li&gt;For apps with multiple components, use Docker Compose to define containers and their relationships in a single Compose Manifest.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Getting Started:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Start containers with:
&lt;/li&gt;
&lt;/ul&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker-compose up
&lt;/code&gt;&lt;/pre&gt;

&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Kubernetes:
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Scaling for Production:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docker eases single-system container deployment.&lt;/li&gt;
&lt;li&gt;Kubernetes addresses challenges in managing hundreds of thousands of containers in production.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Kubernetes Challenges:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Overcoming difficulties in linking docker networks across hosts.&lt;/li&gt;
&lt;li&gt;Managing containers across multiple hosts.&lt;/li&gt;
&lt;li&gt;Lack of built-in solutions for container migration between hosts.&lt;/li&gt;
&lt;li&gt;Production concerns like load balancing and securing traffic are challenging with Docker client alone.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Container Orchestrations:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Solutions like VMware’s vCenter and Rundeck utilize orchestrators for easy scaling, container movement, and traffic routing.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h4&gt;
  
  
  Planet-Scale Container Orchestrator - Kubernetes:
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes automates deployment, scaling, and management of containerized applications.&lt;/li&gt;
&lt;li&gt;It's designed for distributed systems, running components across multiple machines.&lt;/li&gt;
&lt;li&gt;Enables auto-scaling and URL-based traffic routing, making it a platform of platforms.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion:
&lt;/h2&gt;

&lt;p&gt;Navigating Docker troubleshooting, adopting best practices, and exploring advanced orchestration tools are essential steps in mastering containerization. Whether you choose Docker Compose for multi-container simplicity or venture into the vast landscape of Kubernetes, each path unlocks new possibilities for deploying and managing your applications.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fww8sh54qhgn6dzh5yuvm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fww8sh54qhgn6dzh5yuvm.png" alt="Containerization Evolution" width="800" height="368"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Docker Documentation. &lt;a href="https://docs.docker.com/engine/reference/commandline/system_prune/" rel="noopener noreferrer"&gt;Docker system prune.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Docker Documentation. &lt;a href="https://docs.docker.com/storage/" rel="noopener noreferrer"&gt;Manage data in containers.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Clair. &lt;a href="https://github.com/quay/clair" rel="noopener noreferrer"&gt;Vulnerability Static Analysis for Containers.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Trivy. &lt;a href="https://github.com/aquasecurity/trivy" rel="noopener noreferrer"&gt;A Simple and Comprehensive Vulnerability Scanner for Containers.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Dagda. &lt;a href="https://github.com/eliasgranderubio/dagda" rel="noopener noreferrer"&gt;A tool to perform static analysis of known vulnerabilities, trojans, viruses, malware &amp;amp; other malicious threats in Docker images/containers.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Docker Compose Documentation. &lt;a href="https://docs.docker.com/compose/" rel="noopener noreferrer"&gt;Overview of Docker Compose.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kubernetes Documentation. &lt;a href="https://kubernetes.io/docs/tutorials/kubernetes-basics/" rel="noopener noreferrer"&gt;Kubernetes Basics.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;VMware. &lt;a href="https://www.vmware.com/products/vcenter-server.html" rel="noopener noreferrer"&gt;vCenter Server.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rundeck Documentation. &lt;a href="https://docs.rundeck.com/docs/manual/introduction.html" rel="noopener noreferrer"&gt;Introduction to Rundeck.&lt;/a&gt;
&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>docker</category>
      <category>kubernetes</category>
      <category>tutorial</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Create Containers and Interaction in Docker</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Wed, 17 Jan 2024 13:00:00 +0000</pubDate>
      <link>https://dev.to/rkj180220/create-containers-and-interaction-in-docker-3hac</link>
      <guid>https://dev.to/rkj180220/create-containers-and-interaction-in-docker-3hac</guid>
      <description>&lt;h2&gt;
  
  
  Building Containers: The Detailed Way
&lt;/h2&gt;

&lt;p&gt;Containers in Docker are born from container images—a compressed, pre-packaged file system containing your app, its environment, and configuration instructions. Let's embark on the journey of container creation, starting with the meticulous approach.&lt;/p&gt;

&lt;h3&gt;
  
  
  Create Container Using 'docker container create'
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Understanding Images:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Images are compressed file systems with instructions on how to start your app.&lt;/li&gt;
&lt;li&gt;Docker Hub serves as a default repository for images.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Command to Create Container:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker container create &lt;span class="nt"&gt;--help&lt;/span&gt;
   docker container create hello-world:linux
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;The &lt;code&gt;create&lt;/code&gt; command doesn't start the container; it merely generates it.&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Listing Containers:&lt;/strong&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker ps
   docker ps &lt;span class="nt"&gt;--all&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;The status "0 exit code (exited)" indicates successful execution.&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Starting the Container:&lt;/strong&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker container start &amp;lt;container_id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Check logs using &lt;code&gt;docker logs &amp;lt;container_id&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Choosing the detailed way may be preferable in scenarios where you want granular control over container creation, particularly when specific configurations or setups are required.&lt;/p&gt;

&lt;h3&gt;
  
  
  Shortcut: 'docker run' Command
&lt;/h3&gt;

&lt;p&gt;On the flip side, the 'docker run' command offers a more efficient and convenient way to create and run containers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Single Command Magic:&lt;/strong&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker run &amp;lt;image_name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;This command combines create, start, and attach actions.&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Note:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Use &lt;code&gt;docker ps&lt;/code&gt; to list containers and retrieve IDs.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Experimenting with both methods is encouraged. The 'docker run' command is often favored for its simplicity, making it an excellent choice for quick development iterations.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dockerfile and Image Creation
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Dockerfile Basics
&lt;/h3&gt;

&lt;p&gt;Introducing Dockerfile basics is essential for efficient image creation. Let's explain each component succinctly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Example Dockerfile:&lt;/strong&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;   FROM ubuntu:latest
   LABEL maintainer="Your Name"
   USER nobody
   COPY . /app
   RUN apt-get update &amp;amp;&amp;amp; apt-get install -y curl bash
   USER nobody
   ENTRYPOINT ["curl"]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Explanation:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;FROM ubuntu:latest&lt;/code&gt;: Specifies the base image as the latest version of Ubuntu.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;LABEL maintainer="Your Name"&lt;/code&gt;: Adds metadata to the image, specifying the maintainer.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;USER nobody&lt;/code&gt;: Sets the default user for subsequent commands to 'nobody'.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;COPY . /app&lt;/code&gt;: Copies the contents of the current directory to the '/app' directory in the image.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;RUN apt-get update &amp;amp;&amp;amp; apt-get install -y curl bash&lt;/code&gt;: Updates package lists and installs 'curl' and 'bash'.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;USER nobody&lt;/code&gt;: Switches back to the 'nobody' user.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ENTRYPOINT ["curl"]&lt;/code&gt;: Defines the default executable for the container as 'curl'.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Building an Image:&lt;/strong&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker build &lt;span class="nt"&gt;-t&lt;/span&gt; &amp;lt;image-name&amp;gt; &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Use &lt;code&gt;-t&lt;/code&gt; to tag an image.&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Intermediary Images:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docker generates intermediate images for each line in the Dockerfile.&lt;/li&gt;
&lt;li&gt;The final image is squashed and tagged.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Running the Image:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker run &amp;lt;image-name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Removing Images
&lt;/h3&gt;

&lt;p&gt;To maintain a clean and efficient system, it's crucial to remove unused images:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;To remove an image:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; docker rmi &amp;lt;image_name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Use &lt;code&gt;docker images&lt;/code&gt; to list images.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Advanced Container Operations
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Container Naming
&lt;/h3&gt;

&lt;p&gt;Organizing advanced container operations under subheadings for better structure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Assign a name to your container:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  docker run &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nt"&gt;--name&lt;/span&gt; &amp;lt;container_name&amp;gt; &amp;lt;image_name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Port Binding
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Bind ports for network access:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  docker run &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nt"&gt;--name&lt;/span&gt; &amp;lt;server_name&amp;gt; &lt;span class="nt"&gt;-p&lt;/span&gt; 5001:5000 &amp;lt;image_name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Access the server at &lt;a href="http://localhost:5001" rel="noopener noreferrer"&gt;http://localhost:5001&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Volume Mounting
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Preserve data outside containers using volumes:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  docker run &lt;span class="nt"&gt;--rm&lt;/span&gt; &lt;span class="nt"&gt;--entrypoint&lt;/span&gt; sh &lt;span class="nt"&gt;-v&lt;/span&gt; /tmp/container:/tmp ubuntu &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"echo 'Hello There.' &amp;gt; /tmp/file &amp;amp;&amp;amp; cat /tmp/file"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Map a folder on your machine to a folder in the container.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Container Registries
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Understanding Registries:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Container image registries track and store container images.&lt;/li&gt;
&lt;li&gt;Docker Hub is the default registry.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Pushing Images to Docker Hub:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   docker login
   docker tag &amp;lt;image_to_rename&amp;gt; username/new_image_name
   docker push &amp;lt;image-name&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Deleting Images in Docker Hub:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Delete images through the Docker Hub settings in the browser.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Mastering container creation and interaction forms the backbone of Docker. Whether you choose the detailed or shortcut approach, understanding the nuances empowers you to efficiently manage and deploy your applications.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/engine/reference/run/" rel="noopener noreferrer"&gt;Docker Documentation. Docker run reference.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/engine/reference/builder/" rel="noopener noreferrer"&gt;Docker Documentation. Dockerfile reference.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/engine/reference/commandline/cli/" rel="noopener noreferrer"&gt;Docker Documentation. Docker CLI documentation.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/storage/" rel="noopener noreferrer"&gt;Docker Documentation. Manage data in containers.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/compose/compose-file/" rel="noopener noreferrer"&gt;Docker Documentation. Docker Compose overview.&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>docker</category>
      <category>containers</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Anatomy of Docker</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Mon, 15 Jan 2024 13:00:00 +0000</pubDate>
      <link>https://dev.to/rkj180220/anatomy-of-docker-4a4j</link>
      <guid>https://dev.to/rkj180220/anatomy-of-docker-4a4j</guid>
      <description>&lt;p&gt;In this section, we delve into the intricate components that form the backbone of Docker. Understanding the anatomy of Docker is essential for harnessing its true power in containerization.&lt;/p&gt;

&lt;h3&gt;
  
  
  Container Components:
&lt;/h3&gt;

&lt;p&gt;Docker operates on the fusion of Linux namespaces and control groups, providing a unique and isolated environment for applications. Let's simplify this complex structure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Linux Namespace:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A Linux kernel feature providing different "views" of your system to applications.&lt;/li&gt;
&lt;li&gt;Offers a unique and isolated instance of system resources, allowing independent operation of processes or containers.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Linux Control Group (cgroup):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Another Linux kernel feature restricting how much hardware each process can use.&lt;/li&gt;
&lt;li&gt;Utilized for monitoring and restricting CPU, network, disk bandwidth, and memory consumption.&lt;/li&gt;
&lt;li&gt;Assigning disk quotas (not supported by Docker).&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fe06hmff96ykniuc26irx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fe06hmff96ykniuc26irx.png" alt="Anatomy of Docker" width="799" height="329"&gt;&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Note:&lt;/strong&gt; While Docker is native to Linux and some newer versions of Windows, containers can run on any OS. Docker is compatible with Windows Server, and with the introduction of Windows Subsystem for Linux (WSL) from Windows 10 Anniversary Edition onwards, performance on Windows has significantly improved.&lt;/p&gt;

&lt;h3&gt;
  
  
  Linux's Eight Kernels:
&lt;/h3&gt;

&lt;p&gt;Let's briefly explore the eight namespaces in the Linux kernel that provide isolation for processes, networking, interprocess communication, and more:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;PID Namespace (pid):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Isolates the process ID number space, ensuring processes in different namespaces have distinct PIDs.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Network Namespace (net):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provides isolation in terms of network resources.&lt;/li&gt;
&lt;li&gt;Enables each namespace to have its own network interfaces, routing tables, and firewall rules.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Mount Namespace (mnt):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Isolates the set of mount points seen by a group of processes.&lt;/li&gt;
&lt;li&gt;Ensures each namespace has its own filesystem hierarchy.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;IPC Namespace (ipc):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Isolates interprocess communication resources.&lt;/li&gt;
&lt;li&gt;Each namespace has its own System V IPC, message queues, and semaphore arrays.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;UTS Namespace (uts):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Isolates system identifiers like hostname and NIS domain name.&lt;/li&gt;
&lt;li&gt;Enables each namespace to have its own hostname and domain name.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;User Namespace (user):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provides isolation for user and group IDs.&lt;/li&gt;
&lt;li&gt;Allows different namespaces to have different user and group mappings.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Cgroup Namespace (cgroup):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Introduced in Linux kernel 5.10.&lt;/li&gt;
&lt;li&gt;Enables isolation for control groups, allowing each namespace to have its own set of cgroups.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Time Namespace (time):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Introduced in Linux kernel 5.6.&lt;/li&gt;
&lt;li&gt;Ensures each namespace has its own perception of time.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Docker Advantages:
&lt;/h3&gt;

&lt;h4&gt;
  
  
  Dockerfiles:
&lt;/h4&gt;

&lt;p&gt;Dockerfiles simplify configuring and packaging apps and their environments.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Users instruct Docker on container configuration through Dockerfiles.&lt;/li&gt;
&lt;li&gt;Docker utilizes these Dockerfiles to package apps and their environments into images.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Example:
&lt;/h4&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# Dockerfile example (more details in the next blog)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; ubuntu:latest&lt;/span&gt;
&lt;span class="k"&gt;RUN &lt;/span&gt;apt-get update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; apt-get &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-y&lt;/span&gt; nginx
&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="s"&gt; ["nginx", "-g", "daemon off;"]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  Sharing Images:
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Docker Hub, a global repository, facilitates easy image sharing.&lt;/li&gt;
&lt;li&gt;Users can utilize alternative registries like AWS Elastic Container Registry (ECR), GCP Google Container Registry (GCR), Quay.io, Harbor, Azure Container Registry (ACR) etc., and share images effortlessly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Docker CLI:
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Docker CLI makes starting apps in containers a breeze.&lt;/li&gt;
&lt;li&gt;No worries about UID mappings, network interfaces, or Docker configuration files.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Docker Limitations:
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Docker relies on Linux namespaces and control groups, making it technically native to Linux and some newer versions of Windows.&lt;/li&gt;
&lt;li&gt;Containers can run on any OS, but their images are bound to their parent OS. Workarounds exist for these limitations.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Docker Alternatives:
&lt;/h3&gt;

&lt;h4&gt;
  
  
  Container Runtime Interface (CRI):
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes introduced CRI, enabling developers to create their container runtimes.&lt;/li&gt;
&lt;li&gt;Examples: CRI-O, RUNC, Firecracker from AWS.&lt;/li&gt;
&lt;li&gt;Podman by RedHat for highly secure, rootless containers.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Docker Machine:
&lt;/h4&gt;

&lt;p&gt;Docker Machine serves as a workaround for Mac and Windows users, allowing them to run Docker on their systems. However, if you are using Linux, there are specific distributions optimized to support container runtimes more efficiently.&lt;/p&gt;

&lt;h5&gt;
  
  
  Docker Machine for Mac/Windows:
&lt;/h5&gt;

&lt;p&gt;Designed as a workaround for non-Linux environments.&lt;br&gt;
Utilizes VirtualBox to create virtual machines running Docker.&lt;br&gt;
Offers a solution for those operating systems lacking native Docker support.&lt;/p&gt;

&lt;h5&gt;
  
  
  Linux Distributions Optimized for Containers:
&lt;/h5&gt;

&lt;p&gt;When using Docker on Linux, consider distributions optimized for container runtimes. Examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ubuntu: A widely used and well-supported Linux distribution with good compatibility for Docker.&lt;/li&gt;
&lt;li&gt;Alpine Linux: Known for its lightweight nature, making it an excellent choice for containerized applications.&lt;/li&gt;
&lt;li&gt;Fedora CoreOS: A minimal OS designed explicitly for running containerized applications at scale.&lt;/li&gt;
&lt;/ul&gt;

&lt;h5&gt;
  
  
  Performance Considerations:
&lt;/h5&gt;

&lt;p&gt;Docker Machine on Mac and Windows may exhibit slower performance due to the virtualization layer and dependency on VirtualBox.&lt;br&gt;
On Linux, choosing a distribution optimized for containers can enhance Docker's performance and resource utilization.&lt;br&gt;
When selecting a Linux distribution for Docker, it's essential to consider factors like compatibility, resource efficiency, and the level of support provided for container runtimes. Each distribution may have its strengths, so choose the one that aligns best with your specific requirements.&lt;/p&gt;

&lt;h4&gt;
  
  
  Docker Desktop:
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;A more refined solution introduced in 2016.&lt;/li&gt;
&lt;li&gt;Smaller and faster VM running on Apple's native hypervisor or Hyper-V on Windows.&lt;/li&gt;
&lt;li&gt;Improved GUI for configuration, mounting volumes, and exposing network ports.&lt;/li&gt;
&lt;li&gt;Addresses the limitations of Docker Machine.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Conclusion:
&lt;/h3&gt;

&lt;p&gt;Understanding Docker's anatomy is crucial for harnessing its power. From Linux namespaces to Dockerfiles, we've peeled back the layers. Stay tuned for the next blog, where we'll dissect Dockerfiles and dive deeper into crafting containerized environments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://lwn.net/Articles/531114/#:~:text=The%20namespaces,instance%20of%20the%20global%20resource" rel="noopener noreferrer"&gt;Linux Documentation. Namespaces in operation, part 1: namespaces overview.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/engine/reference/builder/" rel="noopener noreferrer"&gt;Docker Documentation. Dockerfile reference.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/" rel="noopener noreferrer"&gt;Kubernetes Documentation. Container runtimes.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://podman.io/" rel="noopener noreferrer"&gt;Podman Documentation. Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/technologies/cloud-computing/openshift/red-hat-openshift-kubernetes" rel="noopener noreferrer"&gt;RedHat vs Kubernetes&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>docker</category>
      <category>containers</category>
      <category>tutorial</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Why Docker?</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Sat, 13 Jan 2024 12:55:40 +0000</pubDate>
      <link>https://dev.to/rkj180220/why-docker-4hpn</link>
      <guid>https://dev.to/rkj180220/why-docker-4hpn</guid>
      <description>&lt;p&gt;Have you ever found yourself frustrated with the "It works on my machine" dilemma during your software development journey? Despite careful testing in various environments, your application misbehaves elsewhere, falling victim to the common challenges of software engineering.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Landscape of Solutions:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Configuration Management:&lt;/strong&gt;&lt;br&gt;
Tools like Chef, Ansible, and Puppet tackle the issue by enabling developers to write code in configuration languages. This code describes precisely what machines need to run your app.&lt;br&gt;
&lt;em&gt;Drawback: Requires in-depth knowledge of underlying OS and hardware for tools like Chef, Puppet, and Ansible.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Virtual Machines as Code (Vagrant):&lt;/strong&gt;&lt;br&gt;
HashiCorp’s Vagrant empowers developers to script entire virtual machines for their apps.&lt;br&gt;
&lt;em&gt;Drawback: Involves understanding the virtual machine's hardware requirements and often configuring the OS before app installation. Heavy, relatively slow, and demands inconvenient configurations.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;A Paradigm Shift: Docker&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why Docker?&lt;/strong&gt;&lt;br&gt;
Amidst these challenges, Docker emerges as a game-changer. It empowers developers to package their applications into images that seamlessly run on containers. These containers act as virtualized operating systems, configured with just enough resources to run your app.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Virtual Machines vs Containers:&lt;/strong&gt;&lt;br&gt;
&lt;strong&gt;Virtual Machines:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Run on a platform called Hypervisor.&lt;/li&gt;
&lt;li&gt;Hypervisor emulates hardware operations, translating them to real hardware operations on hosts.&lt;/li&gt;
&lt;li&gt;Takes up a lot of space.&lt;/li&gt;
&lt;li&gt;Requires OS installation and configuration.&lt;/li&gt;
&lt;li&gt;Can run multiple apps simultaneously but can't directly interact with the host securely.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Containers:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Run on container runtimes.&lt;/li&gt;
&lt;li&gt;Runtimes work with the OS to allocate hardware and copy files, running like any other app on the system.&lt;/li&gt;
&lt;li&gt;Do not emulate hardware, don't need to "boot up," and use the same hardware and OS.&lt;/li&gt;
&lt;li&gt;Do not require OS installation, enabling quick startup.&lt;/li&gt;
&lt;li&gt;Take up less space, allowing more apps to run.&lt;/li&gt;
&lt;li&gt;Run only one app at a time by design but can interact with the host, posing security challenges (mostly resolved).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu02h24v7yf2ayluvn67x.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu02h24v7yf2ayluvn67x.jpg" alt="Containers vs VMs" width="800" height="405"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Docker Advantage:&lt;/strong&gt;&lt;br&gt;
Docker streamlines the development process by creating images from lightweight configuration files, detailing everything your app needs to run. Unlike traditional virtual machines, containers hide hardware details, ensuring consistent app performance across different environments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Portability and Efficiency:&lt;/strong&gt;&lt;br&gt;
Build your app once, and if the machine can run Docker, your app will run consistently, irrespective of its location. Docker brings efficiency to the development process, enabling you to build, deploy, and scale your applications quickly, safely, and cost-effectively.&lt;/p&gt;

&lt;p&gt;In a nutshell, Docker leverages images and containers to offer a universal solution—run your apps anywhere with consistent behavior. Say goodbye to the chaos of environment discrepancies, and embrace the power to build, deploy, and scale your applications effortlessly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Next Steps:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In this beginner's guide to Docker, we'll unravel the world of containerization, exploring how Docker empowers developers to craft, share, and run applications seamlessly. Let's decode Docker and unlock its potential for your development endeavors.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/" rel="noopener noreferrer"&gt;Docker Docs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.chef.io/" rel="noopener noreferrer"&gt;Chef Software Inc. Chef Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.vagrantup.com/docs" rel="noopener noreferrer"&gt;HashiCorp. Vagrant Documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>docker</category>
      <category>tutorial</category>
      <category>softwaredevelopment</category>
      <category>containers</category>
    </item>
    <item>
      <title>Persistent Data Grids in Angular: A Comprehensive Guide</title>
      <dc:creator>Ramkumar Jayakumar</dc:creator>
      <pubDate>Wed, 20 Dec 2023 10:15:57 +0000</pubDate>
      <link>https://dev.to/rkj180220/persistent-data-grids-in-angular-a-comprehensive-guide-4jbf</link>
      <guid>https://dev.to/rkj180220/persistent-data-grids-in-angular-a-comprehensive-guide-4jbf</guid>
      <description>&lt;p&gt;Ever wondered how web applications keep your search data and grid configurations intact as you navigate through pages? This seamless experience is thanks to data grid implementations—a cornerstone of modern web development.&lt;/p&gt;

&lt;p&gt;Data grids are essential tools, revolutionizing the presentation and interaction with vast sets of information. Picture online banking platforms allowing effortless transaction filtering or project management tools dynamically updating tasks without a page refresh. These real-world scenarios underscore the power of data grids, offering users intuitive controls for customization, sorting, and real-time updates.&lt;/p&gt;

&lt;p&gt;This blog explores the intricacies of achieving persistent storage for data grids in Angular, eliminating the need for manual data synchronization. Join us as we delve into ensuring a seamless user experience where apps effortlessly retain grid configurations and search data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Understanding the Challenge
&lt;/h2&gt;

&lt;p&gt;Before delving into the technical details, let's briefly outline the key elements of an application's data grids. Typically, a data grid consists of an external filter that searches a database and a grid that displays the retrieved records. Two critical pieces of information need to be preserved:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Search Data&lt;/strong&gt;: Retaining all the searches made in external filters whenever the page is visited.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grid Configuration&lt;/strong&gt;: Storing options like selected columns, column width, and column order, which should persist across visits.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The next important thing is how long this data must be stored. Deciding when to retain or discard the data depends on the context. Search data is often considered temporary, while grid configurations are expected to persist across user sessions.&lt;br&gt;
So, the search data should be stored for a particular session against a data grid, while grid configurations are stored against a data grid and the user ID.&lt;/p&gt;
&lt;h2&gt;
  
  
  Choosing the Right Storage Option
&lt;/h2&gt;

&lt;p&gt;Now that we've identified what data to store and when, let's explore the options available to achieve persistent storage. Angular offers an array of options for persistence, each with its own strengths and quirks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;LocalStorage&lt;/strong&gt;: Persistent even after closing the tab, ideal for large data and simple settings. Performance would be a concern for large chunks of data since we will be parsing and stringifying the large data.&lt;br&gt;
&lt;em&gt;Example: User preferences like language or theme persist across logins.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;SessionStorage&lt;/strong&gt;: Similar to LocalStorage, but data vanishes when the browser window closes.&lt;br&gt;
&lt;em&gt;Example: Shopping cart contents during a single shopping session.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;IndexedDB&lt;/strong&gt;: A more powerful local database with richer object storage capabilities. While IndexedDB offers powerful storage capabilities, its complexity and overhead, encompassing factors like error handling challenges, potential performance limitations, and reduced compatibility across browsers and modes, make it a less straightforward choice for simple persistent data grid storage.&lt;br&gt;
&lt;em&gt;Example: Offline data persistence for complex applications.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cookies&lt;/strong&gt;: small data packages sent between browser and server, useful for tracking user sessions, but their privacy concerns, technical limitations, and potential negative impact on the user experience make them a less ideal choice.&lt;br&gt;
&lt;em&gt;Example: Remembering login credentials for returning users.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Angular Routes&lt;/strong&gt;: Embed data directly in the URL, making it bookmarkable and SEO-friendly. Can be used for simple configurations&lt;br&gt;
&lt;em&gt;Example: Sharing specific content configurations with a link.&lt;br&gt;
&lt;code&gt;/analytics/user-report?start_date=2023-10-01&amp;amp;end_date=2023-12-19&lt;/code&gt; (shows specific date range for analysis)&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Angular Services&lt;/strong&gt;: Services act as data managers, making your app cleaner, more organized, and easier to work with. Its key advantages are code reusability, abstraction, and synchronization. Data will be lost upon page refresh/ tab closure.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;&lt;em&gt;Remember&lt;/em&gt;&lt;/strong&gt;: The best choice depends on your application's needs. Consider data size, persistence duration, security implications, and potential limitations.&lt;/p&gt;
&lt;h2&gt;
  
  
  Storage Options Implementation
&lt;/h2&gt;

&lt;p&gt;In my implementation, I've opted for a hybrid approach, utilizing both local storage and a backend database to store different types of data. Here's a glimpse of the structure used for storing grid configurations.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
    "grid_name": {
        "external_filters": {
            "name": null,
            "customer": [ 1234, 1235],
            "email": null,
            "some_boolean_filter": false,
            "created_at": "2023-12-13T18:30:00.000Z"
        },
        "columns_config": [
            // Individual Column Configurations
            {
                "field": "name",
                "width": 120,
                "title": "Name",
                "orderIndex": 1
            },
            {
                "field": "amount",
                "width": 100,
                "title": "Amount(₹)",
                "cssClass": "text-end",
                "orderIndex": 3
            },
            {
                "field": "email",
                "width": 100,
                "title": "Email",
                "hidden": true,
                "orderIndex": 2
            }

        ], 
        "state": {
            // Grid State Information
            "skip": 0,
            "take": 10,
            "sort": []
        }
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Saving Grid Configurations
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Functional Breakdown
&lt;/h3&gt;

&lt;p&gt;Let's get to the exact details as to how and when we save the data. First, we need to break down each of the operations into separate functions so that we can reuse them. We will have the following functions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;saveGridSettings&lt;/strong&gt;: This function builds a data structure containing column configurations, grid state, and external filter data. It then saves this data to LocalStorage.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    saveGridSettings() {
        const external_filters = this.searchForm.value;
        const grid_config = {
            contractor_list: {
                state: this.grid_state,
                external_filters: external_filters,
                columns_config: this.columnsConfig
            }
        };
        this.persistentStorageService.setGridSettings('grid_name', grid_config);
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;setGridSettings(grid_key: string, gridConfig: any): void {
    localStorage.setItem(grid_key, JSON.stringify(gridConfig));
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;saveColumnConfigSettings&lt;/strong&gt;: This function saves the user-specific column configuration for the current grid to the backend via an API call. It's triggered when the "Apply" button is clicked.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    saveColumnConfigSettings(event: any) {
        const input_data = {
            grid: 'grid_name',
            columns_config: event
        };
        this.saveGridSettings();

        this.persistentStorageService.saveColumnConfigSettings(input_data).subscribe({
            next: (response: any) =&amp;gt; {
                this.toast.showSuccess(response.message);
                this.getColumnConfigSettings();
                this.contractor_list_grid_loading = false;
            },
            error: (error: any) =&amp;gt; {
                this.toast.showError(error.message);
            }
        });
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;getColumnConfigSettings&lt;/strong&gt;: This function retrieves the user-specific column configuration for the current grid from the backend. It's used during component initialization.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    getColumnConfigSettings() {
        const input_data = {
            grid: 'grid_name'
        };

        this.persistentStorageService.getColumnConfigSettings(input_data).subscribe({
            next: (response: any) =&amp;gt; {
                if (response &amp;amp;&amp;amp; response['columns_config'].length !== 0) {
                    this.mapGridSettings(response['columns_config']);
                } else {
                    this.initialGridSettings();
                }
            },
            error: (error: any) =&amp;gt; {
                this.toast.showError(error.message);
                this.initialGridSettings();
            }
        });
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;initializeGridRememberSettings&lt;/strong&gt;: This function checks for existing grid settings in LocalStorage. If unavailable, it fetches the user's column configuration via API or uses the default configuration if none exists.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    ngOnInit(): void {
        // Other initializations
        this.initializeGridRememberSettings();
    }

    initializeGridRememberSettings() {
        if (!!this.savedState) {
            this.mapGridSettings();
        } else {
            this.getColumnConfigSettings();
        }
    }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Application Flow
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjkk20kcat00bfuzl2hdq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjkk20kcat00bfuzl2hdq.png" alt="Persistent data grids in angular workflow" width="800" height="904"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;User interaction&lt;/strong&gt;: When the user applies filters or modifies the grid layout, the updated state (including external filters, column configurations, and grid state) is saved to LocalStorage with the help of state change events.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Local Storage cache&lt;/strong&gt;: This cached data ensures quick restoration of configurations when the user revisits the page.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;API calls&lt;/strong&gt;: Saving of user-specific column configurations to the backend happens only when the user explicitly confirms with the "Apply" button, triggering the saveColumnConfigSettings function.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Component initialization&lt;/strong&gt;: During initial page load, the initializeGridRememberSettings function checks for LocalStorage data. If unavailable, it retrieves the user's configuration via API or uses the default configuration.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Overall, this approach leverages LocalStorage for efficient caching and utilizes the backend for persistent, user-specific configuration storage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Choosing the right approach for persistent storage in Angular data grids involves careful consideration of your application's requirements. In my case, LocalStorage with a backend database for grid configurations provided a flexible and scalable solution.&lt;/p&gt;

&lt;p&gt;However, LocalStorage is not without drawbacks. Performance concerns arise due to the parsing and stringification processes involved. Instead, leveraging an Angular service to store grid data can optimize performance by avoiding these processes. This comes at the cost of losing data upon tab closure. Ultimately, the choice depends on your specific needs.&lt;/p&gt;

&lt;p&gt;To further enhance this solution, consider maintaining a queue in LocalStorage with a maximum capacity of 10 data grids. This minimizes storage size by evicting older entries, assuming users navigating beyond 10 pages are unlikely to revisit those specific searches.&lt;/p&gt;

&lt;p&gt;Remember, thorough analysis and planning before diving into code is crucial for ensuring a robust and efficient solution for persistent data grids in your Angular application. Happy coding!&lt;/p&gt;

&lt;h2&gt;
  
  
  Further Resources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;What are data grids? &lt;a href="https://medium.com/swlh/introducing-in-memory-data-grid-hazelcast-imdg-1f2af2be5344" rel="noopener noreferrer"&gt;Introducing In-Memory Data Grid — Hazelcast IMDG | by Dina Bogdan | The Startup | Medium&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;For a deeper dive into Angular Routes, see &lt;a href="https://www.mygreatlearning.com/blog/angular-routing-tutorial/" rel="noopener noreferrer"&gt;Angular Tutorial&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Learn more about LocalStorage and SessionStorage &lt;a href="https://levelup.gitconnected.com/beginners-guide-to-web-storage-a88b6e5b522e" rel="noopener noreferrer"&gt;Beginner's Guide to Web Storage&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;A Developer's Guide to HTTP Cookies: &lt;a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/cookies" rel="noopener noreferrer"&gt;cookies - Mozilla | MDN&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Introducing IndexedDB: &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API" rel="noopener noreferrer"&gt;IndexedDB API&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>angular</category>
      <category>storage</category>
      <category>softwaredevelopment</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
