<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Shahein Ockersz</title>
    <description>The latest articles on DEV Community by Shahein Ockersz (@shahein_ockersz).</description>
    <link>https://dev.to/shahein_ockersz</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4235014%2F64d1833c-f9a9-4b54-8128-bc02ecef4bac.png</url>
      <title>DEV Community: Shahein Ockersz</title>
      <link>https://dev.to/shahein_ockersz</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/shahein_ockersz"/>
    <language>en</language>
    <item>
      <title>Why Vibe-Coded AI Agents Silently Corrupt Microservices (and How I Built an Open-Source SAGA Undo Proxy)</title>
      <dc:creator>Shahein Ockersz</dc:creator>
      <pubDate>Sat, 21 Nov 2026 16:24:03 +0000</pubDate>
      <link>https://dev.to/shahein_ockersz/why-vibe-coded-ai-agents-silently-corrupt-microservices-and-how-i-built-an-open-source-saga-undo-11fa</link>
      <guid>https://dev.to/shahein_ockersz/why-vibe-coded-ai-agents-silently-corrupt-microservices-and-how-i-built-an-open-source-saga-undo-11fa</guid>
      <description>&lt;p&gt;Suppose you wire up an autonomous LLM workflow to handle an everyday enterprise task—say, employee offboarding across three microservices:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;HR Service:&lt;/strong&gt; Mark employee &lt;code&gt;EMP-00104&lt;/code&gt; as &lt;code&gt;INACTIVE&lt;/code&gt; and revoke login access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Payroll Service:&lt;/strong&gt; Record a &lt;code&gt;25,000 LKR&lt;/code&gt; equipment settlement deduction in the ledger.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Notification Service:&lt;/strong&gt; Dispatch the formal clearance email.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Everything works cleanly on the happy path. Then on a Tuesday afternoon, the Notification Service's SMTP relay stalls and throws an &lt;strong&gt;&lt;code&gt;HTTP 504 Gateway Timeout&lt;/code&gt;&lt;/strong&gt; on Step 3.&lt;/p&gt;

&lt;p&gt;What happens next in most agent frameworks?&lt;/p&gt;

&lt;p&gt;Your Python/TypeScript runtime catches an unhandled tool error, maybe retries once, and exits. Meanwhile, out in your real infrastructure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The employee is already locked out in HR.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;25,000 LKR&lt;/code&gt; is already deducted in Payroll.&lt;/li&gt;
&lt;li&gt;No clearance email ever went out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And the worst part? Asking the agent to "retry the task" often tries to run Step 1 or Step 2 a second time—creating duplicate charges on top of partial state.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Why don't Pydantic, Guardrails AI, or NeMo Guardrails fix this?&lt;/strong&gt;&lt;br&gt;
Because those libraries execute &lt;em&gt;inside&lt;/em&gt; the agent's local memory. When the agent process hits an exception, times out, or gets subverted, in-process guardrails vanish with it. More fundamentally, a single-turn payload validator has no concept of distributed state, prior committed database writes, or how to invoke an inverse compensating API in another service.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To solve this properly, contract verification and rollback logic have to move &lt;strong&gt;out of the LLM runtime&lt;/strong&gt; and sit at the &lt;strong&gt;network perimeter&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So I built &lt;strong&gt;&lt;a href="https://github.com/Ockersz/agent-saga" rel="noopener noreferrer"&gt;AgentSaga-Gateway&lt;/a&gt;&lt;/strong&gt;—an open-source (Apache 2.0) zero-trust reverse proxy in &lt;strong&gt;TypeScript / NestJS&lt;/strong&gt; backed by &lt;strong&gt;OpenAPI 3.1&lt;/strong&gt;, &lt;strong&gt;Ajv&lt;/strong&gt;, and &lt;strong&gt;PostgreSQL 16&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Moving Defense to the Perimeter
&lt;/h2&gt;

&lt;p&gt;Instead of trusting agents to self-police their REST calls, all tool traffic flows through &lt;code&gt;/proxy/...&lt;/code&gt; with three lightweight correlation headers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;x-saga-workflow-id: &amp;lt;workflow-uuid&amp;gt;
x-saga-step-index:  1 | 2 | 3
x-agent-id:         hr-bot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before opening an upstream TCP socket to any internal microservice, the gateway enforces four sequential gates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[ AI Agent ]
     │  POST /proxy/api/v1/payroll/settlement
     ▼
┌────────────────────────────────────────────────────────┐
│            AGENTSAGA-GATEWAY (REVERSE PROXY)           │
│                                                        │
│  1. Pre-compiled OpenAPI 3.1 Ajv Gate  ──▶ HTTP 400    │
│  2. Dynamic Risk &amp;amp; Monetary Ceiling    ──▶ HTTP 403    │
│  3. Pivot Invariant &amp;amp; Abort Lock       ──▶ HTTP 409    │
│  4. Append-Only SAGA Journal (PG16/WAL)                │
└──────────────────────────┬─────────────────────────────┘
                           │
         ┌─────────────────┴─────────────────┐
         ▼ (2xx)                             ▼ (5xx / Timeout)
  [ Enterprise APIs ]             [ Automated SAGA Rollback ]
   Commits step state             Resolves JSONPath bindings
                                  Runs S_(k-1)^-1 ... S_1^-1
                                  Returns HTTP 502 audit report
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let's unpack how each piece works in practice.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Declaring the "Undo Button" Inside OpenAPI 3.1
&lt;/h2&gt;

&lt;p&gt;Rather than hardcoding rollback orchestration inside every client agent, microservices declare their transaction semantics right inside standard OpenAPI 3.1 YAML files via vendor-neutral &lt;code&gt;x-agentsaga-*&lt;/code&gt; extensions.&lt;/p&gt;

&lt;p&gt;Here is the real snippet from &lt;code&gt;contracts/payroll-service.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;/api/v1/payroll/settlement&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;post&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Execute employee settlement deduction or payout&lt;/span&gt;
    &lt;span class="na"&gt;operationId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;executeSettlement&lt;/span&gt;
    &lt;span class="na"&gt;x-agentsaga-action-type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;compensatable&lt;/span&gt;
    &lt;span class="na"&gt;x-agentsaga-risk-level&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;high&lt;/span&gt;
    &lt;span class="na"&gt;x-agentsaga-compensating-action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/api/v1/payroll/settlement/{settlementId}/reversal&lt;/span&gt;
      &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;POST&lt;/span&gt;
      &lt;span class="na"&gt;mapping&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;settlementId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"$.response.body.settlementId"&lt;/span&gt;
        &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Automated SAGA rollback due to downstream workflow abort"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice that subtle engineering problem: &lt;strong&gt;before Step 2 runs, the &lt;code&gt;settlementId&lt;/code&gt; doesn't even exist yet&lt;/strong&gt;—the Payroll Service generates a fresh UUID (&lt;code&gt;201 Created&lt;/code&gt;) in its response body.&lt;/p&gt;

&lt;p&gt;At runtime, &lt;code&gt;AgentSaga&lt;/code&gt; persists:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;forwardPathParams&lt;/code&gt; (for example, &lt;code&gt;employeeId: "EMP-00104"&lt;/code&gt; from Step 1's URL)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;forwardRequestBody&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;forwardResponseBody&lt;/code&gt; (including the newly created &lt;code&gt;settlementId&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;compensatingMapping&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When Step 3 crashes, the compensator resolves &lt;code&gt;"$.response.body.settlementId"&lt;/code&gt; and &lt;code&gt;"$.request.path.employeeId"&lt;/code&gt; dynamically into the exact inverse REST URLs and payloads in &lt;strong&gt;reverse topological order&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Calls &lt;code&gt;POST /api/v1/payroll/settlement/{settlementId}/reversal&lt;/code&gt; (reversing Step 2 first)&lt;/li&gt;
&lt;li&gt;Calls &lt;code&gt;POST /api/v1/hr/employees/EMP-00104/status&lt;/code&gt; with &lt;code&gt;{ "status": "ACTIVE" }&lt;/code&gt; (restoring Step 1 second)&lt;/li&gt;
&lt;li&gt;Returns an &lt;strong&gt;HTTP &lt;code&gt;502 Bad Gateway&lt;/code&gt;&lt;/strong&gt; JSON audit report detailing every compensated step.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  2. Three Action Semantics &amp;amp; the Pivot Guard (&lt;code&gt;HTTP 409&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Not everything in the physical or financial world can be rolled back. You cannot un-send an email or recall a wire transfer.&lt;/p&gt;

&lt;p&gt;Every endpoint in &lt;code&gt;AgentSaga&lt;/code&gt; declares one of three action tiers:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Extension Value&lt;/th&gt;
&lt;th&gt;Execution Rule&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;compensatable&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Paired with an inverse endpoint (&lt;code&gt;x-agentsaga-compensating-action&lt;/code&gt;). Rolled back automatically in reverse order on downstream abort.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;pivot&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Irreversible terminal side-effect (e.g., &lt;code&gt;/api/v1/notifications/clearance-email&lt;/code&gt;).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;retriable&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Guaranteed idempotent endpoint (e.g., &lt;code&gt;/settlement/{settlementId}/reversal&lt;/code&gt;) safe under compensation retries.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In &lt;code&gt;gateway/src/proxy/gateway-proxy.controller.ts&lt;/code&gt;, any endpoint flagged &lt;code&gt;pivot&lt;/code&gt; must satisfy two runtime invariants before forwarding:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The workflow cannot be in &lt;code&gt;FAILED&lt;/code&gt;, &lt;code&gt;COMPENSATING&lt;/code&gt;, or &lt;code&gt;COMPENSATED&lt;/code&gt; state.&lt;/li&gt;
&lt;li&gt;All antecedent steps (&lt;code&gt;stepIndex &amp;lt; currentStep&lt;/code&gt;) must be logged in the journal with status &lt;code&gt;EXECUTED&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If a buggy or looping agent tries to replay a clearance email after Step 2 failed—or after Step 3 already rolled back—the gateway locks the request at the edge with &lt;strong&gt;&lt;code&gt;HTTP 409 Conflict&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Stopping Schema Drift &amp;amp; Excess Payouts Pre-Socket
&lt;/h2&gt;

&lt;p&gt;LLM tool calls frequently suffer from subtle parameter bugs: strings instead of numbers (&lt;code&gt;"5000"&lt;/code&gt;), invented enum values (&lt;code&gt;"BITCOIN"&lt;/code&gt;), or malformed path IDs (&lt;code&gt;BAD_ID&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;At startup, &lt;code&gt;ContractRegistryService&lt;/code&gt; compiles both &lt;strong&gt;URL path parameters&lt;/strong&gt; and &lt;strong&gt;request bodies&lt;/strong&gt; into strict &lt;strong&gt;Ajv&lt;/strong&gt; validators with type coercion turned off (&lt;code&gt;coerceTypes: false&lt;/code&gt;). Malformed requests fail immediately at the network edge in sub-millisecond time—downstream databases see &lt;strong&gt;zero&lt;/strong&gt; connections.&lt;/p&gt;

&lt;p&gt;Valid schemas then hit &lt;code&gt;PolicyEvaluatorService&lt;/code&gt; before touching upstream services:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Agent Denylist:&lt;/strong&gt; Rejects caller identities prefixed with &lt;code&gt;untrusted-&lt;/code&gt; or &lt;code&gt;blocked-&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monetary Thresholds:&lt;/strong&gt; Rejects payloads whose &lt;code&gt;amount&lt;/code&gt; exceeds &lt;code&gt;POLICY_MAX_AMOUNT&lt;/code&gt; (default &lt;code&gt;500000&lt;/code&gt;) unless the request presents an explicit &lt;code&gt;x-agentsaga-approval-token&lt;/code&gt; header.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Critical-Risk Escalation:&lt;/strong&gt; Endpoints marked &lt;code&gt;x-agentsaga-risk-level: critical&lt;/code&gt; similarly require an approval token, otherwise returning &lt;code&gt;HTTP 403 Forbidden&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  4. Try It Yourself in 60 Seconds (&lt;code&gt;docker compose&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Everything—PostgreSQL 16 (initialized from &lt;code&gt;gateway/src/database/schema.sql&lt;/code&gt;), three decoupled Express microservices (&lt;code&gt;HR&lt;/code&gt;, &lt;code&gt;Payroll&lt;/code&gt;, &lt;code&gt;Notification&lt;/code&gt;), and the NestJS Gateway—runs out of the box on non-conflicting host ports (&lt;code&gt;5433&lt;/code&gt;, &lt;code&gt;3010&lt;/code&gt;, &lt;code&gt;8090&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/Ockersz/agent-saga.git
&lt;span class="nb"&gt;cd&lt;/span&gt; agent-saga
docker compose up -d --build
curl http://127.0.0.1:8090/health
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Watch a real mid-workflow crash auto-recover in PostgreSQL-backed mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Reset downstream microservice state&lt;/span&gt;
curl -s -X POST http://127.0.0.1:3010/api/v1/reset

&lt;span class="nv"&gt;WF_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"11111111-2222-3333-4444-555555555555"&lt;/span&gt;

&lt;span class="c"&gt;# Step 1: Deactivate employee in HR (200 OK)&lt;/span&gt;
curl -s -X POST http://127.0.0.1:8090/proxy/api/v1/hr/employees/EMP-00104/status \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-workflow-id: $WF_ID"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-step-index: 1"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-agent-id: hr-bot"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"status":"INACTIVE","reason":"Employee resignation approved"}'&lt;/span&gt;

&lt;span class="c"&gt;# Step 2: Record settlement in Payroll (201 Created -&amp;gt; returns settlementId)&lt;/span&gt;
&lt;span class="nv"&gt;SETTLEMENT_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s -X POST http://127.0.0.1:8090/proxy/api/v1/payroll/settlement \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-workflow-id: $WF_ID"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-step-index: 2"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-agent-id: hr-bot"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"employeeId":"EMP-00104","amount":25000,"currency":"LKR","settlementType":"EQUIPMENT_DEDUCTION","reason":"Laptop dock return"}'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; python -c &lt;span class="s2"&gt;"import sys,json; print(json.load(sys.stdin)['settlementId'])"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Step 3: Inject an SMTP 504 timeout -&amp;gt; Gateway rolls back Step 2 &amp;amp; Step 1, returns HTTP 502&lt;/span&gt;
curl -i -X POST http://127.0.0.1:8090/proxy/api/v1/notifications/clearance-email \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-workflow-id: $WF_ID"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-saga-step-index: 3"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-agent-id: hr-bot"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"x-simulate-failure: notification"&lt;/span&gt; \
  -d &lt;span class="s2"&gt;"{\"employeeId\":\"EMP-00104\",\"recipientEmail\":\"emp104@example.com\",\"settlementId\":\"$SETTLEMENT_ID\",\"subject\":\"Clearance\"}"&lt;/span&gt;

&lt;span class="c"&gt;# Inspect live microservice DBs: EMP-00104 is back to ACTIVE, settlement is REVERSED!&lt;/span&gt;
curl -s http://127.0.0.1:3010/api/v1/state
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fax33qx0n6pbhk4s308d6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fax33qx0n6pbhk4s308d6.png" alt=" " width="800" height="1000"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Benchmark Receipts: 2,800 Live Workflows
&lt;/h2&gt;

&lt;p&gt;I wanted real TCP numbers—no mocked HTTP layers—so the repository includes a full Python benchmark harness (&lt;code&gt;eval-harness/&lt;/code&gt;) that hits the live NestJS Gateway and Express cluster over loopback TCP and audits downstream database state via &lt;code&gt;GET /api/v1/state&lt;/code&gt; on every run.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dual-Backend End-to-End Evaluation (&lt;code&gt;1,000&lt;/code&gt; Workflows per Backend)
&lt;/h3&gt;

&lt;p&gt;Each &lt;code&gt;1,000&lt;/code&gt;-workflow run splits across &lt;code&gt;300&lt;/code&gt; nominal 3-step flows, &lt;code&gt;300&lt;/code&gt; schema/type faults, &lt;code&gt;100&lt;/code&gt; policy threshold violations (&lt;code&gt;750,000&lt;/code&gt; &amp;gt; &lt;code&gt;500,000&lt;/code&gt;), and &lt;code&gt;300&lt;/code&gt; mid-workflow Step 3 &lt;code&gt;504&lt;/code&gt; aborts + pivot replay attempts:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Direct Baseline&lt;/th&gt;
&lt;th&gt;AgentSaga (Mem WAL)&lt;/th&gt;
&lt;th&gt;AgentSaga (PostgreSQL 16 ACID)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Schema Defect Interception (&lt;code&gt;HTTP 400&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Policy Threshold Enforcement (&lt;code&gt;HTTP 403&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mid-Flight SAGA State Recovery (&lt;code&gt;HTTP 502&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Post-Abort Pivot Replay Block (&lt;code&gt;HTTP 409&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Schema + Policy Gate (&lt;code&gt;P50 / P95&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.03 / 0.04 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.03 / 0.04 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;In-Gateway Mediation/Step (&lt;code&gt;P50 / P95&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;0.17 / 0.21 ms&lt;/td&gt;
&lt;td&gt;11.76 / 13.53 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;End-to-End Proxy Hop Delta (&lt;code&gt;P50 / P95&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;1.80 / 2.42 ms&lt;/td&gt;
&lt;td&gt;13.45 / 15.72 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Full 3-Step SAGA Rollback (&lt;code&gt;P50 / P95&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;10.91 / 12.43 ms&lt;/td&gt;
&lt;td&gt;60.93 / 75.50 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(In the containerized PostgreSQL 16 run, direct SQL audits on &lt;code&gt;saga_workflows&lt;/code&gt; and &lt;code&gt;saga_step_journal&lt;/code&gt; confirmed &lt;code&gt;310&lt;/code&gt; committed workflows including warm-up, &lt;code&gt;300&lt;/code&gt; compensated workflows, and &lt;code&gt;1,830&lt;/code&gt; step rows with preserved forward responses and latencies.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two takeaways stood out:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Validation &amp;amp; policy gating stays 0.04 ms (&lt;code&gt;P95&lt;/code&gt;) in both modes&lt;/strong&gt; because invalid calls never write to PostgreSQL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Synchronous ACID durability costs ~11.6 ms (&lt;code&gt;P50&lt;/code&gt;)&lt;/strong&gt; per forward step in single-node PostgreSQL 16—tiny next to a 1.5–4 second LLM turn, and crash-safe if the gateway pod restarts mid-saga.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Multi-Agent Concurrency Scaling (&lt;code&gt;800&lt;/code&gt; Mixed Workflows)
&lt;/h3&gt;

&lt;p&gt;To test race safety when multiple agents commit and roll back simultaneously (&lt;code&gt;50%&lt;/code&gt; nominal + &lt;code&gt;50%&lt;/code&gt; Step 3 &lt;code&gt;504&lt;/code&gt; aborts across &lt;code&gt;EMP-10000&lt;/code&gt;..&lt;code&gt;EMP-10199&lt;/code&gt;), &lt;code&gt;concurrency_benchmark.py&lt;/code&gt; ran &lt;code&gt;200&lt;/code&gt; workflows at each worker tier:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concurrent Agents (&lt;code&gt;C&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Throughput&lt;/th&gt;
&lt;th&gt;Step &lt;code&gt;P50&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Step &lt;code&gt;P95&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;State Isolation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;264.3 req/s&lt;/td&gt;
&lt;td&gt;3.36 ms&lt;/td&gt;
&lt;td&gt;5.89 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;10&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;708.1 req/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;12.67 ms&lt;/td&gt;
&lt;td&gt;22.48 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;25&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;684.7 req/s&lt;/td&gt;
&lt;td&gt;30.85 ms&lt;/td&gt;
&lt;td&gt;59.09 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;50&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;634.5 req/s&lt;/td&gt;
&lt;td&gt;63.23 ms&lt;/td&gt;
&lt;td&gt;125.10 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Zero cross-workflow leakage and zero stray clearance emails across all &lt;code&gt;800&lt;/code&gt; concurrent runs.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where This Goes Next
&lt;/h2&gt;

&lt;p&gt;AgentSaga already ships with a GitHub Actions CI pipeline running Jest tests and live PostgreSQL 16 benchmarks on every push, plus an optional Gemini REST agent evaluator (&lt;code&gt;eval-harness/live_llm_agent_eval.py&lt;/code&gt;). Next on the roadmap:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ephemeral Shadow Sandbox Router&lt;/strong&gt; for dry-running &lt;code&gt;critical&lt;/code&gt;-risk mutations before production dispatch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kafka &amp;amp; gRPC Support&lt;/strong&gt; so declarative &lt;code&gt;x-agentsaga-*&lt;/code&gt; compensation works across async event buses, not just synchronous REST.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP (Model Context Protocol) Bridge&lt;/strong&gt; to expose verified, SAGA-wrapped enterprise tools directly to MCP clients.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check out the full codebase, run the benchmark suite yourself, and let me know how you're handling partial tool failures in production:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;GitHub Repo (Apache 2.0):&lt;/strong&gt; &lt;a href="https://github.com/Ockersz/agent-saga" rel="noopener noreferrer"&gt;github.com/Ockersz/agent-saga&lt;/a&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>microservices</category>
      <category>typescript</category>
      <category>ai</category>
    </item>
    <item>
      <title>Solving the Undocumented ZKTeco ADMS Protocol: Hardware Timezones, Infinite Loops, and Python HTTP Push</title>
      <dc:creator>Shahein Ockersz</dc:creator>
      <pubDate>Mon, 09 Nov 2026 11:57:03 +0000</pubDate>
      <link>https://dev.to/shahein_ockersz/solving-the-undocumented-zkteco-adms-protocol-hardware-timezones-infinite-loops-and-python-http-482e</link>
      <guid>https://dev.to/shahein_ockersz/solving-the-undocumented-zkteco-adms-protocol-hardware-timezones-infinite-loops-and-python-http-482e</guid>
      <description>&lt;p&gt;If you have ever been tasked with integrating physical biometric terminals (like ZKTeco F22, IN01, K40, or uFace) with an enterprise web application or ERP, you likely started where everyone does: &lt;strong&gt;legacy TCP polling over port 4370&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;And like most engineers, you quickly hit a brick wall:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The device requires a static LAN IP or VPN tunnel at every site.&lt;/li&gt;
&lt;li&gt;It fails when terminals are deployed in remote branch offices behind NAT firewalls or cellular 4G modems.&lt;/li&gt;
&lt;li&gt;Polling in batch intervals means punch logs can take 15 to 60 minutes to appear in your payroll system.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Fortunately, modern ZKTeco firmware includes an outbound cloud sync feature called &lt;strong&gt;ADMS (Automatic Data Master Server)&lt;/strong&gt;. Instead of your server reaching into the device, the device opens an outbound HTTP connection and pushes attendance punches in real-time.&lt;/p&gt;

&lt;p&gt;However, the protocol is notoriously undocumented. While attempting to implement a production ADMS server, I ran into two critical firmware quirks that cost days of packet-sniffing and debugging.&lt;/p&gt;

&lt;p&gt;Here is how the protocol works under the hood, how I resolved the hardcoded firmware bugs, and how you can run a production-ready ADMS push server in Python.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. High-Level Architecture: Outbound Push vs. Inbound Polling
&lt;/h2&gt;

&lt;p&gt;In legacy polling, your server acts as the client and dials into port 4370 of each device. In ADMS HTTP Push, the roles are reversed:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwrbiz0zt4sumkpkq1ntf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwrbiz0zt4sumkpkq1ntf.png" alt="ZKTeco ADMS Real-Time HTTP Push Architecture" width="800" height="457"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Why HTTP Push Wins:
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Zero Inbound Firewall Configuration:&lt;/strong&gt; The terminal establishes an outbound HTTP connection (&lt;code&gt;0.0.0.0:8080&lt;/code&gt;) to your server. No port forwarding or public static IPs are needed on the device side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sub-Second Latency:&lt;/strong&gt; When an employee scans a fingerprint or RFID card, the punch is POSTed to your server within 1 second.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resilient Offline Buffering:&lt;/strong&gt; If the factory internet connection drops, the terminal stores records in flash memory and streams the entire backlog upon reconnecting.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  2. Engineering Discovery #1: The Factory UTC+8 Timezone Bug
&lt;/h2&gt;

&lt;p&gt;This was the most elusive bug I encountered.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Problem
&lt;/h3&gt;

&lt;p&gt;When you configure an ADMS server, the device periodically issues a heartbeat GET request to &lt;code&gt;/iclock/getrequest&lt;/code&gt;. Standard HTTP (RFC 7231) mandates that the &lt;code&gt;Date&lt;/code&gt; header in server responses must always be in GMT/UTC.&lt;/p&gt;

&lt;p&gt;However, ZKTeco's proprietary firmware parses the HTTP &lt;code&gt;Date&lt;/code&gt; header and &lt;strong&gt;blindly adds +8 hours&lt;/strong&gt; (assuming the server is operating in China Standard Time, UTC+8). &lt;/p&gt;

&lt;p&gt;Even if you manually configure the device's clock or local timezone in its physical settings menu, the next HTTP heartbeat overrides the clock, jumping the device forward or backward into the wrong time. If your company operates in Sri Lanka or India (UTC+5:30), for example, the device will jump by &lt;strong&gt;2.5 hours&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Solution: Compensatory Date Header Shifting
&lt;/h3&gt;

&lt;p&gt;Rather than fighting the terminal's proprietary firmware, we can compensate at the server level by shifting the HTTP &lt;code&gt;Date&lt;/code&gt; header backwards:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Compensatory Date Shift Formula:&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
&lt;code&gt;Date Header Shift = Device Factory Offset (UTC+8) - Target Local Offset&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For Sri Lanka or India (UTC+5:30):  &lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;Shift = 8.0 - 5.5 = 2.5 hours (2 hours, 30 minutes backward)&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;We implement this in Python Flask using an &lt;code&gt;@app.after_request&lt;/code&gt; middleware:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timezone&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utils&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;formatdate&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;flask&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="no"&gt;DEVICE_UTC_OFFSET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;8.0&lt;/span&gt;   &lt;span class="c1"&gt;# Factory hardware default&lt;/span&gt;
&lt;span class="no"&gt;SERVER_UTC_OFFSET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;5.5&lt;/span&gt;   &lt;span class="c1"&gt;# Target timezone (e.g., UTC+5:30)&lt;/span&gt;

&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;after_request&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fix_date_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Compensate for the hardcoded UTC+8 firmware bug."""&lt;/span&gt;
    &lt;span class="n"&gt;shift_hours&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;DEVICE_UTC_OFFSET&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="no"&gt;SERVER_UTC_OFFSET&lt;/span&gt;
    &lt;span class="n"&gt;adjusted_utc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;shift_hours&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Format according to RFC 7231 GMT specification&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Date"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;formatdate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;adjusted_utc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timestamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usegmt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the device reads this shifted header and adds its factory +8 hours, it lands on your exact local time down to the second.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Engineering Discovery #2: The Infinite Retransmission Loop
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The Problem
&lt;/h3&gt;

&lt;p&gt;When a device records attendance punches, it sends a &lt;code&gt;POST&lt;/code&gt; request to &lt;code&gt;/iclock/cdata?table=ATTLOG&lt;/code&gt; with a tab-delimited payload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10024   2026-11-09 08:31:14 1   0   0   0
10118   2026-11-09 08:31:22 2   0   0   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Standard HTTP conventions would suggest returning &lt;code&gt;200 OK&lt;/code&gt; or &lt;code&gt;{"status": "success"}&lt;/code&gt;. &lt;/p&gt;

&lt;p&gt;If you do this, the device will &lt;strong&gt;never acknowledge the transmission as complete&lt;/strong&gt;. It will resend the exact same batch of punches on every single heartbeat interval (every 10 seconds), forever. Over time, the device's network buffer exhausts, leading to packet drops, terminal freeze-ups, and duplicate database rows.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Solution: The &lt;code&gt;OK: &amp;lt;count&amp;gt;&lt;/code&gt; Acknowledgment
&lt;/h3&gt;

&lt;p&gt;The ADMS push firmware specifically looks for a response in the exact format:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OK: &amp;lt;number_of_records_processed&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the payload contained 2 punches, the server must respond with plain text:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OK: 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/iclock/cdata"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;methods&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"POST"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;receive_attendance_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"table"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"ATTLOG"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"OK"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mimetype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;raw_body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"replace"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;parse_attlog&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw_body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Save to database and deduplicate&lt;/span&gt;
    &lt;span class="n"&gt;inserted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duplicates&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;save_punches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# CRITICAL: Return the count to prevent retransmission loop&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"OK: {len(records)}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mimetype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once the terminal receives &lt;code&gt;OK: 2&lt;/code&gt;, it purges the processed records from its outbound transmission queue and resumes normal operation.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. End-to-End Implementation
&lt;/h2&gt;

&lt;p&gt;To help developers avoid these headaches, I have open-sourced a complete, production-ready implementation:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GitHub Repository:&lt;/strong&gt; &lt;a href="https://github.com/Ockersz/zkteco-adms-server" rel="noopener noreferrer"&gt;https://github.com/Ockersz/zkteco-adms-server&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Included Features:
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Zero-Setup SQLite Default:&lt;/strong&gt; Runs out of the box with &lt;code&gt;python app.py&lt;/code&gt; (optional MySQL support via &lt;code&gt;.env&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timezone Middleware:&lt;/strong&gt; Fully configurable offsets and per-device overrides.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heartbeat &amp;amp; Handshake Engine:&lt;/strong&gt; Implements &lt;code&gt;/iclock/cdata&lt;/code&gt; and &lt;code&gt;/iclock/getrequest&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Async Webhook Forwarder:&lt;/strong&gt; Automatically pipes incoming punches into ERP, SAP, or payroll APIs in real time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Legacy TCP Polling Fallback:&lt;/strong&gt; A standalone script for older non-ADMS terminals (Port 4370).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Production Templates:&lt;/strong&gt; Includes ready-to-use Linux &lt;code&gt;systemd&lt;/code&gt; service units and &lt;code&gt;nginx&lt;/code&gt; reverse proxy configs.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  5. Physical Terminal Setup (In 60 Seconds)
&lt;/h2&gt;

&lt;p&gt;To connect any ZKTeco device to this server:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Open the device menu and authenticate as Administrator.&lt;/li&gt;
&lt;li&gt;Go to &lt;strong&gt;Comm.&lt;/strong&gt; &amp;gt; &lt;strong&gt;Cloud Server Setting&lt;/strong&gt; (or &lt;strong&gt;ADMS&lt;/strong&gt; / &lt;strong&gt;Web Server&lt;/strong&gt;).&lt;/li&gt;
&lt;li&gt;Set &lt;strong&gt;Server Address&lt;/strong&gt; to your server IP or domain (e.g. &lt;code&gt;attendance.yourcompany.com&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Set &lt;strong&gt;Server Port&lt;/strong&gt; to &lt;code&gt;8080&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Set &lt;strong&gt;Enable Domain Name&lt;/strong&gt; to &lt;code&gt;ON&lt;/code&gt; if using a hostname, or &lt;code&gt;OFF&lt;/code&gt; if using a numeric IP.&lt;/li&gt;
&lt;li&gt;Save and reboot. A cloud icon will appear in the device header once connected.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Conclusion &amp;amp; Takeaways
&lt;/h2&gt;

&lt;p&gt;Working with proprietary IoT and enterprise hardware often requires reading between the lines of sparse documentation. By understanding the HTTP Date header clock synchronization mechanics and implementing exact acknowledgment counts, you can transform legacy biometric terminals into reliable, modern cloud IoT clients.&lt;/p&gt;

&lt;p&gt;Feel free to star or fork the repository on &lt;a href="https://github.com/Ockersz/zkteco-adms-server" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt; or connect with me on &lt;a href="https://www.linkedin.com/in/shahein-ockersz/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Author:&lt;/strong&gt; &lt;strong&gt;Shahein Noel Ockersz&lt;/strong&gt; | Senior Software Engineer &amp;amp; Systems Architect&lt;br&gt;&lt;br&gt;
&lt;strong&gt;Live Portfolio:&lt;/strong&gt; &lt;a href="https://shahein-portfolio.vercel.app/" rel="noopener noreferrer"&gt;shahein-portfolio.vercel.app&lt;/a&gt;&lt;br&gt;&lt;br&gt;
&lt;strong&gt;GitHub:&lt;/strong&gt; &lt;a href="https://github.com/Ockersz" rel="noopener noreferrer"&gt;github.com/Ockersz&lt;/a&gt; | &lt;strong&gt;LinkedIn:&lt;/strong&gt; &lt;a href="https://www.linkedin.com/in/shahein-ockersz/" rel="noopener noreferrer"&gt;linkedin.com/in/shahein-ockersz&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>iot</category>
      <category>architecture</category>
      <category>backend</category>
    </item>
  </channel>
</rss>
