<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: ZoomApps</title>
    <description>The latest articles on DEV Community by ZoomApps (@zoomapps).</description>
    <link>https://dev.to/zoomapps</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4209265%2F3203cc66-ada1-406d-a5ce-4a9f5d037262.png</url>
      <title>DEV Community: ZoomApps</title>
      <link>https://dev.to/zoomapps</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/zoomapps"/>
    <language>en</language>
    <item>
      <title>On-prem AI infrastructure for a mid-size company: what to buy and what to rent</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Mon, 16 Nov 2026 06:01:34 +0000</pubDate>
      <link>https://dev.to/zoomapps/on-prem-ai-infrastructure-for-a-mid-size-company-what-to-buy-and-what-to-rent-287a</link>
      <guid>https://dev.to/zoomapps/on-prem-ai-infrastructure-for-a-mid-size-company-what-to-buy-and-what-to-rent-287a</guid>
      <description>&lt;p&gt;A company with a few hundred employees usually reaches the same point about a year after the first chatbot pilot. The pilot worked, legal does not want customer data leaving the country, the API bill keeps growing, and someone asks whether it would be cheaper to "just buy a GPU server". The honest answer is that it depends less on the hardware than on which workload you are talking about.&lt;/p&gt;

&lt;p&gt;This post walks through the decision the way an engineering team should make it: split the workloads, size memory from the model, calculate cost per hour actually used, and only then look at price lists. There is a small Python model you can run with your own quotes, a serving config, and the pitfalls that tend to show up after the purchase order is signed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two workloads with opposite profiles
&lt;/h2&gt;

&lt;p&gt;Most AI infrastructure discussions go wrong by treating "AI" as one workload. It is at least two.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Training and fine-tuning&lt;/strong&gt; are bursty. You run a LoRA fine-tune for a day, evaluate for a week, maybe run another job next month. While a job runs it wants as much memory and interconnect bandwidth as you can give it, because gradients, optimizer state and activations all compete for VRAM, and multi-GPU jobs spend real time moving tensors between cards.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Inference&lt;/strong&gt; is steady. It runs every working hour, often around the clock, and it is judged on two numbers: latency per request and cost per response. A single mid-range card can serve a quantized 8B model to a few dozen concurrent users comfortably. What matters is that it is always on and always close to the data.&lt;/p&gt;

&lt;p&gt;Those profiles map naturally onto the buy-or-rent question:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;Utilization pattern&lt;/th&gt;
&lt;th&gt;Usual answer&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Production inference&lt;/td&gt;
&lt;td&gt;High, predictable&lt;/td&gt;
&lt;td&gt;Buy, or rent long-term with fixed capacity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fine-tuning experiments&lt;/td&gt;
&lt;td&gt;Low, bursty&lt;/td&gt;
&lt;td&gt;Rent by the hour or week&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full pre-training&lt;/td&gt;
&lt;td&gt;Very high, short-lived&lt;/td&gt;
&lt;td&gt;Almost never a mid-size company's job&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Embeddings and batch jobs&lt;/td&gt;
&lt;td&gt;Medium, schedulable&lt;/td&gt;
&lt;td&gt;Run on inference hardware off-peak&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;If you buy one big configuration for everything, you pay for training-class interconnect that sits idle while serving requests, or you get a cheap inference box that cannot fit a fine-tuning job. Planning the two separately is the single biggest cost lever.&lt;/p&gt;

&lt;h2&gt;
  
  
  Size memory from the model, not from the spec sheet
&lt;/h2&gt;

&lt;p&gt;VRAM is the constraint that decides everything else. If the model does not fit on one card, you have to split it with tensor or pipeline parallelism, and splitting across cards over PCIe costs throughput. So start from the model.&lt;/p&gt;

&lt;p&gt;Weights are easy: parameters times bytes per parameter.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;8B model: about 16 GB at FP16, 8 GB at FP8, roughly 5 GB at 4-bit&lt;/li&gt;
&lt;li&gt;70B model: about 140 GB at FP16, 70 GB at FP8, roughly 40 GB at 4-bit&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The part people underestimate is the KV cache, which grows with context length and the number of concurrent requests. For a transformer with grouped-query attention:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;kv_cache_bytes_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes_per_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# 2 = one tensor for keys, one for values&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bytes_per_value&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;kv_cache_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context_len&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes_per_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;per_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;kv_cache_bytes_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes_per_value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;per_token&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;context_len&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt;

&lt;span class="c1"&gt;# Llama-style 8B: 32 layers, 8 KV heads, head_dim 128&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;kv_cache_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context_len&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# ~17.2 GB&lt;/span&gt;

&lt;span class="c1"&gt;# Llama-style 70B: 80 layers, 8 KV heads, head_dim 128&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;kv_cache_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context_len&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# ~42.9 GB&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So an 8B model at FP16 with 16 users each holding an 8K context needs around 16 GB for weights plus 17 GB for cache, plus a few GB of runtime overhead. That is a 48 GB card, not a 24 GB one. Quantize the weights to FP8 (8 GB) and store the cache in FP8 (about 9 GB) and it fits on a 24 GB card again. These trade-offs are worth making on paper before hardware arrives.&lt;/p&gt;

&lt;p&gt;The 70B case is the classic trap. At FP8 the weights alone are 70 GB, so an 80 GB card leaves almost nothing for cache. In practice you either quantize to 4-bit, split across two cards, or move to a card with more memory. Each option has a different cost per response, which is the number you should compare.&lt;/p&gt;

&lt;h2&gt;
  
  
  The break-even model
&lt;/h2&gt;

&lt;p&gt;The question "buy or rent" reduces to cost per hour of useful GPU time. Owned hardware has a fixed monthly cost whether you use it or not; rented hardware costs roughly the same per hour regardless of how much you use it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;HOURS_PER_MONTH&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;730&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;owned_cost_per_used_hour&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;capex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;months_of_life&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opex_per_month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;utilization&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""capex: server price; opex: power, rack space, support, admin time."""&lt;/span&gt;
    &lt;span class="n"&gt;monthly&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;capex&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;months_of_life&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;opex_per_month&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;monthly&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;HOURS_PER_MONTH&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;utilization&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;breakeven_utilization&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;capex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;months_of_life&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opex_per_month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rent_per_hour&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;monthly&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;capex&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;months_of_life&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;opex_per_month&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;monthly&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;HOURS_PER_MONTH&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;rent_per_hour&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Illustrative inputs only. Replace with your own quotes.&lt;/span&gt;
&lt;span class="n"&gt;capex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60_000&lt;/span&gt;          &lt;span class="c1"&gt;# server with GPUs, in your currency units&lt;/span&gt;
&lt;span class="n"&gt;life&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;36&lt;/span&gt;               &lt;span class="c1"&gt;# months before it is obsolete for your workload&lt;/span&gt;
&lt;span class="n"&gt;opex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;900&lt;/span&gt;              &lt;span class="c1"&gt;# power + colocation + share of an engineer&lt;/span&gt;
&lt;span class="n"&gt;rent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;3.5&lt;/span&gt;              &lt;span class="c1"&gt;# comparable rented server, per hour&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"break-even utilization: {breakeven_utilization(capex, life, opex, rent):.0%}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;owned_cost_per_used_hour&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;capex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;life&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With these placeholder inputs the break-even sits around 100%, which means renting wins at any utilization. Change the life to 48 months and rent to 5.0 and it drops to roughly 60%. The exact numbers are not the point; the shape is. Three things move the result much more than the GPU price:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Utilization.&lt;/strong&gt; A production inference server running 18 hours a day sits at 75%. A fine-tuning box used for two jobs a month sits under 10%. The same hardware is a good purchase for the first and a poor one for the second.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Useful life.&lt;/strong&gt; Teams plan for five years and replace in three because a new model family needs more memory than the card has. Be pessimistic here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Opex you forget.&lt;/strong&gt; Power at full load, cooling, rack space, spare parts, the engineer who patches drivers at night. If you do not have a server room built for this density, colocation fees go into opex too.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then convert to the metric the business cares about: cost per response. Measure tokens per second at your real concurrency on a candidate card, divide hourly cost by responses per hour, and compare cards on that number. A cheaper card that serves half the throughput is often the more expensive option.&lt;/p&gt;

&lt;h2&gt;
  
  
  Serving configuration that respects the budget
&lt;/h2&gt;

&lt;p&gt;Whatever you buy or rent, the serving layer decides how much of that VRAM turns into throughput. A typical vLLM launch for an 8B model on one 48 GB card:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --port 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And for a 70B model split across two cards:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;your-70b-awq-checkpoint&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.92
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notes from running this in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;--max-model-len&lt;/code&gt; is a memory decision, not a feature flag. Setting it to the model's maximum (often 128K) reserves capacity for contexts your users never send. Set it to what your prompts actually need, measured from logs.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--max-num-seqs&lt;/code&gt; caps concurrency. Too high and requests queue inside the engine with poor latency; too low and the card idles. Load test with your real prompt length distribution.&lt;/li&gt;
&lt;li&gt;Tensor parallelism over PCIe works, but expect noticeably lower scaling than on cards linked with a high-bandwidth bridge. If a model fits on one larger card, that is usually faster than two smaller ones.&lt;/li&gt;
&lt;li&gt;Pin your stack. Write down the driver version, CUDA version, PyTorch version and serving engine version that passed load tests, and build images from that. A driver update that silently changes behaviour is a common cause of the "it was fast last week" ticket.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pitfalls that appear after delivery
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Power and cooling.&lt;/strong&gt; A 4-GPU inference server can draw 2 to 3 kW under load; an 8-GPU training-class machine can pass 10 kW. A typical office server room circuit will not carry that, and air conditioning sized for a few file servers will not remove the heat. Check this before ordering, not when the breaker trips.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Noise and physical space.&lt;/strong&gt; GPU servers are loud. "We will put it in the office" ends the first day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lead times.&lt;/strong&gt; High-end cards can have delivery times measured in months. If the project has a deadline, rent for the first phase and buy once the workload is proven.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Driver and environment work.&lt;/strong&gt; Getting CUDA, cuDNN, PyTorch and the serving engine to agree is a day of work on a good day. Budget for it or buy hardware delivered with a tested environment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data lifecycle.&lt;/strong&gt; For owned hardware, plan disk sanitization at end of life. For rented hardware, get the provider to confirm in writing that storage is wiped on release, and agree how long you have to copy data out before shutdown.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data residency.&lt;/strong&gt; In banking, healthcare and the public sector, regulations often forbid sending personal data abroad. That rules out many hosted APIs regardless of price, and it is often the actual reason on-prem or in-country hosting gets approved. Check this first, since it can make the cost comparison irrelevant.&lt;/p&gt;

&lt;h2&gt;
  
  
  A sensible default architecture
&lt;/h2&gt;

&lt;p&gt;For most mid-size companies the pattern that holds up looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Inference on dedicated capacity you control&lt;/strong&gt;, owned or rented long-term, sized from the KV cache math above, inside the jurisdiction your data must stay in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fine-tuning on rented capacity by the hour or week&lt;/strong&gt;, spun up when there is a measured quality gap between a generic model and your domain, and released when the job is done.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start with an open model as-is.&lt;/strong&gt; Fine-tune only after you have an evaluation set that shows the generic model failing on your tasks. Many projects never need the second step.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Review utilization quarterly.&lt;/strong&gt; If the rented inference server is busy 80% of the time and the workload is stable, that is the moment to price a purchase. If an owned box sits at 20%, consolidate.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Buying hardware is a bet on a workload staying stable for three years. Renting is paying a premium for the right to change your mind. For production inference with steady traffic the bet often pays; for experiments it rarely does.&lt;/p&gt;

&lt;p&gt;If you are working through this decision and want a second opinion on sizing for a specific model, request volume and data residency requirement, here is how we plan &lt;a href="https://zoomapps.kz/en/ai-infrastructure" rel="noopener noreferrer"&gt;AI server infrastructure with training and inference sized separately&lt;/a&gt;, including hosting in a Kazakhstan data centre.&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>llm</category>
      <category>devops</category>
      <category>infrastructure</category>
    </item>
    <item>
      <title>Multi-tenant SaaS on PostgreSQL: schema-per-tenant vs row-level security</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Thu, 12 Nov 2026 06:01:37 +0000</pubDate>
      <link>https://dev.to/zoomapps/multi-tenant-saas-on-postgresql-schema-per-tenant-vs-row-level-security-332f</link>
      <guid>https://dev.to/zoomapps/multi-tenant-saas-on-postgresql-schema-per-tenant-vs-row-level-security-332f</guid>
      <description>&lt;p&gt;Every B2B SaaS product eventually gets the same bug report: "I can see another company's invoices." It rarely comes from a sophisticated attack. It comes from a missing &lt;code&gt;WHERE tenant_id = ?&lt;/code&gt; in a report query written at 11 p.m., or from a background job that reused a connection with the wrong context.&lt;/p&gt;

&lt;p&gt;The way you separate tenant data in PostgreSQL decides how likely that bug is, how painful migrations become at 500 customers, and whether you can move one large customer to its own server later. This post compares the two approaches most teams argue about, schema-per-tenant and a shared schema with row-level security (RLS), with the configs and pitfalls that only show up in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three options in one paragraph each
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Shared schema, &lt;code&gt;tenant_id&lt;/code&gt; column, RLS.&lt;/strong&gt; All tenants live in the same tables. Every tenant-owned table has a &lt;code&gt;tenant_id&lt;/code&gt;, and PostgreSQL policies filter rows based on a session variable. One set of tables, one migration run, one place to index.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema-per-tenant.&lt;/strong&gt; Each tenant gets its own schema (&lt;code&gt;tenant_0042.orders&lt;/code&gt;, &lt;code&gt;tenant_0043.orders&lt;/code&gt;), and the application switches &lt;code&gt;search_path&lt;/code&gt; per request. Isolation comes from namespaces, not from filters.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Database-per-tenant.&lt;/strong&gt; Each tenant gets its own database, sometimes its own server. Strongest isolation, highest operational cost. Usually reserved for customers with regulatory or contractual requirements.&lt;/p&gt;

&lt;p&gt;Most teams end up with a hybrid: shared schema with RLS for the long tail, and a dedicated database for the few tenants who need it. Keep that end state in mind, because it affects the choice you make on day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Shared schema with RLS: a setup that actually holds
&lt;/h2&gt;

&lt;p&gt;Here is a minimal setup that fails closed, meaning that if the application forgets to set the tenant, it sees nothing rather than everything.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;tenants&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;         &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nf"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;name&lt;/span&gt;       &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;plan&lt;/span&gt;       &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="s1"&gt;'starter'&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;tenant_id&lt;/span&gt;  &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;REFERENCES&lt;/span&gt; &lt;span class="nf"&gt;tenants&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;         &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nf"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;email&lt;/span&gt;      &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;tenant_id&lt;/span&gt;   &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;          &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nf"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt;      &lt;span class="kt"&gt;numeric&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;created_at&lt;/span&gt;  &lt;span class="kt"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;FOREIGN&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;REFERENCES&lt;/span&gt; &lt;span class="nf"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;invoices_tenant_created&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="no"&gt;ENABLE&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="no"&gt;LEVEL&lt;/span&gt; &lt;span class="no"&gt;SECURITY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="no"&gt;FORCE&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="no"&gt;LEVEL&lt;/span&gt; &lt;span class="no"&gt;SECURITY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt;  &lt;span class="no"&gt;ENABLE&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="no"&gt;LEVEL&lt;/span&gt; &lt;span class="no"&gt;SECURITY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt;  &lt;span class="no"&gt;FORCE&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="no"&gt;LEVEL&lt;/span&gt; &lt;span class="no"&gt;SECURITY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;POLICY&lt;/span&gt; &lt;span class="n"&gt;tenant_isolation&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt;
  &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;current_setting&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'app.tenant_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="kt"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="k"&gt;CHECK&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;current_setting&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'app.tenant_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="kt"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;POLICY&lt;/span&gt; &lt;span class="n"&gt;tenant_isolation&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt;
  &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;current_setting&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'app.tenant_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="kt"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="k"&gt;CHECK&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;current_setting&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'app.tenant_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="kt"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the application role:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt; &lt;span class="no"&gt;LOGIN&lt;/span&gt; &lt;span class="no"&gt;PASSWORD&lt;/span&gt; &lt;span class="s1"&gt;'...'&lt;/span&gt; &lt;span class="no"&gt;NOSUPERUSER&lt;/span&gt; &lt;span class="no"&gt;NOBYPASSRLS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;DELETE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every detail here exists because of a real failure mode.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;FORCE ROW LEVEL SECURITY&lt;/code&gt;.&lt;/strong&gt; Without it, the table owner bypasses policies. If your application connects as the same role that ran the migrations (very common in small teams), RLS is silently off. Superusers and roles with &lt;code&gt;BYPASSRLS&lt;/code&gt; skip policies regardless, so the app must connect as a dedicated role with neither.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;current_setting(..., true)&lt;/code&gt; plus &lt;code&gt;NULLIF&lt;/code&gt;.&lt;/strong&gt; The second argument makes a missing setting return NULL instead of raising an error. &lt;code&gt;tenant_id = NULL&lt;/code&gt; is never true, so no rows match. The &lt;code&gt;NULLIF&lt;/code&gt; matters because once a custom setting has been set in a session and the transaction ends, PostgreSQL reverts it to an empty string, not NULL, and &lt;code&gt;''::uuid&lt;/code&gt; throws an error. Without &lt;code&gt;NULLIF&lt;/code&gt; you get intermittent "invalid input syntax for type uuid" errors on pooled connections.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;WITH CHECK&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;USING&lt;/code&gt; filters what you read. &lt;code&gt;WITH CHECK&lt;/code&gt; stops an insert or update from writing a row with someone else's &lt;code&gt;tenant_id&lt;/code&gt;. Skip it and a bug in the API layer can plant data into another tenant.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Composite foreign keys.&lt;/strong&gt; This one surprises people: referential integrity checks bypass RLS. A plain &lt;code&gt;customer_id REFERENCES customers(id)&lt;/code&gt; lets tenant A's invoice point at tenant B's customer, and the database will accept it. Making the key &lt;code&gt;(tenant_id, customer_id)&lt;/code&gt; makes cross-tenant references structurally impossible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;tenant_id&lt;/code&gt; as the leading index column.&lt;/strong&gt; Almost every query is "this tenant's rows, sorted or filtered by something". Indexes that start with &lt;code&gt;tenant_id&lt;/code&gt; keep those queries on a narrow slice of the table.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setting the tenant per request, and why &lt;code&gt;SET&lt;/code&gt; is a trap
&lt;/h2&gt;

&lt;p&gt;The tenant context must be scoped to a transaction, not a connection:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="nf"&gt;set_config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'app.tenant_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;  &lt;span class="c1"&gt;-- true = local to this transaction&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt; &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In application code, wrap it so nobody can forget:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;contextlib&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;contextmanager&lt;/span&gt;

&lt;span class="nd"&gt;@contextmanager&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tenant_tx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="s2"&gt;"SELECT set_config('app.tenant_id', %s, true)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A plain &lt;code&gt;SET app.tenant_id = '...'&lt;/code&gt; (session level) survives after the request ends. With PgBouncer in transaction mode, the next transaction on that server connection may belong to a different client, and it inherits the previous tenant. That is exactly the "I see another company's data" bug, and it only appears under load, when connections are being reused aggressively. &lt;code&gt;set_config(..., true)&lt;/code&gt; or &lt;code&gt;SET LOCAL&lt;/code&gt; disappears at commit or rollback, so it is safe with transaction pooling.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls that RLS does not cover
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Views.&lt;/strong&gt; By default a view runs with the privileges of its owner. If the owner is the migration role, queries through the view skip the policies. On PostgreSQL 15 and later, create views with &lt;code&gt;WITH (security_invoker = true)&lt;/code&gt;. On older versions, make sure views are owned by a role subject to RLS.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;SECURITY DEFINER&lt;/code&gt; functions.&lt;/strong&gt; Same problem: they run as their owner. Audit every one of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Background workers and admin tools.&lt;/strong&gt; Cron jobs, queue consumers and internal dashboards often connect with a privileged role "just to make it work". Give them their own role with RLS on, and set the tenant explicitly per job. For genuinely cross-tenant tasks (billing runs, platform analytics), use a separate role with &lt;code&gt;BYPASSRLS&lt;/code&gt; and keep its credentials out of the web app.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Caches and search indexes.&lt;/strong&gt; RLS protects PostgreSQL only. Redis keys, Elasticsearch documents and files in object storage need the tenant in the key or path, and their own checks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Planner surprises.&lt;/strong&gt; Policies are added as extra predicates. With simple equality on &lt;code&gt;tenant_id&lt;/code&gt; and a good index, the overhead is negligible. If a policy calls a non-trivial function (for example, a lookup of the user's tenants), it may be evaluated per row. Mark helper functions &lt;code&gt;STABLE&lt;/code&gt;, wrap them in a subselect so they are computed once per query, and check &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; on your top queries before and after enabling RLS.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Schema-per-tenant: what it really costs
&lt;/h2&gt;

&lt;p&gt;Schema-per-tenant looks attractive: no &lt;code&gt;tenant_id&lt;/code&gt; columns, no policies, and a tenant's data is easy to dump or delete with &lt;code&gt;DROP SCHEMA tenant_0042 CASCADE&lt;/code&gt;. The costs appear as tenants grow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Catalog size.&lt;/strong&gt; Every table, index, sequence and TOAST table is a row in the system catalogs. A product with 150 tables and 300 indexes, times 2,000 tenants, means well over a million relations. &lt;code&gt;pg_dump&lt;/code&gt; slows down sharply, &lt;code&gt;\d&lt;/code&gt; in psql becomes sluggish, and each backend's relation cache grows as it touches more schemas, which multiplies memory across hundreds of connections.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Migrations.&lt;/strong&gt; One schema change becomes N schema changes. A migration that takes 200 ms per schema takes almost 7 minutes for 2,000 tenants, and some of them will fail midway on data you did not expect. You need per-schema version tracking, a runner that can resume, and application code that tolerates tenants being on different schema versions during the rollout.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- a per-schema version table you will end up writing anyway&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;tenant_0042&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema_version&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;applied_at&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Pooling.&lt;/strong&gt; The &lt;code&gt;search_path&lt;/code&gt; problem is the same as the &lt;code&gt;SET&lt;/code&gt; problem above. Use &lt;code&gt;SET LOCAL search_path TO tenant_0042, public&lt;/code&gt; inside each transaction, never a session-level &lt;code&gt;SET&lt;/code&gt;, if a transaction pooler sits in front of the database. Also, unqualified names resolve to &lt;code&gt;public&lt;/code&gt; if a table is missing from the tenant schema, which can quietly read shared data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-tenant queries.&lt;/strong&gt; "Revenue by plan across all customers" becomes a generated &lt;code&gt;UNION ALL&lt;/code&gt; over thousands of schemas, or an ETL job into a separate warehouse. With a shared schema it is a &lt;code&gt;GROUP BY&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Schema-per-tenant does have real advantages: per-tenant restores are simple, a missing filter cannot leak data, and per-tenant customization (an extra table for one enterprise customer) is possible. It fits products with tens or low hundreds of tenants, each with substantial data, for example an ERP-like system sold to mid-size companies. It fits poorly with self-serve products that sign up hundreds of small tenants a month.&lt;/p&gt;

&lt;h2&gt;
  
  
  A rough decision guide
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Factor&lt;/th&gt;
&lt;th&gt;Shared + RLS&lt;/th&gt;
&lt;th&gt;Schema-per-tenant&lt;/th&gt;
&lt;th&gt;DB-per-tenant&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Comfortable tenant count&lt;/td&gt;
&lt;td&gt;thousands to millions&lt;/td&gt;
&lt;td&gt;tens to low hundreds&lt;/td&gt;
&lt;td&gt;a few to tens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Migration effort&lt;/td&gt;
&lt;td&gt;one run&lt;/td&gt;
&lt;td&gt;N runs&lt;/td&gt;
&lt;td&gt;N runs, N servers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Leak risk from a forgotten filter&lt;/td&gt;
&lt;td&gt;low with RLS on and forced&lt;/td&gt;
&lt;td&gt;very low&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cross-tenant analytics&lt;/td&gt;
&lt;td&gt;simple SQL&lt;/td&gt;
&lt;td&gt;UNION or ETL&lt;/td&gt;
&lt;td&gt;ETL only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per-tenant backup and restore&lt;/td&gt;
&lt;td&gt;needs tooling&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pg_dump -n&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;native&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Noisy neighbor control&lt;/td&gt;
&lt;td&gt;weak&lt;/td&gt;
&lt;td&gt;weak&lt;/td&gt;
&lt;td&gt;strong&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note the last row. Schemas share CPU, memory and I/O on one server just as rows do. If one tenant runs heavy reports, schema-per-tenant does not protect the others. Only a separate database (on separate resources) does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Plan for the hybrid from day one
&lt;/h2&gt;

&lt;p&gt;The design that ages best is a shared schema with RLS plus a routing table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;tenant_placement&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;tenant_id&lt;/span&gt;  &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="k"&gt;REFERENCES&lt;/span&gt; &lt;span class="nf"&gt;tenants&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;dsn_alias&lt;/span&gt;  &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="s1"&gt;'shared-1'&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application looks up &lt;code&gt;dsn_alias&lt;/code&gt; and connects to the right cluster. Because every row already carries &lt;code&gt;tenant_id&lt;/code&gt;, moving one customer to a dedicated database is a scripted export:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="no"&gt;COPY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;invoices&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'...'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="no"&gt;STDOUT&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;FORMAT&lt;/span&gt; &lt;span class="n"&gt;binary&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;followed by an import into an identical schema on the new server and a change to one row in &lt;code&gt;tenant_placement&lt;/code&gt;. Going the other way, from thousands of schemas back to shared tables, means adding &lt;code&gt;tenant_id&lt;/code&gt; to everything and rewriting the data, which is effectively a re-platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test isolation like a feature
&lt;/h2&gt;

&lt;p&gt;Put a test in CI that connects as &lt;code&gt;app_user&lt;/code&gt;, sets tenant A, and tries to read, update and insert rows belonging to tenant B through every table, view and API endpoint. Also run one test with no tenant set and assert it returns zero rows. Add a migration check that fails if any table with a &lt;code&gt;tenant_id&lt;/code&gt; column lacks &lt;code&gt;relrowsecurity&lt;/code&gt; and &lt;code&gt;relforcerowsecurity&lt;/code&gt; in &lt;code&gt;pg_class&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;pg_attribute&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attrelid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'tenant_id'&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;pg_namespace&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relnamespace&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nspname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'public'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relkind&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'r'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relrowsecurity&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relforcerowsecurity&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An empty result is the only acceptable one. This catches the most common regression: a new table added six months later by someone who never read the original design notes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;For most SaaS products, start with a shared schema, &lt;code&gt;tenant_id&lt;/code&gt; on every tenant-owned table, forced RLS, composite foreign keys and transaction-scoped tenant context. Reach for schema-per-tenant only when you have few, large tenants who need per-tenant restores or customization. Keep database-per-tenant as an option for the customers whose contracts demand it, and make it reachable through a routing table rather than a rewrite.&lt;/p&gt;

&lt;p&gt;Isolation is only one of the decisions that are hard to change later; how plans and limits are counted and what happens to data on non-payment shape the schema just as much. If you are scoping a subscription product and want those questions settled before the first line of code, this overview of &lt;a href="https://zoomapps.kz/en/razrabotka-saas-platformy" rel="noopener noreferrer"&gt;how we approach SaaS platform development&lt;/a&gt; lists what goes into a first release.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>saas</category>
      <category>database</category>
      <category>security</category>
    </item>
    <item>
      <title>Voice Robot for Outbound Calls: A Latency Budget from ASR to TTS</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Mon, 09 Nov 2026 06:01:26 +0000</pubDate>
      <link>https://dev.to/zoomapps/voice-robot-for-outbound-calls-a-latency-budget-from-asr-to-tts-5hhk</link>
      <guid>https://dev.to/zoomapps/voice-robot-for-outbound-calls-a-latency-budget-from-asr-to-tts-5hhk</guid>
      <description>&lt;p&gt;A voice robot that confirms appointments can be perfectly accurate and still feel broken. The customer says "yes, I'm coming", then hears nothing for two seconds, says "hello?", and the robot starts answering on top of them. The transcript looks fine. The call does not.&lt;/p&gt;

&lt;p&gt;On the phone, latency is the product. People tolerate a short pause after they finish speaking, and they start filling the silence themselves somewhere around one second. So the useful question is not "which speech model is best" but "where do the milliseconds go between the end of the customer's phrase and the first sound of our reply". This post breaks that path into stages, gives each stage a budget, and lists the traps that quietly eat it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The metric that matters
&lt;/h2&gt;

&lt;p&gt;Measure one number per turn: &lt;strong&gt;time from end of user speech to first audio byte played back&lt;/strong&gt;. Call it turn latency. Not the ASR processing time, not the LLM time, not the TTS real-time factor. Those are components. The customer experiences only the sum, plus whatever the telephony network adds.&lt;/p&gt;

&lt;p&gt;A workable target for short outbound scenarios:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Turn latency (p50)&lt;/th&gt;
&lt;th&gt;How it feels&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;under 700 ms&lt;/td&gt;
&lt;td&gt;natural, like a quick human operator&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;700-1200 ms&lt;/td&gt;
&lt;td&gt;acceptable, slightly "robotic"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;over 1500 ms&lt;/td&gt;
&lt;td&gt;people talk over the robot or repeat themselves&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Track p90 as well. A robot with a 600 ms median and a 2.5 s p90 will produce a talk-over in roughly one turn out of ten, and in a four-turn confirmation call that means a large share of calls have at least one awkward moment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pipeline, stage by stage
&lt;/h2&gt;

&lt;p&gt;A typical outbound call path looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SIP trunk -&amp;gt; media server (RTP) -&amp;gt; VAD / endpointing -&amp;gt; streaming ASR
  -&amp;gt; dialog logic (rules or LLM) -&amp;gt; streaming TTS -&amp;gt; RTP back to caller
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here is a budget that fits under 800 ms at the median, with realistic ranges for each stage:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Budget&lt;/th&gt;
&lt;th&gt;Typical range&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Network + jitter buffer (both directions)&lt;/td&gt;
&lt;td&gt;120 ms&lt;/td&gt;
&lt;td&gt;80-250 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Endpointing (deciding the user stopped)&lt;/td&gt;
&lt;td&gt;300 ms&lt;/td&gt;
&lt;td&gt;200-800 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ASR final result after endpoint&lt;/td&gt;
&lt;td&gt;80 ms&lt;/td&gt;
&lt;td&gt;50-400 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dialog decision&lt;/td&gt;
&lt;td&gt;100 ms&lt;/td&gt;
&lt;td&gt;5 ms (rules) to 1500 ms (LLM)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TTS time to first audio chunk&lt;/td&gt;
&lt;td&gt;150 ms&lt;/td&gt;
&lt;td&gt;80-600 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playback start (buffering, codec)&lt;/td&gt;
&lt;td&gt;40 ms&lt;/td&gt;
&lt;td&gt;20-100 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~790 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The two biggest and most variable items are endpointing and the dialog decision. That is where most of the engineering goes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Endpointing: the silent budget killer
&lt;/h2&gt;

&lt;p&gt;Before you can answer, you need to decide that the person has finished. The naive approach is a fixed silence timeout: if the VAD reports no speech for N milliseconds, the turn is over.&lt;/p&gt;

&lt;p&gt;Set N to 1000 ms and every turn starts one full second in the hole before any model even runs. Set it to 200 ms and the robot will interrupt people who pause mid-sentence: "I'd like to... reschedule".&lt;/p&gt;

&lt;p&gt;Practical settings that work for short confirmation scripts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Adaptive silence threshold by expected answer.&lt;/strong&gt; After a yes/no question ("Will you come tomorrow at 15:00?") use 250-350 ms. After an open question ("Which day suits you?") use 600-800 ms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Early finalization on a confident short answer.&lt;/strong&gt; If streaming ASR already has a partial "yes", "да", "иә" with high confidence and 200 ms of silence follows, end the turn. Short confirmations are exactly where this pays off.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Minimum speech duration.&lt;/strong&gt; Ignore bursts shorter than ~150 ms. Coughs, a door, a car horn and line clicks otherwise trigger a turn.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simplified endpointing loop with a frame-level VAD:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;FRAME_MS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;endpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;asr_partial&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expect_short&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;silence_limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;expect_short&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;700&lt;/span&gt;
    &lt;span class="n"&gt;speech_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;silence_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;frame&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                      &lt;span class="c1"&gt;# 20 ms PCM frames, 8 kHz
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;vad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_speech&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frame&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;speech_ms&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;FRAME_MS&lt;/span&gt;
            &lt;span class="n"&gt;silence_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;silence_ms&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;FRAME_MS&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;speech_ms&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;150&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;                          &lt;span class="c1"&gt;# not real speech yet
&lt;/span&gt;        &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;asr_partial&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;expect_short&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_short_answer&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.85&lt;/span&gt; \
                &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;silence_ms&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;early&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;silence_ms&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;silence_limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;silence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The values are starting points. Tune them on recorded calls, not on your own voice in a quiet office.&lt;/p&gt;

&lt;h2&gt;
  
  
  ASR: streaming or nothing
&lt;/h2&gt;

&lt;p&gt;Batch recognition ("record the turn, then send the file") adds the full processing time after the endpoint. Streaming ASR processes audio as it arrives, so when the endpoint fires the hypothesis is already almost done and only finalization remains.&lt;/p&gt;

&lt;p&gt;Things to check when choosing or self-hosting an engine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Telephony audio is 8 kHz narrowband&lt;/strong&gt;, usually G.711. Many models are trained on 16 kHz. Upsampling does not restore lost frequencies; test accuracy on real call recordings, not on podcast clips.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Language switching.&lt;/strong&gt; In Kazakhstan a customer may answer a Russian script in Kazakh, or mix both in one sentence. Either run a model that handles both or detect the language on the first turn and keep the session in that language. Detection on every turn adds latency and flips on short words.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keyword boosting.&lt;/strong&gt; Confirmation calls revolve around a few dozen words: dates, weekdays, "reschedule", clinic or branch names. Phrase hints or a constrained grammar for yes/no questions cut both errors and finalization time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPU contention.&lt;/strong&gt; A self-hosted model that answers in 60 ms with one call can take 400 ms at 40 concurrent calls if you did not batch or size the hardware. Load test at your real peak concurrency.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The dialog decision: rules first, LLM where needed
&lt;/h2&gt;

&lt;p&gt;For a booking confirmation the set of meaningful answers is small: confirm, decline, reschedule, "who is this?", "call me later", and everything else. A classifier or a rule set over the transcript decides in single-digit milliseconds.&lt;/p&gt;

&lt;p&gt;An LLM is useful for the long tail: "I'm on the road, can my wife come instead?" But a full generation call easily costs 500-1500 ms to first token, which blows the whole budget. Patterns that keep it under control:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Intent first, generation second.&lt;/strong&gt; Run a fast intent classifier on every turn. Only route "other" to the LLM.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pre-written replies for known intents.&lt;/strong&gt; The response to "confirm" is fixed text, so its TTS audio can be cached in advance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hard timeout with a fallback.&lt;/strong&gt; If the LLM has not produced the first sentence in 900 ms, play a neutral filler and offer a transfer or callback.
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;decide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;intent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;            &lt;span class="c1"&gt;# ~5 ms
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;intent&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;SCRIPTED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;SCRIPTED&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;intent&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;intent&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;llm_reply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mf"&gt;0.9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;llm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;TimeoutError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Let me pass this to a colleague, they will call you back.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;escalate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not hide a slow LLM behind fillers like "hmm, let me think" on every turn. One filler is human. Three in a minute sound like a stalling machine.&lt;/p&gt;

&lt;h2&gt;
  
  
  TTS: time to first chunk, not total time
&lt;/h2&gt;

&lt;p&gt;A TTS engine that synthesizes a 3-second phrase in 300 ms looks fast, but if it returns audio only after the whole phrase is ready, you wait the full 300 ms. Streaming TTS that emits the first 100-200 ms of audio early lets playback start while the rest is still being generated.&lt;/p&gt;

&lt;p&gt;Cheap wins:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cache everything static.&lt;/strong&gt; Greetings, the confirmation reply, the goodbye, the "I didn't catch that" prompt. Synthesize once per voice and language, store as 8 kHz G.711 so no transcoding happens at call time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Template with slots.&lt;/strong&gt; "Your appointment is on {day} at {time}" can be split into a cached prefix plus a short synthesized slot. The prefix starts playing immediately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split long LLM replies by sentence.&lt;/strong&gt; Send the first sentence to TTS as soon as it is complete instead of waiting for the full answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match the output format.&lt;/strong&gt; Generating 24 kHz audio and resampling to 8 kHz on every chunk costs CPU and a few milliseconds per chunk. Ask the engine for telephony format directly if it supports it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Barge-in: when the customer talks over the robot
&lt;/h2&gt;

&lt;p&gt;Even with a good budget, people interrupt. "Your appointment at the clinic tomorrow at..." "Yes, yes, I know." The robot must stop speaking within roughly 200-300 ms, or it sounds like a recorded message.&lt;/p&gt;

&lt;p&gt;That requires:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Running VAD and ASR on the inbound stream &lt;strong&gt;while&lt;/strong&gt; TTS is playing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Echo handling.&lt;/strong&gt; On some lines the robot hears its own voice back. Without echo cancellation or a raised VAD threshold during playback, it interrupts itself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flushing the outbound buffer.&lt;/strong&gt; If you queued 3 seconds of audio into the media server, "stop" must clear that queue, not just stop sending new chunks.&lt;/li&gt;
&lt;li&gt;Deciding whether the interruption is meaningful. A short "mm-hm" during playback is acknowledgement, not a turn.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Network and telephony: the part you don't control
&lt;/h2&gt;

&lt;p&gt;The SIP trunk and mobile network add their own delay, often 100-250 ms round trip, plus the jitter buffer on your media server. Two settings matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Jitter buffer size.&lt;/strong&gt; A fixed 200 ms buffer adds 200 ms to every turn. An adaptive buffer starting at 40-60 ms is usually better for voice bots.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Media server location.&lt;/strong&gt; Host the media server and the speech models close to the trunk provider. Sending RTP across continents to a cloud ASR region adds a round trip every turn.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Measuring it in production
&lt;/h2&gt;

&lt;p&gt;Log timestamps per turn, in one clock domain, on the media server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"call_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"c-20261109-0412"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"turn"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"user_speech_end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.420&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"endpoint_fired"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.720&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"asr_final"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.790&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"decision_ready"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.796&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"tts_first_chunk"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.930&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"playback_start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;18.965&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"intent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"confirm"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"barge_in"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;user_speech_end&lt;/code&gt; comes from the VAD, so it is an estimate, but it is consistent, which is what you need for trends. From this you get the per-stage breakdown, p50 and p90 per stage, and a direct view of which stage regressed after a deploy or a model update.&lt;/p&gt;

&lt;p&gt;Also record the business outcome per call: confirmed, declined, rescheduled, no answer, escalated. A latency improvement that does not move the confirmation rate or reduce talk-overs probably was not the bottleneck.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls that show up only in real calls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Answering machines and voicemail.&lt;/strong&gt; A greeting recording will happily "answer" your question. Detect long uninterrupted speech in the first seconds and end the call or leave a short message.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Background noise.&lt;/strong&gt; Markets, cars, children. Raise VAD aggressiveness and rely more on confident ASR partials than on pure energy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The first turn.&lt;/strong&gt; People often say "hello" twice before listening. Start the script after the first short utterance, not after the pickup event.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time zones and calling hours.&lt;/strong&gt; If the customer base spans several countries or time zones, a reminder sent at 9:00 server time can land at 7:00 for some customers. Store the customer's time zone and schedule by their local time, with a calling window per scenario.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consent.&lt;/strong&gt; Only call people who left their number and expect the call. Beyond legal risk, cold lists destroy answer rates and get trunk numbers flagged as spam.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Turn latency measured as end of speech to first audio, p50 and p90&lt;/li&gt;
&lt;li&gt;Adaptive endpointing by expected answer type, early finalization on short confident answers&lt;/li&gt;
&lt;li&gt;Streaming ASR tested on 8 kHz call recordings in every language the script uses&lt;/li&gt;
&lt;li&gt;Intent classifier first, LLM only for the long tail, with a hard timeout&lt;/li&gt;
&lt;li&gt;Cached static prompts in telephony format, streaming TTS for the rest&lt;/li&gt;
&lt;li&gt;Barge-in with buffer flush and echo handling&lt;/li&gt;
&lt;li&gt;Per-turn timing logs tied to the call outcome&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Short, predictable scenarios like booking confirmations, visit reminders and one-question surveys are where these budgets are easiest to hit, because most replies can be scripted and cached. The usual path is to start with one scenario, listen to the test call recordings, tune pauses and wording, and only then scale the volume. If you need this for a clinic, salon or delivery service with scripts in Russian and Kazakh, here is how we set up an &lt;a href="https://zoomapps.kz/en/ai-voice-robot-for-calls" rel="noopener noreferrer"&gt;AI voice robot that handles confirmation and reminder calls&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>voip</category>
      <category>speechrecognition</category>
      <category>architecture</category>
      <category>ai</category>
    </item>
    <item>
      <title>Syncing 1C and Bitrix24 Without Duplicates: Designing an Idempotent Exchange</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Thu, 05 Nov 2026 06:07:14 +0000</pubDate>
      <link>https://dev.to/zoomapps/syncing-1c-and-bitrix24-without-duplicates-designing-an-idempotent-exchange-dhg</link>
      <guid>https://dev.to/zoomapps/syncing-1c-and-bitrix24-without-duplicates-designing-an-idempotent-exchange-dhg</guid>
      <description>&lt;p&gt;Most 1C and Bitrix24 integrations work fine in the demo. The first real problem shows up a month later: the same customer exists three times in the CRM, a payment was applied twice to one deal, and an invoice that was cancelled in accounting is still "awaiting payment" in the pipeline. Nobody wrote a bug on purpose. The exchange simply assumed that every message arrives exactly once, in order, and that both sides agree on what "the same customer" means.&lt;/p&gt;

&lt;p&gt;None of those assumptions hold. This post describes how we design the exchange so that repeating any message is harmless. That property, idempotency, is what separates an integration you can leave running from one you babysit.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure modes, concretely
&lt;/h2&gt;

&lt;p&gt;Before designing anything, it helps to list the ways duplicates appear. In our experience they come from five places:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Retry after a lost response.&lt;/strong&gt; Your connector calls &lt;code&gt;crm.company.add&lt;/code&gt;, Bitrix24 creates the company, the response is lost on the way back. The connector retries and creates a second one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overlapping runs.&lt;/strong&gt; The scheduled job runs every 5 minutes, one run takes 7 minutes on a busy day, and two runs process the same batch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Matching by name.&lt;/strong&gt; "Romashka LLP" in the CRM and "Romashka, LLP" in 1C look like different customers to any string comparison.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two creators.&lt;/strong&gt; A manager creates the company in Bitrix24 while the accountant creates the same counterparty in 1C. Both sync to the other side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Additive updates.&lt;/strong&gt; "Payment received: add 50,000 to the paid amount" is applied twice, and the deal shows 100,000 paid on a 50,000 invoice.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each needs a different fix, and together they define the design.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 1: one stable key per entity, stored on both sides
&lt;/h2&gt;

&lt;p&gt;Every synced object needs an identifier that never changes and is known to both systems. Names, phone numbers and invoice numbers are not that identifier. Numbers get renumbered at the start of the year, names get edited, phones change.&lt;/p&gt;

&lt;p&gt;1C already gives you a good one: every reference and document has a GUID (&lt;code&gt;Ссылка.УникальныйИдентификатор()&lt;/code&gt;). Bitrix24 gives you an integer &lt;code&gt;ID&lt;/code&gt; per entity type. Store both, in both places:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In Bitrix24, add a custom field to companies, smart invoices and products, for example &lt;code&gt;UF_CRM_1C_GUID&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In 1C, add an attribute or a separate information register &lt;code&gt;ВнешниеИдентификаторы&lt;/code&gt; with dimensions &lt;em&gt;object&lt;/em&gt; and &lt;em&gt;external system&lt;/em&gt;, and resource &lt;em&gt;external ID&lt;/em&gt;. A register is cleaner than adding attributes to every catalog, and it survives configuration updates better on standard configurations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The connector in the middle keeps its own mapping table. It is the source of truth for "which object over there corresponds to this one over here":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;entity_map&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;entity_type&lt;/span&gt;   &lt;span class="nb"&gt;text&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- 'company', 'invoice', 'product', 'payment'&lt;/span&gt;
    &lt;span class="n"&gt;onec_guid&lt;/span&gt;     &lt;span class="n"&gt;uuid&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;b24_id&lt;/span&gt;        &lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;onec_version&lt;/span&gt;  &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                           &lt;span class="c1"&gt;-- 1C ВерсияДанных at last sync&lt;/span&gt;
    &lt;span class="n"&gt;b24_hash&lt;/span&gt;      &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                           &lt;span class="c1"&gt;-- hash of fields we last wrote&lt;/span&gt;
    &lt;span class="n"&gt;updated_at&lt;/span&gt;    &lt;span class="n"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;onec_guid&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b24_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The two unique constraints matter more than they look. They make it impossible, at the database level, for one 1C object to point to two Bitrix24 records or the other way round.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 2: create is "find or create", never just "create"
&lt;/h2&gt;

&lt;p&gt;The classic retry duplicate (failure mode 1) happens because &lt;code&gt;add&lt;/code&gt; is not idempotent. Wrap every create in a lookup, and do the lookup and the create under a lock keyed by the stable identifier:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;upsert_company&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;onec&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;guid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;onec&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;guid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="c1"&gt;# Serialises concurrent workers on the same GUID only
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SELECT pg_advisory_xact_lock(hashtext(%s))&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;company:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;guid&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;

        &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SELECT b24_id, b24_hash FROM entity_map &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;WHERE entity_type=&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;company&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; AND onec_guid=%s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;guid&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;

        &lt;span class="n"&gt;fields&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;map_company_fields&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;onec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# includes UF_CRM_1C_GUID = guid
&lt;/span&gt;        &lt;span class="n"&gt;new_hash&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;stable_hash&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;b24_hash&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;new_hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;b24_id&lt;/span&gt;                      &lt;span class="c1"&gt;# nothing changed, no API call
&lt;/span&gt;
        &lt;span class="n"&gt;b24_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;b24_id&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nf"&gt;find_in_b24&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;onec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;b24_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;b24&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.company.update&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;b24_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fields&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;b24_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b24&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.company.add&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fields&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;

        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
            INSERT INTO entity_map (entity_type, onec_guid, b24_id, b24_hash)
            VALUES (&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;company&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;, %s, %s, %s)
            ON CONFLICT (entity_type, onec_guid)
            DO UPDATE SET b24_id=EXCLUDED.b24_id, b24_hash=EXCLUDED.b24_hash, updated_at=now()
        &lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;guid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b24_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new_hash&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;b24_id&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;find_in_b24&lt;/code&gt; is the safety net for the lost-response case: before creating, search Bitrix24 by &lt;code&gt;UF_CRM_1C_GUID&lt;/code&gt;. If the previous attempt did create the company, the search finds it and the retry becomes an update. Because we write the GUID into the company in the same &lt;code&gt;add&lt;/code&gt; call, there is no window where the record exists without its key.&lt;/p&gt;

&lt;p&gt;The advisory lock takes care of overlapping runs (failure mode 2) without a global "job is running" flag, which tends to get stuck after a crash. Two workers can process different companies in parallel; they only wait for each other on the same GUID.&lt;/p&gt;

&lt;p&gt;The hash check is a bonus that pays for itself quickly. Bitrix24 REST has request rate limits, and most scheduled syncs touch thousands of records that did not change. Skipping unchanged records usually removes the bulk of API calls after the initial load.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 3: match on legal identity, then clean up before go-live
&lt;/h2&gt;

&lt;p&gt;Failure mode 3 is not solvable by better string comparison. Normalising "LLP", commas and quotes helps you &lt;em&gt;find candidates&lt;/em&gt;, but the decision must rest on a legal identifier. In Kazakhstan that is the BIN for legal entities and the IIN for individual entrepreneurs. In 1C it sits on the counterparty card; in Bitrix24 it lives in the company's requisites (&lt;code&gt;crm.requisite.*&lt;/code&gt;, the BIN field in Kazakhstan presets), not in the company itself.&lt;/p&gt;

&lt;p&gt;The initial matching pass looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Export all counterparties from 1C with GUID, BIN/IIN and name.&lt;/li&gt;
&lt;li&gt;Export all Bitrix24 companies with their requisites.&lt;/li&gt;
&lt;li&gt;Join on BIN/IIN. Exact matches go straight into &lt;code&gt;entity_map&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Records without a BIN on one side go to a review list with fuzzy name candidates. A person decides.&lt;/li&gt;
&lt;li&gt;Duplicates &lt;em&gt;inside&lt;/em&gt; each system (two 1C counterparties with the same BIN) get merged in that system first.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 5 takes most of the project time. An exchange running on dirty reference books does not remove duplicates, it replicates them both ways. The same applies to items: if the product catalogue has "Cable 3x2.5" and "Cable 3x2,5 (old)" as separate items, decide which survives before prices start flowing into deals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 4: one master per field, not per system
&lt;/h2&gt;

&lt;p&gt;Failure mode 4, two creators, is a policy problem disguised as a technical one. The fix is to assign ownership per entity and per field, write it down and enforce it in the connector:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Data&lt;/th&gt;
&lt;th&gt;Master&lt;/th&gt;
&lt;th&gt;Other side&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Counterparty legal details (BIN, legal name, bank)&lt;/td&gt;
&lt;td&gt;1C&lt;/td&gt;
&lt;td&gt;read-only in CRM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contact persons, phones, responsible manager&lt;/td&gt;
&lt;td&gt;Bitrix24&lt;/td&gt;
&lt;td&gt;not synced back or informational only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Items and prices&lt;/td&gt;
&lt;td&gt;1C&lt;/td&gt;
&lt;td&gt;read-only in deals&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invoice&lt;/td&gt;
&lt;td&gt;created from the deal, posted in 1C&lt;/td&gt;
&lt;td&gt;status from 1C&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Payment&lt;/td&gt;
&lt;td&gt;1C&lt;/td&gt;
&lt;td&gt;read-only&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;When a manager needs a new customer, the CRM creates a company with a BIN, the connector checks 1C by BIN, and either links to the existing counterparty or creates a new one there. It never creates a second company in Bitrix24 because the counterparty "came back" from 1C: that GUID is already in &lt;code&gt;entity_map&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Make fields the CRM does not own read-only for managers, otherwise someone will "fix" a legal name and wonder why it reverts every night.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 5: send state, not deltas
&lt;/h2&gt;

&lt;p&gt;Failure mode 5 is the most expensive one, because it touches money. Never send "add 50,000". Send "invoice 9f3c... has payments totalling 50,000, status partially paid", computed from 1C's own records at the time of the message:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"message_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"b3c1e7a2-6f0d-4c8e-9a51-2d7e0f9c4a10"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entity"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"invoice"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"onec_guid"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"9f3c4d2e-1a7b-11ef-8c3a-005056a1b2c3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"onec_version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AAAAAQAAAAA="&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;120000.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"paid_total"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;50000.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"partially_paid"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"payments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"onec_guid"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"c41d..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-11-03"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;50000.00&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Applying this message twice gives the same result as applying it once. Applying an older message after a newer one is the remaining risk, and &lt;code&gt;onec_version&lt;/code&gt; handles it: 1C changes &lt;code&gt;ВерсияДанных&lt;/code&gt; on every write, and the connector stores the version it last applied. If the incoming version matches, skip. For ordering, keep a monotonic sequence on the sending side (the exchange message number, see below) and drop anything older than what you have already applied for that entity.&lt;/p&gt;

&lt;p&gt;Partial payments and refunds become trivial: the receiver never does arithmetic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Using 1C exchange plans as the outbox
&lt;/h2&gt;

&lt;p&gt;On the 1C side you do not need to invent change tracking. Exchange plans (&lt;code&gt;ПланыОбмена&lt;/code&gt;) already register changes per node and support an acknowledgement protocol based on message numbers. A minimal HTTP service handler that serves changes to the connector looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight bsl"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Returns changes registered for the Bitrix24 node&lt;/span&gt;
&lt;span class="k"&gt;Функция&lt;/span&gt; &lt;span class="nv"&gt;ПолучитьИзменения&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;Запрос&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;Узел&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;ПланыОбмена&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;ОбменБитрикс24&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;НайтиПоКоду&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;"B24"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;НомерПодтвержденного&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Число&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;Запрос&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;ПараметрыЗапроса&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;Получить&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;"ack"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

    &lt;span class="c1"&gt;// Connector confirmed everything up to ack: drop those registrations&lt;/span&gt;
    &lt;span class="k"&gt;Если&lt;/span&gt; &lt;span class="nv"&gt;НомерПодтвержденного&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;0&lt;/span&gt; &lt;span class="k"&gt;Тогда&lt;/span&gt;
        &lt;span class="nv"&gt;ПланыОбмена&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;УдалитьРегистрациюИзменений&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;Узел&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;НомерПодтвержденного&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;КонецЕсли&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="nv"&gt;НомерСообщения&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;Узел&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;НомерОтправленного&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nv"&gt;Выборка&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;ПланыОбмена&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;ВыбратьИзменения&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;Узел&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;НомерСообщения&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="c1"&gt;// ... serialise objects to JSON with GUID and ВерсияДанных ...&lt;/span&gt;

    &lt;span class="nv"&gt;ОбъектУзла&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;Узел&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;ПолучитьОбъект&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="nv"&gt;ОбъектУзла&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;НомерОтправленного&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;НомерСообщения&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nv"&gt;ОбъектУзла&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;Записать&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="c1"&gt;// return JSON with message_no = НомерСообщения&lt;/span&gt;
&lt;span class="k"&gt;КонецФункции&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key property: changes are removed from the registration only after the connector confirms it has applied them, by passing the message number back on the next request. If the connector crashes mid-batch, the next request has the old &lt;code&gt;ack&lt;/code&gt;, 1C returns the same changes again, and the idempotent upserts on the connector side make the replay harmless. At-least-once delivery plus idempotent receivers is the whole trick.&lt;/p&gt;

&lt;p&gt;For the opposite direction (deal created in Bitrix24, invoice to be issued in 1C), use the outbound event (&lt;code&gt;ONCRMDEALUPDATE&lt;/code&gt; and similar) only as a trigger. Bitrix24 events carry an ID, not the data, and can arrive twice or out of order. The handler should re-read the deal via REST and run the same find-or-create logic against 1C, with the deal ID stored on the 1C document.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deletions and cancellations
&lt;/h2&gt;

&lt;p&gt;Hard deletes break mapping tables. In 1C, documents are usually marked for deletion or unposted rather than deleted, and the exchange should treat those as status changes: an unposted invoice becomes "cancelled" in the CRM, not removed. If an object is actually deleted, the exchange plan still registers the deletion, and the connector should send a tombstone that sets the CRM side to an archived state and keeps the &lt;code&gt;entity_map&lt;/code&gt; row. Otherwise the next full resync will recreate the object from whichever side still has it.&lt;/p&gt;

&lt;p&gt;A note on real time: a schedule of a few minutes is enough for invoices and payments, easier to monitor and naturally batches requests under the REST limits. If you do add events, keep the scheduled run as a backstop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monitoring that catches silent failures
&lt;/h2&gt;

&lt;p&gt;An idempotent exchange can still fail quietly, by simply not running. The minimum we put in place:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Queue age alert.&lt;/strong&gt; If the oldest unacknowledged 1C change is older than, say, 30 minutes, someone gets a message.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Error log per entity&lt;/strong&gt;, with the 1C GUID and the Bitrix24 ID, so support can find the record in seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nightly reconciliation report.&lt;/strong&gt; Count invoices and paid totals for the last 30 days on both sides and compare. A mismatch of even one record is investigated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unmapped records list.&lt;/strong&gt; Companies without a BIN or without a mapping row are listed daily instead of piling up.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Stable GUID stored on both sides and in a mapping table with unique constraints&lt;/li&gt;
&lt;li&gt;Every create is find-or-create under a per-key lock&lt;/li&gt;
&lt;li&gt;Counterparties matched by BIN/IIN, reference books cleaned before launch&lt;/li&gt;
&lt;li&gt;One master per field, enforced in the connector and in CRM field permissions&lt;/li&gt;
&lt;li&gt;Messages carry full state and a version, never deltas&lt;/li&gt;
&lt;li&gt;1C exchange plans with acknowledged message numbers as the outbox&lt;/li&gt;
&lt;li&gt;Cancellations as status changes, deletions as tombstones&lt;/li&gt;
&lt;li&gt;Alerts on queue age and a nightly reconciliation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A realistic project of this kind, covering counterparties, invoices and payments, takes from about three weeks, and most of that time goes to matching and cleaning reference books rather than writing code. If you want a team that also works inside non-standard 1C configurations to design and run it, here is how we approach &lt;a href="https://zoomapps.kz/en/bitrix24-1c-integration" rel="noopener noreferrer"&gt;connecting Bitrix24 and 1C for invoices, payments and items&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>bitrix24</category>
      <category>integration</category>
      <category>database</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Building a WhatsApp AI Chatbot That Hands Off to a Human: Architecture and Pitfalls</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Mon, 02 Nov 2026 06:01:21 +0000</pubDate>
      <link>https://dev.to/zoomapps/building-a-whatsapp-ai-chatbot-that-hands-off-to-a-human-architecture-and-pitfalls-23le</link>
      <guid>https://dev.to/zoomapps/building-a-whatsapp-ai-chatbot-that-hands-off-to-a-human-architecture-and-pitfalls-23le</guid>
      <description>&lt;p&gt;Getting a language model to answer WhatsApp messages takes an afternoon. Getting it to stop answering at the right moment, pass the conversation to a person with enough context, and never talk over that person afterwards takes considerably longer. Most of the bugs we have seen in production chatbots are not about answer quality. They are about state: who owns the conversation right now, which messages have already been processed, and what happens when the phone, the model or the manager goes quiet.&lt;/p&gt;

&lt;p&gt;This post walks through an architecture that holds up under real traffic, with the parts that usually break.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shape of the system
&lt;/h2&gt;

&lt;p&gt;At a high level there are five components:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Transport.&lt;/strong&gt; Receives incoming messages and sends replies. This is either the official WhatsApp Business Platform with webhooks, or a session linked to an existing number through Linked Devices (the same mechanism WhatsApp on a computer uses).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Inbox queue.&lt;/strong&gt; Every incoming event goes into a durable queue before anything else touches it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Conversation store.&lt;/strong&gt; One record per chat: owner (bot or human), recent history, flags, timestamps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Assistant worker.&lt;/strong&gt; Pulls from the queue, builds the prompt from the knowledge base and the history, decides between "answer" and "hand off".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Operator side.&lt;/strong&gt; Wherever managers actually reply: the phone itself, a shared inbox, or a CRM.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The rule that saves you later: the transport layer does nothing except normalise events and enqueue them. No model calls inside a webhook handler. Webhook senders retry on slow responses, and an LLM call that takes eight seconds will happily produce duplicate replies.&lt;/p&gt;

&lt;h2&gt;
  
  
  Normalise and deduplicate first
&lt;/h2&gt;

&lt;p&gt;Whatever transport you use, convert events into one internal shape:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;InboundEvent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;message_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;      &lt;span class="c1"&gt;# provider's ID, used for dedup
&lt;/span&gt;    &lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;         &lt;span class="c1"&gt;# stable per customer
&lt;/span&gt;    &lt;span class="n"&gt;from_me&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;        &lt;span class="c1"&gt;# sent from the business number
&lt;/span&gt;    &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;            &lt;span class="c1"&gt;# "text", "voice", "image", "document"
&lt;/span&gt;    &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;media_ref&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;              &lt;span class="c1"&gt;# provider timestamp, seconds
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then deduplicate on &lt;code&gt;message_id&lt;/code&gt; before enqueueing. A Redis &lt;code&gt;SET NX&lt;/code&gt; with a TTL of a day is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;accept&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;InboundEvent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seen:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nx&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ex&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;86400&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;          &lt;span class="c1"&gt;# duplicate delivery, drop it
&lt;/span&gt;    &lt;span class="n"&gt;queue&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Webhook retries, reconnects of a linked session and your own worker restarts will all deliver the same message twice. Without dedup the customer gets two slightly different answers to one question.&lt;/p&gt;

&lt;p&gt;Note the &lt;code&gt;from_me&lt;/code&gt; field. On a linked number, messages the manager types on the phone arrive as events too. You need them, because they are the most reliable signal that a human has taken over (more on that below).&lt;/p&gt;

&lt;h2&gt;
  
  
  Debounce bursts before calling the model
&lt;/h2&gt;

&lt;p&gt;People do not write in single messages. They write "hi", then "how much is", then "delivery to Almaty?" within ten seconds. If the worker answers each fragment, you get three replies, the first two of them useless.&lt;/p&gt;

&lt;p&gt;A simple debounce: when a message arrives for a chat, schedule processing a few seconds later and reset the timer on every new message.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;DEBOUNCE_SEC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;on_event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append_pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;scheduler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reply:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;run_at&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;DEBOUNCE_SEC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;replace_existing&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the job fires, it takes all pending messages for that chat as one turn. Three to five seconds is the usual sweet spot: shorter and you still split sentences, longer and the bot feels sluggish. Voice notes need special handling here, because transcription adds its own delay; start the debounce timer from the moment the transcript is ready, not from the moment the audio arrived.&lt;/p&gt;

&lt;h2&gt;
  
  
  Voice notes are text with a confidence problem
&lt;/h2&gt;

&lt;p&gt;A large share of WhatsApp traffic is voice. The pipeline is straightforward: download the media, transcribe it, treat the transcript as a text message. The pitfall is noise. A voice note recorded on a busy street produces a transcript that is grammatically fine and semantically wrong, and the model will confidently answer the wrong question.&lt;/p&gt;

&lt;p&gt;Two mitigations that work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keep the per-segment confidence or average log-probability from the speech model and, below a threshold, have the assistant ask the customer to clarify instead of answering.&lt;/li&gt;
&lt;li&gt;Store the original audio reference with the transcript, so a manager who takes over can listen to it rather than trust the text.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conversation ownership is a state machine
&lt;/h2&gt;

&lt;p&gt;This is the core of the handoff design. Each chat has exactly one owner at a time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;            customer asks for human / trigger fires
   BOT  ───────────────────────────────────────────▶  PENDING_HUMAN
    ▲                                                     │
    │                                     manager replies │
    │                                                     ▼
    └──────────── idle timeout or manual release ─────  HUMAN
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BOT&lt;/strong&gt;: the assistant answers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PENDING_HUMAN&lt;/strong&gt;: the assistant has told the customer a person will reply, and stays silent. Managers are notified.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HUMAN&lt;/strong&gt;: a manager is active. The assistant is fully muted for this chat.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Store it explicitly and check it at the last possible moment, right before sending, not only when the job starts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;send_reply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generated_at_version&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;conv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;conv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BOT&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;  &lt;span class="c1"&gt;# a human took over while the model was thinking
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;conv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;generated_at_version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;  &lt;span class="c1"&gt;# new messages arrived, this answer is stale
&lt;/span&gt;    &lt;span class="n"&gt;transport&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The version check matters more than it looks. A model call can take several seconds; during that time the customer may add a message or a manager may jump in. Sending a reply generated from an outdated state is the single most common source of "the bot interrupted me" complaints from managers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What triggers a handoff
&lt;/h2&gt;

&lt;p&gt;Hard-coding "if the message contains 'operator'" catches only a fraction of cases. A better approach is to let the model return a structured decision alongside the answer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"action"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"handoff"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reason"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"custom_terms"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Wants 40 units with installation in Shymkent, asks for a discount on volume."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reply_to_customer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Thanks, I'm passing this to a manager who can confirm volume terms."&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Typical handoff reasons worth defining up front:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the customer explicitly asks for a person;&lt;/li&gt;
&lt;li&gt;the customer is ready to buy and the next step needs a human (payment, contract, scheduling);&lt;/li&gt;
&lt;li&gt;non-standard terms: discounts outside the price list, unusual quantities, custom deadlines;&lt;/li&gt;
&lt;li&gt;complaints and negative sentiment;&lt;/li&gt;
&lt;li&gt;the question is outside the knowledge base after one clarification attempt.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Make the list configurable per business rather than baked into the prompt. A clinic and a furniture shop draw the line in very different places.&lt;/p&gt;

&lt;p&gt;Equally important are hard boundaries that never depend on the model's judgement. The assistant should not promise discounts beyond the price list, quote deadlines that depend on workload, or give professional conclusions. Encode these both in the system prompt and as post-generation checks: a regex for percentages and currency amounts that do not appear in the knowledge base catches a surprising number of invented prices.&lt;/p&gt;

&lt;h2&gt;
  
  
  Detecting that a human has taken over
&lt;/h2&gt;

&lt;p&gt;If managers reply from a CRM or a shared inbox, you know when they send something. If they reply from the phone itself, which is common with small teams, you only know through &lt;code&gt;from_me&lt;/code&gt; events from the transport.&lt;/p&gt;

&lt;p&gt;The rule: any &lt;code&gt;from_me&lt;/code&gt; message that was not sent by the bot moves the chat to &lt;code&gt;HUMAN&lt;/code&gt;. To tell the bot's own messages apart, record the provider message IDs of everything the bot sends:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;on_from_me&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sismember&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bot_sent:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message_id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;                      &lt;span class="c1"&gt;# our own echo
&lt;/span&gt;    &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_owner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HUMAN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;since&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not rely on comparing message text; managers copy bot phrases more often than you would expect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Giving the manager context
&lt;/h2&gt;

&lt;p&gt;A handoff that dumps the customer on a person with no context forces them to ask everything again, which defeats the purpose. Send the manager:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the model's one-paragraph summary from the handoff decision;&lt;/li&gt;
&lt;li&gt;the reason code;&lt;/li&gt;
&lt;li&gt;the last N messages verbatim, including transcripts of voice notes;&lt;/li&gt;
&lt;li&gt;anything already collected: name, city, items and quantities mentioned.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The summary is generated once at handoff time and stored, so it stays consistent if several managers look at the chat.&lt;/p&gt;

&lt;h2&gt;
  
  
  Releasing the chat back to the bot
&lt;/h2&gt;

&lt;p&gt;The forgotten transition is HUMAN back to BOT. If you never release, the bot slowly goes silent across your whole customer base. If you release too eagerly, the bot answers in the middle of a human negotiation.&lt;/p&gt;

&lt;p&gt;What works in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An idle timeout measured from the last &lt;strong&gt;manager&lt;/strong&gt; message, not the last customer message. Something between 2 and 12 hours depending on the business.&lt;/li&gt;
&lt;li&gt;A manual release command from the operator side.&lt;/li&gt;
&lt;li&gt;On release, the bot does not proactively write anything. It simply answers the next incoming message, with the human part of the conversation included in its history so it does not contradict what the manager promised.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last point is easy to miss. If the manager agreed to a delivery date, the bot needs to see that agreement, or it will cheerfully quote the standard terms on the next question.&lt;/p&gt;

&lt;h2&gt;
  
  
  Knowledge base instead of a decision tree
&lt;/h2&gt;

&lt;p&gt;The reason menu-driven bots age badly is maintenance: every new service or price change means editing the tree. With an LLM, the price list and service descriptions become the knowledge base, and a new line in the price list is available immediately.&lt;/p&gt;

&lt;p&gt;For a typical small business catalogue (a few hundred items) you often do not need vector search at all. The whole price list fits in the context window, and putting it there directly removes a class of retrieval failures where the right item simply is not fetched. Move to retrieval when the catalogue grows past what fits comfortably, and even then keep exact-match lookup for SKUs and product names alongside embeddings.&lt;/p&gt;

&lt;p&gt;Feed corrections back from real conversations. Log every handoff with reason &lt;code&gt;outside_knowledge_base&lt;/code&gt;; reviewing that list weekly is the cheapest way to improve answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Operational pitfalls
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The linked phone goes offline.&lt;/strong&gt; With Linked Devices the link runs through the phone. If it loses connectivity, the session stops receiving messages. Monitor last-event time per number and alert when it exceeds a sensible window during working hours. When the link comes back, the replayed backlog hits your dedup layer, which is another reason that layer has to exist.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bulk messaging gets numbers banned.&lt;/strong&gt; Answering inbound messages on a business number is normal usage. Sending to cold lists is what triggers restrictions. Keep the bot strictly reactive.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Late answers.&lt;/strong&gt; If processing fails and recovers an hour later, do not answer an hour-old question as if nothing happened. Drop or rephrase replies to messages older than a cutoff, and route those chats to a human instead.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Several numbers.&lt;/strong&gt; Branches often have different prices. Key the conversation store and the knowledge base by number, and allow a shared base with per-number overrides.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Logging and privacy.&lt;/strong&gt; Decide retention up front and mask phone numbers in application logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist before going live
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Dedup on provider message ID before the queue&lt;/li&gt;
&lt;li&gt;Debounce of 3 to 5 seconds per chat, started after transcription for voice&lt;/li&gt;
&lt;li&gt;Explicit owner state per chat, checked again right before sending&lt;/li&gt;
&lt;li&gt;Version check to drop stale model answers&lt;/li&gt;
&lt;li&gt;Structured handoff decision with reason and summary&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;from_me&lt;/code&gt; detection with a record of bot-sent IDs&lt;/li&gt;
&lt;li&gt;Idle-based release measured from the last manager message&lt;/li&gt;
&lt;li&gt;Hard checks against invented prices and deadlines&lt;/li&gt;
&lt;li&gt;Alert on silent numbers, cutoff for late answers&lt;/li&gt;
&lt;li&gt;A sandbox chat where answers are reviewed before the first real customer&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these pieces is complicated on its own. The difficulty is that skipping any one of them produces failures that look like "the AI said something weird", when the real cause is a race between a model call and a manager's thumb. If you would rather start with a setup that already handles connection, voice notes and handoff, and test it on your own price list first, take a look at our &lt;a href="https://zoomapps.kz/en/whatsapp-ai-chatbot" rel="noopener noreferrer"&gt;WhatsApp AI chatbot that works on your existing number&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>whatsapp</category>
      <category>chatbot</category>
      <category>architecture</category>
      <category>ai</category>
    </item>
    <item>
      <title>Bitrix24 REST API in Practice: Webhooks, Batch Calls and Rate Limits</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Thu, 29 Oct 2026 06:02:09 +0000</pubDate>
      <link>https://dev.to/zoomapps/bitrix24-rest-api-in-practice-webhooks-batch-calls-and-rate-limits-49i0</link>
      <guid>https://dev.to/zoomapps/bitrix24-rest-api-in-practice-webhooks-batch-calls-and-rate-limits-49i0</guid>
      <description>&lt;p&gt;The Bitrix24 REST API looks simple on day one. You create an inbound webhook, call &lt;code&gt;crm.deal.list&lt;/code&gt;, get JSON back and the prototype works. The trouble starts on the first bulk sync, when a few thousand records need to move and the portal answers with &lt;code&gt;QUERY_LIMIT_EXCEEDED&lt;/code&gt;, or a week later, when someone notices that deals updated over the weekend never reached the warehouse system.&lt;/p&gt;

&lt;p&gt;This post collects what we have learned building exchanges between Bitrix24 and external systems: how the two kinds of webhooks behave, how to use &lt;code&gt;batch&lt;/code&gt; without shooting yourself in the foot, how the rate limits actually work, and the handful of field-level quirks that cause silent data corruption.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two kinds of webhooks, two different jobs
&lt;/h2&gt;

&lt;p&gt;Bitrix24 uses the word "webhook" for two unrelated things.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Inbound webhook.&lt;/strong&gt; A URL with a secret embedded in it that lets your code call REST methods on behalf of a specific user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://yourcompany.bitrix24.kz/rest/1/x7k2abc9def0ghij/crm.deal.get.json?id=42
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;1&lt;/code&gt; is the user ID, the next segment is the secret. Everything this URL does runs with that user's permissions and is limited to the scopes you ticked when creating it (&lt;code&gt;crm&lt;/code&gt;, &lt;code&gt;user&lt;/code&gt;, &lt;code&gt;task&lt;/code&gt; and so on). Two practical consequences:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Create a dedicated technical user for integrations. If the webhook belongs to a sales manager who later leaves the company and gets deactivated, the integration dies with their account.&lt;/li&gt;
&lt;li&gt;Grant the minimum scopes. The URL is a bearer credential. It ends up in logs, environment files and occasionally in screenshots sent to a chat.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Outbound webhook (event handler).&lt;/strong&gt; Bitrix24 sends a POST to your URL when something happens: &lt;code&gt;ONCRMDEALADD&lt;/code&gt;, &lt;code&gt;ONCRMDEALUPDATE&lt;/code&gt;, &lt;code&gt;ONCRMCONTACTDELETE&lt;/code&gt; and many more. The body is form-encoded, not JSON, and looks roughly like this once parsed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;event=ONCRMDEALUPDATE
data[FIELDS][ID]=1287
ts=1761712345
auth[domain]=yourcompany.bitrix24.kz
auth[application_token]=k4q9...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice what is missing: the deal itself. You get an ID and nothing else. Your handler has to call &lt;code&gt;crm.deal.get&lt;/code&gt; to find out what changed, which means every event costs at least one API request.&lt;/p&gt;

&lt;p&gt;A minimal handler in Python (Flask) that does the right things:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;flask&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;abort&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;queue_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;enqueue&lt;/span&gt;  &lt;span class="c1"&gt;# Redis, RabbitMQ, a DB table, anything durable
&lt;/span&gt;
&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Flask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;APP_TOKEN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B24_APP_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nd"&gt;@app.post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/b24/events&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;b24_event&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;form&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;form&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;form&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;auth[application_token]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;APP_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;abort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;form&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;entity_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;form&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data[FIELDS][ID]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;entity_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;abort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entity_id&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;form&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three rules are baked in here:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Check &lt;code&gt;application_token&lt;/code&gt;.&lt;/strong&gt; The handler URL is public; without the check anyone can push fake events.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not call the API inside the handler.&lt;/strong&gt; Acknowledge fast and queue the work. A slow or failing handler loses events, and Bitrix24 keeps no replay log for you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store only the ID.&lt;/strong&gt; By the time the worker runs, the deal may have changed again, and you want the latest state anyway.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Events are a hint, not a guarantee
&lt;/h3&gt;

&lt;p&gt;Treat outbound events as "something probably changed, go look". Deliveries get lost during your downtime, network blips or a deploy that broke the route. Two mitigations we use together:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Offline events.&lt;/strong&gt; For applications, Bitrix24 can store events on its side; you pull them with &lt;code&gt;event.offline.get&lt;/code&gt; and confirm processing. Push becomes pull and survives your downtime.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A periodic reconciliation pass.&lt;/strong&gt; Every N minutes, request entities with &lt;code&gt;DATE_MODIFY&lt;/code&gt; after your last checkpoint and run them through the same worker. Cheap, and it catches whatever the event stream dropped.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The echo loop
&lt;/h3&gt;

&lt;p&gt;If your integration writes back to the same entity it listens to, you will create a loop: you update deal 1287, Bitrix24 fires &lt;code&gt;ONCRMDEALUPDATE&lt;/code&gt;, your worker fetches it, sees a "change", pushes it to the external system, which pushes a status back, which updates the deal again.&lt;/p&gt;

&lt;p&gt;Break it explicitly. Options that work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Compare a hash of the fields you care about with the last hash you synced. If nothing relevant changed, stop.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;MODIFY_BY_ID&lt;/code&gt; (or &lt;code&gt;updatedBy&lt;/code&gt; in &lt;code&gt;crm.item.*&lt;/code&gt;) and ignore changes made by your technical integration user.&lt;/li&gt;
&lt;li&gt;Write a sync marker into a custom field, but only if the business is fine with an extra field on the card.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The first option is the most robust, because it also filters out updates that touched fields you do not sync.&lt;/p&gt;

&lt;h2&gt;
  
  
  batch: up to 50 calls in one request
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;batch&lt;/code&gt; takes a set of named commands and executes them on the server in one HTTP round trip. The limit is 50 commands per batch.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;WEBHOOK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://yourcompany.bitrix24.kz/rest/1/x7k2abc9def0ghij/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;WEBHOOK&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# read the body even on 503: the error code says which limit you hit
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;error_description&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;

&lt;span class="n"&gt;cmd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;d&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;deal_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.deal.get?id=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;deal_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;deal_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;103&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;batch&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;halt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cmd&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;        &lt;span class="c1"&gt;# successful commands
&lt;/span&gt;&lt;span class="n"&gt;errors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result_error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;   &lt;span class="c1"&gt;# failed commands, keyed the same way
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Things to know before relying on it:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check &lt;code&gt;result_error&lt;/code&gt; every time.&lt;/strong&gt; With &lt;code&gt;halt: 0&lt;/code&gt; the batch continues past failures, and the top level of the response looks successful. A batch of 50 updates where 7 failed returns HTTP 200. If you only read &lt;code&gt;result&lt;/code&gt;, those 7 are lost silently.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;halt: 1&lt;/code&gt; is not a transaction.&lt;/strong&gt; It stops at the first error, but the commands that already ran are not rolled back. There are no transactions in the REST API. Design updates so that re-running them is safe.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Commands are query strings.&lt;/strong&gt; Each value in &lt;code&gt;cmd&lt;/code&gt; is a URL-encoded string, not JSON. Hand-built strings break on &lt;code&gt;&amp;amp;&lt;/code&gt;, &lt;code&gt;=&lt;/code&gt; or Cyrillic. Use an encoder for PHP-style nested arrays (&lt;code&gt;fields[TITLE]=...&lt;/code&gt;, &lt;code&gt;filter[&amp;gt;ID]=...&lt;/code&gt;); plain &lt;code&gt;urlencode&lt;/code&gt; on a nested dict gets it wrong.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Results can feed later commands.&lt;/strong&gt; You can reference an earlier result inside the same batch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;cmd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;contact&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.contact.add?fields[NAME]=Aigerim&amp;amp;fields[PHONE][0][VALUE]=%2B77001234567&amp;amp;fields[PHONE][0][VALUE_TYPE]=WORK&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.deal.add?fields[TITLE]=Order%20123&amp;amp;fields[CONTACT_ID]=$result[contact]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Handy, but it makes the batch order-dependent and harder to retry. We use it for create-and-link pairs and avoid it everywhere else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A batch is one request for the rate limiter, but not free.&lt;/strong&gt; It counts as one hit against the request frequency limit, but every command inside still consumes server execution time, which is limited separately (see below). Fifty heavy &lt;code&gt;list&lt;/code&gt; calls with complex filters in one batch can exhaust the time budget faster than fifty simple &lt;code&gt;get&lt;/code&gt; calls spread over a minute.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the rate limits actually work
&lt;/h2&gt;

&lt;p&gt;There are two independent limits, and integrations usually discover them in this order.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Request frequency: a leaky bucket
&lt;/h3&gt;

&lt;p&gt;Each portal has a request counter that behaves like a leaky bucket. Requests fill it, and it drains at a fixed rate per second. While the bucket has room, requests go through instantly. When it is full, you get HTTP 503 with error &lt;code&gt;QUERY_LIMIT_EXCEEDED&lt;/code&gt;. The common published figures are a drain rate of 2 requests per second with a burst capacity of 50 on standard plans, and higher values on top plans and on-premise installations. Check the current numbers for your plan, they have changed over the years.&lt;/p&gt;

&lt;p&gt;A script firing 200 requests in a tight loop gets the first 50 through, then hits errors, and a naive retry loop only adds more requests. The fix is a client-side limiter that mirrors the server's bucket, plus backoff when the server disagrees:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threading&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LeakyBucket&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;capacity&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;capacity&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;threading&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;acquire&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lock&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
                &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
                    &lt;span class="k"&gt;return&lt;/span&gt;
                &lt;span class="n"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;
            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LeakyBucket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# stay below the real limit
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call_limited&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;RuntimeError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;QUERY_LIMIT_EXCEEDED&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;OPERATION_TIME_LIMIT&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
                &lt;span class="k"&gt;raise&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: rate limited after &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; attempts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter. The limiter runs below the real limit, because telephony, marketplace apps and other scripts share the same bucket. And it must be shared across workers: four processes with their own in-memory bucket send four times the allowed rate. Beyond one process, keep the counter in Redis.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Execution time per method
&lt;/h3&gt;

&lt;p&gt;Newer versions of the platform also limit how much server time a single method may consume over a sliding window. Responses carry a &lt;code&gt;time&lt;/code&gt; block, with fields such as &lt;code&gt;operating&lt;/code&gt; and &lt;code&gt;operating_reset_at&lt;/code&gt; telling you how much time the method has used. When a method exceeds its budget, it is blocked for a while with &lt;code&gt;OPERATION_TIME_LIMIT&lt;/code&gt;, while other methods keep working.&lt;/p&gt;

&lt;p&gt;This bites bulk exports. &lt;code&gt;crm.deal.list&lt;/code&gt; filtering on a custom field with a large &lt;code&gt;select&lt;/code&gt; is expensive, and a loop of them eats the budget. Log the &lt;code&gt;time&lt;/code&gt; block during test runs to see which calls are heavy before production does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pagination that does not get slower on every page
&lt;/h2&gt;

&lt;p&gt;List methods return 50 records per page with &lt;code&gt;next&lt;/code&gt; and &lt;code&gt;total&lt;/code&gt;. The obvious loop passes &lt;code&gt;start=0, 50, 100...&lt;/code&gt;. On large tables this degrades, because the server still counts and offsets through everything before your page, and it also computes &lt;code&gt;total&lt;/code&gt; on every call.&lt;/p&gt;

&lt;p&gt;The faster pattern: sort by &lt;code&gt;ID&lt;/code&gt;, filter by &lt;code&gt;&amp;gt;ID&lt;/code&gt; from the last record you saw, and pass &lt;code&gt;start=-1&lt;/code&gt; to turn off the count.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;iter_deals&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;last_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;call_limited&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crm.deal.list&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ASC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;filter&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;gt;ID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;last_id&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;select&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;
        &lt;span class="n"&gt;last_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each request is now an indexed range scan, and page 2,000 costs the same as page 1. On exports of hundreds of thousands of records this decides whether the job finishes overnight at all.&lt;/p&gt;

&lt;p&gt;Always pass &lt;code&gt;select&lt;/code&gt;. Without it you get the default field set, which excludes custom &lt;code&gt;UF_CRM_*&lt;/code&gt; fields, and you will wonder why they are empty. With &lt;code&gt;["*", "UF_*"]&lt;/code&gt; you get everything, which is expensive. List the fields you need.&lt;/p&gt;

&lt;h2&gt;
  
  
  Field-level pitfalls
&lt;/h2&gt;

&lt;p&gt;These do not throw errors. They corrupt data quietly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multi-fields append by default.&lt;/strong&gt; &lt;code&gt;PHONE&lt;/code&gt;, &lt;code&gt;EMAIL&lt;/code&gt;, &lt;code&gt;WEB&lt;/code&gt; and &lt;code&gt;IM&lt;/code&gt; are arrays of &lt;code&gt;{ID, VALUE, VALUE_TYPE}&lt;/code&gt;. If you call &lt;code&gt;crm.contact.update&lt;/code&gt; with &lt;code&gt;PHONE: [{"VALUE": "+77001234567"}]&lt;/code&gt;, the number is added, not replaced. Run that on every sync and a contact collects fifty copies of the same phone. To change a value, pass the existing item's &lt;code&gt;ID&lt;/code&gt;; to delete one, pass its &lt;code&gt;ID&lt;/code&gt; with an empty &lt;code&gt;VALUE&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two naming conventions.&lt;/strong&gt; &lt;code&gt;crm.deal.*&lt;/code&gt; uses &lt;code&gt;UPPER_SNAKE&lt;/code&gt; (&lt;code&gt;STAGE_ID&lt;/code&gt;, &lt;code&gt;ASSIGNED_BY_ID&lt;/code&gt;). The newer universal &lt;code&gt;crm.item.*&lt;/code&gt; methods with &lt;code&gt;entityTypeId&lt;/code&gt; (2 for deals, 3 for contacts, 4 for companies, plus IDs for smart processes) use &lt;code&gt;camelCase&lt;/code&gt; (&lt;code&gt;stageId&lt;/code&gt;, &lt;code&gt;assignedById&lt;/code&gt;) and wrap results in &lt;code&gt;item&lt;/code&gt; or &lt;code&gt;items&lt;/code&gt;. Mixing them in one codebase leads to fields that are "set" but never saved. Pick one family per entity and stick to it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage IDs depend on the pipeline.&lt;/strong&gt; In the default pipeline a stage is &lt;code&gt;NEW&lt;/code&gt; or &lt;code&gt;WON&lt;/code&gt;; in additional pipelines it becomes &lt;code&gt;C3:NEW&lt;/code&gt;, &lt;code&gt;C3:WON&lt;/code&gt;, where 3 is the category ID. Hardcoding &lt;code&gt;WON&lt;/code&gt; works until sales creates a second pipeline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dates come with a timezone offset.&lt;/strong&gt; Values look like &lt;code&gt;2026-10-29T11:00:00+05:00&lt;/code&gt;. Parse them as aware datetimes. Comparing them as strings or stripping the offset produces off-by-hours bugs in checkpoints, which means missed or duplicated records in reconciliation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;List fields store IDs, not labels.&lt;/strong&gt; A custom enum field returns &lt;code&gt;"45"&lt;/code&gt;, not &lt;code&gt;"Wholesale"&lt;/code&gt;. Load the field definitions with &lt;code&gt;crm.deal.userfield.list&lt;/code&gt; once and map both ways.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to log
&lt;/h2&gt;

&lt;p&gt;Log method, entity ID, direction, duration, the &lt;code&gt;time&lt;/code&gt; block, the error code and a correlation ID linking event, queue job and API call. Add one alert on the age of the oldest unprocessed job. Integrations rarely crash; they drift until the numbers stop matching a week later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Dedicated technical user, minimal webhook scopes&lt;/li&gt;
&lt;li&gt;Event handler: token check, enqueue, return 200&lt;/li&gt;
&lt;li&gt;Reconciliation pass by &lt;code&gt;DATE_MODIFY&lt;/code&gt; to catch lost events&lt;/li&gt;
&lt;li&gt;Echo-loop protection by field hash or author check&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;batch&lt;/code&gt; with &lt;code&gt;result_error&lt;/code&gt; checked on every call&lt;/li&gt;
&lt;li&gt;Shared client-side limiter below the published rate, backoff on 503&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;&amp;gt;ID&lt;/code&gt; pagination with &lt;code&gt;start=-1&lt;/code&gt; and explicit &lt;code&gt;select&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Multi-fields updated by &lt;code&gt;ID&lt;/code&gt;, stage IDs resolved per pipeline&lt;/li&gt;
&lt;li&gt;Logs with timing data and an alert on queue age&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The REST API itself is the same for cloud and on-premise portals; only authorisation and access setup differ, so these patterns carry over. If you would rather hand the exchange to a team that builds queues, retries and monitoring into every project, here is how we approach &lt;a href="https://zoomapps.kz/en/bitrix24-rest-api" rel="noopener noreferrer"&gt;Bitrix24 REST API integration with warehouse, logistics and banking systems&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>bitrix24</category>
      <category>api</category>
      <category>python</category>
      <category>integration</category>
    </item>
    <item>
      <title>Sizing VRAM for a Self-Hosted LLM: 7B to 70B on a Rented GPU Server</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Tue, 27 Oct 2026 18:29:47 +0000</pubDate>
      <link>https://dev.to/zoomapps/sizing-vram-for-a-self-hosted-llm-7b-to-70b-on-a-rented-gpu-server-m53</link>
      <guid>https://dev.to/zoomapps/sizing-vram-for-a-self-hosted-llm-7b-to-70b-on-a-rented-gpu-server-m53</guid>
      <description>&lt;p&gt;Most "will it fit" questions about self-hosted models get answered with a rule of thumb: two bytes per parameter, add some headroom, done. That rule tells you whether the weights load. It does not tell you whether the server survives fifty concurrent users with long prompts, which is the question that actually decides what card you rent.&lt;/p&gt;

&lt;p&gt;This post walks through the arithmetic I use before picking hardware for inference and fine-tuning, with a small Python calculator, a working vLLM launch config and the mistakes that cost the most time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four things that live in VRAM
&lt;/h2&gt;

&lt;p&gt;During inference, GPU memory is split between:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Model weights.&lt;/strong&gt; Fixed size, determined by parameter count and precision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;KV cache.&lt;/strong&gt; Grows with context length and with the number of sequences served in parallel. This is the part people forget.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Activations and temporary buffers.&lt;/strong&gt; Relatively small for inference, but spikes during prefill of long prompts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime overhead.&lt;/strong&gt; CUDA context, cuBLAS workspaces, NCCL buffers if you use several cards. Budget roughly 1 to 2 GB per GPU.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Training adds gradients and optimizer state on top, which changes the picture completely. We will get to that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Weights: the easy part
&lt;/h2&gt;

&lt;p&gt;Bytes per parameter by precision:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Precision&lt;/th&gt;
&lt;th&gt;Bytes/param&lt;/th&gt;
&lt;th&gt;7B&lt;/th&gt;
&lt;th&gt;13B&lt;/th&gt;
&lt;th&gt;34B&lt;/th&gt;
&lt;th&gt;70B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;FP16 / BF16&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;14 GB&lt;/td&gt;
&lt;td&gt;26 GB&lt;/td&gt;
&lt;td&gt;68 GB&lt;/td&gt;
&lt;td&gt;140 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FP8 / INT8&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;7 GB&lt;/td&gt;
&lt;td&gt;13 GB&lt;/td&gt;
&lt;td&gt;34 GB&lt;/td&gt;
&lt;td&gt;70 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4-bit (AWQ, GPTQ)&lt;/td&gt;
&lt;td&gt;~0.55&lt;/td&gt;
&lt;td&gt;~4 GB&lt;/td&gt;
&lt;td&gt;~7 GB&lt;/td&gt;
&lt;td&gt;~19 GB&lt;/td&gt;
&lt;td&gt;~39 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The 4-bit column is not exactly 0.5 because quantized formats store scales and zero points per group of weights, and some layers (embeddings, the output head) often stay in higher precision. Check the actual size of the safetensors files rather than trusting the multiplication.&lt;/p&gt;

&lt;p&gt;So a 70B model in BF16 does not fit on one 80 GB card, needs two of them with tensor parallelism, and fits on a single 141 GB card with room for cache. The same model in 4-bit fits on a 48 GB card, but with very little left over for users.&lt;/p&gt;

&lt;h2&gt;
  
  
  KV cache: the part that decides the card
&lt;/h2&gt;

&lt;p&gt;For every token in every active sequence, each transformer layer stores a key and a value vector. The formula:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;kv_bytes_per_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;num_layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;num_kv_heads&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bytes_per_element&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;2&lt;/code&gt; is key plus value. &lt;code&gt;num_kv_heads&lt;/code&gt; is the important term: models with grouped-query attention (GQA) share key/value heads across several query heads, which shrinks the cache dramatically.&lt;/p&gt;

&lt;p&gt;Real numbers, FP16 cache:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Llama 2 7B&lt;/strong&gt; (32 layers, 32 KV heads, head_dim 128, no GQA): 2 × 32 × 32 × 128 × 2 = &lt;strong&gt;512 KiB per token&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Llama 3 8B&lt;/strong&gt; (32 layers, 8 KV heads, head_dim 128): 2 × 32 × 8 × 128 × 2 = &lt;strong&gt;128 KiB per token&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Llama 3 70B&lt;/strong&gt; (80 layers, 8 KV heads, head_dim 128): 2 × 80 × 8 × 128 × 2 = &lt;strong&gt;320 KiB per token&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now multiply by load. Say you serve Llama 3 70B with an 8,192-token context and want 32 sequences in flight:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;320 KiB * 8192 tokens = 2.5 GiB per sequence
2.5 GiB * 32 sequences = 80 GiB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is 80 GiB of cache on top of 140 GB of weights. The "it fits on two 80 GB cards" plan turns into "it loads on two cards and serves about four users at full context". The model starts, the demo works, and throughput collapses under real traffic.&lt;/p&gt;

&lt;p&gt;In practice not every request uses the full context, and paged attention (vLLM, SGLang, TensorRT-LLM) allocates cache in blocks only for tokens that exist. So size against your real prompt length distribution, not the theoretical maximum. Log prompt plus completion lengths from a pilot and use the 90th percentile.&lt;/p&gt;

&lt;h2&gt;
  
  
  A calculator you can adapt
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;

&lt;span class="n"&gt;GiB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;params_b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;      &lt;span class="c1"&gt;# billions of parameters
&lt;/span&gt;    &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;inference_vram_gib&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;weight_bytes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_bytes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="n"&gt;ctx_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="n"&gt;overhead_gib&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;activation_frac&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;weights&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;params_b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;weight_bytes&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;GiB&lt;/span&gt;
    &lt;span class="n"&gt;kv_per_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kv_heads&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;head_dim&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv_bytes&lt;/span&gt;
    &lt;span class="n"&gt;kv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;kv_per_token&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ctx_tokens&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;GiB&lt;/span&gt;
    &lt;span class="n"&gt;activations&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weights&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;activation_frac&lt;/span&gt;
    &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;weights&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;activations&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;overhead_gib&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weights&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;llama3_8b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;llama3_70b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;70.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;inference_vram_gib&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llama3_8b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;    &lt;span class="c1"&gt;# BF16 weights, FP16 cache
&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;inference_vram_gib&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llama3_70b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;inference_vram_gib&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llama3_70b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.55&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c1"&gt;# 4-bit weights, FP8 cache
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(14.9, 16.0, 34.4)
(131.5, 80.0, 224.1)
(36.2, 40.0, 82.0)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the last line carefully. Quantizing weights to 4-bit and the cache to FP8 brings a 32-user 70B deployment from "three or four 80 GB cards" down to "one 80 GB card, slightly over budget". Drop concurrency to 24 or context to 6k and it fits. Whether the quality loss from 4-bit weights is acceptable is a separate question that you answer with your own eval set, not with a benchmark table.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;activation_frac&lt;/code&gt; of 5% is a rough allowance. Long-prompt prefill on large batches can need more; vLLM handles this by reserving memory during its profiling run at startup, which is why its numbers differ from naive math.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning the math into a vLLM config
&lt;/h2&gt;

&lt;p&gt;vLLM takes a fraction of GPU memory, loads weights, and gives everything left to the KV cache. The flags that matter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve meta-llama/Meta-Llama-3-70B-Instruct &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--tensor-parallel-size&lt;/span&gt; 2 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--gpu-memory-utilization&lt;/span&gt; 0.90 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--max-model-len&lt;/span&gt; 8192 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--max-num-seqs&lt;/span&gt; 32 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--kv-cache-dtype&lt;/span&gt; fp8 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--enable-prefix-caching&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;--gpu-memory-utilization 0.90&lt;/code&gt; leaves 10% for the CUDA context and fragmentation. Going to 0.95 works until a monitoring agent or a second process grabs memory, then you get OOM at 3 a.m.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--max-model-len&lt;/code&gt; caps context. If you leave it at the model's native 128k, vLLM may refuse to start because one maximum-length sequence does not fit in the cache, or it starts with very few concurrent slots.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--max-num-seqs&lt;/code&gt; limits how many sequences are scheduled at once. Too high and requests get preempted and recomputed when the cache fills; too low and the GPU idles.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--kv-cache-dtype fp8&lt;/code&gt; halves cache memory. Supported on Hopper and Ada cards; measure output quality before shipping it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--enable-prefix-caching&lt;/code&gt; reuses cache for shared prompt prefixes. If every request starts with the same 2,000-token system prompt, this is a large win.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At startup vLLM logs the number of GPU blocks it allocated for the cache and the maximum concurrency it can sustain at full context. Read that line before running a load test. It is the single best check of whether your sizing was right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fine-tuning needs a different budget
&lt;/h2&gt;

&lt;p&gt;Full fine-tuning with AdamW in mixed precision costs about 16 bytes per parameter before activations:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;BF16 weights: 2&lt;/li&gt;
&lt;li&gt;BF16 gradients: 2&lt;/li&gt;
&lt;li&gt;FP32 master weights: 4&lt;/li&gt;
&lt;li&gt;Adam first and second moments in FP32: 8&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a 7B model that is about 112 GB, so full fine-tuning even a small model needs several cards or ZeRO/FSDP sharding plus optimizer offload. Activations come on top and scale with batch size and sequence length; gradient checkpointing trades roughly 30% more compute for a large cut there.&lt;/p&gt;

&lt;p&gt;LoRA freezes the base weights and trains small adapter matrices, so optimizer state almost disappears. A 7B model with LoRA fits on a 24 to 48 GB card. QLoRA loads the base model in 4-bit and trains adapters on top: the original QLoRA work fine-tuned a 65B model on a single 48 GB GPU. For most business use cases (tone, format, domain vocabulary) LoRA on an 8B or 70B base is the sensible starting point.&lt;/p&gt;

&lt;h2&gt;
  
  
  Matching models to cards
&lt;/h2&gt;

&lt;p&gt;Rough mapping for inference with moderate concurrency, assuming you size the cache as above:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;7B to 13B, BF16:&lt;/strong&gt; a 48 GB card such as the L40S handles this comfortably with plenty of cache. Good fit for internal assistants and RAG.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;30B to 34B, BF16:&lt;/strong&gt; one 80 GB A100 or H100.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;70B, 4-bit:&lt;/strong&gt; one 80 GB card with limited concurrency, or a 48 GB card for a low-traffic internal tool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;70B, BF16 or FP8 with real traffic:&lt;/strong&gt; two 80 GB cards with tensor parallelism, or a single H200 with 141 GB, which avoids inter-GPU communication entirely.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Memory bandwidth matters as much as capacity. Token generation is bandwidth-bound: every new token reads all the weights once. A card with more HBM bandwidth generates tokens faster at the same model size, which is a big part of why H100 and H200 are worth the premium for latency-sensitive serving.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls that cost real time
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Multi-GPU without NVLink.&lt;/strong&gt; Tensor parallelism across PCIe-only cards works, but the all-reduce on every layer eats into throughput. Ask how the cards are connected before you plan a two-card split.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Driver and CUDA mismatch.&lt;/strong&gt; A wheel built for one CUDA version on a host driver that is too old gives cryptic errors. Pin your container base image and check &lt;code&gt;nvidia-smi&lt;/code&gt; for the driver's maximum supported CUDA version first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Disk and download time.&lt;/strong&gt; A 70B BF16 checkpoint is around 140 GB. On a fresh server, downloading it can take longer than the first experiment. Put model weights on local NVMe, not a network mount, or load times stretch into tens of minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Benchmarking with one user.&lt;/strong&gt; A single-stream test measures latency, not capacity. Use a load generator with your real prompt length distribution and watch time-to-first-token as concurrency grows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignoring data residency.&lt;/strong&gt; If the prompts contain customer records, where the GPU physically sits can be a legal constraint, not a preference. Check it before choosing a provider, not after the security review.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rent or buy
&lt;/h2&gt;

&lt;p&gt;The utilisation threshold is the deciding number. A card that is busy less than about half the time is usually cheaper to rent, because owned hardware depreciates while idle. Prototyping, evaluation of several model sizes and periodic fine-tuning runs almost always fall below that line. A model serving production traffic around the clock for several years can justify buying.&lt;/p&gt;

&lt;p&gt;The practical sequence: rent, measure real utilisation and real VRAM use for a month with the calculator above as your starting estimate, then decide. If you need hardware located inside Kazakhstan with CUDA, PyTorch and Docker already set up, billed by the hour for experiments or monthly for long runs, here is how our &lt;a href="https://zoomapps.kz/en/gpu-server-rental" rel="noopener noreferrer"&gt;GPU server rental with H100, H200, A100 and L40S cards&lt;/a&gt; works.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>gpu</category>
      <category>machinelearning</category>
      <category>devops</category>
    </item>
    <item>
      <title>Finding duplicate photos on iPhone without uploading a single one</title>
      <dc:creator>ZoomApps</dc:creator>
      <pubDate>Mon, 26 Oct 2026 17:22:05 +0000</pubDate>
      <link>https://dev.to/zoomapps/finding-duplicate-photos-on-iphone-without-uploading-a-single-one-1okg</link>
      <guid>https://dev.to/zoomapps/finding-duplicate-photos-on-iphone-without-uploading-a-single-one-1okg</guid>
      <description>&lt;p&gt;Most photo cleaners send your library somewhere or lock the useful part behind a weekly subscription. We wanted the opposite for our app, Fast Cleaner: everything runs on the iPhone, and the free tier covers a normal daily cleanup. Here is how the duplicate finder works and what we learned shipping it to 50 languages.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. One embedding per photo with Apple Vision
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;VNGenerateImageFeaturePrintRequest&lt;/code&gt; returns a feature vector for an image. Two photos of the same scene end up close to each other, even with small changes in framing or exposure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;request&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;VNGenerateImageFeaturePrintRequest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;handler&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;VNImageRequestHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;cgImage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;thumb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[:])&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perform&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;print&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;?&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt; &lt;span class="k"&gt;as?&lt;/span&gt; &lt;span class="kt"&gt;VNFeaturePrintObservation&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We request a 160×160 thumbnail from PhotoKit instead of the full image. Vision resamples to its own input size anyway, and the smaller request loads about 30% faster. Embeddings are cached on the device, so the second scan only processes new photos.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Compare only what can be similar
&lt;/h2&gt;

&lt;p&gt;Comparing every photo with every other one is O(n²). On a 50,000-photo library that is far too slow. We bucket photos by the month they were taken and compare pairs inside a bucket. Duplicates and burst-like series are almost always taken close in time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;Float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;computeDistance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;0.3&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;unionFind&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;union&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A distance threshold of 0.3 gave the best balance between "same moment" and "similar but different". Union-find turns the pairs into groups.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Pick the best shot, offer the rest
&lt;/h2&gt;

&lt;p&gt;Inside each group a simple weighted score picks the photo to keep:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;face capture quality (&lt;code&gt;VNDetectFaceCaptureQualityRequest&lt;/code&gt;): 0.35&lt;/li&gt;
&lt;li&gt;sharpness, Laplacian variance on luminance: 0.30&lt;/li&gt;
&lt;li&gt;resolution, log-scaled: 0.15&lt;/li&gt;
&lt;li&gt;open eyes and smile from face landmarks: 0.15&lt;/li&gt;
&lt;li&gt;newer date as a tiebreak: 0.05&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Photos without faces get a neutral face score, so landscapes are not penalised.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Things that bit us
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Memory.&lt;/strong&gt; Batches of 50 images inside &lt;code&gt;autoreleasepool&lt;/code&gt;, otherwise large libraries run out of memory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The simulator lies.&lt;/strong&gt; In the iOS simulator Vision returned the same vector for every image, so all distances were 0 and every photo landed in one group. We now detect "collapsed" embeddings (different sizes, distance exactly 0) and skip such buckets.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Main thread.&lt;/strong&gt; Reading the photo library during view rendering caused watchdog kills on big libraries. All PhotoKit reads moved off the main thread.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deletion safety.&lt;/strong&gt; Deletions go through PhotoKit, so everything lands in Recently Deleted for 30 days.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pricing without a weekly plan
&lt;/h2&gt;

&lt;p&gt;Scanning, the report, swipe-to-clean and up to 300 MB of cleanup a day are free. Pro removes the limit and adds video compression: $1.99/month (first month $0.99), $9.99/year or $19.99 once.&lt;/p&gt;

&lt;p&gt;Fast Cleaner is on the App Store: &lt;a href="https://apps.apple.com/app/id6772155675" rel="noopener noreferrer"&gt;https://apps.apple.com/app/id6772155675&lt;/a&gt; — more at &lt;a href="https://zoomapps.kz/fastcleaner/" rel="noopener noreferrer"&gt;https://zoomapps.kz/fastcleaner/&lt;/a&gt;&lt;/p&gt;

</description>
      <category>iosswiftprivamobilecy</category>
    </item>
  </channel>
</rss>
