<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Suryansh Chaudhary</title>
    <description>The latest articles on DEV Community by Suryansh Chaudhary (@dev_suryansh).</description>
    <link>https://dev.to/dev_suryansh</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1257065%2Fbc0b85f1-1f88-4ce0-8502-dc9b6a457ffe.png</url>
      <title>DEV Community: Suryansh Chaudhary</title>
      <link>https://dev.to/dev_suryansh</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dev_suryansh"/>
    <language>en</language>
    <item>
      <title>We're deleting our vector database and our graph database. Both of them.</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Tue, 18 Aug 2026 09:41:01 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/were-deleting-our-vector-database-and-our-graph-database-both-of-them-lgf</link>
      <guid>https://dev.to/dev_suryansh/were-deleting-our-vector-database-and-our-graph-database-both-of-them-lgf</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5j2engh3yangzhgpbqth.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5j2engh3yangzhgpbqth.png" alt=" " width="800" height="640"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not because they were bad. Because we counted, and we were running &lt;strong&gt;three retrieval&lt;br&gt;
substrates for one product&lt;/strong&gt;: a managed vector RAG service, a hosted graph database, and&lt;br&gt;
the application's own Postgres. Three vendors. Three sets of credentials. Three failure&lt;br&gt;
modes to reason about at 2am. Two of them billing us to embed &lt;em&gt;the same text twice&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;This is the write-up of why both are going, why one data warehouse is replacing both, and&lt;br&gt;
the four things that cost us the most time to discover. If you're standing up RAG on&lt;br&gt;
BigQuery, sections 4–7 are the ones that will save you a week.&lt;/p&gt;

&lt;p&gt;The honest status: the graph migration is built. The RAG half is a decision we've&lt;br&gt;
committed to &lt;em&gt;measuring&lt;/em&gt;, with criteria written down in advance. I'll show you those&lt;br&gt;
criteria at the end, including the outcome where we delete the whole thing instead.&lt;/p&gt;


&lt;h3&gt;
  
  
  1. Why we're leaving a managed RAG service
&lt;/h3&gt;

&lt;p&gt;The seductive thing about managed RAG is that chunking, embedding, indexing and retrieval&lt;br&gt;
arrive as one API call. The expensive thing is that when retrieval goes wrong, you own the&lt;br&gt;
symptom and not the machinery.&lt;/p&gt;

&lt;p&gt;Ours went wrong like this. A workspace with ~500 ingested documents was surfacing &lt;strong&gt;one to&lt;br&gt;
three of them&lt;/strong&gt;, over and over, no matter what was asked. Not low-quality answers —&lt;br&gt;
&lt;em&gt;starved&lt;/em&gt; answers, grounded in a handful of sources while 497 sat there unread.&lt;/p&gt;

&lt;p&gt;The root cause is a shape you should go check for in your own stack right now:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;One shared corpus for every tenant. Retrieve the global top-50 across all tenants.&lt;br&gt;
Then filter down to the current tenant, client-side.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's &lt;strong&gt;post-filtering&lt;/strong&gt;. The top-50 is a lottery drawn from everyone's data, and your&lt;br&gt;
tenant wins however many tickets it wins — which, on a shared corpus, is a handful.&lt;br&gt;
Ingesting more documents does not help. It makes it worse, because it adds tickets to a&lt;br&gt;
draw you're already losing.&lt;/p&gt;

&lt;p&gt;The obvious fix — server-side metadata filtering — was a closed door at our configuration:&lt;br&gt;
metadata search was only available in a beta API surface and explicitly unsupported in&lt;br&gt;
serverless mode. Not an oversight on our part; a documented limitation.&lt;/p&gt;

&lt;p&gt;The next fix — pass explicit document IDs to scope the search — collided with something&lt;br&gt;
even more mundane. The serverless vector backend caps the request payload at ~10 KB, and&lt;br&gt;
&lt;strong&gt;the query embedding alone was ~9.6 KB of it.&lt;/strong&gt; That left room for roughly fifteen IDs.&lt;br&gt;
Fifteen, out of five hundred. Which is how you end up maintaining ~150 lines of ID&lt;br&gt;
batching, adaptive request-splitting, and an all-or-nothing fallback that only fires at&lt;br&gt;
&lt;em&gt;exactly zero&lt;/em&gt; results — so "three documents out of five hundred" was being counted as&lt;br&gt;
success by our own code.&lt;/p&gt;

&lt;p&gt;Add a regional quota of about 25 requests per minute, shared across import, list and&lt;br&gt;
delete, and the picture resolves: we weren't tuning a retrieval system. We were building&lt;br&gt;
scaffolding around one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What we lose by leaving, stated plainly:&lt;/strong&gt; native reranking, corpus lifecycle&lt;br&gt;
management, and grounded generation as a single call. The first one is real and we have to&lt;br&gt;
rebuild it. More on that in section 8.&lt;/p&gt;


&lt;h3&gt;
  
  
  2. Why we're leaving the graph database
&lt;/h3&gt;

&lt;p&gt;This one is less dramatic and more useful, because &lt;strong&gt;the graph database was working&lt;br&gt;
fine.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There was no incident. No performance cliff. It was a paid third-party service living&lt;br&gt;
outside the cloud project everything else runs in, with its own credentials, its own&lt;br&gt;
uptime, its own invoice. Consolidation was the whole motivation, and I'd rather say that&lt;br&gt;
plainly than invent a technical grievance after the fact.&lt;/p&gt;

&lt;p&gt;But auditing it did surface something worth generalising. Here was our graph model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(:Resource)-[:HAS_CHUNK]-&amp;gt;(:Chunk {text, embedding})
(:Resource)-[:MENTIONS]-&amp;gt;(:Entity {name, type})
(:Entity)-[:RELATED {type}]-&amp;gt;(:Entity)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And here is what the read path actually did: vector-search the chunks, join to the&lt;br&gt;
resource, collect &lt;strong&gt;one level&lt;/strong&gt; of entity relationships. That's it.&lt;/p&gt;

&lt;p&gt;No variable-length patterns. No &lt;code&gt;*1..5&lt;/code&gt;. No path finding, no centrality, no traversal of&lt;br&gt;
unknown depth. &lt;strong&gt;A vector search and two joins.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So the question stopped being "how do we migrate our graph database" and became:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If none of your Cypher contains a variable-length pattern, what are you paying a graph&lt;br&gt;
database for?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A graph database earns its price on traversals whose depth you don't know in advance —&lt;br&gt;
that's the query class where the index-free adjacency actually beats a join. One hop is&lt;br&gt;
not that query class. One hop is a join, and every database does joins.&lt;/p&gt;

&lt;p&gt;If you have a knowledge graph in production, go grep your queries for &lt;code&gt;*&lt;/code&gt; between two&lt;br&gt;
node patterns. The answer is genuinely informative either way. If you find them, keep your&lt;br&gt;
graph database — this post doesn't apply to you. We didn't find any.&lt;/p&gt;


&lt;h3&gt;
  
  
  3. Why one substrate, and why a warehouse
&lt;/h3&gt;

&lt;p&gt;Once both moves are on the table, the third fact becomes impossible to ignore: &lt;strong&gt;we were&lt;br&gt;
running two complete chunk → embed → search stacks over identical text.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The ingestion service scraped a page, wrote it to object storage, and imported it into the&lt;br&gt;
managed corpus, which chunked and embedded it internally. Then the application read &lt;em&gt;the&lt;br&gt;
same text back out of object storage&lt;/em&gt;, chunked it again with its own splitter, and&lt;br&gt;
embedded it again. Same source. Two chunkings. Two embedding passes. Two bills. Two things&lt;br&gt;
that can drift apart silently and give different answers to the same question.&lt;/p&gt;

&lt;p&gt;Neither vendor was wrong. The &lt;em&gt;seam between them&lt;/em&gt; was.&lt;/p&gt;

&lt;p&gt;Collapsing both into one BigQuery dataset gives one text corpus, one chunking, one&lt;br&gt;
embedding model, one query surface — and, structurally, it can't starve:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- pre-filter, then top-k&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;VECTOR_SEARCH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;chunk_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;embedding&lt;/span&gt;
     &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;v_chunk&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;v_chunk_embedding&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;workspace_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="n"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;        &lt;span class="c1"&gt;-- &amp;lt;&amp;lt;&amp;lt; this runs FIRST&lt;/span&gt;
  &lt;span class="s1"&gt;'embedding'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="n"&gt;qvec&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="n"&gt;top_k&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="n"&gt;probe&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;distance_type&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'COSINE'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tenant predicate is inside the search subquery. The &lt;em&gt;k&lt;/em&gt; nearest are computed &lt;strong&gt;within&lt;br&gt;
the tenant&lt;/strong&gt;, not filtered down to it afterwards. The starvation failure mode from&lt;br&gt;
section 1 is not mitigated here; it is unrepresentable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The cost model that makes this viable:&lt;/strong&gt; in a warehouse, query cost &lt;em&gt;is&lt;/em&gt; bytes scanned.&lt;br&gt;
A 5,000-chunk workspace at 768 dimensions scans roughly 30 MB — about &lt;strong&gt;$0.0002 per&lt;br&gt;
query&lt;/strong&gt; — and brute-force vector search is competitive below a few thousand rows anyway.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;And the alternative we turned down,&lt;/strong&gt; because a decision post that only lists the&lt;br&gt;
winner's virtues isn't a decision post: Postgres + &lt;code&gt;pgvector&lt;/code&gt; was the better fit on the&lt;br&gt;
merits. The database was already deployed, already credentialed everywhere, &lt;code&gt;MERGE&lt;/code&gt; maps&lt;br&gt;
1:1 onto &lt;code&gt;ON CONFLICT&lt;/code&gt;, HNSW works at any row count, and latency stays in milliseconds&lt;br&gt;
instead of BigQuery's 0.5–2s floor. We went the other way for warehouse-side batch&lt;br&gt;
embedding and analytics colocation. It's a real trade, not a slam dunk, and if latency is&lt;br&gt;
your binding constraint you should probably pick differently.&lt;/p&gt;


&lt;h3&gt;
  
  
  4. Trap one: the embedding parameter that is accepted and ignored
&lt;/h3&gt;

&lt;p&gt;This is the most dangerous thing in the post, so it goes first.&lt;/p&gt;

&lt;p&gt;Newer embedding models have &lt;strong&gt;deprecated the &lt;code&gt;task_type&lt;/code&gt; parameter — and the backend&lt;br&gt;
silently ignores it.&lt;/strong&gt; You pass &lt;code&gt;RETRIEVAL_DOCUMENT&lt;/code&gt;. You get no error. You get no&lt;br&gt;
warning. You get raw-text embeddings with no task optimisation, and retrieval that is&lt;br&gt;
quietly worse than it should be with &lt;em&gt;nothing to debug&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Degraded quality with a clean log is the worst failure shape there is. A crash you can fix&lt;br&gt;
in an afternoon.&lt;/p&gt;

&lt;p&gt;The task now goes &lt;strong&gt;in the text&lt;/strong&gt;, as a prefix, and it stays deliberately asymmetric:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# one shared constant — two call sites that must never drift
&lt;/span&gt;&lt;span class="n"&gt;DOC_PREFIX&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;task: retrieval_document | &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;QUERY_PREFIX&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;task: retrieval_query | &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- ingest&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;AI&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GENERATE_EMBEDDING&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="nv"&gt;`ds.embed_model`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;chunk_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'task: retrieval_document | '&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;v_chunk&lt;/span&gt; &lt;span class="p"&gt;...),&lt;/span&gt;
  &lt;span class="n"&gt;STRUCT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;768&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;output_dimensionality&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two rules we now enforce with unit tests, because production won't tell you:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Ingest and query must use the &lt;strong&gt;same model at the same dimensionality&lt;/strong&gt;. Break it and
you don't get an exception, you get meaningless neighbours.&lt;/li&gt;
&lt;li&gt;Ingest and query must use &lt;strong&gt;different task prefixes&lt;/strong&gt;. That asymmetry is the entire
point of task types; symmetry silently costs you quality.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Test that no code path passes &lt;code&gt;TASK_TYPE&lt;/code&gt; at all. It's three lines and it protects you&lt;br&gt;
from a bug that has no other detector.&lt;/p&gt;


&lt;h3&gt;
  
  
  5. Trap two: the warehouse's model endpoint name is a different surface than the model
&lt;/h3&gt;

&lt;p&gt;The embedding model went GA on the ML platform in April. As of docs updated at the end of&lt;br&gt;
July, the warehouse's remote-model endpoint list &lt;strong&gt;still named it with a &lt;code&gt;-preview&lt;/code&gt;&lt;br&gt;
suffix.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;These are two independent surfaces and the warehouse's lags. A wrong endpoint string&lt;br&gt;
doesn't degrade — it fails outright at &lt;code&gt;CREATE MODEL&lt;/code&gt;, which is at least honest, but it&lt;br&gt;
will burn an afternoon if you're confident the model is GA and therefore assume the&lt;br&gt;
endpoint string matches.&lt;/p&gt;

&lt;p&gt;Do this before you build anything on top of it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="nv"&gt;`ds.embed_model`&lt;/span&gt;
  &lt;span class="n"&gt;REMOTE&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="k"&gt;CONNECTION&lt;/span&gt; &lt;span class="nv"&gt;`proj.region.conn`&lt;/span&gt;
  &lt;span class="k"&gt;OPTIONS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ENDPOINT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'model-name'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;          &lt;span class="c1"&gt;-- rejected? try 'model-name-preview'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then put whichever one worked in config and never infer it again. Related gotcha from the&lt;br&gt;
same family: a globally-available model can expand to a multi-region path that conflicts&lt;br&gt;
with your dataset's region. That reads like an IAM error and isn't one.&lt;/p&gt;


&lt;h3&gt;
  
  
  6. Trap three: mutating DML doesn't queue past its limit — it fails
&lt;/h3&gt;

&lt;p&gt;This is the most portable idea in the post, and it applies to anyone doing&lt;br&gt;
fire-and-forget background writes into a warehouse.&lt;/p&gt;

&lt;p&gt;Everyone knows the old per-day DML quota is gone. Here's what replaced it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Concurrency&lt;/th&gt;
&lt;th&gt;Queue&lt;/th&gt;
&lt;th&gt;Past the queue&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;UPDATE&lt;/code&gt; / &lt;code&gt;DELETE&lt;/code&gt; / &lt;code&gt;MERGE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;2 per table&lt;/td&gt;
&lt;td&gt;up to 20&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;jobs FAIL&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;INSERT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10 (after the first 1,500/table/24h run immediately)&lt;/td&gt;
&lt;td&gt;up to 100&lt;/td&gt;
&lt;td&gt;queued&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Our ingestion is background tasks fanned out across many tenants. A large batch plus a&lt;br&gt;
concurrent reconcile sweep can plausibly stack more than 20 &lt;code&gt;MERGE&lt;/code&gt;s on one table — and&lt;br&gt;
those don't wait their turn, they error. Recently-written rows aren't reliably mutable&lt;br&gt;
either.&lt;/p&gt;

&lt;p&gt;So we removed mutating DML entirely. &lt;strong&gt;Append-only writes, plus a dedup view:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="k"&gt;VIEW&lt;/span&gt; &lt;span class="n"&gt;v_chunk&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt;
&lt;span class="n"&gt;QUALIFY&lt;/span&gt; &lt;span class="n"&gt;ROW_NUMBER&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;PARTITION&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;chunk_id&lt;/span&gt; &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;ingested_at&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Re-ingesting appends a newer generation. The view returns only the newest. &lt;strong&gt;Every reader&lt;br&gt;
gets &lt;code&gt;MERGE&lt;/code&gt; semantics with zero mutating DML&lt;/strong&gt; — the upsert logic didn't move into the&lt;br&gt;
application, it moved into the read path, where the warehouse is happy to do it.&lt;/p&gt;

&lt;p&gt;Two things this pattern quietly buys you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Load jobs instead of DML.&lt;/strong&gt; They're free, they have no streaming-buffer semantics, and
their few-second latency is irrelevant on an async path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A free work queue.&lt;/strong&gt; Embeddings fill via &lt;code&gt;INSERT … SELECT&lt;/code&gt; over chunks that have no
embedding row yet. Rows whose embedding call failed are simply &lt;em&gt;absent&lt;/em&gt;, so the next
sweep retries them. The &lt;code&gt;NOT EXISTS&lt;/code&gt; join &lt;strong&gt;is&lt;/strong&gt; the retry bookkeeping. There is no
other retry bookkeeping.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One ordering rule, learned the hard way: load jobs are not transactional across tables, so&lt;br&gt;
&lt;strong&gt;write your commit-marker table last.&lt;/strong&gt; If the "this document is ingested" row lands&lt;br&gt;
first and a later load fails, that document is permanently marked done with zero chunks —&lt;br&gt;
never retried, never retrievable, and invisible to your gap metrics. Write it last and a&lt;br&gt;
partial failure leaves no marker, so the sweep retries and the system converges. Orphan&lt;br&gt;
rows from the failed attempt are harmless; the retry's newer generation supersedes them&lt;br&gt;
through the same view.&lt;/p&gt;




&lt;h3&gt;
  
  
  7. Trap four: your embedding dimension is a cost decision, not a quality decision
&lt;/h3&gt;

&lt;p&gt;The default output is 3072 dimensions. We use 768. That's not a quality compromise we&lt;br&gt;
grudgingly accepted — it's the single biggest cost lever in the design.&lt;/p&gt;

&lt;p&gt;In a warehouse, the vector dominates the row:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;3072 dims&lt;/strong&gt; → ~24 KB per chunk → a 5,000-chunk tenant scans &lt;strong&gt;~120 MB per query&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;768 dims&lt;/strong&gt; → ~6 KB per chunk → the same tenant scans &lt;strong&gt;~30 MB per query&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;4× on every single query, forever.&lt;/strong&gt; And modern embedding models are trained with&lt;br&gt;
Matryoshka Representation Learning, so truncated outputs are automatically normalised —&lt;br&gt;
768 is a &lt;em&gt;recommended&lt;/em&gt; size, not a hack. You trade a little quality for four times the&lt;br&gt;
throughput per dollar on the hot path.&lt;/p&gt;

&lt;p&gt;Two companions to it:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cluster on your tenant key.&lt;/strong&gt; Every read is tenant-scoped; without clustering, each&lt;br&gt;
query scans every vector in the table. This is the difference between scanning one&lt;br&gt;
tenant's data and scanning everyone's, and it's one line of DDL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Skip the vector index on purpose.&lt;/strong&gt; It sounds like the obvious optimisation and it's a&lt;br&gt;
trap here: a vector index requires a materialised table and forces you back into&lt;br&gt;
&lt;strong&gt;post-filtering by tenant&lt;/strong&gt; — which is exactly the starvation pattern from section 1,&lt;br&gt;
faithfully reimplemented in a new database. Brute force below a few thousand rows per&lt;br&gt;
tenant is fast, cheap and, more importantly, &lt;em&gt;correct by construction&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Note also: two independent embedding stacks don't need to agree with each other. Vectors&lt;br&gt;
from different stores are never compared as geometry — results are combined as text. The&lt;br&gt;
only invariant that matters is internal consistency within one stack.&lt;/p&gt;




&lt;h3&gt;
  
  
  8. What we haven't decided — and the bar we set in advance
&lt;/h3&gt;

&lt;p&gt;Here's where I stop claiming a win.&lt;/p&gt;

&lt;p&gt;Ending the duplicate stack has &lt;strong&gt;exactly two coherent end-states, and they're mutually&lt;br&gt;
exclusive:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Path A — keep the managed RAG service, share its embeddings.&lt;/strong&gt; Point the corpus at a&lt;br&gt;
feature-store backend that writes chunks and embeddings into a table you own. The service&lt;br&gt;
chunks and embeds once; the graph reads those rows and adds only the entity layer. Our own&lt;br&gt;
embedding fill disappears. Reranking, grounded generation and corpus lifecycle all survive&lt;br&gt;
untouched.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Path B — remove the managed service, warehouse only.&lt;/strong&gt; One stack, so no duplication by&lt;br&gt;
construction. The ingestion service drops its import step and gets strictly simpler.&lt;br&gt;
Corpus lifecycle code is &lt;em&gt;deleted&lt;/em&gt;, not replaced. Grounded generation becomes&lt;br&gt;
retrieve-then-generate at three call sites. And reranking has to be rebuilt — an LLM&lt;br&gt;
reranker over the top-N candidates, temperature 0, falling back to score order on any&lt;br&gt;
failure.&lt;/p&gt;

&lt;p&gt;They can't be combined, because the feature-store option &lt;strong&gt;is&lt;/strong&gt; the managed service.&lt;/p&gt;

&lt;p&gt;We wrote the decision criteria down &lt;em&gt;before&lt;/em&gt; collecting data, which is the only way this&lt;br&gt;
isn't rationalisation:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Why it decides the fork&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Graph-hit rate&lt;/td&gt;
&lt;td&gt;If retrieval rarely hits, Path B isn't viable at all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Results per hit&lt;/td&gt;
&lt;td&gt;The old path starves to 1–3. This is the axis pre-filtering should win on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Answer quality, new vs fallback&lt;/td&gt;
&lt;td&gt;The only thing that actually matters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval latency&lt;/td&gt;
&lt;td&gt;The warehouse has a 0.5–2s floor; the managed call is faster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bytes scanned per query&lt;/td&gt;
&lt;td&gt;Validates the clustering claim, which is currently asserted and unmeasured&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Path B is viable&lt;/strong&gt; if hit rate is high, results-per-hit meaningfully beats the old
path, and answer quality is at least equal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Path A wins&lt;/strong&gt; if hits are inconsistent or quality drops — because then reranking and
managed retrieval are carrying weight we didn't replace.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Neither&lt;/strong&gt; if the hit rate is low enough that the graph isn't earning its keep. In
which case the honest move is deleting the graph outright, which is strictly cheaper
than either migration.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That third option is the one I'd encourage you to keep on your own table. Every call site&lt;br&gt;
in our system already degrades gracefully to the fallback — which means "turn the flag off&lt;br&gt;
and delete 800 lines" was always available, and &lt;em&gt;measuring first&lt;/em&gt; would have been cheaper&lt;br&gt;
than either migration. We're doing the measurement after the build. Do it before.&lt;/p&gt;




&lt;h3&gt;
  
  
  The five-line version
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Post-filtering starves multi-tenant retrieval.&lt;/strong&gt; Filter &lt;em&gt;before&lt;/em&gt; top-k, not after.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One hop is a join.&lt;/strong&gt; Grep for variable-length patterns before you pay for a graph
database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deprecated parameters that are silently ignored&lt;/strong&gt; are worse than ones that error.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Append-only + a dedup view&lt;/strong&gt; gives you &lt;code&gt;MERGE&lt;/code&gt; semantics with zero mutating DML.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the decision criteria down before the data arrives.&lt;/strong&gt; Otherwise you'll find the
data agrees with whatever you already built.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;em&gt;If you've run RAG on a warehouse in production — especially the latency floor on&lt;br&gt;
interactive paths — I'd genuinely like to hear how it went.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>bigquery</category>
      <category>ai</category>
      <category>development</category>
      <category>rag</category>
    </item>
    <item>
      <title>I deleted the smartest feature in my download manager</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Thu, 30 Jul 2026 09:23:41 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/i-deleted-the-smartest-feature-in-my-download-manager-4m5g</link>
      <guid>https://dev.to/dev_suryansh/i-deleted-the-smartest-feature-in-my-download-manager-4m5g</guid>
      <description>&lt;p&gt;In 1.3, I deleted half of it. The half I was proudest of.&lt;/p&gt;

&lt;h2&gt;
  
  
  The feature
&lt;/h2&gt;

&lt;p&gt;Downloads opened at 4 connections. Every 3 seconds the engine added one more,&lt;br&gt;
measured aggregate throughput, and kept the new connection only if throughput&lt;br&gt;
improved by at least 15%. Otherwise it dropped back and stopped climbing.&lt;/p&gt;

&lt;p&gt;Read that back. It sounds like the responsible thing to do. It measures instead&lt;br&gt;
of assuming. It has a control loop. It even has a tunable threshold.&lt;/p&gt;
&lt;h2&gt;
  
  
  The bug
&lt;/h2&gt;

&lt;p&gt;"Throughput didn't improve by 15%" has two completely different causes, and the&lt;br&gt;
probe couldn't tell them apart:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The host is refusing to serve more connections. → Stop climbing. Correct.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your link is already saturated.&lt;/strong&gt; → Also stop climbing. Catastrophically
wrong.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;On a 100 Mbit connection pulling from a fast CDN, four connections will happily&lt;br&gt;
saturate the pipe. Connection five adds nothing measurable — not because the&lt;br&gt;
server refused it, but because there's no headroom left to measure it &lt;em&gt;with&lt;/em&gt;.&lt;br&gt;
The probe reads a flat line and concludes the ceiling is 4.&lt;/p&gt;

&lt;p&gt;Then the link frees up. Another download finishes, the VPN reconnects, whatever.&lt;br&gt;
Now there &lt;em&gt;is&lt;/em&gt; headroom — and the probe already stopped climbing. It settled low&lt;br&gt;
precisely in the case where extra connections were free.&lt;/p&gt;

&lt;p&gt;The heuristic's failure mode was invisible because it never errored. It just&lt;br&gt;
quietly returned a smaller number than it should have, forever, and every&lt;br&gt;
download looked "fine."&lt;/p&gt;
&lt;h2&gt;
  
  
  The fix: back off on evidence, not on absence of evidence
&lt;/h2&gt;

&lt;p&gt;A download now opens at its full effective thread count immediately, staggered&lt;br&gt;
100 ms apart so a burst of SYNs doesn't trip anti-abuse middleboxes. No ramp, no&lt;br&gt;
probe.&lt;/p&gt;

&lt;p&gt;Backing off is still there, but it now requires a &lt;em&gt;positive&lt;/em&gt; signal instead of a&lt;br&gt;
missing one. &lt;code&gt;DemotionPolicy&lt;/code&gt; halves the worker count when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;4 chunk attempts fail without progress&lt;/li&gt;
&lt;li&gt;inside a 10-second window&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;and bytes are still flowing&lt;/strong&gt; on the other connections&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last condition is the whole fix. It's what separates a hostile host —&lt;br&gt;
refusing some connections while happily serving the rest — from a dead local&lt;br&gt;
link, where everything fails at once because your Wi-Fi dropped.&lt;/p&gt;

&lt;p&gt;Without it, a 3-second network blip wrote a permanent per-host cap against a&lt;br&gt;
server that had done nothing wrong. Learned caps now also expire after 7 days and&lt;br&gt;
are clearable from Settings, so a cap mislearned during an outage heals itself.&lt;/p&gt;

&lt;p&gt;The lesson I keep re-learning: &lt;strong&gt;a control loop that treats "I measured nothing"&lt;br&gt;
as "there is nothing" will always fail toward doing less.&lt;/strong&gt; If your only feedback&lt;br&gt;
signal is absence, you don't have feedback.&lt;/p&gt;
&lt;h2&gt;
  
  
  Endgame chunk splitting
&lt;/h2&gt;

&lt;p&gt;Related tail problem. MacGet slices range-capable downloads into more pieces than&lt;br&gt;
workers (8 MB target, up to 256 pieces), so a finished worker steals the next&lt;br&gt;
outstanding piece instead of idling.&lt;/p&gt;

&lt;p&gt;That works right up until every piece is assigned. Then the freed workers have&lt;br&gt;
nothing to steal, and the download waits on whichever single piece landed on the&lt;br&gt;
slowest path.&lt;/p&gt;

&lt;p&gt;So now, when there's nothing left to steal, a freed worker &lt;strong&gt;splits the largest&lt;br&gt;
in-flight piece in half&lt;/strong&gt; and takes the back half — BitTorrent's endgame mode,&lt;br&gt;
basically, applied to HTTP ranges. Floored at 1 MB so you don't spend a fresh TCP&lt;br&gt;
handshake and an HTTP round trip on a 40 KB scrap.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SlotFiller.next(...) →
  .steal(piece)   // an outstanding piece exists
  .split(piece)   // nothing outstanding: halve the largest in-flight one
  .none           // everything is too small to bother splitting
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last slow chunk no longer sets the finish time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Book catalogs, and three APIs that didn't work
&lt;/h2&gt;

&lt;p&gt;1.3 also adds a catalog browser (⇧⌘B): search a book catalog, pick a format, it&lt;br&gt;
goes to the download queue like any other URL. The interesting part was that the&lt;br&gt;
obvious integration path was wrong for all three sources — and I only found out&lt;br&gt;
by hitting the live services.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Project Gutenberg.&lt;/strong&gt; It publishes OPDS, so: use OPDS, right? Its search feed&lt;br&gt;
returns &lt;em&gt;navigation&lt;/em&gt; entries — one sub-feed per result — so rendering a 50-book&lt;br&gt;
grid with download links would cost 50 extra requests. Those per-book feeds were&lt;br&gt;
also returning 504s, and the OPDS 2.0 endpoint is a 404. Gutendex returns every&lt;br&gt;
format's direct URL inline, so that's what the grid uses.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Internet Archive.&lt;/strong&gt; IA's OPDS BookServer doesn't just return errors —&lt;br&gt;
&lt;code&gt;bookserver.archive.org&lt;/code&gt; no longer resolves at all. Replaced with&lt;br&gt;
&lt;code&gt;advancedsearch.php?output=json&lt;/code&gt; for browse plus &lt;code&gt;metadata/&amp;lt;id&amp;gt;/files&lt;/code&gt; for the&lt;br&gt;
per-item file list, fetched lazily on selection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Standard Ebooks.&lt;/strong&gt; Every OPDS feed returns 401 to anonymous clients; access is a&lt;br&gt;
donor benefit. It ships as a built-in source that's disabled by default, which&lt;br&gt;
forced a small design point: &lt;code&gt;CatalogStore&lt;/code&gt; has to track both explicitly-enabled&lt;br&gt;
&lt;em&gt;and&lt;/em&gt; explicitly-disabled built-ins, so a shipped default can apply until the user&lt;br&gt;
actually expresses a preference.&lt;/p&gt;

&lt;p&gt;Everything still normalizes into one &lt;code&gt;CatalogFeed&lt;/code&gt; model, so the UI never branches&lt;br&gt;
on source. And &lt;code&gt;AcquisitionLink.isDownloadable&lt;/code&gt; is a single gate — direct-download&lt;br&gt;
rel, no price, http(s), known format. DRM fulfilment documents get parsed and&lt;br&gt;
displayed but never fetched, because that href is a license token, not a book.&lt;br&gt;
Archive.org lists &lt;code&gt;LCP Encrypted EPUB&lt;/code&gt; right next to the free files.&lt;/p&gt;

&lt;p&gt;One more: catalog requests use a &lt;em&gt;different&lt;/em&gt; &lt;code&gt;URLSession&lt;/code&gt; than downloads do. The&lt;br&gt;
download session sets &lt;code&gt;waitsForConnectivity = true&lt;/code&gt; and&lt;br&gt;
&lt;code&gt;timeoutIntervalForResource = .infinity&lt;/code&gt;, which is exactly right for a 6 GB file&lt;br&gt;
and exactly wrong for a metadata fetch behind a spinner.&lt;/p&gt;

&lt;h2&gt;
  
  
  BitTorrent
&lt;/h2&gt;

&lt;p&gt;Magnets and &lt;code&gt;.torrent&lt;/code&gt; files now run through the same queue as everything else —&lt;br&gt;
same concurrency limit, same pause/resume, same quiet hours. It's off by default&lt;br&gt;
and asks once before enabling, because unlike every other download type it&lt;br&gt;
&lt;em&gt;uploads&lt;/em&gt; on your connection and puts your IP in front of a swarm.&lt;/p&gt;

&lt;p&gt;Two things worth stealing if you ever wrap aria2:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MacGet owns the queue, aria2 doesn't.&lt;/strong&gt; &lt;code&gt;--save-session&lt;/code&gt; and &lt;code&gt;--input-file&lt;/code&gt; are&lt;br&gt;
deliberately absent. &lt;code&gt;queue.json&lt;/code&gt; is the source of truth and MacGet re-adds&lt;br&gt;
torrents itself on launch; if aria2 also restored its own session, every info hash&lt;br&gt;
would be registered twice and aria2 fails the duplicate outright. Resume comes&lt;br&gt;
from &lt;code&gt;--continue=true&lt;/code&gt; plus the &lt;code&gt;.aria2&lt;/code&gt; control file.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Magnets have a metadata phase that looks exactly like success.&lt;/strong&gt; The first GID&lt;br&gt;
downloads only the metainfo — a few hundred KB — and then reports &lt;code&gt;complete&lt;/code&gt; with&lt;br&gt;
&lt;code&gt;completedLength == totalLength&lt;/code&gt;. Indistinguishable from a finished download&lt;br&gt;
unless you're looking for it. An &lt;code&gt;isAwaitingMetadata&lt;/code&gt; flag suppresses completion&lt;br&gt;
and byte reporting until &lt;code&gt;followedBy&lt;/code&gt; hands off to the real GID. Get it wrong and&lt;br&gt;
your 6 GB torrent shows "Completed" at 500 KB.&lt;/p&gt;

&lt;p&gt;Also: bind ports as a &lt;em&gt;range&lt;/em&gt; (&lt;code&gt;6881-6890&lt;/code&gt;), not a single value. aria2 hard-fails&lt;br&gt;
with "Errors occurred while binding port" the moment another client holds it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Install
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;brew &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--cask&lt;/span&gt; suryansh-codes2209/macget/macget
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or grab the DMG: &lt;a href="https://macget.suryansh.work" rel="noopener noreferrer"&gt;https://macget.suryansh.work&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;MIT licensed, and the engine is genuinely fun to poke at — clone it, open&lt;br&gt;
&lt;code&gt;Macget.xcodeproj&lt;/code&gt;, ⌘R. No package manager step, no codegen:&lt;br&gt;
&lt;a href="https://github.com/Suryansh-Codes2209/Macget" rel="noopener noreferrer"&gt;https://github.com/Suryansh-Codes2209/Macget&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Issues and PRs welcome. I'm especially interested in ideas for what belongs in a&lt;br&gt;
download manager in 2026 — if you've got a feature you've always wanted and never&lt;br&gt;
found, open an issue.&lt;/p&gt;

&lt;p&gt;And if you've ever shipped a heuristic that looked smart and was quietly costing&lt;br&gt;
you something, I'd like to hear about it. I clearly can't be trusted to spot my&lt;br&gt;
own.&lt;/p&gt;

</description>
      <category>swift</category>
      <category>download</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Why more download threads make your downloads slower (and how I fixed it)</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Mon, 29 Jun 2026 07:07:16 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/why-more-download-threads-make-your-downloads-slower-and-how-i-fixed-it-1i37</link>
      <guid>https://dev.to/dev_suryansh/why-more-download-threads-make-your-downloads-slower-and-how-i-fixed-it-1i37</guid>
      <description>&lt;p&gt;There's a myth baked into almost every "download accelerator": open more connections, download faster. It's intuitive, and it's wrong for most of the modern web.&lt;/p&gt;

&lt;p&gt;I learned this the hard way building &lt;a href="https://github.com/Suryansh-Codes2209/Macget" rel="noopener noreferrer"&gt;MacGet&lt;/a&gt;, a free, open-source, native macOS download manager — and the fix turned into the most interesting part of the project.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: parallelism looks like an attack
&lt;/h2&gt;

&lt;p&gt;When you split a file into N chunks and open N HTTP-Range connections at once, a naive downloader assumes the server will happily serve all of them. Modern CDNs don't. To them, one IP suddenly opening 16 connections and pulling ranges looks exactly like leech/abuse behavior. So they fight back:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;TCP-RST&lt;/strong&gt; your connections after a few bytes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;throttle&lt;/strong&gt; your IP for a while&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;403&lt;/strong&gt; new requests outright&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The result is counterintuitive: crank the thread count up and your download gets &lt;em&gt;slower&lt;/em&gt;, or fails entirely. More threads ≠ more speed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: discover each host's real capacity at runtime
&lt;/h2&gt;

&lt;p&gt;Instead of trusting a fixed thread count, MacGet's engine treats parallelism as something to &lt;em&gt;discover&lt;/em&gt; per host:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Adaptive up-scaling.&lt;/strong&gt; Downloads start at 4 connections and probe upward one at a time, keeping each added connection only when aggregate throughput improves by ≥15%. So it climbs toward the host's real ceiling instead of guessing.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Demotion on rejection.&lt;/strong&gt; When ≥4 chunk attempts fail without making progress inside a 10-second window the signal that the host is rejecting parallelism the engine halves its worker count, cancels the lowest-progress chunks, and carries on. It repeats until it stabilizes at a level the host actually allows.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Per-host memory.&lt;/strong&gt; That learned cap is persisted (&lt;code&gt;host_caps.json&lt;/code&gt;). The next download from the same host starts at the right level — no rediscovery tax. Caps only ratchet downward.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Staggered spawns.&lt;/strong&gt; Workers start ~100ms apart, so anti-abuse middleboxes see a steady ramp instead of a SYN burst from one IP.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Smart retry classification.&lt;/strong&gt; Permanent failures (401/403/404/410/451, range refusals, malformed responses) fail fast. Transient ones (mid-stream RSTs, server-side stream kills, 5xx) retry with full-jitter exponential backoff under a hard cap — and 429/503 honor the server's &lt;code&gt;Retry-After&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There's also a macOS-specific gotcha: &lt;strong&gt;App Nap&lt;/strong&gt;. If you switch apps, macOS will happily throttle your "background" download into the ground. MacGet holds a &lt;code&gt;ProcessInfo&lt;/code&gt; activity assertion while any download runs, so the OS leaves it alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Killing the slow-chunk tail
&lt;/h2&gt;

&lt;p&gt;Even when a host allows N connections, fixed N-way chunking has a long tail: split a file into N equal pieces and the whole download waits on whichever piece landed on the slowest path. MacGet slices range-capable downloads into &lt;strong&gt;more pieces than workers&lt;/strong&gt; (8 MB target), and a finished worker immediately &lt;em&gt;steals&lt;/em&gt; the next&lt;br&gt;
outstanding piece. No worker sits idle while another grinds through the tail.&lt;/p&gt;
&lt;h2&gt;
  
  
  How it's built
&lt;/h2&gt;

&lt;p&gt;The engine is modeled with Swift's actor concurrency, which made the&lt;br&gt;
correctness story dramatically easier than locks-and-queues would have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DownloadEngine (actor)
  └─ DownloadCoordinator (actor)   // one per download: probe → plan → stream → finalize
       ├─ ChunkWorker              // one HTTP-Range request, streamed via AsyncThrowingStream
       └─ FileWriter (actor)       // serializes positional writes so chunks don't race
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few more design notes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Probe first.&lt;/strong&gt; A HEAD request (falling back to &lt;code&gt;GET Range: bytes=0-0&lt;/code&gt;, since some servers 405 on HEAD) establishes size, &lt;code&gt;Accept-Ranges&lt;/code&gt;, &lt;code&gt;ETag&lt;/code&gt;, and &lt;code&gt;Last-Modified&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;HTTP/3 when offered.&lt;/strong&gt; Requests opt into QUIC via &lt;code&gt;assumesHTTP3Capable&lt;/code&gt;, with graceful fallback to HTTP/2 → HTTP/1.1.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Integrity at finalize.&lt;/strong&gt; SHA-256 / MD5 is verified before the partial is promoted to the final filename; a mismatch fails the download and keeps the partial instead of handing you a corrupt file.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Sparse partials.&lt;/strong&gt; The partial file is &lt;code&gt;truncate&lt;/code&gt;d to full size up front; APFS keeps it sparse until bytes actually land.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Resumable across restarts.&lt;/strong&gt; Per-chunk byte offsets are persisted, and workers send the recorded &lt;code&gt;ETag&lt;/code&gt;/&lt;code&gt;Last-Modified&lt;/code&gt; as &lt;code&gt;If-Range&lt;/code&gt; so a file that changed server-side fails fast instead of silently corrupting your partial.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Survives a dropped connection.&lt;/strong&gt; An &lt;code&gt;NWPathMonitor&lt;/code&gt; pauses active downloads on network loss ("Waiting for network…") and auto-resumes when it's back, instead of burning retries through an outage.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Live speed/ETA.&lt;/strong&gt; A 3-second rolling window over &lt;code&gt;(time, bytes)&lt;/code&gt; samples; ETA goes &lt;code&gt;nil&lt;/code&gt; below 1 KB/s instead of lying to you.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The whole thing is SwiftUI on top, with &lt;code&gt;@Observable&lt;/code&gt; view models draining an &lt;code&gt;AsyncStream&lt;/code&gt; of engine events — the UI never mutates download state directly, only through the engine actor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Beyond the engine
&lt;/h2&gt;

&lt;p&gt;The latest release (1.2) builds a real download manager around that core:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Video &amp;amp; audio downloads.&lt;/strong&gt; A conservative host classifier routes known video/audio sites to a bundled yt-dlp + ffmpeg extractor (with a quality/format picker); ordinary links still download as plain HTTP through the engine above.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Authenticated downloads.&lt;/strong&gt; A Keychain-backed credential store answers Basic/Digest/NTLM challenges and remembers them per host across launches.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A browser extension&lt;/strong&gt; (Chrome/Edge/Brave/Firefox) that hands off downloads you start in the browser — carrying the cookies, referrer, and user-agent — so logged-in downloads actually work.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Auto-sort&lt;/strong&gt; finished files into category folders, &lt;strong&gt;High/Normal/Low priorities&lt;/strong&gt;, &lt;strong&gt;bandwidth throttling&lt;/strong&gt;, and &lt;strong&gt;signed auto-updates&lt;/strong&gt; via Sparkle (EdDSA-verified — a tampered update is rejected).&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Just want to use it?&lt;/strong&gt; Download the DMG: &lt;a href="https://suryansh.work/macget" rel="noopener noreferrer"&gt;https://suryansh.work/macget&lt;/a&gt; (free, un-notarized — right-click → Open on first launch for the one-time Gatekeeper step).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Want to read or hack the code?&lt;/strong&gt; It's MIT licensed and the engine is genuinely fun to poke at: &lt;a href="https://github.com/Suryansh-Codes2209/Macget" rel="noopener noreferrer"&gt;https://github.com/Suryansh-Codes2209/Macget&lt;/a&gt; — clone, open &lt;code&gt;Macget.xcodeproj&lt;/code&gt;, ⌘R. No setup.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>macos</category>
      <category>swift</category>
      <category>opensource</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Unleashing the Power of Django on AWS: Best Practices for Seamless Deployment!</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Tue, 30 Jan 2024 04:48:26 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/unleashing-the-power-of-django-on-aws-best-practices-for-seamless-deployment-53n</link>
      <guid>https://dev.to/dev_suryansh/unleashing-the-power-of-django-on-aws-best-practices-for-seamless-deployment-53n</guid>
      <description>&lt;p&gt;&lt;strong&gt;Hey Dev Community! 💻 Excited to share some golden nuggets on deploying Django web apps on AWS. Let's make your deployment journey smoother with these best practices:&lt;br&gt;
**&lt;br&gt;
**AWS App Runner Magic:&lt;/strong&gt; Dive into the simplicity of deploying and scaling Django apps on AWS App Runner. Check out this AWS blog for a quick guide.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Elastic Beanstalk Brilliance:&lt;/strong&gt; Discover the art of deploying Django apps on Elastic Beanstalk with AWS's official guide here. It's like a magic wand for streamlined deployment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Kubernetes Mastery:&lt;/strong&gt; Elevate your Django deployment game by containerizing with Kubernetes. Explore the scalability and security in this DigitalOcean tutorial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;PostgreSQL Power:&lt;/strong&gt; Level up your Django app by integrating PostgreSQL on AWS Elastic Beanstalk. Learn the ropes in this insightful Real Python tutorial.&lt;/p&gt;

&lt;p&gt;Remember, the key to a successful deployment is a mix of AWS services tailored to your app's needs. Keep it efficient, keep it scalable! 🔗✨&lt;/p&gt;

&lt;h1&gt;
  
  
  Django #AWS #DevOps #DeploymentMastery
&lt;/h1&gt;

</description>
      <category>webdev</category>
      <category>cloud</category>
      <category>aws</category>
      <category>django</category>
    </item>
    <item>
      <title>AWS Elastic Beanstalk: ⚡Simplifying Web Application Deployment and Scaling</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Mon, 15 Jan 2024 11:34:02 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/aws-elastic-beanstalk-simplifying-web-application-deployment-and-scaling-4jfm</link>
      <guid>https://dev.to/dev_suryansh/aws-elastic-beanstalk-simplifying-web-application-deployment-and-scaling-4jfm</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F67lh8abvb0gthwhrcjkl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F67lh8abvb0gthwhrcjkl.png" alt=" " width="800" height="800"&gt;&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;AWS Elastic Beanstalk is a fully managed service that streamlines the deployment and scaling of web applications.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Easy Deployment:&lt;/strong&gt; Simply upload your code, and Elastic Beanstalk takes care of the deployment process. It eliminates the need for manual intervention in tasks like capacity provisioning and configuration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Automatic Scaling:&lt;/strong&gt; Elastic Beanstalk dynamically scales your application based on demand. It leverages Auto Scaling, ensuring that your application can handle varying traffic loads efficiently.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Load Balancing:&lt;/strong&gt; The service includes Elastic Load Balancing, distributing incoming traffic across multiple instances to enhance application availability and fault tolerance.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Health Monitoring:&lt;/strong&gt; Elastic Beanstalk continuously monitors the health of your application. It automatically replaces unhealthy instances, ensuring high availability and reliability.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Traffic-Splitting Deployments:&lt;/strong&gt; With features like traffic-splitting deployments, Elastic Beanstalk minimizes downtime by creating a new set of instances for updates while preserving the existing environment.&lt;/p&gt;

&lt;p&gt;In summary, AWS Elastic Beanstalk abstracts the complexities of infrastructure management, allowing developers to focus on building robust applications without worrying about the intricacies of deployment, scaling, and monitoring.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>webdev</category>
      <category>devops</category>
      <category>student</category>
    </item>
    <item>
      <title>Launching 🚀Your AWS Application Journey with EC2: A Student's Guide⚡</title>
      <dc:creator>Suryansh Chaudhary</dc:creator>
      <pubDate>Mon, 15 Jan 2024 11:13:24 +0000</pubDate>
      <link>https://dev.to/dev_suryansh/launching-your-aws-application-journey-with-ec2-a-students-guide-35o</link>
      <guid>https://dev.to/dev_suryansh/launching-your-aws-application-journey-with-ec2-a-students-guide-35o</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1hs4zp0ui2hh5kcg5i5m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1hs4zp0ui2hh5kcg5i5m.png" alt=" " width="800" height="800"&gt;&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Are you a student eager to embark on your AWS application journey? Start by mastering Amazon EC2, a fundamental service that provides virtual machines in the cloud. Here's a step-by-step guide to kickstart your production-making adventure:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Set Up Your AWS Account&lt;/strong&gt;:&lt;br&gt;
Begin by creating an AWS account. Follow the official guide to ensure a seamless setup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Launch Your First EC2 Instance:&lt;/strong&gt;&lt;br&gt;
Learn the basics of launching an EC2 instance with this tutorial. Choose an Amazon Machine Image (AMI), select an instance type, and configure your settings.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Understand Virtual Machines:&lt;/strong&gt;&lt;br&gt;
Delve into the world of virtualization. Understand how EC2 instances function as virtual machines, providing scalable computing power for your applications.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Explore CodeDeploy for Deployments:&lt;/strong&gt;&lt;br&gt;
Familiarize yourself with CodeDeploy, a service that automates application deployments. Follow this guide for deployments on EC2.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Enhance Security with Security Groups:&lt;/strong&gt;&lt;br&gt;
Master the art of managing security groups to secure your EC2 instances. Follow best practices outlined in the official documentation.&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
