<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Fh Ndiritu</title>
    <description>The latest articles on DEV Community by Fh Ndiritu (@fh_ndiritu).</description>
    <link>https://dev.to/fh_ndiritu</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3726954%2F8958e3bc-72ff-48a8-8ad4-164775aaca16.png</url>
      <title>DEV Community: Fh Ndiritu</title>
      <link>https://dev.to/fh_ndiritu</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fh_ndiritu"/>
    <language>en</language>
    <item>
      <title>AI Landscape Design in 2026: Why the Generative Model Is the Easy Part</title>
      <dc:creator>Fh Ndiritu</dc:creator>
      <pubDate>Sun, 02 Aug 2026 11:32:18 +0000</pubDate>
      <link>https://dev.to/fh_ndiritu/ai-landscape-design-in-2026-why-the-generative-model-is-the-easy-part-2c23</link>
      <guid>https://dev.to/fh_ndiritu/ai-landscape-design-in-2026-why-the-generative-model-is-the-easy-part-2c23</guid>
      <description>&lt;p&gt;If you've used any modern image model, you already know it can redesign a backyard photo into something gorgeous. Feed it "make this yard a modern Japanese garden" and you get a convincing render in seconds. Problem solved, right?&lt;/p&gt;

&lt;p&gt;Not even close. The pretty picture is the &lt;em&gt;easy&lt;/em&gt; 20% of the problem. The hard 80% — the part that decides whether the output is a usable product or a toy — is everything the generative model doesn't know: whether those plants can survive in the user's climate, how to turn an unstructured photo into a spatial map, and how to emit a structured spec a contractor can actually build from.&lt;/p&gt;

&lt;p&gt;I want to walk through the real engineering pipeline behind a production AI landscape tool, because it's a nice case study in the gap between "a model that generates" and "a system that ships." I'll use &lt;a href="https://hadaa.app/" rel="noopener noreferrer"&gt;Hadaa&lt;/a&gt; as the concrete example since it's one of the more complete implementations in this space, but the architecture lessons generalize to almost any "generative + real-world constraints" product you might build.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem, stated like an engineer
&lt;/h2&gt;

&lt;p&gt;Input: 1–14 phone photos of a messy real yard.&lt;br&gt;
Output the user actually needs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Photorealistic renders in multiple styles and angles&lt;/li&gt;
&lt;li&gt;A planting list where every plant will survive at that location&lt;/li&gt;
&lt;li&gt;A contractor blueprint with zones, labels, and path widths&lt;/li&gt;
&lt;li&gt;A bill of quantities (plants + materials) that can be priced&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Only the first bullet is a generative-vision problem. The rest are structured-data problems wearing a trench coat. That's the whole insight.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage 1 — Photo to structured understanding
&lt;/h2&gt;

&lt;p&gt;You can't constrain or annotate what you don't understand spatially. So before any "make it pretty" step, the system has to convert pixels into structure: segment the scene (lawn, fence, patio, existing trees, structures), estimate rough geometry, and — when given several photos — stitch them into a coherent area map of the site.&lt;/p&gt;

&lt;p&gt;This is the unglamorous computer-vision plumbing: segmentation, depth/consistency estimation, and multi-image synthesis into a single top-down understanding. Get it wrong and everything downstream inherits the error. Get it right and you now have a canvas with &lt;em&gt;known regions&lt;/em&gt; you can preserve, edit, or replace deliberately instead of letting the model hallucinate over the whole frame.&lt;/p&gt;

&lt;p&gt;Practical takeaway if you build anything like this: &lt;strong&gt;masking and region-preservation are features, not afterthoughts.&lt;/strong&gt; Users need "redesign the beds but keep my house and pool exactly where they are." That's only possible if Stage 1 produced real regions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage 2 — Constrained generation (the "easy" part)
&lt;/h2&gt;

&lt;p&gt;Now the generative step. Given the structured scene and a style prompt, produce photorealistic variations. Modern diffusion-based image-to-image handles this well; realism in 2026 is basically a solved commodity.&lt;/p&gt;

&lt;p&gt;The engineering effort here isn't "can it look good" — it's control: honoring the preserved masks, keeping edits local (swap the bamboo for a palm without redrawing the house), and generating a consistent &lt;em&gt;set&lt;/em&gt; (same design across 8 camera angles and seasonal/night variants) rather than 8 unrelated images. Consistency across a set is the actual hard sub-problem, and it's a very different beast from one-shot generation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage 3 — The biology layer (the actually hard part)
&lt;/h2&gt;

&lt;p&gt;Here's where most "AI yard design" tools quietly fail, and where the interesting engineering lives.&lt;/p&gt;

&lt;p&gt;A generative model has no idea what grows where. It draws tropical palms into a render for a yard in USDA zone 4 because palms &lt;em&gt;look&lt;/em&gt; right for "lush garden," and the model optimizes for plausibility, not survivability. The user finds out nine months later when everything's dead.&lt;/p&gt;

&lt;p&gt;Fixing this is not a prompt-engineering problem — you can't just append "use appropriate plants" and trust it. It's a &lt;strong&gt;constraint-satisfaction problem layered on top of generation&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Resolve the user's location to a hardiness zone, plus rainfall and frost-date data.&lt;/li&gt;
&lt;li&gt;Maintain a plant database keyed on those environmental tolerances.&lt;/li&gt;
&lt;li&gt;When the render calls for "tall flowering shrub in the back corner," resolve that &lt;em&gt;role&lt;/em&gt; to an actual species that satisfies the location's constraints — not just any plant that looks similar.&lt;/li&gt;
&lt;li&gt;Verify the final planting list against the constraints before it ever reaches the user.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Hadaa calls this its "Biological Engine," and it's the piece competitors have had the hardest time replicating — precisely because it's not a model tweak, it's a whole data-and-rules subsystem bolted onto the generative core. This is the classic pattern: &lt;strong&gt;the moat isn't the model, it's the domain constraints you wrap around it.&lt;/strong&gt; (&lt;a href="https://hadaa.app/blog/best-ai-landscape-design-apps" rel="noopener noreferrer"&gt;Their 11-tool teardown&lt;/a&gt; is a decent map of which tools do and don't have this.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage 4 — From pixels back to a structured spec
&lt;/h2&gt;

&lt;p&gt;The last mile is turning an approved render into something buildable: a color-coded blueprint (zones, labels, path widths) and a bill of quantities (how many of each plant, how much material). This is essentially an information-extraction and layout-generation problem running &lt;em&gt;in reverse&lt;/em&gt; of Stage 2 — going from the visual design back to structured, quantified data.&lt;/p&gt;

&lt;p&gt;It's also where a lot of business value hides, because it's the deliverable a professional can hand to a crew. For a solo designer, the difference between "here's a nice picture" and "here's a blueprint, a zone-verified planting guide, and a priced BOQ generated in two minutes" is the difference between a sales cycle measured in days and one measured in minutes. That's the whole pitch of &lt;a href="https://hadaa.app/blog/best-ai-landscape-design-software-professionals-2026" rel="noopener noreferrer"&gt;Hadaa's Pro Studio&lt;/a&gt;, and the &lt;a href="https://hadaa.app/what-hadaa-offers-professional-landscapers" rel="noopener noreferrer"&gt;feature breakdown for professionals&lt;/a&gt; is basically a spec sheet for this stage.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Why not just use a general image model?"
&lt;/h2&gt;

&lt;p&gt;Because a raw diffusion model gives you Stage 2 and nothing else. To ship a real product you still have to build:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Capability&lt;/th&gt;
&lt;th&gt;Raw image model&lt;/th&gt;
&lt;th&gt;Full pipeline&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Photorealistic render&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Preserve existing structures&lt;/td&gt;
&lt;td&gt;⚠️ manual masking&lt;/td&gt;
&lt;td&gt;✅ region-aware&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consistent multi-angle set&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Climate-valid plant list&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅ constraint-checked&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contractor blueprint&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bill of quantities&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every ❌ in that first column is a subsystem, not a prompt. That's why "I could rebuild this with an API key in a weekend" is the famous last words of this category.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you're evaluating (not building) one
&lt;/h2&gt;

&lt;p&gt;Most people reading a piece like this just want to redesign their own yard or use one professionally. The engineering lens still gives you the buying criteria:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ask what happens in Stage 3.&lt;/strong&gt; Does it verify plants against your actual climate zone, or just draw greenery? This is the single highest-signal question.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask what comes out of Stage 4.&lt;/strong&gt; A render alone is wallpaper; a render plus a zone-verified planting guide plus a buildable blueprint is a project.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match the pricing model to your usage.&lt;/strong&gt; Pay-per-project (Hadaa starts around $9/render, no subscription) fits a one-time redesign; a monthly seat only makes sense if you're producing designs every week.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want to poke at a working implementation of the full pipeline, the fastest way to build intuition is to &lt;a href="https://hadaa.app/" rel="noopener noreferrer"&gt;run a yard photo through one&lt;/a&gt; and inspect what the planting guide and blueprint actually contain — that's where you can feel which stages a tool has really built versus faked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;The lesson generalizes well beyond gardens: in most "AI does X" products of 2026, the generative model is a commodity, and the durable engineering — and the moat — is the layer that constrains generation to real-world truth and emits structured, actionable output. In landscape design that layer is spatial understanding, climate-aware constraint satisfaction, and spec generation. The render is just the part that demos well.&lt;/p&gt;

&lt;p&gt;If you're comparing tools in this space, judge them by the parts that &lt;em&gt;don't&lt;/em&gt; screenshot nicely.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Have thoughts on constraint layers over generative models? I'd love to hear how you'd architect the biology engine differently — drop a comment.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>computervision</category>
      <category>startup</category>
    </item>
    <item>
      <title>I migrated 15M records of 13GB data from Mongo to Rails Postgres on a 512MB budget</title>
      <dc:creator>Fh Ndiritu</dc:creator>
      <pubDate>Thu, 22 Jan 2026 20:04:07 +0000</pubDate>
      <link>https://dev.to/fh_ndiritu/need-for-speed-from-swiss-cheese-heaps-to-a-15m-record-flow-state-4735</link>
      <guid>https://dev.to/fh_ndiritu/need-for-speed-from-swiss-cheese-heaps-to-a-15m-record-flow-state-4735</guid>
      <description>&lt;h2&gt;
  
  
  The "Complexity Tax" of Polyglot Persistence
&lt;/h2&gt;

&lt;p&gt;Scaling is exciting; it’s even more exciting before the grounding reality of the first month in production sets in and the thousands of customers you scaled for actually come knocking and you find out the actual numbers. Our Polyglot persistence setup was not unique, nor was it unexpected for an application handling hundreds of thousands of documents through extraction pipelines. But like most startups, we ended up scaling our costs long before we scaled our user base.&lt;/p&gt;

&lt;p&gt;Early on, we designed the system to store extracted paragraphs on a MongoDB instance while keeping the core relational data on Postgres. This delivered the expected flexibility of very fast writes and schema-less retrieval—until the day we had to critique the bill. We were paying a &lt;strong&gt;"Complexity Tax"&lt;/strong&gt;: the maintenance overhead of running two databases, the cognitive load of context switching, and the realization that we were leaving Postgres's best features—joins, foreign keys, and optimized includes—on the table.&lt;/p&gt;

&lt;p&gt;Postgres 18 and recent versions have introduced the most significant optimizations since the debut of TOAST decades ago. The switch from memory-heavy PGLZ to LZ4 compression for text columns, reliance on hardware acceleration (SIMD), and Async I/O with copy scans has changed the calculus. These updates promised near-similar speed to MongoDB, lower costs, and zero network latency between our data and our metadata.&lt;/p&gt;

&lt;p&gt;We also had early expectations that users would be constantly editing paragraph contents, leading to massive write spikes. This turned out to be false. After extraction, a paragraph is touched less than three times over its lifecycle. Our write-throughput bottleneck had disappeared, and so had our justification for the overhead of running two databases.&lt;/p&gt;




&lt;h2&gt;
  
  
  Designing the Migration Architecture
&lt;/h2&gt;

&lt;p&gt;Moving data is easy to judge: is it there or is it not? But getting it there without downtime is a different beast. We needed to account for potential rollbacks and the reality that our extraction pipelines would continue pouring new data into MongoDB while we were trying to empty it. This demanded a &lt;strong&gt;"Dual Write"&lt;/strong&gt; pattern combined with a &lt;strong&gt;Hot Swap&lt;/strong&gt; approach to ensure we kept exactly the same table names, models and relationships.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Dual Write Sequence
&lt;/h3&gt;

&lt;p&gt;The application needed to write to the new Postgres table without breaking existing code. The strategy involved creating a shadow Active Record model, &lt;code&gt;RailsParagraph&lt;/code&gt;, pointing to the standard paragraphs table.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;RailsParagraph&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;ApplicationRecord&lt;/span&gt;
  &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;table_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'paragraphs'&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We mirrored the MongoDB fields exactly but made one crucial decision: the &lt;code&gt;mongo_id&lt;/code&gt; would become a unique index. This was the anchor. It allowed migration scripts to be replayed infinitely without creating duplicates.&lt;/p&gt;

&lt;p&gt;We had more indexes but each B-tree index adds an extra write request, indexes like page_id, section_id and contains_images could wait and be added concurrently when doing the hot swap after data migration which involves renaming AR models and removing the MongoId models. &lt;/p&gt;

&lt;p&gt;To capture live data, we hooked into the Mongoid lifecycle. Using &lt;code&gt;after_save&lt;/code&gt; callbacks, every new or updated document was synced to Postgres immediately.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Inside the Mongo model&lt;/span&gt;
&lt;span class="n"&gt;after_save&lt;/span&gt; &lt;span class="ss"&gt;:sync_to_postgres&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sync_to_postgres&lt;/span&gt;
  &lt;span class="c1"&gt;# We use upsert to safely handle replays&lt;/span&gt;
  &lt;span class="no"&gt;RailsParagraph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upsert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="ss"&gt;mongo_id: &lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;id&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;to_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="ss"&gt;content: &lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;content&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="ss"&gt;unique_by: :mongo_id&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Idempotency is King
&lt;/h3&gt;

&lt;p&gt;For the backfill script, simple inserts were insufficient. If the script crashed halfway through, it needed to be restartable without manual cleanup. We utilized the &lt;code&gt;upsert_all&lt;/code&gt; method with the &lt;code&gt;unique_by: :mongo_id&lt;/code&gt; parameter. This ensured that if a record had already been synced by the Dual Write process, the backfill script would simply skip it or update it, preventing race conditions.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Poison Pill (Sanitization)
&lt;/h3&gt;

&lt;p&gt;Migration scripts are excellent at finding data corruption you didn't know you had. MongoDB, being schema-less, had happily accepted strings containing Null Bytes (&lt;code&gt;\u0000&lt;/code&gt;). PostgreSQL is stricter; a single null byte in a text column is a violation that aborts the entire transaction.&lt;/p&gt;

&lt;p&gt;A sanitization boundary was required, stripping these characters before they ever touched the SQL payload.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# The Sanitization Boundary&lt;/span&gt;
&lt;span class="ss"&gt;text: &lt;/span&gt;&lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'text'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;delete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u&lt;/span&gt;&lt;span class="s2"&gt;0000"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  The "Naive" Implementation &amp;amp; The Crash
&lt;/h2&gt;

&lt;p&gt;A separate PR would remove the MongoId index which would not be needed anymore, add the remaining indexes by disabling ddl transaction which allows us to add concurrently indexes without locking tables and finally insert our relationships between paragraphs, pages and sections. &lt;/p&gt;

&lt;p&gt;With the architecture set, the code was written—clean, idiomatic Ruby. A Sidekiq job was built to process 15 million historical records in batches of 2,000, spawning subsequent jobs in a neat daisy-chain. It looked perfect in the Pull Request.&lt;/p&gt;

&lt;p&gt;Then it was deployed.&lt;/p&gt;

&lt;p&gt;The jobs ran for a few minutes, processed a few batches, and then the worker would silently die. No error logs, just a vanished process. On Heroku, the graphs revealed the dreaded &lt;strong&gt;Error R14 (Memory Quota Exceeded)&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It wasn't a flat line; it was a "staircase." Every batch pushed the memory baseline higher. Ruby was releasing the objects, but the operating system wasn't reclaiming the RAM. We weren't fighting a bug in the code; we were fighting &lt;strong&gt;Memory Fragmentation&lt;/strong&gt;. The heap looked like Swiss Cheese—full of holes too small to reuse.&lt;/p&gt;




&lt;h2&gt;
  
  
  Debugging Physics
&lt;/h2&gt;

&lt;p&gt;At this point, standard advice is "Just install jemalloc." It’s the easy button that swaps the memory allocator to mask the problem. But rather than masking it, we chose to fix the physics of the container.&lt;/p&gt;

&lt;h3&gt;
  
  
  Taming the Arenas
&lt;/h3&gt;

&lt;p&gt;The standard Linux allocator creates multiple memory pools ("arenas") to optimize for multi-threaded speed. In a constrained 512MB container, these arenas compete for RAM and hoard pages. We forced glibc to be frugal by strictly limiting the number of arenas via the environment config.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Force the allocator to reuse memory chunks&lt;/span&gt;
heroku config:set &lt;span class="nv"&gt;MALLOC_ARENA_MAX&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The Hack: Manual Garbage Collection
&lt;/h3&gt;

&lt;p&gt;High-churn string processing is the enemy of Ruby's lazy Garbage Collector. By the time the GC wakes up, the OS has already allocated new pages, increasing RSS (Resident Set Size). We manually managed memory inside the loop by forcing the GC to sweep immediately after releasing the batch reference.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# 1. Clear the reference explicitly&lt;/span&gt;
&lt;span class="n"&gt;batch_docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kp"&gt;nil&lt;/span&gt; 

&lt;span class="c1"&gt;# 2. Force Ruby to clean up NOW&lt;/span&gt;
&lt;span class="no"&gt;GC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;full_mark: &lt;/span&gt;&lt;span class="kp"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="ss"&gt;immediate_sweep: &lt;/span&gt;&lt;span class="kp"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The Defragmenter
&lt;/h3&gt;

&lt;p&gt;Just cleaning up isn't enough if the heap is fragmented. We introduced compaction to physically move objects in memory, smashing the "Swiss Cheese" holes closed every few thousand records.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processed&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;zero?&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="no"&gt;GC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;respond_to?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:compact&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="no"&gt;GC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compact&lt;/span&gt; 
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  The "Crowded Door" (Connection Limits)
&lt;/h2&gt;

&lt;p&gt;With memory stabilized, ambition took over. We scaled the worker count to speed up the migration. That’s when the second wall appeared: &lt;strong&gt;Redis Connection Limits&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Running on a Redis "Mini" plan meant a hard limit of 20 connections. The workers crashed immediately on boot with &lt;code&gt;ERR max number of clients reached&lt;/code&gt;. Worse, attempting to debug via &lt;code&gt;rails console&lt;/code&gt; grabbed 10 connections instantly, effectively performing a "Murder by Console" on the production workers.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Fix: Connection Diet
&lt;/h3&gt;

&lt;p&gt;The application had to go on a strict diet. We capped &lt;code&gt;RAILS_MAX_THREADS&lt;/code&gt; to 2 and &lt;code&gt;SIDEKIQ_CONCURRENCY&lt;/code&gt; to 1. We traded theoretical capacity for actual stability.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Force Rails and Sidekiq to fit in the box&lt;/span&gt;
heroku config:set &lt;span class="nv"&gt;RAILS_MAX_THREADS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2
heroku config:set &lt;span class="nv"&gt;SIDEKIQ_CONCURRENCY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Optimizing Optimizations
&lt;/h2&gt;

&lt;p&gt;With the system stable, we hunted for bottlenecks. The biggest one wasn't the database; it was the &lt;strong&gt;logs&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Writing "INSERT INTO..." 15 million times is a massive I/O operation. The CPU was spending half its cycles waiting for the disk to acknowledge the log write. We silenced ActiveRecord for the migration block, and throughput instantly increased by 40%.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="no"&gt;ActiveRecord&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="no"&gt;Base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;logger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;silence&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="c1"&gt;# Pure signal, no noise.&lt;/span&gt;
  &lt;span class="n"&gt;migrate_batch!&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Counter-intuitively, setting &lt;code&gt;SIDEKIQ_CONCURRENCY=1&lt;/code&gt; also made the migration faster. With 3 threads, there was high context-switching overhead and constant contention for the database lock. With 1 thread, the worker entered a "flow state"—reading from Mongo and writing to Postgres in a tight, uninterrupted loop without swapping to disk.&lt;/p&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;We successfully migrated 15 million records, consolidated our database stack, and reduced our monthly infrastructure spend. But the real lesson wasn't about PostgreSQL or MongoDB.&lt;/p&gt;

&lt;p&gt;It was about &lt;strong&gt;Physics&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;We could have installed jemalloc and called it a day. But by choosing the hard path—tuning the arenas, manually compacting the heap, and mathematically sizing our batches—we actually learned how our runtime interacts with the hardware. We moved from a fragile, complexity-taxed system to a &lt;strong&gt;boring, standard stack&lt;/strong&gt;. And in infrastructure, "boring" is the highest compliment you can get.&lt;/p&gt;

</description>
      <category>ruby</category>
      <category>rails</category>
      <category>heroku</category>
      <category>postgres</category>
    </item>
  </channel>
</rss>
