<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Wycliffe Rotich</title>
    <description>The latest articles on DEV Community by Wycliffe Rotich (@wycliffrotichdev).</description>
    <link>https://dev.to/wycliffrotichdev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4040322%2F8ec5d690-d886-4db6-928f-b39761c80fe7.png</url>
      <title>DEV Community: Wycliffe Rotich</title>
      <link>https://dev.to/wycliffrotichdev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/wycliffrotichdev"/>
    <language>en</language>
    <item>
      <title>Your worker lost its lease. So why is it still running?</title>
      <dc:creator>Wycliffe Rotich</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:44:54 +0000</pubDate>
      <link>https://dev.to/wycliffrotichdev/your-worker-lost-its-lease-so-why-is-it-still-running-4i9f</link>
      <guid>https://dev.to/wycliffrotichdev/your-worker-lost-its-lease-so-why-is-it-still-running-4i9f</guid>
      <description>&lt;p&gt;Imagine this.&lt;/p&gt;

&lt;p&gt;You have a distributed compute cluster running AI workloads. One of your workers suddenly becomes unreachable.&lt;/p&gt;

&lt;p&gt;Your control plane stops receiving heartbeats. The execution lease expires. The scheduler assumes the worker is no longer available and reallocates the job.&lt;/p&gt;

&lt;p&gt;But there's a problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The worker never actually stopped executing&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It might be partitioned from the control plane, experiencing network problems, or simply unable to communicate. Meanwhile, it's still consuming CPU, memory, and potentially expensive GPU resources.&lt;/p&gt;

&lt;p&gt;Then it reconnects.&lt;/p&gt;

&lt;p&gt;It reports that the job completed successfully. But another worker may already have executed the same job.&lt;/p&gt;

&lt;p&gt;Whose result do you trust? Who owns the execution? And how do you prevent an old worker from modifying state it no longer owns?&lt;/p&gt;

&lt;p&gt;These aren't theoretical questions. They're the kinds of problems that make distributed infrastructure difficult to get right.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I decided to build around these failure scenarios.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I'm the founder and primary engineer behind AetherGrid, an open-source distributed AI workload orchestration platform.&lt;/p&gt;

&lt;p&gt;Rather than treating worker failures as simple offline events, I built the execution model around explicit ownership and recovery.&lt;/p&gt;

&lt;p&gt;Some of the mechanisms include:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Time-bound execution leases&lt;/strong&gt;: Workers renew ownership throughout job execution.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lease fencing&lt;/strong&gt;: Renewal, release, and outcome reporting are checked against lease identity, preventing stale owners from modifying execution state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multidimensional scheduling&lt;/strong&gt;: Workloads are assigned based on CPU, memory, and VRAM requirements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reconciliation&lt;/strong&gt;: Expired leases, dead workers, offline nodes, and abandoned jobs are handled through bounded recovery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tenant isolation&lt;/strong&gt;: Isolation is enforced in both application logic and PostgreSQL using composite foreign keys.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Correctness matters more than the happy path.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One engineering decision that shaped AetherGrid was testing failure conditions against real PostgreSQL.&lt;/p&gt;

&lt;p&gt;I don't want a test suite that only proves a worker can successfully execute a job. I want it to catch what happens when ownership changes, processes become unreachable, operations race, or a worker reports an outcome after losing its lease.&lt;/p&gt;

&lt;p&gt;AetherGrid currently has more than 1,600 passing tests and over 100 architecture decision records.&lt;/p&gt;

&lt;p&gt;It's built with Python, FastAPI, PostgreSQL, Go-based infrastructure experience, React, TypeScript, and Docker.&lt;/p&gt;

&lt;p&gt;The project is live and open source.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Explore the implementation:&lt;/strong&gt;&lt;br&gt;
&lt;a href="https://github.com/wycliffRotich-dev/aethergrid" rel="noopener noreferrer"&gt;https://github.com/wycliffRotich-dev/aethergrid&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Project website:&lt;/strong&gt;&lt;br&gt;
&lt;a href="https://aethergrid.co.ke" rel="noopener noreferrer"&gt;https://aethergrid.co.ke&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgsgjemrlwmvl7d7i4o0m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgsgjemrlwmvl7d7i4o0m.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Here's a question I'm genuinely interested in discussing:&lt;/p&gt;

&lt;p&gt;When a worker loses its lease but continues executing, how do you handle the resources it is still consuming while preventing it from reporting a stale result?&lt;/p&gt;

&lt;p&gt;Do you terminate immediately, wait for execution to finish, or use another mechanism?&lt;/p&gt;

&lt;p&gt;I'm interested in hearing how other infrastructure teams approach this problem, and I'm open to technical collaborations, design partnerships, and contributors who want to challenge the implementation.&lt;/p&gt;

&lt;p&gt;Because distributed systems don't become reliable just because everything works when the network behaves.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>devops</category>
      <category>infrastructure</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>Distributed Systems</title>
      <dc:creator>Wycliffe Rotich</dc:creator>
      <pubDate>Fri, 31 Jul 2026 05:24:41 +0000</pubDate>
      <link>https://dev.to/wycliffrotichdev/distributed-systems-4h3p</link>
      <guid>https://dev.to/wycliffrotichdev/distributed-systems-4h3p</guid>
      <description>&lt;p&gt;Most distributed schedulers pull work. Clients ask for tasks. It works until thousands of workers swamp your database with continuous polling.&lt;br&gt;
For AetherGrid, I took the opposite path. I built an active orchestration plane using a continuous, asynchronous cluster tick loop.&lt;br&gt;
Instead of workers constantly querying for state changes, the core engine drives the cluster loop forward from a centralized control plane. Every single second, a tick evaluates global cluster capacity, reconciles node heartbeats, and pushes deterministic work assignments to idle nodes.&lt;br&gt;
To maintain system reliability, I decoupled the domain. Infrastructure layers like SQLite and PostgreSQL handles persistence, while ephemeral lease states run entirely in memory to prevent execution deadlocks. If a worker goes quiet, the lease expires naturally. The cluster sweep catches it, marks the node offline, and gracefully frees the orphaned job for rescheduling on the next tick.&lt;br&gt;
Building distributed systems means planning for failure from line one. When you decouple your domain logic from your storage backend, handling race conditions becomes a predictable engineering task rather than an operational nightmare.&lt;br&gt;
The code is open source, fully containerized, and backed by a contract-tested architecture.&lt;br&gt;
Take a look at the architecture here: &lt;a href="https://github.com/wycliffRotich-dev/aethergrid" rel="noopener noreferrer"&gt;https://github.com/wycliffRotich-dev/aethergrid&lt;/a&gt;&lt;br&gt;
Are you a senior backend or infrastructure engineer working on distributed orchestration? How are you handling scheduler polling bottlenecks in your current stack? Let's talk engineering in the comments.&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxt08co036eosjfwd4abg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxt08co036eosjfwd4abg.png" alt=" " width="800" height="576"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>I built a distributed AI scheduler to prove DDD isn't just enterprise buzzword bingo</title>
      <dc:creator>Wycliffe Rotich</dc:creator>
      <pubDate>Wed, 22 Jul 2026 15:50:41 +0000</pubDate>
      <link>https://dev.to/wycliffrotichdev/i-built-a-distributed-ai-scheduler-to-prove-ddd-isnt-just-enterprise-buzzword-bingo-5eah</link>
      <guid>https://dev.to/wycliffrotichdev/i-built-a-distributed-ai-scheduler-to-prove-ddd-isnt-just-enterprise-buzzword-bingo-5eah</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm0jmo3hmljbjo8w2m2j6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm0jmo3hmljbjo8w2m2j6.png" alt=" " width="800" height="517"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
