DEV Community: Anton Shvein

Supervising What You Don't Own: A Controller Loop in OTP

Anton Shvein — Tue, 27 Oct 2026 11:52:15 +0000

One morning I opened Camelot and saw a card stuck in the executing column.
The agent had been "working" on it since the previous day.

The funny part is that the work was already done. The agent had written the
code, pushed the branch, opened a pull request, and I had merged it myself the
evening before. The board just didn't know about any of that.

This post is about why that happened, and about the piece of code I had to
write to make it stop. It seems to be a piece of code that many Elixir projects
eventually need and few of us plan for: a controller loop that reconciles what
the database believes with what the infrastructure is actually doing.

How Camelot runs agents

Camelot AI runs coding agents on a kanban board. You
create a task, an agent plans it, you approve the plan, the agent implements it
and opens a PR.

Each agent runs inside its own container. In production that means a Docker
Swarm service per task, and the BEAM talks to the Swarm manager over an HTTP
socket proxy. A single run can take forty minutes.

Inside the application, each task gets a TaskRunner GenServer. It builds the
spec, starts the runner, streams output over PubSub to the LiveView, and
finalises the session when the process exits.

It looks like a normal OTP design. For a while it behaved like one.

Where OTP stops

OTP supervises processes inside your VM. If a TaskRunner crashes, its
supervisor restarts it. If a process it monitors dies, it gets a :DOWN
message. This is the part everybody knows and loves.

None of it applies to the agent.

The agent is not a BEAM process. It is a CLI inside a container, possibly on
another machine. My TaskRunner does not supervise it directly. It watches the container, over
a network, through a proxy, with no delivery guarantees.

And here is the thing that broke me: a deploy kills my processes but not my
containers.

When I push to develop, the app container is replaced. Every TaskRunner
dies. Meanwhile the agent containers are still out there, happily compiling
somebody's Elixir project, with nobody left in the world listening to them.

The database still says status: :running. The world says something else.

The row outlives the process

Once I accepted that, the design got a bit simpler to reason about.

The durable thing is a Session row in Postgres. The TaskRunner GenServer is
just the thing that happens to be attending to it right now. Processes are
disposable. Rows are not.

So after a restart, somebody has to walk the rows and ask, for each session that
claims to be running: is this still true?

That somebody is Camelot.Runtime.Reconciler. A GenServer that runs once at
boot and then every sixty seconds. Probably the most valuable module in the
whole runtime.

def handle_info(:tick, state) do
  state = do_reconcile(state)
  Process.send_after(self(), :tick, @tick_ms)
  {:noreply, state}
end

If you have written a Kubernetes controller, this looks familiar. Observe the
desired state, observe the actual state, take the smallest action that moves
one towards the other. Then do it again in a minute, because you were probably
wrong about something.

Adopting a run that lost its owner

The first thing the reconciler does is find :running sessions whose owning
TaskRunner no longer exists.

The naive move is to mark them failed. Correct, and also terrible: you just
threw away thirty minutes of an agent's work and real money because you
deployed a CSS fix.

So instead Camelot tries to adopt them. If the container is still alive, we
start a fresh TaskRunner and re-attach it to the run already in progress.

The problem is that after a restart we no longer have the original docker exec
id, so we cannot ask Docker for its exit status. The exec is gone as far as we
are concerned, even though the process inside the container is fine.

The solution lives in the container. Every agent invocation goes through a small
wrapper script:

out="/tmp/camelot-output-${CAMELOT_SESSION_ID:-session}.log"
set +e
"$@" 2>&1 | tee "$out"
code=${PIPESTATUS[0]}
set -e

echo "$code" > "/tmp/camelot-exit-${CAMELOT_SESSION_ID:-session}"
exit "$code"

Two files, and the order matters:

The output is tee'd to a per-session log, so the complete result can be fetched with a short docker exec cat after the fact, instead of depending on a long-lived, mostly idle exec stream.
The exit code is written to a marker file last, so its presence strictly implies the output file is already complete.

An adopting session polls for that marker. When it appears, we know the run
finished and with what code, and we read the tee'd output next to it.

I was quite pleased with this. It survived deploys. It recovered work. It felt
robust.

The poll that never ends

Here is the poll loop, roughly as I first wrote it: check for the marker, and if
it is not there, sleep and check again.

Read that sentence again and see if you can spot what is missing.

There is no way out.

If the marker never appears, the loop runs forever. And there is a very ordinary
situation where the marker can never appear: the container was replaced.

Swarm reschedules a task. A node runs out of memory and kills the container.
The boot sweep rolls the service onto a newer runner image. In all three cases a
brand new container comes up with the same service name, perfectly healthy, and
completely useless to us — the marker and the tee'd output lived in the previous
container's /tmp.

My test cluster is two very small nodes. Reschedules are not a rare event there.
They are Tuesday.

So the sequence that produced my eighteen-hour card:

A session started executing.
I deployed. The TaskRunner died, the container kept running.
The container got rescheduled onto another node. The marker died with it.
The reconciler adopted the session and started polling for a file that no longer existed anywhere in the universe.
It polled. And polled.

The session stayed :running, so the task stayed in executing. And every
other recovery path politely stepped around it: the stale-session sweep saw a
live owning process, so it skipped it. The stale-handle sweep saw a live
service, so it skipped it. Everything was working exactly as designed.

Meanwhile the agent had finished long ago and opened the PR. I merged it by
hand. The board never noticed.

The fix: a decision with no I/O

The fix itself is not complicated. What matters is where I put it.

I pulled the decision out of the polling loop entirely, into its own module with
no I/O in it at all:

def decide(container_started_at, session_started_at, elapsed_ms, budget_ms) do
  cond do
    container_replaced?(container_started_at, session_started_at) ->
      {:give_up, :container_replaced}

    elapsed_ms >= budget_ms ->
      {:give_up, :timeout}

    true ->
      :poll
  end
end

And the check underneath it:

def container_replaced?(%DateTime{} = container_started, %DateTime{} = session_started) do
  DateTime.after?(container_started, session_started)
end

def container_replaced?(_container_started, _session_started), do: false

If the live container started after the session's exec did, the marker is
provably gone. Give up now, fail the session so it is user-recoverable, and
re-queue the task.

Two details I got wrong on the first attempt, and would like to save you:

The fallback clause returns false, not true. If either timestamp is
unknown, we keep polling and let the wall-clock budget decide. A missing
timestamp is not evidence of a dead container, and I would rather wait fifteen
minutes than abandon a healthy run because Docker hiccuped on one request.

The container start time is re-read on every iteration, not sampled once at
the beginning. A deploy can replace the container while the adoption is
already running. A sample taken a moment too early would keep the poll going for
the whole budget.

The wall-clock budget stays as a backstop. Two independent give-up conditions,
because the precise one depends on timestamps I might not have.

Why the pure function matters

The polling loop needs Docker, a node proxy, a running Swarm, and forty minutes
of patience. The decision needs four values.

Once they are separated, the test for the incident that cost me a day is this:

test "gives up when the container was replaced after the session exec" do
  container = dt("2026-07-14T12:47:00Z")
  session = dt("2026-07-14T10:44:00Z")

  assert AdoptPolicy.decide(container, session, 0, 900_000) ==
           {:give_up, :container_replaced}
end

Real timestamps from the real incident. No mocks, no Docker, milliseconds to
run. And because the module is pure, the moduledoc examples are doctests and the
documentation cannot drift away from the behaviour.

This is the part I would most like you to take from the article. A controller
loop is mostly policy, and policy has no business being tangled up with the
I/O that feeds it. The reconciler does the same trick for recovery:

def recovery_action(LocalPort, _kind, _handle, _presence), do: :fail
def recovery_action(_backend, :bootstrap, _handle, _presence), do: :fail
def recovery_action(_backend, _kind, nil, _presence), do: :fail
def recovery_action(_backend, _kind, _handle, :present), do: :adopt
def recovery_action(_backend, _kind, _handle, _presence), do: :fail

Five clauses. No side effects. Exactly the kind of code Elixir is good at, and
exactly the kind that ends up buried in the middle of a handle_info if you are
not deliberate about it.

The order of the steps matters

One more trap, because it took me a second incident to see it.

The boot sweep re-pins runner images to the newest digest. That replaces
containers — which means the boot sweep itself creates the very condition that
dooms an adoption.

So the recovery of rolled tasks has to run before the stale-session sweep.
Otherwise the sweep hands a session we already know is doomed to an adoption
that will poll for fifteen minutes before reaching the same conclusion.

In a controller loop, the order of your reconciliation steps is not an
implementation detail. It is part of the contract, and it deserves a comment
saying so.

What I'd tell myself six months ago

Every wait needs a deadline. Not most waits. Every one. If you write a poll loop with no exit condition, you have written a hang and just not met it yet.
The database row is the process. Anything you want to survive a deploy has to be a row, not GenServer state. The GenServer is a fast index over the rows, rebuilt on boot.
Polling is your Process.monitor. You do not get :DOWN messages from another machine. Adopt, fail, retry — you are writing a restart strategy by hand, so write it like one.
Never assume you finish in the container you started in. Sounds obvious. Responsible for most of the bugs in this post.

None of these are new ideas. Kubernetes controllers have worked this way for
years. What surprised me is how naturally the whole thing sits in OTP once you
stop expecting supervision to reach across the network for you. A GenServer, a
sixty-second tick, some pure decision functions and a couple of Ash queries make
a genuinely small controller.

Conclusion

The stuck card was not a Docker bug or a Swarm bug. It was me assuming that a
file I wrote would still be there when I came back for it.

OTP is very good at supervising things it owns. The moment your real workload
lives somewhere else — a container, another node, somebody else's API — you have
to rebuild the guarantees yourself, and it seems better to rebuild them
explicitly: durable state, a bounded wait, a policy you can test, and a loop
that runs again in a minute because you were probably wrong about something.

Next in this series: what happens when a polling trigger sits in front of something that costs money on every tick.
That one cost me $63 in a single night.

The process is going on...

OpenTelemetry Collector on Docker Swarm

Anton Shvein — Thu, 22 Oct 2026 06:24:48 +0000

Two clusters, running for months, and no logs anywhere except docker service logs and my own patience. I decided to fix that.

I thought it is a configuration task. Write some YAML, point it at a backend, go
and do something else. It seems it was not.

Every version of that YAML passed otelcol validate. Every one. Including the
version that lost data, the one that shipped every line twice, the one that
filled a disk with its own output, and the one whose regex could never match
anything at all.

Idea

Camelot AI runs on two small Docker Swarm clusters. Test
and production, two nodes each: a 1 GB manager that is always a bit close to the
edge, and a big ARM worker. The goal was ordinary — container logs, node and
container metrics, and an OTLP endpoint for traces I haven't written yet.
Everything to PostHog, which I already use.

Architecture

I ended with two components:

an agent on every node, as a Swarm global service, reading container logs off that node's own disk;
a gateway, one replica, holding the credentials and talking out.

The agent-per-node is not a design preference. Container logs live in
/var/lib/docker/containers on the machine that ran the container. One central
collector can't see other nodes' logs, no matter how you configure it.

It ingested its own logs

The first run on a real node produced 1.8 million lines of debug output in about
seventy seconds. On a node with 31 GB free it is not a comfortable rate.

Almost all of it came from one container — the collector itself. It discovers
containers over the Docker socket and tails each one's log file. It discovered
itself. So it read its own output, and printing anything about a log record
produced another record, which it then read. A small loop, running at whatever
speed the disk allowed.

The fix is two lines of excluded_images. I would never have written them from
the documentation — you need to see the loop first.

extensions:
  docker_observer:
    endpoint: unix:///var/run/docker.sock
    # most containers here publish no ports, and without this they are
    # never discovered at all
    include_all_containers: true
    # globs, because the deployed tag is a commit sha
    excluded_images:
      - "*otel-agent*"
      - "*otel-gateway*"
      - "*opentelemetry-collector-contrib*"

Later I excluded the gateway too, for the mirror reason: a failing export makes
the gateway log, and shipping that log back through the gateway amplifies exactly
the failure it reports.

The duplicate

The numbers still looked high. Every line from one container appeared exactly
twice.

The observer has an option include_all_containers. I read it as "emit an
endpoint for every container", which is true, and missed the point: it emits a
port-less endpoint in addition to one per exposed port. A container
publishing two ports is matched three times. Three filelog receivers on one
file. Every line shipped three times, to a backend that bills by volume.

The fix is port == 0 in the discovery rule. Obvious once you see the
duplicates, invisible before.

receivers:
  receiver_creator/containers:
    watch_observers: [docker_observer]
    receivers:
      filelog:
        # port == 0 is the single port-less endpoint - matches once,
        # not once per exposed port
        rule: type == "container" && port == 0
        config:
          include:
            - /var/lib/docker/containers/`container_id`/`container_id`-json.log
          start_at: beginning

Making it safer lost data

This one I like the most, because I did it to myself on purpose.

A review comment suggested lowering the memory limiter from 160 MiB to 128, to
leave more headroom on the 1 GB manager. It sounds sensible — the node really is
OOM-prone. I agreed and changed it.

Then I ran it, and the log said data refused due to high memory usage.

With the tighter limit, the limiter refused batches while the agent worked
through the backlog of existing logs — and the filelog receivers, when refused,
dropped those records instead of waiting. I made the collector safer against OOM
by making it lose logs. The whole point of the collector is to not lose logs.

The real fix was not a number. It was retry_on_failure on the receivers, so
back-pressure replaces dropping: when the limiter bites, the receiver stops and
waits.

processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 160
    spike_limit_mib: 40

receivers:
  receiver_creator/containers:
    receivers:
      filelog:
        config:
          # without this the receiver drops records outright whenever
          # the limiter refuses a batch, instead of waiting for it
          retry_on_failure:
            enabled: true
            initial_interval: 1s
            max_interval: 30s

Then 160 MiB was fine. A real node settles around 137 MiB.

I would have merged the 128 and felt responsible about it.

A fix that was not a fix

In PostHog every metric arrived with service set to unknown. That part was
easy — metrics carried no service.name at all, and unknown is what a backend
shows when it's missing. Logs had one, metrics never did.

So I wrote a transform to fill it from the Swarm service label, ran it, and
unknown was gone. Fixed.

Except the names read srv-captain--db.1.mms89736vzjrjpsfn6yagtb0w. The Swarm
label branch never matched. docker_stats promotes those labels to resource
attributes, not data point attributes, despite the option being called
container_labels_to_metric_labels, and I read them in the wrong context, where
they are always nil. Everything fell through to the fallback.

receivers:
  docker_stats:
    # this is where docker.service.name comes from
    container_labels_to_metric_labels:
      com.docker.swarm.service.name: docker.service.name

processors:
  # wrong - datapoint context, where the label is always nil
  transform/service_name_wrong:
    metric_statements:
      - context: datapoint
        statements:
          - set(resource.attributes["service.name"], attributes["docker.service.name"])

  # right - despite the option's name, docker_stats puts it on the resource
  transform/service_name:
    metric_statements:
      - context: resource
        statements:
          - set(resource.attributes["service.name"], resource.attributes["docker.service.name"])

If I only checked that unknown disappeared, I would have shipped that.

The obvious alternative would have failed too, for another reason. Trimming the
task suffix off the container name works for replicated services, named
service.slot.task, and quietly breaks for global ones, named
service.node-id.task — which is exactly what my own agent is.

The escape hatch

The agent has to run on every node, which in Swarm means a global service.
CapRover, which manages my clusters, has a Service Update Override field for
exactly this. So I put the mode in it, and it refused to save: must specify only one service mode.

Its merge is additive. It merges Global next to the generated Replicated
instead of replacing it, and Docker rejects a spec carrying both. There is an
open issue from 2024 with a
suggested workaround using a pre-deploy script.

The workaround can't work, and it took reading CapRover's source on the manager
to see why. It creates every app's service from a placeholder image first, so
the service always exists as replicated before any deploy, and the daemon
refuses mode changes on an existing service: service mode change is not allowed. No update-time manipulation converts it.

A service has to be created global. So: let CapRover deploy normally, then
recreate the service from the spec CapRover itself generated, with only the mode
swapped.

#!/usr/bin/env bash
set -euo pipefail

SERVICE="${1:-otel-agent}"

# idempotent: a global service is already what we want
docker service inspect "$SERVICE" --format '{{json .Spec.Mode}}' \
  | grep -q Global && exit 0

# reuse the spec CapRover generated, and change only the mode. rebuilding
# it by hand would drop the labels, env, limits and placement it set.
docker service inspect "$SERVICE" --format '{{json .Spec}}' > /tmp/spec.json
python3 -c '
import json
spec = json.load(open("/tmp/spec.json"))
spec["Mode"] = {"Global": {}}
json.dump(spec, open("/tmp/global.json", "w"))
'

docker service rm "$SERVICE"
curl -s --unix-socket /var/run/docker.sock -X POST \
  -H "Content-Type: application/json" \
  -d @/tmp/global.json \
  http://localhost/v1.44/services/create

After that CapRover manages it happily, because merging Global into
an already-global spec is a no-op.

A short bash script, and both clusters survived several redeploys since. I put
the findings on the upstream issue — it felt honest after taking the
maintainer's time indirectly.

Everything was info

By this point the pipeline was healthy by every measure I had. Zero export
errors, memory where I expected it, both clusters green. Then I looked at the
data, and every record said info.

Not most of them. Every one. A Postgrex disconnect and a failed schema migration
sat in PostHog at the same level as a 200 response:

body: "07:11:40.138 [error] Postgrex.Protocol ... disconnected: ... timed out"
severity_text: info    severity_number: 9

The container operator unwraps Docker's JSON envelope — log, stream,
time — and stops there. The application's own line stays raw text, so nothing
sets severity, and PostHog shows unset severity as info. The collector was
working perfectly and the result was useless.

So I wrote a regex to pull the level out of the Elixir format. It matched
nothing. I had written it as a folded YAML scalar:

# matches nothing, and validates fine
regex: >-
  ^\d{2}:\d{2}:\d{2}\.\d{3}
  (?:request_id=(?P<request_id>\S+) )?
  \[(?P<level>[a-z]+)\]

# works - one line, single-quoted so the backslashes survive
regex: '^\d{2}:\d{2}:\d{2}\.\d{3} (?:request_id=(?P<request_id>\S+) )?\[(?P<level>[a-z]+)\]'

>- joins its lines with spaces. That puts an extra space before the bracket,
so the pattern can't match a real line. otelcol validate accepted it, because
it is a perfectly valid regex. It just fires never.

Then the regex worked, and it still wasn't right. Docker gives the collector one
record per physical line, so a text-formatted stack trace arrives as several
records and only the first carries [error]. The rest strand at the default
level:

"09:58:10.874 [error] Could not create schema migrations table..."  -> error
"  * The database does not exist"                                   -> info

No pattern fixes that — the lines are separate records before the collector ever
sees them. So released builds log JSON now. JSON escapes the newlines, so one
event stays one line and one record, and every metadata key arrives as a field
instead of being recovered by guesswork.

# config/runtime.exs, :prod only - mix phx.server stays readable
config :logger, :default_handler,
  formatter: LoggerJSON.Formatters.Basic.new(metadata: [:request_id, :mfa, :crash_reason])

One last trap on the way out. Severity parsers keep the word the application
used, and Elixir says warning. PostHog filters severity by exact match against
six lowercase buckets, so warning never matches warn. The level looked
parsed and stayed unfilterable.

What I learned

otelcol validate accepted every broken version of this config. It was right
to — the schema was valid every time.

Validation proves shape. It doesn't know that a discovery rule matches three
times, that a receiver can see its own output, that a memory limit interacts
with a backlog, or that a label lives in a different context than you assumed.
Those are behaviours, and behaviour only exists on a real node with real
containers. Most of the problems here would have passed any review I could
write, because the YAML looks correct. It is correct, in the sense a linter
means. The folded scalar is the purest case: a valid regex, a valid config, a
green check, and a pattern that can never match a single line.

There is one nice detail in all this. Containers here arise and cease all the
time — a task container lives forty minutes, a probe container lives one second
— and the only thing that persists across them is a small file of read offsets
on the host. Impermanence (མི་རྟག་པ།) is not a metaphor in a Swarm cluster. It
is the scheduling model, and if you want to remember anything you have to write
it down somewhere the scheduler can't reach.

Both clusters now ship logs and metrics, with zero export errors and about 137
MiB on the node I was worried about. Traces are ingesting and empty, because I
haven't instrumented the app yet. That's the next piece.

Neither cluster has Docker log rotation configured, which I found when the
parser choked on six truncated records left by a disk-full event two weeks
earlier. It is still not configured. It needs a dockerd restart per node, and
I haven't decided when.

If there is one thing worth taking from this, it is the memory limiter. A
sensible suggestion, applied carefully, made the system lose the exact data it
exists to collect — and the only reason I know is that I ran it and read the
output, instead of trusting that a smaller number is a safer number.

The process is going on...

$63 in One Night: When Your Elixir Polling Loop Costs Money

Anton Shvein — Tue, 13 Oct 2026 08:06:53 +0000

I opened the board one morning and one card looked wrong. Not broken — busy.
It had a wall of activity behind it from the previous evening, and I had not
touched it since lunch.

My stomach did the thing it does before I even know why. I went digging to find
out how much that evening had cost me.

243 agent runs in eight hours. About 31 million input tokens. $64.25 spent, of
which $63.10 was pure waste. Zero commits produced.

I read that number a few times. Sixty-three dollars, for nothing, while I slept.

Then the second part landed, and that one was worse. I did not write the code
that did this. Claude Code wrote it, inside Camelot, on a task I created myself.
I read the diff. I approved it. I merged it. It looked fine to me.

The loop did not stop because I noticed it, or because a guard caught it. It
stopped because the runner eventually crashed on malformed output and pushed the
task into an error state. Even the crash felt, in hindsight, like the most
responsible member of the team.

So this is the story of a three-line cron trigger, of what happens when you put
something expensive behind it, and of a code review where I was looking at all
the wrong things.

A two-minute GitHub poller in Ash and Oban

In Camelot AI
a task moves through stages on a kanban
board. When the agent opens a pull request, the task lands in the pr stage and
a poller takes over: every two minutes it checks GitHub and reacts. Merged →
done. CI failing → send the agent back in. Reviewer requested changes → send the
agent back in.

That poller was itself a Camelot task. I described what I wanted, the agent went
away, opened a PR, and I read it on a Tuesday evening. With Ash and Oban the
trigger is almost nothing to write, and what came back was this:

triggers do
  trigger :check_pr_status do
    action(:check_pr_status)
    scheduler_cron("*/2 * * * *")
    queue(:github)
    max_attempts(3)

    where(
      expr(
        not is_nil(pr_number) and
          stage == :pr and
          project.status == :active
      )
    )
  end
end

That is the whole thing. AshOban takes the where
expression, finds matching
rows every two minutes, and runs the action on each of them. I have written
dozens of these by hand. They are boring and reliable and I like them very much.

Which is exactly why I read this one so fast. I recognised the shape, and the
moment I recognised it I stopped reading and started scrolling.

The problem is that this one is boring and reliable in front of something that
starts a container and buys LLM tokens.

What actually happened?

Further down the same diff — well past the point where I was still reading
carefully — there was a branch that read, in spirit:

cond do
  # ...
  has_review_state?(reviews, "CHANGES_REQUESTED") ->
    request_changes(task, comments, 0)
  # ...
end

reviews is what GitHub's reviews endpoint
returns. And here is the thing
neither of us thought about carefully enough — though only one of us was supposed
to:

GitHub keeps every review ever submitted, forever.

A review is not a flag you clear. It is an immutable historical record. If a
reviewer requested changes once, that entry is in the list on every subsequent
call, for the rest of the pull request's life. Dismissing it adds a new review
with state DISMISSED — the old one stays.

So has_review_state?(reviews, "CHANGES_REQUESTED") was not asking "does this PR
need work?". It was asking "has anyone ever asked for changes on this PR?". Once
true, it was true forever.

I read that sentence three times before it landed. The function name is honest,
by the way. has_review_state? says precisely what it does. I read it as "does
this PR need work?" because that is what I wanted it to mean, and because the
line under it did the thing I expected the line to do.

Every two minutes, the trigger matched the row. Every two minutes, the condition
was true. Every two minutes, Camelot dispatched an agent to go fix feedback that
had been addressed hours earlier.

And the agent, to its credit, kept looking, finding nothing to do, and exiting
cleanly. 243 times. More discipline than I showed in that review, honestly.

Level-triggered, with a price tag

The name for this is a level-triggered check. The code was testing a state
("changes are requested") instead of an event ("changes were requested, and I
have not acted on it yet").

Level-triggered is usually the right default in a reconciliation loop. It is what
makes the loop self-healing: if you miss an event, the next pass still sees the
state and fixes it. The first article in this series
is an argument in favour of exactly that style.

The difference is the actuator on the other end.

In an ordinary Ash application, a level-triggered check that fires redundantly
costs you a database query every two minutes, and you will never notice. Here the
actuator provisions a container, clones a repository, and hands a language model
about 130 thousand tokens of context.

A redundant poll in an ordinary app is a rounding error. A redundant poll here is
an invoice. That sentence is still a bit hard for me to write.

There is a second-order effect that makes it worse. The prompt includes the PR
conversation. Every useless run added its own noise to that conversation. So the
cost per run climbed steadily through the night — from about $0.12 early on to
$0.45 by the end. The loop was not just wasteful, it was accelerating.

And I did not know any of this while it was happening. Nobody was watching. That
is probably the loneliest part of this kind of failure: it happens at 3am, at its
own quiet pace, and it sends you the bill at breakfast.

Why none of the guards helped

This thing was not built without limits. There were three mechanisms that should
have stopped it, and the loop walked straight past all three.

The freshness check was dead code on that path. There is a
pr_comments_seen_at timestamp precisely so feedback is not acted on twice. But
it was only consulted by the comments branch, which sits below the review
branch in the same cond. The review branch never reached it. The guard existed,
was correct, and was unreachable.

The branch reset the counter that was supposed to bound it. Look again at the
call:

request_changes(task, comments, 0)

That third argument is the auto-fix attempt counter. There is a cap on
consecutive automatic re-dispatches — default 2 — so a PR the agent cannot fix
gets left for a human. But the cap only counts merge-conflict and CI-failure
fixes, and this branch passed 0, resetting the counter on every single pass.

The guard was not bypassed by accident. The code actively disarmed it, twice an
hour, all night. And that 0 sat in the diff, in plain sight, three characters
wide.

The order of the branches made recovery impossible. CHANGES_REQUESTED was
checked before APPROVED. So even if the reviewer had come back and approved the
PR, the loop would have kept firing — the earlier branch matched first and the
approval was never reached.

I want to be clear that none of these are exotic mistakes. Each one is the kind
of thing that reads fine in review — I know that for a fact, because each one was
on my screen and I read it fine. For about half an hour that morning I tried to
make this the agent's failure. It is not. The agent wrote plausible code, which
is what it does, and it wrote the pattern I have written myself dozens of times.
That is exactly why it got through: it looked like my code. I am the approval
gate, and the gate opened.

The fix: compare the review against the last commit

Two changes, and the important one is not the obvious one. This time I wrote them
by hand — not because that is the lesson, but because that is where my confidence
was that morning.

First, reduce the review list to each reviewer's current verdict before asking
anything about it:

@review_states ~w(APPROVED CHANGES_REQUESTED DISMISSED)

def latest_reviews(reviews) do
  reviews
  |> Enum.filter(&(&1["state"] in @review_states))
  |> Enum.group_by(&reviewer_login/1)
  |> Enum.map(fn {_login, submitted} ->
    Enum.max_by(submitted, &review_date/1)
  end)
end

Group by reviewer, take the latest state-bearing review from each. COMMENTED
and PENDING carry no verdict and never supersede one. Now a reviewer who
requested changes and later approved reads as approved, which is what a human
would say if you asked them.

Second — and this is the part that actually stops the loop — apply the same
staleness guards to reviews that the comment path already used:

def changes_requested?(reviews, last_commit_date, seen_at) do
  reviews
  |> latest_reviews()
  |> Enum.any?(fn review ->
    review["state"] == "CHANGES_REQUESTED" and
      unaddressed?(review_date(review), last_commit_date, seen_at)
  end)
end

A verdict has to be newer than the last commit — pushing a fix addresses it —
and unseen since the last dispatch. Two independent reasons to consider a
review handled.

On the actual incident this alone would have ended it in one pass. The review was
submitted at 11:46:46Z. The head commit was 11:52:34Z. The review was already
stale before the loop even started. One timestamp comparison. That is the whole
tragedy.

The other bug: an optional call inside a `with` chain

While I was in there I found a second failure with the same flavour — same PR,
same review, also mine to catch and also missed. It is worth its own paragraph
because it is a pure Elixir idiom mistake, and the kind that is very easy to
approve.

The poller makes five GitHub calls, chained in one with:

with {:ok, pr_data} <- Client.get_pull_request(...),
     {:ok, reviews} <- Client.list_pull_request_reviews(...),
     {:ok, comments} <- Client.list_pull_request_comments(...),
     {:ok, commits} <- Client.list_pull_request_commits(...),
     {:ok, check_runs} <- fetch_check_runs(...) do
  # ...
else
  {:error, reason} ->
    Logger.warning("Failed to check PR for task #{task.id}: #{inspect(reason)}")
end

This looks like good Elixir. It mostly is. That is the trap — it is idiomatic
enough that your eye slides over it. But the last call needs a Checks: Read
permission on the GitHub App installation, and without it GitHub returns 403.

One 403, and the whole with collapses into a log line. Merge-conflict handling,
review handling, comment handling, merged detection, closed detection — all of it
stopped, for every task in the pr stage, silently, because of an optional
enrichment call.

The fix is to be explicit that this one call is allowed to fail:

@spec best_effort_check_runs({:ok, [map()]} | {:error, term()}) :: {:ok, [map()]}
def best_effort_check_runs({:ok, runs}), do: {:ok, runs}

def best_effort_check_runs({:error, reason}) do
  Logger.warning("check-runs unavailable (#{inspect(reason)}); treating as no checks")
  {:ok, []}
end

Degrade to "no checks" and keep going. CI failures go undetected until the
permission is granted, which is annoying but survivable. Everything else keeps
working, which is the point.

with gives you one error channel for a whole chain. That is the feature. It
becomes a bug the moment one link in the chain is not actually essential, and it
fails quietly because the else clause is usually where you put the logging and
stop thinking.

What I changed in how I work

I was reviewing agent diffs for the wrong things. I read for naming, for
structure, for whether it fits the codebase, for whether I would have written it
that way. All of that was fine here, which is precisely why I merged it. Now I
try to ask three questions of any diff that touches a loop: what makes it fire,
what makes it stop, and what does one pass cost. If the diff does not answer the
second and the third, it is not ready, however good it looks.

A level-triggered check needs a termination condition, not just a trigger
condition. "Is this state true?" is not enough. "Is this state true and have I
not already handled it?" is the real question, and the second half has to be
durable — a timestamp or a counter in the database, not something in memory.

Never reset a guard inside the branch the guard protects. If I had to name the
single line that cost me $63, it is the 0 in that function call. One character,
which I did not write and did not question. I keep coming back to that.

Best-effort calls do not belong in a with chain. If a step is optional,
write a function that makes it always succeed and say so in the docs. The type
signature becomes the documentation: {:ok, term} | {:error, term} going in,
{:ok, term} coming out.

Cost is a first-class signal. I had logs. I had session records. What I did
not have was anything that would say "this task has run 243 times today" or "this
task has spent $60". A counter and a threshold would have caught this in twenty
minutes instead of eight hours. The
OpenTelemetry collector I put on those Swarm nodes
ships logs and traces already — it just does not know anything about dollars yet.
That is now on the list, near the top.

Conclusion

The trigger was three lines. The branch that looped was four. Nothing in this
incident was complicated, and nothing in it would have mattered at all in a
normal CRUD application — a redundant poll would have cost me a few hundred
cheap queries and I would never have known.

What changed is what sits on the other end of the loop. When the actuator costs
money, "harmless redundant work" stops being a category. Every pass has to earn
itself.

And there is the part I find harder to write. Camelot exists because I think an
agent's work should pass a human gate before it becomes real. The gate was there.
It worked exactly as designed. The gate is me, on a Tuesday evening, looking at a
diff that resembled something I would have written myself, and clicking approve.
A human approval loop is not a guarantee. It is a place where a human has to
actually be present, and that is a harder promise than the one I thought I was
making.

My practice says: notice the thought that repeats, and stop feeding it. It seems
to be my poller needed that teaching. So did my reading.

Next time I want to write about the layer underneath all of this: what happened
when I tried to run long-lived, stateful agent containers on an orchestrator that
is designed around the opposite assumption.

And if an agent wrote a loop for you recently and you approved it because it
looked right — go check what it did last night. I really wish somebody had told
me that a week earlier.

Lama Bot (དླ་མ་བོཏ།) - helps to coup with depression with the help of Buddhist practice

Anton Shvein — Sat, 27 Apr 2024 06:08:26 +0000

This is a submission for the Coze AI Bot Challenge: Bot Innovator.

What I Built

According to WHO depression and burnout is a rapidly growing mental condition. They even emphasise that it is going to beat cancer and AIDS by number of deaths.
I had a deep depression recently and Buddhist practice helped me a lot.
The bot is my small contribution in order to make this world more peacefull. It is and it's going to be completely free to use and free from any advertisement.
This is a very first PoC in order to collect data and start the process of user acquisition. There is a lot more to do in future.

Demo

Lama bot on Coze (direct link)
Lama Bot in Telegram

Your Configuration

ATM, bot's configuration is easy enough but I'm going to enrich it as soon as I get feedback. Here is the prompt:

# Персонаж
Ты Буддистский лама по имени Бот (ངོཏ་). Ты обладаешь бесконечным принятием, любящей добротой и состраданием.  Твоя главная цель помочь всем живым существам предодлеть страдания. Для этого ты подбираешь каждому обратившемуся подходящие практики в зависимости от их состояния ума.

## Навыки
### Навык 1: Выявление ядов ума
### Навык 2: Подбор практик для устранения выявленных ядов ума
### Навык 3: Бесконечное принятие
### Навык 3: Бесконечное сострадание

Journey

The journey started in 2018 when I lost my major client and started to burn myself trying to get my business back to appropreate level. As a result in the middle of 2021 despite of good enough financial state I feeled myself so wrong that I regreted that I haven't died from COVID. At this point I understood that I need to do something with it and found a therapist. She helped a bit. But what really helped me is Buddhist practice and lectures by Lobsang Tenpa about Buddhist wisdom.
For now I practice daily and learn from different sources. As far as I'm a Master of Biology (on PhD studies) I'm trying to compare practice with neuroscience and psychology data.

I have a channel in Telegram about it but currently only in Russian.

This Bot is a part of my Bodhisattva contribution to this world. What it really needs is promotion and early adoptors to collect data.

ATM, I'm the only contributor but I want to thank all Sangha and especially Foundation for Contemplative Studies.

DEV Community: Anton Shvein

Supervising What You Don't Own: A Controller Loop in OTP

How Camelot runs agents

Where OTP stops

The row outlives the process

Adopting a run that lost its owner

The poll that never ends

The fix: a decision with no I/O

Why the pure function matters

The order of the steps matters

What I'd tell myself six months ago

Conclusion

OpenTelemetry Collector on Docker Swarm

Idea

Architecture

It ingested its own logs

The duplicate

Making it safer lost data

A fix that was not a fix

The escape hatch

Everything was info

What I learned

$63 in One Night: When Your Elixir Polling Loop Costs Money

A two-minute GitHub poller in Ash and Oban

What actually happened?

Level-triggered, with a price tag

Why none of the guards helped

The fix: compare the review against the last commit

The other bug: an optional call inside a with chain

What I changed in how I work

Conclusion

Lama Bot (དླ་མ་བོཏ།) - helps to coup with depression with the help of Buddhist practice

What I Built

Demo

Your Configuration

Journey

The other bug: an optional call inside a `with` chain