DEV Community: Anis Meziani

[Boost]

Anis Meziani — Wed, 14 Oct 2026 09:53:50 +0000

David Elom GNAGLO

Oct 14

Running ADK agents with MCP tools from a REST API

#python #ai #agents #opensource

3 min read

Amnesia and déjà vu: how our AI agents failed after every restart (and how we fixed it)

Anis Meziani — Tue, 13 Oct 2026 15:14:02 +0000

An agent that crashes is annoying.

An agent that crashes and comes back is where things get dangerous.

When we started running our agents in production, we discovered that a restart could go wrong in two completely opposite ways. Sometimes the agent came back with amnesia: it had forgotten everything, the conversation, the context, what it was in the middle of doing. And sometimes, when we tried to fix that by simply relaunching the work, it came back with déjà vu: it happily redid everything from the start, including the things it had already done in the real world.

Like sending an email. Twice.

This post is about both failures, why the obvious fixes don't work, and what we ended up building. It's the part of an agent nobody puts in a demo, and it's the part that decides whether you can trust it.

Everything below comes from ApowerB, the open-source agent runtime we build at thaink², and every fix links to the real code. If you're new to the project, this overview of the architecture is a good place to start.

TL;DR

Amnesia: by default, our agent framework kept every conversation in RAM. Every restart erased everything. We moved the agent's memory into Postgres.
Déjà vu: "just relaunch it" is a trap. An agent acts on the world through tools. Relaunching blindly repeats those actions.
You can't cleanly "resume from step 3" with an LLM agent. So we stopped asking how do we resume? and started asking is it safe to start again?
Every run is now written down when it starts, with its input. Every tool it calls is recorded. A failed run can be replayed without a second thought only if we know it touched nothing. Otherwise the replay is refused, loudly, with the list of what it already did, until a human decides.

Part 1: Amnesia

Eight restarts a day, zero memory

At some point we ran an internal audit on one of our deployments, and one number jumped out: around eight restarts a day.

That's not unusual. Every deploy restarts the app. Kubernetes moves pods around. A node gets drained. In a healthy cluster, processes come and go all the time.

What was a problem: after each of those restarts, every conversation was gone. No history, no context. A user could be halfway through a task with an agent, and the next message would land on an agent that had never heard of them.

The cause was a default. We build on Google's Agent Development Kit (ADK). Each agent conversation lives in a session: the list of events that happened (what the user said, what the model answered, which tools it called, what came back). That session is the agent's short-term memory. And if you don't configure where sessions live, ADK falls back to an in-memory session service. A Python dict, basically. It works perfectly on your laptop, and it evaporates every time the process restarts.

The tab that killed the workflow

The second kind of amnesia was even sillier.

Our visual workflow builder (you chain agents on a canvas, and each one passes its output to the next) originally ran in the browser. The browser was the orchestrator. Which meant that if you closed the tab in the middle of a long workflow, the workflow just... stopped. Halfway. With no record of where it was.

Runs that left nothing behind

And the third one was the most fundamental.

Only one path in the whole product could retry a failed run: incoming emails, because the webhook queue kept the payload and an attempt counter. Everything else (a workflow launched from the canvas, a scheduled agent, an API call) left nothing behind when it failed. The workflow router literally kept its state in a Python dict that was emptied at the end of the stream.

So when something failed, we couldn't relaunch it, because we didn't know what it had been asked to do. We had traces, logs, token counts. We could see the failure in detail. We just couldn't redo the work.

That was the moment it clicked for us:

What was missing wasn't observability. It was the input.

Fix 1: give the agent a memory that survives restarts

The first fix is one setting, and it's the most important line in this post. We point ADK's session service at Postgres instead of RAM (see it in main.py):

app = get_fast_api_app(
    agents_dir=agents_dir,
    session_service_uri="postgresql+asyncpg://...",   # sessions in Postgres
    memory_service_uri=MEMORY_SERVICE_URI,             # long-term memory too
    session_db_kwargs={
        "pool_size": 3, "max_overflow": 2,              # 5 connections per worker
        "pool_recycle": 300, "pool_pre_ping": True,
    },
)

Now every session and every event is a row in the database. A pod can restart ten times a day; the conversation is still there when it comes back. And as a bonus, every conversation is now auditable, which matters a lot once real businesses run on it.

We did the same for long-term memory (what an agent remembers across conversations). The default there was also in RAM, per worker, and never fed. It now lives in the application database, scoped by agent and by user, so one user's memories never leak into another's.

Two small details that only show up in production:

Mind the connection pool. Out of the box, the session engine would open up to 15 connections per worker (SQLAlchemy's default 5 plus 10 overflow). On a database shared with other services, that adds up fast. We capped it at 5.
Memory grows forever if you let it. ADK's database session service has no retention policy: the events table gets one row per agent turn, each holding the full event as JSON. Left alone, it slowly eats the database. A small background task now deletes events older than 90 days, once a day. Sessions keep their shell; they just lose old turn details.

Fix 2: take the orchestrator out of the browser

The workflow engine moved to the server, built on ADK's own Workflow primitives (chaining, fan-out, join) instead of a home-made scheduler. Every step emits step_start, step_complete or step_error, and the run's outcome is written to the database in a finally block.

To be precise about what changed: closing the tab still stops the run. That's deliberate. A client that disconnects cancels its run, and the cancellation is propagated to everything the workflow started (sub-workflows, retry waits), so nothing keeps running unsupervised in the background. What changed is that the run no longer vanishes. It's recorded as cancelled, with its input, and a cancelled run can be replayed. If it's safe, that is. Hold that thought until Part 2.

Fix 3: write every run down before it starts

This is the one that made everything else possible. Every run (a workflow, a scheduled agent, a chat turn) now gets one row, written the moment it starts. Incoming webhooks keep their own durable queue, with its own idempotency; the two will merge the day the execution engine is unified.

Here's what that row holds (run_store.py):

Table("agent_runs", metadata,
    Column("run_id", String, primary_key=True),
    Column("trigger", String),        # workflow | schedule | chat | webhook
    Column("owner_id", String),
    Column("config", String),         # the input. No input, no replay.
    Column("input_file_path", String),
    Column("status", String),         # running | success | error | cancelled
    Column("attempts", Integer),
    Column("error_message", String),
    Column("replay_of", String),      # the run this one replays
    Column("tools_executed", String), # more on this one below
)

If the run came with a file, we copy that file to disk right away. That one came from experience: the email path already kept attachments, precisely so we could replay an email that had since been deleted from the mailbox. If you only keep a pointer to the user's file, the day it disappears, your replay is impossible.

Fix 4: memory on a diet

Persisting memory solved amnesia. It also created a new problem: an agent that remembers everything can choke on it.

ADK sends the full conversation history to the model on every turn. Now imagine a tool that converts a PDF into images so the model can read it. Each page is a base64 blob worth roughly 100k to 150k tokens. After five turns, the model has been sent the same image five times. You're paying, five times, for something it already read.

So before every model call, a callback strips large base64 payloads from every message except the latest one:

def before_model_callback(*, callback_context, llm_request):
    contents = llm_request.contents or []
    for content in contents[:-1]:            # never touch the latest turn
        for part in content.parts or []:
            fr = getattr(part, "function_response", None)
            if fr and fr.response:
                _strip_payload(fr.response)  # big base64 -> "<stripped: N chars>"
    return None

The model already looked at the image when it arrived. From the next turn on, its own text about the image is the source of truth. It still sees that the tool ran and what it produced, just not the raw bytes again.

Amnesia: solved. Our agents now remember. Which brings us to the second, sneakier problem.

Part 2: Déjà vu

The tempting fix

Once runs are written down with their input, the obvious next idea is: when a run fails, just relaunch it. Same input, same agent, go again. Fully automatic. Problem solved, right?

Picture an agent whose job is to process an incoming customer request. It reads the message, drafts a reply, sends the email, then updates the CRM. Halfway through the CRM update, the pod gets evicted.

Relaunch it from the start, and it reads the message, drafts a reply, and sends the email again. The customer now has two answers. Maybe two slightly different answers, because the model doesn't write the same thing twice.

That's the core of the problem. An agent isn't a pure function. It acts on the world through its tools: it sends emails, writes to databases, calls APIs. Those are side effects. And a blind relaunch repeats every one of them.

Why you can't just "resume from step 3"

The natural reaction is: fine, then don't restart from the beginning, resume from where it stopped. Skip the steps that are already done.

With a classic workflow engine, that works, because steps are fixed and deterministic. Step 3 is always step 3.

With an LLM agent, there is no step 3. The "plan" only exists inside that particular run, in the model's reasoning at that moment. Run it again and the model may take a different path, call the tools in a different order, or decide it needs a tool it didn't use the first time. There's no clean way to replay an agent while skipping the calls it already made, because you can't replay the reasoning that led to them.

So we stopped trying to answer "how do we resume?" and asked a different question:

How do we know whether it's safe to start again?

To answer that, you need exactly one piece of information: did this run already touch the world?

Fix 5: record every tool call (the call, not the answer)

Every agent run now records the list of tools it called. We read it straight from the session events (run_main.py):

def executed_tools(events) -> list[str]:
    """Tools called in a list of ADK events, in order.

    The call (functionCall) is what counts, not the response:
    a tool that crashes midway may have acted without ever answering.
    """
    names = []
    for event in events:
        for part in (event.get("content") or {}).get("parts") or []:
            call = part.get("functionCall") or part.get("function_call")
            if call and call.get("name"):
                names.append(call["name"])
    return names

That docstring hides the most important detail in this whole post. We count the call, not the response. If the email tool sends the email and then times out before answering, there's no response in the history. But the email is gone. If you only counted tools that answered, you'd conclude "nothing happened" and send it again.

Fix 6: the memory is what makes this possible after a crash

Here's where the two halves of this post connect.

When a run fails with an exception, its response is lost. We don't get the nice list of events back. But the session is in Postgres now. It kept everything that happened before the crash. So after a failure, we go read the session itself and extract the tool calls from it.

One subtlety: a chat session contains the whole conversation, including tools called in previous turns. Those aren't side effects of this run. So we only look at what happened after the user's last message.

Without Fix 1, this is impossible. Amnesia doesn't just lose the conversation, it destroys the evidence you need to decide whether relaunching is safe.

Fix 7: "unknown" is not "none"

The tools_executed column has three possible states, and the difference between two of them is the whole safety model:

[]: we checked, and no tool was called. Safe to relaunch.
["send_email", "update_crm"]: we checked, and these were called. Relaunching repeats them.
NULL: we couldn't check (the session was unreadable, for instance). We don't know.

It's very tempting to treat "unknown" as "probably nothing". We treat it as "probably something", because the cost of being wrong isn't symmetric. Refusing a safe relaunch costs a click. Approving an unsafe one costs a duplicate email to a customer.

Fix 8: the replay rules

With all of that in place, every replay goes through a single endpoint, POST /runs/{run_id}/replay, and the rules for replaying an agent run (a scheduled agent or a chat turn) fit in a table:

The original run...	What happens when you ask to replay it
succeeded	Refused. Its side effects already happened.
is still running	Refused. Wait for it to settle.
failed, and we know no tool was called	Replayed. Nothing to repeat.
failed after calling tools	Refused, with the list of tools it called.
failed, and we can't tell	Refused. Unknown is not none.

And the refusals aren't silent. They're an HTTP 409 with a message a human can act on:

def _refuse_if_side_effects(tools_executed: str | None) -> None:
    if tools_executed is None:
        raise HTTPException(409,
            "It is unknown whether this run called any tool before failing, "
            "so a replay could repeat a side effect. "
            "Pass force=true to run it again anyway.")
    tools = json.loads(tools_executed or "[]")
    if tools:
        raise HTTPException(409,
            f"This run already called {', '.join(tools)} before failing: "
            "a replay would call them again. "
            "Pass force=true to run it again anyway.")

force=true exists, on purpose. Sometimes repeating the action is fine, or even what you want. But that's a decision for a human who has read the message, not for an automatic retry loop at 3 a.m.

Three more rules that matter:

A replay is a new run that cites the original (replay_of). We never overwrite history. If the replay fails too, you can tell it apart from the original failure.
A replay starts in a fresh session. The original session may contain a half-finished exchange, and for chat it's the user's own conversation, where a replay has no business writing.
We never replay "almost". If the input file has disappeared from disk, the replay is refused (410 Gone). Relaunching without it would produce a run that silently differs from the one you think you're reproducing.

Fix 9: runs that die without saying goodbye

One last trap. A run marks itself as finished in a finally block. But when Kubernetes kills a pod, the finally never runs. The row stays running. Forever. And a run that's "still running" can never be replayed.

So whenever we read a run, we reconcile it: if it's been running for more than six hours with no completion recorded, we mark it error with an explicit message, "no completion was recorded within 6h; its actual outcome is unknown":

conn.execute(
    run_table.update()
    .where(run_table.c.run_id == run_id,
           run_table.c.status == "running")   # only if still running
    .values(status="error", error_message=message, finished_at=now())
)

The WHERE status = 'running' makes it safe even when several replicas reconcile the same run at the same time: only the first write lands. Six hours is deliberately generous, so we never close a genuinely long run by mistake. And once reconciled, the run goes through exactly the same safety rules as any other failure. If it had already sent that email, it won't send it again.

What does get retried automatically

None of this means nothing is automatic. Things are retried silently when, and only when, it's provably harmless:

Empty model turns. Sometimes the model returns an empty turn (it happens, especially with "thinking" models). By definition an empty turn contains no tool call, so asking again can't repeat a side effect. We retry it once before falling back.
Event-triggered runs. Incoming webhooks go through a durable queue: each event is processed once, a crashed event is picked up again by another worker, and retries back off and stop after a ceiling, so nothing spins forever.

One honest footnote on the empty-turn retry: ADK appends the user's message to the session on every request, so a retry leaves that message twice in the history. It's rare and harmless today, but it's on our watch list for very long sessions.

The mental model

If you remember one thing from this post, make it these three questions. Ask them before your agent relaunches anything:

Do I remember? Is the agent's memory (sessions, events, inputs) stored somewhere that survives a restart?
Do I know what it already did? Do I record every tool call, not just every tool answer?
Is doing it again harmless? And if I can't tell, do I refuse, or do I gamble?

Amnesia is a "no" to the first question. Déjà vu is skipping the other two.

What this doesn't solve (yet)

What we have today is safe replay: a failed run is either relaunched because we know it's harmless, or refused with an explanation. What we don't have yet is true resume: picking up a run mid-flight, reusing the results of tools that already ran instead of calling them again.

Getting there probably means making tools themselves idempotent (an idempotency key on every side-effecting call, so "send this email" twice is still one email) and caching tool results per run. That's the next frontier, and it's a hard one. If you've built it cleanly, I'd genuinely love to read about it.

Is this overkill?

It's mostly one table (agent_runs), one setting (session_service_uri), one callback, and a handful of rules. No new infrastructure.

And it's what lets our agents run in production for months, across industrial use cases, on Kubernetes with our Helm chart and fully on-prem with Docker Compose, restarting several times a day without anyone noticing. Which is exactly how it should be.

Try it yourself

The whole stack, database included, comes up with three commands from the apowerb-hosting repository:

git clone https://github.com/apowerb/apowerb-hosting.git && cd apowerb-hosting
cp .env.example .env && ./scripts/generate-secrets.sh
docker compose -f docker-compose/docker-compose.yml --env-file .env up -d

The interface is on http://localhost:3000 and the API on port 8000. Add a model key in the interface and you can start breaking agents on purpose: kill the container mid-run, restart it, and watch the session come back.

Going to Kubernetes? The same repository holds the Helm chart (also on Artifact Hub) and plain Kubernetes manifests. The Docker Compose guide and the Kubernetes guide walk through both, and I wrote up the honest version of the Helm install, traps included.

Before you let your agent relaunch itself, check this

[ ] Are sessions stored somewhere other than RAM?
[ ] Is long-term memory scoped per agent and per user?
[ ] Is every run written down, with its input, before it starts?
[ ] Are input files copied, not just referenced?
[ ] Do you record tool calls, including those that never answered?
[ ] Do you treat "unknown" as "unsafe"?
[ ] Is a replay a new run that cites the original, in a fresh session?
[ ] Do runs killed mid-flight get reconciled instead of staying "running" forever?
[ ] Are old events purged, and old payloads stripped from the context?

What I'd tell myself before we shipped

Defaults are written for demos. In-memory sessions are perfect until your first restart. Check every default that stores state.
Never trust "nothing happened" unless you checked. Unknown is not none.
A refusal with a clear message beats a clever automatic retry. Your users will forgive a button that says "this already sent an email, are you sure?". They won't forgive the second email.

Come break it with us

All of this lives in ApowerB, our open-source agent runtime (Apache 2.0). The replay logic is in run_main.py, the run table in run_store.py, and the memory diet in history_compaction.py. The comments explain the scars; go read them, and tell us where we got it wrong.

Issues and PRs are welcome. If you want a first contribution in this area, open an issue or pick an existing one, and I'm happy to pair on your first PR. And if this post was useful, a star on the repo helps us a lot.

Source code: github.com/apowerb/apowerb
Self-hosting (Docker Compose, Helm chart, Kubernetes manifests): github.com/apowerb/apowerb-hosting
Helm chart on Artifact Hub: artifacthub.io/packages/helm/apowerb/apowerb-chart
Docker image: hub.docker.com/r/apowerb/apowerb
Documentation: docs.apowerb.com
Product page: thaink2.com/apowerb
See an agent run end to end (75s): youtu.be/L3KoK0gqekY

Now I'm curious: what does your agent do when it crashes halfway through a task?
Start over, resume, give up? Tell me in the comments, I read every one.

Deploying an open-source AI agent platform to Kubernetes: the honest one-command version

Anis Meziani — Mon, 05 Oct 2026 14:13:00 +0000

Every "deploy X to Kubernetes in one command" article is technically true and practically a lie. The helm install really is one line. What the post skips is the empty cluster with no storage class, no ingress controller, no cert-manager, and no DNS. That 90% is what stands between you and a working URL. So here's the honest version for ApowerB, the open-source agent platform: the chart genuinely installs in one command, and here is everything around it, traps included.

What the chart actually deploys

ApowerB ships an official Helm chart (apowerb-chart, OCI, cosign-signed) that stands up the whole stack, not just the app:

Backend and Next.js frontend: the agent runtime and its UI
PostgreSQL: where your agents live (they're rows, materialized into executable modules at load time)
th2etl: pipeline and orchestration, with a seed job
th2pulse: log and trace storage
otel-collector: telemetry routing
th2forecast: optional forecasting engine (disabled by default)
A PersistentVolumeClaim for uploads, RAG artifacts and agent pools

That's a platform, not a container. Which is exactly why the cluster prerequisites below matter more than for a toy app.

The fast path (local or dev cluster)

If you already have a cluster with a working storage class, two commands get you running.

1. Generate the secrets. The chart refuses to start without them, on purpose:

umask 077
cat > values-secrets.yaml <<YAML
backend:
  env:
    encryptKey: "$(openssl rand -base64 32)"
th2etl:
  apiKey: "$(openssl rand -hex 32)"
th2pulse:
  ingestToken: "$(openssl rand -hex 32)"
  queryToken: "$(openssl rand -hex 32)"
postgres:
  password: "$(openssl rand -hex 16)"
YAML
chmod 600 values-secrets.yaml

2. Install:

helm upgrade --install apowerb \
  oci://registry-1.docker.io/apowerb/apowerb-chart --version 0.4.23 \
  --namespace apowerb --create-namespace \
  --values values-secrets.yaml --timeout 10m

Then port-forward and open the UI:

kubectl -n apowerb port-forward svc/apowerb-frontend 13000:3000 &
# http://localhost:13000

Add a model key in the UI and you have a working agent platform on your cluster. That's the "one command" part, and it's real.

The one rule that bites everyone: pass --values values-secrets.yaml on every helm upgrade. Drop it and you rotate your own credentials out from under a running database. Keep that file safe and version it in your secret store, not in Git.

The honest production path (empty cluster to HTTPS URL)

A real cluster rarely has everything wired. Here's the full walk, which is where most tutorials go quiet.

Prerequisites

Kubernetes 1.28+. Three nodes at 4 vCPU / 8 GB give comfortable headroom. Check the ground first:

kubectl get nodes          # all Ready
kubectl get storageclass   # must not be empty
kubectl get ingressclass   # must exist

Step 1: Storage that actually binds

Lean managed clusters often ship with no default storage class. If get storageclass is empty, install one:

curl -sL -o local-path-storage.yaml \
  https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl apply -f local-path-storage.yaml
kubectl -n local-path-storage rollout status deploy/local-path-provisioner --timeout=120s

kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Trap: most storage classes are WaitForFirstConsumer, so a lone PVC stays Pending and looks broken when it's fine. Test with a PVC and a pod together:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: probe-pvc}
spec:
  accessModes: ["ReadWriteOnce"]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: probe-pod}
spec:
  restartPolicy: Never
  containers:
    - name: writer
      image: busybox:1.37
      command: ["sh","-c","echo ok > /data/marker && cat /data/marker && sleep 20"]
      volumeMounts: [{name: data, mountPath: /data}]
  volumes:
    - {name: data, persistentVolumeClaim: {claimName: probe-pvc}}
YAML
sleep 25 && kubectl get pvc probe-pvc && kubectl logs probe-pod
kubectl delete pvc probe-pvc pod probe-pod

Want Bound and ok in the logs. If this fails, stop. Nothing else will work.

Step 2: Ingress

kubectl get ingressclass
kubectl get svc -A | grep LoadBalancer

Note the class name (Traefik and ingress-nginx are the usual managed add-ons) and the EXTERNAL-IP for DNS.

Step 3: cert-manager and a staging issuer

Let's Encrypt rate-limits at 5 failures/hour, so always start on staging:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik   # your actual class from Step 2

Trap that costs an afternoon: use ingressClassName, not the deprecated class annotation. Otherwise the ACME challenge Ingress gets no recognised class and the certificate sits at Ready: False forever.

Step 4: Deploy with ingress and TLS

Same install command, now with public access turned on:

helm upgrade --install apowerb \
  oci://registry-1.docker.io/apowerb/apowerb-chart --version 0.4.23 \
  --namespace apowerb --values values-secrets.yaml \
  --set ingress.enabled=true \
  --set ingress.className=traefik \
  --set ingress.host=apowerb.example.com \
  --set ingress.tlsEnabled=true \
  --set ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt-staging

Create the first admin while you're at it:

  --set superadmin.email=admin@example.com \
  --set superadmin.password='<a strong one>'

Step 5: DNS and verify

Point an A record at the EXTERNAL-IP, then:

kubectl -n apowerb get certificate
kubectl -n apowerb describe certificate apowerb-tls

Stuck at Ready: False? It's almost always one of: DNS not resolving to the load balancer, port 80 closed (HTTP-01 needs it), the class vs ingressClassName trap, or a staging rate limit. Once it's True, swap the issuer to production (https://acme-v02.api.letsencrypt.org/directory) and reapply.

Configuring the platform

The same values mechanism covers the rest:

Default LLM: defaultLlm.model and defaultLlm.apiKey for agents without their own key (LiteLLM under the hood, so OpenAI / Anthropic / Mistral / Gemini / local all work)
Integrations: GOOGLE_INTEGRATION_* (Drive, Gmail, Calendar), MICROSOFT_INTEGRATION_* (Outlook, webhooks, mail), SMTP_*
Storage: storage.mode and S3_* to push bi_store, uploads, artifacts_store and agents_pool to object storage instead of the PVC

Verdict

The install itself is genuinely one command against an OCI chart, and the chart is honest: it won't boot without real secrets, and it bundles the orchestration, log store and telemetry rather than leaving them as "exercises for the reader." The work that remains isn't ApowerB's, it's the cluster's: storage, ingress, cert-manager, DNS. And that's true of anything serious you deploy.

If you want to see the easy version first, the Docker Compose stack comes up locally in a few commands. But if you're putting agents in front of real users, the Helm chart is the path, and now you've seen all of it, traps included.

Your agent framework stops at the demo. This open-source stack ships to production.

Anis Meziani — Thu, 24 Sep 2026 12:47:05 +0000

Building an AI agent is easy. Building the system around it is where things get complicated.

You can create an agent in a few lines of Python. Then you need an API, persistence, authentication, tool integrations, RAG, scheduling, observability, a UI, deployment, and a way to operate the whole thing. That's the gap I wanted to explore when I started digging into apowerb, an open-source AI agent framework from thaink².

Rather than treating it as another agent SDK, I wanted to answer a more specific question:

What does an AI agent runtime actually need to provide once the prototype becomes a real system?

This article walks through apowerb's architecture, its runtime model, RAG engine, Text-to-SQL support, event-driven execution, deployment model, and how it compares with other popular approaches.

apowerb / apowerb

The open-source agentic framework to build, orchestrate, and operate production AI agents.

Build, run and govern AI agents that work while you don't.

The open-source agentic framework to build, orchestrate, and operate production AI agents.

Documentation • Quickstart • API Reference • Deployment • thaink2

apowerb is an open-source platform to build, run and govern AI agents in production. It covers multi-agent orchestration, RAG and Text-to-SQL, MCP servers and built-in business tools, email- and schedule-triggered runs, token quotas and session audit. You can self-host it with Docker Compose or Helm. The core is Apache-2.0; a few enterprise modules (billing, identity-provider sign-in, MFA agent evaluation, the supervision screen) are sold separately by thaink².

Build agents from the web interface or the REST API — instructions, model tools, knowledge bases — and compose them: sequential, parallel, loop, sub-agents.
Connect them to your systems: any MCP server, plus built-in tools for Google Workspace, Microsoft 365, GitHub, HubSpot, Pipedrive, Odoo, SQL databases Oracle, MongoDB, S3,…

View on GitHub

What is apowerb, exactly?

Most agent frameworks start from a library. You import the framework, define an agent, give it tools, connect an LLM, and build your application. That model is useful, but it leaves a lot of infrastructure outside the framework: API layer, persistence, authentication, UI, deployment, scheduling, event handling, observability, RAG, integrations.

apowerb takes a different approach. Instead of treating the agent as an object embedded inside an application, it treats the agent as something that can be defined, stored, materialized and operated by a runtime. The stack:

FastAPI : REST APIs and Server-Sent Events
Typer : the CLI (apowerb serve, apowerb agents list, …)
Next.js : the web UI (build agents, give them tools/knowledge, run and inspect them)
PostgreSQL : agent and configuration state
Google ADK : the execution engine
LiteLLM : the model routing layer
Docker Compose / Helm : deployment

The result is closer to an application runtime than to a standalone Python library.

The architecture in one picture

┌────────────────────────────────────────────────────────┐
│                     Clients / UI                        │
│    Next.js UI  •  Typer CLI  •  REST API  •  SSE         │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│                 FastAPI Orchestrator                    │
│   Agent materialization   •   Webhooks (Pub/Sub)        │
│   Cron / event scheduling •   SSE output stream         │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│              Execution Engine Google ADK              │
│   Base • Sequential • Parallel • Loop                   │
│   Hierarchical sub-agents  •  Human-in-the-loop         │
└─────────────┬───────────────────────────┬──────────────┘
              │                           │
              ▼                           ▼
┌─────────────────────────┐    ┌─────────────────────────┐
│   LiteLLM Model Router  │    │     Tools & Services     │
│ OpenAI • Anthropic      │    │ M365 • Google • SQL      │
│ Gemini • Mistral • local│    │ RAG • Webhooks           │
└─────────────────────────┘    └─────────────────────────┘

Four orchestration patterns are built into the runtime:

base : single-turn conversational agents with tool execution
sequential : deterministic step-by-step pipelines passing context downstream
parallel : concurrent branches aggregated for synthesis
loop : evaluator/optimizer cycles that iterate until a target condition or stop token

Hierarchical sub-agents let a coordinator delegate work to specialized agents. The interesting part, though, isn't the list of patterns, it's what happens before execution.

The interesting choice: agents as runtime objects

In a typical application, an agent is defined directly in Python and the application owns that definition:

agent = SomeAgent(
    model="...",
    tools=[...],
    instructions="...",
)

apowerb takes another route. Agent definitions, tools, skills and their configuration live in PostgreSQL. At startup, the backend uses those definitions to materialize executable Python modules.

Agent definition ─► PostgreSQL ─► Materialization ─► Python module ─► Google ADK
                                                                        ├─► Tools
                                                                        ├─► RAG
                                                                        └─► LLM

That changes the operational model. The application isn't rebuilt every time the agent catalogue changes; the runtime loads definitions from the database and turns them into executable agents. For an organisation managing many agents, that's a meaningful distinction, the agent becomes closer to a runtime-managed resource than a static piece of application code. It's the detail I found most interesting in the project.

Model access through LiteLLM

The execution layer is deliberately separated from model access. apowerb uses LiteLLM as the routing layer, so the runtime can work with multiple providers without coupling an agent's tools to one vendor. The documented stack includes Anthropic, OpenAI, Mistral, Gemini, OVHcloud, Groq, and local Ollama/vLLM deployments.

Agent ─► Google ADK ─► LiteLLM ─┬─► OpenAI
                                ├─► Anthropic
                                └─► Local (Ollama / vLLM)

The tools and orchestration logic don't need to know which provider ultimately executes the model call, useful for teams that want to swap or benchmark models.

RAG is part of the stack

RAG isn't treated as an external SaaS dependency. apowerb includes th2rag, a standalone FastAPI service for ingestion and retrieval:

Document ─► Conversion (Docling) ─► Chunking ─► Embeddings ─► LanceDB ─► Retrieval ─► Generation

The current implementation uses Docling for document conversion, its HybridChunker for semantic chunking, sentence-transformers/gtr-t5-large for embeddings, LanceDB as the vector store, a LanceDB retriever for similarity search, and Mistral AI for generation. Each stage sits behind a base class (BaseEmbedding, BaseGenerator, …), so swapping the embedding model or the LLM is a matter of subclassing, not surgery.

The service can index local files, URLs, database query results and S3 objects, into knowledge bases associated with individual agents rather than one global index. That makes RAG a runtime capability instead of something each application assembles separately.

Native Text-to-SQL

An agent can connect to PostgreSQL or MySQL, introspect the schema, generate SQL from a natural-language question, execute it and return the result:

"Which customers generated the most revenue last quarter?"
        │
        ▼
   Agent / LLM ─► Schema context ─► SQL generation ─► Query ─► Results ─► NL response

The interesting part isn't that an LLM can write SQL, it's that the connection, schema introspection, generation and execution all happen inside the agent runtime. Text-to-SQL becomes another runtime capability rather than an integration every application builds from scratch.

Event-driven agents

Most agent demos are request/response:

HTTP request ─► Agent ─► Response

apowerb can also work in the opposite direction:

External event ─► Webhook ─► Agent ─► Tools / RAG / LLM ─► Action / Result

The current implementation includes email-triggered agents. Gmail uses Google Cloud Pub/Sub push notifications; Outlook uses Microsoft Graph change subscriptions with clientState validation.

Email arrives ─► Provider notification ─► apowerb webhook ─► Fetch message ─► Run agent ─► Log result

Subscription renewal is handled in the background: Gmail watches expire after seven days, Outlook after three, and the runtime periodically renews anything approaching expiration. Combined with cron-style scheduled runs and SSE streaming (token-by-token output, RAG ingestion progress, notifications), this moves the runtime beyond chatbots, the agent becomes a component in an event-driven system.

The open-source boundary

The core stack is Apache 2.0 ; the framework plus supporting services like the RAG engine, ETL components, observability and web UI. The project follows an open-core model: some commercial extensions (billing, MFA, identity-provider SSO, agent evaluation, supervision, organisation management) are maintained separately.

A TH2_EXTENSIONS environment variable declares which commercial bricks to load; if a brick isn't installed or enabled, the core doesn't depend on it. Routes belonging to an absent brick return a clean 404. The useful distinction: the open-source runtime remains fully usable without the commercial extensions, the core isn't an artificially limited demo.

It's more than one repository

A common pattern in AI tooling is "open-source framework + several proprietary services around it." Here, much more of the surrounding infrastructure is exposed. The thaink² GitHub org holds 10 repositories, all Apache 2.0:

Repository	Role	Technology
`apowerb`	FastAPI backend, CLI and ADK runtime	Python
`apowerb-ui`	Visual agent builder and monitoring	Next.js
`th2rag`	RAG, parsing, embeddings, vector search	LanceDB / Docling / Mistral
`th2etl`	ETL pipelines	Python
`th2pulse`	Observability and logs	OpenTelemetry
`th2forecast`	Time-series forecasting	R
`apowerb-hosting`	Docker Compose / Kubernetes / Helm	—
`apowerb-docs`	Documentation	Mintlify / MDX
`agent-hub`	Community agent templates	—

You can read and run the RAG engine, the scheduler and the telemetry pipeline, there's no hidden proprietary service doing the heavy lifting behind an API.

Run the full stack

The fastest way to understand the project is to run it. The hosting repository provides a Docker Compose setup:

git clone https://github.com/apowerb/apowerb-hosting.git
cd apowerb-hosting

cp .env.example .env
./scripts/generate-secrets.sh

docker compose \
  -f docker-compose/docker-compose.yml \
  --env-file .env \
  up -d

The UI comes up on http://localhost:3000, the API on :8000, and PostgreSQL runs inside the stack. You still provide an LLM API key, through the UI, or via DEFAULT_LLM_MODEL / DEFAULT_LLM_API_KEY. If you only want the Python package:

uv add apowerb   # or: pip install apowerb

Two entry points, two purposes:

apowerb package   ─► embed the runtime in your application
apowerb-hosting   ─► run the complete platform

How it compares

Comparisons between agent frameworks are easy to get wrong because they solve different problems. The useful question isn't "which framework is best?" but "which layer does each project expect you to own?" Below, "built-in" means the capability is part of the platform rather than something you assemble around it.

Capability	apowerb	LangGraph	CrewAI	n8n + LLM
Multi-model	LiteLLM	Yes	Yes	Partial
Built-in RAG	th2rag	Yes	Partial	Partial
Email webhooks	Gmail + Outlook	No	No	Yes
Native Text-to-SQL	Yes	Via tools	No	Partial
Included UI	Next.js	No	No	Yes
Helm deployment	Yes	DIY	DIY	Yes
Main approach	Runtime / platform	Framework	Framework	Workflow automation

The distinction isn't one checkbox, it's the architecture. LangGraph and CrewAI are frameworks you embed into an application. n8n comes from workflow automation, where an LLM is one node in a larger graph. apowerb provides the runtime around the agent: API, persistence, UI, integrations, scheduling, RAG and deployment. Different design choices, not just different implementations of the same product.

What I found interesting

1. Agents are treated as managed runtime resources. The database-backed definition and materialization model is the most distinctive architectural choice.

2. RAG isn't an external dependency. The project ships its own RAG service instead of requiring every deployment to assemble one.

3. Event-driven execution is first-class. Gmail, Outlook, schedules and SSE make "proactive" agents implementable, not just request/response chatbots.

Trade-offs

Not everything is settled. The release cadence is high, so the API isn't frozen, expect to track changes if you build on it now. Schema is created by ensure_* functions at startup rather than Alembic, which is fine today but means no reversible migrations. And the external community is still small; most contribution looks internal so far, so you'd be an early adopter.

None of that is a dealbreaker, it's the normal shape of a young project that's visibly running in production.

Worth a clone if…

you want RAG, tool auth, an API and a UI to already exist so you can build the actual agent;
you're a data team and Text-to-SQL over your warehouse plus scheduled runs maps to your use case;
you need everything self-hosted and auditable, your models, your data, Apache 2.0;
you're benchmarking agent frameworks and want to see where the "runtime, not library" line falls.

Three experiments that take minutes each: point an agent at a Postgres and query it in English; drop a folder of PDFs into a knowledge base and ask questions; set a Gmail trigger that summarizes and routes incoming mail.

DEV Community: Anis Meziani

[Boost]

Running ADK agents with MCP tools from a REST API

Amnesia and déjà vu: how our AI agents failed after every restart (and how we fixed it)

TL;DR

Part 1: Amnesia

Eight restarts a day, zero memory

The tab that killed the workflow

Runs that left nothing behind

Fix 1: give the agent a memory that survives restarts

Fix 2: take the orchestrator out of the browser

Fix 3: write every run down before it starts

Fix 4: memory on a diet

Part 2: Déjà vu

The tempting fix

Why you can't just "resume from step 3"

Fix 5: record every tool call (the call, not the answer)

Fix 6: the memory is what makes this possible after a crash

Fix 7: "unknown" is not "none"

Fix 8: the replay rules

Fix 9: runs that die without saying goodbye

What does get retried automatically

The mental model

What this doesn't solve (yet)

Is this overkill?

Try it yourself

Before you let your agent relaunch itself, check this

What I'd tell myself before we shipped

Come break it with us

Deploying an open-source AI agent platform to Kubernetes: the honest one-command version

What the chart actually deploys

The fast path (local or dev cluster)

The honest production path (empty cluster to HTTPS URL)

Prerequisites

Step 1: Storage that actually binds

Step 2: Ingress

Step 3: cert-manager and a staging issuer

Step 4: Deploy with ingress and TLS

Step 5: DNS and verify

Configuring the platform

Verdict

Links

Your agent framework stops at the demo. This open-source stack ships to production.

apowerb / apowerb

The open-source agentic framework to build, orchestrate, and operate production AI agents.

What is apowerb, exactly?

The architecture in one picture

The interesting choice: agents as runtime objects

Model access through LiteLLM

RAG is part of the stack

Native Text-to-SQL

Event-driven agents

The open-source boundary

It's more than one repository

Run the full stack

How it compares

What I found interesting

Trade-offs

Worth a clone if…

Links