Agents in the database, not in the repo: a tour of apowerb

The usual way to define an agent is to write it: a Python file that declares a model, attaches tools, carries the instruction. You commit it, you deploy it. That is a good default — the definition is versioned, reviewable, testable — and it holds right up to the moment a non-engineer needs to change a prompt, or you are running forty agents for twelve teams and every wording fix has become a release.

apowerb takes the other road: an agent is a row in Postgres. The Python that the runtime imports is generated, disposable, and seven lines long — two of which do anything. Changing an agent is an UPDATE, not a deploy.

It's Apache-2.0, built on FastAPI and Google ADK (the Agent Development Kit, Google's agent SDK), and it self-hosts with Docker Compose. This post walks through the parts of the design that are actually interesting — the ones we got wrong at least once first.

Creating an agent in the UI. Name, category, provider — no file, no commit. What this form writes is a database row.

The generated module

When you create an agent, the framework writes a directory under agents_pool/ named after the agent's id, and puts this in it (src/apowerb/core/adk_agent_builder.py):

# agents_pool/agent1/agent.py, read out of a running container

# th2agent modules
from apowerb.core.agent_helpers import to_agent

# declare env variables from stored in agent_model_params
root_agent = to_agent(agent_name = 'agent1')
# Create the agent

That's the whole module, comments and stale package name included. It is a stub: a name ADK can import, pointing at a loader. Everything that makes agent1 what it is — instruction, model, tools, sub-agents, guardrails, output schema — is read from the database at load time by to_agent (src/apowerb/core/agent_helpers/agent_utils.py, abridged):

def to_agent(agent_name: str) -> LlmAgent:
    """Convert the schema to an Agent instance."""
    agent_id = int(agent_name.replace("agent", ""))
    agent_details = get_agent_details(agent_id=agent_id)

    tools_ids = _parse_string_list(agent_details.get("agent_tools"))
    tools_names, tools_funcs = load_agent_tools_functions(
        tools=tools_ids, owner_id=owner_id
    )
    load_mcp_servers(agent_details.get("mcp_servers"), tools_funcs)
    load_agent_skills_toolset(agent_name, agent_details.get("agent_skills"), tools_funcs)
    ...

We checked rather than trusting the code: on a stack booted from the published images, we created one agent named article_demo in the interface, and then looked at both ends of the chain. In Postgres:

 agent_id |  agent_name  | agent_type |             agent_model
----------+--------------+------------+--------------------------------------
        1 | article_demo | base       | anthropic/claude-sonnet-4-5-20250929

And on disk, agents_pool/agent1/agent.py — the stub above, written by the server the moment the row appeared. The model is a string in a column. The agent type is a column. Note also what the form lets you leave empty: the API key. A definition can exist without a credential; it just can't answer until one is there.

The payoff is that the editing surface of an agent is an API and a UI, not a repository. The cost is that the filesystem now holds derived state — and derived state drifts.

Derived state drifts, so boot repairs it

An agents_pool/ directory and a database can disagree. Someone restores a volume without the database. Someone syncs one environment onto another and loses a folder. A package rename leaves a stub that imports a module that no longer exists. All three failures look identical at runtime: a ModuleNotFoundError the first time a user talks to that agent.

So the boot sequence reconciles instead of trusting. The repair function says what it covers:

def ensure_agent_modules(agents_pool_path: str | None = None) -> None:
    """Auto-repair: regenerate missing *or stale* agent.py files for every agent.

    Called at startup, so an environment heals itself on its next restart
    rather than waiting for someone to save each agent by hand.
    """

"Stale" is the interesting half. A present-but-broken file is worse than a missing one, because a naive os.path.exists check calls it healthy. The repair reads the stub and looks for the single import line every generated stub carries; a file that no longer contains it is rewritten like a missing one. Deliberately coarse, and deliberately narrow: a stub somebody has genuinely customised, which still imports the core, is left alone.

The whole startup path lives in one bootstrap() behind the ASGI lifespan, and the docstring says why (it is written in French in the repo; translation mine):

This all used to run at module level: importing apowerb.main triggered 10
migrations against a real database and wrote into agents_pool/. A plain import was
worth a boot — which made the package unusable as a dependency.

If you ship a Python package whose import does migrations, you don't have a library, you have a side effect with a name.

The third state: the agent already in memory

Two states have to agree, and that is what boot repairs. There is a third, and it is the one that makes the opening claim true or false: the agent object a running process has already built.

to_agent reads the database at load time. Load it once and the process keeps it — the module in sys.modules, the agent in the loader's cache, an ADK runner wrapping it. Change the row and nothing moves. A user with a chat open would keep talking to yesterday's instruction until somebody restarted the backend, which is precisely the deploy the design was supposed to remove.

So a save invalidates all three. POST /api/agents/{id}/reload (src/apowerb/routers/agent_reload.py) drops the agent's modules from sys.modules and from the loader cache, then queues its runner for cleanup; the next get_runner_async closes the old runner and rebuilds it, which re-imports the module, which re-runs to_agent against the current row. The frontend calls it right after saving the row. Once that invalidation has gone through, the next message in an already-open conversation gets an agent rebuilt from the new configuration — no "New chat", no restart.

Three honest caveats, all visible in the code. The reload is best-effort at every step: the frontend fires it without awaiting it, the endpoint wraps both invalidations in try/except, and it answers 204 either way. So a save can succeed while the reload fails. That failure is logged — console.warn in the browser, logger.warning on the server — but nothing in the interface raises it: you get the save toast, and the old definition can stay live. Second, because the frontend never awaits it, that toast is not a promise the runtime has caught up; a message sent the instant it appears can reach the server before the invalidation does. Third, the rebuild is deferred to the next runner request rather than done on the spot — and what happens to a run already in flight at that moment is not something we have exercised end to end.

There is also a scar in it worth keeping:

# Only queue the runner for cleanup if one actually exists: ADK's
# close_runners([None]) crashes with ``'NoneType' object has no
# attribute 'close'`` otherwise.

An agent that has never been talked to in this process has no runner, and asking ADK to close it raised an AttributeError on a route whose whole job was to be invisible.

What that costs

The objection lands immediately, and the repo does not answer all of it. A definition that lives in a table is a definition that has left git, so:

• Concurrent edits are last-write-wins. The agent table — th2agents_store, a name older than the project's own — carries created_at and updated_at and no revision column; there is no optimistic-locking token and no If-Match on the update route. Two people saving the same agent is a race, and the loser gets no warning.
• History is a rollback log, not git. Since apowerb#171, every overwrite — an edit or a template resync — first copies the stored row into agent_revisions, in the same transaction. GET /agents/{id}/revisions tells you who changed what, when, and which fields differ; POST /agents/{id}/revisions/{revision_id}/restore puts an older definition back in service without a restart, archiving the current one on the way so a restore can itself be undone. What you don't get is what git gave you: no branches, no review before a change goes live, one linear log per agent.
• Testing an agent means having a database. The definition is a row, so a unit test that wants a real agent wants Postgres. The suite works around it, but that is a cost the file-based approach doesn't have.

Worth saying plainly, because the honest comparison is not "database beats files". It's that you are trading the guarantees git gave you for the ability to change an agent without a deploy. If your agents are written once by engineers and rarely touched, a YAML file and a hot reload is the smaller tool and you should use it.

The same agent on the canvas. The graph is derived from the stored definition, not the other way round.

One column for the model

Agent code never names a provider. The agent_model column does, and build_litellm_model turns it into an ADK LiteLlm instance (src/apowerb/core/agent_helpers/llm_model_builder.py). Anthropic, OpenAI, Mistral, Google, OVHcloud, or anything OpenAI-compatible behind a custom api_base — same agent, different row.

Two details worth stealing:

Custom endpoints get forced onto the OpenAI path. A self-hosted mistral/... model behind an OpenAI-compatible gateway will fail tool calling if litellm routes it through the Mistral SDK, so the provider prefix is rewritten:

if model_api_base:
    _model_name = agent_details["agent_model"]
    # When using a custom OpenAI-compat api_base, force the openai/ provider prefix
    # so litellm uses the OpenAI tool-calling path instead of the Mistral SDK path.
    if not _model_name.startswith("openai/"):
        _model_name = "openai/" + _model_name.split("/", 1)[-1]

A shared model's credentials are overwritten, not merged. When an agent points at the instance's default model, the key and api_base come from the server environment and clobber whatever the agent row carries — an agent cannot redirect the shared model's traffic to an endpoint of its choosing. Credentials stored per agent are encrypted at rest, and the server refuses to boot without a Fernet key rather than silently falling back to plaintext.

Five shapes of agent, one column again

agent_type selects the ADK primitive:

agent_type	What it builds	Use
base	LlmAgent	one agent, its tools
router	LlmAgent + generated routing instruction	dispatch to sub-agents
sequential	SequentialAgent	pipeline: each step reads the previous one's result under the name its output_key gave it
parallel	ParallelAgent	fan out, then join
loop	LoopAgent	iterate until a condition or a bound

The loop is bounded twice: a per-agent loop_max_iterations (default 3) and a HARD_LIMIT = 100 it cannot exceed. An unbounded agent loop is not a feature, it's a bill.

Composition is data too — sub_agents is a column, and a sub-agent is loaded by the same to_agent, so hierarchies are built without a line of user code.

The tool catalogue in a stock instance: 108 tools across 32 categories, plus 8 skills and MCP servers.

Tools: 31 modules, 108 tools, plus MCP, plus skills

The tool store ships 31 modules in src/apowerb/tools_store/portfolio/: Google Workspace (Drive, Gmail, Calendar, Sheets, Docs), Microsoft 365 (Outlook, OneDrive, Teams), generic SQL, text-to-SQL, RAG, S3, HubSpot, charting, web search, Odoo, and so on. A module is a family, not a single function — a clean install of this stack reports 108 available tools in 32 categories, plus 8 reusable skills, before you connect anything. An agent's agent_tools column lists what it may use; OAuth tokens for the integrations are encrypted in the integrations table.

Beyond that, mcp_servers on the agent row instantiates ADK McpToolsets at load time, so any MCP server you run becomes tools for that agent without touching apowerb, and agent_skills attaches reusable prompt+tool bundles.

RAG is a first-class route family rather than a tool you wire yourself: POST /api/rag/index-files, index-url, index-db, index-db-nl, index-s3, with progress on an SSE stream. The URL indexer has SSRF protection (localhost and private ranges refused), which is the kind of thing you only add after someone points a crawler at 169.254.169.254.

Integrations are OAuth apps you declare; the agent gets the credentials, not your code.

The door problem

Here is our favourite bug in this codebase, because the tests were green the whole time.

A run can start from several places: the chat endpoints (/api/adk/run, /run_sse), scheduled runs driven by an external orchestrator, and inbound webhooks (Gmail via Pub/Sub, Outlook via Graph). Each entry point resolved its own guards. Two of them did. The other two had been written later, somewhere else, by someone who didn't know the first two existed.

The fix is a single choke point, src/apowerb/core/run_gate.py, and then a test that makes the choke point stay one — by reading the sources instead of exercising a path:

# tests/test_run_gate_couverture.py — the two names the test looks for
RUNNERS = {"run_adk_agent", "stream_adk_agent"}   # calling either one is "running an agent"
PORTIER = "apply_run_guards"                      # ...and this must be called too

Its docstring, in French like the rest of this codebase's reasoning, says the point better than we can: it re-reads the sources rather than exercising a path, because the door somebody adds in six months is one no behavioural test will have been written for.

The test walks the AST of every module under src/apowerb, collects the names each one calls, and fails when a module that calls a runner never names the gate. Be clear about what that proves: it matches at module scope, so it does not check that the gate runs before the runner, or on every branch — a call sitting in another function of the same file satisfies it. It is a coarse net, and it is aimed at the failure that actually happened: a new door written somewhere else, by somebody who never heard of the gate. That one fails CI on the day it's written rather than the day it's abused. It complements the behavioural tests rather than replacing them: a regression signal for the forgotten door, not proof that every execution path is guarded.

The token cap

run_gate doesn't hard-code its guards: it asks an extension registry for them (src/apowerb/core/extensions/registry.py). The core registers its own guards there, and an extension package adds its own through the same seam — so guards an extension brings leave with it, while the ones the core registers stay whatever you uninstall. The main one is a token cap, wired in src/apowerb/core/usage_wiring.py: every model call is recorded into llm_usage, and a run can be refused before it starts.

What the cap covers is worth stating precisely, because it's a self-hosting decision. Only consumption of the shared default model is bounded — the rows flagged billed_to_thaink2 in llm_usage. Tokens you spend through your own API key are yours to pay for, so capping them would be someone else's policy, not a feature. The cap is a monthly token quota per account, counted over the calendar month in the billing timezone, and it is a setting: a quota of 0 means unlimited.

Be precise about what that guard is, though, because it is not a budget you cannot exceed. It runs before a run starts and never mid-flight — cutting an SSE stream in half would leave a truncated conversation and a half-billed turn, so an overage is a clean 402 refusal at the door rather than a cut-off. It is deliberately fail-open: if the agent can't be resolved or the usage read fails, the run goes through, on the view that losing a cap matters less than making the product mute. And it inspects the agent you called, not the tree below it. agent_uses_default_llm reads the called agent's own agent_model column and returns before the quota is ever read when that column is not the shared model — so a sequential or parallel agent whose sub-agents alone sit on the shared model takes that early return on every run, not merely the first. Its consumption is recorded all the same, because the recorder runs on each sub-agent, and that recorded usage can get a later quota-checked run on the same account refused, provided the quota read succeeds. The container itself is not stopped by this guard, then or later. Worth knowing before you plan a budget around it.

The pipe that was open and empty

One honest note on observability, straight from src/apowerb/configs/observability.py.

ADK exports its GenAI spans over OTLP as soon as OTEL_EXPORTER_OTLP_ENDPOINT is set — model calls, tool calls, durations. Standard Python logging records are not part of that, and nobody noticed, because the collector was plainly reachable:

Measured on 4 September 2026 (translated from the French docstring) with the full stack running and that variable set: th2pulse
held {"count": 0} on /logs after six minutes and several served requests, while a
synthetic OTLP record pushed to the same collector arrived fine. The pipe was open;
nothing was writing to it.

A reachable endpoint is not a working pipeline. The bridge is now explicit, it's an optional extra (apowerb[otel], backed by th2pulse), it's opt-in on the endpoint variable, and every path in it swallows its exception and logs it — observability that takes the application down is worse than no observability.

The dashboard of the instance these screenshots come from — public images, one compose file.

Run it

git clone https://github.com/apowerb/apowerb-hosting.git && cd apowerb-hosting
cp .env.example .env && ./scripts/generate-secrets.sh
docker compose -f docker-compose/docker-compose.yml --env-file .env up -d

UI on http://localhost:3000, API on 8000, Postgres inside the stack, secrets generated for you. Nothing to fill in — except a model key, which is the one thing it can't invent: add yours in the interface, or set DEFAULT_LLM_MODEL and DEFAULT_LLM_API_KEY in .env. Until then everything else works and the model simply doesn't appear in the list.

The same repo carries Kubernetes manifests, a Helm chart and a Traefik overlay. Images are on Docker Hub under the apowerb namespace.

Links

• Core: github.com/apowerb/apowerb — Apache-2.0
• Deployment: apowerb/apowerb-hosting
• Web UI: apowerb/apowerb-ui
• ETL / orchestration: apowerb/th2etl
• Observability: apowerb/th2pulse
• Docs: docs.apowerb.com — quickstart, API reference

Issues and PRs welcome. If you self-host it and something in this post doesn't match what you see, that's a bug in one of the two — tell us which.