DEV Community: Oluwaseyi Ogunsola

Designing agent-broker: Collision-Safe Redis Keys, Budget Integrity, and Zero-Trust Delegation

Oluwaseyi Ogunsola — Wed, 14 Oct 2026 18:47:46 +0000

A permission broker for concurrent AI agents must do more than check whether a request is allowed.

It needs to coordinate shared budgets, enforce concurrency limits, manage reservations, and prevent delegated agents from bypassing the constraints imposed on their parent or root agent.

These responsibilities become difficult when several processes operate concurrently and when external failures leave shared state in unexpected conditions.

While building agent-broker, I focused on several less obvious failure modes: ambiguous composite Redis keys, budget pools that disappear before reservations are resolved, and caller-controlled identity or delegation metadata.

This article examines the design decisions behind those cases.

Composite Redis Keys Need Unambiguous Boundaries

Redis keys are often constructed by concatenating several values with a delimiter.

For example:

"concurrency:{target}:{budgetKey}"

This is convenient until the values themselves contain the delimiter.

Suppose two requests use these inputs:

"target = "a:b"" and "budgetKey = "c""
"target = "a"" and "budgetKey = "b:c""

A naive delimiter-based scheme can serialize both combinations into the same string:

"concurrency🅰️b:c"

The application intended these to represent different logical resources, but Redis sees identical keys.

In a system that tracks concurrency or budget state by key, this ambiguity can cause unrelated operations to share counters or interfere with one another.

Length-prefixing the segments

agent-broker addresses this problem through explicit length-prefixing.

Each caller-supplied string segment is encoded as:

"seg(value) = "${value.length}:${value}""

For example:

"target = "a:b"" becomes "3🅰️b"
"budgetKey = "c"" becomes "1:c"

The resulting composite key is:

"concurrency:3:a:b1:c"

For the second input:

"target = "a"" becomes "1:a"
"budgetKey = "b:c"" becomes "3🅱️c"

The key becomes:

"concurrency:1:a3🅱️c"

The segment boundaries remain distinguishable even when values contain colons.

This is the central property of the approach: serialization preserves the boundaries between components rather than relying on the assumption that their contents will never contain a separator.

The implementation is documented in ADR-0008.

Handling a Missing Budget Pool During Reservation Resolution

A second problem arises when a budget pool disappears while an agent still has an outstanding reservation.

agent-broker uses budget pools that do not have a TTL by design, allowing them to outlive individual agent lifetimes. However, the absence of a TTL does not make a key impossible to delete. An operator, an administrative operation, or an unexpected Redis state change can still remove it.

This matters when resolving a reservation.

Redis's "INCRBY" command creates a missing key with the requested increment. That behavior can be dangerous if the increment is intended to refund previously reserved budget.

If the original pool has disappeared, applying a refund without checking the pool's existence can recreate a budget key whose state no longer represents the original pool.

The resulting value may look valid at the Redis level while being inconsistent with the broker's intended accounting.

Detecting the missing pool atomically

agent-broker's reservation-resolution Lua script checks whether the budget pool exists before processing a refund.

If the pool is missing, the script skips the refund instead of allowing "INCRBY" to recreate the key.

The reservation still resolves its other responsibilities: concurrency slots are released and applicable expiring entries are cleaned up. This avoids leaving unrelated concurrency state stranded merely because the budget pool is missing.

The response explicitly reports:

"poolMissing: true"

That signal gives the caller or observability layer a way to detect the unexpected state and investigate it.

There is an important distinction here: the broker can detect that the pool is missing, but that observation alone does not establish whether it was deleted intentionally, evicted, or removed by another mechanism.

The design prevents a missing pool from being silently recreated through the refund path while keeping the reservation lifecycle recoverable.

This behavior is documented in ADR-0003.

Broker-Issued Identity and Computed Delegation Depth

A permission system loses much of its value if callers can supply the identity or hierarchy metadata used to decide whether their requests are allowed.

Consider a delegation tree with a maximum permitted depth.

If a caller can register a child agent and claim that its depth is zero, the broker may evaluate that child against the wrong position in the hierarchy.

Similarly, accepting caller-supplied parent or root identifiers without validating them against stored relationships can undermine the intended delegation boundaries.

The broker owns identity and hierarchy

agent-broker applies three related rules:

Agent identity UUIDs are issued by the broker during registration rather than accepted from callers.
Delegation depth, parent ID, and root ID are derived by the broker from the referenced parent's stored state.
Callers attempting to supply protected fields such as "depth", "rootId", "budgetKey", or "agentId" during child registration receive a "BrokerArgumentError".

These decisions make the broker responsible for establishing identity and hierarchy instead of trusting clients to describe their own permissions.

The distinction is fundamental: clients can request an operation, but they should not be allowed to redefine the security context in which that operation is evaluated.

This is one part of the broker's trust-boundary design, not a claim that these rules alone constitute a complete security model.

The relevant decisions are recorded in ADR-0012.

The Broader Design Principle

These three problems look unrelated at first:

Composite keys can collapse distinct resources into the same identifier.
A missing budget pool can be recreated by an otherwise ordinary Redis operation.
Caller-controlled hierarchy metadata can undermine permission enforcement.

But each problem has the same underlying structure: the system is relying on an assumption that may not hold when inputs are adversarial or state changes unexpectedly.

The response is to make the assumption explicit and enforce the relevant invariant at the layer that owns it.

For key construction, that means unambiguous serialization.

For reservation resolution, that means checking shared state within the atomic script before applying a refund.

For delegation, that means deriving identity and hierarchy from broker-controlled state rather than trusting caller-provided values.

These mechanisms do not eliminate every possible failure. They make specific failure modes harder to trigger and easier to detect.

Conclusion

agent-broker is a distributed concurrency and resilience library for coordinating agent permissions, budget reservations, and delegation.

Building it highlighted an important lesson: correctness depends on more than the happy path. It also depends on how a system represents identity, serializes resource names, and behaves when shared state no longer matches expectations.

Length-prefixed Redis keys, missing-pool detection during reservation resolution, and broker-controlled delegation metadata are three examples of turning those concerns into explicit architectural rules.

The most valuable design decisions are often the ones that remove ambiguity before it becomes a production incident.

Reengineering Family Finance: Building Financial Integrity Into Kith

Oluwaseyi Ogunsola — Wed, 14 Oct 2026 18:27:49 +0000

Families manage money and responsibilities in ways that often resemble small organizations. They coordinate recurring contributions, fund shared projects, track obligations, resolve disputes, and follow up with members who miss payments.

Yet these activities frequently happen through group chats, spreadsheets, handwritten records, and informal agreements. The problem isn't simply that these tools lack automation. It's that they make it difficult to maintain a reliable financial history, coordinate responsibilities, and establish accountability as a family grows.

Kith was built to address that problem: bringing structured financial records, automated workflows, and accountable collaboration to family finance without imposing the complexity of enterprise software.

Building it required more than creating forms and database endpoints. Financial coordination introduces difficult engineering requirements around concurrent writes, duplicate submissions, historical integrity, scheduled work, and recoverable notifications.

This article explores the architectural decisions behind those requirements.

Financial History Must Remain Auditable

In an ordinary CRUD application, updating a record in place is often sufficient. In a financial system, silently changing a confirmed transaction can undermine the history that other decisions depend on.

Kith uses an append-oriented ledger model. Once an entry is confirmed, corrections are represented through new entries linked to the original rather than by silently replacing the original transaction.

The relationship between an original entry and its correction is cryptographically linked, providing a mechanism for connecting the financial history and detecting inconsistencies according to the implemented verification rules.

Database-level immutability enforcement provides another layer of protection. The database, rather than application conventions alone, prevents confirmed entries from being modified in place through the protected database operations.

This distinction matters because application-level validation cannot protect a record against every other database access path. Integrity rules are stronger when they are enforced at the layer responsible for storing the data.

The objective is straightforward: preserve the original financial event, make corrections explicit, and maintain an auditable history of how the record evolved.

Idempotency and Duplicate-Submission Defense

Financial applications must account for unreliable networks, repeated button clicks, client retries, and requests that succeed even when the client never receives the response.

Consider a contribution request that reaches the server and commits successfully. If the response is lost, the client may retry. Without duplicate protection, the same logical contribution could be recorded twice.

Kith uses client-supplied idempotency keys to associate retries with the original logical operation.

The important part is not merely checking whether a key already exists. Concurrent requests can both pass an application-level check before either inserts its record. That check-then-insert pattern leaves a race condition.

Kith's duplicate-submission handling has been hardened to close that race, so the idempotency mechanism must be evaluated as a complete concurrency-safe operation rather than as a preliminary lookup.

A separate ten-minute heuristic identifies likely accidental duplicate contributions. This heuristic serves a different purpose from idempotency: it can warn users about suspiciously similar submissions without automatically rejecting every legitimate repeated contribution.

The distinction is important:

Idempotency identifies repeated attempts at the same logical operation.
Duplicate heuristics identify separate operations that may represent an accidental repeat.

Keeping these mechanisms separate avoids confusing network retries with legitimate contributions.

Atomic State Transitions With PostgreSQL

Financial coordination involves operations that update several related records. Dispute resolution, invitation acceptance, and converting an event into a recurring pool are examples where partial completion can leave the application in an inconsistent state.

If two requests modify related data concurrently, individually valid operations can still produce an invalid combined result.

Kith addresses these cases using PostgreSQL transactions and atomic RPC functions for critical multi-step writes. Where appropriate, row locking coordinates competing operations so that they cannot independently proceed on stale assumptions about the same protected state.

The objective is to keep related changes within a single transactional boundary.

For example, a critical state transition should not update one record successfully and then leave its related records unchanged because a subsequent operation failed.

Moving these transitions into the database reduces the number of application-level steps that can interleave with competing requests and makes the consistency boundary explicit.

This approach does not eliminate the need to reason about isolation levels, lock ordering, or transaction failures. It provides a mechanism for enforcing the invariants that those operations require.

Proxy Membership: Designing for Families With Different Technical Abilities

Not every family member can or wants to use a mobile application.

A household may need to record contributions or assign responsibilities for a baby, an elderly relative, or someone who does not use digital tools independently.

Kith addresses this through proxy membership.

An authorized administrator can manage participation on behalf of a proxy member while maintaining an explicit relationship between the proxy and the accountable manager.

Actions performed on behalf of a proxy are separately recorded, preserving accountability without requiring every participant to operate an account independently.

This is more than a convenience feature. It introduces a distinct authorization question: who is performing an action, and on whose behalf is that action being performed?

Maintaining that distinction allows the application to support family circumstances without treating every action as if the affected member performed it personally.

Unattended Automation With BullMQ

Recurring financial activities should not depend on an administrator remembering to trigger every operation manually.

Kith uses nine Redis-backed BullMQ queues and scheduled jobs to coordinate recurring pools, reminders, notifications, cleanup tasks, and member-engagement updates.

Recurring pool management

Recurring pools require predictable lifecycle transitions. Kith generates upcoming cycles three months ahead of time and automates scheduled opening and closing.

Where configured, unpaid balances can carry forward into the next cycle as adjusted targets rather than being represented as payments that never occurred.

This preserves the distinction between an outstanding obligation and an actual contribution.

Recoverable notifications

Notification delivery introduces an external dependency. A job can be processed successfully by the application while the downstream email or push provider rejects the delivery or becomes unavailable.

Kith uses an outbox-oriented workflow to record delivery state and a recovery sweep that runs every five minutes to identify and re-enqueue eligible stuck deliveries.

This makes failures observable and creates a recovery path for transient problems instead of silently discarding work.

It does not mean every external delivery is guaranteed to succeed. Permanent provider failures and other terminal errors still require appropriate failure handling.

Scheduled maintenance

Background workers also handle overdue-payment reminders, task-status updates, invitation cleanup, and member-engagement recomputation.

Moving this work out of user-facing requests keeps routine operations from depending on a user remaining online or an administrator manually initiating each step.

Security Beyond Authentication

Financial records require protection at several levels, not just a login screen.

Kith includes authorization and validation mechanisms designed to reduce the risk of information disclosure and unauthorized changes.

Tenant isolation

Workspace membership checks return a uniform not-found response when access cannot be established. This avoids revealing whether a particular resource exists to an unauthorized requester.

File integrity validation

Payment proofs, task evidence, and milestone photographs undergo file-signature validation using actual file contents rather than relying solely on client-declared content types.

This helps detect files whose contents do not match their claimed formats.

Administrative safeguards

The application prevents the last remaining active workspace administrator from being removed or demoted.

Administrative dashboard access is also restricted through IP allowlisting and timing-safe Basic Authentication verification.

These controls address different risks: authorization mistakes, misleading file metadata, accidental loss of administrative access, and unauthorized access to privileged interfaces.

They should be understood as layers of protection rather than as a claim that any single mechanism makes the entire application secure.

Conclusion

Kith began with a familiar problem: families need to coordinate shared finances and responsibilities, but informal tools become difficult to manage as those relationships grow more complex.

Solving that problem required engineering beyond ordinary CRUD operations. Financial history needed explicit integrity rules, concurrent writes needed safe state transitions, repeated requests needed duplicate protection, and background workflows needed observable failure handling and recovery.

The broader lesson is that trustworthy financial software depends on making important invariants explicit and enforcing them at the appropriate layer.

For Kith, the goal is to combine that engineering discipline with an interface and workflow that remain accessible to the people using it: families coordinating money, responsibilities, and trust.

Not All AI Provider Failures Are the Same: Building a Fault-Aware LLM Fallback System

Oluwaseyi Ogunsola — Wed, 14 Oct 2026 18:21:19 +0000

Not All AI Provider Failures Are the Same: Building a Fault-Aware LLM Fallback System

A common approach to multi-provider LLM integration is straightforward: send a request to a provider, catch an error if it fails, and try the next available key or provider.

That works for some failures, but treating every error identically introduces unnecessary problems.

An invalid API key, a provider-wide outage, a deprecated model identifier, and a malformed application request are not the same failure. They have different causes and require different responses.

For example, if a provider returns an HTTP 503 because its service is temporarily unavailable, cooling down a perfectly valid API key does not solve the underlying problem. It simply reduces the key's availability after a failure that was unrelated to its validity.

In StudyHub, I built a multi-provider LLM routing system that classifies provider failures before deciding whether to cool a key, switch providers, invalidate a cached model, or stop the request.

This article explores that design.

Why Failure Classification Matters

A fallback system needs to answer two questions:

What most likely caused the failure?
Which action is appropriate for that failure?

Without classification, the router can waste valid credentials, repeatedly request models that no longer exist, or continue retrying requests that will fail regardless of which provider receives them.

StudyHub represents provider failures through a custom "ProviderCallError" that carries diagnostic information such as status codes, provider identifiers, and parsed error details.

The system then classifies failures into four categories:

"KEY_FAULT"
"PROVIDER_TRANSIENT"
"BAD_MODEL"
"NON_RETRYABLE"

The taxonomy determines how the routing system should respond.

KEY_FAULT: Cool the Affected Key

The "KEY_FAULT" category represents failures associated with the credential or account state.

In the current classification rules, these include HTTP statuses 401, 402, 403, and 429.

Examples include invalid credentials, account restrictions, billing problems, and rate limits associated with the account or key.

When the router classifies a failure this way, it cools the affected key and attempts another eligible key or provider.

The intention is to avoid repeatedly sending requests through a credential that is currently unable to serve them.

However, status codes are signals rather than universal explanations. A 429 response, for example, may represent a key-specific rate limit, an account-level quota, or another provider-specific restriction.

For that reason, the classification should be understood in the context of the provider's API behavior and the error information available to the implementation.

PROVIDER_TRANSIENT: Change Providers Without Penalizing the Key

The "PROVIDER_TRANSIENT" category handles failures such as HTTP 500, 502, 503, and 504, along with qualifying network-level connection failures.

These failures commonly indicate temporary provider or network problems rather than invalid credentials.

When such a failure is classified as transient, StudyHub advances to another provider without unnecessarily cooling the current key.

This distinction helps preserve credential availability during external outages.

It also illustrates why retry behavior should follow the cause of a failure rather than treating every exception as evidence that the current key is unusable.

The exact handling still depends on the provider's response semantics and the router's classification rules.

BAD_MODEL: Invalidate the Model, Not the Credential

Providers change their model catalogues. Models may be renamed, removed, deprecated, or made unavailable to a particular endpoint.

A model identifier that worked yesterday may therefore fail even though the provider and credential remain usable.

StudyHub classifies certain HTTP 400 responses as "BAD_MODEL" when structured error information indicates that the requested model is invalid, missing, or decommissioned.

The classifier examines fields such as:

"error.type"
"error.code"
"error.message"

It then checks for known indicators of an unavailable model.

When a response matches the classification rules, the router evicts the affected model from the shared Redis discovery cache rather than cooling the key.

This prevents other application instances from continuing to select the same cached model identifier after the system has determined that it is no longer usable.

The distinction is important: invalidating a model and invalidating a credential are different operations because they address different underlying resources.

The classification also needs to remain conservative. Not every HTTP 400 means that a model has disappeared; malformed payloads and other client-side errors can produce the same status.

NON_RETRYABLE: Fail Fast When Fallback Is Unlikely to Help

The final category, "NON_RETRYABLE", acts as the conservative default for errors that do not match the known classifications.

The reasoning is that some failures originate in the application's request construction rather than in the provider's availability.

If the application sends a malformed payload, rotating through several providers may simply repeat the same mistake.

In those cases, failing fast avoids wasting requests and obscuring the original error behind a chain of unsuccessful fallback attempts.

This policy depends on the classifier being accurate. An unfamiliar provider error may be misclassified as non-retryable even when another provider could have handled the request.

Unknown errors should therefore remain observable, and the taxonomy should evolve as new provider response patterns are understood.

Normalizing Python Request Exceptions

A practical implementation challenge is that Python's "requests" library exposes failures through different exception types.

An HTTP error may include a response with a status code and structured body. A connection failure or timeout may occur without any HTTP response at all.

StudyHub uses a helper, "_wrap_request_exception", to normalize these failures into a common representation.

For HTTP errors, the wrapper extracts the response status code and attempts to parse the response body as JSON so the classifier can inspect structured error details.

For network failures, the wrapper captures the exception information even when no response was received. A fallback regular expression, "_NETWORK_MESSAGE_FALLBACK_RE", can inspect the error message when necessary.

Normalizing the information allows the classification layer to reason about failures consistently without requiring every routing decision to understand the details of each Python exception type.

The goal is to separate exception handling from the policy that determines what the router should do next.

Sharing Failure State Across Application Instances

A failure classification is less useful if every application instance maintains an independent view of provider health.

Suppose one instance identifies a key as temporarily unusable. If another instance immediately retries that same key because it has not observed the first instance's state change, the system can generate unnecessary failures.

StudyHub centralizes runtime state in Redis using dedicated key families for cooldowns and provider blacklists, including prefixes such as:

"sh:1:ai:cooldown:"

"sh:1:ai:blacklist:"

This gives multiple instances a shared mechanism for observing relevant failure state.

The system also provides an environment-variable-controlled switch, "AI_PROVIDER_REDIS_STATE_ENABLED", that can disable Redis-backed provider-state tracking and fall back to process-local in-memory state.

That fallback can preserve some operation when Redis is unavailable, but it changes the coordination model. Individual processes may then hold different cooldown and blacklist information.

In other words, the fallback favors continued local operation over consistent shared state. It should not be mistaken for equivalent distributed coordination.

The Trade-Offs Behind Fault-Aware Routing

The classification system improves the router's ability to respond appropriately, but it also introduces design responsibilities.

Classification must evolve

Providers do not necessarily use status codes and error bodies in identical ways. A classifier that works for one provider may need additional rules for another.

Fallback is not always the correct answer

A second provider may not support the requested model, modality, or request format. The router must still respect provider compatibility and request constraints.

Shared state introduces its own dependency

Redis allows instances to coordinate cooldowns and blacklists, but Redis availability and the behavior of the local-state fallback must be considered explicitly.

Diagnostics must remain useful

When the classifier cannot confidently identify a failure, preserving diagnostic information and monitoring unknown errors helps prevent silent misclassification.

These trade-offs are part of building a reliable routing layer. Provider fallback is not simply a list of API keys; it is a policy for responding to different failure conditions.

Conclusion

Multi-provider LLM routing becomes more reliable when the system distinguishes the reasons requests fail.

StudyHub's failure taxonomy separates credential-related problems, transient provider failures, invalid model identifiers, and errors that should stop further fallback attempts.

Each category leads to a different response: cool the relevant key, switch providers without penalizing the credential, invalidate a stale model entry, or fail fast.

Combined with normalized Python exceptions and Redis-backed shared state, this design makes provider failure handling more deliberate and observable.

The broader lesson is that resilience does not come from retrying everything. It comes from understanding what failed, choosing a response appropriate to that failure, and preserving enough information to investigate cases the system does not yet understand.