DEV Community: Tural Muradov

Head-of-Line Blocking in HTTP

Tural Muradov — Mon, 10 Aug 2026 04:39:53 +0000

An asynchronous HTTP/1.1 server can handle many connections concurrently while still serializing work within each connection. Send a slow request followed by a fast request over the same persistent connection, and the fast response remains trapped behind the slow one. Send those requests over separate connections, and the fast response can arrive immediately.

That behavior is head-of-line blocking: one unit of work at the front of an ordered sequence delays independent work behind it.

HTTP/2 addresses this problem at the application layer by dividing one connection into independently identified streams. HTTP/3 goes further by running over QUIC, whose transport-level streams avoid the cross-stream blocking imposed by TCP's single ordered byte stream.

The differences are easier to understand when reduced to their scheduling and framing mechanisms. The examples below are not implementations of HTTP/2 or QUIC. They isolate the ordering constraint each protocol changes.

Reproducing the HTTP/1.1 Limitation

HTTP/1.1 supports persistent connections and request pipelining. A client may send another request without waiting for the previous response, but the server must return responses in request order. Responses do not carry request or stream identifiers, so the client associates the first response with the first request, the second response with the second request, and so on.

The following client pipelines a slow request and a fast request on one connection:

writer.write(b"GET /slow HTTP/1.1\r\nHost: localhost\r\n\r\n")
writer.write(b"GET /fast HTTP/1.1\r\nHost: localhost\r\n\r\n")
await writer.drain()

The observed timing is:

== one connection, /slow then /fast pipelined ==
  /slow response arrived at t=1.00s -> HTTP/1.1 200 OK
  /fast response arrived at t=1.00s -> HTTP/1.1 200 OK

== two separate connections, same two requests ==
  /fast response arrived at t=0.00s
  /slow response arrived at t=1.00s

On the persistent connection, /fast cannot produce a visible response before /slow. On two connections, each connection has its own ordering sequence, so the fast request is unaffected.

There are two related effects in a minimal server. A simple connection loop reads one request, awaits its handler, writes the response, and only then reads the next request. In that implementation, /fast is not dispatched until /slow completes. A more sophisticated HTTP/1.1 server could parse and execute pipelined requests concurrently, but it would still have to buffer completed responses and transmit them in request order. Either design exposes the same protocol constraint to the client.

This is why HTTP/1.1 clients historically opened several parallel TCP connections per origin. Browsers commonly used limits around six connections, although the exact policy varied by browser and changed over time. Multiple connections created several independent ordering lanes, reducing the chance that one slow response would delay every other resource. The workaround added connection setup, congestion-control state, memory use, and competition between parallel TCP flows.

HTTP/2 Adds Logical Streams

HTTP/2 replaces HTTP/1.1's textual message stream with binary frames. Every request and response belongs to a logical stream, and each frame includes a stream identifier. Frames from several streams can therefore be interleaved on one TCP connection and reassembled independently by the receiver.

The stream identifier removes the ambiguity that forces HTTP/1.1 responses into request order. If stream 1 is producing a large or slow response while stream 3 has a small response ready, the server may transmit stream 3's frames first or interleave them with stream 1's frames.

The core mechanism can be illustrated with a queue standing in for a shared connection:

class MiniConnection:
    def __init__(self):
        self.wire = asyncio.Queue()

    async def send_frame(self, stream_id, chunk, end_stream=False):
        await self.wire.put((stream_id, chunk, end_stream))
        await asyncio.sleep(0)  # let other stream producers interleave their frames too

Two producers place tagged chunks on the same queue. One response is generated slowly; the other completes in two frames:

wire: frame for stream 2: b'HTTP/1.1 200'
wire: frame for stream 2: b' DONE'
wire: frame for stream 1: b'HTTP/1.1 200'
wire: frame for stream 1: b' slow-part-1'
wire: frame for stream 1: b' slow-part-2'
wire: frame for stream 1: b' DONE'

reassembled:
  stream 2: b'HTTP/1.1 200 DONE'
  stream 1: b'HTTP/1.1 200 slow-part-1 slow-part-2 DONE'

The payload strings are illustrative rather than valid HTTP/2 messages; real HTTP/2 uses binary HEADERS and DATA frames. The relevant property is the stream ID. Stream 2 can complete while stream 1 is still active because the receiver no longer depends on global response order to interpret the bytes.

Multiplexing made one connection capable of carrying many concurrent exchanges and removed much of the motivation for a pool of HTTP/1.1 connections. It also enabled stream prioritization and flow control, although real-world prioritization behavior has evolved across clients and servers.

HTTP/2 introduced HPACK alongside multiplexing. HPACK compresses repeated header fields using static and dynamic tables, substantially reducing redundant metadata such as cookies and user-agent values. Header compression improves efficiency, but it is separate from the stream framing that removes HTTP/1.1 application-layer head-of-line blocking.

The Ordering Constraint Moves Down to TCP

HTTP/2 streams are independent within the HTTP layer, but every frame is still encoded into one TCP byte stream.

TCP guarantees reliable, ordered delivery. If a TCP segment is lost, later bytes may already be present in the receiver's network stack, but TCP cannot expose them to the application until the missing range has been retransmitted. HTTP/2 never gets a chance to inspect those later frames, even when they belong to an unrelated stream.

This is transport-level head-of-line blocking. The HTTP/2 multiplexer knows that its streams are independent; TCP does not. From TCP's perspective, the entire connection is one ordered sequence of bytes.

The distinction matters:

HTTP/1.1 head-of-line blocking comes from ordered responses within the application protocol.
HTTP/2 removes that constraint by identifying and interleaving streams.
TCP loss can still prevent HTTP/2 from receiving available data for every stream on the connection.

HTTP/2 still provides significant benefits. On stable networks, multiplexing, header compression, and connection reuse can materially improve page loading and API behavior. Its remaining limitation appears when packet loss creates a gap in the underlying TCP stream.

HTTP/3 Moves Streams into the Transport

HTTP/3 runs over QUIC rather than TCP. QUIC uses UDP datagrams as its substrate but implements reliable delivery, congestion control, encryption, connection management, and multiplexed streams in user space.

Because QUIC understands streams, an ordering gap in one stream does not prevent the application from receiving contiguous data from another stream. If data for stream A is lost, stream A waits for the missing bytes; already available data for stream B can continue to the HTTP/3 layer.

This does not eliminate every form of interference. Ordering still applies within an individual stream, and streams commonly share connection-level congestion control and network capacity. Packet loss may reduce the sending rate for the connection as a whole. QUIC specifically removes TCP's cross-stream delivery-order dependency: missing bytes in one stream do not mechanically withhold complete bytes from another.

QUIC also changes connection management in several useful ways:

TLS 1.3 is integrated into the protocol rather than layered over a completed TCP handshake.
Connection IDs allow a connection to survive some network-path changes, such as a device moving between Wi-Fi and cellular networks.
A previously established connection may use 0-RTT data, subject to server policy and replay-safety restrictions.
Transport behavior can evolve without waiting for operating-system TCP stacks to implement a new protocol version.

0-RTT is not a universal zero-latency mode. It applies only to resumed connections, can be rejected by the server, and carries replay risk. Applications should send only replay-safe requests until the handshake is confirmed.

Comparing the Protocols

Property	HTTP/1.1	HTTP/2	HTTP/3
Transport	TCP	TCP	QUIC over UDP
Multiplexing	No independent streams; pipelined responses remain ordered	Stream-ID-tagged frames	Streams implemented by QUIC and used by HTTP/3
Application-layer response blocking	Present within a connection	Removed across streams	Removed across streams
Cross-stream transport ordering	TCP can block the entire connection after loss	TCP can block the entire connection after loss	Loss in one stream does not block delivery on another stream
Header compression	Repeated textual headers	HPACK	QPACK
Traditional concurrency strategy	Several parallel connections	Primarily one multiplexed connection per origin	Primarily one multiplexed connection per origin

QPACK is not simply HPACK renamed for HTTP/3. HPACK relies on the ordered HTTP/2 connection when synchronizing dynamic header state. QUIC streams may be delivered independently, so QPACK separates encoder and decoder instructions in a way designed to limit header-decoding blockage while preserving compression.

In the Real World

Head-of-line blocking is most visible when many independent resources share limited connection capacity or when a multiplexed connection experiences loss.

Under HTTP/1.1, a browser can distribute requests across several connections, but each connection remains an ordered lane. Slow responses occupy those lanes, and additional connections bring their own setup and transport costs.

Under HTTP/2, streams share one connection and application responses can complete independently. A slow API response no longer has to delay an unrelated small asset merely because its request arrived first. Packet loss can still pause delivery across all streams until TCP fills the missing byte range.

Under HTTP/3, QUIC confines that ordering dependency to the affected stream. This can improve responsiveness on lossy or frequently changing networks, particularly mobile paths. The result is workload- and network-dependent: QUIC does not make loss free, and HTTP/3 is not automatically faster for every connection.

The progression from HTTP/1.1 to HTTP/3 is therefore not a sequence of failed attempts at the same fix. Each version changes the layer that still imposes unnecessary ordering:

HTTP/1.1 binds responses to connection-wide order.
HTTP/2 introduces application-layer streams but carries them through one ordered TCP byte stream.
HTTP/3 uses a transport that preserves ordering within each stream without imposing that order across unrelated streams.

The governing design principle is simple: preserve ordering where correctness requires it, and avoid extending that dependency to work that could progress independently.

Inside an Async Web Server

Tural Muradov — Mon, 10 Aug 2026 04:10:16 +0000

Frameworks such as FastAPI and aiohttp make HTTP servers straightforward to build. They also hide most of the machinery that explains why asynchronous servers handle large numbers of concurrent connections efficiently.

Beneath routing, middleware, validation, and dependency injection, a server follows a small cycle for every connection: read a request, dispatch it to application code, write a response, and either wait for another request or close the connection.

Most of that lifecycle is waiting. The server waits for a client to send bytes, for a socket to accept more output, and often for a database or upstream service. An event loop can use those waiting periods to make progress on other connections without assigning an operating-system thread to each one.

This article builds a minimal HTTP/1.1 server directly on asyncio.start_server(). It is intentionally not production-ready. Its purpose is to expose the architecture that frameworks normally keep out of sight: one Task per connection, non-blocking stream I/O, flow control, keep-alive, and cooperative scheduling.

The Server's Core Responsibilities

Once the framework layers are removed, the connection handler has four responsibilities:

Read and parse one HTTP request.
Select an application handler.
Serialize and write an HTTP response.
Decide whether to read another request from the same connection.

The application handler may perform CPU work, but network servers commonly spend far more time waiting on I/O. A conventional blocking architecture can overlap that waiting with multiple threads or processes. An asynchronous architecture instead represents each connection as a lightweight Task that suspends whenever its next operation cannot proceed.

This distinction matters at scale. A Task is not an operating-system thread: it is a Python object containing coroutine state. Thousands of Tasks can wait on sockets without requiring thousands of thread stacks or equivalent scheduler activity. The kernel monitors socket readiness, and the event loop resumes only the Tasks that can make progress.

Async is therefore not a shortcut to faster computation. It is a resource-efficient way to coordinate large amounts of concurrent waiting.

Accepting Connections with `start_server()`

The listening side of the server is compact:

async def main():
    server = await asyncio.start_server(handle_connection, "127.0.0.1", 8899)
    async with server:
        await server.serve_forever()

asyncio.start_server() creates and binds a listening socket, then registers it with the event loop. For each accepted connection, it invokes handle_connection() with a StreamReader and StreamWriter. Because the callback is a coroutine function, asyncio schedules the returned coroutine as a Task.

Each connection consequently has an independent execution path. When one handler waits for a request, a timer, or an asynchronous database call, its Task suspends and the loop can run another connection's Task.

The qualification around database work is important: awaiting helps only when the operation itself integrates with asynchronous I/O. Calling a synchronous database client inside an async def handler still blocks the event-loop thread.

Parsing an HTTP Request from a Byte Stream

TCP provides an ordered stream of bytes, not HTTP messages. A read may return part of a request, exactly one request, or bytes belonging to several requests. StreamReader supplies buffering helpers that let the parser work with protocol boundaries rather than TCP packet boundaries.

An HTTP/1.1 request starts with a request line, followed by headers, a blank line, and an optional body:

async def read_request(reader):
    line = await reader.readline()
    if not line:
        return None  # client closed the connection
    method, path, version = line.decode().strip().split(" ")

    headers = {}
    while True:
        header_line = await reader.readline()
        if header_line in (b"\r\n", b""):
            break
        name, _, value = header_line.decode().partition(":")
        headers[name.strip().lower()] = value.strip()

    body = b""
    if "content-length" in headers:
        body = await reader.readexactly(int(headers["content-length"]))

    return {"method": method, "path": path, "version": version, "headers": headers, "body": body}

readline() may suspend several times before a complete line is available. readexactly() does the same until it has collected the requested number of body bytes. A single low-level recv() is never guaranteed to return an entire request body, regardless of how the client wrote it.

This parser is suitable only as a teaching example. It does not limit line, header, or body sizes; validate the request target or HTTP version; reject malformed headers; support chunked transfer encoding; preserve duplicate headers; or handle conflicting message-length fields. Those omissions are security boundaries in a public server, not optional protocol polish.

Dispatching Requests and Preserving Connections

A minimal router can map request paths to coroutine functions:

ROUTES = {}

def route(path):
    def wrap(fn):
        ROUTES[path] = fn
        return fn
    return wrap

@route("/")
async def index(req):
    return make_response("200 OK", b"hello from the toy server\n")

@route("/slow")
async def slow(req):
    await asyncio.sleep(1)  # simulate an I/O-bound handler: DB call, upstream request, etc.
    return make_response("200 OK", b"done sleeping\n")

async def handle_connection(reader, writer):
    try:
        while True:
            req = await asyncio.wait_for(read_request(reader), timeout=30)
            if req is None:
                break

            handler = ROUTES.get(req["path"])
            resp = await handler(req) if handler else make_response("404 Not Found", b"not found\n")

            writer.write(resp)
            await writer.drain()

            if req["headers"].get("connection", "").lower() == "close":
                break
    except (asyncio.IncompleteReadError, ConnectionResetError, asyncio.TimeoutError):
        pass
    finally:
        writer.close()
        await writer.wait_closed()

The loop implements persistent connections: after sending a response, the handler returns to read_request() instead of closing the socket. That avoids a new TCP connection for every request.

For HTTP/1.1, persistence is the default unless the client requests Connection: close. A complete implementation must also account for HTTP/1.0 semantics, parse comma-separated connection tokens, and ensure every response has an unambiguous message boundary—typically Content-Length for a small fixed body.

A minimal response serializer can make that boundary explicit:

def make_response(status, body):
    headers = (
        f"HTTP/1.1 {status}\r\n"
        f"Content-Length: {len(body)}\r\n"
        "Content-Type: text/plain; charset=utf-8\r\n"
        "\r\n"
    )
    return headers.encode("ascii") + body

The 30-second timeout limits how long an idle or extremely slow client can retain a connection Task while sending no complete request. Real servers apply several independent controls: header timeouts, body timeouts, keep-alive timeouts, maximum request sizes, connection limits, and often per-client limits.

With the loop above, three requests can reuse the same connection:

/ -> HTTP/1.1 200 OK (same connection)
/ -> HTTP/1.1 200 OK (same connection)
/ -> HTTP/1.1 200 OK (same connection)

`drain()` Applies Backpressure

writer.write(resp) passes bytes to the transport, which attempts to write them and buffers anything the socket cannot currently accept. The method does not wait for the remote client to receive those bytes.

await writer.drain() participates in flow control. It returns quickly while the transport's write buffer remains below its high-water mark. If the buffer grows too large, it suspends the connection Task until enough buffered data has been handed to the operating system for the buffer to fall below the low-water mark.

That distinction prevents a fast producer from allocating memory indefinitely while writing to a slow client. drain() is not a delivery acknowledgement and does not guarantee that the peer has read the response. It bounds producer pressure against the local transport buffer.

For tiny responses, the buffer may never reach its limit, so drain() often appears to do nothing. It becomes essential when responses are large, streamed, or sent to slow consumers.

Measuring Concurrent Waiting

The /slow handler provides a controlled way to observe scheduling. It spends one second waiting on an asynchronous timer, which approximates the scheduling shape of an asynchronous database or upstream request without introducing external variability.

The client opens five connections concurrently:

async def fetch(path, n):
    reader, writer = await asyncio.open_connection("127.0.0.1", 8899)
    writer.write(f"GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n".encode())
    await writer.drain()
    data = await reader.read()
    writer.close()
    await writer.wait_closed()
    return n, data.decode().splitlines()[0]

async def main():
    start = time.monotonic()
    results = await asyncio.gather(*[fetch("/slow", i) for i in range(5)])
    print(f"took {time.monotonic() - start:.2f}s total")

A representative run looks like this:

client 0: HTTP/1.1 200 OK
client 1: HTTP/1.1 200 OK
client 2: HTTP/1.1 200 OK
client 3: HTTP/1.1 200 OK
client 4: HTTP/1.1 200 OK
5 concurrent /slow requests (each sleeps 1s) took 1.01s total

All five handlers start, reach await asyncio.sleep(1), and suspend. The event loop registers their timers and remains available to accept connections or run other callbacks. Roughly one second later, the timers become ready and the five Tasks continue.

The correct comparison is not a thread-per-request server: five threads blocked concurrently for one second can also finish in approximately one second. The contrasting implementation is a sequential blocking server, or an async server whose handlers call time.sleep(1) on the single event-loop thread. Either would process these sleeps serially and take roughly five seconds.

The result demonstrates concurrency, not higher execution speed. Async reduces the resources required to keep many waiting operations in flight. Whether it improves throughput depends on the workload, downstream capacity, connection limits, and how consistently handlers avoid blocking the loop.

When Async Stops Helping

The event loop can switch Tasks only when running code suspends. Replace the simulated I/O with a blocking call and every connection stalls:

@route("/blocking")
async def blocking(req):
    time.sleep(1)  # blocks the event-loop thread
    return make_response("200 OK", b"done\n")

The same applies to CPU-heavy Python code. A long computation that contains no await monopolizes the loop just as effectively as time.sleep(). The usual remedies are an asynchronous client for I/O, asyncio.to_thread() for blocking I/O that cannot be replaced, and a process pool or external worker for sustained CPU-bound work.

Async also introduces its own failure modes. Unbounded concurrency can overwhelm databases and upstream APIs even when the event loop remains responsive. Semaphores, bounded queues, connection pools, timeouts, and admission control are part of a production async architecture because cheap Tasks do not make downstream capacity infinite.

In the Real World

The minimal server exposes the scheduling model, but it leaves out the work that makes an HTTP implementation safe and interoperable:

complete request-line and header validation;
request-line, header, body, and connection limits;
chunked transfer encoding and robust message framing;
HTTP pipelining behavior and ordered responses;
TLS, shutdown coordination, and signal handling;
structured access and error logging;
overload protection and graceful connection draining;
defenses against slow clients and request smuggling;
correct cancellation and exception reporting;
HTTP/2 or HTTP/3 protocol support.

Frameworks and protocol servers such as aiohttp, Uvicorn, Hypercorn, and their underlying HTTP libraries handle these concerns so application code does not have to. Alternative event loops such as uvloop can improve throughput for some workloads, but they are an optimization to benchmark, not a substitute for sound protocol handling or non-blocking application code.

The architecture underneath remains compact. The listener turns accepted connections into Tasks. Stream operations suspend those Tasks when sockets cannot make progress. The kernel reports readiness to the event loop. Completed Futures place Tasks back on the ready queue. Each Task then runs until it reaches another suspension point.

That cycle is what allows one event-loop thread to coordinate many open connections—and what makes a single blocking handler capable of delaying all of them.

Understanding Raft

Tural Muradov — Fri, 07 Aug 2026 09:48:01 +0000

Distributed consensus becomes easier to reason about when the implementation is reduced to its essential state transitions.

This article builds a minimal three-node Raft cluster in a single process. There are no sockets, threads, or wall-clock dependencies. Nodes exchange messages through a simulated network, and a virtual clock controls delivery latency and election timeouts.

The implementation is intentionally small, but it captures the mechanisms at the center of Raft:

leader election
log replication
commitment
leader failover

Production implementations also need persistent storage, snapshots, membership changes, network transport, and careful recovery logic. Those concerns matter, but they are easier to understand once the core protocol is visible.

The Goal of Consensus

A replicated service needs its nodes to agree on one ordered sequence of operations. A key-value store, for example, might represent each write as a command in a replicated log. If every node applies the same committed commands in the same order, each state machine reaches the same result.

Raft organizes this problem around a strong leader:

One node acts as leader for a given term.
Clients send writes to the leader.
The leader appends those writes to its log and replicates them to followers.
An entry becomes committed only after the protocol establishes that it is safe.

Two mechanisms make that model work: leader election determines which node may coordinate writes, and log replication brings the other nodes into agreement with the leader's history.

Terms: Raft's Logical Clock

Every node tracks a monotonically increasing current_term. A term identifies a period in which an election occurs and, if the election succeeds, one leader coordinates the cluster.

Terms are logical rather than wall-clock time. A node that receives a message containing a term greater than its own updates current_term and becomes a follower. Messages from older terms cannot establish authority over a node in a newer term.

The minimal node state looks like this:

FOLLOWER, CANDIDATE, LEADER = "follower", "candidate", "leader"

class Node:
    def __init__(self, node_id, peers):
        self.id = node_id
        self.peers = peers
        self.state = FOLLOWER
        self.current_term = 0
        self.voted_for = None
        self.log = []              # list of (term, command)
        self.commit_index = -1
        self.next_index = {}
        self.match_index = {}
        self.votes_received = set()
        self.election_deadline = self.reset_election_deadline()
        self.leader_id = None

    def reset_election_deadline(self):
        return random.uniform(150, 300)  # ms

The randomized election timeout is essential to liveness. If every follower timed out at the same interval, multiple nodes could repeatedly become candidates together and split the vote. Randomization makes it likely that one candidate starts first and gathers a majority before competing elections begin.

In a real implementation, current_term, voted_for, and the log must be persisted before the node sends responses that depend on them. The in-memory simulation omits storage so the protocol remains easy to inspect.

Leader Election

A follower starts an election when it receives no valid leader communication before its election deadline. It increments its term, transitions to candidate, votes for itself, and requests votes from its peers:

def become_candidate(self, now):
    self.state = CANDIDATE
    self.current_term += 1
    self.voted_for = self.id
    self.votes_received = {self.id}
    self.election_deadline = now + self.reset_election_deadline()
    return [("RequestVote", self.id, peer, {
        "term": self.current_term,
        "candidate_id": self.id,
        "last_log_index": len(self.log) - 1,
        "last_log_term": self.log[-1][0] if self.log else -1,
    }) for peer in self.peers]

A node grants at most one vote per term. It also checks whether the candidate's log is at least as up to date as its own, comparing the last entry's term first and its index second:

def handle_request_vote(self, sender, msg):
    grant = False
    if msg["term"] > self.current_term:
        self.current_term = msg["term"]
        self.state = FOLLOWER
        self.voted_for = None

    log_ok = (msg["last_log_term"], msg["last_log_index"]) >= (
        (self.log[-1][0] if self.log else -1), len(self.log) - 1
    )

    if (
        msg["term"] == self.current_term
        and self.voted_for in (None, msg["candidate_id"])
        and log_ok
    ):
        grant = True
        self.voted_for = msg["candidate_id"]

    return ("RequestVoteResponse", self.id, sender, {
        "term": self.current_term,
        "vote_granted": grant,
    })

The log comparison is a safety constraint, not an optimization. A candidate with a log older than a voter's log cannot receive that vote. Combined with majority voting, this prevents a candidate that is missing a committed entry from becoming leader.

The complete protocol also resets the election timer when a vote is granted and steps down whenever a higher term appears in any RPC or response. Those transitions should be centralized in production code; the excerpts here focus on the decision itself.

Becoming the Leader

A candidate becomes leader after receiving votes from a majority of the cluster. In a three-node cluster, that requires two votes, including the candidate's own.

The new leader initializes replication state for every follower and immediately sends AppendEntries RPCs. With no log entries attached, these messages act as heartbeats:

def become_leader(self, now):
    self.state = LEADER
    self.leader_id = self.id
    for p in self.peers:
        self.next_index[p] = len(self.log)
        self.match_index[p] = -1
    return self.make_heartbeats(now)

The leader maintains two indices per follower:

next_index is the next log position the leader will send.
match_index is the highest position known to match the leader's log.

next_index begins optimistically at the end of the leader's log. If a follower rejects an append because its preceding entry does not match, the leader moves that index backward and retries.

Log Replication

When a client submits a command, the leader first appends it to its own log:

leader.log.append((leader.current_term, "SET x=1"))

The entry is not committed at this point. It exists only on the leader in this simulation and must still be replicated.

AppendEntries serves both as the heartbeat mechanism and the replication RPC. For each follower, the leader sends any entries beginning at next_index, together with the index and term immediately before them:

def make_heartbeats(self, now):
    msgs = []
    for p in self.peers:
        prev_index = self.next_index[p] - 1
        prev_term = self.log[prev_index][0] if prev_index >= 0 else -1
        entries = self.log[self.next_index[p]:]
        msgs.append(("AppendEntries", self.id, p, {
            "term": self.current_term,
            "leader_id": self.id,
            "prev_log_index": prev_index,
            "prev_log_term": prev_term,
            "entries": entries,
            "leader_commit": self.commit_index,
        }))
    return msgs

The follower accepts the entries only if its log contains a matching term at prev_log_index, or if the leader is appending from the beginning of the log. That check establishes a common prefix.

def handle_append_entries(self, sender, msg):
    success = False
    if msg["term"] >= self.current_term:
        self.current_term = msg["term"]
        self.state = FOLLOWER
        self.leader_id = msg["leader_id"]
        self.election_deadline = now + self.reset_election_deadline()

        prev_ok = (msg["prev_log_index"] == -1) or (
            msg["prev_log_index"] < len(self.log)
            and self.log[msg["prev_log_index"]][0] == msg["prev_log_term"]
        )
        if prev_ok:
            success = True
            self.log = (
                self.log[:msg["prev_log_index"] + 1]
                + msg["entries"]
            )
            if msg["leader_commit"] > self.commit_index:
                self.commit_index = min(
                    msg["leader_commit"], len(self.log) - 1
                )

    return ("AppendEntriesResponse", self.id, sender, {
        "term": self.current_term,
        "success": success,
        "match_index": len(self.log) - 1,
    })

Here, now is supplied by the simulation's virtual clock. Resetting the deadline after a valid AppendEntries prevents followers from starting an election while they are receiving communication from the current leader.

If the prefix does not match, the follower rejects the request. The leader then decrements next_index for that follower and retries from an earlier position. Once it finds a matching prefix, the follower removes conflicting entries after that point and appends the leader's entries.

This is the operational form of Raft's Log Matching Property: if two logs contain an entry with the same index and term, the entries before it are identical.

When Is a Log Entry Committed?

An entry is not committed merely because it appears in a log. In production Raft, commitment implies that the entry has been durably replicated in a way that allows it to survive leader replacement. The simulation models the replication rule but uses volatile memory rather than durable storage.

The leader uses each follower's match_index to find the highest entry replicated on a majority. It may advance commit_index through this counting rule only for an entry from its current term:

def handle_append_entries_response(self, sender, msg):
    if self.state != LEADER:
        return

    if msg["term"] > self.current_term:
        self.current_term = msg["term"]
        self.state = FOLLOWER
        self.voted_for = None
        return

    if msg["term"] != self.current_term:
        return

    if msg["success"]:
        self.match_index[sender] = msg["match_index"]
        self.next_index[sender] = msg["match_index"] + 1

        for n in range(len(self.log) - 1, self.commit_index, -1):
            replicas = 1 + sum(
                1 for p in self.peers if self.match_index[p] >= n
            )
            cluster_size = len(self.peers) + 1
            if (
                replicas > cluster_size // 2
                and self.log[n][0] == self.current_term
            ):
                self.commit_index = n
                break
    else:
        self.next_index[sender] = max(0, self.next_index[sender] - 1)

The current-term condition is one of Raft's less obvious safety rules. A leader does not directly commit an entry from an earlier term merely by observing that it now exists on a majority. Once an entry from the current term is committed, all preceding entries become committed with it.

After advancing commit_index, the leader includes the new value in subsequent AppendEntries messages. Followers then advance their own commit indices to the smaller of leader_commit and the last entry they hold. Each node can apply newly committed entries to its state machine in order.

A client should receive success only after the corresponding entry is committed and applied according to the service's response policy. A leader crash before that point may leave an uncommitted entry in some logs; a later leader may preserve or overwrite it. A reply sent earlier would therefore be unsafe.

Running the Simulation

The simulated network can be represented by messages containing a delivery time, sender, recipient, type, and payload. The main loop advances a virtual clock, delivers due messages, triggers election timeouts, and schedules periodic heartbeats.

A successful run produces a trace similar to this:

t=161 A starts election for term 1
t=181 A became leader for term 1
t=202 client command appended to A's log at index 0
t=310 command committed on all nodes: {'A': 0, 'B': 0, 'C': 0}

final state:
  A: state=leader term=1 log=[(1, 'SET x=1')] commit_index=0
  B: state=follower term=1 log=[(1, 'SET x=1')] commit_index=0
  C: state=follower term=1 log=[(1, 'SET x=1')] commit_index=0

The timestamps are properties of this simulation, not protocol guarantees. They make the message sequence observable: election, leadership, replication, majority acknowledgement, and propagation of the commit index.

Stopping the leader introduces the next important path. After the followers stop receiving heartbeats, one election deadline expires, a new term begins, and a candidate requests votes. Raft's voting restriction ensures that a node missing a committed entry cannot win the election. The new leader can then reconcile follower logs through the same AppendEntries consistency check.

What the Minimal Model Demonstrates

The simulation exposes several protocol properties that can be obscured by a production implementation:

Randomized election timeouts support progress by reducing repeated split votes.
Terms invalidate stale authority without relying on synchronized clocks.
Majority voting and the up-to-date-log check protect committed history during elections.
prev_log_index and prev_log_term let the leader discover a common prefix and repair divergent logs.
Followers learn the commit boundary from the leader; they do not independently commit entries by replica counting.
Entries from older terms become committed indirectly when a current-term entry is safely committed after them.

These mechanisms are closely connected. Election safety depends on log freshness, log repair depends on a legitimate leader, and client-visible durability depends on commitment surviving the next election.

Using Raft in Real-World Systems

The prototype models protocol decisions, not a complete distributed system. A production implementation must add several layers around the same state machine:

Durable state: persist the current term, vote, and log before acknowledging dependent RPCs.
State-machine application: apply committed entries exactly once and in log order.
Snapshots and compaction: bound storage and transfer a compact state image to followers that fall far behind.
Efficient recovery: use conflict metadata to move next_index backward by terms rather than one entry per retry.
Membership changes: use a safe reconfiguration protocol, such as joint consensus, so two disjoint majorities cannot form.
Client semantics: handle retries, deduplication, leader redirection, and responses whose delivery is uncertain after a crash.
Operational safeguards: add pre-vote, leadership transfer, observability, backpressure, and limits on replication batches where appropriate.

Failure injection is the most useful next step for the simulation. Delay selected messages, partition one node, crash a leader at each point in the replication sequence, and restart nodes with persisted state. Then assert the central safety property: no two nodes ever apply different commands at the same log index.

The Raft paper provides the complete safety argument and the edge cases deliberately omitted here. With the election and replication paths reduced to executable state transitions, its terminology maps directly onto observable behavior.

Conclusion

Raft is built from a small set of rules that reinforce one another. Terms establish authority. Elections restrict leadership to candidates with sufficiently up-to-date logs. AppendEntries repairs divergence around a verified prefix. Majority replication and the current-term rule establish a safe commit boundary.

The minimal cluster does not replace the formal specification or the engineering required for a production implementation. It provides a compact model for reasoning about both. Once the message flow is explicit, failures such as a leader crash after replication but before commitment are no longer special cases; they are ordinary state transitions governed by the same invariants.

How asyncio Really Works Under the Hood

Tural Muradov — Fri, 07 Aug 2026 05:25:23 +0000

Python's asyncio is usually introduced through its public API: define a coroutine with async def, suspend it with await, and run several operations concurrently with create_task() or gather(). That is enough to write useful programs, but it does not explain why those programs behave as they do.

The difficult questions sit below the API:

Why does one blocking function stall every coroutine on the loop?
What resumes a coroutine after an await?
Why is cancellation cooperative rather than immediate?
How can one thread manage thousands of network connections?
Why can an apparently harmless CPU loop make an entire service unresponsive?

The core execution model is smaller than the production implementation suggests. Strip away transports, protocols, signal handling, context propagation, debugging support, and platform-specific optimizations, and the scheduler reduces to a short cycle: resume a coroutine, observe what it is waiting for, and arrange to resume it when that dependency completes.

A minimal event loop makes that cycle concrete. The code below is deliberately incomplete, but it contains the essential relationship between coroutines, Futures, Tasks, the ready queue, timers, and operating-system I/O readiness.

Coroutines Build on Python's Suspension Protocol

Cooperative scheduling requires code that can suspend in the middle of a function and later continue with its local state intact. Python had that capability before native coroutines: generators already implement it through yield, send(), and throw().

def gen():
    print("start")
    x = yield 1
    print("got", x)
    yield 2

g = gen()
print(next(g))       # start -> 1
print(g.send("hi"))  # got hi -> 2

next() starts the generator and runs it until the first yield. send() resumes it, injects a value at the suspension point, and runs until the next yield or completion.

Native coroutines are distinct from generators at the language level, but they use the same underlying idea: an execution frame can be driven from the outside, suspended, resumed with send(), interrupted with throw(), and closed with close().

async def hello():
    return 1

coro = hello()
coro.send(None)  # raises StopIteration(1)

Calling hello() creates a coroutine object; it does not execute the function body. The first send(None) starts it. Because this coroutine returns immediately, the return value is carried by StopIteration, the normal completion signal for the protocol.

In an asyncio application, application code rarely calls send() directly. A Task owns that responsibility.

What `await` Actually Does

await does not block the event-loop thread. It allows the current coroutine to suspend while preserving its frame, then passes control back through the chain of awaiting coroutines until it reaches the Task driving the outermost coroutine.

Consider a socket read:

async def fetch():
    data = await sock_recv(sock, 1024)
    return data

If the socket already has data available, the operation may complete without suspending. If it is not ready, the awaitable exposes a pending dependency to the scheduler and the coroutine stops running. The thread is then free to execute another ready callback or Task.

Conceptually, await expr obtains an iterator from expr.__await__() and delegates to it. Values yielded by that iterator propagate toward the Task. When the dependency becomes ready, the Task resumes the suspended coroutine, and the awaitable produces its result or raises its exception.

The important distinction is between a coroutine and a thread. A suspended coroutine consumes no CPU. It is an object holding execution state, waiting for the scheduler to drive it again.

Futures Represent Incomplete Work

A Future is a state container for a result that may not exist yet. It does not usually perform the operation. Instead, it connects the producer of a result—such as a timer, socket callback, thread-pool completion, or another Task—to the code waiting for that result.

A minimal Future needs a completion state, a stored result, completion callbacks, and an await protocol:

class MiniFuture:
    def __init__(self):
        self._done = False
        self._result = None
        self._callbacks = []

    def set_result(self, result):
        self._done = True
        self._result = result
        for cb in self._callbacks:
            cb(self)

    def add_done_callback(self, cb):
        self._callbacks.append(cb)

    def __await__(self):
        if not self._done:
            yield self          # bubbles up to the event loop
        return self._result

When pending, __await__() yields the Future so the Task can register a wake-up callback. After set_result() runs, those callbacks make the waiting Task runnable again. The resumed await expression then returns _result.

Production asyncio.Future adds exception storage, cancellation, state validation, callback scheduling through the loop, context propagation, and safeguards against invalid transitions. Those details matter in real code, but they do not change the Future's architectural role: it is the handoff point between a producer and a suspended consumer.

Tasks Drive Coroutines

A coroutine describes computation but does not schedule itself. A Task wraps the coroutine, schedules its first step, and continues driving it until it returns or raises. A Task is also a Future, which allows other coroutines to await its eventual result.

The control flow is compact:

Resume the coroutine.
If it completes, store its return value on the Task.
If it raises, store the exception on the Task.
If it suspends on a Future, register a callback and stop.
When that Future completes, schedule the next step.

class MiniTask(MiniFuture):
    def __init__(self, coro, loop):
        super().__init__()
        self._coro = coro
        self._loop = loop
        loop.call_soon(self._step)

    def _step(self, exc=None):
        try:
            if exc is None:
                yielded = self._coro.send(None)
            else:
                yielded = self._coro.throw(exc)
        except StopIteration as e:
            self.set_result(e.value)
            return
        except Exception as e:
            self.set_exception(e)
            return

        yielded.add_done_callback(lambda f: self._loop.call_soon(self._step))

This sketch assumes that every suspension yields a compatible Future and omits result and exception propagation from the yielded Future. A real Task validates what the coroutine yielded and uses a dedicated wake-up path that retrieves the dependency's result before scheduling the next step. If that retrieval raises, the exception is thrown into the coroutine.

One property remains the same: a Task does not execute continuously. It runs only while its step callback owns the event-loop thread. At the next suspension point, it returns control to the loop and consumes no CPU until another callback makes it ready.

There is no operating-system preemption between Tasks on the same loop. This makes many state transitions easier to reason about, but it does not make shared state automatically safe. If an invariant spans an await, another Task can run before the first Task restores that invariant; asyncio.Lock and related primitives exist for exactly these cases.

Building the Event Loop

The production event loop has a broad API and substantial platform-specific behavior. Its scheduling core can still be modeled with three structures:

a ready queue for callbacks that can run now;
a min-heap for callbacks scheduled at a future time;
an I/O selector that reports file descriptors ready for non-blocking operations.

import heapq
import time
import selectors

class MiniLoop:
    def __init__(self):
        self._ready = []
        self._scheduled = []
        self.selector = selectors.DefaultSelector()

    def call_soon(self, callback):
        self._ready.append(callback)

    def call_later(self, delay, callback):
        heapq.heappush(self._scheduled, (time.monotonic() + delay, callback))

    def _run_once(self):
        now = time.monotonic()
        while self._scheduled and self._scheduled[0][0] <= now:
            _, cb = heapq.heappop(self._scheduled)
            self._ready.append(cb)

        timeout = 0 if self._ready else (
            self._scheduled[0][0] - now if self._scheduled else None
        )

        for key, _ in self.selector.select(timeout):
            self._ready.append(key.data)

        todo, self._ready = self._ready, []
        for cb in todo:
            cb()

    def run_forever(self):
        while True:
            self._run_once()

Each iteration promotes expired timers, computes how long it can wait, asks the selector for ready I/O, and executes a bounded snapshot of ready callbacks. Callbacks scheduled while that snapshot is running remain in the ready queue for a later iteration, preventing a single pass from expanding without limit.

selector.select() is not a busy loop. selectors.DefaultSelector chooses an efficient platform mechanism—commonly epoll on Linux or kqueue on macOS and BSD systems. The kernel watches registered file descriptors and wakes the process when an operation can make progress. The application therefore does not need one thread per connection or repeated user-space polling.

The simplified timer calculation can produce a small negative timeout if the clock advances between checks; production code clamps it to zero and handles timer precision, cancellation, and clock resolution. Those are implementation details around the same scheduling decision: block only until either I/O becomes ready or the next timer expires.

Tracing One `await`

The interaction becomes clearer when followed through a complete socket-read path:

async def fetch(loop, sock):
    fut = MiniFuture()
    loop.selector.register(sock, selectors.EVENT_READ, lambda: fut.set_result(sock.recv(1024)))
    data = await fut
    loop.selector.unregister(sock)
    return data

The sequence is:

MiniTask schedules its first _step with call_soon().
The loop invokes _step, which starts fetch() with coro.send(None).
fetch() registers the socket and reaches await fut.
Because the Future is pending, MiniFuture.__await__() yields it to the Task.
The Task adds a completion callback and returns control to the loop.
With no runnable application callback, the loop waits in selector.select().
The kernel reports that the socket is readable.
The registered callback receives the data and calls fut.set_result(...).
Completing the Future schedules the Task's next step.
The Task resumes the coroutine. await fut obtains the stored result, and fetch() continues after the await.
fetch() returns, represented to the Task by StopIteration(data), and the Task stores that result on itself.

The source of readiness changes across operations, but the scheduling shape does not. A sleep uses a timer instead of a selector registration. Awaiting another Task uses that Task's completion callbacks. An executor bridges a worker completion back into an event-loop Future.

`sleep(0)` Is an Explicit Scheduling Point

asyncio.sleep(0) is not merely an empty delay. It takes an optimized path that suspends the current Task and allows other ready work to run before the Task is scheduled again.

That makes it a deliberate cooperative yield. It can improve fairness in a long-running coroutine, but it does not turn CPU-heavy Python code into an appropriate event-loop workload. Repeatedly inserting sleep(0) may reduce starvation while preserving the underlying CPU contention and latency. Substantial CPU work belongs outside the loop, often in a process pool or separate service.

Cancellation Is Cooperative

Task.cancel() cannot terminate a coroutine at an arbitrary instruction. It requests cancellation and arranges for asyncio.CancelledError to be raised in the coroutine on a subsequent event-loop cycle, normally at an await where control returns to the Task.

The coroutine may use try/finally to release resources, and it may catch CancelledError to perform cleanup. If it catches the exception without re-raising it—or without calling uncancel() when intentionally suppressing cancellation—the Task may continue running. Suppression should therefore be rare and explicit.

In current Python versions, asyncio.CancelledError inherits from BaseException, not Exception. A conventional except Exception: does not catch it. An overly broad except BaseException: can still swallow it, as can an explicit except CancelledError: that fails to propagate the request.

A coroutine executing a tight CPU loop never returns control to its Task, so the cancellation cannot be delivered. This is the same failure mode as event-loop starvation: cooperative scheduling depends on application code reaching suspension points promptly.

Where Threads and the GIL Enter the Picture

The event loop does not require multiple threads to create concurrency. It interleaves non-blocking operations on one thread, switching only when a coroutine suspends. The Global Interpreter Lock is therefore not what enables or limits the core scheduling model.

Threads become relevant when synchronous work is delegated with asyncio.to_thread() or run_in_executor(). The worker runs elsewhere, and its completion is reported back to the loop through a thread-safe scheduling path and a Future. This is useful for blocking I/O and for extension code that releases the GIL.

Pure Python CPU-bound work is different. Moving it to a thread prevents the event-loop thread from executing the function directly, but contention for the GIL can still limit parallel throughput and affect latency. A process pool, native code that releases the GIL, or a separate worker service is generally a better boundary for sustained CPU-heavy work.

Reading Production Failures Through the Scheduler

Once the ready queue and Task/Future relationship are explicit, common production symptoms become easier to classify.

A service that stops responding while connections remain open often has synchronous code blocking the event-loop thread. A synchronous database driver, file operation, DNS path, logging handler, or accidental time.sleep() can prevent both I/O polling and every other Task step.

Latency spikes can come from callbacks or coroutines that run too long before suspending. Even if each unit of work eventually awaits, a large batch in the ready queue introduces queueing delay for everything behind it. Slow-callback diagnostics and event-loop lag metrics are often more useful here than network-level investigation.

Cancellation delays usually indicate that the target Task has not reached a suspension point, is waiting on code that does not propagate cancellation correctly, or has deliberately suppressed CancelledError.

Coordination helpers follow the same model. gather() schedules or observes several awaitables and exposes an aggregate awaitable whose completion depends on them. A Task waiting, directly or indirectly, on its own completion creates a dependency cycle that no callback can resolve.

These are not unrelated edge cases. They are consequences of one scheduler running callbacks until they return, Tasks driving coroutines until they suspend, and Futures reconnecting suspended consumers to completed work.

Comparing the Model with CPython

The minimal classes omit most of what makes asyncio suitable for production:

exception and cancellation propagation;
callback contexts and contextvars integration;
thread-safe scheduling;
signal and subprocess support;
transports, protocols, and platform-specific event loops;
debug instrumentation and slow-callback reporting;
protection against invalid Future and Task states;
optimized implementations in the _asyncio extension module.

Despite that additional machinery, the execution flow remains recognizable in CPython's Lib/asyncio/tasks.py, Lib/asyncio/futures.py, and Lib/asyncio/base_events.py: a Task resumes a coroutine; the coroutine either completes or suspends on an awaitable; completion of that dependency schedules the Task to continue.

That cycle is the useful mental model. It explains both the performance characteristics of asyncio and its operational constraints: the event loop scales when work suspends quickly, I/O is non-blocking, and dependency completion is converted into callbacks. It stalls when application code prevents the scheduler from regaining control.

Why Log Masking Matters in Kubernetes (and How We Enforced PCI Safety with Fluent Bit)

Tural Muradov — Mon, 29 Dec 2025 05:57:10 +0000

Logging is one of the most underestimated security risks in systems.

Not because engineers are careless, but because logs change faster than security reviews, and production pressure always wins over perfect hygiene.

We will go through:

why sensitive data ends up in logs
why relying on developers alone is not enough
how we enforced PCI-safe logging at the infrastructure level
a complete Fluent Bit + Fluentd configuration
a production-ready Lua masking script

This solution is used in a real Kubernetes production environment.

The uncomfortable truth about logs

Every experienced engineer has seen logs like these:

POST /pay?pan=4111111111111111&cvv=123
password=secretpass
Authorization failed for user=admin pass=admin123

These usually appear not because someone intended to leak data, but because of:

temporary debug logging during incidents
forgotten log statements after hotfixes
third-party SDKs logging request payloads
rushed troubleshooting under pressure
team or ownership changes

Why this is dangerous

Logs often:

live longer than databases
are accessible to more people
are copied across multiple systems
bypass encryption and access reviews
sit outside normal data-protection controls

A single leaked PAN or password in logs can result in PCI DSS violations, GDPR incidents, and audit findings that are hard to fix retroactively.

Why “just don’t log sensitive data” does not work

Even strong engineering teams eventually log sensitive data:

humans forget
incidents force shortcuts
third-party libraries log unexpected payloads
reviews do not cover every log line

Security controls must assume mistakes will happen.

That is why we moved responsibility from application code into the logging pipeline.

Enforcing security at the logging layer

We defined one rule:

Sensitive data must never leave the node unmasked, even if an application logs it.

Architecture:

Application
   ↓
Fluent Bit (masking and filtering)
   ↓
Fluentd
   ↓
Elasticsearch / Kibana

Masking happens before logs are forwarded anywhere.

Why Fluent Bit with Lua

Fluent Bit runs:

on every node
close to the log source
before logs leave the host

Lua allows:

conditional logic
validation (for example, Luhn checks)
partial masking instead of blind replacement
zero application changes

What we mask and why

Data type	Reason
PAN (card number)	PCI DSS 3.4
CVV / PIN	PCI DSS 3.3
Track1 / Track2	PCI DSS
Passwords	PCI DSS 8.x
IBAN	GDPR / PII
IP addresses	GDPR / PII

Example

Before:

password=secretpass pan=4111111111111111&cvv=123 ip=10.233.105.74

After:

password=****retpass pan=411111******1111&cvv=*** ip=xxx.xxx.xxx.xxx

Fluent Bit configuration (production-safe)

Input: exclude system namespaces early

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Exclude_Path /var/log/containers/*_kube-system_*.log,                 /var/log/containers/*_logging_*.log,                 /var/log/containers/*_monitoring_*.log,                 /var/log/containers/fluent-bit-*.log,                 /var/log/containers/fluentd-*.log
    Parser cri
    Tag kubernetes.*
    DB /tail-db/tail-containers-state.db
    Mem_Buf_Limit 100MB

This prevents Fluent Bit from ingesting its own logs, kube-system components, and logging feedback loops.

Kubernetes metadata enrichment

[FILTER]
    Name kubernetes
    Match kubernetes.*
    Merge_Log On
    K8S-Logging.Exclude On

Namespace and container exclusion

[FILTER]
    Name grep
    Match *
    Exclude kubernetes.namespace_name ^kube-system$
    Exclude kubernetes.namespace_name ^logging$
    Exclude kubernetes.namespace_name ^monitoring$
    Exclude kubernetes.container_name ^fluent-bit$
    Exclude kubernetes.container_name ^fluentd$

Lua masking filter

[FILTER]
    Name lua
    Match kubernetes.*
    script /fluent-bit/scripts/mask_pci.lua
    call filter

Masking occurs before logs leave the node.

Fluentd output configuration (example)

<match **>
  @type elasticsearch
  host elasticsearch.logging.svc
  port 9200
  logstash_format true
  include_tag_key true
</match>

At this point, all sensitive data has already been sanitized.

Full Lua masking script

-- /fluent-bit/scripts/mask_pci.lua
-- Centralized PCI / PII log masking

local function luhn(num)
    local sum, alt = 0, false
    for i = #num, 1, -1 do
        local n = tonumber(num:sub(i, i))
        if alt then
            n = n * 2
            if n > 9 then n = n - 9 end
        end
        sum = sum + n
        alt = not alt
    end
    return sum % 10 == 0
end

function filter(tag, ts, record)
    local msg = record["message"]
    if not msg then return 1, ts, record end

    -- PAN masking
    msg = msg:gsub("(%d%d%d%d%d%d%d%d%d%d%d%d%d?%d?%d?%d?%d?)",
        function(num)
            if #num >= 12 and #num <= 19 and luhn(num) then
                return num:sub(1,6) .. string.rep("*", #num - 10) .. num:sub(-4)
            end
            return num
        end
    )

    -- CVV / PIN
    msg = msg:gsub("(?i)(cvv|cvc|cv2|cvn)=\d{3,4}", "%1=***")
    msg = msg:gsub("(?i)(pin|pinblock)=\d+", "%1=****")

    -- Expiry dates
    msg = msg:gsub("(?i)(exp|expiry|expires)=\d{2}[/\-]\d{2}", "%1=**/**")

    -- Track data
    msg = msg:gsub("(?i)%B%d+%^[^%s]+", "track1=***")
    msg = msg:gsub("(?i);%d+=%d+", "track2=***")

    -- IBAN
    msg = msg:gsub("(?i)\b([A-Z]{2}\d{2}[A-Z0-9]{11,30})\b",
        function(iban)
            return iban:sub(1,6) .. string.rep("*", #iban - 10) .. iban:sub(-4)
        end
    )

    -- Passwords (partial)
    msg = msg:gsub(
        "(?i)\b(pass(word)?|passwd|pwd)\s*([:=])\s*([^\s"&]+)",
        function(key, _, sep, value)
            if #value <= 4 then
                return key .. sep .. string.rep("*", #value)
            end
            return key .. sep .. "****" .. value:sub(5)
        end
    )

    -- IPv4 addresses
    msg = msg:gsub("(%d+%.%d+%.%d+%.%d+)", "xxx.xxx.xxx.xxx")

    record["message"] = msg
    return 1, ts, record
end

PCI DSS alignment

Requirement	Enforcement
3.3	CVV and PIN fully masked
3.4	PAN partially masked
3.5	PAN never stored unmasked
8.x	Passwords protected
GDPR	PII minimized

Final result

No PANs in Elasticsearch
No credentials in Kibana
No application code changes
No reliance on human perfection
Auditors satisfied

Designing a Dynamic Loan Decision Engine in a Regulated FinTech Environment

Tural Muradov — Mon, 22 Dec 2025 03:46:10 +0000

In regulated fintech environments, loan decisioning is often treated as a mathematical problem.
In reality, it is much closer to a system design problem.

When you operate under banking regulations, external audits, and constantly changing compliance rules, the biggest challenge is not calculating a score. The challenge is building a decision engine that can adapt without breaking production, without violating regulations, and without slowing down the business.

I’ve seen decision logic become a bottleneck more than once — not because it was wrong, but because it was too rigid to evolve. This article describes how I approached the design of a dynamic loan decision engine that treats credit decisioning as a configurable, auditable system rather than a static model.

The Constraints of Regulated FinTech

Designing a loan decision engine in a regulated fintech environment means accepting a set of constraints from day one.

Regulations change. Banking requirements evolve. Audit and compliance teams require full explainability of every decision. At the same time, the business expects fast iteration, experimentation, and growth.

These forces often pull the system in opposite directions. A design that optimizes only for flexibility usually fails compliance. A design that optimizes only for compliance quickly becomes unusable for the product.

The decision engine must sit exactly in the middle — flexible enough to adapt, but strict enough to remain predictable and auditable.

Core Design Principles

Before thinking about metrics or rules, I focused on a few core principles that guided the entire architecture.

First, configuration must be treated as a first-class citizen. Decision logic should not live inside application code. If changing a rule requires a redeployment, the system will not scale operationally.

Second, customer data and product logic must be separated. Who the customer is and what a loan product allows are two different concerns. Mixing them leads to fragile and opaque decision-making.

Finally, explainability must be designed in from the start. In regulated environments, it is not enough to make the correct decision — you must be able to explain why it was made, even months later.

Card-Based Decision Model

One of the most important architectural decisions was to avoid treating loan decisioning as a single scoring function.

Instead, I designed the system around a card-based model that separates concerns and keeps decision logic understandable at scale.

Each loan request generates a Client Card — a snapshot of the customer’s current state. It contains signals derived from minimal user input and a large amount of data retrieved automatically from external systems such as government platforms and banking integrations.

Loan products are represented as Product Cards. Each product card defines its own eligibility rules, constraints, and risk parameters. This makes product logic explicit and independent from individual customers.

Between them sits the Pre-Match layer. Its purpose is simple but critical: quickly exclude requests that clearly do not meet minimum requirements, before triggering expensive or paid checks. In practice, this significantly reduces operational cost and unnecessary load on external systems.

The matching engine evaluates client cards against product cards through a configurable ruleset, producing either a loan offer or a clear rejection reason.

Metrics-Driven Decisioning

A dynamic decision engine lives or dies by the quality and structure of its metrics.

In this system, every loan request is evaluated against a large set of metrics — not to create a single opaque score, but to build a transparent decision context. The engine combines behavioral signals, external data, and internal platform information to understand both eligibility and risk.

A key design choice was to minimize required user input. The less a customer needs to manually provide, the better the conversion. At the same time, relying purely on user-provided data introduces obvious risk.

The solution was to treat external integrations as first-class data sources and design the engine so that most signals are resolved automatically. This allows the platform to make informed decisions while keeping the user experience simple and fast.

Dynamic Risk Control Without Redeployments

In regulated fintech environments, risk appetite is not static.

What is acceptable today may be too risky tomorrow, or too conservative next month. Hardcoding risk thresholds into application logic makes this problem almost impossible to manage.

To avoid this, the engine was designed so that all decision metrics and rules are fully configurable. Product teams can tighten or relax eligibility criteria, exclude specific customer segments, or temporarily hide products — without redeploying code or changing the core system.

This approach shifts credit strategy from an engineering problem into a controlled configuration problem, while still preserving technical safeguards and auditability.

Observability & Auditability

One lesson became clear early: in regulated lending, observability is not optional.

Every automated decision must be explainable — not just at the moment it is made, but months later during audits or investigations. A simple “score exceeded threshold” is rarely sufficient.

The engine was designed to record not only the final decision, but also the decision context — which metrics were evaluated, which rules were applied, and why a particular outcome was produced.

This makes automated decisioning defensible and builds trust with compliance, risk, and business stakeholders.

Lessons Learned

Building a dynamic decision engine is not about finding the perfect model.

It is about designing a system that can evolve without becoming fragile. Flexibility without structure leads to chaos. Structure without flexibility leads to stagnation.

The biggest lesson was that credit decisioning should be treated as a living system — one that needs continuous tuning, visibility, and clear ownership — rather than a static artifact delivered once and forgotten.

In modern fintech platforms, especially in regulated environments, loan decisioning is a system design problem before it is a data science problem.

Dynamic, configurable decision engines allow platforms to scale responsibly, adapt to regulation, and evolve with the market — without sacrificing control or transparency.

Static models may still have their place, but they are no longer sufficient on their own. The future of automated lending belongs to systems that can change as fast as the world around them.

Why Static Credit Scoring Fails in Modern FinTech Platforms

Tural Muradov — Sun, 21 Dec 2025 09:59:52 +0000

Introduction

I’ve worked with credit decisioning systems long enough to see the same pattern repeat itself.

A scoring model is built, validated, approved — and at some point, it quietly turns into the biggest obstacle to product growth. Not because it’s inaccurate, but because it can’t keep up with how fast reality changes.

Static credit scoring works well on paper. In real production fintech systems, especially in regulated or emerging markets, it often fails in ways that are not immediately obvious. The problem is rarely the math itself — it’s the rigidity around it.

What Is Static Credit Scoring?

Static credit scoring typically relies on a predefined model where:

input parameters are fixed or change infrequently
decision thresholds are hardcoded or rarely adjusted
scoring logic is tightly coupled to the application code or external vendors
changes require model retraining, redeployment, or long approval cycles

These models are often treated as “black boxes” and integrated as a single decision point in the loan approval process.

While this approach simplifies governance and validation, it assumes that risk profiles, regulations, and product requirements remain relatively stable over time — an assumption that no longer holds true for most fintech platforms.

Why Static Scoring Breaks in Modern FinTech

In theory, static scoring models provide stability and predictability. In practice, they break down once a fintech platform starts operating at scale.

A small change — a new regulatory requirement, a new loan product, or entry into a new market — often turns into weeks of coordination between risk, engineering, and compliance teams. Even minor adjustments may require model retraining, vendor involvement, or full redeployment.

By the time the scoring logic is updated, the business context has already changed again.

What makes this worse is that static models usually treat credit decisioning as a single checkpoint rather than a process. They don’t adapt well to multi-step onboarding flows, product-specific rules, or real-time signals coming from external systems.

Over time, this rigidity forces teams to choose between speed and control — a trade-off that modern fintech platforms can’t afford.

The Reality of Regulated and Emerging Markets

In regulated and emerging markets, the limitations of static scoring become even more visible.

Regulatory requirements may vary significantly between countries and can change faster than traditional scoring models can adapt. In addition, data availability and quality often differ across markets, making one-size-fits-all scoring models ineffective.

Static scoring systems are poorly suited for environments where:
• minimum required data may change
• external data sources evolve over time
• compliance rules differ per product or region
• explainability and auditability are mandatory

In these scenarios, static models introduce operational risk and slow down market expansion.

Business and Product Impact

From a business perspective, static credit scoring often leads to:
• over-rejection of potentially good customers
• increased operational costs due to manual reviews
• slower time-to-market for new products
• limited ability to experiment with user journeys

Instead of enabling growth, the scoring system becomes a constraint that limits both conversion and innovation.

What Modern FinTech Platforms Actually Need

Modern fintech platforms require dynamic decisioning systems, not static scoring models.

Key architectural principles include:
• configuration-driven rules instead of hardcoded logic
• separation of customer data from product eligibility logic
• explainable and auditable decisions
• ability to adjust risk parameters without redeployment
• product-aware and market-aware decisioning

These principles allow fintech platforms to remain compliant, scalable, and adaptable in a constantly changing environment.

Conclusion

Static credit scoring models were built for a different era — one where markets were stable, products were limited, and regulatory change was slow.
In modern fintech platforms, especially those operating across multiple markets and regulatory regimes, static scoring increasingly fails to meet business, risk, and technical requirements.
This doesn’t mean static credit scoring is useless. In some contexts, it’s still the right choice.
But for fintech platforms that need to adapt quickly, operate under changing regulation, and scale across markets, treating credit decisioning as something fixed is increasingly a liability rather than a safeguard.

MadMigration

Tural Muradov — Mon, 15 Feb 2021 10:21:36 +0000

I would like to introduce a recently started open source project designed for database migration. Why would we need such a tool when there are many database migration tools that exist? Let's dive into details. We were asked to add some new features to a project which was made back in time. Upon entering the project we realized that the database structure was not designed very well, had complicated issues on tables. We decided to redesign the database structure and keep data as it was.
We looked at many tools to produce solution for this case, however, they did operate almost the same. Basically, the procedure of the tools, obtaining old database structure and convert data types,change structure of database and transfer it to another database. We have designed new database structure and our main goal to keep data. We needed to take data from different tables and transfer them to a new table. For sure, another option could be dumping old database and change everthing there and import it into new a database. Fortunately we did not apply this tecnnique, we decided to make a tool that does all of it. That is where story begins MadMigration.

MadMigration is written entirely in python. We used the power of the SQLAlchemy library. In this tutorial, we will explain the details of how to use MadMigration.

Installation

pip install madmigration
or
pip3 install -U madmigration

You can check version:
madmigration --version

Let's get start.

For example you have old MySQL database table with name user :

#Mysql
CREATE TABLE `user` (
  `id` int NOT NULL AUTO_INCREMENT,
  `fname` varchar(10) NOT NULL,
  `lname` varchar(10) NOT NULL,
  `nick` varchar(10) NOT NULL,
  `email` varchar(15) NOT NULL,
  `pass` varchar(15) NOT NULL,
 `updated` TIMESTAMP NULL,
 `created` TIMESTAMP NOT NULL,
  `companyID` INT,
  PRIMARY KEY (`id`),
  FOREIGN KEY (companyID)
  REFERENCES company(`id`)
  ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

And you want to migrate this table to new postgresql table with different column names:

#postgresql

CREATE TABLE public."User" (
created_at timestamp NOT NULL,
updated_at timestamp NULL,
id serial NOT NULL,
"name" varchar NOT NULL,
surname varchar NOT NULL,
nickname varchar NOT NULL,
email varchar NOT NULL,
"password" varchar NOT NULL,
CONSTRAINT "User_email_key" UNIQUE (email),
CONSTRAINT "User_pkey" PRIMARY KEY (id),
company_id integer REFERENCES company (id)
ON DELETE SET NULL
);

Then create yaml file schema.yml for declaring tables and columns. At first stage add Configs section. This section is for connecting to database. Configs section have two subsections SourceConfig,DestinationConfig:

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

The migrationTables section is intended to describe information about tables:

migrationTables:
  - migrationTable:
      SourceTable:
        name: user
      DestinationTable:
        name: User
        create: true

Nextly, write information about columns in MigrationColumns section.

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn:
      name: id
      options:
        primary_key: true
        type_cast: bigint

In case of creating a foreign key you can specify it in the column parameters. By defining foreign key options, such as ondelete,onupdate. The table_name parameter is a reference to the table, column_name is the name of the referenced column.

  - sourceColumn:
      name: companyID
    destinationColumn:
      name: company_id
      options:
        type_cast: int
        nullable: true
        foreign_key:
          table_name: company
          column_name: id
          ondelete: "SET NULL"

No options added sourceColumn section, however in destinationColumn section we could have primary_key,nullable,default,index,unique,autoincrement,foreign_key,length,type_cast options. The final yaml sturtcure will look like this:

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

migrationTables:
  - migrationTable:
      SourceTable:
        name: user
      DestinationTable:
        name: User
        create: True 

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn: 
            name: id
            options:
              primary_key: true
              type_cast: bigint

        - sourceColumn:
            name: fname
          destinationColumn:
            name: name
            options:
              type_cast: string

        - sourceColumn:
            name: lname
          destinationColumn:
            name: surname
            options:
              type_cast: string

        - sourceColumn:
            name: email
          destinationColumn:
            name: email
            options:
              type_cast: string
              unique: true

        - sourceColumn:
            name: nick
          destinationColumn:
            name: nickname
            options:
              type_cast: string

        - sourceColumn:
            name: pass
          destinationColumn:
            name: password
            options:
              type_cast: string

        - sourceColumn:
            name: updated
          destinationColumn:
            name: updated_at
            options:
              type_cast: datetime

        - sourceColumn:
            name: created
          destinationColumn:
            name: created_at
            options:
              type_cast: datetime

        - sourceColumn:
            name: companyID
          destinationColumn:
            name: company_id
            options:
              type_cast: int
              nullable: true
              foreign_key:
                table_name: company
                column_name: id
                ondelete: "SET NULL"

In addition, you can migrate your data from RDBMS to NOSQL. Below a small example for migration data from PostgreSQl to MongoDB. Work on PostgreSQl to MongoDB still goes, we will add other features as soon as possible.

version: 0.1.8
Configs:
  - SourceConfig:
      dbURI: "postgresql://root:admin@localhost:5432/olddb"
  - DestinationConfig:
      dbURI: "mongodb://localhost:27017/mydb"

migrationTables:
  - migrationTable:
      SourceTable:  #postgresql table name
        name: company  
      DestinationTable: #the collection name:
        name: Company

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn:
            name: id
            options:
              type_cast: uuid

        - sourceColumn:
            name: name
          destinationColumn:
            name: NAME
            options:
              type_cast: varchar

        - sourceColumn:
            name: created
          destinationColumn:
            name: CREATED
            options:    
              type_cast: datetime
        - sourceColumn:
            name: email
          destinationColumn:
            name: EMAIL
            options:     
              type_cast: string
        - sourceColumn:
            name: updated
          destinationColumn:
            name: UPDATED
            options:
              type_cast: datetime
         - sourceColumn:
            name: code
          destinationColumn:
            name: CODE
            options:      
              type_cast: string

Upon finishing declaration of yaml file run the following command.

madmigrate -f schema.yaml

Alternatively, you can separate migration logic into multiple .yaml or .json files. Importantly, one main .yaml file should exist, other files should be imported into this main file.

main.yaml file:

version: 1.1
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

migrationTables:
  - migrationTable: !import company.yaml
  - migrationTable: !import op_cond.json

company.yaml file

SourceTable:
  name: company
DestinationTable:
  name: company
  create: true 

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn: 
      name: id
      options:
        primary_key: true
        type_cast: uuid

  - sourceColumn:
      name: name
    destinationColumn:
      name: name
      options:
        length: 120
        type_cast: varchar
        nullable: false

  - sourceColumn:
      name: created
    destinationColumn:
      name: created
      options:
        type_cast: datetime
  - sourceColumn:
      name: updated
    destinationColumn:
      name: updated
      options:
        type_cast: datetime
        nullable: true

op_conds.json file

{
    "SourceTable": {
      "name": "operation_conditions"
    },
    "DestinationTable": {
      "name": "operation_conditions",
      "create": true
    },
    "MigrationColumns": [
      {
        "sourceColumn": {
          "name": "id"
        },
        "destinationColumn": {
          "name": "id",
          "options": {
            "primary_key": true,
            "type_cast": "uuid"
          }
        }
      },
      {
        "sourceColumn": {
          "name": "interest"
        },
        "destinationColumn": {
          "name": "interest",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "FIFD"
        },
        "destinationColumn": {
          "name": "FIFD",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "comission"
        },
        "destinationColumn": {
          "name": "comission",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      }
    ]
  }

We are not currently fully implemented all functionality for SQL and NOSQL, therefore we are still developing and adding new features on it.
Thank you for amazing work to our team for implementing this tool.

Building Authorization in FastAPI with Open Policy Agent (OPA)

Tural Muradov — Sat, 01 Feb 2020 21:46:11 +0000

Authorization logic shouldn't be scattered across your Python
services. In this article, we'll build a complete authorization
service using Open Policy Agent (OPA) and opa-python-client,
from writing your first Rego policy to integrating it with FastAPI.

Why Another Authorization Article?

If you've built more than one backend service, you've probably written
code like this:

if current_user.role == "admin":
    return True

if current_user.id == document.owner_id:
    return True

raise HTTPException(status_code=403, detail="Forbidden")

Initially, this seems perfectly reasonable.

Then your application grows.

Now you need to support:

Multiple user roles
Organization-level permissions
Temporary permissions
Country-specific regulations
Customer-specific rules
Feature flags
Auditable authorization decisions

Soon, every service has its own authorization implementation.

Business rules become duplicated.

Changing one permission requires modifying multiple services.

Testing authorization becomes difficult.

This is exactly the problem Open Policy Agent (OPA) solves.

What is Open Policy Agent?

Open Policy Agent is an open-source policy engine.

Instead of embedding authorization logic inside your application, your
application asks OPA a question.

Can Alice read this document?

OPA evaluates your policies written in Rego and returns either:

true

false

Your application no longer needs to know how authorization works.

It simply enforces the decision.

Architecture

                +------------------+
                |   FastAPI App    |
                +------------------+
                         |
                         |
               Authorization Request
                         |
                         ▼
                +------------------+
                | opa-python-client|
                +------------------+
                         |
                     HTTP API
                         |
                         ▼
                +------------------+
                |      OPA         |
                +------------------+
                         |
                    Rego Policies
                         |
                    Data Documents

Running OPA

docker run \
  --name opa \
  -p 8181:8181 \
  openpolicyagent/opa:latest \
  run --server

Installing the Python Client

pip install opa-python-client

Writing Our First Policy

Create authz.rego

package authz

default allow := false

allow if {
    input.user.role == "admin"
}

allow if {
    input.user.id == input.document.owner
}

Uploading Policies

from opa import OPAClient

client = OPAClient("http://localhost:8181")

with open("authz.rego") as file:
    client.create_policy(
        policy_name="authz",
        policy=file.read(),
    )

Evaluating Authorization

decision = client.check_policy(
    "authz/allow",
    {
        "user": {
            "id": 10,
            "role": "customer"
        },
        "document": {
            "owner": 10
        }
    }
)

print(decision)

Expected output:

True

Dynamic Data

client.update_data(
    "users",
    {
        "admins": [
            "alice",
            "bob",
            "charlie"
        ]
    }
)

package authz

default allow := false

allow if {
    input.user.name in data.users.admins
}

FastAPI Integration

from fastapi import FastAPI, HTTPException
from opa import AsyncOPAClient

app = FastAPI()

client = AsyncOPAClient("http://localhost:8181")

@app.get("/documents/{document_id}")
async def get_document(document_id: int):

    payload = {
        "user": {
            "id": 15,
            "role": "customer"
        },
        "document": {
            "owner": 15
        }
    }

    allowed = await client.check_policy(
        "authz/allow",
        payload
    )

    if not allowed:
        raise HTTPException(
            status_code=403,
            detail="Forbidden"
        )

    return {
        "id": document_id,
        "title": "My Document"
    }

Production Recommendations

Enable HTTPS
Configure request timeouts
Use mutual TLS where appropriate
Store policies in Git
Test Rego policies
Separate policies from data
Monitor policy evaluation latency

Why I Built opa-python-client

The goal of opa-python-client is to provide a clean, Pythonic,
production-ready interface for interacting with Open Policy Agent.

Resources

GitHub: https://github.com/Turall/OPA-python-client
Open Policy Agent: https://www.openpolicyagent.org/

DEV Community: Tural Muradov

Head-of-Line Blocking in HTTP

Reproducing the HTTP/1.1 Limitation

HTTP/2 Adds Logical Streams

The Ordering Constraint Moves Down to TCP

HTTP/3 Moves Streams into the Transport

Comparing the Protocols

In the Real World

Inside an Async Web Server

The Server's Core Responsibilities

Accepting Connections with start_server()

Parsing an HTTP Request from a Byte Stream

Dispatching Requests and Preserving Connections

drain() Applies Backpressure

Measuring Concurrent Waiting

When Async Stops Helping

In the Real World

Understanding Raft

The Goal of Consensus

Terms: Raft's Logical Clock

Leader Election

Becoming the Leader

Log Replication

When Is a Log Entry Committed?

Running the Simulation

What the Minimal Model Demonstrates

Using Raft in Real-World Systems

Conclusion

How asyncio Really Works Under the Hood

Coroutines Build on Python's Suspension Protocol

What await Actually Does

Futures Represent Incomplete Work

Tasks Drive Coroutines

Building the Event Loop

Tracing One await

sleep(0) Is an Explicit Scheduling Point

Cancellation Is Cooperative

Where Threads and the GIL Enter the Picture

Reading Production Failures Through the Scheduler

Comparing the Model with CPython

Why Log Masking Matters in Kubernetes (and How We Enforced PCI Safety with Fluent Bit)

The uncomfortable truth about logs

Why this is dangerous

Why “just don’t log sensitive data” does not work

Enforcing security at the logging layer

Why Fluent Bit with Lua

What we mask and why

Example

Fluent Bit configuration (production-safe)

Input: exclude system namespaces early

Kubernetes metadata enrichment

Namespace and container exclusion

Lua masking filter

Fluentd output configuration (example)

Full Lua masking script

PCI DSS alignment

Final result

Designing a Dynamic Loan Decision Engine in a Regulated FinTech Environment

Why Static Credit Scoring Fails in Modern FinTech Platforms

MadMigration

Installation

Building Authorization in FastAPI with Open Policy Agent (OPA)

Why Another Authorization Article?

What is Open Policy Agent?

Architecture

Running OPA

Installing the Python Client

Writing Our First Policy

Uploading Policies

Evaluating Authorization

Dynamic Data

FastAPI Integration

Production Recommendations

Why I Built opa-python-client

Resources

Accepting Connections with `start_server()`

`drain()` Applies Backpressure

What `await` Actually Does

Tracing One `await`

`sleep(0)` Is an Explicit Scheduling Point