DEV Community: M. Alwi Sukra

TIL - Choosing Between Code, an LLM Call, and an AI Agent

M. Alwi Sukra — Mon, 27 Jul 2026 02:16:42 +0000

Two questions sent me down this path.

Question one: what is an "AI agent," really? Most job posts mention them. I had not looked into it deeply, and from the outside I could not tell what it referred to. Is an agent a different endpoint? A different model? A library we install? Or is it just a name for calling an LLM API in some particular way?

Question two came later, and it stopped me cold. Once I understood the mechanics and sat down to pick something to build, I could not come up with a single idea that I could not also write as ordinary Go: a function that calls some APIs, hits a database, and returns a result. If plain code can do it, why pay a model to do it worse?

This is what I found. There is a real answer to both, and the second one has a trap in it.

So I built both

Code: github.com/Arkoes07/llm

One Go service, one interface, four endpoints of increasing complexity:

POST /chat/no-memory        one stateless LLM call
POST /chat                  multi-turn, history per session
POST /chat/agent/weather    tool-calling loop, one tool
POST /chat/agent/log-triage tool-calling loop, three tools

No framework. Hand-rolled net/http first, then a second implementation behind the same interface using an SDK, so I could see exactly what the SDK was doing for me. (Answer: it removed boilerplate, not thinking. The loop logic was identical in both). Worth noting: Groq's official SDKs are Python and JS only, so in Go we are on community libraries or an OpenAI-compatible client.

Starting at a plain call and walking up to an agent one step at a time turned out to be the whole answer to question one.

An agent is a while-loop

Here is the version with no vocabulary.

A plain LLM call is a stateless HTTP request. We send messages, we get text back. The model remembers nothing. "Conversation history" is state we store and resend every time. That is /chat/no-memory and /chat above, and the only difference between them is who keeps the array.

An agent adds one thing: we also send a list of tools, described as JSON schemas. Now the model can respond with "call get_metrics with {"service":"checkout"}" instead of a final answer. So we loop:

Send messages plus tool schemas.
Model replies with either a final answer, or a tool request.
Our code executes the tool. The model executes nothing, it only asks.
Append the result to the history, call again.
Repeat until it stops asking, or we hit an iteration cap.

That is the difference. Same endpoint, same model, about 150 lines of extra Go. Every agent framework is this loop plus state persistence, retries, and tracing.

What I realized while building it is that most of the hard parts are things we already know under different names:

Agent problem	What it actually is
Tool dispatch + schema validation	RPC dispatch
Model calls the same tool twice	Idempotency
Multi-step work that must not half-complete	Saga / compensation
Context window overflow	Bounded cache with an eviction policy
Untrusted text in the prompt issuing instructions	An authz boundary
Runaway loop	Circuit breaker
"Did this actually work?"	Regression testing, for non-deterministic output

There is a reason the table lines up like that, and the loop hides it.

From inside the code it looks like one API call repeated. But we call the model, the model asks for a tool, we call that tool (in a real system, another service: a logging backend, a metrics API, a runbook store), then we call the model again. One request fans out across several independent services, in an order nobody fixed in advance. That is orchestration, and we already have a name for it: a distributed system.

The difference is which part we cannot trust. Normally the orchestrator is our own code, the most reliable thing in the system, and we save our suspicion for the dependencies. In an agent, the orchestrator is the model. The component deciding what to call next is now the least reliable one: it picks a different sequence on identical input, it can emit a malformed request, and when it fails it usually fails plausibly instead of loudly.

So the instincts still apply, we just point them somewhere new. Validate every argument. Make writes idempotent, because the caller may repeat them. Bound the loop. Treat anything a tool returns as data, never as instructions. What we already do at the edge of a system, applied to the orchestration we are used to trusting.

What is actually new is a short list: prompting, structured output, evals, and retrieval.

Question one, answered. Which is when question two showed up.

Why would I ever use this?

I needed something to build. And every idea I had, I could immediately picture as normal Go.

"Generate a study plan" is one LLM call with a structured output, wrapped in a function. No loop. "Classify support tickets" is one call, then a switch on the result. "Summarize a document" is one call. Every time I reached for the loop, plain code was cheaper, faster, deterministic, and testable.

That is not a small objection. It is the whole question. The loop costs us money (tokens), latency (seconds per iteration), determinism (same input, different output), and debuggability (behavior lives in prose, not code). What are we buying with all that?

Exactly one thing: the model chooses the next step at runtime, using information that did not exist when we wrote the code.

Not intelligence, and not language. Those come from a single call. The loop buys runtime branch selection and nothing else.

So the question becomes: when do we actually need that?

The criterion I landed on

The common framing is "simple problems go to code, complex problems go to an agent." Too vague to act on. Here is the version I would use:

Can we draw the flowchart before we see the input?

If yes, the decision points are enumerable and we can write the branches, so we should write them. Code is deterministic, testable, debuggable, and roughly free. An agent there is pure overhead, and the LLM's job, if it has one, is a single call inside one box: classify, extract, summarize. No loop.

If no, meaning the path depends on what each step reveals and the real tree has thousands of branches we would only discover in the moment, that is the narrow case where the loop earns its cost.

That criterion is why I picked log triage. Consider:

if latencyHigh {
    checkDatabase()
    if dbSlow {
        checkConnectionPool()
    } else {
        checkDownstream()  // which one? there are 30
    }
}

We can write this. But which downstream service? Depends on what the logs said. What if the DB check reveals a deploy changed a query plan? We did not have a branch for that. The order we investigate depends on what each step returns.

That is the trigger. Not "it is complicated," but "I cannot enumerate the branches at code-writing time."

Keep this criterion in mind. The rest of the post is about what happens when we pass it and then slowly undo our own answer.

The test case

A fake incident with a deliberate lie in it:

checkout p99 spiked at 14:03. But checkout is fine. Its DB pool is healthy (4/20) and traffic is normal. It calls payment, whose max_pool_size was quietly lowered 20 → 5 by a config reload at 13:58. Payment saturated its pool and started timing out. Checkout is the victim, not the cause.

Three tools over hardcoded data: query_logs, get_metrics, search_runbook. Getting it right means ruling out checkout's own resources, hopping to the downstream service, and (the trap) widening the time window past the incident to catch that 13:58 config change.

Mocked data is a feature, not a shortcut. Every failure is then unambiguously the loop's, not flaky infrastructure.

Then it went wrong, five times

1. It cited a runbook it never opened

iteration 1: query_logs(checkout, 13:55-14:05)
iteration 2: get_metrics(checkout, latency)
iteration 3: get_metrics(checkout, error_rate)
iteration 4: get_metrics(payment, latency)
iteration 5: get_metrics(checkout, request_rate)
iteration 6: query_logs(payment, 13:55-14:05)
iteration 7: get_metrics(payment, db_pool)
iteration 8: get_metrics(checkout, db_pool)

Note what is missing: search_runbook, never called. Now the answer it produced:

...These symptoms match the known runbook pattern for "Database connection-pool exhaustion".

Remediation (per the runbook)

Restore / increase the DB pool size for the payment service

Restart the payment service to clear lingering connection-wait states

The root cause was correct. The provenance was invented. It attributed general knowledge to an internal document it never opened, and step 2 was not in my runbook at all.

Nothing in the output looked wrong. Plausible content, authoritative formatting, correct conclusion. The only way to catch it was diffing the answer against the trace. Now imagine a real runbook saying "do NOT restart, drain connections first, page the DBA."

Fix (system prompt): "Only cite the runbook if you called search_runbook. Otherwise say the recommendation is based on general knowledge."

The lesson: an agent can reach a correct answer through invalid reasoning and give us no signal that it did. That is the argument for evals, and it is checkable. "Every source cited must appear in the tool trace" is an assertion we can write.

2. It stopped at the first plausible explanation

iteration 1: query_logs(checkout, 13:55-14:05)
iteration 2: get_metrics(checkout, error_rate)
iteration 3: search_runbook("payment.authorize timeout")

Three calls, then it concluded: tune the client timeout, add a circuit breaker. Plausible, and wrong. It never checked whether checkout's own DB was healthy or whether traffic was abnormal. It gathered evidence for its first hypothesis instead of testing alternatives. A human had to type "continue, check the downstream."

Fix (system prompt): "Do not stop at the first plausible explanation. Before concluding, rule out alternatives: check whether the service's own resources are healthy and whether traffic is abnormal."

3. It searched the runbook before it knew what to search for

iteration 6: search_runbook("payment.Authorize timeout")   ← searched here
iteration 7: get_metrics(payment, latency)
iteration 8: query_logs(payment, 13:55-14:05)              ← found the cause here

It searched using the symptom it had at iteration 6, found the actual root cause at iteration 8, and never searched again. A human does this reflexively: it is pool exhaustion, so pull up that runbook. The loop does not re-plan by default.

Fix (system prompt): "After identifying a root cause, search the runbook again using the root-cause terms, not the original symptom."

4. The caller emitted an invalid call

{"error":{"code":"tool_use_failed","failed_generation":
"<function=query_logs{\"service\": \"checkout\"}</function>"}}

Right tool, right arguments, malformed syntax. It is missing one > after the function name. A gRPC client cannot do this; the wire format is generated and guaranteed. Here the caller's ability to form a valid request is itself probabilistic.

It also broke my retry logic:

attempt 1: <function=query_logs{...
attempt 2: <function=query_logs{...   ← identical
attempt 3: <function=query_logs={...  ← one char different
attempt 4: <function=query_logs{...   ← identical again

Four attempts, 15 seconds of backoff, the same malformed output. Retry only helps if the output would differ. At low temperature the model is near-deterministic, so an identical request produces an identical failure.

Fix (code, then model): drop tool_use_failed from the retryable set, so we fail fast instead of burning 15 seconds. Then switch models. I did not see it again on openai/gpt-oss-120b.

5. Context growth is the cost model

Rate limit reached ... tokens per minute (TPM): Limit 8000, Used 6886, Requested 1311

Every iteration resends the entire history, so iteration 8 carries all seven prior tool results. Eight iterations, under a minute, into a hard wall. Not because any single call was large, but because they compound. Log prompt_tokens per iteration; that curve is the most honest thing we can show about agent economics.

Fix (code): exponential backoff on 429, honoring the Retry-After header rather than our own curve. This is a mitigation, not a fix. The growth is inherent to the loop.

The two kinds of guardrail

Look back at the labels on those fixes. Two were code, and code fixes are just engineering: better retry classification, better backoff. Nothing interesting there.

The other three were system prompt. And those three are not the same species.

Type A: guardrails that improve judgment.
Better tool descriptions. "Rule out alternatives before concluding." "Cite the runbook only if you called it." These do not shrink the option space, they help the model navigate the same space better. Nearly free. Keep them, add more.

Type B: guardrails that encode the flowchart.
"Always call query_logs before get_metrics." "After identifying a root cause, search the runbook again." These remove options. Each one is a branch, written in English instead of Go.

The test is mechanical:

If we can state the guardrail as a deterministic rule, we can write it as code. And if we can write it as code, it does not belong in the prompt.

Fix #3 is mine, and it is Type B. "After identifying a root cause, search again with the root-cause terms" is a sequencing rule. I could enforce it in code: if the loop is about to terminate and search_runbook was never called with the root-cause terms, force one more iteration. Deterministic, testable, free. Instead I wrote a branch in English and paid a model to interpret it.

That is the paradox. Remember what the loop buys us: the model choosing at runtime. Every Type B guardrail takes some of that back. We are paying full price for a capability we are actively suppressing. Push far enough and we have a complete flowchart written in English, executed by a probabilistic interpreter, at a hundred times the cost of an if, with no type checking, no test coverage, and no way to diff a behavior change in code review.

We did not build an agent. We built the world's most expensive switch statement and moved it out of version control.

And notice the shape of it. I passed the flowchart criterion honestly (log triage really is not enumerable) and then spent three fixes quietly re-enumerating it anyway.

Which means the two ideas in this post were never separate. The criterion tells us whether the flowchart exists before we start. The guardrails tell us whether we were right, after. Same spine, checked from both ends. If we find ourselves writing the flowchart in English one rule at a time, the criterion already had the answer.

The diagnostic

The number of Type B guardrails we need is a diagnostic on whether we picked the right tool.

The agent works with almost none, which means the problem really was not enumerable. The loop earned its cost.
We keep adding them and reliability keeps improving. That is not tuning. That is the problem telling us it was enumerable, and we should port those rules back into code and shrink the agent to whatever judgment is left.

One caveat: how many we need is not a fixed property of the problem. Compare the two traces above. Finding #2, the three-call run that stopped at checkout, was llama-3.3-70b-versatile. Finding #1, the eight-call run that ruled out checkout's pool and traffic, hopped to payment, and widened the window to 13:55 on its own, was openai/gpt-oss-120b. Same tools, same fake world, same prompt. The only change was the model string.

I only ran each model once, so treat this as a signal and not a benchmark. But the swing matters: the same code was a supervised tool on one model and an autonomous one on another. A guardrail we need this quarter may be dead weight next quarter.

(Related: the Llama model I started on was deprecated on Groq partway through this project. Model availability is a dependency that disappears).

The answer to both questions

Three tools, not two. Most of the confusion in this space comes from collapsing the middle one.

1. Pure code, no LLM at all.
The flowchart is drawable and every step is mechanical. Parse, transform, query, branch. If we can specify it, we should specify it. This is still most software.

2. A single LLM call inside code we control.
The flowchart is drawable, but one box in it needs judgment over unstructured input: classify this ticket, extract these fields, summarize this text, draft this reply. One call, structured output, and our code decides everything else. This is the overwhelming majority of real "AI features," and it is not a lesser agent. It is the correct architecture for an enumerable problem.

3. The agent loop.
We genuinely cannot draw the flowchart, because step three depends on what step two returned and there are too many possibilities to enumerate. Incident triage, code review, open-ended research. Here the loop reaches things branching would not, and we pay for it in tokens, latency, and non-determinism.

The line between 2 and 3 is the flowchart criterion. The line between 2 and 1 is just whether any single step needs judgment over unstructured input.

And the sweet spot inside option 3, which is what the guardrail paradox is really about:

Constrain in code everything that can be enumerated. Leave the loop wrapping only the irreducible judgment.

The agent should be as small as possible. That is the principle I would take into the next one, and it follows from the paradox rather than from measurement.

The observation underneath it is smaller and more concrete. Every idea I started with turned out to be a code problem in disguise. That is what question two was. I could not find a use case that plain Go would not handle until I went looking for one deliberately, and even then the loop only survived because I stopped short of writing the last three rules in code.

What I have not resolved

Two things I could not settle from this build.

Where the crossover actually sits. I can say the loop earns its cost when the flowchart is not drawable, but I cannot yet say what that costs in dollars and milliseconds compared to the deterministic version of the same task. I have the argument and not the numbers.

When to stop adding Type B guardrails and rewrite in code. The diagnostic says a rising count means we picked wrong, but there is no threshold. Two Type B rules is clearly fine. Fifteen is clearly a flowchart. I do not know where the line is, and I suspect it depends on how much the remaining judgment step is worth.

Answering either one properly would mean building the same task twice, deterministic and agentic, and measuring cost, latency, correctness, and the inputs the deterministic version cannot handle at all. I have not done that, so for now both stay open.

If you have hit different failure modes running agents in production, I would like to hear them.

TIL - Compatibility Direction in Schema Evolution

M. Alwi Sukra — Tue, 21 Jul 2026 06:08:21 +0000

This week I read DDIA Chapter 4, on encoding and evolution, and the part that stuck with me was how to reason about schema compatibility during a rolling upgrade.

In a large system, code changes don't happen all at once. We do a rolling upgrade, so old and new versions of the code run side by side for a while. For the system to keep working through that window, the data has to survive being read by whichever version happens to pick it up. That's what forward and backward compatibility are for.

Which direction we actually need isn't fixed. It depends on the dataflow, and on the order we roll things out.

Through a database

Say we add a field and update the code that reads and writes the row.

Backward compatibility is obviously needed: data written by the old code shouldn't break when the new code reads it. But forward compatibility is needed too, because during a rolling upgrade an old instance can read a row that a new instance already wrote.

And there's a twist a database adds that services don't have: old data just sits there. A row written two years ago is still there, in the old shape, and we have to keep supporting it. We could migrate (rewrite) the data to avoid that, but on a large database that's expensive, so most of the time we don't. So the database case needs both directions, and it needs them for a long time.

Through services (REST and RPC)

Now say we change a request or response schema. Here it splits by who's processing what.

On the server side, we need backward compatibility: the new server code has to handle old requests from clients that haven't upgraded yet. What we usually don't strictly need is forward compatibility, because we always deployed the server fully first, then told clients to upgrade. So an old server never sees a new request. I'd been doing this by instinct; the chapter just gave me the reason it holds, the rollout order is what makes that direction safe to skip.

On the client side it's the mirror image. The client needs forward compatibility: old client code has to handle new responses, because the server upgraded first and is already sending the new shape. It doesn't need backward compatibility, because by the time the client upgrades, the server is already new.

So "the direction we care about" was never really a rule about APIs. It was a consequence of how we rolled out. Server first, clients after.

Through message passing

Producer and consumer versions coexist, and we can't force the order the way we can with server-then-client. A newer consumer might read a message from an older producer, and an older consumer might read a message from a newer producer. So the consumer needs both directions.

And even if we decide to upgrade the producer fully first, we still can't drop forward compatibility, because messages live in the broker. They sit there for retries and replays, so a message written by the new producer can still be waiting when an old consumer picks it up. We can't tell a message already sitting in the queue to upgrade itself.

That's the difference. With an API, the rollout order is something we control. With a queue, the message outlives the moment it was written, so the order isn't ours to enforce anymore.

The summary

	Backward (old data → new code)	Forward (new data → old code)
Database	Yes	Yes
Service (server)	Yes	No, because server upgrades first
Service (client)	No, because server upgrades first	Yes
Message passing	Yes	Yes

The thing I took from it: I'd been handling both directions in practice, but as a set of habits, not a clear model. What this chapter gave me was the definition, backward and forward as two independent directions, and the reason the direction we need shifts with the dataflow and the rollout order. The moment the dataflow stops letting us control that order, like a message queue does, the shortcut of leaning on rollout order stops working, and we have to actually support both.

TIL - Picking a Database by Its Read/Write Pattern

M. Alwi Sukra — Sun, 14 Jun 2026 04:36:15 +0000

I've shipped systems on Postgres, BigTable, and a column store. If you asked me which one to reach for, I could answer from experience. But until I read DDIA Chapter 3, I couldn't have told you why they behave so differently, because I'd never actually thought about what the storage engine does with the bytes underneath.

That turned out to be the whole lesson. The internal data structure of a database isn't an implementation detail we can ignore. It's designed and tuned around a read/write access pattern. So once we understand the structures, "which database" stops being a vibe and starts being a consequence of how our workload reads and writes.

Two worlds: OLTP and OLAP

DDIA splits database workloads into two shapes. These aren't database types, they're descriptions of how a workload behaves:

OLTP (Online Transaction Processing): point reads/writes, low latency, high concurrency, few rows touched per query, needs fresh data. (The app serving user requests)
OLAP (Online Analytical Processing): large scans, few columns across many rows, throughput over latency, bulk/batch writes, tolerates staleness. (The analytics and reporting)

The thing that finally clicked for me: these labels are something we put on a workload after we've described it, not a bucket we pick first. And one system usually has both. The write path can be analytical-shaped while the read path is transactional-shaped.

Picking an engine is asking two questions about the workload

If the storage engine is tuned to the access pattern, then choosing one is really about describing how our workload reads and writes, then matching it. Two questions get us most of the way there.

One quick note before that. I'm only scratching the surface of each technology here. This is what clicked for me, not a deep dive into compaction internals or page layouts. If you want the full mechanics, the book itself (and plenty of good articles) go far deeper. Treat the names below as starting points to go read about, not as the last word.

Q1: write-heavy or read-heavy? (LSM vs B-tree)

Inside the OLTP world, there are two ways to organize data on disk, and the difference is entirely about how each one handles a write.

Log-structured (LSM-tree). New and updated data is appended. Writes land in an in-memory sorted structure (the memtable), which gets flushed to disk as an immutable, sorted file (an SSTable). Background compaction merges those files and throws away superseded values. Because every write is a sequential append, write throughput is high. The cost is read amplification, where a key might live in several SSTables, so reads check multiple files (Bloom filters exist to skip the ones that definitely don't have the key). Databases built this way: Cassandra, RocksDB, LevelDB, HBase, and BigTable.

Update-in-place (B-tree). Data lives in fixed-size pages, and an update rewrites the page where the key already is. Reads are predictable (a bounded number of page lookups), and this structure fits transactions cleanly. The cost is write amplification: writes are random I/O, and a single update often rewrites a whole page (plus the write-ahead log entry, and sometimes a page split). Databases built this way: Postgres, MySQL (InnoDB), and most traditional relational databases.

So the heuristic "LSM for write-heavy, B-tree for read-heavy or transactional" isn't a rule to memorize. It falls out of how each structure treats a write. That's the thesis in miniature: the engine is shaped by the access pattern. This is why Cassandra can absorb a firehose of writes while Postgres gives us clean transactions. They picked different sides of this tradeoff.

Q2: point access or analytical scan? (and why "row vs column" is the wrong axis)

This is the part that reframed how I think about it.

I used to assume the storage axis was "row-oriented vs column-oriented." After this chapter I think that's the wrong axis. The real distinction is whether our storage unit is explicitly keyed or positionally implied.

In a row store like Postgres or MySQL, a row is one keyed record with all its columns fused together. Update one column, and it rewrites the whole record.
In a wide-column store like BigTable, Cassandra, or HBase, the unit isn't row -> all columns. The on-disk key is the full cell coordinate (row key, column, timestamp) -> value. Each cell is independently keyed. A row's cells are sorted to sit next to each other (that's the only thing "row-oriented" really means here, just locality), but each cell is written and updated on its own.
In a column store like Parquet, ClickHouse, Vertica, or BigQuery, each column is a separate file of bare values, with no key stored next to each value. Row N is whatever sits at position N in every file. Position is the implicit key.

For example, let's say we have these rows:

row A: impression=11, click=12, cost=13
row B: impression=14, click=15, cost=16
row C: impression=17, click=18, cost=19

The storage visualization for each type is:

That single distinction explains the write behavior the orientation framing can't:

A row store updates one column by rewriting the whole row, since the columns are fused into a single keyed record. Cheap when you're touching one row at a time, but two writers updating different columns of the same row still contend on the same record.
A wide-column store can update one column of one row without touching the others, because the cell is its own keyed thing. Independent writers writing different columns to the same row never collide.
A column store can't cheaply insert one row, because there's no key to address it by. Position ties every value to its row, so inserting means realigning every file (and the compressed, sorted columns reject cheap in-place edits). That same positional layout is exactly what makes scans and compression spectacular.

Strength and weakness come from the same design decision. Keyed buys us independent writes and locality. Positional buys us compression and scan speed. Neither is "better." Each is tuned to a different access pattern.

The read side mirrors it:

A row store fetches one whole record in a single read, but scanning one column means dragging every full row off disk.
A wide-column store reads one row as a contiguous scan over its adjacent cells, but it's still reading row by row, not scanning one column across everything.
A column store reads one column across millions of rows by touching just that file, but reassembling one whole row means gathering a value from every file.

Point access wants keyed; analytical scan wants positional. Read and write pull the same direction, because they're the same access-pattern bet.

Where I'd been relying on this without naming it

A system I worked on stored predictions for an ads insights platform: three prediction types (impression uplift, click uplift, cost saving), produced by several suggestion engines, kept in BigTable. We did reason our way there from the read/write pattern. What this chapter gave me was sharper vocabulary for the thing we were already doing.

Run the workload through the two questions:

Writes were high-volume and batch: several engines aggregating offline, then loading results in. That's append-heavy ingest, the LSM side of Q1.
Reads were point and prefix lookups on a composite key like shop#group (read one group, or a range of groups under a shop). Not analytical scans across one column, so keyed, not positional, on Q2.

Append-heavy writes, keyed point and prefix reads. That shape fits a wide-column store like BigTable almost exactly, so that's where we landed. None of it is "row vs column" or "SQL vs NoSQL." It's just the workload described honestly, and the engine fell out of the description.

The fit wasn't perfect, either. Some read paths still needed online aggregation, which traded read latency for write-path simplicity, but that's a story for another post.

The part the access pattern doesn't decide

Here's what I only noticed reading this chapter. We assessed the read/write pattern, it pointed at BigTable, and we stopped there. I never actually asked the next question: could Postgres have done this too?

The honest answer is, probably yes.

Whole-row update contention from multiple writers? Postgres has a workaround. Split each writer's columns into separate tables keyed on (shop, group). No shared row, no contention.

Avoid joins on large tables? That join only exists because I split the tables. Keep one wide table and there's no join at all. And even the split join is cheap when it's indexed on a bounded set of rows.

The data too big for one machine? It wasn't, at the time. It fit comfortably on a single node.

So the thing that pointed me at BigTable, the access pattern, wasn't actually the thing that ruled Postgres out. Postgres could have served the same reads and writes. I just never ran the comparison, because I'd found something that fit and didn't look back.

That's the lesson I took from it. The access pattern narrows you to an engine class, and that narrowing is real and useful. But it doesn't rule out the alternatives within reach of that class. What actually separates BigTable from Postgres here is how each one grows past a single machine. Postgres is single-node by design: scaling out means sharding it yourself, picking a shard key, routing queries in the application, and rebalancing by hand as it grows. BigTable partitions automatically, splitting data into tablets by row-key range and spreading them across machines on its own. So once the data outgrows one node, Postgres turns into an operational project and BigTable just keeps going. That's the real dividing line, and it's the question I never got around to asking out loud.

Takeaway

So the honest shape of database selection, after this chapter:

The read/write access pattern selects the engine class: LSM vs B-tree, keyed vs positional. This part is real and reasoned, and it's what I actually did.
Scale and transactional needs finish the selection, and these are independent of the access pattern. Two stores can match the access pattern perfectly and differ entirely on whether they shard or do multi-row ACID. This is the step it's easy to skip once you've found something that fits.

The data structure follows the access pattern. The product follows the data structure plus scale.

This is the storage-engine layer of the decision. Replication, partitioning, and consistency (later chapters in the book) add their own factors, but those sit on top of this layer, they don't replace it.

TIL - Graph Thinking Without a Graph Database

M. Alwi Sukra — Thu, 28 May 2026 17:19:16 +0000

This week I read DDIA Chapter 2 related to data models. Most of it felt familiar. Relational vs document, many-to-many with junction tables, schema-on-read vs schema-on-write. These were things I had opinions about already.

But the graph data model section was a blind spot. I assumed graph databases were for social networks, interesting but not relevant to anything I was doing.

What graph data models actually are

One key aspect that the chapter emphasizes is how different data models handle many-to-many relationships. In a relational data model, we usually have several tables and a junction table that connects them. We also can add some additional columns to that junction table.

For a graph data model, we can think of it having 2 different tables: nodes and edges. Nodes are entities. Edges are relationships between them. Both can have properties.

There is not much difference between a relational and graph data model for a single relationship at a fixed depth. For example, a friendships table with user_a_id, user_b_id, since, is_close_friend is basically an edge with properties. Relational handles that fine.

The difference shows up when we start traversing.

Say we want "friends of friends". With a junction table, that's a self-join. "Friends of friends of friends" is another join. "Anyone reachable from me through any number of friendship hops" is a recursive CTE. It works, but the query complexity has nothing to do with how simple the question sounds.

In a graph query language, traversal is the native operation. Here's friends of friends in Cypher (Neo4j's query language):

MATCH (me:User {id: $userId})-[:FOLLOWS]->(friend)-[:FOLLOWS]->(fof)
RETURN fof

We can read it almost like a sentence: match the pattern where I follow a friend, who follows a friend-of-friend. The arrows are edges; [:FOLLOWS] is the edge type to traverse.

And arbitrary depth is just one more character:

MATCH (me:User {id: $userId})-[:FOLLOWS*]->(reachable)
RETURN reachable

The * means "follow any number of these edges." Same query shape whether it's one hop or ten. In SQL, that jump from fixed depth to arbitrary depth means rewriting our query as a recursive CTE.

The second difference is that a graph treats different relationship types uniformly. In relational, follows, blocks, and memberships are usually separate tables, and traversing across them means a different join per table. In a graph, they're all just edges, and we can traverse across types in a single pattern.

So my take is that the real distinction is the traversal. Especially variable-depth traversal across multiple relationship types. It's a first-class operation in a graph model and an awkward bolt-on in SQL.

The shape that fits

The chapter convinced me that graph models suit problems where:

Relationships are recursive or variable-depth (friends of friends, transitive dependencies, reachability)
Multiple paths can exist between the same two entities
The type of relationship matters as much as the entities themselves
Queries are about traversal and reachability, not just lookup

If our data is mostly "fetch a row by ID" or "join two tables on a foreign key", relational is fine. But the moment we start asking "what's reachable from here through any valid path?", that's a graph question, whether we store it in a graph database or not.

Looking at my own work through this lens

I worked on an ads management system. The schema looked like this:

Some queries this service needed to answer:

Find all keywords in a shop.
Find all keywords in a group.
Find all keywords in an ad. (Keywords directly on the ad, plus keywords on the group that contains it.)

Reasonable schema, queries, and I'd worked with this code. But when I tried drawing the data as a graph, this is what I got:

There are multiple paths from a shop to a keyword: through an ad group, through an ad, through both. When I query "all keywords in a shop", I'm doing a graph traversal: "find every Keyword reachable from this Shop through any path of contains edges". I just hadn't been calling it that.

What actually changed for me

1. I started thinking about reachability instead of joins

Before: every question about the data felt like a question about which tables to join. To get keywords in a shop, I join keyword tables with ad/ad_group tables and filter by shop_id. The query was a sequence of join operations.

After: every question about the data feels like a question about which nodes are reachable from which. Find all keyword nodes reachable from this shop. The traversal is the question, and the join is just one implementation of the traversal.

The shift sounds subtle, but it's what made other approaches (denormalization, recursive CTEs, even just rephrasing the SQL) become visible. Once the question is "what's reachable from here?", the answer doesn't have to be "join these tables." It can be anything that gets us the same set of reachable nodes.

2. I noticed the multiple paths problem

In a tree, every node has exactly one parent. The hierarchy I was working with isn't a tree. A keyword can be reached from a shop through an ad group, or through an ad, or through both. An ad can belong to a shop directly, or be inside a group.

I'd been treating this as a quirk of the schema. The nullable columns, the join table, the two separate keyword tables, these were just "how things are." But the graph lens names it clearly: the data has multiple paths between the same kinds of nodes. That's a structural property, not a quirk.

And it explains why my SQL queries kept needing unions. Each UNION branch is one path. The graph is telling me up front that I'm going to need multiple branches; the schema was hiding that until the query made it visible.

3. I saw edges that weren't in my schema

The graph diagram has a contains edge from ad_group to ad. In my schema, that relationship lives in the ad_group_ad junction table.

But the graph also has implicit relationships that my schema doesn't model directly. The "keyword in shop X" relationship is real (and we query for it constantly), but no column or table represents it directly. It's a derived relationship, computed every time we run the traversal.

That's where the option space opens up. Once I can see shop-to-keyword as a meaningful relationship, I can ask whether to materialize it (denormalize shop_id onto every keyword) or keep deriving it (current approach with traversal). Both are valid; the graph view is what made the choice visible.

What it would look like as a graph query

In Cypher, "all keywords in a shop" is:

MATCH (s:Shop {id: $shopId})-[:CONTAINS*]->(k:Keyword)
RETURN k

In our actual schema, the same query takes two branches:

-- Keywords on groups in this shop
SELECT k.* FROM ad_group_keyword k
JOIN ad_group g ON k.ad_group_id = g.id
WHERE g.shop_id = $shopId

UNION

-- Keywords on ads in this shop
SELECT k.* FROM ad_keyword k
JOIN ad a ON k.ad_id = a.id
WHERE a.shop_id = $shopId AND a.status = 'active'

It works. But this query got slow on us in a way that took us a while to understand.

The issue wasn't the JOIN itself, or the UNION. It was the query planner. When a shop has many keywords, the planner sometimes picks an index path that ends up scanning across the ad tables (soft-deleted rows included), even when the request is for a small page of results.

The behavior was hard to predict because it depended on the shop's data distribution. Small shops were fine. Large shops sometimes triggered the bad path. And because the keyword list is a batch API, a single slow query multiplied across the batch and put real pressure on the database.

The team's fix was to stop letting the planner choose. We took the joins out of SQL and resolved them in application code: query each table separately with WHERE shop_id = X AND status = 'active' (which uses clean indexes predictably), then stitch the results in Go.

It works. But it's a workaround for a query that's conceptually one thing: find all keywords reachable from this shop. The graph traversal is happening, just spread across multiple queries and some application code, with the planner taken out of the loop entirely.

Would I actually use a graph database?

Constraints:

The hierarchy is only 3-4 levels deep. Graph databases shine on deep or unbounded traversal. Mine is bounded.
Nobody on the team has run Neo4j in production. PostgreSQL we know cold.
Our source of truth is already in Postgres. Adding a graph database means syncing two stores or migrating the source of truth, both big commitments.
Our queries are predictable. We're not doing pattern matching or shortest-path.

I'm not sure I'd reach for Neo4j here. The elegance of the Cypher query is real, but the operational cost feels high for the shape of problem I have.

What's more interesting is that the graph lens opened up another option.

A different option

What if every entity in the hierarchy carried its ancestor IDs directly?

ad:               id, shop_id, ad_group_id (nullable), status
ad_group_keyword: id, ad_group_id, tag, shop_id
ad_keyword:       id, ad_id, shop_id, ad_group_id (nullable), status

ad_group_ad still records the actual ad-to-group membership, but ad.ad_group_id is a maintained denormalization. Same idea on the keyword tables: each keyword carries shop_id, and ad_keyword also carries ad_group_id and status. The ID columns are nullable where the relationship doesn't apply.

Now the "find" queries get simpler:

-- All keywords in a shop
SELECT * FROM ad_group_keyword WHERE shop_id = $shopId
UNION
SELECT * FROM ad_keyword WHERE shop_id = $shopId AND status = 'active'

-- All keywords in a group
SELECT * FROM ad_group_keyword WHERE ad_group_id = $groupId
UNION
SELECT * FROM ad_keyword WHERE ad_group_id = $groupId AND status = 'active'

-- All keywords in an ad
-- Step 1: get the ad's group (single PK read)
SELECT ad_group_id FROM ad WHERE id = $adId
-- Step 2: pull keywords from both sources
SELECT * FROM ad_keyword WHERE ad_id = $adId AND status = 'active'
UNION
SELECT * FROM ad_group_keyword WHERE ad_group_id = $adGroupId

These are direct lookups on indexed columns. There's no join for the planner to mis-optimize, no intermediate result set whose size depends on data distribution. The query behavior is the same whether a shop has 50 keywords or 50,000.

The third case is two reads, but step 1 is just a primary-key lookup on the ad row we'd usually be fetching anyway.

The cost is that denormalization is now a system, not a single column. Moving an ad updates ad_group_id across its keywords; soft-deleting an ad updates status across its keywords. Two operations, both fan out to the keyword rows, both have to be transactional or the data drifts.

Drift is the part that worries me. The query patterns get dramatically cleaner, but the write paths get more places they could go wrong. Six months later, someone adds a new way to move ads between groups and forgets to update the keywords. The reads quietly return wrong results. So I'm not sure denormalization is the answer either.

What surprised me is that I'd been treating it as a normalization problem ("where should the foreign keys go?") instead of a modeling problem ("what shape does the data actually have?"). The graph perspective is what reframed it for me.

The takeaway

The data was always graph-shaped. The queries were always graph queries. The schema and the application code were doing graph work without the vocabulary to describe it. And once I could see the shape, alternatives I hadn't been considering became visible, even if I haven't decided which one is right.

I'm not sure I'll ever reach for a graph database. But learning about them is already changing how I think about modeling, even though I'm not using one.

TIL - What Response Time Metrics Really Mean

M. Alwi Sukra — Sun, 10 May 2026 07:55:53 +0000

I always thought high percentiles didn't really matter, they only impact a small number of users, right? I interpreted them as the worst case (something unlikely to affect most users).

This week I read DDIA and came across the part describing how Amazon sets their response time requirements at p99.9. That means the requirement is based on 1 in 1000 users :). But the reason is something I never thought of: the users in the high percentiles are most likely the ones with the most data, which makes them important users for Amazon.

I reflected on this with my experience working on an Ads Platform. Some processes were slow and it was almost always the same small group of users with many ads, which I assume also correlates with ads revenue contribution. I wonder if we had designed the system around those high-percentile users, maybe we could have made the platform better for all users, and best for our most important sellers.

Response time isn't the same as latency

I don't know why, but somehow I just know that response time and latency are different:

Service time: how long the server actually spends processing the request.
Latency: time the request spends waiting (queued, in transit, blocked).
Response time: what the caller sees: service time + network + queueing + everything else.

Response time is from the caller's perspective. Service time is from the callee's. They're almost never equal. I think this is important because most of us only track one side.

Average hides the shape

Response times aren't a single number, they're a distribution. Most requests are fast, a few are very slow, and the average sits somewhere awkward between them.

Average doesn't tell us how many users actually experienced the delay. An average of 200ms can mean everyone gets ~200ms, or that most get 50ms while a few get 2 seconds. The average doesn't tell us which one we have.

That's why averages aren't enough. We need a metric that respects the shape.

Percentiles, properly

A percentile shows "what response time were X% of requests faster than?"

p50: half were faster.
p95: 95% were faster, 5% were slower.
p99: 99% were faster, 1% were slower.
p99.9: 99.9% were faster, 0.1% were slower.

If p99 = 500ms, it means 1 out of every 100 requests took longer than 500ms. That's the part I used to dismiss as noise.

Which percentile to chase

Once we accept the tail matters, the next question is how far in?

Honestly, I don't know how to answer the question. Maybe the choice isn't really technical and it's a business question: which users have we decided to serve well? p99 means we're serving 99% of requests well. p99.9 means we're including heavy users (the ones who, going back to the Amazon insight, probably matter most).

A few things I wish I'd known earlier

Measure at both caller and callee. Callee might report p99 = 50ms while caller sees p99 = 300ms for the same calls. The 250ms gap is in the network, the connection pool, queueing, or the caller's own thread pool. If we only look at one side, we miss it.
Timeouts decouple the metrics. If the caller times out at 200ms and the callee takes 500ms, the callee's dashboard shows a successful 500ms response to a request the caller already gave up on. Both metrics are technically correct but are misleading on their own.
Don't average percentiles across servers. This is my second confession. For years, when our dashboard showed p99 from multiple servers, I'd take the average of those numbers and call it our "global p99." That's mathematically meaningless. The average of ten p99s is not the p99 of the combined population. The right way is to merge the underlying histograms first, then compute the percentile.

Takeaway

A metric isn't just a number. It's a statement about which users we've decided to serve well.

An average says "I care about the typical user." p99 says "I care about almost everyone." p99.9 says "I care about the heavy users too, the ones who probably matter most to the business."

For years, I was implicitly choosing the first one without realizing I was choosing anything.