DEV Community: Ebrahim Arian

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 5: Coordinating Two Zones at Once

Ebrahim Arian — Sat, 08 Aug 2026 01:24:29 +0000

This is Part 5, the last part of this series. Part 4 let a human step into the loop before a risky decision executes. Every part so far, though, has shared one assumption: a zone is evaluated completely on its own.

That assumption hides a real inconsistency. driver_bonus and surge_pricing both attract "new drivers," as if from an unlimited outside pool. But a real regional driver pool is finite, and it's shared. Two zones both running an aggressive incentive at the same hour can't both be right about where their new drivers are actually coming from. Part 5 is the first part where the agent has to notice that.

Two zones, evaluated together every cycle — fixed at 2, not a general N, more on that below. Each zone gets two new supply channels, on top of its existing zone-local response:

A local dormant pool. Off-platform drivers a zone can entice with a big enough incentive. It's fully local, finite, and it depletes as it's used.
A cross-zone pull request. Asking to draw from the adjacent zone's genuine surplus. This is only ever a request at evaluation time — granting it depends on a resource the requesting zone doesn't unilaterally control, so nothing is finalized until both zones' evaluations are in.

One interrupt, not three. Part 4 had three interrupt categories: editing, approval, debugging. Part 5 keeps only approval, gated on a new condition specific to this stage: does a zone's cross-zone pull request exceed what the adjacent zone can comfortably spare? "Comfortably" is the key word — it's a conservative line, not a hard limit, so crossing it isn't dangerous, just tighter than usual. Most cycles, the answer is no, and both zones' policies run straight through. When a request does cross that line, a human decides: approve it anyway, reject it back down to the comfortable amount, or override it with a specific number.

The Graph

                 ┌──▶ evaluate_zone_a ──┐
start_cycle ─────┤                      ├──▶ reconcile_and_approve ──▶ simulate_and_report
                 └──▶ evaluate_zone_b ──┘        (gate: cross-zone request > safely_pullable)

Fixed at exactly 2 zones, not a general N. With a known, fixed number of parallel branches, two plain static edges are the right tool. LangGraph's Send API is for when the number of branches is only decided at runtime — that's not the case here.

Why each zone's evaluation is its own compiled sub-graph, invoked as one atomic node, instead of two duplicate top-level nodes. Look at the per-zone chain: a balanced zone takes the short branch (trivial_do_nothing) and finishes in a handful of steps. A deficit or surplus zone takes the long branch, including an LLM call, and needs several more. Same chain, different path through it — decided by route_llm_or_skip.

If those steps were exposed directly as top-level nodes duplicated per zone — detect_imbalance_a/_b, classify_severity_a/_b, and so on, all feeding into one shared reconcile_and_approve — the two zones would finish in a different number of steps in the same cycle. LangGraph's fan-in only reliably waits for parallel branches that complete in the same step. It does not wait for branches that legitimately take longer. A short branch finishing early would fire the fan-in node immediately, using whatever partial state the slower branch happened to have at that moment, not once both zones were actually done.

Wrapping each zone's whole chain as a sub-graph, invoked from a single parent-level node, hides that difference entirely — the same way calling a Python function hides how many lines ran inside it from the caller. Whether a zone took 4 internal steps or 7, the parent graph only ever sees one call that returns a result. Both branches become exactly one parent-level step each, which is the case LangGraph's fan-in handles correctly.

reconcile_and_approve is the only node with visibility into both zones at once. It's where the new coordination mechanic actually happens, and where the interrupt lives.

The State

The per-zone state (ZoneEvalState) is identical to Part 2's state. That's deliberate — each zone's evaluation reuses Part 1 and Part 2's nodes unchanged:

class ZoneEvalState(TypedDict):
    zone:                dict
    ops_note:            str
    imbalance_ratio:     float
    imbalance_type:      str
    severity:            str
    candidate_policies:  list
    policy_evaluations:  dict
    policy_resolutions:  dict
    recommended_policy:  str
    explanation:         str
    messages:            Annotated[list[BaseMessage], operator.add]

The parent state holds both zones' fields side by side, each suffixed _a/_b, rather than a nested {zone_name: {...}} structure. With exactly two zones, state["severity_a"] is a direct, one-line read off a result dict — no lookup, no if zone_name == ... branching, no risk of a typo'd key silently returning nothing instead of raising. That's a deliberate trade specific to a fixed count of 2. It wouldn't hold up for a general N zones, where a list or a dict keyed by zone name is the right shape instead — the fixed two-zone scope is exactly what makes the suffixed approach the more readable choice here, not a general pattern to reach for by default:

class AgentState(TypedDict):
    zone_a: dict
    zone_b: dict
    ops_note_a: str
    ops_note_b: str
    initial_hour: int
    cycle_number: int
    history_a: Annotated[list[dict], operator.add]
    history_b: Annotated[list[dict], operator.add]
    imbalance_ratio_a: float
    imbalance_ratio_b: float
    imbalance_type_a: str
    imbalance_type_b: str
    severity_a: str
    severity_b: str
    candidate_policies_a: list
    candidate_policies_b: list
    policy_evaluations_a: dict
    policy_evaluations_b: dict
    policy_resolutions_a: dict
    policy_resolutions_b: dict
    recommended_policy_a: str
    recommended_policy_b: str
    explanation_a: str
    explanation_b: str
    messages_a: Annotated[list[BaseMessage], operator.add]
    messages_b: Annotated[list[BaseMessage], operator.add]
    cross_zone_pull_final_a: float
    cross_zone_pull_final_b: float
    outcome_a: dict
    outcome_b: dict
    outcome_delta_a: dict
    outcome_delta_b: dict
    report_a: str
    report_b: str

The Per-Zone Sub-Graph

Only one node here is genuinely new: resolved_imbalance_regional. It calls evaluate_policy_regional — the dormant-pool, cross-zone-aware version — instead of Part 1's plain evaluate_policy:

def resolved_imbalance_regional(state: ZoneEvalState) -> ZoneEvalState:
    zone = state["zone"]
    evaluations, resolutions = {}, {}
    for policy in state["candidate_policies"]:
        result = evaluate_policy_regional(zone, policy, noise=False)
        evaluations[policy] = result["profit"]
        resolutions[policy] = "N/A" if state["severity"] == "none" else result["resolved"]
    return {"policy_evaluations": evaluations, "policy_resolutions": resolutions}

Same shape and role as Part 1's resolved_imbalance: evaluate every candidate once, record its profit, record whether it resolves the imbalance. Only the function it calls is different.

The rest of the sub-graph is wired exactly like Part 2's build_agent — same nodes, same edges, this new node dropped in where resolved_imbalance used to be. So it isn't rebuilt from scratch a third time. _build_zone_evaluator does that wiring, and each zone gets its own instance:

def _build_zone_evaluator(llm_with_reconcile_tool, llm):
    def _reconcile(state):
        return reconcile_inputs(state, llm_with_reconcile_tool)

    def _explain(state):
        return generate_explanation(state, llm)

    g = StateGraph(ZoneEvalState)
    g.add_node("detect_imbalance",     detect_imbalance)
    g.add_node("classify_severity",    classify_severity)
    g.add_node("set_candidates",       set_candidates)
    g.add_node("trivial_do_nothing",   trivial_do_nothing)
    g.add_node("reconcile_inputs",     _reconcile)
    g.add_node("resolved_imbalance",   resolved_imbalance_regional)
    g.add_node("choose_best_policy",   choose_best_policy)
    g.add_node("generate_explanation", _explain)

    g.add_edge(START, "detect_imbalance")
    g.add_edge("detect_imbalance",  "classify_severity")
    g.add_edge("classify_severity", "set_candidates")
    g.add_conditional_edges("set_candidates", route_llm_or_skip, {
        "trivial_do_nothing": "trivial_do_nothing",
        "reconcile_inputs":   "reconcile_inputs",
    })
    g.add_edge("reconcile_inputs",     "resolved_imbalance")
    g.add_edge("resolved_imbalance",   "choose_best_policy")
    g.add_edge("choose_best_policy",   "generate_explanation")
    g.add_edge("generate_explanation", END)
    g.add_edge("trivial_do_nothing",   END)

    return g.compile()

The Parent Graph

reconcile_and_approve is where the actual coordination happens, and where interrupt() gets called for this stage. It's the one node worth seeing in full:

def reconcile_and_approve(state: AgentState) -> AgentState:
    requested = {}
    for suffix in ("_a", "_b"):
        zone = state[f"zone{suffix}"]
        policy = state[f"recommended_policy{suffix}"]
        result = evaluate_policy_regional(zone, policy, noise=False)
        requested[suffix] = result.get("cross_zone_pull_requested", 0.0)

    zone_name = {s: state[f"zone{s}"]["zone_name"] for s in ("_a", "_b")}
    safe_cap, conflicts = {}, []
    for suffix in ("_a", "_b"):
        if requested[suffix] <= 0:
            continue
        cap = safely_pullable(state[f"zone{_OTHER[suffix]}"])
        safe_cap[suffix] = cap
        if requested[suffix] > cap:
            conflicts.append({
                "zone": zone_name[suffix],
                "adjacent_zone": zone_name[_OTHER[suffix]],
                "policy": state[f"recommended_policy{suffix}"],
                "requested": round(requested[suffix], 2),
                "safe_cap": round(cap, 2),
            })

    finalized = dict(requested)

    if conflicts:
        answer = interrupt({
            "kind": "cross_zone_approval",
            "question": (
                "A zone's cross-zone driver pull would exceed what the adjacent "
                "zone can safely give up. For each conflict: approve (grant the "
                "full request anyway), reject (cap it at the safe amount), or "
                "override (set a specific amount)."
            ),
            "conflicts": conflicts,
        })
        decisions = {d["zone"]: d for d in (answer or {}).get("decisions", [])}
        for suffix in ("_a", "_b"):
            if suffix not in safe_cap or requested[suffix] <= safe_cap[suffix]:
                continue
            decision = decisions.get(zone_name[suffix], {"action": "approve"})
            action = decision.get("action", "approve")
            if action == "reject":
                finalized[suffix] = safe_cap[suffix]
            elif action == "override":
                finalized[suffix] = decision["amount"]
            # action == "approve" (or unspecified): keep the full requested amount

    return {
        "cross_zone_pull_final_a": round(finalized["_a"], 2),
        "cross_zone_pull_final_b": round(finalized["_b"], 2),
    }

Both zones have already picked a policy by the time this node runs. It checks each zone's cross-zone request against what the other zone can actually spare, using safely_pullable(). That function returns a conservative number — how many drivers a zone can give up while staying at or above a comfortably balanced ratio, not the absolute most it could survive losing. Crossing that line doesn't mean anything breaks. It means the adjacent zone would end up running tighter than its comfort margin, which is exactly the kind of tradeoff worth a second opinion instead of a hard rule. evaluate_policy_regional itself never caps a request against the adjacent zone — deliberately. This is the only place that check happens.

It only pauses when a request genuinely crosses that comfortable amount. Otherwise, both zones' requests are granted in full, and the cycle runs straight through — the same "only interrupt when it matters" principle as Part 4.

Now the parent graph itself, wrapped in a build(checkpointer) function rather than built inline once — the same reusable pattern as Part 4's build(), keeping the graph and its checkpointer cleanly separable. evaluate_zone_a/evaluate_zone_b are the "sub-graph as atomic node" piece — each one just calls .invoke() on the compiled per-zone sub-graph, reading and writing only its own zone's suffixed fields:

def build(checkpointer):
    llm = ChatOllama(model="qwen2.5:14b", temperature=0)
    zone_evaluator = _build_zone_evaluator(llm.bind_tools([report_context]), llm)

    def _zone_view(state, suffix):
        return {f: state.get(f"{f}{suffix}") for f in ZoneEvalState.__annotations__}

    def _suffix(update, suffix):
        return {f"{k}{suffix}": v for k, v in update.items()}

    def evaluate_zone_a(state):
        return _suffix(zone_evaluator.invoke(_zone_view(state, "_a")), "_a")

    def evaluate_zone_b(state):
        return _suffix(zone_evaluator.invoke(_zone_view(state, "_b")), "_b")

    g = StateGraph(AgentState)
    g.add_node("start_cycle",           start_cycle)
    g.add_node("evaluate_zone_a",       evaluate_zone_a)
    g.add_node("evaluate_zone_b",       evaluate_zone_b)
    g.add_node("reconcile_and_approve", reconcile_and_approve)
    g.add_node("simulate_and_report",   simulate_and_report)

    g.add_edge(START, "start_cycle")
    g.add_edge("start_cycle", "evaluate_zone_a")
    g.add_edge("start_cycle", "evaluate_zone_b")
    g.add_edge("evaluate_zone_a", "reconcile_and_approve")
    g.add_edge("evaluate_zone_b", "reconcile_and_approve")
    g.add_edge("reconcile_and_approve", "simulate_and_report")
    g.add_edge("simulate_and_report",   END)

    return g.compile(checkpointer=checkpointer)


memory = MemorySaver()
app = build(memory)

A 10-Cycle Run: The Agent Asks, You Decide

Downtown Core and Midtown, set up so a real cross-zone conflict is likely — not guaranteed. reconcile_and_approve computes each zone's request against how much the adjacent zone can comfortably spare, fresh every cycle, and only pauses when a request crosses that line. Nothing here is scripted to pause on a specific cycle; whether it pauses, and how many times, comes entirely from how the two zones actually evolve.

In the notebook, this is the one interactive example in this stage: run a cell to start the thread, and it advances automatically — cycle by cycle, printing each one — until it hits a real conflict, and then it actually stops. You read the pause, then run the next cell to decide: approve, reject, or override. That resumes the thread and keeps advancing automatically again, until either the next conflict stops it or the run reaches cycle 10. Whenever a new pause shows up, you rerun that same "decide" cell with a fresh decision. Here's what one full run looks like, approving every conflict:

[Cycle 1] [Downtown Core] policy=surge_pricing | driver_count 6→8  | resolved=YES
[Cycle 1] [Midtown]       policy=surge_pricing | driver_count 3→4  | resolved=NO
  -> 2 conflicts this cycle: Downtown Core and Midtown each ask for more than the other's comfortable buffer. Approved anyway — both zones can absorb running a bit tighter.

[Cycle 2] [Downtown Core] policy=do_nothing    | driver_count 8→9
[Cycle 2] [Midtown]       policy=surge_pricing | driver_count 4→4  | resolved=NO
  -> Midtown's request crosses Downtown Core's comfortable buffer. Approved anyway.

[Cycle 3] [Downtown Core] policy=do_nothing    | driver_count 9→11
[Cycle 3] [Midtown]       policy=surge_pricing | driver_count 4→5  | resolved=YES
  -> Midtown's request crosses Downtown Core's comfortable buffer. Approved anyway.

[Cycles 4-8] both zones settle into do_nothing, no conflicts — the regional channels simply aren't needed

[Cycle 9] [Downtown Core] policy=do_nothing    | driver_count 12→10
[Cycle 9] [Midtown]       policy=surge_pricing | driver_count 5→5  | resolved=YES
  -> Midtown's request crosses Downtown Core's comfortable buffer. Approved anyway.

[Cycle 10] [Downtown Core] policy=do_nothing   | driver_count 10→9
[Cycle 10] [Midtown]       policy=do_nothing   | driver_count 5→5

Five real conflicts fired across these 10 cycles — two at once on cycle 1 (both zones asking at the same time), then cycle 2, cycle 3, a five-cycle quiet stretch, then one more at cycle 9. None of that count was decided in advance. It's the coordination check catching ordinary demand drift eroding each zone's slack, exactly when the numbers say it should, and staying silent the rest of the time.

Decide differently at the first pause in the notebook — reject instead of approve — and everything downstream changes with it. A rejected pull caps the zone at the comfortable amount instead of the full request, which changes its driver count, which changes what it's able to do next cycle, including whether the later pauses even happen the same way.

What We Built

	Part 4	Part 5
Scope	One zone, reviewed by a human	Two zones, coordinated with each other
New LangGraph concept	`interrupt()` / `Command`	Parallel fan-out/fan-in, sub-graph as atomic node
Supply model	Zone-local only	+ a local dormant pool, + a cross-zone pull request
What's shared	Nothing	Each zone's safely-pullable surplus, contended by the other
Interrupt gate	`severity == "critical"`	A cross-zone request crossing the adjacent zone's comfortable buffer
Interrupt categories used	Approval, editing, debugging (3)	Approval only (1), deliberately

The single-zone decision core — detect_imbalance, classify_severity, choose_best_policy — is still exactly Part 1's code, still untouched. What changed is that a policy's supply effect is no longer assumed to come from nowhere. It's either a finite local pool that visibly depletes, or a real request against a neighbor that has to be reconciled, not just asserted.

Five Parts, Five Additions

That's the series. Each part added exactly one thing the previous one genuinely couldn't do:

Part 1 — rules, because the problem started out fully structured.
Part 2 — an LLM, for exactly two jobs a lookup table couldn't do.
Part 3 — memory, because one decision often wasn't enough.
Part 4 — a human pause, because automatic wasn't always safe.
Part 5 — coordination, because a zone's supply was never really its own.

None of it showed up before the problem actually demanded it. That was the whole point.

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 4: Letting a Human Step In

Ebrahim Arian — Sat, 08 Aug 2026 01:24:26 +0000

This is Part 4 of a 5-part series. Part 3 gave the agent memory. It can now run for hours on its own, cycle after cycle, without forgetting what it already tried.

That's exactly the problem. A genuinely severe deficit might call for an aggressive surge multiplier, or a costly driver bonus. Right now, the agent applies whatever it decides immediately. Nobody has looked at it first. Nothing pauses for a second opinion, no matter how expensive the call.

Part 4 adds that pause. It uses LangGraph's dynamic interrupt():

A node calls interrupt(payload) — but only conditionally, when it actually wants to pause.
Execution stops at that exact point.
app.invoke(Command(resume=answer), config=thread) continues from there. answer becomes interrupt()'s return value, right at the call site.

Two gated pauses, not one:

Early — request_data_edit. Gated on cycle_number == 1. A human validates or corrects the raw starting snapshot once, before detect_imbalance even runs. Later cycles use the agent's own carried-forward numbers, not a fresh unverified reading — so re-confirming every cycle would just be noise.
Late — request_approval. Gated on severity == "critical". A wrong automatic call is most expensive in a critical deficit, so that's where a human checkpoint earns its cost. Mild, moderate, balanced, and surplus cycles still run fully autonomously, exactly like Part 3.

One property worth naming: no new fields got added to the state. Every interrupt here only reads state Part 3 already has, and writes back a field Part 3 already has — zone, recommended_policy, explanation. Human review is a new control-flow capability. It's not new data.

The Graph

start_cycle ──▶ apply_scheduled_conditions ──▶ request_data_edit ──▶ detect_imbalance ──▶ classify_severity ──▶ set_candidates
   (gate: cycle_number == 1)                                                                                          │
                                                                                                           ┌──────────┴──────────┐
                                                                                                           ▼ (balanced)          ▼ (deficit/surplus)
                                                                                                   trivial_do_nothing      reconcile_inputs  ← LLM #1
                                                                                                           │                     ▼
                                                                                                           │              resolved_imbalance
                                                                                                           │                     ▼
                                                                                                           │              choose_best_policy
                                                                                                           │                     ▼
                                                                                                           │              generate_explanation ← LLM #2
                                                                                                           └──────────┬──────────┘
                                                                                                                       ▼
                                                                                                             request_approval
                                                                                                        (gate: severity == "critical")
                                                                                                                       ▼
                                                                                                              simulate_and_report

Everything from detect_imbalance through generate_explanation is unchanged from Part 3 — same nodes, reused directly. Two new nodes bracket that core: request_data_edit right after the cycle starts, request_approval right before the outcome is simulated.

Both are plain Python. And both return {} on any cycle where their gate doesn't apply — so an ungated cycle looks exactly like Part 3.

The Nodes That Pause

Each one is a plain function with an if guard at the top. No LangGraph machinery decides whether to pause. The node itself does, then calls interrupt(payload) when it wants to:

def request_data_edit(state: AgentState) -> AgentState:
    if state["cycle_number"] != 1:
        return {}

    zone = state["zone"]
    answer = interrupt({
        "kind": "data_edit",
        "question": (
            f"Review the starting snapshot for {zone['zone_name']}. "
            "Resume with {'corrections': {...}} to fix fields, or {'corrections': {}} to accept it."
        ),
        "zone": zone,
    })
    corrections = answer.get("corrections", {})
    if not corrections:
        return {}
    return {"zone": {**zone, **corrections}}

interrupt()'s return value becomes whatever gets passed to Command(resume=...) — here, a corrections dict.

One real gotcha, worth knowing before you use this: Command(resume={}) is falsy in Python. LangGraph treats a falsy resume as if no answer was given at all. It just re-fires the same interrupt, instead of continuing. Always resume with a non-empty dict — {"corrections": {}} is how you accept the snapshot as-is.

The late pause has the same shape, but a real decision to make:

def request_approval(state: AgentState) -> AgentState:
    if state["severity"] != "critical":
        return {}

    answer = interrupt({
        "kind": "approval",
        "question": (
            f"Critical imbalance in {state['zone']['zone_name']} (cycle {state['cycle_number']}). "
            "Approve, reject, or override the recommended policy."
        ),
        "zone_name": state["zone"]["zone_name"],
        "cycle": state["cycle_number"],
        "imbalance_ratio": state["imbalance_ratio"],
        "recommended_policy": state["recommended_policy"],
        "explanation": state["explanation"],
        "policy_evaluations": state["policy_evaluations"],
        "candidate_policies": state["candidate_policies"],
    })
    action = (answer or {}).get("action", "approve")

    if action == "reject":
        return {
            "recommended_policy": "do_nothing",
            "explanation": "Rejected by human reviewer — falling back to do_nothing.",
        }
    if action == "override":
        chosen = answer["policy"]
        return {
            "recommended_policy": chosen,
            "explanation": f"Overridden by human reviewer: {chosen} chosen instead of "
                            f"{state['recommended_policy']}.",
        }
    return {}  # approve — no change

Three possible resume values:

{"action": "approve"} — keep the recommended policy.
{"action": "reject"} — fall back to do_nothing.
{"action": "override", "policy": "<name>"} — force a specific policy.

Building It

Same graph as Part 3, with these two nodes inserted at the right points. It's wrapped in a build(checkpointer) function, rather than built inline once — the next section needs a second graph object sharing the exact same checkpointer. The structure never changes. Only whether it's the same Python object does:

def build(checkpointer):
    llm = ChatOllama(model="qwen2.5:14b", temperature=0)
    llm_with_reconcile_tool = llm.bind_tools([report_context_and_schedule])

    def _reconcile(state):
        return reconcile_inputs(state, llm_with_reconcile_tool)

    def _explain(state):
        return generate_explanation(state, llm)

    g = StateGraph(AgentState)
    g.add_node("start_cycle",                start_cycle)
    g.add_node("apply_scheduled_conditions", apply_scheduled_conditions)
    g.add_node("request_data_edit",          request_data_edit)
    g.add_node("detect_imbalance",           detect_imbalance)
    g.add_node("classify_severity",          classify_severity)
    g.add_node("set_candidates",             set_candidates)
    g.add_node("trivial_do_nothing",         trivial_do_nothing)
    g.add_node("reconcile_inputs",           _reconcile)
    g.add_node("resolved_imbalance",         resolved_imbalance)
    g.add_node("choose_best_policy",         choose_best_policy)
    g.add_node("generate_explanation",       _explain)
    g.add_node("request_approval",           request_approval)
    g.add_node("simulate_and_report",        simulate_and_report)

    g.add_edge(START, "start_cycle")
    g.add_edge("start_cycle", "apply_scheduled_conditions")
    g.add_edge("apply_scheduled_conditions", "request_data_edit")
    g.add_edge("request_data_edit", "detect_imbalance")
    g.add_edge("detect_imbalance",  "classify_severity")
    g.add_edge("classify_severity", "set_candidates")
    g.add_conditional_edges("set_candidates", route_llm_or_skip, {
        "trivial_do_nothing": "trivial_do_nothing",
        "reconcile_inputs":   "reconcile_inputs",
    })
    g.add_edge("reconcile_inputs",     "resolved_imbalance")
    g.add_edge("resolved_imbalance",   "choose_best_policy")
    g.add_edge("choose_best_policy",   "generate_explanation")
    g.add_edge("generate_explanation", "request_approval")
    g.add_edge("trivial_do_nothing",   "request_approval")
    g.add_edge("request_approval",     "simulate_and_report")
    g.add_edge("simulate_and_report",  END)

    return g.compile(checkpointer=checkpointer)


memory = MemorySaver()
app = build(memory)

Correcting the Starting Snapshot

Cycle 1 always pauses at request_data_edit. Downtown Core's raw snapshot understates its driver count here — it's recorded as 9, but it was actually 15.

Reaching the pause:

downtown = get_zone(zones, DEFAULT_ZONE_NAME, driver_count=9)
r1 = app.invoke(make_initial_state(downtown), config=thread_edit)

PAUSED — kind=data_edit
question: Review the starting snapshot for Downtown Core. Resume with {'corrections': {...}} to fix fields, or {'corrections': {}} to accept it.
zone as recorded: driver_count=9, rider_request_count=16
uncorrected ratio: 1.78

At 1.78, that's a deficit. But it's a wrong one — a data problem, not a real supply problem. Correcting it and resuming:

MY_CORRECTION = {"driver_count": 15}
r2 = app.invoke(Command(resume={"corrections": MY_CORRECTION}), config=thread_edit)

(no interrupt — cycle ran straight through)
ratio now: 1.07 (balanced)
[Cycle 1] [Downtown Core] ratio=1.07 | severity=none | policy=do_nothing | wait 3.3min → 4.6min | resolved=N/A

One correction, and the zone flips from deficit to balanced. That's the entire point of this pause — catching a bad reading before it drives a real decision.

request_data_edit is gated on cycle_number == 1, so it doesn't fire again on cycle 2 of the same thread. The edit gate only ever applies once per thread.

Reviewing a Critical Decision

request_approval only fires when severity == "critical" — a ratio bad enough that the wrong automatic call is expensive.

Reaching the pause, on a zone with 4 drivers against 80 rider requests:

PAUSED — kind=approval
question: Critical imbalance in Airport (cycle 1). Approve, reject, or override the recommended policy.
severity=critical | recommended=surge_pricing | candidates=['surge_pricing', 'driver_bonus', 'demand_redirect', 'do_nothing']
explanation: The surge pricing policy was selected for the Airport zone in Cycle 1 because it generated the highest profit of $142.93, even though it did not resolve the imbalance.

Approving and resuming — but against a different graph object this time:

MY_DECISION = {"action": "approve"}
app_resume = build(memory)   # a different graph object, sharing only `memory`
r3 = app_resume.invoke(Command(resume=MY_DECISION), config=thread_approve)

final policy: surge_pricing
[Cycle 1] [Airport] ratio=20.0 | severity=critical | policy=surge_pricing | wait 5.6min → 13.3min | resolved=NO

That app_resume detail is worth pausing on. It's built fresh from the same build() function, and it only shares one thing with the original app: the MemorySaver. The paused state isn't living inside the app Python object — it's living in the checkpointer. Any graph object built the same way, sharing that checkpointer, can pick the pause back up.

A Third Kind of Pause: Debugging

request_data_edit and request_approval are both gated. They only pause under a specific condition, and the resume value changes what happens next.

A debugging pause is different in kind:

It's unconditional — it always pauses, whenever it's wired in.
It's purely observational — the resume value is discarded, and it never changes state.

def debug_checkpoint(state: AgentState) -> AgentState:
    interrupt({
        "kind": "debug",
        "question": "Inspect state before the approval gate. Resume with anything to continue.",
        "state_snapshot": {
            "cycle_number": state["cycle_number"],
            "zone": state["zone"],
            "imbalance_ratio": state["imbalance_ratio"],
            "severity": state["severity"],
            "candidate_policies": state.get("candidate_policies"),
            "policy_evaluations": state.get("policy_evaluations"),
            "recommended_policy": state.get("recommended_policy"),
            "explanation": state.get("explanation"),
        },
    })
    return {}

It exists to inspect the full in-flight state right before the approval gate acts on it — not to make or revise a decision. It's opt-in via build_graph(debug_mode=True), kept out of the default graph so it doesn't add a pause to every single run.

What We Built

	Part 3	Part 4
Scope	A sequence of cycles on one zone	Same, plus two review checkpoints per thread
Control flow	Fully autonomous, no pauses	`interrupt()` pauses execution; `Command(resume=...)` continues it
Starting snapshot	Trusted as generated	Reviewable once, on cycle 1
Critical decisions	Applied automatically	Held for approve / reject / override — your call, not a script
State	—	Unchanged — no new fields

The decision core still hasn't changed. choose_best_policy is exactly Part 1's function, called exactly the same way. What Part 4 adds is a new way for a human to sit inside the loop, instead of only reading a report after the fact — a real pause, resumable from a different Python object entirely, gated so it only interrupts when it's actually worth someone's attention.

What's still missing: every zone here is still evaluated alone. A policy that pulls a driver from a neighboring zone has been mentioned since Part 1, but never actually built. That's where Part 5 picks up.

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Next — Part 5: coordinating two zones at once, so a driver pulled into one zone is a decision that actually accounts for both.

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 3: Giving the Agent Memory

Ebrahim Arian — Sat, 08 Aug 2026 01:24:22 +0000

This is Part 3 of a 5-part series. Part 1 built a rule-based agent for one zone. Part 2 added an LLM for two narrow jobs: reading an ops note, and explaining the decision. Neither part changed one basic fact: the agent looks at a zone exactly once, then stops.

That's a real problem. A severe deficit often needs more than one 15-minute window to fix. But calling the agent again is first contact all over again:

No memory of what was already tried
No sense of whether this is cycle one or cycle six
The zone's own numbers reset, instead of carrying forward

Part 3 fixes that with MemorySaver and a thread_id. One thread becomes one zone's ongoing story. One .invoke() call becomes one 15-minute cycle.

The decision logic itself doesn't change at all. detect_imbalance, classify_severity, resolved_imbalance, and choose_best_policy are the exact same deterministic functions from Part 1, reused directly.

What memory actually buys here:

The zone's real state carries forward correctly, instead of resetting. That includes drivers, riders, and a little organic replenishment each cycle.
A note can schedule something for future cycles — "traffic bad for the next hour," "the event starts at 6pm" — without being repeated every time. The LLM extracts the fact once. Code applies it automatically from then on.

The Graph

Two new nodes get added at the front. Both are plain Python — no LLM:

start_cycle ──▶ apply_scheduled_conditions ──▶ detect_imbalance ──▶ classify_severity ──▶ set_candidates
                                                                                                │
                                                                                   ┌────────────┴────────────┐
                                                                                   ▼ (balanced)               ▼ (deficit/surplus)
                                                                           trivial_do_nothing            reconcile_inputs  ← LLM #1
                                                                                   │                          ▼
                                                                                   │                   resolved_imbalance
                                                                                   │                          ▼
                                                                                   │                   choose_best_policy
                                                                                   │                          ▼
                                                                                   │                   generate_explanation ← LLM #2
                                                                                   │                          │
                                                                                   └──────────────┬───────────┘
                                                                                                    ▼
                                                                                           simulate_and_report

start_cycle clears the previous cycle's raw conversation. This keeps the message log from growing forever across separate .invoke() calls. (History still persists — it's kept in its own history field, not in the message log.) It also advances the clock, so demand and supply use the right hour's baseline.
apply_scheduled_conditions checks whether anything a past note scheduled should expire or trigger this cycle. No LLM call — just comparing the current cycle number against a stored expiry or trigger.

reconcile_inputs also gets extended, not replaced. It's the same single-shot call from Part 2. Now it also picks up a stated duration ("for the next hour") or a future trigger time ("starts at 6pm"), when the note mentions one.

The LLM extracts the raw fact once. apply_scheduled_conditions applies it automatically on every cycle after that.

The State

AgentState grows four fields past Part 2's:

initial_hour and cycle_number anchor the clock
history accumulates one record per cycle
scheduled_conditions holds whatever a past note scheduled for later

None of this is MemorySaver itself — it's just ordinary state. MemorySaver is what makes this state survive between separate calls, instead of resetting every time:

class AgentState(TypedDict):
    zone:                 dict
    ops_note:             str
    initial_hour:         int   # set once on cycle 1, anchors the clock
    cycle_number:         int
    history:              Annotated[list[dict], operator.add]
    scheduled_conditions: dict  # e.g. {"traffic": {"expires_at_cycle": 5}}
    imbalance_ratio:      float
    imbalance_type:       str
    severity:             str
    candidate_policies:   list
    policy_evaluations:   dict
    policy_resolutions:   dict
    recommended_policy:   str
    explanation:          str
    messages:             Annotated[list[BaseMessage], add_messages]
    outcome:              dict
    outcome_delta:        dict
    report:               str

The Extended Tool

report_context_and_schedule is Part 2's report_context tool, with two fields added. That lets the LLM report timing, not just a current condition — a stated duration, or a future trigger hour.

Code still owns all the arithmetic that follows: minutes to cycles, hour to cycles-until. The LLM only extracts the raw fact:

@tool
def report_context_and_schedule(
    rain_flag: bool,
    event_flag: bool,
    traffic_level: Literal["none", "light", "moderate", "heavy"],
    traffic_duration_minutes: int = 0,
    event_starts_at_hour: int = -1,
) -> str:
    """
    Call this exactly once with your interpretation of the ops note for THIS
    cycle. If the note doesn't affect a value, report the zone's current value
    unchanged.

    traffic_duration_minutes: ONLY if the note states how long this traffic
    condition will last (e.g. "for the next hour" -> 60). 0 if no duration is
    stated -- traffic_level then applies to this cycle only.

    event_starts_at_hour: ONLY if the note describes an event scheduled for a
    specific clock time that hasn't started yet (e.g. "starts at 6pm" -> 18).
    -1 if no future start time is stated.
    """
    return (
        f"rain_flag={rain_flag}, event_flag={event_flag}, traffic_level={traffic_level}, "
        f"traffic_duration_minutes={traffic_duration_minutes}, event_starts_at_hour={event_starts_at_hour}"
    )

Wiring In the Checkpointer

Every node here is a plain function, same as before. The one genuinely new piece is MemorySaver, passed in at compile() time.

That's what turns zone/history/scheduled_conditions from ordinary state into something that survives across separate .invoke() calls on the same thread_id:

llm = ChatOllama(model="qwen2.5:14b", temperature=0)
llm_with_reconcile_tool = llm.bind_tools([report_context_and_schedule])

def _reconcile(state):
    return reconcile_inputs(state, llm_with_reconcile_tool)

def _explain(state):
    return generate_explanation(state, llm)

g = StateGraph(AgentState)

g.add_node("start_cycle",                start_cycle)
g.add_node("apply_scheduled_conditions", apply_scheduled_conditions)
g.add_node("detect_imbalance",           detect_imbalance)
g.add_node("classify_severity",          classify_severity)
g.add_node("set_candidates",             set_candidates)
g.add_node("trivial_do_nothing",         trivial_do_nothing)
g.add_node("reconcile_inputs",           _reconcile)
g.add_node("resolved_imbalance",         resolved_imbalance)
g.add_node("choose_best_policy",         choose_best_policy)
g.add_node("generate_explanation",       _explain)
g.add_node("simulate_and_report",        simulate_and_report)

g.add_edge(START, "start_cycle")
g.add_edge("start_cycle", "apply_scheduled_conditions")
g.add_edge("apply_scheduled_conditions", "detect_imbalance")
g.add_edge("detect_imbalance",  "classify_severity")
g.add_edge("classify_severity", "set_candidates")
g.add_conditional_edges("set_candidates", route_llm_or_skip, {
    "trivial_do_nothing": "trivial_do_nothing",
    "reconcile_inputs":   "reconcile_inputs",
})
g.add_edge("reconcile_inputs",     "resolved_imbalance")
g.add_edge("resolved_imbalance",   "choose_best_policy")
g.add_edge("choose_best_policy",   "generate_explanation")
g.add_edge("generate_explanation", "simulate_and_report")
g.add_edge("trivial_do_nothing",   "simulate_and_report")
g.add_edge("simulate_and_report",  END)

app = g.compile(checkpointer=MemorySaver())

app here is reused for everything in the rest of this article. One graph, many independent zone stories — told apart by thread_id, not by rebuilding anything.

Running Multiple Cycles

MemorySaver plus a thread_id is the whole mechanism. Call .invoke() again on the same thread_id, and the graph picks up exactly where the last call left off.

Only the first call needs a full initial state. Every call after that only needs whatever's new — an ops note, or nothing at all.

Downtown Core, starting in deficit, no ops notes:

[Cycle 1] ratio=1.93 | policy=surge_pricing | wait 6.3min → 4.6min | resolved=NO
    Highest profit at $230.78, even though it did not resolve the imbalance.
[Cycle 2] ratio=1.41 | policy=surge_pricing | wait 4.6min → 4.3min | resolved=YES
    RESOLVED

Two calls, same thread_id. The second one already knows this is cycle 2 of an ongoing deficit — nothing had to be passed back in by hand.

Scheduling a Condition Across Cycles

An ops note on cycle 1 says traffic will be bad for the next hour — that's 4 cycles. reconcile_inputs extracts traffic_duration_minutes=60. Code converts that to 4 cycles, and schedules the expiry.

No further notes are needed. apply_scheduled_conditions keeps traffic_level="heavy" active through cycle 4, then reverts it automatically at cycle 5 — exactly on schedule:

notes = {1: "There's a bad accident on the main road backing up traffic for the next hour."}
results = run_cycles(app, "downtown-traffic-duration", downtown, ops_notes=notes)

cycle=1 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}
cycle=2 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}
cycle=3 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}
cycle=4 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}
cycle=5 traffic=0.00 scheduled={}

One note on cycle 1 shapes five cycles of behavior. Nobody has to remind the agent about the traffic on cycles 2 through 4.

A Future Trigger, and Overriding a Schedule

Two more cases worth showing:

A future trigger, not an immediate correction. "The concert starts at 6pm" — it's currently 5pm. That means event_flag should stay False for now, and flip to True only once that hour actually arrives.

reconcile_inputs extracts event_starts_at_hour=18. Code works out how many cycles away that is, and schedules the flip:

cycle=1 hour=17 event_flag=False
cycle=2 hour=17 event_flag=False
cycle=3 hour=17 event_flag=False
cycle=4 hour=17 event_flag=False
cycle=5 hour=18 event_flag=True
cycle=6 hour=18 event_flag=True

A later note overriding an earlier schedule. Say a new note arrives before a scheduled condition would have expired on its own — "actually the accident's been cleared." It replaces the schedule entirely, instead of waiting for the original timer. New information wins:

notes = {
    1: "Traffic is bad for the next hour due to an accident.",
    3: "Actually the accident has been cleared, traffic is back to normal.",
}

cycle=1 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}
cycle=2 traffic=1.00 scheduled={'traffic': {'expires_at_cycle': 5}}

By cycle 2 the zone had already resolved on its own. So the schedule never actually got overridden here. But the mechanism — a new note wiping and replacing an existing schedule — is the same one that fires whenever a correction genuinely arrives mid-schedule.

Does the Policy Actually Change?

Every example so far stops at the first resolution. That hides an important question: does the agent keep surging forever, or does it actually back off once a zone balances?

Running 20 cycles straight through, without stopping early, answers that directly:

cycle=1  type=deficit   ratio=1.93  policy=surge_pricing   drivers=16  riders=18  wait= 3.7min
cycle=2  type=balanced  ratio=1.12  policy=do_nothing      drivers=15  riders=19  wait= 4.4min
cycle=4  type=deficit   ratio=1.36  policy=surge_pricing   drivers=14  riders=15  wait= 3.8min
cycle=8  type=deficit   ratio=1.31  policy=surge_pricing   drivers=17  riders=18  wait= 3.5min
cycle=12 type=deficit   ratio=1.46  policy=surge_pricing   drivers=14  riders=14  wait= 3.3min
cycle=20 type=balanced  ratio=1.08  policy=do_nothing      drivers=11  riders=12  wait= 3.6min

Over the full 20 cycles, Downtown Core moves between deficit (surge_pricing) and balanced (do_nothing) four separate times. Not once and done — a genuine, ongoing back-and-forth. Ordinary drift periodically pushes the ratio back over the deficit line, before do_nothing settles it again.

Driver and rider counts stay in a stable, bounded range the whole time — drivers 11–17, riders 12–21. Not drifting, not piling up.

What Changed From Part 2

	Part 2	Part 3
Scope	One call, one decision	A sequence of cycles on the same zone, via `thread_id`
State	Stateless — every call is first contact	`MemorySaver` carries `zone`/`history`/`scheduled_conditions` forward automatically
Policy selection	Deterministic, reused from Part 1	Same, unchanged
Ops note	Corrects this decision only	Can also schedule a duration or future trigger across cycles

The decision itself never got smarter. choose_best_policy is identical to Part 1.

What changed: the agent can now tell "still working on it, three cycles in" apart from "first time we've looked at this." And a note like "traffic bad for the next hour" doesn't need to be repeated four times.

What's still missing: every decision here still executes unattended, with nobody checking it first. That's where Part 4 picks up.

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Next — Part 4: a human-in-the-loop pause with interrupt(), so a risky or unusual decision doesn't just run without a second opinion.

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 2: Teaching the Agent to Read Ops Notes

Ebrahim Arian — Sat, 08 Aug 2026 01:24:17 +0000

This is Part 2 of a 5-part series. Part 1 built a rule-based agent for a single ride-share zone — no LLM, just structured numbers and deterministic rules.

It worked, but with one obvious hole. An ops person doesn't type structured fields. They write something like "the event got cancelled, and there's an accident backing up traffic near the venue." Stage 1's agent only understands rain_flag/event_flag/traffic as raw values. It has no way to read that sentence at all.

This is where an LLM earns its place — for exactly two jobs a lookup table genuinely can't do, and nothing more:

Reconcile an ops note. Read the free text and correct rain_flag/event_flag/traffic before the decision runs.
Explain the decision. Write a plain-English justification for a human reviewer, after the policy is already chosen.

Policy selection itself doesn't change at all. It's the exact same deterministic logic from Part 1, reused directly — never redone, never second-guessed by the LLM.

A small local model isn't automatically trustworthy, even at a narrow job like this. So I also compare five of them side by side, to see which one actually holds up.

Why Not Just Hand the LLM Everything?

An earlier version of this project tried that. It used a ReAct loop, where the LLM called several tools and picked the final policy itself, via free-text output.

It didn't hold up. Even qwen2.5:14b sometimes compared the wrong pair of candidates, or dropped the best option from its own answer. That wasn't a prompt-tuning problem — it was the wrong tool for the task.

An LLM doesn't compute an argmax. It generates plausible-sounding text, with no guarantee every candidate actually got compared. max() never makes that mistake.

Here's the rule I use throughout this project: if a task can be solved with plain numbers and a fixed answer, write it in code. Code gets it right every time. An LLM only earns its place when the input is a sentence a lookup table can't parse, or the output has to be a sentence a lookup table can't write. Handing a plain numeric decision to an LLM doesn't add capability — it just adds a new way to be wrong.

reconcile_inputs reads a real sentence, so that's free text in. generate_explanation writes one, so that's free text out. Picking the most profitable policy out of four numbers is neither, so it stays in plain Python:

# Scenario: 4 drivers, 14 rider requests, event nearby
for policy in ["surge_pricing", "driver_bonus", "demand_redirect", "do_nothing"]:
    r = evaluate_policy({"driver_count": 4, "rider_request_count": 14,
                          "rain_flag": False, "event_flag": True}, policy, noise=False)
    print(f"{policy:<20} profit=${r['profit']:>8.2f}  resolved={r['resolved']}")

surge_pricing        profit=$   75.61  resolved=False
driver_bonus         profit=$   56.66  resolved=False
demand_redirect      profit=$   62.96  resolved=False
do_nothing           profit=$   51.60  resolved=False

Same evaluate_policy function as Part 1, called directly. No LLM anywhere near it.

The Graph

Two single-shot LLM calls get added, and nothing else changes:

detect_imbalance ──▶ classify_severity ──▶ set_candidates
                                                  │
                                     ┌────────────┴────────────┐
                                     ▼ (balanced)               ▼ (deficit/surplus)
                             trivial_do_nothing            reconcile_inputs  ← LLM #1
                                     │                          ▼
                                     │                   resolved_imbalance
                                     │                          ▼
                                     │                   choose_best_policy
                                     │                          ▼
                                     │                   generate_explanation ← LLM #2
                                     │                          │
                                     └──────────────┬───────────┘
                                                     ▼
                                            simulate_and_report

detect_imbalance, classify_severity, resolved_imbalance, and choose_best_policy are imported directly from Part 1's module. Same deterministic logic — not redone here.

The two new nodes are single-shot, not a loop:

reconcile_inputs only runs if there's an ops note. It calls a tool, but only to get structured data back. The tool's code never actually runs — calling it is just how the model reports its answer, instead of writing a paragraph.
generate_explanation runs after choose_best_policy has already decided. It's plain text, no tools. It can't get the decision wrong, because it never touches it.

Balanced zones skip both entirely. One candidate, nothing to reconcile or explain — zero model calls.

The State

Same idea as Part 1's state. Three fields get added, for what an LLM now contributes:

ops_note — the input
explanation — an output
messages — the raw LLM conversation, kept around for inspection

class AgentState(TypedDict):
    zone:                dict
    ops_note:            str
    imbalance_ratio:     float
    imbalance_type:      str
    severity:            str
    candidate_policies:  list
    policy_evaluations:  dict
    policy_resolutions:  dict
    recommended_policy:  str
    explanation:         str
    messages:            Annotated[list[BaseMessage], operator.add]
    outcome:             dict
    outcome_delta:       dict
    report:              str

Structured Output via `bind_tools`

reconcile_inputs doesn't let the model write back a free-text paragraph. Instead, it gives the LLM exactly one tool, and asks it to call that tool once — with the three fields it needs to report:

@tool
def report_context(rain_flag: bool, event_flag: bool,
                    traffic_level: Literal["none", "light", "moderate", "heavy"]) -> str:
    """
    Call this exactly once with your interpretation of rain_flag, event_flag, and
    traffic_level for this decision, after considering the ops note. If nothing in
    the note changes a value, report the zone's original value unchanged.

    traffic_level anchors: "none" = no congestion mentioned/normal; "light" = some
    congestion, minor delays; "moderate" = noticeable backup, routes slower than
    usual; "heavy" = gridlock, accident, or road closure — significant delays.
    Only road/vehicle traffic counts here — foot traffic (pedestrian crowds) is a
    separate concept and should NOT affect this value.
    """
    return f"rain_flag={rain_flag}, event_flag={event_flag}, traffic_level={traffic_level}"

The tool's body never actually runs. Calling it is just a clean way for the model to hand back exactly three values: two booleans and one fixed label. That beats a paragraph I'd have to parse by hand.

Wiring It Up

Every node here is a plain function, same as Part 1. reconcile_inputs and generate_explanation just take an extra llm argument. So building the graph means binding a specific model first.

I wrap this in a build_agent(llm) function, instead of building it inline once. Here's why: the model comparison below needs the exact same graph, rebuilt five separate times, once per LLM. The structure never changes. Only the model bound to it does.

def build_agent(llm):
    llm_with_reconcile_tool = llm.bind_tools([report_context])

    def _reconcile(state):
        return reconcile_inputs(state, llm_with_reconcile_tool)

    def _explain(state):
        return generate_explanation(state, llm)

    g = StateGraph(AgentState)
    g.add_node("detect_imbalance",     detect_imbalance)
    g.add_node("classify_severity",    classify_severity)
    g.add_node("set_candidates",       set_candidates)
    g.add_node("trivial_do_nothing",   trivial_do_nothing)
    g.add_node("reconcile_inputs",     _reconcile)
    g.add_node("resolved_imbalance",   resolved_imbalance)
    g.add_node("choose_best_policy",   choose_best_policy)
    g.add_node("generate_explanation", _explain)
    g.add_node("simulate_and_report",  simulate_and_report)

    g.add_edge(START, "detect_imbalance")
    g.add_edge("detect_imbalance",  "classify_severity")
    g.add_edge("classify_severity", "set_candidates")
    g.add_conditional_edges("set_candidates", route_llm_or_skip, {
        "trivial_do_nothing": "trivial_do_nothing",
        "reconcile_inputs":   "reconcile_inputs",
    })
    g.add_edge("reconcile_inputs",     "resolved_imbalance")
    g.add_edge("resolved_imbalance",   "choose_best_policy")
    g.add_edge("choose_best_policy",   "generate_explanation")
    g.add_edge("generate_explanation", "simulate_and_report")
    g.add_edge("trivial_do_nothing",   "simulate_and_report")
    g.add_edge("simulate_and_report",  END)

    return g.compile()

One Ops Note, Five Models

Policy selection can't go wrong here. It's still deterministic. What actually varies from model to model is narrower: whether it correctly reads the ops note and reports the corrected event_flag and traffic_level.

The scenario: Stadium Area's raw snapshot has event_flag=True and no traffic. The ops note says the event was cancelled, and separately describes an accident backing up the main road. That's two corrections. It also has a trap phrase — "foot traffic" — that shouldn't affect traffic_level at all:

OPS_NOTE = (
    "The stadium event just got cancelled — foot traffic will be much lower than "
    "usual tonight. There's also a bad accident on the main road backing up traffic "
    "near the venue."
)

A model reading this correctly reports event_flag=False and traffic_level="heavy". And it doesn't let "foot traffic" leak into that second value.

I ran four 7-8B models first — the range most people can comfortably run on a laptop. Then I moved up to qwen2.5:14b, to see what the extra parameters buy:

Model	Reconciled correctly?	Notes
`deepseek-r1:8b`	No	Never called the tool at all — not this run, not any run so far
`llama3.1:8b`	Yes (this run)	In an earlier run, returned `'False'` as a string, not a boolean — `bool("False")` is `True` in Python
`qwen2.5:7b`	Yes (this run)	Documented as inconsistent across repeated testing — sometimes right, sometimes wrong, occasionally empty
`mistral:7b`	No (this run)	Got it right in an earlier run; didn't call the tool at all this time
`qwen2.5:14b`	Yes	Correct on every run so far

The pattern that matters isn't any single row. It's that llama3.1:8b and mistral:7b swapped which one failed, between two runs of the identical notebook — same prompt, same temperature=0. That's concrete proof that one clean run isn't evidence of reliability.

deepseek-r1:8b is the one exception that looks systematic, not random. It never calls the tool at all, on any run. Its <think> reasoning step doesn't reliably turn into structured tool output.

qwen2.5:14b (~9GB, still comfortable on a 16GB M2) is the only model that's held up correctly on every run of this notebook.

Model choice mattered more than prompt tuning here — even though the task itself is scoped tight: read two or three sentences, report two booleans and one bounded label.

Final model for this project: qwen2.5:14b. Every later part defaults to it. Not because it's the biggest model available — because it's the smallest one that's actually held up correctly, every time it's been tested.

What Changed From Part 1

	Part 1	Part 2
Classification & candidates	`detect_imbalance`/`classify_severity`/regime routing	Same, imported directly
Policy selection	Deterministic profit-argmax	Same, imported directly — not redone by the LLM
Balanced zones	`do_nothing`, single candidate	Skips the LLM entirely — zero model calls
Input	Structured only (numbers, flags)	An ops note can correct `rain_flag`/`event_flag`/`traffic` first
Output	A report string	+ a plain-English explanation for a human reviewer
LLM's role in the decision	None	Still none — only translates input and narrates output

The LLM's job stays narrow. Read a sentence a lookup table can't parse. Write a sentence a lookup table can't produce. Neither one touches the actual decision.

The remaining gap: even a correct policy choice often leaves resolved=False. One 15-minute window frequently isn't enough for a severe deficit. And the agent has no memory of what it already tried. That's where Part 3 picks up.

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Next — Part 3: giving the agent memory across cycles with MemorySaver, so it can apply policies over time instead of deciding once and forgetting.

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 1: A Rule-Based Agent, No LLM Yet

Ebrahim Arian — Sat, 08 Aug 2026 01:24:04 +0000

It's 5pm on a weekday, and you're standing outside a stadium after an event just wrapped up. You open your ride-share app. Estimated wait: 12 minutes. Meanwhile, three neighborhoods over, drivers are circling empty streets with nobody requesting a ride.

This happens constantly on any ride-share platform. It's not a bug — it's just supply and demand failing to line up on their own.

Someone, or something, has to watch each part of the city. It needs to notice when a zone is out of balance, then decide what to do:

Raise prices, to pull in more drivers
Pay drivers a bonus to reposition
Redirect riders to a calmer zone
Or just leave it alone, if it'll sort itself out

That decision-maker is what I'm building in this series, using LangGraph — one capability at a time.

This is Part 1 of a 5-part series. Each part adds exactly one thing the previous one genuinely couldn't do, so the agent gets closer to how this would actually have to work in production:

Part 1 (this one): a rule-based agent for a single zone — no LLM, everything is structured numbers
Part 2: an LLM, but only for the two jobs a lookup table can't do — reading a human's free-text note, and explaining the decision back
Part 3: memory, so the agent remembers what it already tried on a zone instead of deciding once and forgetting
Part 4: a human-in-the-loop pause, so a risky or unusual decision doesn't just execute unattended
Part 5: coordinating two zones at once, since pulling a driver from one zone is a decision that affects both

Nothing above is built yet in Part 1. That's deliberate. Every one of those additions is real complexity, and real complexity should only show up once the problem actually needs it — not because it sounds impressive.

So Part 1 stays as plain as the problem allows: one zone, one decision cycle, and rules a human ops person could write on a whiteboard.

The Data: A Snapshot of a City

Before there's an agent, there's a city to look at. I generate a synthetic snapshot of 8 zones — downtown, an airport, a couple of suburbs, a university district, and so on.

Each zone has:

A driver count
A rider request count
An average wait time
A couple of context flags — is it raining, is there an event nearby

zones = generate_all_zones(hour=17, rain=False, event=True, seed=42)

df = pd.DataFrame(zones)[[
    'zone_name', 'zone_type', 'driver_count',
    'rider_request_count', 'avg_wait_time', 'rain_flag', 'event_flag', 'traffic'
]]
df['ratio'] = (df['rider_request_count'] / df['driver_count']).round(2)
df

zone_name	driver_count	rider_request_count	avg_wait_time	ratio
Downtown Core	15	29	6.3	1.93
Airport	8	17	6.7	2.12
Midtown	6	14	8.9	2.33
Stadium Area	6	15	9.2	2.50

The ratio column is rider requests divided by drivers. It's the whole signal the agent needs:

Above 1.3 → deficit (too few drivers)
Below 0.5 → surplus (too many idle drivers)
In between → balanced

Downtown Core is at 1.93 — a real deficit, made worse by a nearby event. That's the zone I'll walk through.

The Graph

Here's the shape of the decision, end to end:

detect_imbalance
      ↓
classify_severity
      ↓ (conditional edge)
 ┌────┴────────────┬──────────────┐
 ↓                 ↓              ↓
demand_action   supply_action  do_nothing
 └────────┬────────┴──────────────┘
          ↓
   resolved_imbalance
          ↓
   choose_best_policy
          ↓
   simulate_and_report

Two nodes run first:

detect_imbalance computes the ratio.
classify_severity labels it mild, moderate, or critical. That label is for reporting only — it doesn't affect the decision.

Then comes the conditional edge, where the actual branching happens. It reads imbalance_type and decides which policies are even worth trying:

deficit → surge_pricing, driver_bonus, demand_redirect, or do_nothing
surplus → reallocation_nudge or do_nothing (a zone with too many idle drivers doesn't need demand-side incentives)
balanced → do_nothing

Two more nodes run after that:

resolved_imbalance evaluates every candidate policy once. It records each one's profit, and whether it actually fixes the imbalance.
choose_best_policy picks the most profitable policy among those that resolve it. If none of them resolve it, it falls back to picking the most profitable one overall.

One more thing worth naming: every policy here is zone-local. None of them look past the selected zone's own numbers.

Pulling a driver in from a neighboring zone is a real idea. But it needs a second zone in the picture first — that's Part 5, not Part 1.

The State

LangGraph's StateGraph runs on a single shared piece of state. Every node reads from it and writes to it. In Python, that's just a TypedDict:

class ZoneState(TypedDict):
    zone:                dict  # the single zone this graph run studies
    imbalance_ratio:     float
    imbalance_type:      str   # "deficit" | "surplus" | "balanced"
    severity:            str   # "none" | "mild" | "moderate" | "critical"
    candidate_policies:  list  # which policies are worth evaluating, set by the routed branch
    policy_evaluations:  dict  # {policy_name: profit} for every candidate considered
    policy_resolutions:  dict  # {policy_name: bool} — did this candidate resolve the imbalance
    recommended_policy:  str
    outcome:             dict
    outcome_delta:       dict
    report:              str

Nothing here is LangGraph-specific magic. It's a plain dictionary shape. The framework's only job is making sure every node agrees on it.

Wiring the Graph

Each node is a plain Python function. It takes the current state in, and returns whichever fields it changed. Nodes don't call each other directly — the graph's edges decide what runs next.

Wiring them together takes three steps:

Register each node under a name.
Connect the straightforward ones with add_edge.
Use add_conditional_edges for the one spot where the next step depends on the state.

g = StateGraph(ZoneState)

g.add_node("detect_imbalance",    detect_imbalance)
g.add_node("classify_severity",   classify_severity)
g.add_node("demand_action",       demand_action)
g.add_node("supply_action",       supply_action)
g.add_node("do_nothing",          do_nothing)
g.add_node("resolved_imbalance",  resolved_imbalance)
g.add_node("choose_best_policy",  choose_best_policy)
g.add_node("simulate_and_report", simulate_and_report)

g.add_edge(START, "detect_imbalance")
g.add_edge("detect_imbalance", "classify_severity")

g.add_conditional_edges("classify_severity", route_action, {
    "demand_action": "demand_action",
    "supply_action": "supply_action",
    "do_nothing":    "do_nothing",
})

g.add_edge("demand_action",       "resolved_imbalance")
g.add_edge("supply_action",       "resolved_imbalance")
g.add_edge("do_nothing",          "resolved_imbalance")
g.add_edge("resolved_imbalance",  "choose_best_policy")
g.add_edge("choose_best_policy",  "simulate_and_report")
g.add_edge("simulate_and_report", END)

app = g.compile()

route_action is the function behind that conditional edge. It reads imbalance_type off the state, and returns the name of the branch to take.

That's the entire routing logic for this stage — one if-shaped decision, expressed as data instead of a chain of if statements.

Running It on Downtown Core

With the graph compiled, running it takes one call: invoke(), with Downtown Core's snapshot as the starting state.

result = app.invoke(initial_state)

Here's what came back:

Policy	Profit	Resolved?
surge_pricing ← chosen	$230.78	No
demand_redirect	$202.46	No
driver_bonus	$181.83	No
do_nothing	$177.84	No

surge_pricing won on profit — same as it would have without the resolution check. But notice that nothing resolved the imbalance in one pass.

That's not a bug. resolved_imbalance only measures a policy's effect over one 15-minute window. A deficit this size — 29 riders against 15 drivers — genuinely needs more than one nudge to close.

The agent still made the best call it could. It just can't promise a full fix in a single shot, and it says so honestly (resolved=NO) instead of pretending otherwise. Giving it the ability to try again, and remember what it already attempted, is exactly what Part 3 adds.

Fast-Forwarding the Outcome

The last node is simulate_and_report. It plays the chosen policy forward 15 minutes.

The simulation is stochastic. Drivers respond to incentives probabilistically, the way real people do — not on a fixed schedule.

Metric	Before	After	Change
driver_count	15	16	+1
rider_request_count	29	25	−4
avg_wait_time (min)	6.3	5.5	−0.8

Directionally correct: more drivers, fewer pending requests, a shorter wait. Just not enough to flip Downtown Core out of deficit territory in one cycle.

What's Missing

Everything the agent looked at here was already structured — numbers and boolean flags. That's exactly why plain rules could handle it. There was no ambiguity for a language model to resolve.

But an ops person doesn't write structured fields. They write something like "the stadium event just got cancelled, and there's an accident backing up traffic near the venue." Stage 1's agent has no way to read that sentence. No lookup table fixes that — it needs an LLM, used for exactly one narrow job. That's where Part 2 picks up.

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Next — Part 2: teaching the agent to read a human's ops note, and to explain its own decision back in plain English — plus a head-to-head comparison of five local LLMs to see which one actually holds up at the job.

When the Picture Doesn't Match the Label — Part 2: Feature-Based Detection with OCR and VQA

Ebrahim Arian — Thu, 23 Jul 2026 02:43:34 +0000

This is Part 2 of a two-part series. Part 1 covered the problem, the CLIP baseline, and BLIP. This part covers the OCR + VQA pipeline that replaced them for fine-grained product feature matching.

The Problem with Scores

At the end of Part 1, the conclusion was clear: CLIP and BLIP are good at catching obvious mismatches but fail at the fine-grained ones that matter most in a marketplace. A wrong brand scores 0.79 on BLIP ITM — comfortably inside "Likely match" territory. A wrong size is nearly invisible to both models.

The reason is architectural. Both CLIP and BLIP compare an image and a description as whole units. They never ask: does the brand in the image match the brand in the description? They encode everything together and produce a single score. That score cannot tell you which attribute caused a mismatch.

What a marketplace actually needs is not a similarity score. It needs answers to specific questions:

Is this the right brand?
Is this the right size?
Is this the right variant?

That insight led to a completely different approach.

The Three Products Used in This Article

All experiments in this article are run against three real grocery products — chosen to represent different label types, packaging materials, and mismatch challenges:

Prairie Farms Whole Milk (1 quart carton) — a text-heavy label with a stylised brand logo, nutritional facts on the back, and a clear front panel. Useful for testing brand and variant detection.
Amazon Fresh Boneless Skinless Chicken Breast (30 oz plastic container) — minimal label text, brand printed on a sticker. Tests whether the model can identify both the brand and the specific cut.
Land O Lakes Salted Butter (8 oz paper wrapper) — compact packaging with multiple size representations printed on the label (8 oz, 226.8 g, 4 half sticks). Tests size matching edge cases.

The Feature Dictionary Approach

Instead of scoring image against description holistically, the idea is to define a feature dictionary — a structured set of attributes that any product listing should have — and verify each one independently.

description_features = {
    "product_type": "milk",
    "brand":        "prairie farms",
    "size":         ["1 quart", "946ml", "1 qt"],  # multiple equivalent representations
    "variant":      "whole",
    "container":    "carton",
}

For each feature, the system extracts the corresponding value from the product image and compares it against the description. If any single feature fails — the listing is flagged.

Two tools do the extraction, each handling what the other cannot:

Tool 1: OCR for Text Features

OCR (Optical Character Recognition) reads printed text directly from an image pixel by pixel and converts it into a string. For product labels, the most important information — brand name, weight, variant — is printed text. Reading it directly is faster and more accurate than asking a vision model to interpret it.

We use EasyOCR (pip install easyocr), a Python library that runs locally with no API calls required.

import easyocr
import numpy as np
from src.utils import fetch_image_from_url

def extract_ocr_text(image_url: str, min_confidence: float = 0.4) -> str:
    img = fetch_image_from_url(image_url)
    img_array = np.array(img)

    reader = easyocr.Reader(["en"], gpu=False, verbose=False)
    results = reader.readtext(img_array)

    texts = [text for _, text, conf in results if conf >= min_confidence]
    return " ".join(texts).lower()

OCR on the Land O Lakes butter label returns:

jarmer-owned landolakes since 1921 tbsp 4 half sticks butter half net wt 8oz (226.8 g) salted

"land o lakes", "butter", "8oz", and "salted" are all there.

The space stripping trick

OCR commonly reads "8oz" as a single token while the description says "8 oz" with a space. A naive substring match would fail. The fix is to strip all spaces before comparing:

ocr_found = any(
    d.replace(" ", "") in ocr_text.replace(" ", "")
    for d in desc_norms
)

Multiple equivalent representations

Size values are inconsistent across listings. A butter listing might say "8 oz", "226.8 g", or "4 half sticks" — all referring to the same product. Requiring an exact match would cause false negatives. The solution is to accept a list of equivalent representations for any feature:

"size": ["8 oz", "226.8 g", "4 half sticks"]

If OCR finds any one of them, the feature passes.

Tool 2: Moondream2 VQA as Fallback

OCR has two hard limits. First, it cannot read stylised logos and custom brand fonts — "Prairie Farms" rendered as a decorative script may return garbled characters or nothing at all. Second, some features are not printed as plain text at all — "container type" is not written on the label; it is a visual property the model must infer.

For these cases, the fallback is VQA (Visual Question Answering) using Moondream2 (1.86B parameters, vikhyatk/moondream2).

VQA models take an image and a natural language question and return a short answer. Unlike captioning — which describes everything and may miss the specific detail you care about — VQA is targeted and controllable.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "vikhyatk/moondream2",
    revision="2025-01-09",
    trust_remote_code=True,
    torch_dtype=torch.float16,
).to(device)
tokenizer = AutoTokenizer.from_pretrained("vikhyatk/moondream2", revision="2025-01-09")

enc_image = model.encode_image(image)   # encode once, ask many questions
answer = model.answer_question(enc_image, "What brand name is shown on this product?", tokenizer)

Why Moondream2?

Model	Size	Why chosen
`blip-vqa-base`	~400MB	Weak instruction-following, poor on product labels
`Salesforce/blip2-opt-2.7b`	~5.5GB	No VQA head, generative only
`vikhyatk/moondream2`	~4GB	Strong instruction-following, efficient on M2, handles logos well

Moondream2 runs in float16 on Apple M2 MPS and uses approximately 4 GB of unified memory — comfortable on a 16 GB machine.

Constrained prompts

Open-ended questions invite unpredictable answers. Constrained prompts force the model to choose from a known vocabulary, which reduces hallucination:

FEATURE_QUESTIONS = {
    "product_type": "Is this milk, coffee, butter, chicken, meat, bread, juice, a beverage, snack, or cleaning product?",
    "container":    "Is this product in a bottle, can, carton, container, canister, bag, or box?",
    "brand":        "What brand name is shown on this product?",
    "size":         "What is the size or quantity? Find a number with these words: oz, quart, gram.",
    "variant":      "What variant or flavor is this? For example: whole, 2%, dark roast, classic roast.",
}

Putting It Together: OCR First, VQA Fallback

The routing logic is simple:

For text features (brand, size, variant, product_type):
  1. Try OCR → if found, done
  2. If not found → ask Moondream2

For visual features (container):
  → Ask Moondream2 directly (not printed as text)

def compare_features(image_features: dict, description_features: dict) -> dict:
    ocr_text = image_features.get("_ocr_text", "")

    for feature, desc_val in description_features.items():
        desc_norms = [v.strip().lower() for v in desc_val] if isinstance(desc_val, list) else [desc_val.lower()]

        if feature in TEXT_FEATURES:
            ocr_found = any(d.replace(" ", "") in ocr_text.replace(" ", "") for d in desc_norms)
            if ocr_found:
                matched, source, img_answer = True, "OCR", "found in OCR"
            else:
                vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
                matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
                source, img_answer = "VQA fallback", vqa_val
        else:
            vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
            matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
            source, img_answer = "VQA", vqa_val

Results on Three Products

Product	brand	product_type	size	variant	container	Overall
Prairie Farms Whole Milk	✓ VQA	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match
Amazon Fresh Chicken Breast	✓ OCR	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match
Land O Lakes Salted Butter	✓ OCR	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match

All correct descriptions matched. Wrong brand, wrong size, and wrong variant correctly triggered mismatches across all three products.

Every Feature Is Critical

An early version of the labelling logic used a ratio: if 75% of features match, the listing passes. This is wrong for a marketplace.

A customer ordering 8 oz of butter who receives 16 oz has received the wrong product. A boneless chicken breast described as bone-in is a real fulfilment failure. There is no threshold at which a wrong size becomes acceptable.

The final labelling logic is a hard veto:

CRITICAL_FEATURES = {"brand", "product_type", "size", "variant", "container"}

def overall_label(feature_results: dict) -> str:
    for feature in CRITICAL_FEATURES:
        if feature in feature_results and not feature_results[feature]["match"]:
            return "Likely mismatch"   # immediate, no exceptions
    return "Likely match"

Any single feature failure returns "Likely mismatch" regardless of what else matched.

Limitations

This approach is meaningfully better than CLIP or BLIP for fine-grained product matching, but several real-world challenges remain:

Image angle: OCR reads only what is visible. If the listing image shows the nutrition facts panel rather than the front label, brand and variant text will be missed entirely. The fix — accepted multiple image URLs per product and merge OCR results — is left for future work.

Image quality: Blurry or compressed images reduce OCR accuracy and may cause Moondream2 to hallucinate answers for features it cannot clearly see.

Prompt coverage: The current VQA prompts cover a handful of product categories. In a real marketplace with thousands of categories, prompts would need to be carefully engineered per category — broader option lists, context-aware phrasing, potentially chain-of-thought reasoning. The current prompts are a proof of concept, not production-ready.

Semantic matching: The comparison logic uses substring matching after space normalisation. "1 qt" and "one quart" are not matched even though they are equivalent. Embedding-based semantic matching would handle this but adds complexity.

Takeaways

The key insight from this project is that the right framing matters more than the right model. CLIP and BLIP are powerful, but they answer the wrong question — "how similar are these overall?" — when the marketplace needs answers to specific questions: wrong brand? wrong size? wrong variant?

Switching to a feature dictionary approach with OCR and VQA fallback made the system interpretable (you can see exactly which feature failed), debuggable (you can inspect the OCR text and VQA answers directly), and strict (a single wrong feature is enough to flag a listing).

The combination of EasyOCR for printed text and Moondream2 for visual reasoning is not perfect — but it is a meaningful step toward a system that could be useful in production.

Code and Experiments

All code, notebooks, and experiment results are available on GitHub:
github.com/ebiarian/product-image-description-alignment

← Back to Part 1: CLIP & BLIP

When the Picture Doesn't Match the Label — Part 1: Using CLIP and BLIP to Catch Product Mismatches

Ebrahim Arian — Thu, 23 Jul 2026 02:40:44 +0000

This is Part 1 of a two-part series on building an automated product image-description mismatch detector for online marketplaces. Part 1 covers the problem, the CLIP baseline, and BLIP. Part 2 covers the OCR + Visual Question Answering pipeline that followed.

The Problem No One Notices Until It's Too Late

Imagine shopping online for a specific brand of milk. The image shows a Prairie Farms carton, but the description says Organic Valley. Or the image matches, but the size is wrong. You hesitate, second-guess yourself, and maybe abandon the cart entirely.

Prairie Farms Whole Milk — 1 quart carton used throughout this article

This small moment of doubt is not an edge case — it is one of the most common pain points in e-commerce. When product images and text descriptions are out of sync, the cost is real: buyers lose confidence, return rates climb, and platforms take a reputational hit.

Across major marketplaces, mismatches between images and descriptions rank among the leading causes of customer frustration, alongside missing or unclear product information [1]. On Amazon, a significant share of returns and complaints stems from products that look different from their listing [2]. On eBay, buyers frequently report disputes triggered by images that don't match what arrived [1]. On Instacart, grocery listings sometimes carry outdated or incorrect images, leading customers to select the wrong item — particularly problematic when products look visually similar [3].

The core issue is scale. Large marketplaces host millions of listings, many uploaded by third-party sellers with inconsistent quality control. Manual review at that scale is not feasible. What is needed is an automated system that can reliably flag mismatched listings before customers encounter them.

This is the problem I set out to tackle in this project.

Why This Is Harder Than It Looks

At first glance, matching an image to a description seems straightforward. But real-world product data is deceptively tricky:

Brand mismatches: Two cartons of milk may look nearly identical in shape and color, but belong to different brands. The distinction lives entirely in the label text — a few pixels in a small font.
Size mismatches: "8 oz" vs. "16 oz" may show visually identical packaging, differing only in fine print on the back of the label.
Variant mismatches: "Salted butter" vs. "unsalted butter" — same product, different SKU, meaningfully different for the customer.
Obvious mismatches: An image of a chicken breast paired with a butter description — easy for a human, but a system still needs to catch it reliably at scale.

Traditional image classification or keyword matching falls apart on the first three categories. What is needed is a model that understands both images and text in a unified way.

Phase 1: CLIP as a Zero-Shot Baseline

CLIP (Contrastive Language–Image Pretraining), developed by OpenAI, was the natural starting point. CLIP is trained on hundreds of millions of image-text pairs to learn a shared embedding space — a mathematical representation where semantically related images and text end up close together, and unrelated ones end up far apart.

In practice:

An image of a milk carton and the text "whole milk" will have a high cosine similarity score.
The same image paired with "a cat playing with a ball" will have a very low score.
Subtle mismatches — wrong brand, wrong size — fall somewhere in between.

How it is implemented

from transformers import CLIPProcessor, CLIPModel
from src.config import Config
from src.utils import fetch_image_from_url

def compute_clip_similarity(configs: Config, image_url: str, description: str) -> float:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    model = CLIPModel.from_pretrained(configs.clip_model).to(device)
    processor = CLIPProcessor.from_pretrained(configs.clip_processor)

    img = fetch_image_from_url(image_url)
    inputs = processor(text=[description], images=img, return_tensors="pt", padding=True).to(device)

    with torch.no_grad():
        outputs = model(**inputs)
        image_emb = outputs.image_embeds / outputs.image_embeds.norm(dim=-1, keepdim=True)
        text_emb  = outputs.text_embeds  / outputs.text_embeds.norm(dim=-1, keepdim=True)

    return (image_emb @ text_emb.T).item()

The similarity score is mapped to a label:

Score	Label
> 0.30	Likely match
0.20 – 0.30	Uncertain
< 0.20	Likely mismatch

What CLIP gets right — and where it falls short

Testing against a Prairie Farms whole milk image with several descriptions:

Description	Score	Label
One quart of whole milk from Prairie Farms (correct)	0.31	Likely match
One quart of whole milk from Organic Valley (wrong brand)	0.29	Uncertain
Half gallon of whole milk from Prairie Farms (wrong size)	0.30	Uncertain
A cat playing with a ball (obvious mismatch)	0.11	Likely mismatch

CLIP catches the obvious mismatch confidently. But wrong brand and wrong size both land in "Uncertain" — CLIP recognises the product type correctly but misses the brand and quantity distinction. This is a known limitation: CLIP encodes descriptions holistically, so quantity words like "half gallon" or brand names like "Organic Valley" may not shift the embedding enough to trigger a clear mismatch signal.

CLIP is a useful zero-shot baseline for catching gross mismatches, but fine-grained product-level distinctions remain a challenge with similarity scoring alone.

Phase 2: BLIP for Richer Image Understanding

The natural next step was BLIP (Bootstrapping Language-Image Pre-training), which brought two new capabilities:

2a. Image Captioning with BLIP-2

Instead of comparing embeddings directly, a captioning model can describe what it sees in an image. This makes the model's perception explicit and debuggable.

The original blip-image-captioning-base had a significant failure mode on text-heavy product images — it produced repetitive output like "classic classic classic classic..." when encountering product labels with bold text. The fix was to upgrade to Salesforce/blip2-opt-2.7b, which uses a large language model (OPT-2.7B) as its language backbone and generates coherent captions.

from transformers import Blip2Processor, Blip2ForConditionalGeneration

def generate_blip_caption(configs: Config, image_url: str) -> str:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    processor = Blip2Processor.from_pretrained(configs.blip2_model)
    model = Blip2ForConditionalGeneration.from_pretrained(
        configs.blip2_model, torch_dtype=torch.float16
    ).to(device)

    img = fetch_image_from_url(image_url)
    inputs = processor(images=img, return_tensors="pt").to(device, torch.float16)

    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=50)

    return processor.decode(output[0], skip_special_tokens=True)

Results on three products:

Product	Actual description	BLIP-2 caption
Prairie Farms Whole Milk	one quart of whole milk from Prairie Farms	a carton of milk with the label on it
Amazon Fresh Chicken Breast	boneless skinless chicken breast, 30 oz	a package of chicken breast
Land O Lakes Salted Butter	salted butter, 8 oz	land o lakes salted butter

Left to right: Prairie Farms Whole Milk, Amazon Fresh Chicken Breast, Land O Lakes Salted Butter — the three products used in all experiments

BLIP-2 correctly reads "Land O Lakes salted butter" from the label — noticeably better than the base model. But it loses granular detail like size and variant for the milk and chicken.

2b. Image-Text Matching (ITM)

BLIP's ITM head is more directly useful for mismatch detection. Rather than generating text, it outputs a probability score between 0 and 1 representing how well an image and text match — directly interpretable without calibration.

from transformers import BlipProcessor, BlipForImageTextRetrieval

def compute_blip_itm_score(configs: Config, image_url: str, description: str) -> float:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    processor = BlipProcessor.from_pretrained(configs.blip_itm_model)
    model = BlipForImageTextRetrieval.from_pretrained(configs.blip_itm_model).to(device)

    img = fetch_image_from_url(image_url)
    inputs = processor(images=img, text=description, return_tensors="pt").to(device)

    with torch.no_grad():
        outputs = model(**inputs, use_itm_head=True)
        score = torch.nn.functional.softmax(outputs.itm_score, dim=1)[:, 1].item()

    return score

Description	ITM Score	Label
Correct (Prairie Farms milk)	0.9998	Likely match
Wrong brand (Organic Valley)	0.7872	Likely match
Wrong size (two quarts)	0.9996	Likely match
Obvious mismatch (cat playing)	0.0000	Likely mismatch

ITM is excellent at catching obvious mismatches. However, wrong brand and wrong size both score very high — the model is too lenient for fine-grained product matching. A wrong brand gets a 0.79 ITM score, which would pass as a likely match in a marketplace context.

This revealed a fundamental limitation: both CLIP and BLIP treat the description as a whole. Neither one systematically checks individual product attributes like brand, size, and variant in isolation.

The Lesson from Phase 1 and 2

CLIP and BLIP are powerful general-purpose models, but neither was designed to answer the specific question a marketplace needs: "Does the brand in the image match the brand in the description? Does the size match?"

What is needed is not a similarity score — it is a structured, per-feature comparison. That insight drives the approach in Part 2.

Why It Matters

Product content quality is not a niche concern. Research consistently links inconsistencies between images and text to reduced customer confidence and lower purchase intent [4]. Platforms that invest in automated content quality tooling see measurable improvements in conversion rates, return rates, and seller accountability.

An automated mismatch detection pipeline does not replace human review — it scales it. By surfacing the listings most likely to be wrong, it directs human attention to where it matters most, at a fraction of the cost of manual auditing.

What's Next

In Part 2, I replace free-form similarity scoring with a feature dictionary approach:

OCR (Optical Character Recognition) reads brand names, sizes, and variants directly from the product label as text
VQA (Visual Question Answering) answers targeted questions about visual features the OCR cannot read (container type, product category) and acts as a fallback when OCR misses stylised fonts or logos
A hard veto on all features ensures any single wrong attribute flags the listing — because in a marketplace, a wrong size is just as damaging as a wrong product

→ Continue to Part 2: OCR + VQA

Code and Experiments

All code and Jupyter notebooks for this project are available on GitHub:
github.com/ebiarian/product-image-description-alignment

References

Buzzin Content. 91.3% of E-Commerce Brands Failed to Address Real Customer Pain Points. https://www.buzzincontent.com/story/91-3-e-commerce-brands-failed-to-address-real-customer-pain-points-in-pdp-content-odn-report/
Besedo. The Effects of Fraud and Poor Content Quality on Online Marketplaces. https://besedo.com/blog/report-the-effects-of-fraud-and-poor-content-quality-on-online-marketplaces/
Perpetua Help. Missing or Incorrect Product Images. https://help.perpetua.io/en/articles/4626446-missing-or-incorrect-product-images
ScienceDirect. Inconsistencies between images and text in online shopping. https://www.sciencedirect.com/science/article/abs/pii/S0969698917307142

DEV Community: Ebrahim Arian

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 5: Coordinating Two Zones at Once

The Graph

The State

The Per-Zone Sub-Graph

The Parent Graph

A 10-Cycle Run: The Agent Asks, You Decide

What We Built

Five Parts, Five Additions

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 4: Letting a Human Step In

The Graph

The Nodes That Pause

Building It

Correcting the Starting Snapshot

Reviewing a Critical Decision

A Third Kind of Pause: Debugging

What We Built

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 3: Giving the Agent Memory

The Graph

The State

The Extended Tool

Wiring In the Checkpointer

Running Multiple Cycles

Scheduling a Condition Across Cycles

A Future Trigger, and Overriding a Schedule

Does the Policy Actually Change?

What Changed From Part 2

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 2: Teaching the Agent to Read Ops Notes

Why Not Just Hand the LLM Everything?

The Graph

The State

Structured Output via bind_tools

Wiring It Up

One Ops Note, Five Models

What Changed From Part 1

Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 1: A Rule-Based Agent, No LLM Yet

The Data: A Snapshot of a City

The Graph

The State

Wiring the Graph

Running It on Downtown Core

Fast-Forwarding the Outcome

What's Missing

When the Picture Doesn't Match the Label — Part 2: Feature-Based Detection with OCR and VQA

The Problem with Scores

The Three Products Used in This Article

The Feature Dictionary Approach

Tool 1: OCR for Text Features

The space stripping trick

Multiple equivalent representations

Tool 2: Moondream2 VQA as Fallback

Why Moondream2?

Constrained prompts

Putting It Together: OCR First, VQA Fallback

Results on Three Products

Every Feature Is Critical

Limitations

Takeaways

Code and Experiments

When the Picture Doesn't Match the Label — Part 1: Using CLIP and BLIP to Catch Product Mismatches

The Problem No One Notices Until It's Too Late

Why This Is Harder Than It Looks

Phase 1: CLIP as a Zero-Shot Baseline

How it is implemented

What CLIP gets right — and where it falls short

Phase 2: BLIP for Richer Image Understanding

2a. Image Captioning with BLIP-2

2b. Image-Text Matching (ITM)

The Lesson from Phase 1 and 2

Why It Matters

What's Next

Code and Experiments

References

Structured Output via `bind_tools`