Building a Ride-Share Zone-Balancing Agent with LangGraph — Part 2: Teaching the Agent to Read Ops Notes

This is Part 2 of a 5-part series. Part 1 built a rule-based agent for a single ride-share zone — no LLM, just structured numbers and deterministic rules.

It worked, but with one obvious hole. An ops person doesn't type structured fields. They write something like "the event got cancelled, and there's an accident backing up traffic near the venue." Stage 1's agent only understands rain_flag/event_flag/traffic as raw values. It has no way to read that sentence at all.

This is where an LLM earns its place — for exactly two jobs a lookup table genuinely can't do, and nothing more:

• Reconcile an ops note. Read the free text and correct rain_flag/event_flag/traffic before the decision runs.
• Explain the decision. Write a plain-English justification for a human reviewer, after the policy is already chosen.

Policy selection itself doesn't change at all. It's the exact same deterministic logic from Part 1, reused directly — never redone, never second-guessed by the LLM.

A small local model isn't automatically trustworthy, even at a narrow job like this. So I also compare five of them side by side, to see which one actually holds up.

Why Not Just Hand the LLM Everything?

An earlier version of this project tried that. It used a ReAct loop, where the LLM called several tools and picked the final policy itself, via free-text output.

It didn't hold up. Even qwen2.5:14b sometimes compared the wrong pair of candidates, or dropped the best option from its own answer. That wasn't a prompt-tuning problem — it was the wrong tool for the task.

An LLM doesn't compute an argmax. It generates plausible-sounding text, with no guarantee every candidate actually got compared. max() never makes that mistake.

Here's the rule I use throughout this project: if a task can be solved with plain numbers and a fixed answer, write it in code. Code gets it right every time. An LLM only earns its place when the input is a sentence a lookup table can't parse, or the output has to be a sentence a lookup table can't write. Handing a plain numeric decision to an LLM doesn't add capability — it just adds a new way to be wrong.

reconcile_inputs reads a real sentence, so that's free text in. generate_explanation writes one, so that's free text out. Picking the most profitable policy out of four numbers is neither, so it stays in plain Python:

# Scenario: 4 drivers, 14 rider requests, event nearby
for policy in ["surge_pricing", "driver_bonus", "demand_redirect", "do_nothing"]:
    r = evaluate_policy({"driver_count": 4, "rider_request_count": 14,
                          "rain_flag": False, "event_flag": True}, policy, noise=False)
    print(f"{policy:<20} profit=${r['profit']:>8.2f}  resolved={r['resolved']}")

surge_pricing        profit=$   75.61  resolved=False
driver_bonus         profit=$   56.66  resolved=False
demand_redirect      profit=$   62.96  resolved=False
do_nothing           profit=$   51.60  resolved=False

Same evaluate_policy function as Part 1, called directly. No LLM anywhere near it.

The Graph

Two single-shot LLM calls get added, and nothing else changes:

detect_imbalance ──▶ classify_severity ──▶ set_candidates
                                                  │
                                     ┌────────────┴────────────┐
                                     ▼ (balanced)               ▼ (deficit/surplus)
                             trivial_do_nothing            reconcile_inputs  ← LLM #1
                                     │                          ▼
                                     │                   resolved_imbalance
                                     │                          ▼
                                     │                   choose_best_policy
                                     │                          ▼
                                     │                   generate_explanation ← LLM #2
                                     │                          │
                                     └──────────────┬───────────┘
                                                     ▼
                                            simulate_and_report

detect_imbalance, classify_severity, resolved_imbalance, and choose_best_policy are imported directly from Part 1's module. Same deterministic logic — not redone here.

The two new nodes are single-shot, not a loop:

• reconcile_inputs only runs if there's an ops note. It calls a tool, but only to get structured data back. The tool's code never actually runs — calling it is just how the model reports its answer, instead of writing a paragraph.
• generate_explanation runs after choose_best_policy has already decided. It's plain text, no tools. It can't get the decision wrong, because it never touches it.

Balanced zones skip both entirely. One candidate, nothing to reconcile or explain — zero model calls.

The State

Same idea as Part 1's state. Three fields get added, for what an LLM now contributes:

• ops_note — the input
• explanation — an output
• messages — the raw LLM conversation, kept around for inspection

class AgentState(TypedDict):
    zone:                dict
    ops_note:            str
    imbalance_ratio:     float
    imbalance_type:      str
    severity:            str
    candidate_policies:  list
    policy_evaluations:  dict
    policy_resolutions:  dict
    recommended_policy:  str
    explanation:         str
    messages:            Annotated[list[BaseMessage], operator.add]
    outcome:             dict
    outcome_delta:       dict
    report:              str

Structured Output via bind_tools

reconcile_inputs doesn't let the model write back a free-text paragraph. Instead, it gives the LLM exactly one tool, and asks it to call that tool once — with the three fields it needs to report:

@tool
def report_context(rain_flag: bool, event_flag: bool,
                    traffic_level: Literal["none", "light", "moderate", "heavy"]) -> str:
    """
    Call this exactly once with your interpretation of rain_flag, event_flag, and
    traffic_level for this decision, after considering the ops note. If nothing in
    the note changes a value, report the zone's original value unchanged.

    traffic_level anchors: "none" = no congestion mentioned/normal; "light" = some
    congestion, minor delays; "moderate" = noticeable backup, routes slower than
    usual; "heavy" = gridlock, accident, or road closure — significant delays.
    Only road/vehicle traffic counts here — foot traffic (pedestrian crowds) is a
    separate concept and should NOT affect this value.
    """
    return f"rain_flag={rain_flag}, event_flag={event_flag}, traffic_level={traffic_level}"

The tool's body never actually runs. Calling it is just a clean way for the model to hand back exactly three values: two booleans and one fixed label. That beats a paragraph I'd have to parse by hand.

Wiring It Up

Every node here is a plain function, same as Part 1. reconcile_inputs and generate_explanation just take an extra llm argument. So building the graph means binding a specific model first.

I wrap this in a build_agent(llm) function, instead of building it inline once. Here's why: the model comparison below needs the exact same graph, rebuilt five separate times, once per LLM. The structure never changes. Only the model bound to it does.

def build_agent(llm):
    llm_with_reconcile_tool = llm.bind_tools([report_context])

    def _reconcile(state):
        return reconcile_inputs(state, llm_with_reconcile_tool)

    def _explain(state):
        return generate_explanation(state, llm)

    g = StateGraph(AgentState)
    g.add_node("detect_imbalance",     detect_imbalance)
    g.add_node("classify_severity",    classify_severity)
    g.add_node("set_candidates",       set_candidates)
    g.add_node("trivial_do_nothing",   trivial_do_nothing)
    g.add_node("reconcile_inputs",     _reconcile)
    g.add_node("resolved_imbalance",   resolved_imbalance)
    g.add_node("choose_best_policy",   choose_best_policy)
    g.add_node("generate_explanation", _explain)
    g.add_node("simulate_and_report",  simulate_and_report)

    g.add_edge(START, "detect_imbalance")
    g.add_edge("detect_imbalance",  "classify_severity")
    g.add_edge("classify_severity", "set_candidates")
    g.add_conditional_edges("set_candidates", route_llm_or_skip, {
        "trivial_do_nothing": "trivial_do_nothing",
        "reconcile_inputs":   "reconcile_inputs",
    })
    g.add_edge("reconcile_inputs",     "resolved_imbalance")
    g.add_edge("resolved_imbalance",   "choose_best_policy")
    g.add_edge("choose_best_policy",   "generate_explanation")
    g.add_edge("generate_explanation", "simulate_and_report")
    g.add_edge("trivial_do_nothing",   "simulate_and_report")
    g.add_edge("simulate_and_report",  END)

    return g.compile()

One Ops Note, Five Models

Policy selection can't go wrong here. It's still deterministic. What actually varies from model to model is narrower: whether it correctly reads the ops note and reports the corrected event_flag and traffic_level.

The scenario: Stadium Area's raw snapshot has event_flag=True and no traffic. The ops note says the event was cancelled, and separately describes an accident backing up the main road. That's two corrections. It also has a trap phrase — "foot traffic" — that shouldn't affect traffic_level at all:

OPS_NOTE = (
    "The stadium event just got cancelled — foot traffic will be much lower than "
    "usual tonight. There's also a bad accident on the main road backing up traffic "
    "near the venue."
)

A model reading this correctly reports event_flag=False and traffic_level="heavy". And it doesn't let "foot traffic" leak into that second value.

I ran four 7-8B models first — the range most people can comfortably run on a laptop. Then I moved up to qwen2.5:14b, to see what the extra parameters buy:

Model	Reconciled correctly?	Notes
deepseek-r1:8b	No	Never called the tool at all — not this run, not any run so far
llama3.1:8b	Yes (this run)	In an earlier run, returned 'False' as a string, not a boolean — bool("False") is True in Python
qwen2.5:7b	Yes (this run)	Documented as inconsistent across repeated testing — sometimes right, sometimes wrong, occasionally empty
mistral:7b	No (this run)	Got it right in an earlier run; didn't call the tool at all this time
qwen2.5:14b	Yes	Correct on every run so far

The pattern that matters isn't any single row. It's that llama3.1:8b and mistral:7b swapped which one failed, between two runs of the identical notebook — same prompt, same temperature=0. That's concrete proof that one clean run isn't evidence of reliability.

deepseek-r1:8b is the one exception that looks systematic, not random. It never calls the tool at all, on any run. Its <think> reasoning step doesn't reliably turn into structured tool output.

qwen2.5:14b (~9GB, still comfortable on a 16GB M2) is the only model that's held up correctly on every run of this notebook.

Model choice mattered more than prompt tuning here — even though the task itself is scoped tight: read two or three sentences, report two booleans and one bounded label.

Final model for this project: qwen2.5:14b. Every later part defaults to it. Not because it's the biggest model available — because it's the smallest one that's actually held up correctly, every time it's been tested.

What Changed From Part 1

	Part 1	Part 2
Classification & candidates	detect_imbalance/classify_severity/regime routing	Same, imported directly
Policy selection	Deterministic profit-argmax	Same, imported directly — not redone by the LLM
Balanced zones	do_nothing, single candidate	Skips the LLM entirely — zero model calls
Input	Structured only (numbers, flags)	An ops note can correct rain_flag/event_flag/traffic first
Output	A report string	+ a plain-English explanation for a human reviewer
LLM's role in the decision	None	Still none — only translates input and narrates output

The LLM's job stays narrow. Read a sentence a lookup table can't parse. Write a sentence a lookup table can't produce. Neither one touches the actual decision.

The remaining gap: even a correct policy choice often leaves resolved=False. One 15-minute window frequently isn't enough for a severe deficit. And the agent has no memory of what it already tried. That's where Part 3 picks up.

---

Code for this series: github.com/ebiarian/zone-balancing-ridesharing-langgraph-agent

Next — Part 3: giving the agent memory across cycles with MemorySaver, so it can apply policies over time instead of deciding once and forgetting.