DEV Community

Cover image for Code Agent Dissected (03): Different LLM Formats — How Does an Agent Unify the Interface?
WonderLab
WonderLab

Posted on

Code Agent Dissected (03): Different LLM Formats — How Does an Agent Unify the Interface?

#agents #agentharness #llm #ai

The Problem Isn't "Can You Call an API?"

Switching to a different model is rarely as simple as changing one base_url:

  • DeepSeek, Zhipu, Kimi, and Qwen all claim to be "OpenAI-compatible," but their API key env var names, default endpoints, and tool_calls field positions aren't entirely the same
  • Some models still use the old function_call format; others return content as a multi-segment list
  • Certain backends restrict temperature, disallow multiple system messages, or don't recognize tool_choice=auto

If these differences are scattered throughout loop.py, the main loop turns into a swamp of if provider == xxx branches. MyCodeAgent's approach: lock away the differences inside core/llm.py and core/openai_compat.py, so the ReAct loop only ever sees a unified text and tool_calls.

The previous post covered how the loop runs turn by turn. This one looks at the boundary layer the loop crosses every time it actually calls a model.

Conclusion First 

RuntimeRunner._react_loop()
        │
        ▼
HelloAgentsLLM.invoke_raw(messages, tools=...)
        │  ① Provider routing: select endpoint / key / default model
        │  ② Request normalization: temperature, multiple system messages, tool_choice quirks, etc.
        ▼
OpenAICompatibleClient          ← stdlib urllib, no openai SDK
        │  POST {base_url}/chat/completions
        ▼
raw response (object or dict, varies slightly by provider)
        │
        ▼
extract_* five-tuple             ← response normalization
  content / tool_calls / usage / meta / reasoning
        │
        ▼
loop consumes unified structure, continues Acting / Reasoning
Enter fullscreen mode Exit fullscreen mode

Two layers, clear responsibilities:

Layer File Responsibility
Transport openai_compat.py Send HTTP, parse JSON/SSE, mimics client.chat.completions.create
Adaptation llm.py Provider routing + request compatibility + response extraction

The loop deliberately calls invoke_raw() instead of invoke() — the former returns the full response object so tool_calls, usage, and finish_reason can be extracted later; the latter just yields a string, which is not enough for the main loop.

Layer One: Roll Your Own "Fake SDK"

The project has no dependency on the official openai package. openai_compat.py uses urllib to implement the minimal subset the harness actually needs:

# core/openai_compat.py — core call path
class OpenAICompatibleClient:
    def __init__(self, api_key, base_url, timeout):
        self.chat = _Chat(self)   # exposes client.chat.completions.create(...)

    def _create_completion(self, payload):
        if payload.get("stream"):
            return self._stream(payload)
        with self._request(payload) as response:
            return ResponseObject(json.loads(response.read().decode("utf-8")))
Enter fullscreen mode Exit fullscreen mode

A request is a plain POST to {base_url}/chat/completions with a Bearer token header. Streaming reads SSE data: lines until [DONE].

The key piece is ResponseObject: it wraps a JSON dict into an object that supports dot-access like .choices[0].message, while also providing model_dump() to recover the raw dictionary.

class ResponseObject:
    def __getattr__(self, name):
        return _to_object(self._value[name])  # dict → dot-accessible object

    def model_dump(self):
        return self._value
Enter fullscreen mode Exit fullscreen mode

Extraction functions higher up don't care whether the response came from the official SDK or this hand-rolled client — both attribute access and dict access work the same way.

Why skip the official SDK? Fewer dependencies; the entire critical path is visible in the repo. When something breaks, there's no third-party wrapper to suspect. The trade-off is implementing only the subset the harness needs, not a complete SDK.

Layer Two: How the Provider Gets Selected

HelloAgentsLLM.__init__ resolves configuration in a fixed priority order:

Explicit parameter: provider
    → Env var: LLM_PROVIDER
        → Auto-detection (detect_envs / URL markers)
            → Fall back to "auto" (generic OpenAI-compatible)
Enter fullscreen mode Exit fullscreen mode

Credentials and default endpoints aren't hard-coded in if/else chains — they're looked up in the PROVIDER_PROFILES table:

# core/llm.py — table-driven routing (simplified)
PROVIDER_PROFILES = {
    "deepseek": {
        "key_envs": ("DEEPSEEK_API_KEY", "LLM_API_KEY"),
        "base_url": "https://api.deepseek.com",
        "model": "deepseek-chat",
        "url_markers": ("api.deepseek.com",),
        ...
    },
    "zhipu": { ... },
    "kimi": { ... },
    # ...
}

# During resolution:
self.provider = self._resolve_provider(provider, api_key, base_url)
self.api_key, resolved_base_url = self._resolve_credentials(api_key, base_url)
Enter fullscreen mode Exit fullscreen mode

Set LLM_PROVIDER=zhipu and the code fetches Zhipu's default base_url and the relevant key env vars directly from the table — no domain names hard-coded in business logic.

Auto-detection also has a safety valve: if keys for multiple providers are detected simultaneously, it raises an error and requires an explicit choice — preventing the silent "you think you're calling A, but you're hitting B" failure.

Layer Three: Response Normalization (The Most Valuable Layer)

After receiving raw_response, the loop does exactly this:

# runtime/loop.py — main loop consumes unified fields
response_text = extract_response_content(raw_response) or ""
reasoning_content = extract_reasoning_content(raw_response)
usage = extract_usage(raw_response)
response_meta = extract_response_meta(raw_response)
tool_calls = extract_tool_calls(raw_response)
Enter fullscreen mode Exit fullscreen mode

Five functions project "slightly-different JSON from each provider" into the harness's stable shape. The one the loop depends on most is extract_tool_calls:

# core/llm.py — normalize old and new Function Calling formats
def extract_tool_calls(response):
    message = _response_message(response)

    # New format: message.tool_calls[] (mainstream)
    calls = response_attr(message, "tool_calls") or []
    if calls:
        return [{"id": ..., "name": ..., "arguments": ...} for call in calls]

    # Old format: message.function_call (single invocation)
    function_call = response_attr(message, "function_call")
    if function_call:
        return [{"id": None, "name": ..., "arguments": ...}]

    return []
Enter fullscreen mode Exit fullscreen mode

Regardless of whether the model returns tool_calls or function_call, the loop always receives [{id, name, arguments}]. The "has tool_calls → execute tool" branch in the Acting step from the previous post is built on top of this normalization.

The other four functions each own a slice:

Function What It Extracts How the Loop Uses It
extract_response_content Body text (handles list-style content) Written to history / candidate answer for completion gate
extract_tool_calls List of tool invocations Acting branch
extract_usage prompt/completion/total tokens Track usage, check token budget
extract_response_meta finish_reason, length, refusal, etc. Empty-response retry, truncation detection, tracing
extract_reasoning_content Chain-of-thought (optional field) Debug display; does not affect control flow

The generic attribute reader is a single line, yet it underpins the entire extraction logic:

def response_attr(value, key):
    # dict → .get; object → getattr — ResponseObject and official SDK share the same extraction code
    return value.get(key) if isinstance(value, dict) else getattr(value, key, None)
Enter fullscreen mode Exit fullscreen mode

The Request Side Has Compatibility Patches Too

Normalization doesn't only happen on the response. Before sending, _build_request() smooths over several known quirks:

  1. Kimi K2 / 2.5: Only accepts temperature=1; other values are automatically replaced and a warning is logged
  2. MiniMax: Merges multiple system messages; drops tool_choice=auto (not recognized)
  3. Accidentally-full paths: If base_url was set to .../chat/completions, the suffix is stripped before constructing the client

All of this logic lives in the adaptation layer. The loop still just passes messages / tools / tool_choice. When a new "mostly-compatible but quirky" backend needs to be added, the rule goes here — not into the main loop.

Retries also live here: _invoke_with_retries wraps non-streaming calls with LLM_MAX_RETRIES plus exponential backoff. Note this is distinct from the "inner while model error recovery" described in the previous post — that layer handles semantic-level recovery like empty responses or PROMPT_TOO_LONG; this layer handles transient network/HTTP failures.

Three Entry Points — the Loop Uses One

Method Returns Used By
invoke_raw() Raw response object Main loop (then passed to extract_*)
invoke() Plain text string Summarization and other "just give me a sentence" scenarios
think() / stream_invoke() Streaming text chunks User-facing streaming output

The main loop chooses invoke_raw deliberately: an agent needs more than "what did the model say" — it also needs "were there tool calls, why did it stop, how many tokens were used." The string entry point throws all of that away.

Design Highlights

  1. Transport separated from adaptation: HTTP mechanics in openai_compat, business compatibility in llm, the loop touches neither
  2. Table-driven providers: Adding a new model is mostly adding a row to PROVIDER_PROFILES, not modifying the call chain
  3. Project rather than rewrite: extract_* only reads the raw response; it never mutates the provider's original payload — full snapshots remain available for tracing
  4. Zero SDK dependency: The critical path is fully visible and portable; the "fake SDK shape" keeps call patterns consistent without the external dependency

Summary

Mechanism Purpose
OpenAICompatibleClient Minimal OpenAI-compatible transport via stdlib
PROVIDER_PROFILES Table-driven routing for endpoints, keys, and default models
invoke_raw + extract_* Main loop consumes only unified text / tool_calls / meta
Old/new tool format compatibility tool_calls and function_call normalized to the same list shape
Request-side compatibility patches temperature, multiple system messages, malformed URLs — all handled centrally

The next post follows "how does the model know which tools are available" deeper: where tool schemas come from, and how Function Calling arguments flow into the execution pipeline.

About the Source Code for This Series

All analysis in this series is based on the open-source project MyCodeAgent.

The source code includes inline comments at key locations, aligned with the order topics are introduced in this series — you can follow along with the code while reading, or clone it and run, modify, and extend it to build your own agent.

git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env   # fill in your LLM API key
uv sync
uv run python main.py
Enter fullscreen mode Exit fullscreen mode

Visit PrimeSkills — a curated AI Agent and skills marketplace where every piece of content is validated against real enterprise workflows. No hype, just things that actually work.

For more practical insights and interesting products, visit my personal homepage.

Top comments (0)