Google's LangExtract on a Custom OpenAI-Compatible API: Model IDs That Route to Ollama, Strict Schemas and Silent Chunks

LangExtract is Google's Python library for pulling structured entities out of long text, with every extraction mapped back to its character position in the source. You describe the task, give one or two examples, and it chunks the document, sends each chunk to a model, parses the JSON and aligns the results. Gemini is the default, but it has an OpenAI provider built on the official OpenAI SDK, so it can run against vLLM, LiteLLM or a hosted gateway.

Getting it to use that provider with a non-OpenAI model name is the first surprise. I tested langextract 1.7.1 (openai 3.35.1, Python 3.13) against a small local mock server that logged every request and could return 400, 429, 503, fenced JSON or truncated output on demand. Everything below is what went over the wire. The examples point at APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions accordingly. Its base URL is https://apiclaw.biz/v1; replace YOUR_MODEL_ID with an exact ID from your provider's model list.

The model name chooses the provider

The obvious call is:

import langextract as lx

result = lx.extract(
    text_or_documents=text,
    prompt_description=prompt,
    examples=examples,
    model_id="qwen3-coder-plus",
    api_key="your-key",
    model_url="https://apiclaw.biz/v1",
)

LangExtract routes on regex patterns over model_id. Only names starting with gpt-3.5, gpt-4, gpt-5 or o1 to o9 go to the OpenAI provider. Names starting with qwen, deepseek, llama, mistral, gemma, phi and gpt-oss, among others, go to the Ollama provider. Here is what happened in my tests:

• qwen3-coder-plus and gpt-oss-120b went to the Ollama client, which POSTed to /v1/api/generate on my base URL and failed with Can't find Ollama qwen3-coder-plus. Try: ollama run qwen3-coder-plus.
• A name that matched nothing, such as plain-model, failed with No provider registered for model_id='plain-model'.
• gpt-4o-mini with model_url worked and hit /v1/chat/completions.

So the code above only works if your gateway's model happens to be named like an OpenAI model.

Force the OpenAI provider with ModelConfig

import os
import langextract as lx

config = lx.factory.ModelConfig(
    model_id="YOUR_MODEL_ID",
    provider="OpenAILanguageModel",   # "openai" also worked
    provider_kwargs={
        "api_key": os.environ["APICLAW_API_KEY"],
        "base_url": "https://apiclaw.biz/v1",
        "temperature": 0.0,
        "max_output_tokens": 2000,
        "max_workers": 4,
    },
)

result = lx.extract(
    text_or_documents=text,
    prompt_description="Extract every person mentioned, with their role. Use exact text.",
    examples=examples,
    config=config,
    show_progress=False,
)
for e in result.extractions:
    print(e.extraction_class, e.extraction_text, e.attributes, e.char_interval)

That sent POST /v1/chat/completions for every model name I tried. Two details:

• Pass api_key explicitly. LangExtract only reads OPENAI_API_KEY from the environment when the model ID contains gpt. With an explicit provider and a name like qwen3-coder-plus, leaving the key out failed with API key not provided. even though OPENAI_API_KEY was set.
• base_url can come from the environment. With base_url omitted, the SDK picked up OPENAI_BASE_URL. Keep /v1 on the end either way.

You'll also see a warning: With 'config', schema constraints are still applied via examples. That matters for the next section.

What each request contains

Every chunk became one request with:

• a system message, You are a helpful assistant that responds in JSON format.
• one user message holding your prompt description, your examples as Q:/A: pairs with the expected JSON, and the chunk text,
• n: 1,
• response_format of type json_schema, named langextract_extractions, with strict: true, built from your examples.

By default there was no temperature and no max_tokens, so the server's defaults applied. Setting temperature and max_output_tokens in provider_kwargs sent temperature: 0.0 and max_tokens: 2000. For extraction I'd always set both: a low temperature for consistency and an output cap that fits a long list of entities.

When the server doesn't support json_schema

Strict json_schema output is not universal. Some models and some OpenAI-compatible servers support only json_object, or nothing. My mock returned a 400 for json_schema, and the whole extract() call raised InferenceRuntimeError: OpenAI API error: Error code: 400.

result = lx.extract(..., config=config, use_schema_constraints=False)

With use_schema_constraints=False the request switched to response_format: {"type": "json_object"} and succeeded. The examples in the prompt still show the model the expected shape.

Parse failures are dropped with a warning

This is the setting I'd change first. By default LangExtract suppresses parse errors per chunk. Each of these runs printed one log line and returned zero extractions with no exception:

• The model wrapped its answer in prose and a fence (Here is the result: followed by a fenced json block). Under json_object mode LangExtract expects raw JSON, so the chunk was omitted: Chunk omitted from results (parse error): Failed to parse JSON content.
• The reply was cut off with finish_reason: "length". The truncated JSON failed to parse and the chunk was dropped the same way.

On a 40-chunk document, a few dropped chunks look like the model just didn't find anything there. Two fixes:

result = lx.extract(
    ...,
    config=config,
    use_schema_constraints=False,
    fence_output=True,                                  # accept fenced json replies
    resolver_params={"suppress_parse_errors": False},   # fail loudly instead
)

fence_output=True parsed the fenced reply correctly. suppress_parse_errors: False turned the silent drop into a ResolverParsingError, which you can catch and retry. A reply that started with a <think>...</think> block parsed fine in my test, so reasoning models weren't a problem there.

Chunks, workers and passes multiply requests

I ran a 5,320-character document with the defaults:

Setting	Requests	Max in flight
defaults (max_char_buffer=1000)	6	6
max_workers: 3 in provider_kwargs	6	3
extraction_passes=3	18	6
max_char_buffer=4000	2	2

Each request repeats your prompt description and examples, so with small chunks the fixed part can be a large share of every request. max_workers defaults to 10, which means a long document can fire 10 requests at once. On a plan with a per-minute or concurrency limit, set it lower. extraction_passes runs the whole document again per pass, one pass after another, so three passes triple both requests and time.

A larger max_char_buffer cuts request count, but long chunks give the model more to track, so test recall on your own documents before raising it a lot.

Retries come from the SDK only

A server that always returned 503 got three requests (the SDK's default two retries) and then extract() raised InferenceRuntimeError. A 429 with Retry-After: 2 was honoured: the SDK waited and the third request succeeded after about 4.6 seconds. LangExtract adds no retry loop of its own, so for big batch jobs wrap the call in your own retry, and remember that a failure in one chunk fails the whole document.

Checklist

• Use ModelConfig(provider="OpenAILanguageModel") unless your model ID starts with gpt- or o1 to o9.
• Pass api_key in provider_kwargs; the env key is only read for gpt model names.
• Set temperature and max_output_tokens yourself.
• Use use_schema_constraints=False if your server rejects json_schema.
• Set suppress_parse_errors: False (and fence_output=True for chatty models) so dropped chunks don't go unnoticed.
• Lower max_workers to fit your rate limit, and count extraction_passes as a multiplier.

Disclosure: I build APIClaw, the OpenAI-compatible gateway in these examples (flat monthly plans; crypto on the site, other payment methods through support). Everything above applies to any compatible endpoint.