LLMs for Time Series Forecasting with Low Latency

#aiinfrastructure #oxlo #ai

Time series forecasting is usually the domain of statistical models and dedicated neural architectures, but large language models are increasingly being used to predict future values by treating numerical sequences as structured text. The basic idea is simple: serialize your historical observations into a prompt, ask the model to extrapolate, and parse the result. The hard part is doing this with low latency and predictable cost when your historical window contains thousands of data points. That is where inference infrastructure becomes the deciding factor.

Why LLMs for Time Series?

LLMs excel at pattern recognition in text, and a time series can be rendered as a delimited string of numbers. Recent research shows that in-context learning with LLMs can match or surpass classical methods on many forecasting benchmarks, especially when the model is given sufficient context and clear instructions. The appeal is not just accuracy. A single LLM can handle heterogeneous data, missing values, and textual covariates without retraining. For shops that already run LLM pipelines for chat or coding, reusing that infrastructure for forecasting eliminates a separate serving stack.

The Latency and Cost Problem

Forecasting with LLMs requires long prompts. A daily sales record spanning two years is over 700 tokens before you add instructions, covariates, or few-shot examples. On token-based providers, input costs scale linearly with prompt length, and first-token latency grows as the model processes the full context. For agentic or streaming workloads that re-query frequently, this becomes expensive and slow.

Oxlo.ai uses request-based pricing: one flat cost per API call regardless of prompt length. For long-context forecasting workloads, this can be significantly cheaper than token-based alternatives. You can feed the model a full year of minutely data without watching your inference budget expand with every extra timestamp. Combine that with no cold starts on popular models, and you get consistent latency from the first request.

See Oxlo.ai pricing for plan details.

Prompt Engineering for Forecasts

The simplest approach is zero-shot text completion. Convert your series into a comma-separated or newline-separated list, then ask for the next N values. Few-shot prompting with similar seasonal patterns often improves accuracy. For structured parsing, use JSON mode.

Example prompt structure:

Historical daily energy consumption (kWh):
2024-01-01: 45.2
2024-01-02: 44.8
...
2024-12-31: 38.1

Forecast the next 7 days as JSON with keys "date" and "predicted_kwh".

Choosing Models and Context Windows

Not every LLM needs to be a 400B parameter giant for forecasting. A 32B or 70B model with strong reasoning and a large context window is often sufficient. Oxlo.ai offers several relevant options:

DeepSeek V4 Flash: 1M context window, efficient MoE architecture, near state-of-the-art open-source reasoning. Ideal for very long historical windows.
Kimi K2.6: 131K context, advanced reasoning, agentic coding, and vision support if you want to combine time series with chart images.
Qwen 3 32B: Multilingual reasoning and strong agent workflow support, good for pipelines that mix forecasting with tool use.
Llama 3.3 70B: General-purpose flagship, reliable for structured JSON output and function calling.

Because Oxlo.ai is fully OpenAI SDK compatible, switching between these models is a one-line parameter change.

Implementation Pattern

Below is a minimal Python example using the OpenAI SDK pointed at Oxlo.ai. It sends a serialized time series and requests a JSON forecast. Use streaming if you want to start parsing partial results before generation finishes.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

series_text = "\n".join([f"{d}: {v}" for d, v in zip(dates, values)])

response = client.chat.completions.create(
    model="qwen3-32b",  # or llama-3.3-70b, deepseek-v4-flash, etc.
    messages=[
        {"role": "system", "content": "You are a forecasting assistant. Output only valid JSON."},
        {"role": "user", "content": f"Historical data:\n{series_text}\n\nForecast the next 7 days as JSON."}
    ],
    response_format={"type": "json_object"},
    stream=False
)

forecast = response.choices[0].message.content

If latency is critical, enable streaming and parse tokens as they arrive, or select a smaller model from the catalog. Oxlo.ai carries 45+ models across seven categories, so you can trade accuracy for speed without changing providers.

Optimizing for Low Latency

Beyond model choice, there are three practical ways to keep forecasting responsive:

Truncate or compress history: Use seasonal subsampling or piecewise aggregation to fit the signal into fewer tokens without losing trend information.
Use JSON mode and function calling: Constraining output format reduces the number of tokens the model must generate and eliminates post-processing regex.
Avoid cold starts: Oxlo.ai serves popular models with no cold starts, so p50 and p99 latencies stay stable even during off-peak hours.

Request-based pricing also removes the penalty for padding prompts with detailed instructions or few-shot examples. You can afford to be verbose in the prompt if it improves forecast quality, because the cost is flat per request.

Conclusion

LLM-based time series forecasting is viable today, but only if your inference layer can handle long contexts without ballooning costs or unpredictable delays. Oxlo.ai offers a developer-first platform with request-based pricing, no cold starts, and a broad model catalog that includes long-context options like DeepSeek V4 Flash and Kimi K2.6. If you are already using the OpenAI SDK, pointing your client to https://api.oxlo.ai/v1 is the fastest way to test whether flat-rate inference improves your forecasting pipeline.

DEV Community