Time series forecasting is traditionally the domain of statistical models and dedicated neural architectures, but LLMs are increasingly being used for zero-shot and few-shot prediction. By encoding temporal sequences as text, models can capture seasonality, trends, and contextual anomalies without task-specific training. The practical challenge is latency. When forecasts must run every minute or feed a real-time dashboard, token-based billing and cold starts can make production deployments expensive and unpredictable. This is where inference infrastructure and pricing models matter as much as the model itself.
Why LLMs for Time Series Forecasting
LLMs treat numerical sequences as structured text. You can prompt a model with a CSV snippet or a narrative description of historical data and ask for the next N steps. Recent research shows that LLMs perform competitively on standard benchmarks when given clear prompts and scalar scaling. They also unify forecasting, anomaly detection, and natural language explanation inside a single API call, which simplifies architecture.
The Latency Problem in Production Pipelines
Production forecasting often requires thousands of inferences per hour. With token-based providers, cost scales directly with input length, so feeding a long history of hourly observations quickly becomes prohibitive. Cold starts add variance to response times, making it difficult to meet SLA targets. For real-time use cases, you need an inference layer that is consistently fast and economically predictable.
Prompt Design for Numerical Forecasting
The simplest approach is to serialize the time series into a delimited string, normalize the values, and ask the model to extend the sequence. A well-structured prompt includes the frequency, the number of steps to predict, and the desired output format. JSON mode is useful here because it forces a parsable schema.
import pandas as pd
def build_forecast_prompt(series: pd.Series, horizon: int) -> str:
history = ", ".join([f"{v:.4f}" for v in series.values])
prompt = (
"You are a forecasting assistant. Given the following daily time series values, "
f"predict the next {horizon} steps. Return only a JSON object with a key 'forecast' "
f"containing a list of {horizon} floats.\n\n"
f"History: {history}\n\n"
"Prediction:"
)
return prompt
Choosing a Model and Inference Stack
Not every LLM endpoint is optimized for low-latency numerical inference. You want a provider that minimizes overhead and offers models with strong reasoning and long-context support.
Oxlo.ai is a developer-first inference platform that fits this workflow well. It hosts 45+ open-source and proprietary models, including DeepSeek V4 Flash, an efficient MoE model with a 1M context window that can ingest very long historical sequences without truncation. For agentic pipelines that combine forecasting with tool use, Qwen 3 32B provides multilingual reasoning and robust function calling. Llama 3.3 70B serves as a general-purpose flagship for balanced latency and accuracy. All models are served with no cold starts on popular models, which keeps p50 and p99 latencies tight.
Because Oxlo.ai is fully OpenAI SDK compatible, you can integrate it with existing Python tooling by changing a single base URL.
Implementation: Streaming Forecasts with Oxlo.ai
The following example uses the OpenAI Python SDK to stream a forecast from Oxlo.ai. Streaming reduces time-to-first-token, which is often the dominant factor in perceived latency for short completions.
from openai import OpenAI
import json
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
def forecast_series(history_values: list[float], horizon: int, model: str = "deepseek-v4-flash") -> list[float]:
history_str = ", ".join(f"{v:.4f}" for v in history_values)
messages = [
{
"role": "system",
"content": "You are a precise forecasting engine. Return only valid JSON."
},
{
"role": "user",
"content": (
f"Daily observations: {history_str}. "
f"Predict the next {horizon} values. "
"Return a JSON object with key 'forecast' and a list of floats."
)
}
]
response = client.chat.completions.create(
model=model,
messages=messages,
response_format={"type": "json_object"},
stream=True,
temperature=0.1,
max_tokens=256
)
chunks = []
for chunk in response:
if chunk.choices[0].delta.content:
chunks.append(chunk.choices[0].delta.content)
result = json.loads("".join(chunks))
return result["forecast"]
# Example usage
if __name__ == "__main__":
history = [120.5, 121.0, 119.8, 122.4, 123.1, 121.5, 122.0]
next_three = forecast_series(history, horizon=3)
print(next_three)
Key details in this snippet: stream=True delivers tokens as they are generated, response_format={"type": "json_object"} guarantees parsable output, and temperature=0.1 keeps the generation deterministic. Because Oxlo.ai does not charge by the token, you can pass a long history string without worrying about input-length penalties.
Cost Predictability at Scale
Time series workloads are inherently rhythmic. A monitoring system might issue one forecast per metric per minute, which means tens of thousands of requests per day. Under token-based pricing, a sudden spike in context length, such as backfilling a year of hourly data, can cause a billing surprise. Oxlo.ai uses flat per-request pricing, so the cost of a forecast is the same whether you pass ten data points or ten thousand. For long-context and agentic forecasting workflows, this model can be significantly cheaper than token-based alternatives. Plan details are available on the Oxlo.ai pricing page.
Conclusion
LLM-based time series forecasting is moving from experiment to production, but only if latency and cost remain under control. By combining structured prompts, JSON mode, and streaming inference on a platform built for speed, you can deploy real-time predictors without architectural complexity. Oxlo.ai provides the OpenAI-compatible endpoint, the model variety, and the request-based pricing structure that this use case demands. If you are evaluating inference providers for your forecasting pipeline, it is worth including Oxlo.ai in your benchmark.
Top comments (0)