LLMs for Cloud Functions

#aiinfrastructure #oxlo #ai

Running large language models from cloud functions forces you to balance latency, cost, and statelessness. Serverless platforms like AWS Lambda, Cloudflare Workers, and Vercel Functions meter execution time and memory, while the majority of inference providers bill by the token. When your workload involves long system prompts, retrieved document chunks, or multi-turn agent traces, token-based costs balloon and your serverless bill becomes unpredictable. Oxlo.ai changes the equation with request-based pricing: a single flat cost per API call no matter how long the input.

The Serverless LLM Problem

Cloud functions are stateless, have strict timeouts, and scale to zero. Every invocation must initialize its environment, deserialize the event, and return a response before the platform cuts it off. Adding an LLM call to that flow introduces network latency and variable processing time. Worse, most inference APIs scale cost linearly with input and output tokens. Providers like Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale all use token-based metering, which means a function that passes a large JSON blob or a long conversation history to the model incurs a disproportionate charge. In agentic or retrieval-augmented workflows, where prompts can grow to tens of thousands of tokens, this pricing model fights against the fine-grained, event-driven nature of serverless compute.

Request Pricing for Serverless

Oxlo.ai is a developer-first AI inference platform that charges one flat cost per API request regardless of prompt length. Unlike token-based providers, the price does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads. For a cloud function that reformats a large log file, summarizes a multi-page document, or runs a ReAct agent loop, the cost is the same whether the prompt is 100 tokens or 100,000 tokens. This makes capacity planning simple: if your application receives 1,000 requests per day, you know exactly how many API requests to budget for.

Oxlo.ai also advertises no cold starts on popular models. In a serverless context, that matters because the inference layer responds immediately instead of adding a warm-up delay on top of your function's cold start.

Drop-In Code Example

Oxlo.ai is fully OpenAI SDK compatible. You can point the official Python or Node.js client at the Oxlo.ai base URL and use the same chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech endpoints. Below is an AWS Lambda handler that calls Llama 3.3 70B through Oxlo.ai using the OpenAI Python SDK.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def lambda_handler(event, context):
    body = event.get("body", {})
    messages = body.get("messages", [{"role": "user", "content": "Hello"}])

    # Example: use Llama 3.3 70B for general-purpose tasks
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        max_tokens=512,
        stream=False
    )

    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": response.choices[0].message.content
    }

Because the API is a drop-in replacement, you can reuse existing middleware for retries, timeouts, and error handling without rewriting your function logic.

Model Selection for Functions

Oxlo.ai hosts 45-plus open-source and proprietary models across 7 categories. For cloud functions, the right model depends on latency and output requirements.

General-purpose chat: Llama 3.3 70B or Qwen 3 32B for multilingual reasoning and agent workflows.
Deep reasoning: DeepSeek R1 671B MoE or Kimi K2.6 for advanced chain-of-thought reasoning, complex coding, and vision tasks.
Fast responses: DeepSeek V4 Flash offers efficient MoE inference with a 1M context window, or Oxlo.ai Coder Fast for low-latency code generation.
Structured output: Any chat model supports JSON mode and function calling, so you can return structured data from your function without parsing free text.
Other modalities: You can call vision models like Gemma 3 27B or Kimi VL A3B, generate images with Flux.1 or Oxlo.ai Image Pro, transcribe audio with Whisper Large v3, or produce embeddings with BGE-Large, all from the same API endpoint pattern.

This breadth lets you keep your architecture uniform. One function can handle summarization, another can generate images, and both use the same base URL and authentication flow.

Cost Predictability

Token-based billing turns every long prompt into a budget risk. If your function accepts user-uploaded documents or recursive agent states, your monthly inference cost scales with user behavior, not request volume. Oxlo.ai's request-based pricing removes that variable. Competitors may be cheaper on ultra-short prompts, but Oxlo.ai can be 10 to 100 times cheaper for long-context workloads. For serverless developers, that means you can pass full context windows into the model without redesigning your prompt to save tokens.

Oxlo.ai publishes its pricing at https://oxlo.ai/pricing. The Free plan offers 60 requests per day across 16-plus models, including DeepSeek V3.2, and includes a 7-day full-access trial. The Pro plan provides 1,000 requests per day, while Premium raises that to 5,000 requests per day with priority queue access. Enterprise plans offer unlimited requests, dedicated GPUs, and a guaranteed 30 percent discount versus your current provider. Because each request is a flat unit, forecasting spend is arithmetic, not algebra.

Best Practices

Stream responses. Use streaming to start sending data back to the client while the model is still generating. This improves perceived latency and helps you stay within HTTP timeout limits.
Use JSON mode. When your function must return structured data, enable JSON mode instead of parsing markdown or free text.
Tool use inside functions. Oxlo.ai supports function calling. A single cloud function can act as an orchestrator: call the model, receive a tool request, execute the tool, and feed the result back in a multi-turn conversation.
Set max tokens. Cap max_tokens to a reasonable ceiling so that a single request cannot exhaust your function timeout or your daily request budget on an accidental loop.
Reuse the client. Initialize the OpenAI client outside the handler scope when possible so that HTTP connection pools persist across warm invocations.

Conclusion

Cloud functions and LLMs are a natural fit for event-driven AI, but token-based pricing introduces friction that works against the serverless model. Oxlo.ai offers a flat, per-request pricing structure that aligns with function-based architectures, removes cost surprises from long inputs, and delivers 45-plus models through a fully OpenAI-compatible API with no cold starts. If you are building agents, document processors, or API gateways on serverless infrastructure, Oxlo.ai is a relevant option worth evaluating.

DEV Community