Serverless platforms like AWS Lambda, Google Cloud Functions, and Azure Functions promise scale-to-zero economics and event-driven scaling, which makes them attractive for sporadic AI workloads. However, deploying large language models directly on these platforms introduces friction around artifact size, memory limits, cold starts, and the lack of GPU acceleration. This guide examines practical patterns for running LLM-backed services on cloud functions, including when to host a small model directly and when to route inference to a managed provider such as Oxlo.ai.
Why Serverless for LLM Workloads
Cloud functions excel at handling variable traffic without provisioning idle GPUs. For applications such as webhook-based summarization, async document processing, or form classification, a function that wakes on demand and sleeps after execution can look cost-efficient on paper. The challenge is that transformer-based models are not naturally serverless-friendly. Weights are large, initialization is slow, and inference without dedicated acceleration is often too latent for real-time use.
Architectural Options
There are two distinct patterns for pairing LLMs with cloud functions.
Self-hosted inference. You package a quantized model, such as Llama 3.2 3B or Gemma 3 4B IT, inside a container image and run it within a function environment that supports larger disk footprints, such as AWS Lambda container images or Google Cloud Run functions. This keeps everything in your cloud account but pushes against memory and CPU ceilings.
External inference API. The cloud function acts as a lightweight orchestration layer. It validates input, manages secrets, preprocesses prompts, and forwards requests to a dedicated inference provider. Oxlo.ai fits this pattern natively. Because Oxlo.ai offers flat per-request pricing and no cold starts on popular models, you get the cost predictability of serverless without forcing a 70B parameter model into a 10 GB container.
Deploying Tiny Models on Cloud Functions
If your workload truly requires on-premise weights or offline execution, you can containerize a small model using llama.cpp or a similar quantized runtime. Below is a representative Dockerfile and handler for AWS Lambda using Python.
FROM public.ecr.aws/lambda/python:3.11
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Download a 4-bit quantized model at build time
RUN curl -L -o /var/task/model.gguf https://huggingface.co/.../model-Q4_K_M.gguf
COPY app.py .
CMD ["app.handler"]
And a minimal handler:
import json
from llama_cpp import Llama
# Load once per execution environment
llm = Llama(model_path="/var/task/model.gguf", n_ctx=2048, verbose=False)
def handler(event, context):
body = json.loads(event.get("body", "{}"))
prompt = body.get("prompt", "")
output = llm(
prompt,
max_tokens=256,
temperature=0.7,
stop=["</s>"]
)
return {
"statusCode": 200,
"body": json.dumps({"text": output["choices"][0]["text"]})
}
This approach works for sub-5B parameter models, but you will still face a multi-second cold start while the model is loaded into memory. Concurrent requests can reuse the execution environment, yet bursts will spawn new instances and repeat the penalty.
The Limits of Serverless Inference
Cloud functions impose hard boundaries that erode the viability of self-hosted LLMs as complexity grows. AWS Lambda allocates up to 10 GB of memory and a 15-minute timeout. Google Cloud Run functions offer up to 32 GiB of memory and a 60-minute timeout, but neither provides GPU access for standard function tiers. Running DeepSeek R1 671B or even Llama 3.3 70B on CPU is impractical, and quantized 7B models often fail to meet latency expectations for user-facing chat.
Security patching, model updates, and scaling logic also become your responsibility. Every revision requires a new container build and redeploy. For teams that want to focus on application logic rather than MLOps, this overhead accumulates quickly.
Managed Inference from Cloud Functions
The more robust pattern is to keep the cloud function thin and stateless while delegating inference to a specialized provider. Oxlo.ai provides fully OpenAI SDK compatible endpoints for over 45 models, including general-purpose LLMs, code models, vision models, and embeddings. Because Oxlo.ai charges a flat cost per API request rather than per token, your cloud function can send long prompts or multi-turn agentic contexts without the cost ballooning associated with token-based providers.
This architecture eliminates model loading delays inside the function. The cold start is limited to your own business logic, which is typically measured in milliseconds. Oxlo.ai handles the model weights, batching, GPU allocation, and scaling.
Implementation: Calling Oxlo.ai from a Cloud Function
Below is a Node.js example for Google Cloud Functions that receives a JSON payload, forwards it to Oxlo.ai, and returns the streamed response.
const { OpenAI } = require('openai');
const client = new OpenAI({
apiKey: process.env.OXLO_API_KEY,
baseURL: 'https://api.oxlo.ai/v1',
});
exports.chat = async (req, res) => {
if (req.method !== 'POST') {
res.status(405).send('Method not allowed');
return;
}
const { messages, model = 'llama-3.3-70b' } = req.body;
try {
const stream = await client.chat.completions.create({
model,
messages,
stream: true,
temperature: 0.7,
});
res.setHeader('Content-Type', 'text/plain; charset=utf-8');
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content || '';
res.write(content);
}
res.end();
} catch (err) {
console.error(err);
res.status(500).json({ error: 'Inference failed' });
}
};
For Python on AWS Lambda, the pattern is identical.
import os
import json
from openai import OpenAI
client = OpenAI(
api_key=os.environ["OXLO_API_KEY"],
base_url="https://api.oxlo.ai/v1",
)
def lambda_handler(event, context):
body = json.loads(event.get("body", "{}"))
messages = body.get("messages", [])
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=messages,
stream=False,
)
return {
"statusCode": 200,
"headers": {"Content-Type": "application/json"},
"body": json.dumps({
"message": response.choices[0].message.content
}),
}
Because the Oxlo.ai endpoint is a drop-in replacement for the OpenAI client, you can test locally and deploy without vendor-specific SDK changes.
Top comments (0)