Deploying LLM Models on Cloud Functions: A Step-by-Step Guide

Deploying large language models inside serverless cloud functions is tempting. You get automatic scaling, pay-per-use billing, and no server management. However, packing multi-gigabyte model weights into a function deployment package, managing GPU runtimes, and surviving cold starts quickly turns a simple prototype into infrastructure heavy lifting. This guide walks through both paths: running a small model directly in a function, and the more robust pattern of using the function as a thin orchestration layer that calls an external inference API. For the latter, Oxlo.ai provides a purpose-built platform that eliminates the operational burden while keeping costs predictable.

Why Serverless for LLMs

Cloud functions excel at event-driven workloads. When you need to summarize a document after an upload, classify a support ticket, or generate code snippets from a CI trigger, a serverless execution environment means you do not pay for idle GPU time. AWS Lambda, Google Cloud Functions, and Azure Functions all offer concurrency controls, managed networking, and integration with object storage and message queues.

The problem is that LLMs are not typical microservices. A quantized 7B parameter model still requires several gigabytes of memory. Loading it from disk into RAM on every cold start can push execution times past timeout limits, and standard serverless tiers rarely offer GPU acceleration. Before you commit to self-hosting, it is worth mapping your latency requirements, payload sizes, and call frequency against the constraints of your chosen provider.

Self-Hosting Small Models

If your use case tolerates longer initialization times and you only need a lightweight model, you can bundle a quantized checkpoint with your function. Below is a minimal Python example for AWS Lambda using a small transformer. It assumes you are working within the 250 MB deployment package limit, or that you have loaded the model from an S3-backed Lambda layer.

import json
import os
from transformers import pipeline

# Loaded outside the handler for reuse across warm invocations
generator = pipeline(
    "text-generation",
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    device="cpu"
)

def lambda_handler(event, context):
    body = json.loads(event.get("body", "{}"))
    prompt = body.get("prompt", "Hello")

    result = generator(prompt, max_new_tokens=50, do_sample=False)
    return {
        "statusCode": 200,
        "body": json.dumps({"output": result[0]["generated_text"]})
    }

This works for toy projects, but it breaks down quickly. Memory limits, lack of GPU, and multi-second cold starts make it unsuitable for production agents, long-context tasks, or anything requiring models larger than a few billion parameters.

External Inference with Oxlo.ai

A more resilient pattern is to keep the cloud function stateless and thin, delegating inference to a managed API. Your function handles authentication, request validation, and pre-processing, then forwards the prompt to a specialized inference provider. This removes model weights from your deployment package, eliminates cold starts caused by model loading, and gives you access to large models such as DeepSeek R1 671B MoE or Llama 3.3 70B without provisioning dedicated hardware.

Oxlo.ai is a developer-first inference platform built for this exact architecture. It offers fully OpenAI SDK-compatible endpoints, flat per-request pricing that does not scale with prompt length, and a catalog of over 45 open-source and proprietary models. Because cost is fixed per request, serverless pipelines that process long documents or multi-turn agent conversations remain predictable. You can explore the exact tiers on the Oxlo.ai pricing page.

Step-by-Step Implementation

The following example uses Google Cloud Functions (2nd gen) with Python, but the Oxlo.ai client works identically in AWS Lambda or Azure Functions. The only requirement is the openai package.

Prerequisites

• A Google Cloud project with Cloud Functions API enabled
• The gcloud CLI installed and authenticated
• An Oxlo.ai API key from your dashboard

Project Structure

oxlo.ai-cloud-function/
├── main.py
└── requirements.txt

Dependencies

In requirements.txt:

functions-framework==3.*
openai>=1.0.0

Function Code

In main.py:

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

def generate_text(request):
    # Parse the incoming JSON payload
    request_json = request.get_json(silent=True)
    if not request_json or "prompt" not in request_json:
        return ("Invalid request: missing 'prompt'", 400)

    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": request_json["prompt"]}
    ]

    try:
        response = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=messages,
            temperature=0.7,
            max_tokens=512
        )
        output = response.choices[0].message.content
        return (
            json.dumps({"output": output}),
            200,
            {"Content-Type": "application/json"}
        )
    except Exception as e:
        return (
            json.dumps({"error": str(e)}),
            500,
            {"Content-Type": "application/json"}
        )

Deployment

Deploy with the gcloud CLI, injecting your Oxlo.ai key as an environment variable:

gcloud functions deploy generate_text \
  --runtime python311 \
  --trigger-http \
  --entry-point generate_text \
  --memory 256Mi \
  --timeout 60s \
  --set-env-vars OXLO_API_KEY=YOUR_OXLO_API_KEY \
  --allow-unauthenticated

Because the function itself does not load a model, memory can stay low and cold starts remain under a second. The heavy lifting happens on Oxlo.ai infrastructure, which offers no cold starts on popular models.

Architectural Patterns

Using a cloud function as a gateway to Oxlo.ai enables several production patterns:

• Request sanitization: Validate and strip PII before sending data to the model provider.
• Multi-modal pipelines: Fetch an image from cloud storage, base64-encode it, and send it to a vision model such as Kimi VL A3B.
• Agentic loops: Use the function to maintain tool state. Call Oxlo.ai with function definitions, execute local business logic based on the model's tool-use response, and iterate.
• Streaming proxies: Forward Server-Sent Events from Oxlo.ai back to the caller without buffering the entire response in memory.

In each case, the function remains small, stateless, and easy to version. The complexity of GPU drivers, model quantization, and batching stays with the inference provider.

Cost Considerations