Mastering Transfer Learning with LLMs: A Step-by-Step Guide

We are building a support ticket triage agent that transfers general LLM reasoning into a domain-specific classification task using few-shot prompting and embedding retrieval. It helps teams that cannot afford to fine-tune but still need accurate, low-latency routing. We will run everything against Oxlo.ai's inference API so cost stays flat per request regardless of how much context we stuff into the prompt.

What you'll need

• Python 3.10+
• pip install openai numpy
• An Oxlo.ai API key from https://portal.oxlo.ai. The free tier includes 60 requests per day, enough to prototype.

Step 1: Gather domain examples

Transfer learning with LLMs at inference time means giving the model enough domain context to adapt without weight updates. I start with five labeled support tickets that represent the target distribution.

examples = [
    {
        "ticket": "My account balance shows $0 after the wire transfer yesterday.",
        "label": "billing",
        "priority": "high"
    },
    {
        "ticket": "How do I export my data to CSV?",
        "label": "product_help",
        "priority": "low"
    },
    {
        "ticket": "The API returns 500 on every /v1/invoices call since 09:00 UTC.",
        "label": "engineering",
        "priority": "critical"
    },
    {
        "ticket": "I was charged twice for my monthly subscription.",
        "label": "billing",
        "priority": "high"
    },
    {
        "ticket": "Do you support SAML SSO?",
        "label": "product_help",
        "priority": "low"
    },
]

Step 2: Index examples with embeddings

We embed the examples with Oxlo.ai's BGE-Large model so we can retrieve the most relevant ones for any new ticket. This simulates a learned domain memory without training.

import numpy as np
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def embed(texts):
    res = client.embeddings.create(
        model="bge-large",
        input=texts,
    )
    return [d.embedding for d in res.data]

example_texts = [ex["ticket"] for ex in examples]
example_vectors = embed(example_texts)

def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, k=2):
    q_vec = embed([query])[0]
    scores = [cosine_similarity(q_vec, ev) for ev in example_vectors]
    top_idx = np.argsort(scores)[-k:][::-1]
    return [examples[i] for i in top_idx]

Step 3: Write the transfer prompt

The system prompt constrains the generalist model to our taxonomy and output format. Treat this as the fixed bias we transfer to the domain.

SYSTEM_PROMPT = """You are a support triage agent. Your job is to classify incoming tickets and assign a priority.

Rules:
- Choose label from: billing, product_help, engineering.
- Choose priority from: low, high, critical.
- Respond with exactly one line: label | priority
- No extra text."""

Step 4: Build the triage agent

We retrieve the top two similar examples, prepend them as few-shot turns, then call Oxlo.ai. Because Oxlo.ai charges per request rather than per token, adding these examples does not inflate cost.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def triage_ticket(text):
    few_shot = retrieve(text, k=2)
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    for ex in few_shot:
        messages.append({"role": "user", "content": ex["ticket"]})
        messages.append({"role": "assistant", "content": f"{ex['label']} | {ex['priority']}"})
    messages.append({"role": "user", "content": text})

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
    )
    return response.choices[0].message.content.strip()

Run it

Initialize the client and pass three unseen tickets through the pipeline. The few-shot examples adapt the model to our vocabulary without any gradient steps.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

if __name__ == "__main__":
    tests = [
        "My webhook endpoint is timing out after 30s.",
        "I need an invoice for last quarter.",
        "How do I reset my password?",
    ]
    for t in tests:
        print(f"Ticket: {t}")
        print(f"Result: {triage_ticket(t)}")
        print()

Expected output:

Ticket: My webhook endpoint is timing out after 30s.
Result: engineering | high

Ticket: I need an invoice for last quarter.
Result: billing | low

Ticket: How do I reset my password?
Result: product_help | low

Next steps

Expand the example bank to a few hundred tickets and switch the chat completion to JSON mode so you can parse labels programmatically. If you later need to scale throughput, Oxlo.ai's per-request pricing means you can stuff even longer retrieved contexts without watching token meters spin up. See https://oxlo.ai/pricing for plan details.