Transfer Learning in LLM: A Comprehensive Guide

#learnai #oxlo #ai

I recently shipped a support triage bot that classifies tickets without any fine-tuning. Instead, I transferred a general-purpose LLM to our domain by packing curated examples into the system prompt. In this guide, I will walk you through the exact setup so you can adapt it to your own data, and I will run it against Oxlo.ai so the input length never affects the per-request cost.

What you'll need

You will need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key.

Python 3.10+
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Initialize the Oxlo.ai client

I keep credentials in environment variables. The Oxlo.ai endpoint is a drop-in replacement for the standard OpenAI client.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Prepare the domain transfer set

The transfer set is the adaptation layer. I collected four representative tickets from our backlog and labeled them. Injecting these into the context transfers the model from general chat to specialist classifier.

TRANSFER_EXAMPLES = [
    {
        "ticket": "I was charged twice for my Pro subscription this month. Can you fix this?",
        "label": "billing_duplicate_charge",
        "urgency": "high"
    },
    {
        "ticket": "How do I export my data to CSV? I can't find the button.",
        "label": "product_how_to",
        "urgency": "low"
    },
    {
        "ticket": "The API returns a 503 error every time I hit the /v1/batch endpoint since 9 AM.",
        "label": "api_outage",
        "urgency": "critical"
    },
    {
        "ticket": "Do you offer SAML SSO on the Enterprise plan?",
        "label": "sales_inquiry",
        "urgency": "medium"
    },
]

Step 3: Construct the system prompt

This system prompt is the only thing that changes the model from general chat to a support triage specialist. It contains the task definition, the output schema, and the few-shot examples.

SYSTEM_PROMPT = """You are a support intent classifier for a B2B SaaS platform.
Your job is to read a customer support ticket and output a JSON object with exactly these keys:
- label: one of [billing_duplicate_charge, product_how_to, api_outage, sales_inquiry, other]
- urgency: one of [low, medium, high, critical]
- confidence: a float between 0.0 and 1.0
- reasoning: one sentence explaining your choice

Here are examples of correct classifications:

Ticket: I was charged twice for my Pro subscription this month. Can you fix this?
Output: {"label": "billing_duplicate_charge", "urgency": "high", "confidence": 0.97, "reasoning": "Duplicate billing errors require immediate refund investigation."}

Ticket: How do I export my data to CSV? I can't find the button.
Output: {"label": "product_how_to", "urgency": "low", "confidence": 0.95, "reasoning": "This is a standard feature question with no service degradation."}

Ticket: The API returns a 503 error every time I hit the /v1/batch endpoint since 9 AM.
Output: {"label": "api_outage", "urgency": "critical", "confidence": 0.99, "reasoning": "Recurring 5xx errors indicate an active incident."}

Ticket: Do you offer SAML SSO on the Enterprise plan?
Output: {"label": "sales_inquiry", "urgency": "medium", "confidence": 0.92, "reasoning": "This is a pre-sales question about plan features."}

Now classify the following ticket. Respond with valid JSON only."""

Step 4: Build the classification function

I wrap the API call in a small function that sends the system prompt and the user ticket. I use JSON mode to enforce the output schema. Because the system prompt is long, flat per-request pricing is useful here. On a token-based provider, every extra example increases the input cost. With Oxlo.ai, the cost stays the same per request, so I can afford to give the model a richer transfer set without surprise bills. You can see the exact plan details at https://oxlo.ai/pricing.

import json

def classify_ticket(ticket_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Ticket: {ticket_text}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )

    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Add a confidence threshold and fallback

In production, I gate anything under 0.85 confidence to human review. This wrapper also catches parsing failures so the pipeline does not crash on malformed responses.

def triage(ticket_text: str, confidence_threshold: float = 0.85):
    try:
        result = classify_ticket(ticket_text)
    except Exception as e:
        return {"label": "escalated", "urgency": "high", "reasoning": f"Parser or API error: {e}"}

    if result.get("confidence", 0.0) < confidence_threshold:
        result["label"] = "escalated"
        result["reasoning"] = f"Confidence {result['confidence']} below threshold {confidence_threshold}."

    return result

Run it

I run the script locally with three unseen tickets. Because the model has been transferred via the domain examples, it correctly recognizes patterns outside generic training distribution.

if __name__ == "__main__":
    test_tickets = [
        "My invoice shows three charges for the same seat. This needs to be resolved today.",
        "Is there a way to sync projects with GitHub Enterprise?",
        "Your status page says all green but I keep getting 500s on the upload endpoint.",
    ]

    for ticket in test_tickets:
        print(f"Ticket: {ticket}")
        print(f"Result: {triage(ticket)}")
        print()

Example output:

Ticket: My invoice shows three charges for the same seat. This needs to be resolved today.
Result: {'label': 'billing_duplicate_charge', 'urgency': 'high', 'confidence': 0.96, 'reasoning': 'Multiple charges for the same seat indicate a duplicate billing error requiring immediate investigation.'}

Ticket: Is there a way to sync projects with GitHub Enterprise?
Result: {'label': 'product_how_to', 'urgency': 'low', 'confidence': 0.88, 'reasoning': 'This is a feature question about integration capabilities.'}

Ticket: Your status page says all green but I keep getting 500s on the upload endpoint.
Result: {'label': 'api_outage', 'urgency': 'critical', 'confidence': 0.95, 'reasoning': 'Discrepancy between status page and actual 5xx errors suggests an unreported incident.'}

Wrap-up

This classifier demonstrates inference-time transfer learning. I took a general model and adapted it to a specific support domain by injecting labeled examples and a strict schema into the context. If you want to go further, replace the static examples with a retrieval step that pulls the most relevant few-shot cases from a vector store based on the incoming ticket. You can also experiment with larger context models on Oxlo.ai, such as DeepSeek V4 Flash or Kimi K2.6, if your transfer set grows beyond what fits into a standard context window.