Transfer learning with LLMs does not always mean fine-tuning. In this guide we will build a DevOps support agent that adapts Llama 3.3 70B to an internal ticket classification and response task using only prompt engineering, few-shot examples, and retrieval. It is useful for any engineering team that needs to specialize open models on private knowledge without managing training clusters.
What you'll need
Python 3.10 or newer, an Oxlo.ai API key from https://portal.oxlo.ai, and the OpenAI SDK.
pip install openai numpyStep 1: Set up the Oxlo.ai client and test the base model
I always start by checking the raw model behavior so I can measure the lift after adaptation. This snippet sends an unmodified support question to Oxlo.ai.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "user", "content": "How do I rotate database credentials in AcmeDeploy?"},
],
)
print(response.choices[0].message.content)
Step 2: Lock the domain with a system prompt
Transfer learning starts with task reformulation. I define a strict JSON schema and domain rules so the model knows exactly what success looks like.
SYSTEM_PROMPT = """You are an AcmeDeploy Level-1 support agent.
Your job is to classify the user's issue and respond with valid JSON.
Rules:
- "category" must be one of: credentials, networking, storage, unknown.
- "severity" must be 1 (low) to 5 (critical).
- "response" must be under 100 words and use only the provided context.
- If you lack facts, set "severity": 3 and ask a clarifying question in "response".
Output only the JSON object. No markdown fences."""Step 3: Verify the reframed behavior
Before adding examples, I check that the system prompt alone changes the output structure.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "How do I rotate database credentials in AcmeDeploy?"},
],
)
print(response.choices[0].message.content)
Step 4: Inject few-shot examples to transfer classification behavior
General models do not know our internal severity scale. I add two labeled ticket and response pairs so the model learns the mapping without any gradient updates.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
few_shot_messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Ticket: I forgot my password and cannot log in."},
{"role": "assistant", "content": '{"category":"credentials","severity":2,"response":"You can reset your password from the login page. Click Forgot password and follow the email link."}'},
{"role": "user", "content": "Ticket: All pods are crash-looping after the latest helm upgrade."},
{"role": "assistant", "content": '{"category":"networking","severity":5,"response":"This is a critical outage. Roll back the helm release immediately with helm rollback and open a P1 incident."}'},
{"role": "user", "content": "Ticket: How do I rotate database credentials in AcmeDeploy?"},
]
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=few_shot_messages,
)
print(response.choices[0].message.content)
Step 5: Ground responses with internal docs using Oxlo.ai embeddings
Next I retrieve facts from our runbooks. I embed three snippets with Oxlo.ai's BGE-Large model and inject the best match into the context window.
from openai import OpenAI
import numpy as np
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
DOCS = [
"To rotate DB credentials in AcmeDeploy, open the Secrets tab, click Rotate, and confirm. The new credentials propagate within 60 seconds.",
"If pods enter CrashLoopBackOff after a Helm upgrade, check imagePullSecrets and run helm rollback.",
"AcmeDeploy storage volumes auto-expand when usage exceeds 80 percent.",
]
def embed(text):
r = client.embeddings.create(model="bge-large", input=text)
return np.array(r.data[0].embedding)
doc_embeddings = [embed(d) for d in DOCS]
def retrieve(query):
q = embed(query)
scores = [np.dot(q, d) for d in doc_embeddings]
return DOCS[np.argmax(scores)]
context = retrieve("How do I rotate database credentials in AcmeDeploy?")
print("Retrieved context:", context)
Step 6: Assemble the full transfer-learned agent
Now I wire retrieval, few-shot examples, and the system prompt into a single function that behaves like a domain-tuned model.
from openai import OpenAI
import json
import numpy as np
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def handle_ticket(ticket_text):
context = retrieve(ticket_text)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Ticket: I forgot my password and cannot log in."},
{"role": "assistant", "content": '{"category":"credentials","severity":2,"response":"You can reset your password from the login page. Click Forgot password and follow the email link."}'},
{"role": "user", "content": "Ticket: All pods are crash-looping after the latest helm upgrade."},
{"role": "assistant", "content": '{"category":"networking","severity":5,"response":"This is a critical outage. Roll back the helm release immediately with helm rollback and open a P1 incident."}'},
{"role": "user", "content": f"Context: {context}\n\nTicket: {ticket_text}"},
]
r = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
response_format={"type": "json_object"},
)
return json.loads(r.choices[0].message.content)
result = handle_ticket("How do I rotate database credentials in AcmeDeploy?")
print(json.dumps(result, indent=2))
Run it
Here is how the agent handles two unseen tickets, including one that is not in the knowledge base.
print(handle_ticket("How do I rotate database credentials in AcmeDeploy?"))
print(handle_ticket("The dashboard shows a purple unicorn error code."))
Example output:
{
"category": "credentials",
"severity": 2,
"response": "Open the Secrets tab, click Rotate, and confirm. The new credentials propagate within 60 seconds."
}
{
"category": "unknown",
"severity": 3,
"response": "I do not recognize that error code. Can you share a screenshot and the request ID from the browser console?"
}Wrap-up
Swap in Qwen 3 32B if you need multilingual ticket classification, or try DeepSeek V3.2 on Oxlo.ai for stronger reasoning on ambiguous inputs. Because Oxlo.ai uses flat per-request pricing, you can stuff the context with long retrieved documents and few-shot examples without the cost scaling you would see on token-based inference providers. See https://oxlo.ai/pricing for plan details.
Top comments (0)