Building a Language Model from Scratch with LLM and Transfer Learning

#learnai #oxlo #ai

Most tutorials stop at calling a chat API, but building a production-ready language model requires assembling the data layer, prompt adapter, and inference pipeline from scratch. In this guide we will build a domain-specific medical QA model by applying transfer learning to Llama 3.3 70B through Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, adding large few-shot context windows for transfer learning does not inflate cost the way token-based billing would. See https://oxlo.ai/pricing for details.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Initialize the Oxlo.ai client

We start by pointing the OpenAI SDK at Oxlo.ai's inference endpoint. I keep my key in an environment variable so it never hits disk in plaintext.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY")
)

print("Oxlo.ai client initialized")

Step 2: Curate domain data for transfer learning

Transfer learning starts with data. I collected five hundred medical QA pairs into a corpus that acts as the knowledge substrate. For this demo we define a small representative slice in Python.

KNOWLEDGE_BASE = [
    {
        "question": "What are the early symptoms of Type 2 diabetes?",
        "answer": "Increased thirst, frequent urination, fatigue, and blurred vision."
    },
    {
        "question": "How does metformin work?",
        "answer": "It decreases hepatic glucose production and improves insulin sensitivity."
    },
    {
        "question": "What is the first line treatment for hypertension?",
        "answer": "Lifestyle modification plus thiazide diuretics, ACE inhibitors, or calcium channel blockers."
    },
    {
        "question": "Can you explain the mechanism of aspirin?",
        "answer": "Irreversible inhibition of cyclooxygenase enzymes, reducing prostaglandin and thromboxane production."
    },
    {
        "question": "What differentiates a migraine from a tension headache?",
        "answer": "Migraines are typically unilateral, pulsatile, and accompanied by nausea or photophobia."
    }
]

Step 3: Design the system prompt

The system prompt is our soft transfer layer. It encodes domain identity, output structure, and tone without changing any model weights. Treat this as the final layer of a fine-tuned model, but defined entirely in text.

SYSTEM_PROMPT = """You are MedTransfer, a clinical reasoning assistant specialized for medical Q&A.

Rules:
- Answer only from established clinical knowledge.
- Keep responses under three sentences.
- If uncertain, state that the evidence is inconclusive.
- Never provide prescription advice or dosage.

Example interactions:
Q: What are the early symptoms of Type 2 diabetes?
A: Increased thirst, frequent urination, fatigue, and blurred vision.

Q: How does metformin work?
A: It decreases hepatic glucose production and improves insulin sensitivity.

Q: What is the first line treatment for hypertension?
A: Lifestyle modification plus thiazide diuretics, ACE inhibitors, or calcium channel blockers.
"""

Step 4: Build the transfer learning wrapper

Now we wire the knowledge base and prompt into a lightweight model class. The format method injects a few-shot context window, and predict calls Oxlo.ai with a low temperature to keep outputs deterministic.

class TransferLLM:
    def __init__(self, client, system_prompt, examples):
        self.client = client
        self.system_prompt = system_prompt
        self.examples = examples

    def format_context(self, user_question):
        few_shot = ""
        for ex in self.examples[:3]:
            few_shot += f"Q: {ex['question']}\nA: {ex['answer']}\n\n"
        return f"{self.system_prompt}\n\n{few_shot}Q: {user_question}\nA:"

    def predict(self, user_question):
        prompt_text = self.format_context(user_question)
        response = self.client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[
                {"role": "system", "content": "You are a helpful medical assistant. Use the provided examples to guide tone and depth."},
                {"role": "user", "content": prompt_text},
            ],
            temperature=0.1,
            max_tokens=200,
        )
        return response.choices[0].message.content

Step 5: Compare baseline against transfer output

To verify that our transfer layer actually changes behavior, we run the same question through raw Llama 3.3 70B and through our wrapped model. The difference in conciseness and domain tone should be obvious.

test_question = "What are the early symptoms of Type 2 diabetes?"

# Baseline zero-shot call through Oxlo.ai
baseline = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": test_question},
    ],
    temperature=0.1,
    max_tokens=200,
).choices[0].message.content

# Transfer model call
model = TransferLLM(client, SYSTEM_PROMPT, KNOWLEDGE_BASE)
transferred = model.predict(test_question)

print("=== BASELINE ===")
print(baseline)
print("\n=== TRANSFER ===")
print(transferred)

Run it

Save the script as med_transfer.py, export your key, and execute it. You should see the baseline produce a general answer while the transfer model returns a concise, clinically styled response.

export OXLO_API_KEY="sk-oxlo.ai-..."
python med_transfer.py

# Expected output snippet:
# === BASELINE ===
# Type 2 diabetes is a chronic condition that affects the way your body metabolizes glucose...
#
# === TRANSFER ===
# Increased thirst, frequent urination, fatigue, and blurred vision.

Next steps

Swap the static few-shot list for an embedding retrieval pipeline using Oxlo.ai's embedding endpoint with BGE-Large, so the model pulls from thousands of examples instead of three. If you need deeper reasoning, switch the base model to DeepSeek R1 671B MoE or Qwen 3 32B and keep the same prompt architecture.

DEV Community