DEV Community

shashank ms
shashank ms

Posted on

Introduction to LLM Inference and Its Applications

#learnai #oxlo #ai

Large language model inference is the process of sending a prompt to a trained model and receiving a generated response. In this tutorial, we will build a Meeting Notes Analyzer that takes a raw transcript and produces a structured summary, extractable action items, and answers to follow-up questions. We will run everything on Oxlo.ai, where flat per-request pricing keeps costs predictable even when we feed long meeting transcripts into the context window.

What you'll need

  • Python 3.10 or newer installed on your machine
  • An Oxlo.ai API key from https://portal.oxlo.ai
  • The OpenAI SDK installed via pip install openai

Step 1: Verify connectivity

Before processing any transcripts, I confirm that the Python environment can reach Oxlo.ai and return a completion. I use Llama 3.3 70B, a reliable general-purpose model.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = "You are a helpful assistant."
user_message = "Explain LLM inference in one sentence."

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

print(response.choices[0].message.content)

Step 2: Define the system prompt and sample transcript

The system prompt constrains the model to a consistent format. I also define a messy sample transcript that we will reuse in every step.

SYSTEM_PROMPT = """You are a Meeting Notes Analyzer. Process raw transcripts and help the team stay organized.

When given a transcript, you must:
1. Summarize the discussion in 3 bullet points.
2. List every action item, assigning an owner if mentioned.
3. Answer follow-up questions using only the transcript content.

Format responses in clean Markdown. If information is missing, say so instead of guessing."""

TRANSCRIPT = """
Alice: We need to ship the auth refactor by Friday. Bob, can you handle the OAuth flow?
Bob: Yes, but I need Carol to review the PR by Thursday.
Carol: I can do Wednesday afternoon. Also, the staging environment is down.
Alice: I'll open a ticket with DevOps. Let's regroup Monday if the staging issue persists.
"""

Step 3: Summarize and extract action items

With the system prompt in place, I send the transcript and ask for the summary and action items together. I switch to Qwen 3 32B because it handles long-context reasoning and agentic workflows cleanly.

SYSTEM_PROMPT = """You are a Meeting Notes Analyzer. Process raw transcripts and help the team stay organized.

When given a transcript, you must:
1. Summarize the discussion in 3 bullet points.
2. List every action item, assigning an owner if mentioned.
3. Answer follow-up questions using only the transcript content.

Format responses in clean Markdown. If information is missing, say so instead of guessing."""
user_message = f"Analyze this meeting transcript:\n\n{TRANSCRIPT}"

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

print(response.choices[0].message.content)

Step 4: Extract structured JSON

For integration with task trackers, I want machine-readable output. I use Kimi K2.6 and JSON mode to enforce a strict schema. Because Oxlo.ai is fully OpenAI compatible, I can pass response_format exactly like the standard SDK.

import json

SYSTEM_PROMPT = "Extract action items from the transcript as JSON. Return a list of objects with keys: task, owner, deadline. If a field is unknown, use null."
user_message = TRANSCRIPT

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"},
)

action_items = json.loads(response.choices[0].message.content)
print(json.dumps(action_items, indent=2))

Step 5: Add follow-up Q&A with conversation history

Finally, I turn the script into an interactive Q&A agent by maintaining message history. I use DeepSeek V3.2 for efficient reasoning, and I append each exchange to the messages list so the model retains context across turns.

history = [
    {"role": "system", "content": "You are a Meeting Notes Analyzer. Answer questions using only the provided transcript."},
    {"role": "user", "content": f"Here is the transcript to reference:\n\n{TRANSCRIPT}"},
]

def ask_question(history, question):
    history.append({"role": "user", "content": question})
    
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=history,
    )
    
    answer = response.choices[0].message.content
    history.append({"role": "assistant", "content": answer})
    return answer

print(ask_question(history, "What deadline did Carol mention?"))
print(ask_question(history, "Who is opening the DevOps ticket?"))

Run it

I save the complete script as meeting_analyzer.py, replace YOUR_OXLO_API_KEY with my key from the Oxlo.ai portal, and run python meeting_analyzer.py. The output below shows the three applications working together.

$ python meeting_analyzer.py

**Summary**
- The team discussed shipping the auth refactor by Friday and assigned Bob to the OAuth flow.
- Carol agreed to review the PR by Wednesday afternoon, though Thursday was initially mentioned.
- Alice will open a DevOps ticket for the staging outage, and the team will regroup Monday if needed.

**Action Items**
- [ ] Bob: Handle OAuth flow
- [ ] Carol: Review PR by Wednesday afternoon
- [ ] Alice: Open DevOps ticket for staging environment

{
  "action_items": [
    {"task": "Handle OAuth flow", "owner": "Bob", "deadline": null},
    {"task": "Review PR", "owner": "Carol", "deadline": "Wednesday afternoon"},
    {"task": "Open DevOps ticket for staging environment", "owner": "Alice", "deadline": null}
  ]
}

Carol mentioned she can review the PR by Wednesday afternoon.

Alice said she will open the ticket with DevOps.

Next steps

Replace the hardcoded TRANSCRIPT with a file reader that loads raw text from stdin or a Zoom export. If you plan to process long recordings, keep in mind that Oxlo.ai charges per request rather than per token, so stuffing a 10,000-word transcript into the context window costs the same as a one-line prompt. For pricing details, see https://oxlo.ai/pricing.

Top comments (0)