Decoding LLMs: Prompt Engineering, API Integration, and Structured Outputs (Part 3)

Mastering LLM API integration is the most critical transition for developers moving from core concepts to building functional software applications. If you missed Part 1: Decoding LLMs: How Large Language Models Work - Fundamentals for Beginners or Part 2: Decoding LLMs: Architecture, Training, and Practical Integration, make sure to check those out first.

When working with Large Language Models (LLMs) in software systems, developers do not train foundation models from scratch. Instead, the daily workload focuses on crafting precise instructions, managing payload configurations, and parsing raw outputs into type-safe, machine-readable data.

In Part 3, we bridge the gap between theoretical AI concepts and functional application code.

---

Core Prompt Engineering Strategies for LLM API Integration

Building predictable applications requires structured prompt engineering techniques that guide model output deterministically.

System Roles vs. User Roles

Modern model providers organize chat interactions into distinct message roles:

• System Role: Defines overall operational rules, output styles, and constraints. > Example: You are a technical documentation assistant. Answer queries concisely using standard Markdown.
• User Role: Supplies dynamic runtime queries, task commands, or input text. > Example: Summarize the following error log into two bullet points: [log output]

Essential Prompting Patterns

1. Zero-Shot Prompting

Requesting a task directly without giving prior examples in the payload.

User: Categorize this support ticket: "Payment failed on checkout screen."
Assistant: Billing

2. Few-Shot Prompting

Providing concrete input-output examples inside the prompt to establish clear formatting and context expectations.

User: Extract key entities into simple key-value pairings.

Input: "Alice purchased 2 books in Seattle."
Output: Name=Alice, Item=books, Quantity=2, Location=Seattle

Input: "Bob returned 1 laptop in Boston."
Output: Name=Bob, Item=laptop, Quantity=1, Location=Boston

3. Chain-of-Thought (CoT) Prompting

Instructing the model to output intermediate logical steps before delivering a final result. This reduces calculation and reasoning errors on complex tasks.

User: A cloud server costs $0.10 per hour. It runs for 24 hours a day for 30 days, with a 10% volume discount applied at the end. Calculate the total cost step by step before stating the final number.

4. Modular Workflows with SKILL.md

When prompts grow long, keeping every instruction in a single system message consumes unnecessary tokens. Modular designs isolate specialized domain guidelines into standardized markdown files, such as SKILL.md:

---
name: code-review-standards
description: Security validation steps for reviewing code snippets.
---
# Instructions
1. Check for unvalidated inputs.
2. Ensure proper error handling and logging.

Why this matters: Using progressive disclosure, an application reads lightweight metadata first and loads the full instructions into the prompt context only when that specific task is invoked. This keeps token costs manageable and allows prompt templates to be version-controlled in Git.

---

Practical Execution and Client Design for LLM API Integration

Understanding the Request Payload

Modern SDKs simplify standard HTTP POST operations. Here is a clean Python example using the standard openai library:

from openai import OpenAI

# Client automatically reads the OPENAI_API_KEY environment variable
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a software engineering assistant."},
        {"role": "user", "content": "Explain REST API rate limiting in two sentences."}
    ],
    temperature=0.2,
    max_tokens=100
)

print(response.choices[0].message.content)

Critical Runtime Parameters

• temperature (0.0 to 2.0): Controls token selection randomness. Use lower values (0.0 - 0.2) for analytical tasks or data extraction. Use higher values (0.7+) for creative text generation.
• top_p (0.0 to 1.0): Nucleus sampling alternative to temperature. A value of 0.1 limits consideration to tokens making up the top 10% probability mass.
• max_tokens: Sets a hard upper bound on generated response length to prevent runaway costs.
• stop: Custom character strings that instantly halt model output generation when encountered.

Handling Token Streaming

To improve user experience, application backends consume Server-Sent Events (SSE) to render output chunks as soon as they are computed:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize Kubernetes pods in three sentences."}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

---

Moving Beyond Unstructured Text: Native Structured Outputs

Free-form string responses are difficult to process in backend applications. Software integrations require reliable, schema-validated JSON structures.

1. Pydantic Parsing with Native Schemas

Modern API endpoints support native response formats that map directly to Pydantic models. This enforces structural validity directly at the API boundary.

from openai import OpenAI
from pydantic import BaseModel, Field

class BugReport(BaseModel):
    title: str = Field(description="Short summary of the issue")
    severity: str = Field(description="Low, Medium, High, or Critical")
    affected_component: str = Field(description="Module or service affected")

client = OpenAI()

completion = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Extract structured bug details from user reports."},
        {"role": "user", "content": "The auth module is throwing HTTP 500 errors during login calls."}
    ],
    response_format=BugReport
)

bug = completion.choices[0].message.parsed
print(f"Title: {bug.title} | Severity: {bug.severity} | Component: {bug.affected_component}")

2. Multi-Provider Validation with Instructor

When deploying across diverse model hosts (such as Anthropic, Gemini, or local endpoints), the instructor library wraps standard SDKs to apply automatic validation retries.

567-labs / instructor

structured outputs for llms

Instructor: Structured Outputs for LLMs

Get reliable JSON from any LLM. Built on Pydantic for validation, type safety, and IDE support.

import instructor
from pydantic import BaseModel


# Define what you want
class User(BaseModel):
    name: str
    age: int


# Extract it from natural language
client = instructor.from_provider("openai/gpt-4o-mini")
user = client.chat.completions.create(
    response_model=User,
    messages=[{"role": "user", "content": "John is 25 years old"}],
)

print(user)  # User(name='John', age=25)

That's it. No JSON parsing, no error handling, no retries. Just define a model and get structured data.

Use Instructor for fast extraction, reach for PydanticAI when you need agents. Instructor keeps schema-first flows simple and cheap. If your app needs richer agent runs, built-in observability, or shareable traces, try PydanticAI. PydanticAI is the official…

View on GitHub

---

Complete Python Example: Production-Ready Data Extraction

The script below demonstrates a reusable integration flow featuring input checks, structured payload execution, error handling, and validation retry loops.

from typing import Optional, Type
from openai import OpenAI, OpenAIError
from pydantic import BaseModel, Field, ValidationError

# 1. Define Target Data Schema
class UserProfile(BaseModel):
    username: str = Field(description="User handle or display name")
    email: str = Field(description="Valid email address")
    role: str = Field(description="Assigned role: Admin, Editor, or Viewer")

# 2. Resilient Data Handler
def extract_user_profile(
    text_input: str,
    schema_class: Type[BaseModel],
    max_retries: int = 3
) -> Optional[BaseModel]:
    # Validate non-empty input
    if not text_input or not text_input.strip():
        print("Input text cannot be empty.")
        return None

    client = OpenAI()

    messages = [
        {
            "role": "system",
            "content": "Extract structured entities strictly adhering to the requested schema."
        },
        {
            "role": "user",
            "content": text_input
        }
    ]

    for attempt in range(max_retries):
        try:
            completion = client.beta.chat.completions.parse(
                model="gpt-4o-mini",
                messages=messages,
                response_format=schema_class
            )

            parsed_data = completion.choices[0].message.parsed
            if parsed_data:
                return parsed_data

        except OpenAIError as api_err:
            print(f"API error on attempt {attempt + 1}: {api_err}")
        except ValidationError as val_err:
            print(f"Validation failure on attempt {attempt + 1}: {val_err}")
            # Append validation error feedback to context for retry
            messages.append({
                "role": "user",
                "content": f"The previous output failed validation: {val_err}. Please output valid JSON matching the schema."
            })

    return None

# Execution Flow
if __name__ == "__main__":
    raw_text = "Please create an account for Sarah Connor ([e-mail]) with Admin privileges."
    profile = extract_user_profile(raw_text, UserProfile)

    if profile:
        print("Successfully parsed profile:")
        print(profile.model_dump_json(indent=2))
    else:
        print("Failed to extract a valid user profile.")

---

Key Takeaways

• Role Design: Use System messages for global constraints and User messages for variable input data.
• Modular Context: Keep prompts efficient using standards like SKILL.md to load instructions dynamically.
• Parameter Tuning: Set temperature low (0.0 - 0.2) for factual tasks and bound outputs using max_tokens.
• Type Safety: Enforce schema validation using native Pydantic integrations instead of relying on manual text parsing.
• Resilient Architecture: Protect API calls using explicit exception handling and validation feedback retry loops.

---

What's Next?

In Part 4 of this series, we shift from isolated API interactions to Building AI Agents & Multi-Step Workflows. We will explore state management, tool execution, memory strategies, and autonomous task loops!

How are you handling structured JSON validation and retries in your current applications? Let us know in the comments below!

Further Reading

Prompt engineering | OpenAI API

Learn strategies and tactics for better results using large language models in the OpenAI API.

developers.openai.com

Pydantic Validation | Pydantic Docs

pydantic.dev

567-labs / instructor

structured outputs for llms

Instructor: Structured Outputs for LLMs

Get reliable JSON from any LLM. Built on Pydantic for validation, type safety, and IDE support.

import instructor
from pydantic import BaseModel


# Define what you want
class User(BaseModel):
    name: str
    age: int


# Extract it from natural language
client = instructor.from_provider("openai/gpt-4o-mini")
user = client.chat.completions.create(
    response_model=User,
    messages=[{"role": "user", "content": "John is 25 years old"}],
)

print(user)  # User(name='John', age=25)

That's it. No JSON parsing, no error handling, no retries. Just define a model and get structured data.

Use Instructor for fast extraction, reach for PydanticAI when you need agents. Instructor keeps schema-first flows simple and cheap. If your app needs richer agent runs, built-in observability, or shareable traces, try PydanticAI. PydanticAI is the official…

View on GitHub