New advancements in Generative AI

If you spent last year fine-tuning massive language models just to get decent JSON output, you probably have some fatigue. Every week brings a new framework, a new context window length, and a new benchmark that somehow claims AGI is arriving next Tuesday.

Most of that noise doesn't matter for day-to-day development. But a few architectural shifts over the last several months actually change how we build software with LLMs. We are moving past the phase of just throwing prompts at an API and hoping for the best.

Here is what is actually working right now, minus the marketing hype.

Structured Outputs Are Finally Reliable

For a long time, forcing an LLM to return valid JSON felt like negotiating with a toddler. You would write elaborate system prompts saying "OUTPUT ONLY VALID JSON OR ELSE," and the model would still casually prepend a cheerful "Sure, here is your data!" right before breaking the syntax.

Now, major providers like OpenAI, Anthropic, and open-weight runners like Ollama support native constrained generation. Instead of hoping the model follows instructions, the API enforces a JSON schema at the token generation level. If the model tries to output a token that violates the schema, that token is literally blocked.

Here is what that looks like in Python using Pydantic, which has become the de facto standard for defining these schemas:

import os
from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

class BugReportAnalysis(BaseModel):
    severity: int = Field(description="Severity score from 1 to 5")
    category: str = Field(description="UI, Backend, Database, or Performance")
    summary: str = Field(description="One sentence summary of the core issue")
    is_actionable: bool

completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Analyze the user bug report."},
        {"role": "user", "content": "The checkout button is throwing a 500 error when clicking submit on Safari."}
    ],
    response_format=BugReportAnalysis,
)

analysis = completion.choices.message.parsed
print(f"Category: {analysis.category}")
print(f"Actionable: {analysis.is_actionable}")

This saves hours of writing regex cleanup scripts and retry loops. If you are still parsing raw string outputs with json.loads() and crossing your fingers, stop. Use structured decoding.

Local Function Calling Without Breaking the Bank

A few months ago, if you wanted local models to do function calling (or tool use), you were largely stuck configuring complex grammars in Llama.cpp or dealing with erratic behavior from smaller models.

That has changed. Models in the 7B to 14B parameter range—like Llama 3.1 and Mistral Nemo—now handle tool definitions natively. You can run them locally via Ollama and get deterministic JSON payloads for function execution without paying per-token cloud costs.

Here is a quick setup using the official Ollama Python client:

import ollama

def get_current_weather(location: str, unit: str = "celsius"):
    """Get the current weather for a given location."""
    # Dummy implementation for demo
    return {"location": location, "temperature": "22", "unit": unit}

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'What is the weather like in Tokyo right now?'}],
    tools=[get_current_weather],
)

# Check if the model wants to call a function
if response.get('message', {}).get('tool_calls'):
    for tool in response['message']['tool_calls']:
        if tool['function']['name'] == 'get_current_weather':
            args = tool['function']['arguments']
            result = get_current_weather(**args)
            print("Function output:", result)
else:
    print(response['message']['content'])

The gotcha here: smaller local models are easily distracted. If you give a 7B model ten different tools with overlapping descriptions, it will hallucinate arguments or pick the wrong tool half the time. Keep your tool definitions sparse and distinct. If your agent needs twenty tools, you are probably doing architecture wrong.

Context Caching for RAG

If you are building Retrieval-Augmented Generation (RAG) applications, you know that stuffing a 50-page technical PDF into the context window for every user query gets expensive and slow very quickly.

Providers are rolling out context caching. Instead of sending the same static reference documents over and over on every API call, you cache the prompt prefix on the provider's servers. You only pay a fraction of the input cost for the cached tokens, and time-to-first-token drops dramatically.

I tried building a custom caching layer with Redis before native provider caching existed. It was more trouble than it was worth—managing token counts, expiration TTLs, and cache invalidation when source docs updated turned into a maintenance nightmare. Letting the API handle it natively is the way to go.

The main thing to watch out for here is minimum token thresholds. Most providers require a minimum payload size (often 32,000 or 64,000 tokens) before caching kicks in. Don't bother trying to cache a two-paragraph system prompt; it won't trigger the discount.

Where This Leaves Us

The wild west phase of generative AI development is cooling down. We have better primitives now: strict schemas, reliable local execution, and cheaper context management. It is starting to feel less like magic and more like standard backend engineering—frustrating at times, but predictable enough to actually build products on.

Next Step

Pick one small script or internal tool you wrote six months ago using raw string prompting. Refactor it to use native structured outputs (either via OpenAI's Pydantic integration or Ollama's format parameter) and measure how many lines of error-handling boilerplate you get to delete.