New advancements in Generative AI

Most AI tutorials start with a 500-word essay about how Alan Turing would be so proud of our current chat bots. Let's skip that.

If you've spent any time building with LLMs over the last year, you already know the core frustration: models are great at generating text tokens in a vacuum, but the second you ask them to interact with a real system, they hallucinate function arguments, forget state, or run out of context window right when things get interesting.

The landscape has shifted past basic prompt engineering. We're moving away from giant, monolithic prompts toward structured reasoning loops, native tool use, and smaller models that punch way above their weight. Here is what actually matters right now if you're building software with AI.

Structured Outputs Without Losing Your Mind

For a long time, getting JSON back from an LLM meant writing a desperate system prompt ("YOU MUST RETURN ONLY VALID JSON OR ELSE") and praying the model didn't add a conversational markdown wrapper at the end anyway. Your parser would inevitably crash at 3:00 AM on a Saturday because the model decided to include a friendly trailing comma.

Major providers finally fixed this at the API level. Instead of hoping the model formats text correctly, you can now enforce a JSON schema directly at the decoding step. The model physically cannot generate tokens that violate your schema.

Here is how you actually do it using the official OpenAI client in Python. No regex hacks required.

import os
from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# Define what you actually want back using Pydantic
class CodeReview(BaseModel):
    summary: str = Field(description="One sentence summary of the code quality")
    severity_score: int = Field(description="Score from 1 to 10 on how broken this is")
    refactored_code: str = Field(description="The fixed version of the code")

completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a ruthless senior code reviewer."},
        {"role": "user", "content": "Review this: `print(x)` where x is undefined."}
    ],
    response_format=CodeReview,
)

# This is a Pydantic object, not a raw string you have to json.loads()
result = completion.choices[0].message.parsed

print(f"Severity: {result.severity_score}/10")
print(f"Fix:\n{result.refactored_code}")

The gotcha here: structured outputs take more compute on the provider's end. Your latency will go up slightly, and if your Pydantic schema is unnecessarily nested with ten levels of optional fields, the API will occasionally throw a 400 error because the constrained decoding path got too complex. Keep your schemas flat.

Local Models Are Actually Usable Now

A year ago, running an open-weight model locally on your laptop felt like watching a slideshow rendered on a Game Boy. Llama 3 and its successors changed that. You can now run a 7B or 8B parameter model locally via Ollama that handles basic classification, entity extraction, and simple code generation fast enough to use in a local development loop without burning through API credits.

This matters for data privacy, obviously, but it also matters for cost and latency when you're processing high-volume, low-complexity tasks.

Here is a quick script to hit a local Ollama instance using the standard OpenAI-compatible endpoint. You don't even need a new SDK.

from openai import OpenAI

# Point the client at your local Ollama instance
client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama', # required, but can be anything
)

response = client.chat.completions.create(
    model="llama3",
    messages=[
        {"role": "system", "content": "Extract all email addresses from the text. Return a comma-separated list."},
        {"role": "user", "content": "Reach out to [e-mail] or [e-mail] for help with the migration."}
    ],
    temperature=0.0
)

print(response.choices[0].message.content)

The reality check: don't expect an 8B local model to write a full microservice architecture from scratch. It will hallucinate imports, forget context after a few thousand tokens, and occasionally get stuck in loops if you set the temperature too high. Use local models for narrow, bounded tasks where failure is cheap and easy to catch.

Agentic Loops and Tool Use

The biggest shift in how we architect AI applications isn't about the model getting smarter—it's about the scaffolding we build around it. Instead of a single prompt-response cycle, modern architectures use loops where the model can inspect its own output, call external tools, and iterate until a condition is met.

Frameworks like LangChain tried to abstract all of this away, and honestly, they created more abstraction layers than most codebases needed. I spent three days debugging a LangChain agent before throwing it out and writing 50 lines of plain Python using native tool calling.

When you write the loop yourself, you can actually see what the model is doing:

import json
import requests
from openai import OpenAI

client = OpenAI()

def get_current_weather(location):
    """Get the current weather for a given city."""
    # Dummy implementation for illustration
    return json.dumps({"location": location, "temperature": "72F", "condition": "Partly Cloudy"})

# Map available tools to their Python functions
available_tools = {
    "get_current_weather": get_current_weather
}

tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather for a given city",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}
            },
            "required": ["location"]
        }
    }
}]

def run_conversation():
    messages = [{"role": "user", "content": "What's the weather like in Seattle right now?"}]

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        tools=tools,
        tool_choice="auto",
    )

    response_message = response.choices[0].message

    # Check if the model wants to call a tool
    if response_message.tool_calls:
        messages.append(response_message)

        for tool_call in response_message.tool_calls:
            function_name = tool_call.function.name
            function_to_call = available_tools[function_name]
            function_args = json.loads(tool_call.function.arguments)

            # Execute our local function
            tool_output = function_to_call(**function_args)

            messages.append({
                "tool_call_id": tool_call.id,
                "role": "tool",
                "name": function_name,
                "content": tool_output,
            })

        # Second round: Send the tool output back to the model so it can answer the user
        second_response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )
        return second_response.choices[0].message.content

print(run_conversation())

The hard part here isn't the code. The hard part is managing infinite loops when the model decides it needs to call the same tool four times in a row because it didn't parse the output correctly. Always set a hard iteration limit (max_loops = 5) on any agentic workflow you write.

Context Windows Are Great, But Watch Your Costs

We now have models that accept millions of tokens of context. You can drop an entire codebase into a prompt and ask questions about it. It feels like magic the first time you do it.

Just remember that with most API providers, you pay for the entire context window on every single request in a conversation thread if you're passing history back and forth. If your context balloons to 100k tokens and you chat back and forth ten times, you are burning through budget fast for simple queries.

Use large context windows for specific ingestion tasks—like summarizing a massive PDF or analyzing a dump of logs—and prune your message histories ruthlessly for standard chat interfaces.

Next Steps

Don't try to build a complex autonomous agent right away. Download Ollama, pull llama3, and write a simple script that uses Pydantic structured outputs to parse messy text input into a clean database record. See where the model fails, adjust your schema, and go from there.