New advancements in Generative AI

If you’ve tried to build anything deterministic with an LLM lately, you know the pain. You write a prompt, cross your fingers, and hope the model doesn’t hallucinate a python method that hasn't existed since 2018. For a long time, building software around generative AI felt less like engineering and more like whispering to a moody psychic.

That is slowly changing.

The conversation around gen AI has shifted away from raw parameter counts and toward reliability, local execution, and structural control. If you've been heads-down writing CRUD apps for the last year and haven't looked up, here is what actually matters right now.

Structured Outputs Are Finally Usable

The biggest bottleneck in AI development used to be parsing the response. You'd ask an LLM for JSON, it would happily wrap it in markdown code blocks, occasionally append "Sure, here is your JSON!", and sometimes casually drop a trailing comma that broke JSON.parse().

We used to spend half our Python files writing regex band-aids to clean up model output.

Now, major providers and local runtimes support constrained generation. Instead of hoping the model formats things correctly, you enforce a schema at the token level. The model literally cannot generate a token that violates your JSON schema or Pydantic model.

Here is what this looks like using the OpenAI client with Pydantic. If you aren't using this yet, stop doing string parsing immediately.

import os
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

class CodeReview(BaseModel):
    summary: str = Field(description="One sentence summary of the code quality")
    severity_score: int = Field(description="Scale of 1 to 10, where 10 is catastrophic")
    suggested_fix: str = Field(description="The corrected code snippet")

completion = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a harsh code reviewer."},
        {"role": "user", "content": "print('hello world')"}
    ],
    response_format=CodeReview,
)

review = completion.choices.message.parsed

# This is a real Python object, not a string you have to clean
print(f"Severity: {review.severity_score}")
print(f"Fix: {review.suggested_fix}")

The gotcha here: structured outputs take longer to initialize on the first request because the provider has to compile the grammar mask. Don't panic if your first request takes an extra second or two. Subsequent requests are fine.

Local Models Grew Up

Sixteen-gigabyte M-series Macs changed local development, but until recently, running open-weights models felt like a compromise. Llama 3 and its variants changed that. You can now run a model locally that actually understands system prompts, handles tool use reasonably well, and doesn't completely lose the plot if your prompt goes over 500 tokens.

Tools like Ollama made this a single-command setup. You pull a model, and it exposes an OpenAI-compatible API on localhost:11434.

Here is how you swap your production client for a local instance without changing your business logic:

import OpenAI from 'openai';

// Point the standard SDK at your local Ollama instance
const openai = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama', // Ollama doesn't care what this is, but the SDK requires it
});

async function runLocal() {
  const response = await openai.chat.completions.create({
    model: 'llama3.2',
    messages: [{ role: 'user', content: 'Why is my database connection dropping?' }],
    temperature: 0.1,
  });

  console.log(response.choices.message.content);
}

runLocal();

The trade-off? Local models are still resource hogs. If you try to run a 70B parameter model on a laptop with 16GB of RAM, your fan is going to sound like a jet engine and generation speed will drop to one word every three seconds. Stick to the 3B to 8B parameter range for local development unless you've got a dedicated GPU rig sitting under your desk.

Context Windows Are Huge, But Mostly Noise

Models now claim context windows of 128k, 200k, or even millions of tokens. The pitch is simple: dump your entire codebase into the prompt and let the model figure it out.

I tried this. I took an entire mid-sized backend repo, concatenated the files, and shoved it into a context window to ask about a routing bug.

It didn't work very well.

While the models technically can read all those tokens, "needle in a haystack" retrieval degrades the further into the context window the important information sits. If your crucial database config is buried on page 40 of a massive prompt, the model is entirely capable of hallucinating an answer based on generic patterns it saw earlier instead of actually reading your code.

RAG (Retrieval-Augmented Generation) isn't dead. Chunking your code into a vector database and pulling relevant snippets is still significantly more reliable and cheaper than dumping an entire repository into a single massive prompt. Treat massive context windows as a nice-to-have for specific documents, not an excuse to stop writing clean code architecture.

Moving Past the Hype

The generative AI landscape is settling into a normal engineering toolset. It's no longer just magic chat interfaces in a browser wrapper. We have better syntax enforcement, faster local execution, and more predictable APIs.

If you want to actually build something with this stuff this weekend, skip the wrapper apps and the complex multi-agent frameworks.

Pick one boring, repetitive task in your daily workflow—like generating test cases for a specific utility function or parsing messy CSV headers—and write a script using Pydantic structured outputs and a local model.