Stop Prompt Engineering Everything: Smarter LLM API Patterns

So you've added an AI feature to your app. Congrats. Now you're debugging why it works great on Tuesdays but eats itself on Fridays. Here's the real talk: it's not about better prompts—it's about better patterns.

I've watched teams throw tokens at the problem for months. The fix? Structuring how they use the API, not begging the model to be smarter.

The Problem With "Better Prompting"

Everyone's obsessed with crafting the perfect prompt. Chain of thought! Few-shot examples! Temperature tweaking! And yeah, that stuff matters. But it's like optimizing your website's CSS when the database query is doing a full table scan.

The actual win is in how you structure your requests:

• Streaming vs. batch
• Context window management
• Error handling and retry logic
• Token budgeting

Pattern 1: Structured Output Instead of Parsing Text

This kills me. Teams send a prompt like "extract the name and email from this text, return it nicely formatted" and then regex their way through the response.

Use function calling or schema validation instead:

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Extract person info from: " + text}
    ],
    tools=[
        {
            "name": "extract_person",
            "description": "Extract person details",
            "input_schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": "string"},
                    "company": {"type": "string"}
                },
                "required": ["name", "email"]
            }
        }
    ]
)

Now you get JSON back. No parsing, no hallucinations, no "oops the model put HTML in there."

Pattern 2: Batching for Cost & Speed

Running inference one-by-one? You're leaving money on the table.

If you're processing lists (emails, documents, support tickets), group them:

# Bad: 1000 API calls
for item in items:
    result = call_api(item)

# Better: Batch messages, single call per batch
batches = [items[i:i+100] for i in range(0, len(items), 100)]
for batch in batches:
    prompt = "Process these 100 items: " + json.dumps(batch)
    result = call_api(prompt)

Fewer round trips, cheaper, faster. The model handles batch processing fine.

Pattern 3: Caching Context for Multi-Turn Interactions

If your user's doing multiple actions in a session, you're re-sending the same context every time.

Use prompt caching:

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a code reviewer. Maintain context across messages."
        },
        {
            "type": "text",
            "text": large_codebase_or_docs,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Review this function..."}
    ]
)

On the first call, it caches. Every followup in that session reuses it—cheaper and faster.

Pattern 4: Streaming for UX, Not Just Tokens

Streaming isn't just for showing live responses. It's for validating early.

with client.messages.stream(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[{"role": "user", "content": user_input}]
) as stream:
    response_text = ""
    for text in stream.text_stream:
        response_text += text
        # Early exit if you detect something wrong
        if "error" in text.lower() and len(response_text) < 100:
            stream.close()
            return None

You can detect bad responses as they stream in, not after 2000 tokens.

Pattern 5: Exponential Backoff + Timeout for Real Systems

Production APIs fail. Rate limits hit. Networks hiccup.

import time
from anthropic import RateLimitError

max_retries = 3
for attempt in range(max_retries):
    try:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=messages,
            timeout=30
        )
        return response
    except RateLimitError:
        wait_time = 2 ** attempt  # 1s, 2s, 4s
        print(f"Rate limited. Waiting {wait_time}s...")
        time.sleep(wait_time)
    except Exception as e:
        if attempt == max_retries - 1:
            raise
        time.sleep(1)

Don't just let it fail. Make it resilient.

Pattern 6: Token Budget Your Prompts

Know how many tokens you're spending per request. It sneaks up:

system_tokens = count_tokens(system_prompt)
context_tokens = count_tokens(user_context)
required_output = 500  # for the response

available = 4096 - system_tokens - context_tokens - required_output
if available < 100:
    # Truncate context or reject the request
    context = truncate_context(user_context, available)

Prevents surprises. Keeps costs predictable.

The Real Win

None of this is rocket science. It's just thinking about the API as a system, not a magic box.

• Structure your outputs so you don't parse text
• Batch when you can
• Cache context for multi-turn flows
• Stream to catch problems early
• Handle failures gracefully
• Know your token budget

Do this stuff and you'll spend less, go faster, and have way fewer "why did it work yesterday but not today" moments.

---

Stay sharp with AI development tips. Subscribe to the LearnAI Weekly newsletter for patterns like these, real dev stories, and tool roundups.

What patterns are you missing? Hit the comments.