You see all these complex guides on prompt engineering, model routing, caching... all promising to shave a few percentage points off your LLM bill. They're not wrong, but they're missing the forest for the trees.
OpenAI and Anthropic are literally offering a 50% discount on their most powerful models. And it seems like nobody is talking about it.
It's called the Batch API.
It's not new, it's not sexy, and it doesn't involve any "AI magic". It's just a different endpoint. Instead of sending requests one by one, you bundle them up and send them in a single file. 24 hours later, you get a file back with the results.
The catch? It's async. Heavily async. You can't use this for a real-time chatbot. The 24-hour SLA is real.
But for any non-interactive workload? It's a no-brainer.
- Summarizing articles for a newsletter.
- Generating embeddings for a RAG pipeline.
- Classifying user-generated content.
- Running batch evaluations on a new prompt.
All these tasks are perfect for batching. You're probably running them on the real-time endpoint right now and paying double for the privilege.
The implementation is a bit different. You upload a JSONL file with your requests, start the batch job, and then poll for completion. It's a bit more work than a simple POST request, but it's not rocket science.
We wrote a more detailed breakdown of the pricing and gotchas here: https://simplifai.tools/blog/llm-batch-api-pricing
Honestly, before you spend a week implementing a caching layer, just check if your workload can be batched. It's the most boring, most effective way to cut your LLM costs. fwiw.
Top comments (1)
I completely agree with the author that the Batch API is an underutilized feature for reducing LLM costs. I've been using the real-time endpoint for my non-interactive workloads and I had no idea I could be saving 50% by switching to the Batch API. One thing I'm curious about is how the 24-hour SLA affects the overall workflow - are there any strategies for handling tasks that require faster turnaround times, or is it simply a matter of scheduling batch jobs during off-peak hours? I'd love to hear more about how others have implemented the Batch API in their workflows.