Running AI inference one request at a time works well for real-time product experiences. But many workloads do not need an immediate response. Data enrichment, classification, extraction, content moderation, summarization, and offline analytics often involve hundreds or thousands of requests that can be processed asynchronously.
That is where the ZeroGPU Batch API comes in.
With Batch Processing, you can upload a JSONL file, submit it as a batch job, and retrieve the results when processing is complete. It is designed for large asynchronous workloads where throughput, reliability, and simplicity matter more than instant response time.
Why Batch Processing?
Many AI workflows are naturally asynchronous.
For example, you might want to:
- Classify thousands of documents.
- Extract structured data from customer records.
- Run content moderation over historical user-generated content.
- Summarize support tickets, reviews, or research notes.
- Process backfills or recurring data pipelines.
Sending each request individually can add unnecessary orchestration complexity. You need retry logic, request tracking, output matching, rate management, and failure handling.
The Batch API gives you a cleaner workflow.
How It Works
Batch Processing in ZeroGPU follows a simple file-based flow:
- Create a JSONL input file.
- Upload it using the Files API.
- Create a batch using the returned file ID.
- Poll the batch until it completes.
- Download the output and error files.
Each line in the JSONL file represents one request. ZeroGPU processes those requests asynchronously and writes the results back to output files.
A minimal input line looks like this:
{“custom_id”:”request-1",”method”:”POST”,”url”:”/v1/chat/completions”,”body”:{“model”:”your-model-id”,”messages”:[{“role”:”user”,”content”:”Classify this text.”}]}}
The custom_id is returned in the output, so you can match every result back to your original input.
Built For AI Workloads At Scale
The Batch API is especially useful when you need to process a large amount of data without holding open client connections or building your own job orchestration layer.
ZeroGPU currently supports batch jobs for /v1/chat/completions, with JSONL files uploaded through /v1/files.
The core endpoints are:
POST /v1/files to upload input JSONL.
POST /v1/batches to create a batch job.
GET /v1/batches/{batch_id} to check status.
GET /v1/files/{file_id}/content to download results.
This makes batch processing easy to integrate into existing backend systems, cron jobs, data pipelines, and internal tools.
OpenAI-Compatible Shape
ZeroGPU’s Batch and Files APIs are wire-compatible with the OpenAI-style batch workflow, while using ZeroGPU authentication headers:
x-api-key: your-api-key
x-project-id: your-project-id
That means developers familiar with OpenAI batch jobs should feel at home, while still getting ZeroGPU’s routing, project isolation, logging, and model infrastructure.
When Should You Use Batch?
Use the real-time API when your user is waiting for a response.
Use the Batch API when the work can happen in the background.
Good fits include:
- Nightly data processing.
- Bulk document classification.
- Large-scale extraction jobs.
- Offline analytics.
- Backfills.
- Evaluation datasets.
- Reprocessing historical data.
Batch jobs are also easier to audit because each request has a stable custom_id, and outputs are written to downloadable files.
Get Started
The fastest way to try it:
- Prepare a JSONL file.
- Upload it with POST /v1/files.
- Create a batch with POST /v1/batches.
- Poll for completion.
- Download the output file.
You can try the new interactive playgrounds in the ZeroGPU docs:
Upload file: /api-reference/batch/upload-file
Create batch: /api-reference/batch/create-batch
Retrieve batch: /api-reference/batch/retrieve-batch
Download file: /api-reference/batch/download-file
Batch Processing makes it easier to run AI workloads at scale without managing queues, workers, retries, or GPU infrastructure.
ZeroGPU handles the execution. You focus on the data.
Top comments (0)