Simulating Traffic for an OCR Service: Part 1

Artyom Makarov — Mon, 09 Nov 2026 13:37:17 +0000

TL;DR

For OCR-like services, docs per second, rather than RPS is a key metrics to plan capacity for.
Look at the hours when users are busiest - in our example morning p99 is twice the all-day p99. The simulation methods like Monte Carlo can give a workload we can employ to see how those peaks affect waiting time.
Use the worker estimate to decide what to benchmark. Micro-batching could roughly halve the worker count (as in our case) in larger scenarios depending on a setup parameters.

When it comes to system design of your services, it's a good idea to start with understanding your traffic. For an invoice OCR service, one HTTP request might contain a single JPEG or a whole folder of invoices collected during a year.

The business wants users to get the value from the service promptly, even during the peak hours. Engineering has to figure out how much capacity that takes and how long a user can afford to wait while a batch fills.

Let’s build a small capacity calculator, then simulate a day of incoming traffic. All numbers below are assumptions for this example. The Python code is in the research notebook.

Step 1. Estimate the incoming load

Suppose our service extracts text from PDF and JPEG invoices. A typical invoice has one page.

Parameter	Baseline	Larger scenario
Daily active users	1,000	10,000
Uploads per user per day	5	10
Documents per upload	7	7
Morning intensity relative to the daily mean	3×	3×

The first calculation fits on paper.

daily_uploads = dau * avg_requests_per_user
mean_rps = daily_uploads / 86400 # seconds in a day
morning_rps = mean_rps * peak_factor_morning
morning_docs_per_second = morning_rps * avg_docs_per_request

The baseline gives us 5,000 uploads and 35,000 documents per day. That’s 0.05787 uploads/s on average and 0.17361 uploads/s during the morning peak. At seven documents per upload, the morning rate is 1.215 documents/s.

Step 2. Compare online and micro-batch processing

An online worker processes documents one at a time.

Suppose one document takes 650 ms.
A continuously busy worker can then process 1/0.65 = 1.54 documents/s.
Assumption: for micro-batching, suppose the same worker can process eight documents together in 2.6 seconds.
That gives 8/2.6 = 3.08 documents/s. On a real deployment, we would benchmark both times on the same hardware.

Here’s the capacity calculation from compare_methods, reduced to the part that determines the worker count.

from math import ceil

def planned_workers(peak_docs_per_second, docs_per_batch, batch_seconds, utilization):
    docs_per_second_per_worker = docs_per_batch / batch_seconds
    return ceil(peak_docs_per_second / (docs_per_second_per_worker * utilization))

With a target worker utilization of 70%, use (docs_per_batch=1, batch_seconds=0.65) for online and (8, 2.6) for micro-batching.

Scenario	Uploads/day	Morning uploads/s	Morning docs/s	Online workers	Micro-batch workers
1,000 users × 5 uploads	5,000	0.174	1.215	2	1
10,000 users × 10 uploads	100,000	3.472	24.306	23	12

Main takeaways here: micro-batching strategy makes more sense with a larger traffic - documents assemble the batch quicker, the potential worker savings come from processing several documents more efficiently together.

Step 3. Give the traffic a daily profile

Let’s say the morning peak runs from 08:00 to 12:00. To keep the expected total at 5,000 uploads per day, we need to balance those four busy hours against the remaining 20.

If k is the off-peak multiplier, the average multiplier over 24 hours must equal one.

\frac{4 \cdot 3 + 20 \cdot k}{24} = 1 \Rightarrow k = 0.6

Period	Multiplier	Expected uploads/s	Expected uploads in period
08:00–12:00	3	0.17361	2,500
Rest of the day	0.6	0.03472	2,500

What would the upload counter actually show at 08:00:00? Let

N_{t}

be the upload count in second t. We draw it from a Poisson distribution with mean

μ_{t}

, the expected count for that second. We assume independent counts across seconds given this daily profile.

P (N_{t} = k) = \frac{e ^{- μ_{t}} μ _{t k}}{k !}

For a morning second, mu_t=0.17361. The probability of an empty second is

e^{- 0.17361} \approx 84.06%

Uploads in a morning second	Probability
0	84.06%
1	14.59%
2	1.27%
3 or more	0.08%

With a baseline parameters, even during the busiest part of the day most seconds still empty (84%).

For this first simulation, we fix the upload size = 7, so a second with two uploads brings 14 documents.

Step 4. Simulate 100 possible days

We can now give each second its expected count and use SciPy to draw the arrivals. Each row is one simulated day and each column - the same second of the day across all runs.

import numpy as np
from scipy import stats

def simulate_uploads(dau, uploads_per_user, days=100, seed=42):
    mean_rps = dau * uploads_per_user / 86400
    hours = np.arange(86400) / 3600
    morning = (hours >= 8) & (hours < 12)
    expected_counts = mean_rps * np.where(morning, 3.0, 0.6)

    requests = stats.poisson.rvs(
        mu=expected_counts,
        size=(days, 86400),
        random_state=np.random.default_rng(seed),
    )
    return expected_counts, requests, morning

expected_counts, requests, morning = simulate_uploads(1000, 5)
t = 8 * 3600  # 08:00:00

print(expected_counts[t])          # 0.17361... expected uploads
print(requests[:10, t])            # [0 0 0 0 0 0 0 1 0 0]
print(requests[:, t].mean())       # 0.22 uploads, averaged over 100 days

The model gives 08:00:00 the same expectation every day, and the counter shows a different possible outcome each day.

If we increase the number of users to 10,000 users and 10 uploads each, the expected count at 08:00:00 rises to 3.472. The first ten draws are now: [5 2 3 3 4 1 0 4 6 2].

To see the wider pattern, pool the one-second counts across all 100 days and calculate the quantiles. Then repeat for the morning alone.

def quantiles(counts):
    return np.quantile(counts, [0.50, 0.95, 0.99], method="inverted_cdf")

for users, uploads in [(1_000, 5), (10_000, 10)]:
    _, counts, morning = simulate_uploads(users, uploads)
    print(users, quantiles(counts), quantiles(counts[:, morning]))

Scenario	Observation window	p50	p95	p99
Baseline	All seconds	0	1	1
Baseline	Morning seconds	0	1	2
Larger	All seconds	1	4	7
Larger	Morning seconds	3	7	8