Bandwidth planning for data collection is often reduced to one question: “How many megabits per second do I need?” The useful answer depends on more than a speed label. Transfer volume, active hours, response size, concurrency, retries, protocol overhead, and target latency all affect the result.
This guide develops a practical model you can reuse for authorized web collection, monitoring, and data-engineering workloads. It also explains why a 1 Gbps connection does not automatically produce 324 TB of useful data every month.
Only collect data you are authorized to access. Respect website terms, robots directives, access controls, rate limits, and applicable law.
1. Start with the units
Network throughput is normally expressed in megabits per second (Mbps), while traffic volume is often expressed in gigabytes or terabytes. Because one byte contains eight bits, the decimal-unit conversion is:
Traffic in TB = Mbps × seconds ÷ 8,000,000
For a workload running continuously for a fixed number of days:
Seconds = days × 86,400
The reverse calculation is:
Required Mbps = target TB × 8,000,000 ÷ seconds
These equations describe theoretical transfer capacity. They assume the selected rate is sustained every second with no idle time, retries, protocol overhead, failed requests, or target-side delays.
| Throughput | 1 day | 7 days | 30 days |
|---|---|---|---|
| 100 Mbps | 1.08 TB | 7.56 TB | 32.4 TB |
| 200 Mbps | 2.16 TB | 15.12 TB | 64.8 TB |
| 400 Mbps | 4.32 TB | 30.24 TB | 129.6 TB |
| 600 Mbps | 6.48 TB | 45.36 TB | 194.4 TB |
| 800 Mbps | 8.64 TB | 60.48 TB | 259.2 TB |
| 1 Gbps | 10.8 TB | 75.6 TB | 324 TB |
The table is a ceiling, not a delivery promise.
2. Add duty cycle and real-world efficiency
Most jobs are not active 24 hours a day. Even while active, they do not convert every bit of available throughput into useful payload. A more realistic model adds two factors:
- Duty cycle: the share of the period during which the workload is actively transferring data.
- Efficiency: the share of available throughput that becomes successful, useful payload after latency, retries, failed responses, and protocol overhead.
The adjusted formula is:
Usable TB = Mbps × seconds × duty cycle × efficiency ÷ 8,000,000
To size a connection for a target volume:
Required Mbps = target TB × 8,000,000 ÷ (seconds × duty cycle × efficiency)
Efficiency is not a universal constant. Measure it with a pilot run against the actual authorized targets.
Worked example: 100 TB in 30 days
Suppose a collection pipeline must deliver 100 TB of useful payload in 30 days. It can run continuously, and a representative pilot shows 65% end-to-end efficiency.
Required Mbps = 100 × 8,000,000 ÷ (2,592,000 × 1.0 × 0.65)
The result is approximately 475 Mbps. A 400 Mbps tier would not meet that target under these assumptions. A 600 Mbps tier provides some operating headroom.
Now assume the same job runs only 12 hours per day. The duty cycle becomes 0.5:
Required Mbps = 100 × 8,000,000 ÷ (2,592,000 × 0.5 × 0.65)
The requirement rises to approximately 950 Mbps. A nominal 1 Gbps connection is now close to the calculated minimum, leaving little room for traffic bursts or worse-than-expected target performance.
This is why monthly volume cannot be inferred from the speed label alone.
3. Estimate bandwidth from request volume
If you know the average successful response size and request rate, estimate payload throughput directly:
Payload Mbps = average response size in kB × requests per second × 8 ÷ 1,000
For example, 100 successful responses per second with an average payload of 250 kB require about 200 Mbps of payload throughput:
250 × 100 × 8 ÷ 1,000 = 200 Mbps
If 15% additional requests are caused by retries, the transfer requirement rises before protocol overhead and idle time are considered.
Concurrency is not the same as requests per second. For a stable I/O-bound workload, a rough relationship is:
Requests per second ≈ active concurrency ÷ average response time in seconds
Fifty concurrent requests with an average response time of 0.5 seconds may approach 100 requests per second. If the average response time rises to two seconds, the same concurrency may produce only about 25 requests per second.
4. Measure the right variables
Before choosing capacity, run a limited and authorized pilot that resembles production. Record at least:
- successful payload bytes;
- requests per second;
- median and 95th-percentile response time;
- success, retry, timeout, and block rates;
- active transfer time versus idle time;
- end-to-end useful throughput;
- peak throughput during bursts.
For browser automation, include downloaded scripts, images, fonts, and API calls if production will load them. Measuring only the initial HTML document can significantly understate traffic.
Do not increase concurrency simply to fill a connection. Target-side rate limits and application behavior remain the controlling constraints. Capacity planning should support a responsible request rate, not override it.
5. Choose headroom deliberately
A practical plan needs room for variation. Instead of selecting the exact calculated minimum, document a headroom policy based on observed volatility.
For a stable workload, 20–30% headroom may be a reasonable starting hypothesis. Workloads with highly variable response sizes, bursty schedules, or frequent retries may require more. These are planning ranges, not guarantees; measurement should decide the final number.
A useful sizing worksheet contains:
| Input | Example |
|---|---|
| Target useful volume | 100 TB |
| Period | 30 days |
| Active hours per day | 24 |
| Pilot efficiency | 65% |
| Calculated minimum | 475 Mbps |
| Selected capacity | 600 Mbps |
| Remaining headroom | about 26% |
Recalculate when target mix, response size, active hours, or retry behavior changes.
6. Compare pricing models using delivered data
A pay-per-GB price and a throughput-based fixed-duration price measure different things. Compare them only after estimating the actual useful volume the workload can deliver.
For a fixed-duration plan, an effective cost per delivered GB can be estimated as:
Effective cost per GB = plan price ÷ measured useful GB delivered
For a metered plan:
Estimated cost = billed GB × provider price per GB
The comparison should use the same workload, period, targets, and success criteria. Avoid comparing a theoretical 24/7 capacity ceiling with a metered bill based on actual use.
You can test different assumptions with the interactive residential proxy bandwidth calculator. The formulas and a bilingual reference are also available in the GitHub bandwidth-calculator repository.
7. A repeatable planning process
- Define the useful data target and deadline.
- Measure average payload, latency, retry rate, and active hours with a limited pilot.
- Calculate the theoretical minimum.
- Apply measured duty cycle and efficiency.
- Add explicit headroom for normal variation.
- Verify that the planned request rate respects target limits.
- Monitor useful throughput after launch and revise the model with real data.
The most important distinction is simple: bandwidth is a rate, while delivered data is an outcome. A reliable estimate connects the two using time, duty cycle, efficiency, and workload measurements rather than assuming that the headline speed will be continuously usable.
Top comments (0)