How I Led Capacity Planning for an Approximately 1,000x E-Commerce Traffic Spike

Eiji — Sat, 03 Oct 2026 16:37:54 +0000

I am the tech lead for an influencer e-commerce team. We were preparing for a traffic event expected to receive approximately 1,000× normal traffic. A previous event had saturated the primary database, so we could not treat this as a routine scaling exercise.

The difficult part was not setting up k6 or listing bottlenecks. It was deciding what to test locally and in staging, estimating the infrastructure cost before scaling the environment, and turning the findings into a remediation plan that multiple teams could execute. The work also required tracing a request from the frontend through the backend and into the database and infrastructure.

Split the tests by cost and fidelity

Scaling staging close to production capacity costs money for every hour it runs. I first used a LocalStack-based local environment to evaluate queries, indexes, transaction boundaries, and single-task throughput.

We ran the actual application image with MySQL and replaced external APIs with deterministic stubs. Instead of copying production data, we generated synthetic data with similar table sizes and index cardinality. For single-task tests, Docker CPU and memory limits matched production.

I separated single-endpoint tests from mixed-flow tests based on traffic patterns observed during the previous incident. A simplified mixed-flow scenario looked like this:

export const options = {
  scenarios: {
    flows: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE),
      timeUnit: '1s',
      duration: __ENV.DURATION,
    },
  },
};

export default function () {
  pickByWeight([
    [__ENV.READ_WEIGHT, readEndpoint],
    [__ENV.WRITE_WEIGHT, writeEndpoint],
    [__ENV.CALLBACK_WEIGHT, asyncCallback],
    [__ENV.AUTH_WEIGHT, authEndpoint],
  ])();
}

MySQL ran with long_query_time=0, and the slow query log was cleared before each run. I compared query counts, rows examined, execution plans, and connection hold time—not only response latency.

This exposed missing indexes, duplicate queries, unnecessary data fetching, and long transactions. In one case, the expensive query was not the SQL under investigation but a full scan triggered by an ORM relationship.

Agree on the staging cost before running the test

The staging test required temporarily scaling the database, container workloads, cache, load generators, and an external-service stub. I estimated the hourly cost, expected duration, and spending cap before provisioning them, then agreed on the plan with the engineers responsible for the environment.

Because the test occupied staging, I also coordinated a window that would not block development or QA. Longer runs were scheduled outside normal working hours, with the start time, affected features, and rollback plan shared in advance.

This preparation mattered as much as the test script. It made the cost explicit and allowed the team to repeat tests when a fix needed verification.

Reproduce the complete request path

Each k6 virtual user represented one isolated client with its own token and test data. We modelled two traffic groups:

Authenticated clients repeatedly exercising the read path before the traffic event
New clients arriving at the start, authenticating, and entering the write path

The simplified flow was:

export function sustainedTraffic() {
  pollReadPathUntilStart();
  executeWritePath();
}

export function newSessionTraffic() {
  authenticate();
  executeWritePath();
}

function executeWritePath() {
  createResource();
  enterAdmissionControl();
  pollUntilAdmitted();
  submitWrite();
}

The scenario covered read-heavy requests, session creation, admission control, state-changing requests, and asynchronous callbacks. The external-service stub also emitted callbacks with realistic retry intervals.

An HTTP 200 was not enough to call a run successful. We reconciled k6 application errors, callback state, and persisted records, including domain-invariant checks.

k6 ran on dedicated EC2 instances in the staging environment. The instances and related IAM and network resources were provisioned with Terraform. We selected their size from the target concurrency and iteration duration, adjusted operating-system limits, and saved the dashboard and summary from every run.

Fix the request path across frontend and backend

The largest issue was in the write-path transaction. It held a primary connection while waiting for a reader connection, exhausting the database connection pool even when the database itself still had capacity. Adding more application tasks gave little improvement. Moving the read before the transaction removed most of the errors under the same load.

Other improvements crossed the frontend/backend boundary. We traced each client action through browser requests, API calls, SQL, and infrastructure metrics. We removed redundant requests from the read-heavy client path and reduced duplicate backend calls.

We also avoided direct database reads where short-lived data did not require them. Admission-control state was cached briefly in Valkey, while signed URLs were reused from an in-memory cache inside each ECS task.

Finally, I increased ECS tasks and EKS pods and nodes step by step to measure the throughput of each configuration. From the database connection limit, I created a connection budget that included rolling deployments and scheduled jobs. This established the safe horizontal-scaling ceiling and the prescale configuration for the event.

Turn findings into team-owned work

I worked as an individual contributor on the transaction, query, cache, and request-flow changes. But the scope was larger than one engineer or one codebase.

I grouped the remaining findings by request path and failure mode, then created remediation issues containing the test evidence, proposed change, and expected impact. I shared the plan with the frontend, backend, and infrastructure owners and led prioritisation across the request path.

During each test, I also showed the engineers how client behaviour translated into request volume, admission-control progress, and error conditions. This let us discuss application behaviour and technical limits using the same evidence.

Result

The traffic event completed with a low error rate and stable request latency, without the database saturation seen previously.

The most useful output was not a single maximum request number. It was a shared plan that separated problems requiring code changes from capacity that should be reserved through prescaling—and assigned each change to an owner.

Delivering that plan required working from frontend behaviour through backend transactions to infrastructure capacity, implementing critical fixes myself, and leading the remaining work across teams.

Open to opportunities

I am currently looking for software engineering opportunities in Canada and the UK. I am authorised to work in both Canada and the UK.

If you are hiring for full-stack engineering, cloud infrastructure, SRE, performance engineering, or developer productivity, please get in touch through my portfolio or X.

DEV Community: Eiji