Designing reliable uploads for large datasets using Playwright

🔥 Chunked Upload Nightmare in Playwright

Can you test a 5 GB CSV upload without loading the whole file into memory?

📌 Problem Statement

Enterprises let users drop massive CSVs (5 GB+) that the UI splits into 10 MB chunks. The browser sends each chunk with X‑Chunk‑Index and X‑Total‑Chunks headers. Traditional page.setInputFiles() can’t emulate this because it streams the whole file and hides the HTTP calls.

💡 Solution & Code Walkthrough

1️⃣ Create on‑the‑fly chunks – use a generator that yields Buffer.alloc(chunkSize, 0x61) (or random data) without persisting a file.

2️⃣ Drive the upload via Playwright’s API request context – page.request lets you fire raw HTTP calls, set custom headers, and inspect responses.

3️⃣ Maintain state – keep totalChunks, increment index, and reuse the same auth cookies/session from the page.

import { test, expect } from '@playwright/test';
const CHUNK_SIZE = 10 * 1024 * 1024; // 10 MB
const TOTAL_SIZE = 5 * 1024 * 1024 * 1024; // 5 GB
const TOTAL_CHUNKS = Math.ceil(TOTAL_SIZE / CHUNK_SIZE);

// Simple async generator for synthetic chunks
async function* chunkGenerator() {
  for (let i = 0; i < TOTAL_CHUNKS; i++) {
    // Fill with repeatable data to keep memory cheap
    const buf = Buffer.alloc(Math.min(CHUNK_SIZE, TOTAL_SIZE - i * CHUNK_SIZE), 0x61);
    yield buf;
  }
}

test('reliable chunked upload', async ({ page }) => {
  // 1️⃣ Navigate & capture auth cookies
  await page.goto('https://app.example.com/upload');
  const cookies = await page.context().cookies();

  // 2️⃣ Prepare request context with same cookies
  const request = page.request;
  await request.setExtraHTTPHeaders({ cookie: cookies.map(c => `${c.name}=${c.value}`).join('; ') });

  // 3️⃣ Stream chunks sequentially
  let index = 0;
  for await (const chunk of chunkGenerator()) {
    const response = await request.post('https://api.example.com/upload/chunk', {
      headers: {
        'Content-Type': 'application/octet-stream',
        'X-Chunk-Index': String(index),
        'X-Total-Chunks': String(TOTAL_CHUNKS),
      },
      data: chunk,
    });
    expect(response.ok()).toBeTruthy();
    index++;
  }

  // 4️⃣ Verify final assembly endpoint
  const finish = await request.post('https://api.example.com/upload/complete', {
    json: { fileName: 'big-data.csv', totalChunks: TOTAL_CHUNKS },
  });
  expect(await finish.json()).toMatchObject({ status: 'SUCCESS' });
});

✅ Why this works

• No full file in RAM – each chunk is generated on demand.
• Full control of headers – mimics the UI’s exact protocol.
• Playwright‑native session – reuses cookies, CSRF tokens, and can still run UI steps (e.g., drag‑and‑drop) before the API loop.

🔑 Key Takeaways

• Use page.request for low‑level HTTP when UI helpers fall short.
• Chunk generators keep memory footprint constant (O(chunkSize)).
• Preserve session state by copying cookies or auth tokens from the page context.

❓ Quick Summary Q&A

❌ Can I still use setInputFiles? – No, it hides the chunk traffic.

✅ Do I need a real 5 GB file? – No, synthetic buffers are enough.

✅ How to assert server‑side assembly? – Call the final “complete” endpoint and check its JSON response.

TAGS: playwright, typescript, e2e-testing, file-upload, automation

────────────────────────────────────────

📲 𝐅𝐑𝐄𝐄 𝐌𝐎𝐁𝐈𝐋𝐄 𝐀𝐏𝐏 — 𝟔𝟎𝟎+ 𝐒𝐃𝐄𝐓 𝐐&𝐀𝐬
Practice real-world interview scenarios offline on the free QA Automation & SDET Prep app:

🤖 𝐆𝐨𝐨𝐠𝐥𝐞 𝐏𝐥𝐚𝐲 (𝐀𝐧𝐝𝐫𝐨𝐢𝐝):
https://play.google.com/store/apps/details?id=com.app.seleniuminterviewquestions&referrer=utm_source%3Ddevto%26utm_medium%3Darticle%26utm_campaign%3Dselenium_20260911

🍎 𝐀𝐩𝐩 𝐒𝐭𝐨𝐫𝐞 (𝐢𝐎𝐒):
https://apps.apple.com/app/id6786760948?pt=128640464&ct=devto_selenium_20260911&mt=8

────────────────────────────────────────