Effortless Data Extraction: Let Cypress Do the Work!

When you think of Cypress, what’s the first thing that comes to mind? Probably end-to-end (E2E) testing, right? That’s what Cypress is built for—automating UI interactions, asserting elements, and ensuring your app works as expected.

But what if I told you Cypress could do more than just testing? 🤯

Recently I found myself in a situation where I needed to extract dynamic page data from a Next.js application across multiple pages and markets. The manual approach? A nightmare

My Use Case
Let’s start with my scenario. I have a Next.js Page Router application that uses server-side rendering (SSR). Content comes from Contentful CMS, and I have multiple popup entries(urgency banner, promotional popup, lead generation popup) dynamically injected into different pages. These popups contents are fetched from Contentful via an API call inside getServerSideProps and passed to the application as props.

Each page can contain three different types of popups, and these popups are rendered dynamically based on the CMS data.

Why Do I Need to Extract This Data?
I am migrating to a new architecture, where content is fetched differently using a new entry point. However, the popup structure in CMS remains the same, meaning we need to map the existing popup IDs to the new system.

To do this, I must extract all popup IDs currently rendered on each page so they can be properly linked in the new CMS setup.

The Manual Approach: A Tedious Task
In theory, this can be done manually by:

1. Visiting each page.
2. Inspecting the NEXT_DATA payload in the browser.
3. Extracting and listing all popup IDs for that page. However, the challenge is that these popups appear throughout the entire customer journey across multiple pages, including: ✅ Homepage → ✅ Plans → ✅ Register → ✅ Checkout → more pages

Now, multiply this by 17 different markets. That means:

1. Visiting every page in every market
2. Manually extracting popup IDs from each page’s NEXT_DATA
3. Keeping track of all extracted data

This quickly becomes an unmanageable and time-consuming process.

The Solution: Let Cypress Automate It! 🚀
Instead of manually checking each page and extracting data, we can leverage Cypress to do this work for us. Cypress can:

1. Visit all necessary pages automatically
2. Extract popup IDs from NEXT_DATA
3. Store and structure the data dynamically
4. Scale the process across 17 markets effortlessly This automation will eliminate human error, save time, and ensure accuracy—allowing us to focus on the actual CMS migration instead of repetitive manual tasks.

It was clear—I needed automation, but not for testing.

Setting Up Cypress for Data Extraction
Install Cypress (if not already installed):
npm install cypress --save-dev

Run Cypress to generate the default folder structure:
npx cypress open

Understanding __NEXT_DATA__ in Next.js

• Next.js exposes NEXT_DATA inside the window object, which contains valuable server-side rendered data.
• Example structure of NEXT_DATA:

{
  "props": {
    "pageProps": {
      "ssrPayload": {
        "popUpEntries": [
          { "items": [{ "fields": { "tagName": "tag1" } }] },
          { "items": [{ "fields": { "tagName": "tag2" } }] }
        ]
      }
    }
  }
}

Creating a Custom Cypress Command for Reusability
A custom Cypress command is a user-defined function that extends Cypress' built-in commands, allowing you to reuse logic across tests

Why use Cypress Commands?

• Keeps the test clean and reusable
• Allows us to easily extract data from multiple pages

let pageDataStore = {}; // Global object to store data for different pages

Cypress.Commands.add('extractNextData', (pageKey) => {
  cy.window().then((win) => {
    const nextData = win.__NEXT_DATA__;

    // Extract list_of_tags from pageProps
    const list_of_tags =
      nextData.props.pageProps.ssrPayload.popUpEntries.map(
        (entry) => entry.items[0].fields.tagName
      );

    cy.log(`NEXT_DATA for ${pageKey}`, list_of_tags);

    // Store in global object
    pageDataStore[pageKey] = list_of_tags;

    return cy.wrap(list_of_tags);
  });
});

Cypress.Commands.add('getPageDataStore', () => {
  return cy.wrap(pageDataStore);
});

Lets Visiting Multiple Pages and Extracting Data
Create a test file (cypress/e2e/readNextData.cy.js) to automate the process:

describe('Extract and Store NEXT_DATA from Multiple Pages', () => {
  it('Visits multiple pages and logs extracted data', () => {
    cy.visit('/'); // Visit home page
    cy.extractNextData('home'); // Store data under "home"

    cy.visit('/plans'); // Visit plans page
    cy.extractNextData('plan'); // Store data under "plan"

    cy.getPageDataStore().then((data) => {
      cy.log('Final Page Data:', JSON.stringify(data, null, 2)); // Logs the full object
    });
  });
});

✅ This test will:

1. Visit multiple pages (/ and /plans)
2. Extract tagName values
3. Store them dynamically in a global object
4. Log the final structured data

Output (Cypress Logs)
After running the test, Cypress will log something like:

NEXT_DATA for home: ["tag1", "tag2", "tag3"]
NEXT_DATA for plan: ["tagA", "tagB", "tagC"]
Final Page Data: {"home":["tag1","tag2","tag3"],"plan":["tagA","tagB","tagC"]}

Now I have structured data from different pages—all automated!

Scaling Up: Running for Multiple Markets
Since I have 17 markets, I can extend this test dynamically:

👉 Update test to loop through multiple markets:

const markets = ['us', 'uk', 'de', 'fr', 'es']; // Example markets

describe('Extract Data from Multiple Market Pages',
 setMarketConfig(market), 
 () => {
  it('Visits each market page and extracts data', () => {
    markets.forEach((market) => {

      cy.visit(`/?locale=${market}`); // Visit home page
      cy.extractNextData('home'); // Store data under "home"

      cy.visit(`/plans?locale=${market}`); // Visit plans page
      cy.extractNextData('plan'); // Store data under "plan"

    });

    cy.getPageDataStore().then((data) => {
      cy.log('Final Market Data:', JSON.stringify(data, null, 2));
    });
  });
});

🔥 Key Takeaways
✅ Manual work eliminated – No need to visit pages one by one
✅ Cypress automates the extraction – Extracts structured data effortlessly
✅ Reusable approach – Can be extended for different pages and markets
✅ Data is stored dynamically – Ready to use for further analysis

💡 No more clicking through 17 markets manually—Cypress does it for me! 🚀