DEV Community: Mersad

Chrome® Powered Web Scraping with Puppeteer: Boosting Speed and Efficiency

Mersad — Sun, 08 Jan 2023 20:10:52 +0000

Chrome® Automation with Puppeteer : Scrape the Web with Style

Puppeteer is a Node.js library that provides a high-level API for controlling headless Chrome or Chromium over the DevTools Protocol. It is a powerful tool for web scraping because it allows you to scrape websites that use JavaScript, cookies, and other complex features that may not be possible to scrape with a traditional web scraper.

To use Puppeteer for web scraping, you will need to install it using npm (the Node Package Manager). Once installed, you can use Puppeteer in your Node.js script to programmatically control a headless Chrome browser and perform web scraping tasks.

Here is a simple example of how to use Puppeteer to scrape a webpage:


const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://radiojavan.com');

  // Extract data from the page
  const data = await page.evaluate(() => {
    const name = document.querySelector('h1').textContent;
    const price = document.querySelector('.price').textContent;
    return { name, price };
  });

  console.log(data);

  await browser.close();
})();

In this example, Puppeteer is used to open a new page in a headless Chrome browser, navigate to the specified URL, and then extract data from the page by using DOM manipulation methods like querySelector. The extracted data is stored in an object and logged to the console.

Puppeteer also provides many other useful features for web scraping, such as the ability to handle cookies, manipulate the DOM, and simulate user events like clicks and form submissions. With these capabilities, Puppeteer can be used to scrape virtually any modern website.

Here's a real-life app to extract artist-names and song-titles and output in array format to a text file...


const puppeteer = require('puppeteer');
const fs = require('fs');
const { Console } = require('console');

(async () => {
    // browser config
    const browser = await puppeteer.launch({headless:true,
        args: [
        '--start-maximized',
        ],
        defaultViewport: null});
    const page = await browser.newPage();
    await page.goto('https://www.radiojavan.com/');

    await page.waitForSelector('.grid');
    await page.click("#featuredPlaylists > div.grid > a:nth-child(3) > img");
    // node
    const info = await page.evaluate(()=>{
        const songs = document.querySelectorAll(".song");
        const artists = document.querySelectorAll(".artist");
        // song array
        let song = []
        songs.forEach(element => {
            let x = element.textContent.trim()
            song.push(x)
        });
        // artist array
        let artist = []
        artists.forEach(element => {
            let x = element.textContent.trim()
            artist.push(x)
        });

        res = song.concat(artist)
        return res
    })
// file write
await fs.writeFileSync("info.txt", info.join("\r\n"))

// console.log(info)
browser.close()
console.log("Success");
})();

Mastering API Testing and Exploring the Power of Postman: A Comprehensive Guide

Mersad — Sun, 08 Jan 2023 19:55:08 +0000

Efficient API Testing with Postman

Testing APIs with Postman is a crucial step in the development process. It allows you to ensure that your APIs are functioning correctly and returning the expected results.

One of the key features of Postman is the ability to create collections of API requests. These collections can be organized into folders and subfolders, making it easy to group and organize your API tests.

One way to use Postman for API testing is to create a collection of test cases. For each test case, you can specify the API endpoint, the request method (e.g. GET, POST, PUT, DELETE), and any necessary request parameters or headers. You can then use Postman's built-in assertions to verify that the API response meets your expectations.

Another useful feature of Postman is the ability to create environments. This allows you to define a set of variables that can be used across multiple requests in a collection. This is particularly useful if you need to test your API with different sets of data, such as different users or different input values.

In addition to testing individual API endpoints, Postman also provides tools for testing the overall performance of your API. You can use the built-in runner to send multiple requests concurrently, helping you to identify bottlenecks and optimize the performance of your API.

Overall, Postman is a powerful tool for testing APIs, with a range of features that make it easy to set up, run, and analyze your API tests. Whether you're working on a small API or a large, complex system, Postman can help you ensure that your API is performing at its best.

To use Postman for API testing, you can follow these steps:

Install Postman on your computer.
Create a new request in Postman. You can choose from various request methods such as GET, POST, PUT, DELETE, etc. depending on the purpose of your request.
Set the request URL to the API endpoint that you want to test.
Add any necessary headers, query parameters, or body data to the request.
Send the request and view the response.
Examine the response to see if it meets your expectations. You can use the built-in testing and validation features in Postman to automate this process.

Effortless UI Testing with CircleCI and Cypress Integration: A Guide to Testing Components and Functionality on a Website

Mersad — Sun, 08 Jan 2023 18:33:21 +0000

A simple <dev>'s guide to Testing Components and Functionality on a Website

Cypress is a powerful tool for automating UI testing of web applications. It is designed to make it easy to write, debug, and maintain tests, and provides a range of features that can be helpful for testing the functionality and components of a website.

To use Cypress for UI testing, you will need to install it and set up your project. Once set up, you can write your tests using JavaScript and the Cypress API.

Here is a simple example of how to use Cypress to test a UI component on a website:


describe('My Component', () => {
  it('displays the correct text', () => {
    cy.visit('https://www.example.com');
    cy.get('.my-component').should('contain', 'Hello World');
  });
});

In this example, the test visits the specified URL and then uses the get command to locate the element with the class my-component. The should command is then used to assert that the element contains the text "Hello World".

Cypress also provides many other commands and features that can be useful for UI testing, such as the ability to interact with elements, simulate user events, and run tests asynchronously. With these capabilities, Cypress can be used to test a wide range of UI components and functionality on a website.

Integration with CircleCI

CircleCI is a continuous integration and delivery (CI/CD) platform that can be used to automate the build, test, and deployment processes for your software projects. It is designed to be flexible and scalable, and can be easily integrated with a variety of tools and services.

One tool that can be integrated with CircleCI is Cypress, a powerful tool for automating UI testing of web applications. By integrating Cypress with CircleCI, you can set up a pipeline that automatically runs your Cypress tests as part of your CI/CD workflow. This can help you catch and fix UI issues early in the development process, and ensure that your application is fully tested and deployed with confidence.

To integrate Cypress with CircleCI, you will need to add a configuration file to your project and set up the necessary dependencies and environment variables. Here is a simple example of a circle.yml configuration file that runs Cypress tests as part of a CircleCI pipeline:


version: 2.1
jobs:
  build:
    docker:
      - image: cypress/base:10
    steps:
      - run:
          name: Install dependencies
          command: npm install
      - run:
          name: Run Cypress tests
          command: npm run cy:run

In this example, the pipeline first installs the necessary dependencies and then runs the Cypress tests using the cy:run script defined in the package.json file. You can customize this pipeline to fit your specific needs and add additional steps as needed.

Overall, integrating Cypress with CircleCI can be a powerful way to automate and streamline your UI testing process and ensure that your application is fully tested and deployed with confidence.