DEV Community: Ian Gabriel Oliveira de Sousa

How to Create a Web Crawler with Puppeteer and Bun

Ian Gabriel Oliveira de Sousa — Wed, 05 Jun 2024 17:13:36 +0000

Web crawling is a powerful technique used to gather data from websites. Whether you're collecting data for research, monitoring prices, or scraping content, building a web crawler can be incredibly useful. In this post, I'll walk you through the process of creating a web crawler using Puppeteer and Bun, two popular JavaScript tools.

Introduction to Puppeteer and Bun

Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium browsers. It's perfect for web scraping and automating browser tasks.

Bun is a fast, modern JavaScript runtime similar to Node.js but optimized for speed and performance. It's designed to work seamlessly with existing JavaScript libraries.

Step-by-Step Guide to Building a Web Crawler

Step 1: Setting Up Your Environment
First, ensure you have Node.js installed. Then, install Bun by following the instructions on the Bun website.

Next, create a new project directory and initialize it with Bun:

# Copy code
mkdir web-crawler
cd web-crawler
bun init

Step 2: Installing Puppeteer
Install Puppeteer using Bun:

# Copy code
bun add puppeteer
bun node_modules/puppeteer/install.js # -> the secret sauce

Step 3: Writing the Web Crawler Script
Create a new JavaScript file, crawler.js, and start by importing Puppeteer:

// Copy code
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');

  // Extract data
  const data = await page.evaluate(() => {
    return document.querySelector('h1').innerText;
  });

  console.log(data);

  await browser.close();
})();

In this script, we launch a headless browser, navigate to a website, and extract the text of an

element.

Step 4: Running the Script
Run your script using Bun:

## Copy code
bun run crawler.js
## You should see data printed in your terminal.

Conclusion

Creating a web crawler with Puppeteer and Bun is straightforward and efficient. Puppeteer handles the browser automation, while Bun provides a fast and modern runtime for your JavaScript code. This combination makes for a powerful tool in your web scraping toolkit.

For more advanced use cases, you can extend your script to handle navigation, interact with page elements, and scrape more complex data structures. Happy crawling!

About the Author

I am Ian, a practiced computer programmer with a strong interest in website design and automation. I have worked extensively with Web Technologies and always follow the latest in technological advancements enabling me to help others create effective scalable applications.

How to Create a Web Crawler with Puppeteer and Bun

Ian Gabriel Oliveira de Sousa — Wed, 05 Jun 2024 17:13:36 +0000

Introduction to Puppeteer and Bun

Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium browsers. It's perfect for web scraping and automating browser tasks.

Bun is a fast, modern JavaScript runtime similar to Node.js but optimized for speed and performance. It's designed to work seamlessly with existing JavaScript libraries.

Step-by-Step Guide to Building a Web Crawler

Step 1: Setting Up Your Environment
First, ensure you have Node.js installed. Then, install Bun by following the instructions on the Bun website.

Next, create a new project directory and initialize it with Bun:

# Copy code
mkdir web-crawler
cd web-crawler
bun init

Step 2: Installing Puppeteer
Install Puppeteer using Bun:

# Copy code
bun add puppeteer
bun node_modules/puppeteer/install.js # -> the secret sauce

Step 3: Writing the Web Crawler Script
Create a new JavaScript file, crawler.js, and start by importing Puppeteer:

// Copy code
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');

  // Extract data
  const data = await page.evaluate(() => {
    return document.querySelector('h1').innerText;
  });

  console.log(data);

  await browser.close();
})();

In this script, we launch a headless browser, navigate to a website, and extract the text of an

element.

Step 4: Running the Script
Run your script using Bun:

## Copy code
bun run crawler.js
## You should see data printed in your terminal.

Conclusion

For more advanced use cases, you can extend your script to handle navigation, interact with page elements, and scrape more complex data structures. Happy crawling!

About the Author

Building a Decentralized Web Application with IPFS: A Step-by-Step Guide

Ian Gabriel Oliveira de Sousa — Wed, 30 Aug 2023 19:41:57 +0000

In an era where data privacy and ownership have become paramount, decentralized technologies are gaining ground. The InterPlanetary File System (IPFS) offers a revolutionary way to build web applications that are not only decentralized but also resilient, efficient, and censorship-resistant. This blog post will delve into IPFS and explore how to build a decentralized web application using this groundbreaking technology.

Understanding IPFS

IPFS, or InterPlanetary File System, is a protocol and network designed to create a peer-to-peer method of storing and sharing hypermedia in a distributed file system. Unlike traditional HTTP, IPFS uses content-addressable storage, which means that files are identified by their content, not their location. This eliminates the need for central servers and ensures data integrity.

Setting Up Your Development Environment

Before building your decentralized web application, you must set up your development environment. Install the IPFS software and any necessary libraries for your chosen programming language. Familiarize yourself with the IPFS command-line interface (CLI) to interact with the network.
Install IPFS CLI
obs: Install IPFS Node

Designing Your Web Application

Begin by outlining the design and features of your decentralized web application. Consider how you'll break down your application into smaller components and files that can be stored on IPFS. This step is crucial as it will help you determine which files must be stored, shared, and updated on the IPFS network.

Storing Data on IPFS

IPFS creates a unique hash for each file based on its content. To store data on IPFS, add your files to the IPFS network using the CLI or libraries available in your programming language. This process generates a hash ,the CID (Content Identifier) that can be used to retrieve the file from any node on the network.

Building the User Interface (UI)

Develop the user interface of your web application as you would with any traditional app. However, instead of referencing files by URLs, you'll reference them using their IPFS hashes. This enables your app to fetch data from the IPFS network directly, making it decentralized and reducing reliance on centralized servers.

Implementing IPFS Integration

Integrate IPFS functionality into your application's code. This involves utilizing the IPFS APIs to interact with the network programmatically. You can upload new files, fetch existing ones, and manage the content seamlessly. Since IPFS is a peer-to-peer network, nodes might come and go, so handle connection and retrieval errors gracefully.

Testing and Debugging

Thoroughly test your decentralized web application. Check if the files are loading correctly from IPFS, and ensure the app behaves as expected. Use IPFS diagnostic tools to identify and address any performance or connectivity issues.

Deployment and Maintenance

When deploying your application, run your own IPFS node or use a reliable node hosting service to ensure consistent availability of your application's files. Regularly monitor your application and update files as needed. As IPFS is a rapidly evolving technology, staying up-to-date with new releases and best practices is crucial.

Conclusion:

Building a decentralized web application with IPFS opens new possibilities for data ownership, privacy, and censorship resistance. By leveraging the power of IPFS, developers can create applications that are both innovative and aligned with the principles of decentralization. Follow the steps outlined in this guide to embark on your journey of crafting a genuinely decentralized web application. Happy coding!

Getting Started with Tekton Pipelines: A Comprehensive Guide

Ian Gabriel Oliveira de Sousa — Sat, 17 Dec 2022 19:53:36 +0000

Tekton Pipelines is an open-source project that provides a cloud-native, Kubernetes-based pipeline system for continuous integration and delivery (CI/CD). In this post, we will explore what Tekton Pipelines is, why it is useful, and how to get started using it for your CI/CD workflow.

What is Tekton Pipelines?

Tekton Pipelines is a flexible and extensible CI/CD system that is designed to run on top of Kubernetes. It is based on the Kubernetes resource model and uses custom resource definitions (CRDs) to define pipeline tasks and workflows.

One of the key benefits of Tekton Pipelines is that it is designed to be easy to use and integrate with other tools and systems. It has a rich set of pre-built integrations and a plugin system that allows you to extend its functionality with custom components.

Why Use Tekton Pipelines?

There are several reasons why you might want to consider using Tekton Pipelines for your CI/CD workflow:

It is cloud-native and runs natively on Kubernetes, which makes it easy to deploy and scale.
It has a simple, declarative syntax for defining pipelines, which makes it easy to understand and maintain.
It has a rich set of built-in integrations and a plugin system that allows you to extend its functionality with custom components.
It is designed to be flexible and extensible, allowing you to customize and tailor your CI/CD workflow to your specific needs.
Getting Started with Tekton Pipelines

To get started with Tekton Pipelines, you will need to have a Kubernetes cluster set up and have the kubectl command-line tool installed on your local machine.

Install Tekton Pipelines: To install Tekton Pipelines, you can use the tkn command-line tool. You can download the tkn binary from the Tekton Pipelines releases page (https://github.com/tektoncd/pipeline/releases) and add it to your PATH, or you can install it using a package manager like homebrew (on macOS) or scoop (on Windows).

Create a pipeline resource: A pipeline resource defines a set of tasks that are executed in a specific order. To create a pipeline resource, you can use the tkn pipeline create command and specify the name of your pipeline and the tasks it should contain.

Run your pipeline: Once you have created a pipeline resource, you can run it using the tkn pipeline start command. This will trigger the execution of the tasks in your pipeline and provide real-time updates on their progress.

Conclusion:

Tekton Pipelines is a powerful and flexible CI/CD system that is designed to run on top of Kubernetes. By following the steps outlined in this post, you can get started using Tekton Pipelines to automate your CI/CD workflow and streamline your software delivery process.

Deno: how to start use it

Ian Gabriel Oliveira de Sousa — Sun, 16 Oct 2022 21:08:51 +0000

Deno is a new JavaScript/TypeScript runtime with secure defaults and a great developer experience. It's built on V8 and Rust.

How to install

You can install Deno using the following command:

curl -fsSL https://deno.land/x/install/install.sh | sh

How to use

After installing Deno, you can create a file with the .ts extension and write your code.

After saving the file, you can run the code by typing the following command:

deno run file.ts

How to use a module

You can use a module by typing the following command:

deno run --allow-net https://deno.land/std/examples/welcome.ts

How to use a module in your code

You can use a module in your code by typing the following command:


import { serve } from "https://deno.land/std/http/server.ts";

const s = serve({ port: 8000 });

console.log("http://localhost:8000/");

for await (const req of s) {

req.respond({ body: "Hello World" });

}

How to install a module

You can install a module by typing the following command:

deno install \
--allow-net --allow-read \
--allow-write --allow-env \
--allow-run --unstable \
https://deno.land/x/velociraptor/cli.ts

How to use a package manager in your code

You can use a package manager in your code by typing the following command:

import { vpm } from "https://deno.land/x/vpm/vpm.ts";

await vpm.install("https://deno.land/x/velociraptor/cli.ts");