DEV Community: Eric Murphy

I Wrote a Script to Download Every NFT

Eric Murphy — Thu, 16 Dec 2021 10:54:25 +0000

By now, you might have learned how to right click and save or screenshot NFTs. But compared to what I'll show you in this post, that'll just look like petty theft. I'm going to show you how to create a script to download tens of thousands of NFTs in just minutes.

Memes aside, I thought this was a fun project to get more familiar with Node.js and Puppeteer, a library you can use for web scraping and more. And if that sounds interesting, read on. If you want to follow along better, here's the GitHub with the code.

(If you'd like to watch this post in video format, click here.)

Casing the Joint

We're going to be lifting the NFTs from Rarible, one of the most popular NFT marketplaces.

Here you can buy JPEG images of monkies, anime girls, cats, and more. But what we're after are the monkies. Some of the most sought-after NFTs. But I don't want to just save one or two of them--I want ALL of them. To be more precise, all 9,800 of them in this collection.

Installing Dependencies

I'm going to be writing this script in Node.js because I never learned a real programming language. And we're going to be using a library called Puppeteer in order to lift the NFTs. What Puppeteer does is basically starts up a remote-controlled version of Chromium that we can program to do our bidding. Let's initialize the project and install Puppeteer.

npm init -y
npm install puppeteer

Writing the Script

Let's create our script. I've created a file called index.js and added the following:

const puppeteer = require("puppeteer");
const fs = require("fs");
const path = require("path");

const url = "https://rarible.com/boredapeyachtclub";

(async () => {
  console.log("Loading...");
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto(url);
  await page.setViewport({
    width: 1200,
    height: 800,
  });
});

Let's explain what's going on here. First, we're importing Puppeteer and a couple of Node.js libraries that will allow us to save files to our local machine.

Next, we're setting the URL of the page that we're going to be lifting the images from. This can be any Rarible collection page.

Finally, we're starting the browser with Puppeteer, navigating to the URL, and setting the viewport dimensions. The await keywords will ensure that the previous command will finish before the next one runs. This is all from the Puppeteer documentation, so it's not rocket science.

If this is all correct, then when we run the script with node index.js, it should open up a Chromium window and navigate to the URL.

Looks good so far. But there's more that needs to be done.

Getting the Name of the Collection

We want to grab the name of the collection that we're downloading and create a folder to deposit all of our loot into.

We can get the name from the page title. However, it doesn't load in the name of the collection until the entire page has been loaded. So, we have to wait until the React app has been completely loaded.

I opened up the devtools and found a class name that we can hook onto. There's a div with the class of ReactVirtualized__Grid that is the grid that holds all the images. Once that's loaded, the name of the collection has already been loaded in the page title. All we need to do is wait for this element to load, then we can proceed.

await page.waitForSelector(".ReactVirtualized__Grid");

const pageTitle = await page.title();
const collection = await pageTitle.split("-").shift().trim();
if (!fs.existsSync(collection)) {
  fs.mkdirSync(collection);
}

We're using the Puppeteer method waitForSelector to hold off until this element is loaded. After that, we grab the page title, do a bit of JavaScript string manipulation to get the right value, and finally create the directory with Node (if it hasn't already been created).

We now have a folder to put all the images in! Let's fill it up.

Downloading the Images

This is the meat of the script. What we want to do is get all the resources that are downloaded to the page. That includes HTML, CSS, JS, and images. We only want the images, but only the NFTs, not any logos, avatars, or other images.

If you look in the network tab of the dev tools, we can see all the images being loaded. We can also see that all of the NFTs are loaded in with a path containing t_preview. None of the other images on this page are from the same path. So, if we sort out these images with these URLs, we can single out the NFTs from all the noise.

let currentImage = 1;

page.on("response", async (response) => {
  const imageUrl = response.url();
  if (response.request().resourceType() === "image") {
    response.buffer().then((file) => {
      if (imageUrl.includes("t_preview")) {
        const fileName = imageUrl.split("/").pop() + ".avif";
        const filePath = path.resolve(__dirname, collection, fileName);
        const writeStream = fs.createWriteStream(filePath);
        writeStream.write(file);
        console.log(`${collection} #${currentImage} saved to ${collection}/${fileName}`);
        currentImage++;
      }
    });
  }
});

There's a lot going on here, but we're getting all the resources, selecting the images, then only getting the ones we want. After that, we use some string manipulation to get the filename and save them as AVIF files (a next-gen image format you can learn more about here). Then we're saving these to the new folder we created with some Node.js methods. Finally, we're just logging to console the image that's just been downloaded and how many images have already been downloaded.

Phew! That works, and we're finally downloading some images. But the images are being lazy-loaded. That means they aren't being loaded until you actually scroll down the page. That's great from a user perspective, as the users are only loading in images they can actually see, but not so much from ours. Let's make an function to scroll down the page for us, and click the "Load more" button that keeps you from seeing all the images.

The Autoscroller

In order to start scrolling, we just want to run some JavaScript on the page to scroll it. We can do that with Puppeteer's evaluate method. This will run some JavaScript on the page, same as if you had written it in the dev tools console.

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let totalHeight = 0;
      let distance = 500;
      let timer = setInterval(() => {
        let scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 1000);
    });
  });
}

What this does is jumps down on the page 500 pixels, and sets a timer to do it again every second. We're scrolling by 500px/second. A little slow, but if I make it faster it could scroll too fast and skip saving some images. Not good. Then, with totalHeight, we're saving how much distance we've scrolled already and comparing it to the total height of the page, scrollHeight. Once we're at the bottom, we're going to stop the setInterval and resolve the promise. No more scrolling.

However, once we're at the bottom, we still need to click the "Load more" button.

We need to tell Puppeteer to run some JavaScript on the page to find all the buttons and narrow it down to the button with the text "Load more". There's no unique ID or class on this button, so we have to find it like this. Finally, we click on the button with help from Puppeteer. Finally finally, we resume the autoscroll function now that there are no more buttons to click.

await autoScroll(page);

await page.evaluate(() => {
  const elements = [...document.querySelectorAll("button")];
  const targetElement = elements.find((e) => e.innerText.includes("Load more"));
  targetElement && targetElement.click();
});

await autoScroll(page);

After all that, we can close the browser once we've gotten to the bottom of this page containing 10,000 NFTs.

await browser.close()

Running the Script

That's it! We can now run the script and see if it works.

node index.js

Actually, this will take a while because we have to scroll down and save 10,000 images. Grab a cup of coffee or something and stretch your legs while you're waiting.

...

All right, we're back. Let's take a look at what we have here...

What a haul! We now have millions of dollars worth of NFTs on our computer. Not bad for a day's work. What am I going to do with all of these monkey pictures??

I've put the code on GitHub if you want to pull an NFT heist like me--or just play around with Puppeteer some more. It's pretty fun.

That's all for now. I think I'm just going to be sitting over here counting my fat stacks of NFTs.

Do You Need to Be "Passionate" About Coding?

Eric Murphy — Wed, 01 Dec 2021 11:30:36 +0000

(If you'd prefer to watch this post in video format, click here.)

Every once in a while, if you're reading some online discussion, you'll see the topic of passion come up in coding. Time and again, you'll probably see the same sort of questions.

Do you need to be passionate about coding?
I'm working a job and I'm just not that passionate about coding anymore. What should I do?
I'm trying to get into coding and I don't have that much passion for it, but I want to get into it because I've heard that it makes a lot of money. Should I continue learning?

In this post, I want to go over passion as it relates to coding and answer all of these questions.

Passion for Your Job vs. Passion for Coding

Let's start off by not confusing passion for your job and passion for your coding. A lot of people have had passion initially learning to code, working on their own projects. They've had a lot of fun doing that. But once they actually get a real job or start taking on some work, then a lot of that passion can go away, because as much as you might like coding, working a job is not always the most fun.

You're not always going to have new and interesting problems to solve. Some of it is just going to be grunt work. Maybe you're feeling stressed out by deadlines or changing requirements. And so you might not be feeling quite as passionate as you did when you first started the job.

And I've definitely felt this before working on lots of different projects. Of course, sometimes I will really enjoy working out how to do something, and maybe even learning something new. And that little dopamine rush that you get whenever you solve a problem and it finally all comes together is just really satisfying. There's nothing like it. So when I'm working a job where I have something like that, it's really enjoyable.

But to be honest, that's not most of it. So sometimes you're just maybe creating the same UI that you've built a million times already. Maybe I'm fixing some small CSS bug or just dealing with some random error from NPM. And a lot of times it can just be frustrating or boring.

So you're not feeling so passionate about your job. And to be honest, that's just part of having a job. That's just part of working. It's not always going to be fun and games the entire time. I know some boomer has probably told you in your life that if you love what you do, you'll never work a day in your life. And to be honest, it's not really true. So even if you love coding, there are going to be days where you're just not having that much fun.

And my philosophy has always been you don't have to necessarily love your work. You just have to not hate it. So if you're just OK with it on a day to day basis, then that's perfectly fine. That's probably my attitude for most of the work I do in order to pay the bills. It's just something that I don't mind doing. It's not particularly hard for me or something that I completely hate doing. It's just something that's okay for me. If you think passion is just loving your work all the time, that's not what it is at all.

Do You Need Passion to Learn Coding?

And that brings me back to the question. Maybe somebody is just getting into the field and they don't necessarily have a whole lot of passion for coding. Maybe they just heard about how much money you can make, and it seems kind of like a fit for their personality. So they're just going to try it. And they're wondering, do you need to be passionate in order to learn programming?

And in this case, I actually think you do. And so by passion, I'm talking about maybe you want to build things. Maybe you have ideas of things that you want to make, and programming will bring them to life for you. And that's what really motivates me. So I started learning to program since I was just a kid. I would make websites for my favorite video game series, and I would make basic games inside Game Maker. And that's how I started to learn programming, just thinking of interesting things that I could build and then building them. And I would love making these websites and building these things. It was never a question about passion for me.

But a lot of people who are just getting into programming or web development are just getting into it because of the hype. Basically, if you don't have any inherent passion for it, I honestly don't think you're going to last for very long. So some people manage to do it. But these are the kinds of people that you'll read about on Reddit maybe ten years later and they just hate their career. They're not happy with it at all. They're completely burned out. They're just very pessimistic kind of people, not really anybody that you'd want to be. And so I can't possibly recommend this if you don't have at least a little bit of passion for it.

If you don't want to build anything, if you've never been interested in programming before, you heard about how much money you can make, then it's probably not the field for you. Sorry, that's just the way it is. Programming can be pretty mentally taxing. Sometimes you're problem-solving difficult issues all day. And if you're not enjoying it at all, then it's probably just going to be miserable for you.

For me personally—well, maybe I don't have that much passion for my work or my job that I do on a daily basis. I do have a lot of passion working on some personal projects on some websites or some side projects that I want to make. At least just thinking of ideas is fun for me, even if I don't actually build them. If I don't have that much free time at the moment, it's still in the back of my mind—ideas for things I can do with programming.

If I want to learn some new technology or learn how to make something new that I've never made before, that's pretty interesting to me, and I'm happy to learn it. So I think that kind of passion is important. If you're trying to dip your toes in and see if you're going to like it, just try it out, build a few things yourself. And if you don't like it at all, then you might want to rethink your career choices.

What to Do if You Don't Feel Any Passion

So while we're talking about passion, let's not get conflated job passion and personal passion for coding. And this is where a lot of problems come in. So if you feel completely burnt out and you just don't have a passion for coding anymore, then it could just be that you're in a toxic job.

You might be in some kind of toxic situation where maybe your boss is just giving you way too much work. You're feeling way too stressed. Maybe they want you to work overtime and learn things on your own free time that you have to learn for work. And while it's okay to be learning things just for fun (that's part of the reason I love programming), it's not really okay to be basically putting in unpaid time for your job, learning something should be on company time.

And maybe you've seen some job posting that the company you're interviewing for and they really want people who are "passionate about coding". And maybe in that case, they're actually just talking about being passionate for the company, and they want you to do all of this nonsense for the company: long hours, weekends, and to put up with a thankless job. They're hoping if you're passionate enough about coding, then you'll put up with all of the nonsense that they're going to put you through. And in that case, you really don't want to be passionate for this job. If you're in that situation, maybe it's just time to start looking for a new job so you can actually relive the passion of coding for you again.

I don't want this to be a complete video about burnout, but sometimes it is just that. Maybe your job isn't even that bad. It's just you're feeling stressed out, frustrated for whatever reason you just need to go outside and log off and get a little bit of time away from the computer.

Closing Thoughts

That's what I think about how much passion you need. Maybe day-to-day working on some job, I don't have that much passion for it, but I have the passion to work on side projects, start a YouTube channel about programming, teach other people what I'm learning about, and I find it really rewarding. If that sounds like you, then I think you're on the right path.