DEV Community: Ayan Pahwa

Headless web exploration is the way to go!

Ayan Pahwa — Mon, 16 Mar 2026 09:21:25 +0000

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa for Extract by Zyte ・ Mar 6

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa — Fri, 06 Mar 2026 14:40:32 +0000

TLDR;

Automate screenshot capture for any URL with JavaScript rendering and anti-ban protection — straight from your AI assistant.

Taking a screenshot of a webpage sounds trivial, until you need to do it at scale. Modern websites throw every obstacle imaginable in your way: JavaScript-rendered content that only appears after a React bundle loads, bot-detection systems that serve blank pages to automated headless browsers, geo-blocked content, and CAPTCHAs that appear the moment traffic patterns look non-human. For a handful of URLs you can get away with Puppeteer or Playwright. For hundreds or thousands? You need infrastructure built for the job.

The Zyte API was designed specifically for this problem. It handles JavaScript rendering, anti-bot fingerprinting, rotating proxies, and headless browser management so you don't have to and what better way to do it straight from your LLM supplying the URLs? Hence I created this zyte-screenshots Claude Skill, which you can use to trigger the entire workflow- API call, base64 decode, PNG save on your filesystem, all just by chatting with Claude.

In this tutorial, we'll walk through exactly how the skill works, how to set it up, and how to use it to capture production-quality screenshots of any URL.

Why Use the Zyte API for Screenshots?

Before diving into the skill itself, it's worth understanding what makes the Zyte API uniquely suited to screenshot capture at scale.

1. Full JavaScript Rendering

Single-page applications built with React, Vue, Angular, or Next.js don't serve their content in the raw HTML response, they render it client-side after the page loads. Tools that capture the raw HTTP response will get a blank shell. Zyte's screenshot endpoint fires a real headless browser, waits for the DOM to fully settle, then captures the final rendered state.

2. Anti-Bot and Anti-Ban Protection

Enterprise-grade sites use fingerprinting libraries to detect automation. They check TLS fingerprints, browser headers, canvas rendering patterns, mouse movement entropy, and dozens of other signals. Zyte's infrastructure is battle-tested to pass these checks so your screenshots won't return a "Access Denied" page.

3. Scale Without Infrastructure

Managing a fleet of headless browser instances, proxy rotation, retries, and residential IP pools is a serious engineering investment. Zyte abstracts all of this into a single API call.

4. One API, Any URL

Whether the target is a static HTML page, a JS-heavy SPA, a behind-login dashboard (with session cookies), or a geo-restricted site, the same API call structure works. The skill you're about to install uses this endpoint.

What Is the zyte-screenshots Claude Skill?

Claude Skills are reusable instruction packages that extend Claude's capabilities with domain-specific workflows. The zyte-screenshots skill teaches Claude how to:

Accept a URL from the user in natural language
Read the ZYTE_API_KEY environment variable
Construct and execute the correct curl command against https://api.zyte.com/v1/extract
Pipe the JSON response through jq and base64 --decode to produce a PNG file
Derive a clean filename from the URL (e.g. https://quotes.toscrape.com becomes quotes.toscrape.png)
Report the exact file path and describe what's visible in the screenshot in one sentence

In practice, this means you can open Claude, say "screenshot https://example.com", and have a pixel-perfect PNG on your filesystem in seconds, no browser, no script, no Puppeteer config.

Prerequisites

Before installing the skill, make sure you have the following:

Tools

curl: Pre-installed on macOS and most Linux distributions. On Windows, use WSL or Git Bash.
jq: A lightweight JSON processor. Install via brew install jq (macOS) or sudo apt install jq (Ubuntu/Debian).
base64: Standard on all Unix-like systems.
Claude desktop app with Skills support enabled.

A Zyte API Key

Sign up at zyte.com and navigate to your API credentials. The free tier includes enough credits to get started with testing. Copy your API key, you'll set it as an environment variable.

💡 Pro tip: Set your ZYTE_API_KEY in your shell profile (~/.zshrc or ~/.bashrc) so it's always available: export ZYTE_API_KEY="your_key_here" or pass it along your prompt

Installing the Skill

Step 1: Download the Skill from GitHub

The skill is open source and available at github.com/apscrapes/claude-zyte-screenshots. Download the latest release ZIP from the repository's Releases page, or clone it directly:

git clone https://github.com/apscrapes/claude-zyte-screenshots.git

Step 2: Import into Claude

Open the Claude desktop app or go to Claude.ai
Navigate to Settings → Skills
Click Import Skill
Select the SKILL.md file from the downloaded ZIP
Claude will confirm the skill has been installed

Step 3: Set Your API Key

In your terminal, export your Zyte API key or pass it along your prompt:

export ZYTE_API_KEY="your_zyte_api_key_here"

To make this permanent across terminal sessions, add the export line to your ~/.zshrc or ~/.bashrc file.

Using the Skill: Step-by-Step

Basic Screenshot

Once the skill is installed and your API key is set, using it is as simple as asking Claude:

Screenshot https://quotes.toscrape.com using zyte-screenshots skill

Claude will:

Verify ZYTE_API_KEY is available in the environment
Derive the output filename: quotes.toscrape.png
Execute the following command:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > quotes.toscrape.png

Verify the file is non-empty
Open and inspect the image
Return the file location and a one-line visual description

Example output from Claude:

✅ Screenshot captured!

📁 Location: /home/you/quotes.toscrape.png

🖼️  What's in it: A clean light-themed page listing inspirational quotes
   with author attributions and tag filters in the sidebar.

Under the Hood: How the API Call Works

Let's break down the exact curl command the skill executes:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://target-site.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > output.png

curl -s — Silent mode; suppresses progress output.

-u "$ZYTE_API_KEY": — HTTP Basic Auth. Zyte uses the API key as the username with an empty password.

-H "Content-Type: application/json" — Tells the API to expect a JSON body.

-d '{...}' — The JSON request body. Setting screenshot: true instructs Zyte to return a base64-encoded PNG of the fully rendered page.

| jq -r '.screenshot' — Extracts the raw base64 string from the JSON response.

| base64 --decode — Decodes the base64 string into binary PNG data.

> output.png — Writes the binary data to a PNG file.

The Zyte API handles everything in between — spinning up a headless Chromium instance, loading the page with real browser fingerprints, waiting for JavaScript execution to complete, and rendering the final DOM to a pixel buffer.

This was a fun weekend project I put together, let me know your thoughts on our Discord and feel free to play around with it. I'd also love to know if you create any useful claude skills or mcp server, so say hi on our discord.

Tags: web scraping • Zyte API • screenshots at scale • JavaScript rendering • anti-bot • Claude AI • Claude Skills • automation • headless browser • site APIs

Raspberry Pi & E-ink scrapes & displays the price of Gold today

Ayan Pahwa — Tue, 24 Feb 2026 08:39:31 +0000

They say that “data is the new oil”, but there’s another hot commodity that’s setting markets alight - precious metals.

In the last 12 months, the value of gold has surged about 75%, while silver has boomed more than 200%. That’s why I, like a growing number of others, now trade in the metal markets.

These days, it is possible to buy digital versions of precious metals. But I think of myself as a collector - I like to buy real, solid coins or bullions whenever I get a chance.

In the last two years, I have acquired a small collection of gold bullions and silver coins, which have appreciated healthily. But I am not planning to sell and book a profit just yet. In fact, I want to buy more, especially when there’s a dip in the price.

There’s just one problem that hits this hobby - prices of actual physical gold and silver bullions are very different in the retail market from stock exchanges’ spot prices and keeping track of them manually is cumbersome specially with a full time job.

To take advantage of the dips and price arbitrage, I need to automate my decisions. To buy gold old-style, I need a key resource from the modern trading toolset - data.

Turn data into gold

The All-India Gem And Jewellery Domestic Council (GJC), a national trade federation for the promotion and growth of trade in gems and jewellery across, is the go-to site listing latest retail rates for gold and silver.

Alas, it doesn’t offer an API to access that data. But fear not - with web scraping skills and Zyte API, I can extract these prices quickly and regularly.

And I can do it using some of the tech I love to tinker with.

I call it ExtractToInk - a custom project that pulls the latest prices on a two-inch, 250x122 e-ink display powered by a retired Raspberry Pi (total cost under US$50).

This is the story of how I power my quest for rapid riches using cheap old hardware and the world’s best web scraping engine - and how you can, too.

Mining for data

Like many modern sites, GJC’s includes both JavaScript rendering for HTML and protection mechanisms- technologies that can break brittle traditional scraping solutions.

This project connects all the dots:

Web → Extract → Parse → Render → Physical display

Tech stack

Hardware

Raspberry Pi (tested on Pi Zero 2 W), it should run on any Raspberry Pi Board
Pimoroni Inky pHAT (Black, SSD1608)

Software

Python 3
Zyte API: to get rendered HTML
BeautifulSoup: to parse HTML
Pillow and Inky Python libraries: for e-ink display stuff

Now let’s get building.

Step 1: Prepare hardware

Setup your Raspberry Pi. In my case, I am using Raspberry Pi OS booted from the SD card.

Depending on which display you use, it most probably will be connected to the Pi over i2c bus or SPI bus protocol - so, enable your display type by entering:

sudo raspi-config

Now attach your e-ink display and do a quick reboot

You might need to install libraries to use your e-ink display.

Step 2: Fetching rendered HTML with Zyte API

The source site, GJC, renders prices dynamically, using JavaScript - something which can make plain HTTP requests unreliable.

No problem. By accessing the page through Zyte API, we can set browserHTML mode to return the page content as though rendered in an actual browser.

Instead of fighting JavaScript, we let Zyte handle it.

html = requests.post(`  
    `"https://api.zyte.com/v1/extract",`  
    `auth=(ZYTE_API_KEY, ""),`  
    `json={"url": URL, "browserHtml": True},`  
`).json()["browserHtml"]

Note: there is no Selenium here, and no headless browsers. This is much more reliable for production-style scraping

Step 3: Parsing with CSS selectors

Once we have clean HTML, parsing becomes straightforward.

Gold prices

Let’s locate the actual prices in the page mark-up.

for row in soup.select(".gold_rate table tr"):`  
    `label = row.select_one("td strong")`  
    `values = row.select("td strong")`

    `if not label or len(values) < 2:`  
        `continue`

    `text = label.get_text(strip=True)`  
    `priceText = values[1].get_text()`

    `if "Standard Rate Buying" in text:`  
        `goldBuying = re.search(r"\d[\d,]*", priceText).group(0)`

    `if "Standard Rate Selling" in text:`  
        `goldSelling = re.search(r"\d[\d,]*", priceText).group(0)

We’re deliberately using:

CSS selectors (easy to find from your browser’s DevTools).
Minimal regular expressions (only for numeric extraction).
Defensive checks to avoid brittle parsing.

Silver prices

Silver appears outside the main table, so we filter it carefully:

for strong in soup.select("p > strong"):`  
    `text = strong.get_text(" ", strip=True)`

    `if "Standard Rate Selling" in text and not strong.find_parent("table"):`  
        `silver = re.search(r"\d[\d,]*", text).group(0)`  
        `break

Step 4: Rendering for e-ink

For this project, I did not want to pipe data into a web dashboard on a computer monitor.

E-ink is always-on, low power, distraction-free and perfect for “ambient information” like this.

So, it’s a great fit for data like prices, weather, status indicators and system health.

But e-ink displays are not normal screens.

They are typically black-and-white, have high contrast and are slow to refresh.

What’s more, no two e-ink displays are made the same way. Every vendor has different support packages so, whichever you end up using, make sure to read the documentation and change the code accordingly.

In my case, I am using Pimoroni inky PHat. The supplied Python library has great built-in examples to get you quickly up and running. I used the helper function to render texts on the display, ex, the build in draw.text() function comes handy:

Draw silver selling price

    draw.text((x, y), f"Silver : {silverPrice}", fill=(0, 0, 0), font=fontBig)

Taking it furtherSection about the finished product

I built this project to use web data thoughtfully, connecting it to the physical world, and building pipelines that feel calm, reliable, and purposeful. When I am at my work-desk the project actively tells me the current prices so I can buy new coins if I see a price drop.

I can further extend this to place automatic orders on the website and secure me a coin at my desired strike price.

If you want to take this further, you could also:

Run it via cron every 10 minutes : The website I am targeting only refreshes prices twice a day, so my cron job runs every 12 hours, but, if you need faster data, you can scrape a site with more real-time updates.
Add more commodities or currencies.
Turn it into a systemd service to run at start time.
Swap e-ink for another output (PDF, LED, dashboard).

If you’re exploring Zyte API, or looking for real-world scraping examples beyond CSVs and JSON files, this project is a great place to start.

You can get my code in the ExtractToInk GitHub repository now.

Holiday Gift Guide 2025: For Developers, Web Scrapers & Everyone In Between

Ayan Pahwa — Fri, 19 Dec 2025 07:55:12 +0000

It’s that time of the year when the coffee gets stronger, commits get messier, and everyone agrees to finally refactor that script in January. And let’s be honest, most of us won’t. But while it lasts let’s celebrate the season of enjoying laid back family time and exchanging gifts.
And to make gifting a little easier, we asked the Zyte team and community to share what they would love to receive. So if you’ve got a developer, a web scraper, or someone who just really enjoys arguing with APIs in your life… Here's your cheat sheet.
Grab a hot drink, settle into your favourite debugging position, and let’s dive in.

Disclaimer : It's not sponsored. All the recommendations are from community and author's personal experience using these products, hence no URL is provided but you should be able find these easily.

1. Book: Soul of a New Machine

For the dev who loves origin stories. It’s a classic engineering tale that reminds us why we fell in love with building things in the first place.

2. Logitech MX Master mouse

Smooth scrolling. Perfect ergonomics. Side buttons that feel like cheat codes for productivity. This is the mouse equivalent of finally finding that one undocumented API endpoint you needed all year.

3. AeroPress + Coffee Subscription

If coffee is the real runtime powering your favourite developer, this combo is basically a performance upgrade. AeroPress means fast, clean brews; fresh beans mean they might actually fix that bug before lunch.

4. Spider Plush Toy

For every web scraper who proudly identifies as “part human, part spider.” it sits silently on your desk reminding you to obey robots.txt (…most of the time).

5. Git Merch Based on Their Commit History

A mug that says “I survived the merge conflict of 2025.”
A T-shirt celebrating their 3,000-day streak.
Or maybe a gentle reminder that “WIP” is not a personality.
Funny, personal, and guaranteed to make them smile during standup.
Link : https://gitmerch.com/ (not-sponsored), just enter their github username

6. Nothing Headphones

Sleek, transparent, and great for tuning out noisy offices or noisy families. Perfect for deep work, debugging, or pretending you can’t hear someone asking, “Can you take a quick look at this?”

7. Mechanical Keyboard (NuPhy / Keychron)

Developers don’t just type on these; they perform.
Clicky keys, gorgeous layouts, RGB that could guide aircraft, what’s not to love? Warning: once they switch, they’ll never stop talking about actuation force.

8. Bambu Lab A1 Mini 3D Printer

For the dev who already has too many hobbies.
They’ll print brackets, cable holders, figurines, replacement parts… and things no one can identify but everyone politely admires. A creator’s playground in miniature.

9. BenQ Monitor Light Bar

Immediate upgrade to any desk setup. Reduces eye strain, looks clean, and helps developers see their code clearly even during late-night “just one more function” sessions without taking extra space on their desk.

10. 100W GaN Charger

Tiny but absurdly powerful, just like that one script they wrote at 3 AM that still runs in production. A GaN charger keeps everything powered: laptop, phone, tablet, e-reader, existential dread, everything.

If you love someone who spends their days coaxing data out of websites, obsessing over keyboard switches, or whispering sweet nothings to their terminal, this list has something that will make them light up.
Here’s to a warm, restful holiday season… and to fewer bugs in 2026. Happy gifting, happy coding, and as always, happy scraping! 🕷️✨

Build Your Own Holiday Deal Tracker with Python, Zyte API & IFTTT 🎁

Ayan Pahwa — Fri, 21 Nov 2025 11:06:18 +0000

(A simple, developer-friendly project to help you catch Black Friday, Cyber Monday, and Christmas deals before anyone else)

Watch the video tutorial :

It’s that time of the year again, everyone’s hunting for discounts, limited-time deals, and that one item you’ve been keeping an eye on all year. You know the drill: tabs open everywhere, price tracker sites breaking under traffic, browser extensions that promise magic but fail right when The Deal drops.

So this year, I decided to build something different, something that actually works when I need it the most.

A personal price-alert system, powered by:

Python 🐍
Zyte’s Automatic Extraction API 🕷️
An IFTTT mobile notification trigger 📱

And honestly? It ended up being one of the simplest, useful and most reliable holiday project I’ve made in a long time.

❌ No HTML parsing.
❌ No complex CSS selectors.
❌ No brittle scrapers that break during peak traffic.
✅Just a clean API call, structured product data, and a push notification when the price hits your set target.

Let me walk you through the whole thing :

🎁 Why Build Your Own Deal Tracker?

Because holiday deals don’t wait for anyone. Every year I get messages from friends asking,

“Bro, what’s the best price tracker? Nothing seems to work today.”

And they’re right. Most free services throttle, break, or go down completely during Black Friday and Cyber Monday because everyone hits them at once.

Meanwhile, with a few lines of Python and Zyte’s AI-powered extraction, you can build something that:

Works only for you
Deals with anti-ban, anti-bot and captchas
Checks as often as you want (make it run on your pc, github-actions, server or a raspberry pi)
Doesn’t rely on brittle selectors
Doesn’t choke on JavaScript
Doesn’t get rate-limited
Sends you a mobile ping instantly when price drops

It’s the perfect mix of practical dev fun + a tool you’ll actually use.

🎄 Why Zyte?

Scraping e-commerce sites for price data is usually a pain: blocking, JavaScript rendering, cookie rules, bot detection, dynamic DOMs, infinite variations in product page layouts... you get it.

The magic here is Zyte’s Automatic Extraction.

You literally tell AI powered Zyte API:

{ 
"url": "<product-url>", 
"product": true 
}

and it returns structured fields like:

price
currency
product name
sku
images
stock availability
description

You don’t write selectors. You don’t parse HTML. You don’t chase CSS changes. You just get clean data. And that makes this holiday project absurdly simple.

🎅 What We’re Building

A script that:

Takes:
a product URL
a target price
your Zyte API key
your IFTTT Webhook key
Fetches structured product data using Zyte API
Checks if the product price is ≤ your target
If yes → triggers an IFTTT event

Your phone instantly notifies you with product URL:
“Your product dropped to your target price. Go get it!”

You can run this:

manually (when you want)
on a cron job (in the background)
in GitHub Actions
on a Raspberry Pi
on a cloud function Your call.

🛠 Setup

Clone your project

git clone https://github.com/apscrapes/zyte-sale-alert.git
cd zyte-sale-alert

Create your virtual environment

python3 -m venv venv
source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Create and add secrets in .env file

ZYTE_API_KEY=your_zyte_key_here (obtained from zyte)
IFTTT_KEY=your_webhooks_key (obtained from IFTTT, see next step)
IFTTT_EVENT=price_drop (name of your ifttt applet, see next step)
TARGET_PRICE=149.99 (example)
PRODUCT_URL=https://example.com/product/123

Create an IFTTT Webhook Applet

a. Download IFTTT mobile app, it’s paid but you get a 7-day trial and it has tons of automation you can build using no-code, so i think it’s worth it.

b. Click create new automation / applet

c. In “IF” field, select “Webhooks” > Add Event Name

d. In “THEN” field select Notification > App Notification > JSON Payload > High Priority

Note : You can instead of notification can also set other automation like getting an email for example

e. Here's how it should look like when it's successfully created:

Once the automation is created add your IFTTT_KEY and IFTTT_EVENT to your .env file

Set product parameters The main script is at src/pricedrop.py, edit the following variables to add your:

PRODUCT_URL = “ ”
DESIRED_PRICE = 250

PRODUCT_URL “ ” is the URL of the product you want to track and DESIRED_PRICE is the price at which you want to be notified, that is it.

Run the project

python src/pricedrop.py

You can run it manually or set up a cronjob to run at regular intervals. Whenever the price drops equal or below your target price you’ll get a notification from the IFTTT app on your phone with product URL so you can order it right away.

Let’s understand how it works.

🧩 Core Logic (Short Version)

resp = client.get({"url": url, "product": True})

“product : True” tells zyte API that the webpage we’re scraping has a product so the Machine Learning powered scrapper gets you all the relevant parameters like price, quantity, description, currency etc.

And that’s literally all you need.

The reason this works so beautifully is Zyte is handling:
JS rendering
blocking
retries
browser simulation
extraction logic
AI-powered field detection

In-detail :

Importing all the necessary libraries

import os
import sys
import requests
from zyte_api import ZyteAPI
from dotenv import load_dotenv

Setting required variables

PRODUCT_URL = "https://outdoor.hylnd7.com/product/a1b2c3d4-e5f6-4a7b-8c9d-000000000293"
DESIRED_PRICE = 250

Here we've setup a sample product link whose price we want to track and a price at which if it goes below we want to be notified.

Loading the API keys from .env file

load_dotenv()

# from Zyte API
ZYTE_API_KEY = os.getenv("ZYTE_API_KEY")

# from IFTTT Service applets
EVENT_NAME= os.getenv("EVENT_NAME")
IFTTT_KEY= os.getenv("IFTTT_KEY")

if not ZYTE_API_KEY:
    print("ERROR: ZYTE_API_KEY not found in environment.")
    sys.exit(1)

In the project root directory, create a .env file and add ZYTE API Key you'll get after logging into zyte.com and IFTTT webhook API key you get after creating the automation applet

Function to trigger the mobile notification :

def trigger_ifttt(event_name, key, value1):
    url = f"https://maker.ifttt.com/trigger/{event_name}/json/with/key/{key}"

    payload = {
        "value1": value1,
    }

What this funciton is doing is basically making an API call to IFTTT and IFTTT applet is set so whenever the API calls comes with payload it sends mobile notification with that payload, which in this case is product URL so you can directly click and open the product page and buy it before it goes out of stock, SMART right? 😉

Scraping init

client = ZyteAPI(api_key=ZYTE_API_KEY)

    payload = {
        "url": PRODUCT_URL,
        "product": True,          
    }

    resp = client.get(payload)

Making a GET request on Zyte API with product : True, we're asking zyte to treat the URL as product page and thus it's uses it's ML capabilities to fetch product relevant details, price in this case.

Compare price to SETPOINT

if price_float <= DESIRED_PRICE:
            trigger_ifttt(EVENT_NAME, IFTTT_KEY, value1 = PRODUCT_URL)

If price of the product reaches to or below our target price it will call the IFTTT function, thus triggering the notification.

🌟 Make It Even Better

You can extend this to:

Track multiple URLs
Log daily prices to CSV
Plot graphs
Send WhatsApp alerts
Push to a Telegram bot
Use GitHub Actions to check every hour
Deploy as a Streamlit dashboard

Zyte handles the extraction. You build the magic on top.

🧘 Final Thoughts

I love projects like this because they hit the sweet spot between:

seasonal usefulness
real-world scraping challenges
a clean developer experience
a fun weekend build

If you're new to the world of web scraping like me, this shows how powerful the right tools can be. If you're experienced, it’s refreshing to skip the boilerplate and let Zyte handle the messy parts.

And honestly, there’s something fun about getting a custom alert on your phone saying:
“Hey, that gadget you wanted all year just dropped to your target price.”

Happy building, happy holidays, and happy deal-hunting! 🎄🎁
Let me know what you end up tracking.

Join Zyte Discord to share what you're building or get any support :
https://discord.com/invite/DwTnbrm83s

@iayanpahwa

Build a crypto miner using Raspberry Pi in 10 minutes

Ayan Pahwa — Wed, 15 Jun 2022 13:44:17 +0000

Lately I’ve been hearing a lot of hype around web3- cryptocurrencies, NFTs, DAOs, DeFi, GameFi and all of these cool jargons giving me a lot of FOMO (Fear of missing out), so I decided that I’m gonna try some of them for myself and see what’s it all about.

I tried a few things like buying some cryptos on an exchange, made a couple of web3 projects and joined a gazillion of discord channels doing the morning ritual of gm(good morning!) and responding to WAGMI(we all gonna make it!) but this one specific project called Monero caught my attention and I wanted to share it with the community here.

Before proceeding I want to put out that this is in no way an endorsement of the project or whole blockchain/web3 ecosystem in general, nor it’s a financial, get rich by mining cryptocurrency advice. This is in fact a fun project which you can build along to learn more about web3 and being a developer advocate, I’m also gonna use this project to share about the concept of :

Using a separate docker images for build-time and run-time within the same Dockerfile, we will see why it’s important and how to achieve that later in this post

Nevertheless, this project is kind of a fun to build, it allows you to mine cryptocurrency called XMR of Monero blockchain, profitable or not you can no doubt brag about mining cryptocurrency at your home to your friends, so let’s get started 😉

You can skip the inner details and jump straight to the build part down in the post if you like.

Background

Without going too much into the detail, for someone new into the web3 ecosystem - A blockchain is a distributed ledger which keeps records of every transaction happening over it. Just like how a bank keeps the record of who sent money to whom, the amount as well as the date and time, similarly in case of blockchain this immutable information is maintained within distributed blocks connected by a network.

This validation of all the transactions on a blockchain is done by certain users who lend their compute power in the form of miners or validator nodes. There are also some programs called smart contracts which can run automatically on blockchain when certain conditions are met but that is out of scope for this blog so will not go into more details of smart contracts.

When you think of Bitcoin or Ethereum miners you might imagine a big server room consisting of massive GPUs or ASICs machines dedicatedly solving complex cryptographic problems sent to them by blockchain and once they solve it they earn rewards in the form of cryptocurrencies The process of successful submission to earn the rewards follows certain consensus mechanism which can vary depending on the type of blockchain

For example: Bitcoin blockchain works on a consensus mechanism known as Proof of Work (PoW), while Solana works on Proof of Stake (PoS) and Proof of History(PoH).
You can read all about types of consensus mechanism here

If you’re feeling disheartened seeing the cost of these mining computers crushing your dream to have a mining rig of yourself, don’t be because the project we’re talking today called Monero allows or in fact encourages mining on CPUs, so even a small single board computer like Raspberry Pi 4 can become a miner to help validate the transactions and get the rewards in form of XMRs, the cryptocurrency of monero blockchain.

The philosophy behind this is since the cost and entry barrier to mine is pretty high, the miners are usually owned by few persons or organizations which may not be good for blockchain’s decentralization (no ownership and trustlesness by design), so monero optimizes to allow more and more people to contribute making their blockchain more decentralized. Monero also works on Proof of Work but instead of your small device solving a really complex cryptography puzzle it can join a mining pool with other devices to lend the compute power and together they can solve it fast and depending on how much your device contributed to the solution it’ll be rewarded suitably for that.

Is it profitable? Maybe not, given the device needs to be powered on and running but it surely is fun and maybe you can really earn if you add a lot of devices in your mining pool with good resources, say Rpi with 8 GB RAM; nevertheless it’s not financial advice so please do your due diligence :)

You can read all about Monero project, how much it pays for mining, costs associated, etc all on official website: here

Build

Hardware Needed

A single board computer like raspberry pi 4 (more ram the better).

The following SBCs were tested by my friend Lambros and this was the hashrate result:

- Raspberry Pi 3  - 20 H/s
- Raspberry Pi 4, 1 GB RAM - 45 H/s
- Raspberry Pi 4, 4 GB RAM - 99 H/s
- Nvidia Jetson Nano 2GB (without GPU enabled) - 62 H/s

Hash rate(H/s) is the number of hashes a device can solve per second. I’ve not enabled CUDA(GPU backend) for Nvidia Jetson since monero encourages mining on CPU though if you want to give it a shot I’d love to see how it performs.

Other things you'll need are

SD card
Power supply
Connectivity to Internet via LAN or WiFi

Software needed

monero wallet (this is where you’ll get the rewards) download from https://www.getmonero.org/downloads/
Information of mining pools : as mentioned earlier we will be joining a mining pool, the address of active mining pools can be found by a simple google query, it’s better to use one near you ex: google search monero mining pools in Los Angeles. Each mining pool has their own threshold after which they start paying out to the wallets.
balenaCloud account to manage miners
balenaEtcher to flash the SD card

Deployment

The easy way (deploy with balena)

Sign up for a free balenaCloud account. Your first ten devices are free and fully-featured! Then use the button below to create and deploy the application:

Note: I have used a Raspberry Pi 4 in the image below but be sure to select the correct device type for the device you are using.

Select the Device
Add OS type: Production vs Development
(optionally) add in your home WiFi credentials, download and flash the OS to SD card using etcher

The Advanced way

If you are already a balena user it might be better for you to use this way. You can clone the project from this github repo and use the balena CLI command

balena push <fleet_name>

to push the application to your devices in the fleet created on dashboard. This is the best option if you want to tinker with the project and have full control. The Getting Started Guide covers this option. After you've created the application and pushed the code using the CLI, follow the steps below.

First device boot and configurations

When the device boots for the first time, it connects to the balenaCloud dashboard, after which you’ll be able to see it listed online. In the meanwhile we need to Install Monero wallet on our computer where we will be getting the mining rewards

Software Setup

Download the Monero wallet from https://www.getmonero.org/downloads/ GUI or CLI as per your preference. I recommend using a GUI wallet and create a new wallet using Simple Mode.

Once the wallet is created you’ll have a unique wallet address copy that to clipboard and head over to:

balena cloud > your device > Device Variables

tab and add the following

Device Variables

VARIABLE NAME	VALUE	CHANGE TYPE	DESCRIPTION
WALLET_ADDRESS	from last step	Must add	This is the wallet address where you’ll earn your mining rewards in the form of XMRs which you can trade on any crypto exchange
MINER_POOL	Default value is: http://xmr.2miners.com:2222/	Optional	This is the miner pool you will join by default. You can change this to another miner pool by searching addresses of miner pools on Google. The one near to your location will be good. For ex:

This will restart the container and your miner will be registered to the mining pool and start getting the jobs. All the rewards will be sent to your monero wallet after your device meets the threshold of the miner pool.

Now as I mentioned above this may not be profitable at all but it's definitely fun and let’s look at some totally unrelated things that we're gonna learn here.

Container Learnings

Here is the project’s Dockerfile.template. If you take a closer look at it, you’ll see there are two different base images being used. One of them is a build image and other one is the run image

Why we do that is simply because we don't want our run image to be bloated with extra packages which were needed to build a source code. So we conduct the build in a separate image and copy the artifacts or binaries in the run image using

COPY --from=build /usr/src/app/xmrig/build/xmrig /usr/local/bin

This way our main image is minimal and less bloated. All of the balena base images are available as build and run. The build image has additional packages such as gcc, build-essential which are needed to build from source whereas the run image is minimal. You can read more about it here.

So this was the container lesson from this blog. Now I hope you do use this project to dip into the world where IoT edge computing meets web3 and as always if you have any suggestions, feedbacks or questions, please write on balenaForums or any social media channel. I'll be back with another interesting project and lesson soon 🍻

Attribution

XMRig project

===