DEV Community: Abdullah Sheikh

How to Scrape Any Website Without Getting Blocked – Proven Techniques for 2025

Abdullah Sheikh — Tue, 09 Jun 2026 12:04:05 +0000

Learn the step‑by‑step methods to scrape data reliably while staying under the radar of anti‑bot defenses

Before We Start: What You'll Walk Away With

By the end of this guide you’ll know exactly how the anti‑bot walls work, which tools cut through them, and how to assemble a scraper that stays online even when the site tightens its defenses.

First, we stay inside legal and ethical borders – you’ll learn when a site’s terms allow crawling, how to respect robots.txt, and why mimicking a real user’s behavior matters more than any fancy code snippet.

Next, you’ll walk away with three concrete results:

Defense insight: a clear picture of rate limits, CAPTCHAs, fingerprinting, and IP bans, like knowing the security checkpoints at an airport before you board.
Ready‑to‑run script: a Python (or JavaScript) starter that incorporates rotating proxies, realistic headers, and adaptive delays, so you can launch it on day one.
Maintenance checklist: a short, repeatable routine that alerts you when a site changes its guard, similar to a weekly car‑inspection list.

This recipe is built for people who can write a for loop but get stuck at the 403 wall – data analysts pulling market trends, growth hackers testing ad creatives, and junior developers automating dashboards.

Understand the “traffic light” system sites use to flag bots.
Pick the right proxy pool, user‑agent rotator, and headless browser for your budget.
Set up automated health checks so your scraper self‑heals before you notice a drop.

Think of it like packing a suitcase: you choose the right clothes (tools), know the airline’s rules (legal limits), and have a checklist so you never forget a sock (maintenance). Follow the steps and you’ll scrape websites without getting blocked.

What Web Scraping Actually Is (No Jargon)

Think of web scraping as hiring a robot assistant to copy the exact bits you need from a web page, just like you’d flip through a printed catalog and jot down product names, prices, and descriptions into a spreadsheet.

The robot reads the page’s HTML—the language browsers use to show the site—and pulls out the pieces that matter. It then formats those pieces into rows and columns you can feed straight into a database or a CSV file.

That’s all there is to it: a program that extracts structured data from HTML pages without you having to type anything manually.

Price monitoring: track competitor pricing daily to stay competitive.
Market research: gather product specs from dozens of sites for a comparison report.
Lead generation: collect contact info from business directories for outreach.
Content aggregation: pull headlines from news sites to build a custom feed.
Academic data collection: scrape public datasets for research projects.

When you scrape a website without getting blocked, you’re simply being smarter about how often you ask for data, where those requests appear to come from, and how you disguise the robot’s fingerprint. The goal isn’t to break anything—just to collect the same info a human could see, but at scale.

Now that you know what scraping really is, let’s see how browsers keep their doors shut.

The 4 Mistakes Everyone Makes With Scraping

Most people hit a wall because they treat scraping like ordering a pizza from the same address every night – the kitchen eventually stops answering.

Using a single static IP – Think of it as showing up at a club with the same fake ID every night; the bouncer will recognize and bar you. When you send every request from one address, the site’s firewall flags and blocks you instantly. Rotate proxies or use residential IP pools to stay under the radar.
Ignoring request headers and user‑agent rotation – It’s like walking into a store dressed in the same uniform every time; the staff knows you’re not a regular shopper. Browsers send dozens of headers (accept‑language, referrer, etc.). Randomize the User-Agent and mimic a real browser’s header set, or sites will serve you a CAPTCHA or a 403.
Flooding the site with rapid requests – Imagine texting a friend every second; they’ll mute you. Sending dozens of requests per second trips rate‑limit alarms. Insert realistic delays, respect “think time,” and batch your calls to mimic human browsing patterns.
Forgetting to respect robots.txt and legal limits – This is like ignoring a “No Entry” sign on a road; you’ll get a ticket. Robots.txt tells crawlers which paths are off‑limits. Skipping it can result in takedown notices or even legal action. Always check the file and stay within the allowed crawl‑delay.
Cheat sheet: Use requests with headers, rotate User-Agent, add time.sleep(random.uniform(1,3)), and proxy through scraperapi.com or similar.

Fix these four mistakes and you’ll stop hitting 403s while you scrape website without getting blocked.

How to Scrape Any Website: Step‑by‑Step

Grab your notebook and follow this checklist; think of it as packing a suitcase for a trip where every item has a purpose.

Set up a rotating proxy pool. Choose residential or datacenter proxies and feed them into a manager like proxy‑pool. It’s like ordering multiple delivery addresses so the restaurant never knows you’re the same customer.
Randomize headers, cookies, and user‑agents per request. Use a list and pick one at random for each call. This mimics a crowd of shoppers each wearing different outfits, making you blend in.
Implement adaptive throttling based on response codes. If you get 429 or 503, back off; if responses are clean, speed up slightly. It works like Google Maps rerouting when traffic slows down.
Use headless browsers for JavaScript‑heavy sites. Spin up Playwright or Puppeteer in headless mode, let the page render, then extract the HTML. Think of it as hiring a driver who knows every back‑street shortcut.
Parse and store results. Feed the markup into BeautifulSoup (Python) or Cheerio (Node) and write rows to a CSV or insert into a database. It’s like sorting groceries into the right bins before you unload the car.
Log failures and auto‑retry with exponential backoff. Record status, URL, and error in a log file, then schedule a retry that doubles the wait each try. This mirrors a friend who keeps calling you back, but waits longer each time you don’t answer.

Cheat sheet:

Proxy manager: proxy‑pool
Header rotator: random‑user‑agent
Throttle logic: check response.status_code
Headless launch: playwright launch --headless
Parse: BeautifulSoup(html, "html.parser")
Retry: time.sleep(2**attempt)

Follow these steps and you’ll scrape any website without getting blocked.

A Real Example: Scraping Competitor Prices for an E‑Commerce Analyst

Lena works as a junior analyst at an online store. Every morning she needs a fresh CSV of the top‑selling competitor’s prices, but the site hides behind Cloudflare and throws a CAPTCHA after a few requests.

Her constraints are simple: run the scraper on a modest AWS t3.micro, keep the script under 10 minutes, and avoid any IP ban that would halt the daily feed.

TaskCode Snippet
Proxy rotation

import random, json, requests

PROXIES = [
    "http://user:pass@proxy1:3128",
    "http://user:pass@proxy2:3128",
    "http://user:pass@proxy3:3128"
]

def get_proxy():
    return {"http": random.choice(PROXIES), "https": random.choice(PROXIES)}

Playwright launch

from playwright.sync_api import sync_playwright

def launch_browser():
    proxy = get_proxy()
    playwright = sync_playwright().start()
    browser = playwright.chromium.launch(
        headless=True,
        proxy=proxy,
        args=["--disable-blink-features=AutomationControlled"]
    )
    return browser, playwright

Data extraction

def fetch_prices(url):
    browser, pw = launch_browser()
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    rows = page.query_selector_all(".product-row")
    data = []
    for r in rows:
        pid = r.get_attribute("data-id")
        price = r.query_selector(".price").inner_text()
        data.append({"id": pid, "price": price, "ts": page.evaluate("()=>Date.now()")})
    browser.close()
    pw.stop()
    return data

Run fetch_prices("https://competitor.com/category") inside a daily cron job.
Append the returned list to prices_2025.csv with columns product_id, price, timestamp.
Because each run picks a random proxy and masks the automation hint, Lena can scrape website without getting blocked consistently.

The Tools That Make This Easier

If you want to keep the scraper humming while the target site tightens its guard, pick tools that already handle the heavy lifting.

ScraperAPI – Think of it as a delivery service that swaps your IP address every time you place an order. It supplies rotating residential proxies and solves CAPTCHAs on the fly. The free tier gives you 1,000 requests a month, enough for testing or small projects.
Playwright Python – This is the Swiss‑army knife of headless browsers. You write ordinary Python, but behind the scenes Playwright mimics a real user, complete with mouse movements and timing. Adding a stealth plugin is like slipping a disguise on your browser, keeping it under the radar.
Apify SDK – Imagine a cloud kitchen where you drop ingredients (your crawling logic) and it serves the dish (data) on a scalable plate. The SDK bundles a proxy pool, so you don’t need to juggle separate services.
OctoParse – For those moments when you need a quick prototype without code, this visual scraper works like a drag‑and‑drop map. Point, click, and it builds the extraction rules, letting you validate a target before committing to a full script.

Putting these together is simple:

Start with ScraperAPI or Apify SDK for reliable rotating IPs.
Switch to Playwright Python when the site detects headless browsers.
Use OctoParse to prototype and confirm selectors before scaling.

With these four tools in your toolbox, you can scrape website without getting blocked and focus on the data that matters.

Quick Reference: Scraping Without Getting Blocked Cheat Sheet

Grab this list, stick it on your monitor, and you’ll stop hitting 403s.

Rotate IPs – think of ordering food from many restaurants; a single address gets flagged, a rotating fleet stays fresh. Use a residential proxy pool and switch every few requests.
Randomize headers & user‑agents – like swapping the driver’s license you show at a checkpoint; vary Accept-Language, Referer, and a fresh UA string for each hit.
Throttle requests – imagine walking through a museum: you don’t sprint from exhibit to exhibit. Keep a 1–2 s pause, then apply exponential backoff when a 429 appears.
Use headless browsers for JS sites – similar to using Google Maps instead of a static paper map; let Playwright or Selenium render the page so scripts run naturally.
Capture and solve CAPTCHAs – picture a concierge handing you a puzzle; send the image to a third‑party service like 2Captcha, wait for the solution, then replay the answer.
Log & retry failures, respect robots.txt – treat each error like a missed package: log the status, pause, and try again later. Skipping disallowed paths avoids unnecessary bans.

Combine these habits into a repeatable routine and you’ll keep the data flowing without the block.

What to Do Next

Grab a free ScraperAPI account, copy the starter script, and watch the first page load without a 403—just like ordering a coffee and getting it instantly.

Easy: Sign up at scraperapi.com, paste the sample python snippet, replace the target URL, and run. If it returns JSON, you’ve already bypassed the basic block.

Medium: Hook Playwright into a rotating proxy pool. Think of it as swapping lanes on a highway to avoid traffic jams. Your script might look like this:

from playwright.sync_api import sync_playwright

proxies = ["http://p1.example:3128","http://p2.example:3128"]
def fetch(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.new_context(proxy={"server": random.choice(proxies)})
        page = context.new_page()
        page.goto(url)
        return page.content()

Run it against a site you control to fine‑tune timing and headers.
Hard: Deploy a scheduled cloud function that scrapes daily and drops results into BigQuery. It’s like setting a smart fridge to restock itself every night. Example outline:

Write a lambda_function.py that calls your Playwright routine.
Package dependencies with a requirements.txt.
Create a CloudWatch rule (or Cloud Scheduler) to trigger the Lambda at 02:00 UTC.
Use the BigQuery client library to insert rows into a dataset.

Pick the step that matches your comfort level, give it a spin, and you’ll stop hitting those dreaded CAPTCHAs.

Which anti‑bot hurdle has slowed you down the most? Share your story below.

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

With 6+ years of experience, Abdullah has built CRMs, Crypto Wallets, DeFi Exchanges, E-Commerce Stores, HIPAA Compliant EMR Systems, and AI-powered systems that drive business efficiency and innovation.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Set Up Traefik as a Reverse Proxy with Docker in Minutes

Abdullah Sheikh — Mon, 08 Jun 2026 12:04:54 +0000

Step‑by‑step guide to install, configure, and run Traefik in Docker so you can route traffic to multiple containers effortlessly

Before We Start: What You'll Walk Away With

When you finish this guide you’ll have a live Traefik reverse proxy Docker setup that starts with a single docker‑compose up command.

Adding another micro‑service will be as simple as dropping a label on its container—Traefik will instantly discover it and route traffic, just like a new restaurant appearing on Google Maps when you open the app.

You’ll also walk away with a clear picture of the three core pieces—entrypoints, routers, and services—without drowning in terminology.

Working container: A ready‑to‑go Traefik instance defined in docker-compose.yml, listening on ports 80 and 443, and serving a dashboard you can reach at http://localhost:8080.
One‑label addition: Any new Docker service gets a traefik.http.routers... label, and Traefik automatically creates the route—think of it as handing a suitcase to a conveyor belt that knows exactly where to send it.

Core concepts simplified:

Entrypoint – the door (port) where traffic enters.
Router – the map that decides which door leads to which service.
Service – the final destination, the container that handles the request.

Cheat sheet:

docker compose up -d – launch everything.
docker compose logs traefik – watch routing in real time.
docker inspect – verify labels are applied.

Tools & tips:

Use docker network create traefik-public so all services share the same network.
Keep the traefik.yml file minimal; defaults cover most needs.
Enable the dashboard only on localhost for security.

Now you’re set to spin up Traefik and let it do the heavy lifting for your Docker stack.

What Traefik Actually Is (No Jargon)

Traefik is a cloud‑native reverse proxy that lives next to your Docker containers and rewrites its routing table the moment a new service appears or disappears. It talks directly to the Docker daemon, reads labels you add to each container, and instantly knows which hostname or path should point at which container.

Imagine a busy downtown intersection with a traffic officer who never sleeps. Cars (incoming HTTP requests) pull up, and the officer flashes a sign directing each driver to the right lane (the right container) based on the destination they typed into their GPS. If a new lane opens, the officer updates the signs on the fly; if a lane closes, the signs disappear. That officer is Traefik, and the intersection is your Docker‑compose network.

It detects containers automatically—no manual IP hunting.
It respects the rules you define in Docker labels, so you control routing where you write your docker-compose.yml.
It handles TLS certificates for you, pulling them from Let’s Encrypt without extra steps.

Because Traefik updates in real time, you can spin up a new microservice, tag it with traefik.http.routers.myapp.rule=Host(app.example.com), and the traffic starts flowing instantly. No restarts, no fiddling with nginx configs, just a single source of truth that lives alongside your containers.

Bottom line: think of Traefik reverse proxy Docker as the ever‑watchful traffic controller that keeps your services reachable and your network tidy.

The 3 Mistakes Everyone Makes With Traefik

Most people hit a wall with Traefik reverse proxy Docker before they even see a single request hit their service.

Forgetting to expose the right entrypoint ports. It’s like ordering a pizza and never opening the delivery door – the driver can’t get the food in. If ports for 80:80 or 443:443 aren’t listed in docker‑compose.yml, Traefik has nowhere to listen and you’ll get 502 errors. Double‑check the expose and ports sections match the entrypoints you defined.
Mislabeling services or using the wrong label prefix. Think of labels as the street signs that tell Google Maps where to go. Using traefik.http.routers... when you’ve set --providers.docker to expect traefik.docker.* makes Traefik blind to your containers. One typo and the whole stack disappears from the dashboard.
Over‑complicating TLS with self‑signed certs. It’s like packing a suitcase with every gadget you own, then forgetting the charger. Instead of generating a maze of certificatesResolvers, start with the built‑in Let’s Encrypt starter: add --certificatesresolvers.myresolver.acme.email and --certificatesresolvers.myresolver.acme.storage. Traefik will handle renewals automatically.

Quick cheat sheet:

Expose ports: ports: - "80:80" - "443:443"
Label prefix: traefik.http. for routers, services, middlewares
Let’s Encrypt starter: --certificatesresolvers.myresolver.acme.email=you@example.com

Fix these three, and Traefik will stop being the mystery box in your Docker stack.

How to Set Up Traefik: Step‑By‑Step

Grab a folder, drop a docker-compose.yml inside, and you’re ready to start.

Create a project directory and touch docker-compose.yml. This is like opening a new notebook for a recipe you’ll follow step‑by‑step.
Paste the Traefik service definition into the file. Use the official image, turn on the Docker provider, and expose ports 80 and 443. It’s the same as telling a delivery driver which streets are open for traffic.
Write a static config file named traefik.yml. Define entryPoints for web (port 80) and websecure (port 443) and enable Let’s Encrypt. Think of this as setting up the highway signs before the cars arrive.
Mount both traefik.yml and the Docker socket into the container:

services:
  traefik:
    image: traefik:v2.10
    command:
      - --providers.docker
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.email=you@example.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - ./traefik.yml:/etc/traefik/traefik.yml
      - /var/run/docker.sock:/var/run/docker.sock
      - ./letsencrypt:/letsencrypt

Add a simple web service, for example an nginx container, and tag it with Traefik labels that specify the host rule and TLS resolver.

nginx:
    image: nginx:alpine
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx.rule=Host(`yourdomain.test`)"
      - "traefik.http.routers.nginx.entrypoints=websecure"
      - "traefik.http.routers.nginx.tls=true"
      - "traefik.http.routers.nginx.tls.certresolver=myresolver"

Spin everything up with docker compose up -d and visit http://localhost:8080 to see the Traefik dashboard.
Open https://yourdomain.test in a browser. If the page loads with a valid certificate, the reverse proxy is working.

That’s the whole setup—quick, clean, and ready for production.

A Real Example: Deploying a Blog and API Behind Traefik

Maya wants her new Hugo blog and Node API reachable at https URLs without fiddling with certificates.

She creates a docker-compose.yml that defines three services.

services:
  traefik:
    image: traefik:v2.10
    command:
      - --providers.docker=true
      - --providers.file.directory=/etc/traefik
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.le.acme.email=maya@example.com
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/etc/traefik
      - traefik-letsencrypt:/letsencrypt
    labels:
      - traefik.enable=true

  blog:
    image: nginx:alpine
    volumes:
      - ./blog/public:/usr/share/nginx/html:ro
    labels:
      - traefik.http.routers.blog.rule=Host(`blog.example.com`)
      - traefik.http.routers.blog.tls=true
      - traefik.http.routers.blog.tls.certresolver=le

  api:
    image: node:14
    working_dir: /app
    command: ["node", "server.js"]
    volumes:
      - ./api:/app
    labels:
      - traefik.http.routers.api.rule=Host(`api.example.com`)
      - traefik.http.routers.api.tls=true
      - traefik.http.routers.api.tls.certresolver=le

volumes:
  traefik-letsencrypt:

She drops a tiny traefik.yml into config/ to tell Traefik where to look for dynamic configs.

log:
  level: INFO
api:
  dashboard: true

Running docker compose up -d is like handing the keys to a valet; Traefik immediately grabs the TLS certificates from Let’s Encrypt and routes traffic.
Now https://blog.example.com serves Maya’s static site, and https://api.example.com handles her Node endpoints, both protected by a Traefik reverse proxy Docker stack.
Tip: Keep the acme.json file readable only by root ( chmod 600 ) to protect private keys.
Tip: Use the Traefik dashboard (http://localhost:8080/dashboard/) to verify routers are live.

The Tools That Make This Easier

Grab the right helpers and the whole Traefik reverse proxy Docker setup feels like ordering a pizza with a clear menu.

Docker Desktop (2025 edition) – The visual dashboard is your restaurant’s order board. You can see containers, volumes, and networks at a glance, start or stop services with a click, and drag‑and‑drop a docker‑compose.yml into the UI to spin everything up instantly.
VS Code with the Docker extension – Think of it as a chef’s knife that’s always sharp. It highlights YAML syntax, offers autocomplete for labels, and lets you tail logs from the sidebar, so you spot mis‑typed routers before they bite.
Portainer Community Edition – This is the Google Maps for your Docker world. The visual Compose editor lets you tweak services, and the health dashboard flags containers that restart or report errors, saving you from hunting through terminal output.
Traefik Pilot (free tier) – Consider it a real‑time kitchen monitor. It displays every router, service, and TLS certificate on a single page, so you instantly know if a rule isn’t matching or a certificate has expired.
mkcert – The fast‑track to trusted TLS for local dev, like a passport office that prints a visa on demand. Run mkcert "*.local" and you get a cert that browsers accept without warnings.

With these tools in your toolbox, wiring Traefik into Docker becomes a smooth, repeatable process.

Quick Reference: Traefik Reverse Proxy Cheat Sheet

Grab this cheat sheet and keep it open while you spin up Traefik – every line maps to a concrete action.

Docker‑Compose file: define a traefik service, publish 80:80 and 443:443, mount /var/run/docker.sock:/var/run/docker.sock. Think of it as ordering a pizza with both crust and cheese sides ready.
traefik.yml: set entrypoints web (port 80) and websecure (port 443), plus ACME block for Let’s Encrypt. It’s like giving Google Maps the start and destination points before the route is drawn.
Labels on each app: add traefik.http.routers.myapp.rule=Host(myapp.example.com) and traefik.http.routers.myapp.tls=true. Labels are the luggage tags that tell the suitcase (your container) which carousel (router) to go to.

Bring it online: run

docker compose up -d

then visit http://localhost:8080 for the dashboard. If the UI pops up, you’ve just turned the key in the ignition.
Common pitfalls:

Forgot to expose ports – Traefik stays hidden like a shop without a sign.
Used label: instead of labels: or missed the traefik. prefix – the router can’t read the address.
Mixed up TLS: set tls=true but left entryPoints on web only – you’ll get a “connection not secure” warning.
Example fix from Alice, a backend engineer: she added traefik.http.routers.myapp.tls.certresolver=myresolver and the HTTPS site worked instantly.

Keep this list handy; it’s the shortcut you’ll reach for every time you deploy a new service behind Traefik.

What to Do Next

Ready to get more mileage out of your Traefik reverse proxy Docker setup? Here are three steps you can take right now.

Easy – add a new service. Think of it like ordering a side dish at a restaurant: you just copy the existing label pattern and specify the new container. For example, to expose a Redis admin UI, drop these labels into the service definition:

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.redis-admin.rule=Host(`redis.local`)"
  - "traefik.http.services.redis-admin.loadbalancer.server.port=80"

Restart docker-compose up -d and the UI appears at http://redis.local.
Medium – enable middleware. Middleware is like adding a traffic cop to a busy intersection. You can limit request bursts or require a password before anyone reaches your internal tools.

labels:
  - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$H6uskkkW$$IgXLP6ewTrSuA1u5cG3c41"
  - "traefik.http.routers.redis-admin.middlewares=auth"

Swap basicauth for ratelimit if you just need to curb traffic spikes.
Hard – scale with Swarm or Kubernetes. Picture packing a suitcase for a long trip: you need compartments, weight limits, and a plan for adding more items later. In a Swarm, define a traefik service with --providers.docker.swarmMode=true; in Kubernetes, deploy the Helm chart and annotate pods with traefik.ingress.kubernetes.io/router.entrypoints.

docker service create \
  --name traefik \
  --publish 80:80 \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
  traefik:latest \
  --providers.docker.swarmMode=true \
  --entrypoints.web.address=:80

After the service is up, add the same label syntax to any Swarm stack to have Traefik route it automatically.

Got stuck or have a creative use‑case? Drop a comment below – I love hearing how you’re using Traefik!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Build a RAG System with Your Own Documents in 7 Simple Steps

Abdullah Sheikh — Sun, 07 Jun 2026 12:03:10 +0000

Create a Retrieval‑Augmented Generation pipeline that answers questions from your private data without writing code

Before We Start: What You'll Walk Away With

By the end of this tutorial you’ll be able to point a chatbot at any collection of PDFs, Word docs, or wiki pages and get accurate answers in seconds.

First, you’ll grasp the three moving parts of a RAG system—the retriever, the generator, and the glue that stitches them together—just like understanding the driver, the map, and the GPS signal when you order a ride.

Next, you’ll spin up a fully functional pipeline using only free or freemium services, so there’s no need to hire a data‑science team or rent pricey GPU servers.

Finally, you’ll have a ready‑to‑plug component you can drop into a Slack bot or a lightweight web UI, turning your knowledge base into an instant assistant.

Component view: Retriever (searches your docs), Generator (writes the answer), Orchestrator (passes the query).
Toolchain: Vector store (e.g., Pinecone free tier), LLM API (OpenAI’s chat model with free credits), No‑code connector (Zapier or Make).
Deployment: Wrap the endpoint with a simple Flask app or use a serverless function, then hook it to Slack or a static HTML page.

Think of it like packing a suitcase: you pick the right items (documents), arrange them efficiently (vector index), and add a travel guide (LLM) that tells you exactly where everything is when you need it.

When you finish, you’ll have a RAG system tutorial you can replicate for any new project, no code beyond a few configuration files.

Ready to start building?

What Retrieval‑Augmented Generation Actually Is (No Jargon)

Think of a RAG system as a two‑person team: one side is a chatty, well‑read friend (the generator) and the other side is a tidy filing cabinet (the retriever) that holds all the docs you care about.

The friend knows how to string words together fluently, but they don’t remember the latest policy updates or product specs. When you ask a question, the friend first asks the filing cabinet for the most relevant pages, then crafts an answer that blends that fresh info with their natural language skill.

In practice, the retriever runs a quick search across your PDFs, Word files, or wiki, returning a handful of snippets. Those snippets are fed to the generator, which writes a response that feels like a conversation yet stays grounded in your actual documents.

Imagine ordering a custom sandwich. The chef (generator) can make any combination, but they need to know which ingredients you have in the fridge (retriever). They glance at the inventory, pick the freshest lettuce and the exact cheese you stocked, then assemble a sandwich that tastes right and matches what you asked for.

The payoff is simple: the AI assistant answers with up‑to‑date, company‑specific facts while keeping the flow of a casual chat. No massive model training, no data‑science squad—just a searchable index and a language model working together.

The 3 Mistakes Everyone Makes With RAG

Most people hit a wall early because they treat a RAG system like a generic search engine.

Using a generic web‑search index instead of a vector store. Imagine ordering a pizza and getting a sushi menu – the results are technically “searchable” but completely useless. A vector store embeds your own PDFs and Word files, so similarity matches stay on topic.
Feeding the whole document into the LLM at once. This is like trying to stuff an entire suitcase into a carry‑on; you’ll exceed the size limit and end up with broken zippers. Large language models have token caps; when you overload them they truncate, guess, and hallucinate.
Ignoring chunk‑size and overlap. Picture Google Maps without street‑level detail: you see the city, but you can’t navigate the alleys. If you split text into chunks that are too big or have no overlap, the similarity search loses the context needed for accurate answers.
Fix #1: Set up faiss or pinecone as your vector store; index only the embeddings from your own docs.
Fix #2: Chunk documents into 300‑500 token pieces before embedding; keep each piece well under the model’s limit.
Fix #3: Add a 50‑token overlap between consecutive chunks so concepts that span sections stay linked.

Skip these pitfalls and your RAG system tutorial will actually deliver the answers you need.

How to Build a RAG System: Step‑by‑Step

Grab your docs, clean them up, and you’re ready to feed them to the AI.

Gather & clean – collect every PDF, DOCX, or Markdown file. If any PDFs are scanned, run an OCR pass so they become searchable plain text. Think of it like washing dishes before you start cooking; you don’t want crumbs in the sauce.
Chunk the text – split the cleaned text into overlapping pieces, about 500 tokens each with a 100‑token overlap. Overlap works like a puzzle border: it gives the model context from the neighboring piece.
Create embeddings – send each chunk to a lightweight model such as text-embedding-ada-002 or a HuggingFace sentence‑transformers model. The output is a dense vector that captures the meaning of the chunk.
Store vectors – push the embeddings into a vector database like Pinecone, Weaviate, or Qdrant. The DB acts as a fast “Google Maps” for similarity: it finds the nearest points (chunks) to a query.

Build the retrieval API – write a tiny endpoint that:

accepts a user query,
asks the vector store for the top‑k similar chunks,
concatenates the query with those chunks, and
calls an LLM to generate the answer.

from fastapi import FastAPI, Request
import pinecone, openai

app = FastAPI()
pinecone.init(api_key="YOUR_KEY")
index = pinecone.Index("rag-index")

@app.post("/ask")
async def ask(req: Request):
    data = await req.json()
    query = data["question"]
    q_emb = openai.Embedding.create(model="text-embedding-ada-002", input=query)["data"][0]["embedding"]
    results = index.query(vector=q_emb, top_k=5, include_metadata=True)
    context = "
".join([r["metadata"]["text"] for r in results.matches])
    answer = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role":"system","content":context},{"role":"user","content":query}]
    )
    return {"answer": answer.choices[0].message.content}

Wrap it in a UI – expose the API through a Slack bot, a Streamlit dashboard, or a Teams connector. This is the “serving plate” that lets users ask questions without seeing the code.
Test & monitor – run sample queries, measure latency, and add a relevance feedback button so users can flag wrong answers. Over time, you can fine‑tune chunk size or top‑k to improve results.

Follow these seven actions and you’ll have a functional RAG system tutorial ready for real users.

A Real Example: Turning a Marketing Playbook Into an AI Coach

Maya, a SaaS marketing manager, wants her team to pull answers from the 2023 Playbook as fast as ordering a coffee.

She drops the 42 PDFs into a shared Google Drive folder and runs the free pdf2txt CLI to pull raw text into a bucket called playbook_raw.
Using a tiny Python script she slices each document into 600‑token chunks with a 150‑token overlap—think packing a suitcase so nothing gets cramped.
Each chunk is sent to OpenAI’s ada‑002 embedding endpoint; the resulting vectors are saved locally.
She spins up a free Qdrant cloud instance, creates a collection, and pushes the embeddings along with their source IDs.
A lightweight FastAPI service reads a query, fetches the top 4 closest chunks from Qdrant, and returns them as JSON.
The API is wired to a Slack slash command /playbook. When a teammate types a question, Slack forwards it to the FastAPI endpoint, which then calls gpt‑4‑turbo with the retrieved snippets to generate the final answer.

The whole flow feels like using Google Maps: you input a destination (the question), the service finds the best roads (relevant chunks), and then gives you turn‑by‑turn directions (the AI‑crafted answer).

Tools: pdf2txt (CLI), Python 3.10+, FastAPI, Qdrant Cloud (free tier), Slack API.
Tips: Keep chunk size under 800 tokens to stay within model limits; use a 150‑token overlap to preserve context across boundaries.

import openai, qdrant_client, pathlib, json

def embed(text):
    return openai.Embedding.create(model="ada-002", input=text)["data"][0]["embedding"]

def chunkify(txt, size=600, overlap=150):
    tokens = txt.split()
    for i in range(0, len(tokens), size - overlap):
        yield " ".join(tokens[i:i+size])

for file in pathlib.Path("playbook_raw").glob("*.txt"):
    for chunk in chunkify(file.read_text()):
        vec = embed(chunk)
        qdrant_client.upsert(collection="playbook", points=[{"id":hash(chunk), "vector":vec, "payload":{"text":chunk}}])

Now any team member can ask, “What’s the recommended email cadence for enterprise leads?” and get a spot‑on answer in under 30 seconds.

The Tools That Make This Easier

Grab the right toolbox and the whole process feels like ordering a meal—you pick the ingredients, the kitchen does the work, and you get a finished dish.

pdf2txt – a free CLI that pulls plain text from PDFs and runs OCR when needed. Think of it as a scanner that not only copies but also translates the printed page into searchable words.

pdf2txt --input report.pdf --output report.txt --ocr

LangChain – open‑source library that chops documents into chunks, builds retrieval logic, and talks to the LLM. It’s like a travel planner that breaks a long trip into stops, then finds the best route between them.
Qdrant Cloud – vector database with a free tier up to 5 M vectors. It stores embeddings and returns the nearest matches in milliseconds, similar to how Google Maps instantly shows the closest coffee shop.
OpenAI API – use text-embedding-ada-002 for cheap, high‑quality embeddings and gpt-4-turbo for completions. It’s the reliable delivery service that brings your answers right to the table without breaking the budget.
Streamlit Community Cloud – spin up a web UI for your RAG endpoint in minutes, free of charge. Imagine packing a suitcase: Streamlit provides the bag, you just drop in the components.

These five tools cover extraction, chunking, storage, reasoning, and presentation, giving you a complete RAG system tutorial without writing a single line of infrastructure code.

With them in place, the next step is wiring everything together.

Quick Reference: RAG System Cheat Sheet

Here’s the whole process at a glance, ready to copy‑paste into your notes.

Gather → clean → plain‑text all docs. Treat it like gathering groceries: pick the items, throw away the wilted ones, then lay everything on the counter in a single list.
Chunk: 500‑600 tokens, 20‑30% overlap. Think of packing a suitcase; each box (chunk) holds a manageable amount, and a little overlap ensures nothing gets left behind.
Embed with Ada‑002 or sentence‑transformers. It’s like converting a photo into a fingerprint—choose a fast, reliable tool to turn text into vectors.
Store in Qdrant / Pinecone / Weaviate. Pick a pantry that keeps your fingerprints organized and searchable.
API flow: query → retrieve top‑k → LLM → answer. Imagine Sam, a product manager, typing a question in Slack. The system pulls the best matches, feeds them to the LLM, and returns a concise reply.
UI options: Slack slash command, Streamlit, Teams bot. Pick the front‑door that your team already uses, just like adding a new lane to an existing road.
Test with 5 real questions, monitor latency **Best‑practice tips:
Keep chunk size consistent; irregular pieces cause uneven search results.
Use a small top_k (e.g., 5) to stay under the latency budget.
Log each query’s response time; alerts on >2 s keep the experience smooth.

Keep this cheat sheet handy; it’s your RAG system tutorial shortcut.

What to Do Next

Kick the tires on your new RAG system with three bite‑size actions, then decide how far you want to push it.

Easy: Grab the run_one_click.py script, point it at a folder of a few PDFs, and hit python run_one_click.py. Within minutes you’ll see vectors land in Qdrant—think of it as watching ingredients pop onto a kitchen counter before you start cooking.
Medium: Hook the FastAPI endpoint to a Slack bot (slack-bot.py in the repo) and invite a teammate to ask five real questions. This is like setting up a walk‑up window: you get instant feedback on whether the assistant is serving the right answers.
Hard: Build a relevance‑feedback loop. Store thumbs‑up/down in a feedback table, re‑rank chunks with ranker = CrossEncoder(...), and periodically fine‑tune a lightweight embedding model (e.g., sentence‑transformers/all-MiniLM-L6‑v2). It’s the equivalent of training a personal barista who remembers your exact coffee preferences.
Tool tip: Use docker-compose up -d qdrant to keep the vector store running locally while you experiment.

Cheat sheet:

python run_one_click.py – ingest PDFs
uvicorn main:app --reload – launch API
python slack-bot.py – start Slack bridge

Got stuck or discovered a shortcut? Drop a comment below – I love hearing how you’ve customized the workflow!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Automate LinkedIn Posts with n8n and the LinkedIn API

Abdullah Sheikh — Sat, 06 Jun 2026 12:03:20 +0000

Step‑by‑step guide to set up a fully automated LinkedIn publishing workflow using n8n

Before We Start: What You'll Walk Away With

By the end of this guide you’ll have a fully‑functional n8n workflow that posts to LinkedIn on autopilot, just like ordering a meal that arrives at your door without you ever picking up the phone.

First you’ll spin up a LinkedIn app, grab the OAuth tokens, and plug them into n8n – think of it as getting the key that opens the restaurant’s kitchen.

Next you’ll stitch together a reusable flow that pulls headlines from a Google Sheet or an RSS feed and pushes them out as LinkedIn updates, similar to loading a suitcase once and reusing it for every trip.

Finally you’ll learn the quick fixes for the most common API hiccups and set the workflow to run on a schedule that matches your posting cadence, whether that’s daily breakfast‑posts or hourly news bursts.

Create a LinkedIn developer app and retrieve clientId, clientSecret, and the refresh token.
Connect those credentials to n8n’s LinkedIn node.
Design a workflow that reads from a source (Google Sheet or RSS) and calls the LinkedIn API to publish.
Add error handling steps and a cron trigger for scheduling.
Tool tip: Use n8n’s built‑in OAuth2 credentials manager – it stores tokens securely, so you don’t have to paste them every time.
Cheat sheet: GET https://api.linkedin.com/v2/me verifies your token; a 401 means you need to refresh.
Tip: Keep your content columns simple (title, url, image) to avoid payload errors.

Now you’re ready to stop the copy‑paste grind and let n8n handle the heavy lifting.

What Automating LinkedIn Posts Actually Is (No Jargon)

Automation is simply a set of instructions that run on their own, just like a coffee machine starts brewing the moment you hit “start.” You set it up once, then it does the work while you focus on other things.

When we talk about automating LinkedIn posts, think of n8n as that coffee machine and the LinkedIn API as the coffee beans. You feed the machine a recipe—fetch a headline, attach an image, schedule a time—and the machine takes the ingredients, mixes them, and delivers a fresh post to LinkedIn without you ever lifting a finger.

Trigger: a new row appears in your Google Sheet, like placing a fresh coffee pod in the slot.
Action: n8n reads the row, formats the text, and hands it off to the LinkedIn API, just as the machine grinds beans and pours the brew.
Result: your post appears on LinkedIn at the exact moment you wanted, without copy‑and‑paste or rate‑limit headaches.

This is what automate LinkedIn posts looks like in plain English: set up a repeatable workflow, let the tools do the heavy lifting, and walk away with a steady stream of content that lands exactly where it should.

The 3 Mistakes Everyone Makes With LinkedIn Automation

Most people hit a wall fast because they skip the basics of LinkedIn’s API.

Using a personal LinkedIn account instead of a Company Page. Think of it like ordering a meal at a fast‑food counter: the personal account only offers a limited menu, while a Company Page unlocks the full buffet of API endpoints. Without a Page, you can’t publish on behalf of your brand or access analytics.
Skipping the OAuth 2.0 refresh‑token step. It’s like relying on a GPS that only works for the first 30 minutes of a road trip—once the token expires, the route disappears and your workflow stalls. A refresh token keeps the session alive so n8n can keep posting.
Over‑loading the API with too many requests at once. Imagine stuffing a suitcase beyond the airline’s weight limit; LinkedIn will refuse the excess and may temporarily block you. Batch your posts, add a short delay, and stay under the rate‑limit thresholds.

Fix these three pitfalls and your automation to automate LinkedIn posts will run smoothly.

How to Automate LinkedIn Posts: Step‑by‑Step

Let’s walk through the exact actions you need to get a hands‑free LinkedIn posting flow.

Create a LinkedIn Developer App and request the w_member_social and rw_organization_admin scopes. Think of it like ordering a meal: you pick the restaurant (app) and tell the server exactly which dishes (permissions) you want.
Generate an OAuth 2.0 authorization code, then exchange it for an access token and a refresh token. This is the “check‑in” at the hotel lobby – you give your reservation number (auth code) and receive the key (access token) plus a spare key (refresh token) for later.
Add the LinkedIn credentials node in n8n and paste the tokens. n8n stores them in its encrypted database, so you don’t have to write them down on a sticky note.
Build a trigger node (for example, Google Sheets → “New Row”). Each new row becomes the content package you want to post, just like a courier delivering a sealed envelope to your front desk.
Add an HTTP Request node aimed at the LinkedIn ‘UGC Posts’ endpoint. Map the title, body, and optional media fields from the trigger. Use the stored access token in the Authorization header.

{
  "method": "POST",
  "url": "https://api.linkedin.com/v2/ugcPosts",
  "headers": {
    "Authorization": "Bearer {{ $credentials.accessToken }}",
    "Content-Type": "application/json"
  },
  "body": {
    "author": "urn:li:person:{{ $json.authorId }}",
    "lifecycleState": "PUBLISHED",
    "specificContent": {
      "com.linkedin.ugc.ShareContent": {
        "shareCommentary": { "text": "{{ $json.postText }}" },
        "shareMediaCategory": "NONE"
      }
    },
    "visibility": { "com.linkedin.ugc.MemberNetworkVisibility": "PUBLIC" }
  }
}

Imagine Maya, a solopreneur, adds a new row with “Launch day! 🎉”. The HTTP node picks up Maya’s text and publishes it instantly.

Test the workflow. Run it once, check that the post shows up on your LinkedIn profile, then activate a schedule or webhook so future rows fire automatically.
Set up error handling. Add an “If” node that looks for status codes 429 or 403. Connect a “Wait” node that pauses for the back‑off period (e.g., 60 seconds) before retrying. This prevents rate‑limit bans and keeps the automation smooth.

Now you have a reliable way to automate LinkedIn posts without touching a single line of server code.

A Real Example: Weekly Thought‑Leadership Post for a SaaS Founder

Every Monday Maya drops a fresh insight into her Google Sheet, and the rest of the workflow does the heavy lifting.

Add the row. Maya opens the sheet called “LinkedIn Queue” and fills in three cells: Date (next Monday), Text (her 150‑word thought‑leadership piece), and Image URL (a link to a custom graphic). Think of it like ordering a lunch combo: you choose the date, the main dish, and the side.
Trigger the workflow. n8n watches the sheet for a “New Row” event. As soon as Maya saves, the trigger fires—just like a doorbell rings when someone steps onto your porch.
Format the payload. A “Set” node trims whitespace, adds a hashtag, and builds a JSON object that matches LinkedIn’s requirements. The image URL is fetched with an HTTP request node, turning the remote file into a base‑64 blob ready for upload.
Post to LinkedIn. The LinkedIn node receives Maya’s Company Page ID, the formatted text, and the image blob, then calls POST /v2/ugcPosts. If the API returns a 201, the post is live.
Notify Maya. A Slack node sends a message to her “#marketing‑updates” channel: “Your Monday insight is up!” and includes the direct link to the article.
Tip: Keep the sheet in the same timezone as your audience to avoid off‑hours posting.
Tool: Use n8n’s built‑in “Google Sheets – Get Row” node to avoid extra API keys.
Cheat sheet: text = text.trim() + " #SaaS" – quick way to add a consistent hashtag.

Now Maya spends a few seconds on Monday morning, and the rest runs itself.

The Tools That Make This Easier

Grab the tools that turn a manual copy‑paste routine into a smooth, code‑free pipeline.

n8n Cloud – Think of it as the Google Maps for workflows. Drag‑and‑drop nodes, connect your LinkedIn API node to a Google Sheets trigger, and you’ve got a route from draft to publish. The free tier lets you run 1,000 executions a month, plenty for a modest posting schedule.
Google Sheets – Your content queue is like a shared kitchen whiteboard. Anyone on the team can drop a headline, body text, and image URL into a row, and n8n will pick it up automatically. No need for a database.
LinkedIn Developer Portal – This is the ticket counter where you order your API access. Register an app, grab the client ID and secret, and set the required permissions (r_liteprofile, w_member_social). All the authentication steps happen behind the scenes in n8n.
Slack (optional) – Use it as the kitchen timer. A quick webhook node sends a green checkmark when a post succeeds or a red alert if LinkedIn throws a rate‑limit error.
Postman – Treat it like a tasting spoon before you serve. Fire a POST https://api.linkedin.com/v2/ugcPosts request with sample data to verify scopes and payload structure before wiring the call into n8n.

With these five pieces you can assemble a reliable system that actually automate LinkedIn posts without writing a single line of full‑stack code.

Quick Reference: LinkedIn Automation Cheat Sheet

Grab this list when you need a quick refresher on automating LinkedIn posts with n8n.

✅ Create LinkedIn App – think of it like signing up for a new loyalty card; you receive a client ID and client secret you’ll use later.
✅ OAuth 2.0 flow – just as you’d exchange a ticket for a seat, request an access token and a refresh token from LinkedIn.
✅ Store tokens in an n8n credentials node – like keeping your keycard in a safe spot so the door opens automatically.
✅ Trigger – use a Google Sheet new‑row webhook (or RSS) as the “order placed” signal, similar to a restaurant receiving a new order.
✅ Action – add an HTTP Request node pointed at https://api.linkedin.com/v2/ugcPosts. This is the kitchen where the post gets cooked.
✅ Map fields – set author to urn:li:organization:{ORG_ID} and lifecycleState to PUBLISHED. It’s like addressing a package before you ship it.
✅ Error handling – catch 429 (rate limit), pause, then retry; log 403 for permission problems, just as you’d note a delivery failure and retry later.
✅ Schedule – run the workflow daily at 09:00 UTC or fire a webhook for real‑time publishing, similar to setting an alarm clock for a regular breakfast.
Tip: Keep the refresh token in a secure secret manager; it’s the spare key you’ll need when the access token expires.
Tip: Test the HTTP Request with a single dummy post before scaling to a full sheet run.
Tip: Use n8n’s built‑in “Wait” node after a 429 response to respect LinkedIn’s rate limits.

Stick this cheat sheet on your monitor and the automation will run itself.

What to Do Next

Grab the template, tweak it, and you’ll have a running automate LinkedIn posts workflow in minutes.

Duplicate the sample n8n workflow from the template link and link your own Google Sheet. It’s like copying a favorite pizza recipe—just swap the toppings you already have.
Swap the Google Sheet trigger for an RSS feed if you want blog excerpts to publish automatically. Think of it as replacing a manual grocery list with a subscription that delivers fresh items right to your cart.
Build a custom UI in Retool so anyone on your team can edit copy, choose images, and set dates, then fire a webhook into the n8n flow. This is the “self‑service kiosk” of posting—no developer needed for each change.
Tip: Test each step with a single post before scaling.
Tool: Use n8n’s “Execute Workflow” node to debug webhook payloads.
Cheat sheet: Keep your LinkedIn API token in an environment variable for easy swapping between dev and prod.

Got a specific use‑case or hit a snag? Drop your question in the comments – I’ll reply within 24 hours.

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Add Web3 Login to Your Website in 5 Simple Steps

Abdullah Sheikh — Fri, 05 Jun 2026 12:03:33 +0000

Implement a secure, wallet‑based authentication flow that works on any modern site

Before We Start: What You'll Walk Away With

By the end of this guide you’ll be able to let users sign into your site with just a crypto wallet, just like they would click “Log in with Google”.

First you’ll grasp how a wallet replaces a password: the wallet holds a private key that signs a message, and the signature proves ownership without ever exposing the key.

Next you’ll drop a ready‑made login button onto any page using the widely‑adopted Web3Auth SDK, no heavy blockchain code required.

Finally you’ll set up a tiny backend endpoint that checks the signature, confirms the address, and creates a session cookie – the same way a server validates an OAuth token.

Concept – Think of the wallet as a digital passport; the signature is the stamp that says “this traveler is who they claim to be”.
Integration – Adding the button is like embedding a “Buy Now” widget: copy the script tag, call Web3Auth.init(), and you’re done.
Verification – Your server acts like a border officer, reading the stamp (signature) and matching it to the passport (address) before letting the user pass.
Tool: @web3auth/web3auth npm package – lightweight, works in any modern browser.
Tip: Test with MetaMask first; it’s the “Google Maps” of wallets, widely installed and easy to debug.

Cheat sheet:

Step 1: Include SDK script.
Step 2: Render Login.
Step 3: Send signed message to /api/verify.

Now you’re ready to start adding a Web3 login that works for real users.

What Web3 Login Actually Is (No Jargon)

Web3 login lets a visitor prove they own a crypto wallet without typing a password. The browser asks the wallet to sign a random message, then sends that signature to your server. Your server checks the signature against the public address – if it matches, the user is who they claim to be.

Think of the wallet like a passport. The signed message is a handwritten signature on the passport page that a guard (your backend) can verify with a stamp. No one else can forge that exact signature because only the genuine passport holder has the private key.

This process replaces the usual email‑and‑password flow. Instead of remembering credentials, users just click “Connect Wallet” in their browser extension or mobile app. The cryptographic proof is instant, and there’s no chance of a password leak.

In practice, the steps are: generate a nonce, ask the wallet to sign it, send the signature and address to your server, and let the server validate. Once validated, you issue a session token just like any other login system.

That’s the whole idea behind a Web3 login – simple, secure, and familiar once you picture a passport and a guard at the door.

The 3 Mistakes Everyone Makes With Web3 Login

Most people hit a wall because they skip the security basics that keep a Web3 login trustworthy.

Skipping the nonce‑based challenge. Without a fresh, random nonce each login, an attacker can replay a previously captured signature—just like ordering the same meal twice by reusing a receipt. The server thinks it’s a new request, but it’s really a copy‑paste.
Storing the raw signature in plain text. Think of a signature as a handwritten password. Saving it as‑is is like leaving that note on a sticky note stuck to your monitor. Hash it first, just as you would lock a diary with a key before storing it.
Ignoring wallet compatibility. Testing only with MetaMask is like navigating with a single map app and assuming it works everywhere. Users might have Trust Wallet, Coinbase Wallet, or a hardware device; each can format the signature slightly differently, and your code will break if you haven’t accounted for it.

Quick cheat sheet:

Generate a crypto.randomBytes(16) nonce for every login request.
Hash signatures with keccak256 before persisting.
Detect provider.isMetaMask, isCoinbaseWallet, etc., and test across at least three wallets.

Fix these three slip‑ups and your Web3 login will stop being a leaky faucet.

How to Add Web3 Login: Step‑by‑Step

Let’s get that Web3 login running in five bite‑size actions.

Install ethers.js (or web3.js). Open a terminal and run:

npm install ethers

This is like adding a new utensil to your kitchen—once it’s there, you can start cooking.

Add a “Connect Wallet” button to your page and ask the user for permission:

Connect Wallet

document.getElementById('connect').onclick = async () => {
  if (window.ethereum) {
    const [addr] = await window.ethereum.request({method: 'eth_requestAccounts'});
    console.log('User address:', addr);
  } else {
    alert('MetaMask not detected');
  }
};

Think of this as the “order now” button at a café – it prompts the user to hand over their wallet ID.

Generate a server‑side nonce and send it to the client.

// server (Node/Express example)
app.get('/nonce', (req, res) => {
  const nonce = Math.floor(Math.random() * 1e6).toString();
  // store nonce with session or DB
  res.json({nonce});
});

The nonce is like a one‑time password you hand to a delivery driver; it proves the request is fresh.

Have the user sign the nonce with their wallet.

async function signNonce(nonce, address) {
  const provider = new ethers.providers.Web3Provider(window.ethereum);
  const signer = provider.getSigner();
  const signature = await signer.signMessage(`Login nonce: ${nonce}`);
  return signature;
}

This mirrors signing a receipt after you’ve paid – the wallet confirms the message belongs to you.

Send the signature back to the server and verify it.

// client
fetch('/verify', {
  method: 'POST',
  headers: {'Content-Type':'application/json'},
  body: JSON.stringify({address, signature, nonce})
});

// server verification
app.post('/verify', async (req, res) => {
  const {address, signature, nonce} = req.body;
  const recovered = ethers.utils.verifyMessage(`Login nonce: ${nonce}`, signature);
  if (recovered.toLowerCase() === address.toLowerCase()) {
    // create session, JWT, etc.
    res.json({ok:true});
  } else {
    res.status(401).json({ok:false});
  }
});

Now you’ve confirmed the user really controls the wallet, just like a bouncer checking an ID.

That’s the whole flow—copy, paste, and you’ve got a production‑ready Web3 login.

A Real Example: Adding Web3 Login to a SaaS Dashboard

Maya runs the product team at a fintech startup and needs her beta users to drop into the dashboard with just a click on MetaMask.

Install dependencies – In her Next.js repo she runs npm i ethers@5 and adds @web3modal/react for the wallet UI.
Create the client side hook – A tiny React component renders a “Connect Wallet” button, requests the user’s address, and asks them to sign a nonce.
Spin up a Vercel serverless function – The API route /api/auth receives the signed message, recovers the address with ethers, and returns a JWT.
Guard the dashboard page – On page load the client checks for the JWT; if it’s missing, it redirects to the connect component.
Persist the session – Maya stores the token in localStorage and sets an Authorization header for future API calls.
Next.js makes the front‑end and API live in the same repo, like ordering a meal and getting the receipt on the same table.
Vercel serverless functions act like a quick‑service counter: they spin up only when the signature arrives.
Ethers.js is the “Google Maps” of blockchain – it handles address recovery without Maya needing to understand the low‑level crypto.

import { useState } from 'react';
import { ethers } from 'ethers';
import { Web3Button } from '@web3modal/react';

export default function Connect() {
  const [addr, setAddr] = useState(null);

  const handleConnect = async (provider) => {
    const signer = provider.getSigner();
    const address = await signer.getAddress();
    const nonce = await fetch('/api/nonce').then(r => r.text());

    const signature = await signer.signMessage(`Login nonce: ${nonce}`);
    const res = await fetch('/api/auth', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ address, signature })
    });
    const { token } = await res.json();
    localStorage.setItem('token', token);
    setAddr(address);
  };

  return addr ? Welcome, {addr} : ;
}

// pages/api/auth.js
import { ethers } from 'ethers';
import jwt from 'jsonwebtoken';

export default async function handler(req, res) {
  const { address, signature } = req.body;
  const nonce = /* retrieve stored nonce for address */;
  const recovered = ethers.utils.verifyMessage(`Login nonce: ${nonce}`, signature);
  if (recovered.toLowerCase() !== address.toLowerCase()) {
    return res.status(401).json({ error: 'Invalid signature' });
  }
  const token = jwt.sign({ address }, process.env.JWT_SECRET, { expiresIn: '1h' });
  res.status(200).json({ token });
}

With these few lines Maya gets a production‑ready Web3 login for her SaaS dashboard.

The Tools That Make This Easier

Grab the right toolbox and the rest falls into place.

ethers.js – Think of it as a Swiss‑army knife for Ethereum. Install with npm i ethers and you get lightweight functions for signing messages and verifying signatures without pulling in a heavyweight SDK.
Alchemy API – Your blockchain “Google Maps.” The free tier gives reliable node access, so you never get stuck waiting for a block to appear. Just plug the endpoint into your code and start querying.
Vercel serverless functions – Like ordering food for delivery: you write the recipe, Vercel handles the kitchen and the delivery driver. Deploy a verify endpoint in seconds, and it scales automatically.
WalletConnect v2 – Extends your reach beyond desktop MetaMask, just as a universal charger works with many phones. It lets mobile wallets scan a QR code and complete the login flow.
Postman – The quick‑check tool for your signature‑verification endpoint. Fire a POST request with a test signature and see the JSON response instantly, no need to spin up a frontend.

Cheat sheet

npm i ethers – add the library.
https://alchemy.com – sign up, grab your API key.
vercel deploy – push your api/verify.js function.
npm i @walletconnect/web3-provider – enable mobile wallets.
Use Postman to POST {address, signature} to /api/verify.

With these five tools, adding a Web3 login becomes a matter of wiring, not wrestling.

Quick Reference: Web3 Login Cheat Sheet

Grab this cheat sheet and copy‑paste each piece as you build your Web3 login.

Install: npm i ethers – like adding a new app to your phone before you can use it.
Frontend – request accounts: call ethereum.request({ method: 'eth_requestAccounts' }) and read the first address – think of it as ordering a coffee and getting the cup number.
Backend – generate nonce: create a random string, store it in Redis (or memory) for a few minutes – similar to a one‑time PIN you get when you check in at a hotel.
Sign: ask the user to sign the message Login to MySite: ${nonce} – like asking them to write their signature on a receipt that includes the one‑time PIN.

Verify: recover the address from the signature with ethers.utils.verifyMessage and compare it to the address you got earlier.

For example, if Alice connects with 0xAbc…123, the recovered address must also be 0xAbc…123 or the login fails.

Session token: on success, issue a JWT or session cookie and send it to the client – like handing over a hotel key card after the front desk checks your ID.

Keep this list handy; it’s the shortcut to a production‑ready Web3 login.

What to Do Next

Grab your dev server and start moving the pieces you just built.

Easy: Drop the login button and nonce flow onto a fresh test page. It’s like ordering a coffee before the full breakfast—quick, no‑frills, and you see the result instantly. Run npm start, open http://localhost:3000, and verify the wallet connects and the nonce validates.
Medium: Swap the in‑memory nonce store for something persistent such as Redis or DynamoDB. Think of it like switching from a paper notebook to a cloud‑based planner—your data survives server restarts and scales with traffic. A few lines of code and a npm install ioredis (or AWS SDK) get you there.
Hard: Tie wallet addresses to real user records and enforce role‑based access. This is similar to packing a suitcase: you decide which items (permissions) go in which compartment (user profile). Create a users table, store wallet_address, and check the address after login to load roles like admin or viewer.
Tool tip: Use dotenv to keep your Redis credentials out of source control.
Cheat sheet: GET /nonce → generate, POST /verify → check signature, then SELECT role FROM users WHERE wallet_address = ?.

💬 Which part of the Web3 login integration gave you the biggest headache?

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Build a Decentralized App (dApp) from Zero to Deployed in 2025

Abdullah Sheikh — Thu, 04 Jun 2026 12:04:00 +0000

Step‑by‑step guide that lets you design, code, test, and launch a production‑ready dApp on Ethereum or Polygon

Before We Start: What You'll Walk Away With

By the time you finish this guide you’ll be able to take a raw idea and ship a fully functional dApp without hitting a dead‑end.

First, you’ll see the whole dApp lifecycle as clearly as a Google Maps route: set up a local blockchain, write and test a Solidity contract, then drive it onto a public network.

Second, you’ll write a smart contract with Hardhat, run unit tests, and push it to a testnet where you’ll verify the bytecode on Etherscan—just like checking a receipt after ordering food.

Third, you’ll stitch a React front‑end to that contract, drop the site onto IPFS, and keep an eye on its health with a block explorer, similar to packing a suitcase and then using a luggage tracker.

Understand every stage from local dev to main‑net launch.
Write, test, and deploy a Solidity contract with Hardhat and verify it on a block explorer.
Connect a React UI, host on IPFS, and set up post‑launch monitoring.
Tools: Hardhat, Node.js, React, IPFS, Etherscan.
Tips: Keep your private keys in .env, use hardhat console for quick contract calls, and always run npm run lint before committing.

Cheat sheet:

npx hardhat compile – compile contracts
npx hardhat test – run tests
npx hardhat run scripts/deploy.js --network sepolia – deploy

This roadmap gives you the confidence to go from zero to a live dApp, ready for users in 2025.

What a dApp Actually Is (No Jargon)

A decentralized app (dApp) is a web‑based interface that talks to a public blockchain instead of a single, owned server. The “backend” lives as a smart contract, so every user sees the same code and the same state.

Imagine a vending machine on the corner. The machine’s logic – which button gives which snack – is hard‑wired inside a metal box. You can’t change the code on the fly, and anyone can walk up, drop a coin, and receive the item. In a dApp, the smart contract is that locked‑in logic, and the token you insert is a blockchain transaction.

Because the contract lives on a public chain, no company can shut it down or rewrite the rules. Everyone can verify the code, and the app continues to work as long as the network does.

When you follow a build dApp tutorial, you’ll be wiring a familiar front‑end (HTML/JS) to this immutable “vending machine” so users can press buttons, send tokens, and get results without ever trusting a central server.

Think of it like ordering food from a self‑service kiosk: you select, you pay, the kitchen (the blockchain) prepares the order, and you receive it – all without a human taking your money.

The 4 Mistakes Everyone Makes With dApp Development

Here’s what trips up most developers before they even see a live dApp.

Skipping local blockchain simulation and jumping straight to a testnet. It’s like ordering a meal without checking the menu first—you’ll end up with something you can’t afford or don’t like. Run hardhat node or ganache-cli locally, write unit tests, and catch bugs before any gas is spent.
Hard‑coding contract addresses or ABI in the front‑end. Imagine saving a Google Maps route with today’s traffic and then using it tomorrow; the directions are useless. Store addresses in a config file that you update after each deployment, and load the ABI dynamically.
Ignoring gas‑optimization. Deploying a contract without checking its size is like packing a suitcase without folding clothes—you’ll pay extra for the overweight baggage. Use tools like solc --optimize or hardhat‑gas‑reporter to keep fees manageable.
Forgetting to verify the contract on a block explorer. It’s the digital equivalent of publishing a research paper without citations; nobody trusts the results. Run npx hardhat verify --network sepolia DEPLOYED_ADDRESS "constructorArg" so others can read the source and debug easily.

Cheat sheet:

Run npm run test on a local node first.
Keep config.js for addresses/ABIs.
Enable optimizer in hardhat.config.js.
Verify with hardhat‑etherscan plugin.

Spot these pitfalls early and your build dApp tutorial will actually reach deployment.

How to Build a dApp: Step‑by‑Step

Install the tooling you’ll need. Grab the latest Node.js LTS, then run

npm install -g yarn

followed by

yarn add --dev hardhat

. Think of it like ordering a coffee: you pick the beans (Node), the milk (Yarn) and the cup (Hardhat) before you can brew.
Scaffold a fresh Hardhat workspace and add a simple contract. In an empty folder run

yarn hardhat init

, choose “create a basic sample project”, then replace contracts/Greeter.sol with a minimal ERC20 token or escrow contract. This is your suitcase: you pack the essential items (the Solidity file) before the journey.
Write unit tests and run them locally. Create test/Token.test.js using Mocha/Chai, then execute

yarn hardhat test

. The in‑memory network is like Google Maps offline: you can explore every route without leaving your desk.
Deploy to a public testnet. Add a Sepolia (or Polygon Amoy) RPC URL and a funded private key to .env, then run

yarn hardhat run scripts/deploy.js --network sepolia

. This step is the flight check‑in: you hand over your luggage (the compiled contract) to the airline (the testnet).
Verify the contract on the explorer. Install the verification plugin with

yarn add --dev @nomiclabs/hardhat-etherscan

and run

yarn hardhat verify --network sepolia  "ConstructorArg"

. Example: Alex, a fintech startup founder, runs the command and sees his token listed on Etherscan within minutes.

Export the ABI and make it available to the front‑end. Hardhat drops artifacts/contracts/Token.sol/Token.json; copy the abi array into src/abi.js. Then import it with import abi from "./abi" in your React code. This is like sharing a recipe so the kitchen (front‑end) knows what ingredients (functions) to use.

Build the UI and connect a wallet. Scaffold with Vite:

yarn create vite my-dapp --template react

, install wagmi and ethers, then add a ConnectButton that calls useAccount. Test a token transfer on Sepolia to confirm the flow works.
Pin the compiled front‑end to IPFS. Run

npm run build

then use Pinata:

pinata-cli add ./dist

. Keep the returned CID and point your ENS name to ipfs://. It’s like putting your restaurant’s menu on a billboard that anyone can see.

Set up a lightweight monitoring alert. Add the Tenderly plugin, configure a webhook, and watch for status: "reverted" events. Now you’ll receive a Slack ping the moment a user’s transaction fails, just like a traffic light warns you of a jam ahead.

Follow these nine moves and you’ll have a fully functional dApp ready for real users.

A Real Example: Token Launch for a Community Artist

Maya wants her fans to earn a token every time they engage, so she treats the launch like a pop‑up gallery opening.

Deploy to Polygon Amoy. She opens MetaMask, selects the Amoy testnet, and hits “Deploy” in Hardhat. The transaction is like ordering a coffee—MetaMask shows the price, she confirms, and the contract lands on the network.
Verify on Polygonscan. After deployment, Maya copies the contract address into Polygonscan and clicks “Verify & Publish.” This step is the receipt that proves the coffee was actually made.
Build the fan dashboard. Using Create‑React‑App she adds web3modal so fans can connect their wallets. The UI shows the ARTIST balance and a “Claim Airdrop” button. It feels like a simple Google Maps view—just a pin and a button.
Deploy the UI to IPFS. Maya runs pinata upload ./build, gets a CID, then points her ENS domain arttoken.eth to that CID. The site now lives on a decentralized file system, just like a suitcase stored in a locker.
Add monitoring. She creates a Tenderly project, adds the contract address, and sets an alert: “If any transaction fails, send a Slack webhook.” Now she gets a ping the moment a fan’s claim hits a snag.
Tool tip: Use npm i @web3modal/react ethers for wallet connection.

Cheat sheet:

Deploy: npx hardhat run scripts/deploy.js --network amoy
Verify: npx hardhat verify --network amoy CONTRACT_ADDRESS
Pin: pinata upload ./build --pin-name artist-dashboard

With these steps Maya’s token is live, fans can claim, and she’s instantly warned if anything goes wrong.

The Tools That Make This Easier

Grab the right toolbox and the rest of the process feels like ordering a pizza—you pick the toppings, the kitchen does the work, and you get a hot slice in minutes.

Hardhat (v3.x, 2025) – spins up a local blockchain, runs tests, and lets you script deployments. Think of it as a kitchen simulator where you can rehearse every dish before serving it live.
Vite + React – a lightning‑fast bundler that understands ES modules out of the box. It’s like using a modern espresso machine: you get a hot UI in seconds instead of waiting for a drip brew.
MetaMask Flask – the dev‑focused edition of the popular wallet, with custom RPC endpoints and snap support. Imagine a travel app that lets you pick any airline’s API on the fly.
Pinata (free tier) – pins your IPFS files so they stay reachable. It works like a reliable locker at a train station: you drop your assets in, and they’re there whenever a passenger needs them.
Tenderly (free sandbox) – streams live transaction data, fires alerts, and lets you step through failed calls. It’s the Google Maps of blockchain debugging, showing you exactly where the road broke.

Here’s a quick cheat sheet to get each tool running:

Hardhat init:

npm install --save-dev hardhat

then

npx hardhat init

Vite + React scaffold:

npm create vite@latest my-dapp --template react

MetaMask Flask install: find “MetaMask Flask” in the Chrome Web Store and enable “Custom RPC”.

Pinata upload: use their UI or run

pinata-cli add ./src/assets

Tenderly connect: sign up, link your Hardhat project via

tenderly login

and

tenderly fork

With these five tools in place, the rest of the build dApp tutorial becomes a smooth ride.

Quick Reference: dApp Build Cheat Sheet

Grab this list, follow it like a recipe, and you’ll have a live dApp without the usual dead ends.

Install the basics. Think of it as setting up a kitchen: run brew install node, then npm install -g yarn and npm install --save-dev hardhat. This gives you the stove, pots, and knives you need.
Write and test your contract. Draft your Solidity file, then spin up npx hardhat node – it’s the local test kitchen. Run npx hardhat test to taste‑test, just like a chef checks seasoning.
Deploy to Sepolia or Amoy. Use npx hardhat run scripts/deploy.js --network sepolia. It’s like sending your dish to a restaurant’s kitchen for the real crowd.
Verify on a block explorer. Paste the tx hash into Etherscan or Polygonscan, click “Verify Contract”, and copy the generated ABI. This is the menu that tells users what you serve.
Build the UI. Create a React app, then hook it up with wagmi and ethers.js. Imagine Maya, a fintech analyst, dragging a widget onto her dashboard; the code below shows how Maya connects:

import { useAccount, useConnect } from 'wagmi';
import { ethers } from 'ethers';
const { address } = useAccount();
const { connect } = useConnect();

Pin the built build/ folder to IPFS (think of it as uploading your menu to a global billboard).
Map an ENS name or custom domain to the IPFS CID so users can type mydapp.eth instead of a long hash.
Set up Tenderly alerts – it’s like a kitchen monitor that warns you when the oven is too hot.
Enable the Solidity optimizer in hardhat.config.js to trim gas, just as you’d trim excess fat from a recipe.
Node/Yarn/Hardhat: Install once, reuse forever.
Contract → Test: Write, run hardhat test, repeat until green.
Deploy: hardhat run --network sepolia or amoy.
Verify & ABI: Verify on explorer, copy ABI.
React + wagmi: Connect wallet, call contract.
IPFS + ENS: Pin UI, point domain.
Monitoring & Optimization: Tenderly + Solidity optimizer.

Keep this cheat sheet handy and you’ll cut the guesswork out of any build dApp tutorial.

What to Do Next

Grab the repo, get your hands dirty, and watch the magic happen.

Fork and test – clone the GitHub repo linked in this guide and point it at Sepolia. It’s like ordering a sample dish before committing to the full menu.

Run git clone https://github.com/yourname/dapp-demo.git
Switch networks with hardhat node --network sepolia
Deploy using npx hardhat run scripts/deploy.js --network sepolia

Add role‑based access – swap the simple owner check for OpenZeppelin AccessControl. Think of it as giving each team member a different key to the office.

Install with npm i @openzeppelin/contracts
Declare roles like bytes32 public constant ADMIN_ROLE = keccak256("ADMIN_ROLE");
Redeploy on Polygon mainnet after confirming gas estimates.

Scale up – integrate Arbitrum Nova for cheap, fast transactions and hook a Graph subgraph for live analytics. It’s like moving from a city bike to a highway express lane.

Deploy the contract to Arbitrum Nova via npx hardhat run scripts/deploy.js --network arbitrumNova
Create a subgraph in TheGraph that indexes Transfer events.
Embed the dashboard in your front‑end with a simple fetch call.
Cheat sheet: fork → access control → L2 + analytics.
Tools: Hardhat, OpenZeppelin, The Graph, Arbitrum Nova.

Which of these steps are you tackling first?

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Use AI to Generate and Test Code Simultaneously: A Practical Step‑by‑Step Guide

Abdullah Sheikh — Mon, 01 Jun 2026 12:03:34 +0000

Learn to write, run, and debug code in one AI‑powered workflow so you can ship features faster.

Before We Start: What You'll Walk Away With

When you finish this guide you’ll be able to fire a single AI prompt and watch code appear, compile, and get its unit tests executed without leaving your editor.

Think of it like ordering a meal where the kitchen not only cooks the dish but also sends you a live status update until it’s on your table. You’ll learn which prompts act like a reliable waiter and which need you to step in and correct the order.

By the end you’ll have a reproducible workflow that writes code and runs unit tests in one AI prompt. The whole process will be scripted so you can drop it into any project and get the same results, just like a recipe you can follow every time you cook.

You’ll also know how to craft prompts that consistently deliver clean, test‑ready code. This means you’ll spot the phrasing that yields a solid solution versus the wording that leaves you fixing syntax errors—similar to knowing which grocery list items keep you in the aisle and which send you back to the store.

Finally, you’ll be ready to embed the workflow into VS Code or GitHub Codespaces. The integration steps are as straightforward as plugging a USB drive into a laptop; once connected, the AI assistant becomes part of your development environment.

Reproducible workflow: one prompt → code + tests.
Prompt mastery: know the wording that works.
IDE integration: VS Code or Codespaces ready to go.

Grab a coffee, follow the steps, and you’ll stop flipping between editor and test runner forever.

What AI‑Generated, Auto‑Tested Code Actually Is (No Jargon)

AI‑assisted coding that blends a large‑language model’s generation with an automated test runner in the same session is what we call AI‑generated, auto‑tested code. The model writes a function, then immediately hands it off to a test framework that executes the relevant unit tests, reports failures, and even suggests fixes—all without you leaving the editor. The result is a single, continuous loop where code and verification live side by side, cutting out the manual copy‑paste and context‑switching that normally eats up your day.

Think of it like a smart kitchen robot. You tell it, “Make a veggie stir‑fry,” and it chops, sautés, and seasons the ingredients. Before the dish hits the plate, the robot takes a quick taste, adjusts the seasoning, and only then serves it to you. In the same way, the AI writes the code (chops the ingredients) and then instantly runs the tests (tastes the dish) so you get a dish—code—that’s already been verified as edible.

The 3 Mistakes Everyone Makes With AI Code Generation

Most developers hit a wall because they treat AI like a magic wand instead of a teammate.

Assuming the first output is production‑ready. It’s like ordering a pizza and expecting the kitchen to deliver a gourmet meal without a taste test. The model will give you syntactically correct code, but you still need a quick sanity check—run it, glance at the logs, and verify edge cases before you commit.
Forgetting to feed test scaffolding into the prompt. Imagine asking Google Maps for directions without specifying “avoid tolls.” If you don’t explicitly request unit tests or validation steps, the AI won’t guess them for you. Include a line like “generate Jest tests for this function” and you’ll get a usable test suite.
Over‑relying on a single tool. It’s like packing only one type of shoe for a trip and then discovering you need both hiking boots and dress shoes. Different AI services excel at different languages and debugging features; switching between, say, Claude for Python snippets and Gemini for TypeScript can fill the gaps.
Quick sanity check: run npm test or pytest -q immediately after generation.
Prompt template tip: “Write a function calcTax and include Mocha tests covering zero, negative, and large inputs.”
Tool mix‑and‑match: combine OpenAI for creative boilerplate with Tabnine for real‑time IDE suggestions.

Fix these habits and your AI code generation and testing loop will finally feel like a smooth coffee break, not a frantic juggling act.

How to Use AI to Generate and Test Code: Step‑By‑Step

Grab your IDE, fire up the AI extension, and let it do the heavy lifting.

Install an AI‑enabled IDE extension. Open your editor’s marketplace and add GitHub Copilot or Cursor. Think of it like adding a smart assistant to your kitchen—once it’s there, you just ask, it responds.
Create a minimal test file. In your project, add tests/test_feature.py with a simple pytest function that spells out the expected behavior. It’s like writing a receipt before you order food, so you know exactly what you should get.
Prompt the AI with the feature description and test location. In a comment or chat window, type something like: “Implement a function calculate_tax that matches tests/test_feature.py.” The AI now has both the recipe and the shopping list.
Review, run, and let the AI fix failures. After the code appears, save the file and execute pytest -q. If an assertion fails, ask the AI “Why is this test failing?” and apply its suggested patch. It’s similar to using Google Maps: you get a route, try it, and the app reroutes when you hit traffic.
Commit the passing code and codify the workflow. Once pytest reports green, run git add . && git commit -m "Add calculate_tax with AI‑generated tests". Then add a short section to CONTRIBUTING.md describing the prompt‑to‑test pattern so teammates can repeat it.
Tip: Keep the test file tiny; the AI fills in the rest.
Cheat sheet: pip install pytest → pytest -q → git push

Now you have a loop where code appears and validates itself—no more back‑and‑forth between IDE and test runner.

A Real Example: Building a URL‑Shortener Endpoint for Lina, a Front‑End Lead

Lina fires up her FastAPI repo and drops a test_shorten.py file with two pytest cases: one happy‑path URL and one that should be rejected.

She prompts the AI: “Create a FastAPI route /shorten that returns a short code and passes the attached tests.” The model spits out a new router with a helper that hashes the URL and a stub validator.
Lina runs pytest -q. The valid‑URL test passes, but the invalid‑URL test flunks because the endpoint never raises a HTTPException for bad input.
She asks the AI to “fix the failing test.” The model adds a simple if not url.startswith('http') check and returns a 400 response.
She runs pytest -q again – now both tests are green. A quick git add . && git commit -m "Add /shorten endpoint with validation" seals the work.
Prompt clarity: Include the exact test file in the prompt so the AI can see expectations.
Iterative fixes: Treat the AI like a pair programmer; each failing test is a cue for a small tweak.
One‑click push: When all tests pass, a single commit captures the entire AI code generation and testing loop.

The Tools That Make This Easier

Pick the right assistant and you’ll stop playing ping‑pong between your editor and the test runner.

GitHub Copilot – works inside VS Code, JetBrains, etc., and suggests code and matching unit tests as you type. Think of it like a sous‑chef who hands you the ingredients and the recipe at the same time. Free for verified students; teams pay per seat.
Cursor – a freemium AI IDE that bundles a terminal you can call from the chat window. You ask it to “run the failing test” and it executes npm test or pytest without you leaving the pane, similar to ordering food and having the kitchen send you updates.
TestGPT – a specialized LLM that turns a plain English description into a full test suite. It’s like Google Maps for test creation: give a destination (“verify login returns a token”) and it plots the route.
VS Code “Code Runner” extension – binds a single shortcut (Ctrl+Alt+N) to run the current file or test block. Imagine packing a suitcase and having one button that zips it closed.
GitHub Actions marketplace “AI‑Test‑Runner” – a workflow step that pulls generated code, runs npm test, and commits the results automatically. It automates the generate‑test‑commit loop in CI, so you can focus on the next feature.

These tools together give you a smooth AI code generation and testing experience without constantly swapping windows.

Quick Reference: AI‑Generated, Auto‑Tested Code Cheat Sheet

Think of the loop like ordering a coffee: you tell the barista (AI) exactly what you want, watch the brew (run tests), and tweak the recipe until it’s perfect.

✅ Define a tiny test first – write the smallest failing pytest that captures the desired behavior.
✅ Prompt with both feature + test path – include the test code and a brief description of the feature in the same request.
✅ Review, run pytest, ask AI to fix failures – let the AI suggest changes, apply them, and re‑run until the suite is green.
✅ Commit only after all tests pass – treat the passing test suite as the gatekeeper before any push.
✅ Store the prompt pattern in CONTRIBUTING.md for team reuse – a single source of truth so everyone follows the same recipe.
Prompt template: Feature: … | Test: … | Current code:
Toolchain: VS Code + GitHub Copilot for generation, pytest for execution, pre‑commit to enforce test pass before commit.
Quick sanity check: run pytest -q locally; if any test fails, ask AI “fix the failing test” before proceeding.
Team tip: add a #ai‑generated tag in the PR description so reviewers know the code came from the loop.
Fail‑fast principle: treat a red test as a “wrong order” – stop, correct, and only then move on.

Keep this cheat sheet on your desk and let AI handle the write‑test dance for you.

What to Do Next

Grab a few minutes and get the ball rolling—pick the action that matches your comfort level and try it right now.

Easy: Install GitHub Copilot or Cursor and fire the one‑line prompt in a sandbox repo. It’s like ordering a coffee with a single tap; you see the result instantly.
Medium: Pick an existing feature, write three unit tests, then ask the AI to generate the implementation. Think of it as packing a suitcase: you list what you need (tests), then let the AI fill the bag (code).
Hard: Add a CI step that runs the AI‑prompt‑to‑test cycle on every pull request. This is the “Google Maps” of your workflow—automatically rerouting code changes through validation without you lifting a finger.
Tip: Keep your prompts concise; start with “Write a function that…”.
Tip: Review AI output before merging; treat it as a draft, not final copy.
Tip: Log AI responses in .ai_log to track improvements over time.

💬 Got a tricky prompt that kept failing? Drop it in the comments and let the community help troubleshoot!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Build a Web Scraper with Node.js and Puppeteer in 8 Simple Steps

Abdullah Sheikh — Sun, 31 May 2026 12:03:27 +0000

Create a reliable, headless-browser scraper from scratch and extract data instantly

Before We Start: What You'll Walk Away With

When you finish this guide you’ll have a ready‑to‑run Node.js project that spins up Puppeteer without opening a browser window.

You’ll be comfortable picking out page elements, clicking “next” buttons, and dropping the scraped rows into a CSV or JSON file.

From there you can point the same code at any site that follows a similar layout and start pulling data in minutes.

Launch a headless browser – think of it as ordering a coffee and having the barista prepare it behind the counter while you watch the receipt appear on your phone.
Select and iterate over elements – like using Google Maps to pinpoint every café on a street, then moving from one pin to the next.
Export results – similar to packing a suitcase: you gather all the items (data) and neatly place them into a CSV or JSON bag for later use.
Install node and npm once; the rest of the setup stays the same.
Use await page.$$ to collect groups of elements, just as you’d scan a menu for all dessert options.
Handle pagination with a simple loop, mimicking the “next page” button you click when scrolling through product listings.

Cheat sheet

npm init -y – creates package.json
npm install puppeteer – pulls in the headless browser
await page.goto(url, {waitUntil: 'networkidle2'}) – lands you on the page
fs.writeFileSync('data.json', JSON.stringify(data, null, 2)) – saves the output

Grab the code, run it, and you’ll be pulling data before your coffee even cools.

What Web Scraping with Puppeteer Actually Is (No Jargon)

Puppeteer is a Node.js library that gives you programmatic control over Chrome or Chromium. It lets your code open a page, wait for content, click buttons, scroll, and then pull out the exact bits of HTML or text you need. In short, a web scraper node.js built with Puppeteer can act like a browser you would use manually, but it runs automatically and at scale.

Imagine a robot with a pair of hands sitting at a café table. You tell it, “Open the menu, click the dessert section, and copy the price of the tiramisu.” The robot follows the steps exactly, even if the site throws a pop‑up or requires a scroll. That’s what Puppeteer does for web pages— it mimics a real user’s actions, so sites that rely on JavaScript or dynamic loading still hand over their data.

Because the robot works inside a real browser, you don’t have to guess how the page renders; you see exactly what a human would see. This means fewer broken scrapers and less time fighting invisible APIs.

Got a list of product pages you need to scrape? Just script the robot to visit each URL, wait for the price element, and write it to a CSV. The same approach works for login flows, infinite scrolls, or extracting tables from dashboards.

Think of Puppeteer as your digital assistant that never gets tired, never clicks the wrong link, and always brings back the data you asked for.

The 3 Mistakes Everyone Makes With Puppeteer Scrapers

Most people hit a wall fast because they miss the three classic traps.

Ignoring headless detection defenses – Think of it like ordering food at a drive‑through with a clearly fake ID. The server spots the default Puppeteer fingerprint and refuses service. Spoof the user‑agent, hide the webdriver flag, and randomize screen size to blend in.
Over‑complicating selectors – It’s like trying to navigate a city with a handwritten map that marks every alley. Using brittle XPaths makes your scraper break on the next layout tweak. Stick to stable CSS selectors such as div.article > h2.title and test them with page.$$(selector) before committing.
Forgetting rate‑limiting – Imagine pounding the door of a house with a hammer; you’ll get shut out fast. Bombarding a site with rapid requests triggers bans. Add await page.waitForTimeout(Math.random()*2000+500) between actions and respect robots.txt where feasible.

Fix these and your web scraper node.js will stay alive longer.

How to Build a Web Scraper with Node.js and Puppeteer: Step‑by‑Step

Let’s get your scraper up and running in eight quick actions.

Open a terminal, run npm init -y to bootstrap a fresh Node project, then install Puppeteer with npm i puppeteer. Think of this as ordering the base ingredients before cooking.
Create a file scraper.js and add an async function, e.g. async function run(). Inside, launch a headless browser via puppeteer.launch() and open a new page with browser.newPage(). This is like turning the ignition and stepping into the driver’s seat.
Before you hit the road, set a realistic userAgent string and apply the puppeteer-extra-plugin-stealth plugin. It masks your scraper the way a disguise hides your identity.
Direct the page to your target URL with await page.goto(url, {waitUntil: 'networkidle2'}) and pause until a required selector appears using await page.waitForSelector('.product'). It’s like waiting for the traffic light to turn green.

Pull the data you need:

const items = await page.$$eval('.product', cards => 
  cards.map(card => ({
    title: card.querySelector('.title').innerText,
    price: card.querySelector('.price').innerText
  }))
);

Meet Alex, a market researcher who needs product names and prices. Alex runs the snippet above and gets an array of objects ready for analysis.
Handle pagination by looping while a “Next” button exists:

Check await page.$('.next').
If found, click it and repeat the extraction.
Break the loop when the button disappears.

This works like flipping pages in a book until you reach the end.
Save the gathered array to disk. For CSV:

const {Parser} = require('json2csv');
const parser = new Parser();
const csv = parser.parse(items);
require('fs').writeFileSync('data.csv', csv);

Or use fs.writeFileSync('data.json', JSON.stringify(items, null, 2)) for JSON.

Finally, close the browser with await browser.close(). Wrap the whole flow in a try/catch block to log errors and guarantee the browser shuts down even if something goes wrong.

Now you have a functional web scraper node.js ready to adapt to any site.

A Real Example: Scraping Product Prices from ExampleStore.com

Maya wants a script that wakes up each morning, grabs the latest prices from ExampleStore.com, and drops them into prices.csv—nothing more.

Install dependencies

npm i puppeteer csv-writer

Launch the browser

const browser = await puppeteer.launch({headless: true});

Open the target page

const page = await browser.newPage();
await page.goto('https://examplestore.com/category/widgets');

Extract product rows

const rows = await page.$$eval('.product-card', cards => 
  cards.map(c => ({
    name: c.querySelector('.title').innerText.trim(),
    price: c.querySelector('.price').innerText.replace('$','')
  }))
);

Handle pagination – click “Next” until it disappears.

while (await page.$('button.next')) {
  await Promise.all([
    page.click('button.next'),
    page.waitForNavigation({waitUntil: 'networkidle0'})
  ]);
  const more = await page.$$eval('.product-card', ...); // same extraction
  rows.push(...more);
}

Write to CSV

const createCsvWriter = require('csv-writer').createObjectCsvWriter;
const csvWriter = createCsvWriter({
  path: 'prices.csv',
  header: [{id:'name',title:'Product'},{id:'price',title:'Price'}]
});
await csvWriter.writeRecords(rows);

Close everything

await browser.close();

Run it daily – add an npm script and a cron entry.

"scripts": { "scrape": "node scraper.js" }

0 6 * * * cd /path/to/project && npm run scrape

Tip: Test the selector '.product-card .price' in Chrome DevTools before coding.
Tip: Use waitForSelector after each page change to avoid race conditions.
Tip: Keep prices.csv in a version‑controlled folder for easy diffing.

With these eight steps, Maya can treat her scraper like a coffee‑order bot—click, collect, and serve fresh data every day.

The Tools That Make This Easier

Grab these five freebies and you’ll spend less time hunting for tools and more time actually scraping.

Puppeteer (npm) – the headless‑browser engine that does the heavy lifting. Think of it as the kitchen appliance that cooks your data soup while you set the timer.
puppeteer‑extra‑stealth – a plugin that hides the browser’s “robot” badge. It’s like ordering a meal with a secret sauce that gets past the picky server.
csv‑writer – a tiny library that turns JavaScript objects into clean CSV files. Imagine packing a suitcase: each object is an item, csv-writer neatly folds them into rows.
VS Code with Prettier – your IDE plus an auto‑formatter. It’s the Google Maps of code layout: you type, Prettier reroutes you to the tidy‑est path.
GitHub Actions (free tier) – schedule your scraper to run nightly without a server. Like setting an alarm clock, it wakes up your script at the right hour.

Quick start commands:

npm install puppeteer puppeteer-extra-stealth csv-writer

const { createObjectCsvWriter } = require('csv-writer');
const csvWriter = createObjectCsvWriter({
  path: 'out.csv',
  header: [{id:'title',title:'Title'},{id:'price',title:'Price'}]
});

With these tools in place, the next step is wiring up the scraper logic.

Quick Reference: Web Scraper with Puppeteer Cheat Sheet

Grab this list and copy‑paste it when you spin up a new scraper.

Setup: npm init -y then npm i puppeteer puppeteer-extra-stealth csv-writer. Think of it like ordering the ingredients before you start cooking.

Launch browser:

async function launchBrowser() {
  return await puppeteer.launch({headless:true});
}

It’s the “turn the stove on” step.
Navigate:

await page.goto(URL, {waitUntil:'networkidle2'});

Like telling Google Maps to drive you to the exact address and wait until traffic clears.
Extract items:

const data = await page.$$eval('.item', els =>
  els.map(e => ({
    title: e.querySelector('.title').innerText,
    price: e.querySelector('.price').innerText
  }))
);

You’re picking the right dishes from a buffet and writing down their names and prices.
Paginate loop (example with Maya): Maya wants every product on a multi‑page catalog.

while (await page.$('.next')) {
  await Promise.all([
    page.click('.next'),
    page.waitForNavigation({waitUntil:'networkidle2'})
  ]);
  // repeat extraction here
}

She clicks “next” just like flipping pages in a book until there’s no more.
Save to CSV:

await csvWriter.writeRecords(data);

Think of it as packing the collected items into a suitcase for easy transport.
Cleanup:

try {
  // main logic
} catch (e) {
  console.error(e);
} finally {
  await browser.close();
}

Wrap everything in a try/catch so the scraper doesn’t leave the kitchen a mess.

Keep this cheat sheet handy and your web scraper node.js project will stay on autopilot.

What to Do Next

Grab the script you just wrote and give it a quick spin on your own machine.

Run it, tweak a selector, export a CSV. Think of it like ordering a coffee: you ask for exactly what you want, take a sip, then adjust the sugar if needed. Open a terminal and fire:

node scraper.js > output.csv

Open output.csv in Excel, confirm the columns line up, then change page.$('selector') until the data matches what you expect.

Make the scraper a little sneaky. For sites that start blocking bots, add the puppeteer-extra-plugin-stealth and sprinkle random delays between actions. It’s like slipping through a crowd by pausing to look at your phone—less likely to be noticed.
Install the plugin: npm i puppeteer-extra puppeteer-extra-plugin-stealth
Wrap actions with await page.waitForTimeout(Math.random()*3000+2000)
Deploy and schedule. Push the repo to GitHub, create a GitHub Action (or Railway job) that runs nightly. This is the suitcase‑packing stage: you’re ready to ship your scraper so it works without you hovering over the keyboard.
GitHub Action snippet:

name: Daily Scrape
on:
  schedule:
    - cron: '0 2 * * *'
jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: node scraper.js > data.csv
      - uses: actions/upload-artifact@v3
        with:
          name: daily-data
          path: data.csv

Now you have a web scraper node.js pipeline that runs on its own.

💬 Got a site that’s giving you trouble? Drop a comment with the URL and I’ll help you debug!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Deploy a Docker App to Any VPS in Under 10 Minutes

Abdullah Sheikh — Sat, 30 May 2026 12:05:19 +0000

Deploy your containerized app to a fresh VPS from zero to live in less than ten minutes

Before We Start: What You'll Walk Away With

By the end of this guide you’ll have a live Docker container humming on a brand‑new VPS, ready to serve traffic.

One command will spin up the whole stack, so you can treat deployment like ordering a pizza: you pick the toppings, hit “place order,” and the kitchen does the rest.

All the tools you’ll touch are free or have a generous freemium tier, meaning you can start without spending a dime.

Connect to a fresh VPS and install Docker in seconds.
Push your app’s image to a public registry.
Run a one‑line script that pulls the image, creates a container, and sets up a basic firewall.
Quick start: curl -sSL https://example.com/deploy.sh | bash
Rollback ready: The script saves the previous container ID, so you can revert with a single command.
Zero‑config networking: Uses the VPS’s default interface, no manual port forwarding needed.

This approach lets you skip the usual “install Docker, configure systemd, tweak iptables” maze and get straight to running code, just like using Google Maps to bypass traffic snarls.

When you finish, deploying a Docker app to VPS will feel as routine as packing a suitcase for a weekend trip—pick, zip, go.

What Deploying a Docker App Actually Is (No Jargon)

Deploying a Docker app means taking the container image you built on your laptop and copying it to a remote server so it runs there nonstop. The server pulls the image, starts a container, and keeps it alive until you decide to stop it.

Think of it like sending a pre‑packed lunch to a coworker’s desk. You’ve already prepared the sandwich, fruit, and napkin (the container). You drop the lunchbox at the office kitchen (the VPS), and a timer makes sure the food stays warm all day. No one has to re‑make the sandwich on site, and the lunch stays ready whenever it’s needed.

Copy the image with docker push or scp – like handing the lunchbox to the mailroom.
Start the container with docker run -d – like placing the lunch on a heated plate.
Monitor it with docker ps – like checking that the timer is still ticking.

When you deploy Docker app to VPS you’re basically automating that hand‑off so the app is up and running the moment the server boots, without manual re‑assembly.

The 3 Mistakes Everyone Makes With Docker VPS Deployments

Most of the time you spend fixing the same things over and over, and it slows you down.

Installing Docker manually on every new server – think of it like ordering a pizza for each friend individually instead of using the party‑size menu. You repeat the same steps, risk a typo, and waste precious minutes. Automate the install with a single script or a cloud‑init snippet, and you’ll get a clean, identical Docker engine on every VPS instantly.
Forgetting to open the right ports – it’s the digital equivalent of locking the front door while leaving the back window open. Your container runs, but the world can’t reach it. Make a habit of adding ufw allow 80/tcp or the specific port your app uses right after the Docker install, then verify with curl http://your‑vps-ip:PORT.
Running the container without a process manager – imagine packing a suitcase without tying the straps; the contents tumble out at the first bump. Without something like systemd or Docker’s built‑in restart policy, a reboot sends your app back to sleep. Use --restart unless-stopped in docker run or create a simple service file so the container boots up automatically.

Fix these three, and you’ll stop chasing ghosts when you deploy Docker app to VPS.

How to Deploy a Docker App: Step-by-Step

Grab a fresh Ubuntu 22.04 VPS and get it ready in five moves.

Pick a provider and launch the server. Think of it like ordering a pizza – you choose the crust (provider) and size (Ubuntu 22.04), then wait for delivery (instance).
SSH in and install Docker + firewall. Run a single command that does the heavy lifting.

ssh root@your-vps-ip
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
sudo ufw allow OpenSSH
sudo ufw allow 2375/tcp
sudo ufw --force enable

Pull your image. This is like grabbing a ready‑made sandwich from the fridge.

docker pull yourusername/yourapp:latest

Create a systemd service. The service is the “alarm clock” that wakes your container on boot.

sudo tee /etc/systemd/system/yourapp.service > /dev/null  /dev/null **UFW (Uncomplicated Firewall)** – Ubuntu’s built‑in guardrail works like a simple gate with three keys. One liner to open the necessary ports:

bash
ufw allow 22 && ufw allow 80 && ufw allow 443 && ufw enable


- **GitHub Actions** – Automate the build pipeline the way a coffee machine brews on schedule. A workflow file builds your Docker image on every push and pushes it to Docker Hub, so the VPS always pulls the latest tag.

With these tools in your toolbox, **deploy Docker app to VPS** becomes a repeatable, low‑effort routine.

## Quick Reference: Docker App Deployment Cheat Sheet

Grab a coffee and copy‑paste this list to get your Docker app running on any VPS in under ten minutes.

- **Prep** – Pick a VPS, write down its IP, and make sure your Docker image is pushed to a registry. Think of it like writing down an address before ordering a pizza.

**Install** – Run the one‑liner that drops Docker on the box and opens the needed ports:

bash
curl -fsSL https://get.docker.com | sh && ufw allow 22 && ufw allow 80 && ufw enable


**Pull Image** – Grab the image you built earlier:

bash
docker pull yourrepo/yourapp:tag


- **Service File** – Create `/etc/systemd/system/yourapp.service` with an `ExecStart` that runs the container. It’s the same as setting a “Start when I turn the key” rule for a car.

**Enable** – Start the service and let it survive reboots. For example, *Alex* a solo founder, runs:

bash
systemctl enable --now yourapp


**Verify** – Confirm the container is alive and reachable:

bash
docker ps
curl http://YOUR_IP:PORT


- **Tip** – Keep the service file minimal; only expose the ports you need.

- **Tip** – Use `--restart unless-stopped` in the `ExecStart` line to auto‑recover from crashes.

- **Tip** – If you change the image tag, just repeat steps 2–5; systemd will pick up the new container.

Copy, paste, and you’re live – that’s all it takes to *deploy Docker app to VPS*.

## What to Do Next

Kick the tires on your new setup by running a tiny “hello‑world” container and seeing it survive a reboot.

**Easy** – SSH into the VPS, pull `nginx:alpine`, and start it in detached mode:

bash
docker run -d --name hello -p 80:80 nginx:alpine


Then reboot the server (`sudo reboot`) and check `docker ps`. If the container is still running, you’ve proven the deployment works—just like ordering a coffee, getting it, and watching it stay hot.
**Medium** – Wire a CI pipeline so every push builds a fresh image and pushes it to your registry. In GitHub Actions add a job that runs on `push` to `main`:

yaml
name: Build & Deploy
on:
push:
branches: [main]
jobs:
docker:
runs-on: ubuntu-latest
steps:

uses: actions/checkout@v3
name: Log in to Docker Hub run: echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USER" --password-stdin
name: Build and push run: | docker build -t $DOCKER_USER/myapp:${{ github.sha }} . docker push $DOCKER_USER/myapp:${{ github.sha }}


Now each merge automatically refreshes the image—think of it as a kitchen that restocks ingredients every time a new order comes in.
**Hard** – Let infrastructure-as-code handle the whole stack. With Pulumi (TypeScript) you can spin up a VPS, open ports, install Docker, and deploy the container in one file:

ts
import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";

const droplet = new digitalocean.Droplet("app", {
region: "nyc3",
size: "s-1vcpu-1gb",
image: "ubuntu-22-04-x64",
userData: #!/bin/bash apt-get update && apt-get install -y docker.io docker run -d --restart unless-stopped -p 80:80 myrepo/myapp:latest,
});

export const ip = droplet.ipv4Address;




Running `pulumi up` provisions everything, like using Google Maps to draw the entire road trip before you leave the house.

- **Cheat sheet**: keep `docker-compose.yml` ready for local testing; copy it to the VPS with `scp` when you need a quick redeploy.

- **Tip**: enable `--restart unless-stopped` on every container so reboots never drop your services.

Got a different stack or cloud provider? Drop a comment and let us know what tweaks you needed!

---

---

## About the Author

**[Abdullah Sheikh](https://abdullah-sheikh.com/)** is the Founder & CEO at [Exteed](https://exteed.com/), where he leads a team of skilled developers specializing in [Web2](https://abdullah-sheikh.com/) and [Web3 applications](https://abdullah-sheikh.com/), [Custom Smart Contracts](https://abdullah-sheikh.com/), and [Blockchain solutions](https://abdullah-sheikh.com/).

With 6+ years of experience, Abdullah has built [CRMs](https://abdullah-sheikh.com/), [Crypto Wallets](https://abdullah-sheikh.com/), [DeFi Exchanges](https://abdullah-sheikh.com/), [E-Commerce Stores](https://abdullah-sheikh.com/), [HIPAA Compliant EMR Systems](https://abdullah-sheikh.com/), and [AI-powered systems](https://abdullah-sheikh.com/) that drive business efficiency and innovation.

His expertise spans [Blockchain](https://abdullah-sheikh.com/), [Crypto & Tokenomics](https://abdullah-sheikh.com/), [Artificial Intelligence](https://abdullah-sheikh.com/), and [Web Applications](https://abdullah-sheikh.com/); building reliable and smooth web apps that fit the client’s goals and requirements.

📧 [info@abdullah-sheikh.com](mailto:info@abdullah-sheikh.com) · 🔗 [LinkedIn](https://www.linkedin.com/in/-abdullah-sheikh/) · 🌐 [abdullah-sheikh.com](https://abdullah-sheikh.com/)

How to Get Free SSL Certificates with Let’s Encrypt in Minutes

Abdullah Sheikh — Fri, 29 May 2026 12:04:29 +0000

A practical, step‑by‑step guide that lets you secure any website for free without a single line of code you can’t understand

Before We Start: What You'll Walk Away With

By the end of this guide you’ll know exactly why Let’s Encrypt is trusted by browsers worldwide—think of it as the “gold star” you get after a restaurant inspection.

You’ll be able to click through an installation wizard, copy a few commands, and confirm that HTTPS is live on Apache, Nginx, or a static‑site host—just like ordering a pizza and watching it arrive at your door.

Finally, you’ll walk away with a one‑page cheat‑sheet you can reuse for any new domain, so you never have to start from scratch again.

Understand the basic idea behind Let’s Encrypt and why it’s trusted.
Install, verify, and set up auto‑renew for a free SSL certificate on Apache, Nginx, or a static‑site host.
Grab a cheat‑sheet that lets you repeat the process for any future domain.
Why it matters: Browsers treat Let’s Encrypt certificates as fully valid, so visitors won’t see warning signs.
What you need: Access to your server’s command line or a control panel that supports custom scripts.
What you’ll avoid: Expired certificates and the headache of manual renewals.

Keep this list handy; it’s the roadmap you’ll follow step by step.

What Let’s Encrypt Actually Is (No Jargon)

Let’s Encrypt is a nonprofit Certificate Authority that hands out domain‑validated SSL certificates for free. It talks to your server through an automated protocol called ACME, so you never have to call a support desk or fill out paperwork.

Imagine a digital passport office that instantly stamps your website’s “identity” the moment you show proof you own the domain. Just as a government ID convinces a barista you’re old enough to order coffee, the let’s encrypt ssl certificate convinces browsers that your site is trustworthy, letting visitors see the padlock icon without a hitch.

The 3 Mistakes Everyone Makes With Let’s Encrypt

Here are the three blunders that turn a smooth let's encrypt ssl setup into a headache.

Forgetting to open port 80/443. The validation server can’t reach your site if those doors are closed, just like a delivery driver blocked by a locked gate. Check your firewall or cloud panel and allow inbound traffic on both ports before you start the challenge.
Using a single‑use certificate and skipping auto‑renew. It’s like ordering a fresh loaf of bread and then never buying another—your site will suddenly serve an expired cert. Enable a cron job or systemd timer that runs certbot renew twice a day so renewal happens automatically.
Running the client as root and overwriting existing configs. Imagine giving a rookie chef unrestricted access to your kitchen; they might replace a trusted recipe with something else. Use a dedicated user or the --deploy-hook option instead, and back up any config files before the client writes to them.

Fix these, and your HTTPS will stay up without a hitch.

How to Get Free SSL With Let’s Encrypt: Step‑by‑Step

Grab a cup and follow these clicks‑and‑type steps; you’ll have HTTPS up faster than ordering a coffee.

Pick a client. Certbot is the default choice, like using the standard app store instead of hunting obscure tools.

Install the client on your server. Use the package manager that matches your OS:

apt install certbot for Debian/Ubuntu
yum install certbot for CentOS/RHEL
brew install certbot for macOS
Or pull the Docker image: docker run -it --rm certbot/certbot
Make sure your domain’s DNS points to the server and that ports 80 and 443 are open. Think of DNS as the address book and the ports as the front‑door keys; both must work for visitors to get in.

Run the appropriate Certbot command for your web server. For Nginx, it looks like this:

certbot --nginx -d example.com -d www.example.com

Replace example.com with your own domain. Certbot will talk to Let’s Encrypt SSL, prove you own the site, and install the certificate automatically.

Verify the result. Open https://yourdomain.com in a browser or use an online SSL checker to confirm the lock icon appears.

Enable auto‑renewal so the cert doesn’t expire. Test the renewal process first:

certbot renew --dry-run

If the dry run succeeds, the built‑in cron job will keep the certificate fresh.

That’s it—your site now enjoys free, trusted encryption.

A Real Example: Securing a Small‑Business Blog on Nginx

Maya, who runs a charming bakery, just wants her WordPress blog to show the padlock without hunting through menus.

First she updates the package list and pulls in Certbot with Nginx support:

sudo apt update && sudo apt install certbot python3-certbot-nginx

Next she asks Certbot to fetch a certificate for both the bare domain and the www subdomain:

sudo certbot --nginx -d bakerydelights.com -d www.bakerydelights.com

Prompt 1: “Enter email?” – Maya types her business email.
Prompt 2: “Agree to Terms of Service?” – she hits Y.
Prompt 3: “Redirect HTTP to HTTPS?” – she chooses the automatic redirect, like telling a waiter to bring the soup before the salad.

Certbot now edits the server blocks in /etc/nginx/sites-available/bakerydelights.com, injects the ssl_certificate directives, reloads Nginx, and prints a success message:

Congratulations! Your certificate and chain have been saved at:
  /etc/letsencrypt/live/bakerydelights.com/fullchain.pem
Your certificate will expire on 2026-08-14.

To be sure the auto‑renew will work, Maya runs a dry‑run:

sudo certbot renew --dry-run

The command simulates a renewal, confirming the timer and Nginx reload are set.
If anything goes wrong, Certbot shows a clear error, letting Maya fix it before the real expiration.
Tip: Schedule a weekly check with systemctl status certbot.timer to verify the renewal timer is active.
Cheat sheet: sudo certbot --nginx -d example.com -d www.example.com handles the whole process for most Nginx sites.

With those few clicks and commands, Maya’s bakery blog now serves pages over HTTPS, and the padlock stays put.

The Tools That Make This Easier

Grab the right helper and the whole process feels like ordering a pizza—pick a menu, confirm the toppings, and the delivery shows up on your doorstep.

Certbot – The official let’s encrypt ssl client. It talks directly to the CA, configures Apache or Nginx, and renews automatically. Think of it as the restaurant’s “set‑it‑and‑forget‑it” menu.
acme.sh – A tiny Bash script that runs on even the smallest VPS. It’s like a pocket‑knife: you can slice, dice, and tighten certs without any heavy dependencies.
Win‑Acme – Windows‑only client with a GUI and CLI for IIS. Imagine a GPS that plots the fastest route on a Windows road map, guiding you step‑by‑step.
SSL Labs’ SSL Test – Free online scanner that grades your HTTPS setup. It’s the “Google Maps” of SSL, showing you exactly where the bumps are.
Docker Let’s Encrypt Nginx Proxy Companion – Container‑aware tool that watches your Docker services and fetches fresh certs for you. Picture it as a suitcase that automatically packs the right clothes for each trip.

Pick one that matches your environment, run the installer, and let the tool handle the heavy lifting.

With any of these, the next time you check your site you’ll see the green padlock without a second thought.

Quick Reference: Let’s Encrypt Cheat Sheet

Grab this cheat sheet and follow it like a recipe—you’ll have a valid Let’s Encrypt SSL in minutes.

🔹 Install client – Think of it as ordering a pizza: you need the right app before you can place the order. Run

apt install certbot

(or pick acme.sh if you prefer a lighter tool).

🔹 Open ports 80/443 – Like making sure the front door is unlocked for delivery. Adjust your firewall so traffic can reach the web server.

🔹 Run certbot – This is the checkout step. Execute

certbot --nginx -d yourdomain.com -d www.yourdomain.com

and let the wizard handle the rest.

🔹 Verify installation – Just as you’d double‑check a map route, visit SSL Labs and enter your domain to confirm everything is green.

🔹 Auto‑renew – Think of setting a calendar reminder so you never forget to restock. Test the renewal with

certbot renew --dry-run

. If your system doesn’t already schedule a cron job, add one that runs daily.

🔹 Common pitfalls – Closed ports are like a blocked driveway; DNS missing is like a wrong address; no auto‑renew is like forgetting to pay a subscription. Check each before you start.

Example – Maria runs a small boutique site on a VPS. After installing certbot, she opens ports, runs the command, and sets a cron entry

0 3 * * * /usr/bin/certbot renew --quiet

. Her site stays HTTPS forever without manual touch.

What to Do Next

Grab the cheat‑sheet you just built and put it somewhere everyone can see.

Wiki entry – copy the three‑step “Install → Verify → Renew” list into your team Confluence, Notion, or internal README.
One‑click link – add a hyperlink to the live https://yourdomain.com/.well-known/acme-challenge/ page so newcomers can test HTTPS instantly.
Tag it – label the page “SSL” so a quick search brings it up.

Next, set up a simple watchdog so you know before a certificate expires.

Sign up at UptimeRobot (free tier works).
Create a monitor pointing to https://yourdomain.com and choose the “HTTPS (SSL)” check.
Configure an email or Slack alert for “Down” events – think of it like a kitchen timer that pings you before the soup boils over.

Finally, if you’re comfortable with pipelines, automate renewal.

Add a step in your CI script that runs certbot renew --quiet after a successful build.
Store the renewed certificate in your artifact repository or secret manager.
Deploy the fresh files to every server – similar to packing a suitcase once and sending the same set of clothes to multiple houses.

Doing these three actions moves you from “I have HTTPS today” to “HTTPS stays alive without me lifting a finger.”

💬 Got stuck or discovered a new tip? Drop a comment below – I’d love to hear your story!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Use GitHub Copilot to Code 3x Faster

Abdullah Sheikh — Thu, 28 May 2026 12:03:56 +0000

Follow this step‑by‑step guide and start writing code three times faster with GitHub Copilot

Before We Start: What You'll Walk Away With

After you finish this guide you’ll have a Copilot setup that feels as quick as ordering a favorite coffee with a single tap, so you never waste minutes staring at half‑written suggestions.

You’ll master a handful of prompt patterns and keyboard shortcuts that cut the time you spend typing and correcting code by up to three times, much like using Google Maps to skip traffic snarls instead of guessing the route.

Finally, you’ll walk away with a repeatable, language‑agnostic workflow that lets you pack a new project’s logic into your editor the way you’d zip a suitcase for a weekend trip—everything in its place, ready to go.

Install and configure Copilot for optimal speed
Use prompt patterns and keyboard shortcuts that cut coding time by up to 3×
Apply a repeatable workflow to any language or project
Turn on Inline Suggestions and set suggestion delay to 0ms so ideas appear instantly.
Map Ctrl+Space to “accept line” and Alt+] to “skip suggestion,” keeping the rhythm steady.
Adopt the “Ask‑Then‑Refine” prompt: start with a brief comment, let Copilot draft, then tweak with a focused question.

Keep this cheat sheet handy; it’s the quick‑reference you’ll reach for whenever you fire up a new repo.

What GitHub Copilot Actually Is (No Jargon)

GitHub Copilot is an AI‑powered autocomplete that watches the code you type and suggests the next line or whole block, matching the surrounding context. It pulls patterns from millions of public repositories, so the suggestions feel like they were written by someone who already knows the project.

Think of it like the predictive text on your smartphone, but instead of finishing a single word, it drafts complete sentences—only those sentences are lines of code. You start a function, and Copilot hands you a ready‑made implementation, just as your phone might finish a message you’ve begun.

The 4 Mistakes Everyone Makes With GitHub Copilot

Most developers hit a wall with GitHub Copilot speed because they repeat the same four missteps.

Ignoring the settings panel and leaving Copilot on default, which yields generic suggestions. It’s like ordering a burger without telling the server you’re vegetarian – you’ll get something, but it won’t suit you.
Accepting every suggestion without reviewing, leading to bugs and wasted time. Think of it as a GPS that reroutes you automatically; you still need to glance at the map before turning.
Using Copilot only for single‑line completions instead of whole‑function generation. It’s comparable to packing a suitcase one item at a time instead of laying out the whole outfit first – you miss the chance to see the bigger picture.
Not leveraging context‑aware comments or ‘// TODO:’ prompts to steer the model. This is like sending a vague text to a friend and expecting a detailed answer; the clearer the prompt, the better the response.
Fix #1: Open the settings (Ctrl+, → Extensions → Copilot) and enable “Suggest whole lines” and “Enable advanced context”.
Fix #2: Treat each suggestion as a draft; run eslint or your test suite before hitting Tab.
Fix #3: Ask for full functions with prompts like “// generate a fetch wrapper for the API”.
Fix #4: Write precise comments such as “// TODO: sort users by lastLogin descending”.

Fix these, and Copilot stops being a hindrance and starts pulling its weight.

How to Use GitHub Copilot: Step‑by‑Step

Ready to turn Copilot into a real pair‑programmer? Follow these exact actions.

Install the GitHub Copilot extension from the VS Code marketplace, then sign in with your GitHub account. It’s like adding a power‑tool to your toolbox – you need to plug it in before you can use it.
Open Settings → Extensions → GitHub Copilot. Set Inline Suggestion Delay to 0.2 s and turn Enable Telemetry off. This makes suggestions appear almost instantly, just as a restaurant’s “fast lane” gets you your dish quicker.
At the top of a new file, type a comment that serves as a Prompt Template. Example: // Write a React component that fetches user data. Think of it as writing a quick order slip for a chef.
Trigger a full‑function suggestion with Ctrl + Enter (Windows) or Cmd + Enter (Mac). It’s like pressing “Enter” on a GPS to get the whole route at once.
Review the code. Press Tab to accept, or Arrow‑Down to scroll through alternatives. This mirrors flipping through menu options until you find the perfect dish.
If the suggestion isn’t clear, run Copilot: Explain. The AI will break down its reasoning, similar to a friend explaining why they chose a particular travel path.
Save your favorite prompts in a personal copilot-prompts.md file. Keep it handy like a cheat‑sheet for recurring orders.
Tip: Keep the prompt concise; extra fluff slows the AI down.
Tip: Use language‑specific comment markers (# for Python, // for JavaScript) to guide the model.
Cheat sheet: Ctrl+Enter / Cmd+Enter → trigger, Tab → accept, Arrow‑Down → next, Copilot: Explain → understand

A Real Example: Building a CRUD API with Node.js

Maya, a junior backend developer, needs a quick CRUD endpoint for her task‑manager app. She opens tasks.js in VS Code, remembers the prompt pattern that gave her the best GitHub Copilot speed, and gets to work.

She types a comment that describes the whole operation:

// Express route for creating a task, with validation

She hits Ctrl + Enter. Copilot instantly suggests a complete handler, including express.Router(), a Joi schema, and a try/catch block.
She accepts the suggestion, runs the existing Jest test suite, and tweaks the Joi rules to match the spec. The test passes on the first run.
She repeats the same three‑step rhythm for the remaining routes—GET, PUT, DELETE—changing only the comment text.
Write a clear comment that states the intent.
Trigger Copilot with Ctrl + Enter (or ⌘ + Enter on macOS).
Run tests immediately; adjust validation or response shape as needed.

By treating each route as a “menu item” she orders from Copilot, Maya cuts the implementation time from roughly four hours to about ninety minutes. The code looks like this:

router.post('/tasks', async (req, res) => {
  const { error, value } = taskSchema.validate(req.body);
  if (error) return res.status(400).json({ error: error.details[0].message });
  const task = await Task.create(value);
  res.status(201).json(task);
});

With the pattern locked down, adding new endpoints feels as fast as ordering a coffee—just describe what you want, confirm, and move on.

The Tools That Make This Easier

Grab the tools that turn Copilot from a novelty into a real speed‑boost.

VS Code (free) – Think of it as the kitchen where you’ll prep everything. The built‑in Copilot extension plugs in instantly, and the marketplace offers dozens of extensions that keep the workflow smooth.
GitHub Copilot – The AI chef. The free tier handles most suggestions; upgrade to $10/mo Pro for faster response and priority access to the newest models.
Tabnine (free tier) – A backup sous‑chef. Install it alongside Copilot to compare autocomplete quality on the same file; you’ll spot which engine writes cleaner snippets for your stack.
CodeQL (free on GitHub) – Your safety inspector. Run a quick scan on any Copilot‑generated pull request; it flags common security smells the way a metal detector spots hidden risks in luggage.
Promptable (freemium) – A personal recipe book. Save, version, and share the exact prompts that consistently give you the fastest results, then pull them into VS Code with a single click.

With these five in place, you’ve set up a workflow that feels like ordering your favorite meal through a well‑tuned app: you select, confirm, and the result arrives ready to eat.

Next, we’ll stitch them together into a repeatable daily routine.

Quick Reference: GitHub Copilot Cheat Sheet

Grab this cheat sheet and keep it open while you code – it’s the quick‑order menu for turning Copilot into a real pair‑programmer.

Install Copilot → VS Code extension. Think of it like adding a new app to your phone; you just click “Install” and you’re ready to use it.
Settings: delay 0.2 s, telemetry off, suggestion count 3. It’s like setting a GPS to give you three route options instantly, without the background data collection.
Prompt pattern: // [Action] + brief description. Write it like a restaurant order: “// Create function fetchData – get JSON from API.”
Shortcut: Ctrl/Cmd + Enter = full‑function suggestion. Press it the way you’d tap “Order now” on a delivery app to get the whole dish delivered.
Review → Tab = accept, Arrow Down = next suggestion. Imagine packing a suitcase: you look at each item (suggestion), slide the one you like into the bag (Tab), or move to the next (Arrow Down). For example, Alex types a loop, hits Arrow Down, and picks the third suggestion that uses map instead of for.
Use ‘Explain’ to validate logic. It works like asking a friend to read your recipe and point out missing steps.
Save favorite prompts in copilot-prompts.md. Treat the file like a sticky note pad for your go‑to orders.
Speed tip: Keep the delay short and suggestion count low; Copilot then feels like a fast‑food drive‑through rather than a sit‑down service.
Safety tip: Turn telemetry off to avoid extra data traffic, just like switching off location services.
Maintenance tip: Review saved prompts monthly; prune anything you no longer use.

Bookmark this list and let GitHub Copilot speed become second nature.

What to Do Next

Pick one of these steps and get the speed boost rolling right now.

Easy: Open VS Code, install the GitHub Copilot extension, and type a simple “Hello World” prompt. It’s like ordering a coffee — you tell the barista what you want and you get it instantly.
Medium: Create a file called copilot-prompts.md and jot down five snippets you reach for daily—API call, loop, validation, etc. Think of it as packing a suitcase: you pre‑place the items you know you’ll need so you don’t waste time rummaging through the closet.
Hard: Build a small feature, such as a login form, using only Copilot‑generated code. When it runs, fire up CodeQL to scan for issues. This is the “Google Maps” test: let Copilot plot the route, then double‑check the directions yourself.

Give one of these a try and watch your GitHub Copilot speed climb.

💬 Got a Copilot tip that saved you time? Drop it in the comments – we’d love to hear it!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

How to Launch Apps on Coolify: The Free Heroku Alternative Step‑by‑Step

Abdullah Sheikh — Wed, 27 May 2026 12:03:55 +0000

Deploy, manage, and scale your web apps on Coolify without spending a dime

Before We Start: What You'll Walk Away With

By the end of this tutorial you’ll be able to spin up a Node.js or Python project on Coolify in under ten minutes, treat the platform’s free‑tier quirks like a familiar restaurant menu, and keep your app’s domain, environment variables, and logs under control without ever leaving the dashboard.

First you’ll grasp the core concepts that Coolify builds on, and you’ll know exactly where the free tier draws the line—think of it as checking the size limits before ordering a large pizza.

Next you’ll push a repo from GitHub, watch the deployment pipeline run, and have a live URL in seconds—just like dropping a pin on Google Maps and getting directions instantly.

Finally you’ll learn to add a custom domain, set secret keys, and read logs directly in the UI, similar to packing a suitcase where every item has its own compartment.

Core concepts & limits – understand projects, services, and the free tier’s CPU, RAM, and container caps.
One‑click deployment – clone a GitHub repo, select runtime (Node.js or Python), and hit “Deploy”.
Management basics – add domains, edit .env variables, and tail logs without SSH.
Tip: Keep your app under 512 MB RAM to stay within the free tier; it’s like ordering a small coffee instead of a venti.
Tool: Use the “Metrics” tab to watch resource usage in real time, just as you’d glance at a speedometer while driving.
Cheat sheet: Deploy → Settings → Logs are the three tabs you’ll visit most often.

Ready to see how easy it really is? Let’s jump into the first steps.

What Coolify Actually Is (No Jargon)

Coolify is an open‑source platform‑as‑a‑service that runs on Docker, giving you a Heroku‑style web UI without the price tag. It abstracts the container plumbing so you can push code, click “Deploy” and let the system spin up a build container, run migrations, and expose a live endpoint.

Imagine a self‑service kitchen: you walk in, drop your recipe card (the source code) on the counter, and the kitchen staff (Coolify) handles chopping, cooking, and plating. You don’t need to pre‑heat ovens or clean the pans – the platform does it all and hands you a ready‑to‑eat dish (your running app).

Because it’s built on Docker, every app lives in its own isolated container, just like each dish gets its own plate. The UI shows logs, env variables, and scaling knobs, so you stay in control without writing docker commands yourself.

Free tier limits are clear: one project, 500 MB RAM, and a 5‑minute build timeout. That’s enough for prototypes, side projects, or a quick demo. If you outgrow those numbers, the open‑source nature lets you spin up your own instance on any cloud.

In a Coolify tutorial you’ll see exactly how to connect a Git repo, set a build command, and launch the app – all from the same dashboard you’d use on Heroku.

Bottom line: Coolify gives you the comfort of a managed PaaS while keeping you in the driver’s seat.

The 3 Mistakes Everyone Makes With Coolify

Here are the three traps that bite most people the first time they try a Coolify tutorial.

Skipping the Docker Compose setup is like ordering a burger without specifying the toppings—you’ll get a mess on the plate. Without a proper docker-compose.yml, Coolify can’t resolve service links, and the build aborts with cryptic errors.

Run coolify init to generate a starter file.
Copy the example from the docs and adjust the image names.

Ignoring the free‑tier container quota is similar to trying to drive a car past the fuel gauge’s empty line; you’ll stall without any warning. Coolify silently drops deployments once you exceed the 1 GB limit, leaving you staring at a “deployment failed” badge.

Check usage in the dashboard before each push.
Keep images under 500MB to stay safe.

Misconfiguring environment variables feels like packing a suitcase without a passport—everything looks fine until you reach the border and get turned away. A missing DATABASE_URL or a typo in NODE_ENV crashes the app at runtime.

Define vars in the “Environment” tab, not just in .env.
Use the “Test” button to validate each entry before saving.

Spot these early and your first Coolify launch will feel like a smooth ride.

How to Deploy on Coolify: Step‑by‑Step

Think of the deployment as ordering a pizza: you pick the crust, add toppings, give the address, and wait for the doorbell.

Sign up for a free Coolify account, then click Connect Repository and link the GitHub repo that holds your code.
Pick a project template that matches your stack – Node.js, Python, Ruby, etc. It’s like choosing the pizza size before you add toppings.

Create a .coolify.yml at the root of the repo. Inside, define build: commands and the port your app listens on. Example for a Node app:

build:
  command: npm install && npm run build
port: 3000

Open the Coolify UI, go to Environment Variables, and add keys such as DATABASE_URL or API_KEY. This step is the sauce that makes everything taste right.
Hit the Deploy button. Coolify will spin up a Docker container and stream the build log – watch it like a live cooking show.
When the log finishes, copy the auto‑generated https://your-app.coolify.app URL and open it. If you have a custom domain, add it under Domain Settings and point your DNS records to the provided CNAME.
Tip: Keep the .coolify.yml simple at first; you can add advanced caching later.

Cheat sheet:

.coolify.yml – build commands + port
Env vars – set in UI, not in code
Deploy – one click, watch logs

Now your app lives on Coolify, ready for traffic.

A Real Example: Deploying a Next.js Blog for Maya the Startup Founder

Maya’s marketing blog is a simple Next.js site, but she wants it live on Coolify without paying a cent.

Create a GitHub repo called maya-blog and push her local next project.
Add a .coolify.yml at the repo root so Coolify knows how to build.
Define the environment variable NEXT_PUBLIC_API_URL in Coolify’s UI.
When the free‑tier warning appears, click “Ignore limit” to keep the app running.

Here’s what Maya drops into .coolify.yml:

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    env:
      - NEXT_PUBLIC_API_URL=https://api.maya.com

GitHub repo – Think of it like ordering a pizza: you pick the place (GitHub) and the toppings (your code), then Coolify delivers.
.coolify.yml – This is the “recipe card” that tells Coolify how to cook your app, just like a recipe tells a chef which ingredients to use.
Environment variable – Similar to setting your GPS destination, it points the Next.js app to the right API endpoint.
Free tier limit warning – It’s the “parking meter” notice; you can acknowledge it and keep the spot for a while.

After committing the file, Maya links the repo in Coolify, clicks “Deploy,” and watches the logs turn green. Within minutes, her blog is live at https://maya-blog.coolify.app.

Cheat sheet

Repo URL: https://github.com/yourname/maya-blog
Env key: NEXT_PUBLIC_API_URL
Ignore limit button: bottom of the “Resources” tab

That’s the whole “Next.js on Coolify” recipe, ready for Maya’s next launch.

The Tools That Make This Easier

Grab the tools you already trust, then treat Coolify like a well‑stocked kitchen where every ingredient has its place.

GitHub – Your code lives here, and a simple push can fire a Coolify build. Think of it like ordering food online; the moment you click “order,” the kitchen starts preparing your dish.
VS Code – Edit the .coolify.yml file with the Docker linting extension. It’s the kitchen’s timer, warning you before a dish burns.
Postman – Shoot requests at your freshly deployed API to confirm it’s tasty. Like tasting a soup before serving, it catches flavor issues early.
Cloudflare DNS – Point your custom domain to Coolify’s load balancer on the free plan. It’s the GPS that directs diners straight to your restaurant’s front door.
Loki + Grafana – Stack these open‑source tools if you want logs served on a dashboard. Imagine a pantry inventory system that tells you exactly what’s running low.

Cheat sheet

git push origin main – triggers the deploy.
code .coolify.yml – opens the config in VS Code.
Run a GET request in Postman to /health after deployment.
Set an CNAME in Cloudflare pointing to app.coolify.io.
Add Loki as a log driver in Docker, then view logs in Grafana.

With these five free‑or‑freemium tools, the Coolify tutorial feels less like a hurdle and more like a smooth ride.

Quick Reference: Coolify Deployment Cheat Sheet

Grab this cheat sheet when you’re ready to push code, and you’ll never wonder which button to click next.

✅ Create a free Coolify account – think of it like signing up for a new coffee loyalty card; you just need an email and then link your GitHub so Coolify can pull your repo.
✅ Add a .coolify.yml file – this is your recipe card. Define build, start, and ports just like you’d list ingredients, cooking time, and serving size.
✅ Set environment variables, Deploy, Verify URL – picture setting the GPS on your phone, hitting “go,” then checking the map’s endpoint to make sure you arrived.
❌ Avoid more than 2 containers on the free tier – the free plan is a compact car; squeeze in a third container and you’ll hit the mileage limit.
🔧 Use VS Code + Docker lint – Alex, a junior dev, lints his Dockerfile in VS Code before committing, catching syntax errors the same way a spell‑checker catches typos.
🔧 Test with Postman – treat each endpoint like a menu item; send a request, taste the response, and confirm the flavor matches expectations.
Start: Sign up → link GitHub.
Configure: Add .coolify.yml (build, start, ports).
Secure: Set env vars in the dashboard.
Deploy: Click “Deploy” and watch the logs.
Validate: Open the provided URL, run a quick Postman check.
Stay within limits: Keep containers ≤ 2, monitor usage in the “Resources” tab.
Iterate fast: Edit locally, push to GitHub, hit “Redeploy.”

Keep this list open and you’ll breeze through any Coolify tutorial.

What to Do Next

First, lock in the process by pushing a tiny “Hello World” app – it’s the “order a coffee” of Coolify.

Deploy a sample app

- Clone a starter repo, run `git push coolify master`, and watch the build log.

- Verify the live URL returns `Hello World`. If it works, you’ve mastered the basic workflow.

Add a custom domain and enable HTTPS

- Think of this like swapping a restaurant’s generic table number for your own vanity plate.

- In Coolify, go to *Settings → Domains*, add `example.com`, then point the DNS A record to the provided IP.

- Flip the *Enable Cloudflare* switch; Coolify will auto‑issue a TLS certificate.

- Test with `curl -I https://example.com` – you should see `200 OK` and `strict-transport-security` headers.

Set up Loki/Grafana monitoring

- Imagine packing a suitcase for a weekend trip: you need the right tools, not every gadget.

- Deploy the `loki` and `grafana` services from Coolify’s marketplace.

- In each app’s *Env* tab, set `LOKI_URL=http://loki:3100` and enable `LOG_LEVEL=info`.

- Open Grafana, add Loki as a data source, then create a dashboard that tracks request latency across all your services.

These three steps take you from a fresh deployment to a production‑ready stack with visibility into every request.

💬 What’s the biggest hurdle you hit when moving from Heroku to Coolify?

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com