DEV Community: Ali

Bypass Bot Detection with Python Selenium. 🤖

Ali — Thu, 21 Aug 2025 07:29:13 +0000

Ali

Nov 4 '24

Bypassing Bot Detection Software with Selenium in Python

#selenium #python #webdev #security

Comments 2

2 min read

ADA Title II Adopting The WCAG 2.1 AA Standard

Ali — Thu, 14 Aug 2025 07:40:58 +0000

The DOJ has updated Title II of the ADA(Americans with Disabilities Act) affecting local and state government institutions, including public universities on April 24 2024.

Which basically enforces the adoption of WCAG 2.1 AA as the standard for web accessibility.

Which essentially means that Web content and mobile applications provided by local and state government will have to implement extra accessibility measures.

These measures aims at making content more accessible to disabled users, including those with mild disabilities such a low vision and photosensitivity.

Key features of WCAG 2.1

Screen Reader Compatibility

Requiring Web content to be readable by screen readers, which means to respect page structure for navigation, context and appropriate headings (H1,H2,H3).

Clear and descriptive link texts and buttons, so instead of having a link say "Click here", it's purpose will have to be conveyed clearly and explicitly such as "Learn more about WCAG 2.1".

Buttons on the other hand will be required to have an aria-label if the button text isn't accessible or doesn't convey a clear purpose. That way screen readers will be able to announce it clearly.

Keyboard Navigation

Web content will have to be keyboard friendly, meaning you can navigate the content with a keyboard instead of mouse. Ensuring that all interactive elements can be selected with the Tab key or other keyboard shortcuts.

Which includes:

Buttons
Links
Inputs
Dropdown menus
Modal dialogs
Custom Widgets

Which will require websites to use HTML elements that can be navigated and selected via a keyboard. Such as <button>, <a> and <input>, and avoid using <div> and <span> elements for interactive components.

Image Alternative Text

This will require every image to have an alternative text, with some exceptions for images that are purely decorative.

This can be added by using the alt attribute to an image.

When to Comply ?

Public entities that serve a population of more than 50,000 must comply by the deadline which is April 24 2026.

Public university are considered as serving the whole population of their current state, which makes all university in all 50 states obligated to comply with the new WCAG 2.1 accessibility standard.

How to Send Emails in Python Using Gmail’s Free SMTP Mail Server API

Ali — Mon, 11 Nov 2024 14:22:56 +0000

This is the easiest way you can start sending emails with Python using only two libraries, smtplib and email.

We will be using Gmail’s free RESTful API in this example.

Here's the code

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

message = MIMEMultipart()
message["To"] = 'To line here.'
message["From"] = 'From line here.'
message["Subject"] = 'Subject line here.'

title = '<b> Title line here. </b>'
messageText = MIMEText('''Message body goes here.''','html')
message.attach(messageText)

email = 'Your Gmail address here.'
password = 'Your app password here.'

server = smtplib.SMTP('smtp.gmail.com:587')
server.ehlo('Gmail')
server.starttls()
server.login(email,password)
fromaddr = 'From line here.'
toaddrs  = 'Address you send to.'
server.sendmail(fromaddr,toaddrs,message.as_string())

server.quit()

How the code works ?

First, let’s import smtplib then MIMEMultipart and MIMEText from email.mime.multipart and email.mime.text respectively:

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

We then call MIMEMultiPart() and instantiate it to the variable Message. We then give a string value to each variable which are Message, To, From and Subject, it should look like this:

Message = MIMEMultipart()
Message["To"] = 'To line here.'
Message["From"] = 'From line here.'
Message["Subject"] = 'Subject line here.'

We'll also give a title to our email through the title variable and add a message through MIMEText() and instantiate it to the variable messageText and finally attach our email message to our main variable Message using the attach() function, it should look like this:

title = '<b> Title line here. </b>'
messageText = MIMEText('''Message body goes here.''','html')
Message.attach(messageText)

Let’s add our Gmail address and our App password, click the link here if you don’t know how to get one:

email = 'Your Gmail address here.'
password = 'Your App password here.'

Then we'll have to connect to Gmail's SMTP server, we'll do that using the smtplib library and we'll need two variables which are the server we want to connect to and the port which are smtp.gmail.com and 587 respectively.

Using the smtplib library we'll be calling the SMTP function and add the server and port variables as its arguments, it should look like this:

smtplib.SMTP('smtp.gmail.com:587') (don't forget the : between them)

We'll then instantiate it to the variable server:

server = smtplib.SMTP('smtp.gmail.com:587')

We then add an ehlo statement using server.ehlo(‘Gmail’), this should be your domain name, this is useful when sending en email to another mail server that supports ESMTP, let’s keep it simple and just put Gmail:

server.ehlo('Gmail')

Let’s also start the TLS protocol with server.starttls(), this tells the mail server we want to send our email through a secure connection:

server.starttls()

We'll then login to the mail sever using this line:

server.login(email,password)

Let’s add a from address and to address(es), using fromaddr and toaddrs respectively:

fromaddr = 'Your Gmail address.'
toaddrs  = 'Destination address.'

Finally, we send our email using server.sendmail(fromaddr,toaddrs,message.as_string()) and we close our connection to the mail server using server.quit():

server.sendmail(fromaddr,toaddrs,message.as_string())
server.quit()

Save that in a file called send_email.py, open a Terminal if you're on Linux or a Command Prompt if you're on Windows and launch it using python send_email.py and you should see this:

If nothing happens, well good news it's working!

You should have received an email to your destination email(s), here's what I got:

Sending email using Python and Gmail's free SMTP server is the easiest way you can start sending email within your Python code, this is exactly what I've done when I built my first online busines ever. You can read more about it here.

How to Scrape Product Pages(Etsy, Amazon, Ebay) in Python Using Scrapy and Request Callbacks

Ali — Tue, 05 Nov 2024 14:08:29 +0000

(I'm not condoning anything illegal, this is for educational purposes only)

Introduction

Scrapy is one of the best web scraping frameworks in Python, it's easy to use, fast and packed with features.

But what if you we wanted to scrape multiple pages recursively? Such as product pages.

Well the easiest way is by adding a simple callback to a Request function.

Here’s a code snippet inside a Scrapy project crawling a website with listed products such as Amazon, eBay and Etsy:

def parse(self, response):
    links = response.css('a.s-item__link::attr(href)').getall()

    for link in links:
        yield Request(url=link, callback=self.parse_item)

    next_page = response.css('a.pagination___next.icon-link::attr(href)').get()
    if next_page:
        print('Next page: %s' % next_page)
        yield Request(url=next_page, callback=self.parse)

def parse_item(self, response):
    title = response.xpath('//h1[@class="x-item-title___mainTitle"]/span/text()').get()
    price = response.xpath('//span[@id="prcIsum"]/text()').get()

    yield {'title':title,
           'price':price}

How it works?

First, it gets the links of each items listed on a products page using this line of code:

links = response.css('a.s-item__link::attr(href)').getall()

It then loops through each one of those links, sends a request with the yield statement and does a callback to our parse_item function with self.parse_item:

for link in links:
     yield Request(url=link,callback=self.parse_item)

Inside the parse_item function, it gets the title and price of the item and returns them with the yield statement:

def parse_item(self, response):
      title = response.xpath('//h1[@class="x-item-title__mainTitle"]/span/text()').get()
      price = response.xpath('//span[@id="prcIsum"]/text()').get()

      yield {'title':title,
             'price':price}

Since the parse function is still running, our code then gets the link to the next page, requests it with a yield statement and does a callback to our parse function with self.parse and starts all over again:

next_page = response.css('a.pagination__next.icon-link::attr(href)').get()
  if next_page:
        print('Next page:%s' % next_page)
        yield Request(url=next_page,callback=self.parse)

Conclusion

There you go, it's that simple!

Scraping product pages recursively with Scrapy can be implemented as easily as adding a callback to a Request function.

Bypassing Bot Detection Software with Selenium in Python

Ali — Mon, 04 Nov 2024 08:10:26 +0000

(Use at your own risk, I'm not condoning anything illegal)

Websites such as Amazon uses bot detection software such as Imperva and FingerprintJS to block automated tools from interacting with their website. And they're rightly doing so to keep web scrapers and hackers away from stealing their precious data.

If they only knew...

Those security controls are fairly easy to bypass if you know what you're doing.

There are a few ways we can bypass those security controls:

Rotate proxies
Using cookies
Rotate User Agents and HTTP Headers
Remote Javascript signatures in your webdriver(we'll explore this one soon! 😎)
Avoiding patterns when interacting with websites

If you're using Selenium and you're getting blocked by automation software.

There's a little trick we can implement that can save us the hassle of setting up rotating proxies, headers or user agents.

Disable Selenium Automation Flag

But here's another simpler way that might do the trick - disabling Selenium's automation flag.

Selenium by default has automation flags enabled which might block you from accessing websites using anti bot software.

To bypass the block you'll have to disable those automation flags in your Selenium code, let's see how we can do that using a simple one liner.

First, let's call the webdriver.Chromeoptions() function and instantiate it to a variable called options.

This enable's us to add custom options to our webdriver:

options = webdriver.ChromeOptions

Now, let’s add the option disabling the automation flag by adding the following line to our Selenium code:

options.add_argument("--disable-blink-features=AutomationControlled")

This simple one-liner disables the automation flag and tells the website that you're not a bot.

Easiest Way to Handle Drop Down Menus in Python Using Selenium ?

Ali — Thu, 31 Oct 2024 05:58:47 +0000

Introduction

Selenium has a neat way to handle drown down menus by using the Select function.

For this example, we will testing it out on:
https://app.endtest.io/guides/docs/how-to-test-dropdowns/

Importing the Select class

First let’s import the Select function.

from selenium.webdriver.support.select import Select

Finding the Drop Down Element

Now let's call the drop down by using its ID, which is pets and name its instance drop_down.

drop_down = driver.find_element_by_id('pets')

Selecting the drop down

Now we've got the drop down selected and will name its instance drop.

drop = Select(drop_down)

There are multiple ways we can select values in a drop down menu, either by index, value or visible text.

Selecting by Index

drop.select_by_index(2)

Selecting by Value

drop.select_by_value('cat')

Selecting by Visible Text

drop.select_by_visible_text("Dog")

How I Bootstrapped My SaaS Business With $0

Ali — Wed, 30 Oct 2024 03:41:10 +0000

Let’s go back in time...

Date: November 21th, 2018

Location: Montreal, Canada

The Idea

I’m thinking about startup ideas, looking at my 13-inch Thinkpad screen with a medium sized coffee in a Tim Horton’s coffee shop at around 10:00 PM. It’s a cold snowy day, snow flakes are slowly falling to the ground, as I’m watching outside through a window facing the street, a beautiful sight. Being a night owl, I always loved working at night, there’s something about night time work that helps me focus. Probably because everyone is sleeping and there’s less distraction.

So here I am thinking about starting a business without spending a dime, I challenged myself.

Tim Horton’s coffee, the best (I'm joking)

At that time, I knew enough of Python to scrape websites, from data extraction to cleaning and loading to a database, that’s the easy part. If something doesn’t work, Google it. Fixing issues in the business world is the hardest part. You can’t Google that and find a straigth forward solution. That’s what business is all about, fixing problems and getting paid for it.

I knew that presenting the right data to the right person would be profitable, that was my vantage point.

We all consume data, like when we look at weather forecasts before going outside, if it’s going to rain, we bring an umbrella. That’s actionable data, information that changes our behavior, it helps us act the right way at the right time. I was looking for a continuous source of data that was monetizable and actionable. I wasn’t interested in building a machine learning model, just a simple data reporting service. I believe there’s plenty of untapped opportunities in the AaaS(Analytics as a Service) domain, going straigth to automated models is a bit too far fetched for what I was looking for.

Keeping it simple, not everyone needs machine learning models and it doesn’t solve everything.

At that time, I was also looking for new ways to make money, and I found out about phone flipping. You’re basically buying used phones and reselling them for profit on sites like eBay. I knew a couple guys in the US that were doing some good money doing that. I joined a couple phone reselling groups on Facebook, and I noticed that many were struggling when estimating phone resale prices. One way that some did it was by going on eBay, type the phone model and estimate the price from the first couple listings. That solved the problem, but still tiresome if you got a lot of phones to sell.

So I thought to myself, why not automate that ?

That’s what I did.

Coding it

Bots running on my laptop turned Linux server… At a coffee shop. lol
Everything was done in Python using Scrapy and running automatically on my laptop with Debian Linux, Thanks to crontab. Sleepless nights and a visit to the hospital due to a panic attack right in the middle of the night. Sacrifices, definitely not a walk in the park, man.

At one point, I was thinking about giving up.

I can’t do that, I’ve put in so many hours in that project

I told myself.

The hardest part by far, was trying to get quality data. Separating each phones (i.e cleaning and separating the iPhone X 64G from iPhone XS 256G), that took me weeks to get it right. I ended up many times with false positives/negatives.

One of the techniques I used to remove bad data, was to look at the title used for the listing. Listings with the keyword “fake”, “not working”, “bad esn” or “iCloud” were separated from the rest.

The naming of my business was the least of my issues at that point, but I ended going with Phone Flipping.

MVP Release

Finally, after months of hard work, I’ve finally got something out and running without issues. I had around 70 bots pulling data from listing websites, then transforming, cleaning and storing it into a database. A fully automated data pipeline.

My bots were pulling data everyday at 8:00 AM from those listing websites. I was rotating my proxies and user-agents for each request. I also added random intervals between requests, to avoid getting detected by anti-bot systems.

I was sending the pricing reports every Friday morning at 8:00 AM by email.

I was using Gmail’s free SMTP server, and some python code to automate the whole thing.

That was my mail server! I really wanted to spend 0$.

On the marketing side, I did everything organically, no paid ads. All I did is post content in Facebook Groups and have an opt-in form on a website to get emails. Why emails ? Because they’re super easy to get, and it’s one of the best marketing channel in terms of ROI.

You want free eBay phone pricing updates ?

We need an email

As simple as that.

There’s one YouTuber that helped me a lot, Kish Israni. He let me post my charts every week in his Facebook Group. A lot of my users were from his following. I leveraged his distribution channel. So shoutout to him !

If you have no idea what’s a distribution channel, it just means a way to distribute your content to a wider audience.

In those Facebook groups, I was helping phone resellers with price estimation and I was answering their questions. Having a close relationship with potential customers is crucial for growth when you’re bootstrapped. Showing authenticity, empathy and understanding is key.

I focused on giving value first, I was not trying to sell anything!

2–3 days later, after sharing free pricing sheets on multiple phone reselling Groups on Facebook.

I got dozens of subscribers messaging me. My inbox was full of interested users wondering what the heck I was doing and where I got the data. In a couple months, it grew to hundreds of users on the free and paid plan. I was charging between 20$–40$ per month.

I still remember the first person wanting to join the paid plan, a store manager. He payed me 40$ a month. A surreal moment for me, seeing that 40$ in my PayPal account.

Here’s the email I sent inviting free user to join the Pro plan:

I played a lot with the pricing to get it right. Should I go with a trial or not?, should I have a single plan or not? A lot of testing, trials and errors.

Before monetizing, I sent out a Google survey to get an idea of how many were interested in a paid plan. Around 20% were interested in giving money to me.

It was weird, because I felt like an imposter:

Do they really like my service ?

I kept thinking.

I couldn’t believe it, people were really giving me money for a PDF file I was sending every Friday morning.

Everything was automated, the only thing that needed some maintenance was the crawling bots that were pulling data from eBay listings. Because as soon eBay was updating their DOM structure, 90% of the time my bots would crash and require me to extract the data again using either CSS or XPATH. That was a real pain in the ass, at one point, I was staying up the whole night debugging my bots and going to sleep when the sun was rising at 5 AM.

This is how it worked:

Pull pricing data from eBay using Scrapy
Store data in text files
Clean data and analyze
Plot data in graphs inside Jupyter Notebook
Generate PDF reports from Jupyter Notebook
Send PDF reports using Gmail's SMTP Mail server

Conclusion

So this is how I launched my first online business, my first 1$ online. The things I’ve learned in the process of building my business is worth more than the money I’ve made. Especially in marketing and sales, I’ve learned a ton !

I ended taking down Phone Flipping, because during the pandemic my paid users stopped paying. I was also dealing with some serious mental health issues. The pandemic had a huge impact on me, which led me to stop everything.

It was unsustainable.

Overall this is how it worked:

There's a couple things I’ve learned in the journey:

Bringing the right information to the right person is monetizable.
People will pay you for automating small parts of their business.
Keep it simple, find solutions to simple problems.
Getting quality data is the hardest part, quality over quantity.
Network, network, network
Have a close relationship with your clients and have a feedback loop.
Just do it ! Like Nike.

How To Find Zombie Hosts With Nmap?

Ali — Tue, 29 Oct 2024 15:52:59 +0000

For educational purposes only

If you are familiar with Nmap, you probably already know that scanning networks can easily be detected by firewals and IDS (Intrusion Detection Systems). Which can blow your cover and get your IP or proxy blacklisted.

There are multiple ways to avoid that to happen. Here we’ll explore a method called idle scan, also known as a ‘zombie scan’. Which is an advanced port scanning method used to avoid detection by having a ‘zombie’ between you and the machine you’re scanning.

This is a blind scanning method used to scan for open ports on the target machine without ever sending a single packet from the attackers IP address.

The scanning works in five stages.

How it works ?

The attacker sends an unrequested SYN/ACK to the zombie machine. Which the zombie didn't expect, so it responds by sending a RST to the attacker and by doing so reveals its IP ID.
The attacker sends a forged SYN to the target making it look like it comes from the zombie.
The target then sends a unrequested SYN/ACK to the zombie.
The zombie then responds with a RST which increments its IP ID. Indicating that the port is open, if the port is closed, the target sends a RST to the zombie and the IP ID will not be incremented.
The attacker repeats step 1 and verifies if the IP ID has been incremented, if so, the port scanned is open, if not the port is closed.

How do you find a zombie ?

Nmap NSE (Nmap Scripting Engine) comes with a handy script called IPIDSEQ. Which we can use to scan for random zombies on the internet. Using the following script, Nmap will scan port 80 of 100 random hosts across the internet.

nmap -p 80 -script ipidseq -iR 100

The hardest part in a zombie scan is to find a suitable machine to act as a zombie as sometimes you can end up with false positives. The candidate host shouldn’t be getting too much traffic because its IP ID won’t be accurate and predictable. Printers, Windows and old Linux servers work fine.

How to launch a idle scan with Nmap ?

Here is a bare bones zombie scan using Nmap:

nmap -sI <zombie host> -p <ports to scan> <target>

Conclusion

We’ve covered the infamous idle scan, incredibly useful for host discovery. It is one of the best ways to scan hosts without getting detected. But keep in mind that some modern IDS’s can detect when a idle scan is happening on their network.

Be careful guys!

This is for educational purposes only. Scanning network without explicit permission can land you in jail. I am not responsible for any damages incurred!

How to Encrypt a USB drive Using Cryptsetup with LUKS on Linux

Ali — Tue, 29 Oct 2024 14:54:57 +0000

Do you even encrypt ?

Bitlocker is Windows default disk encryption software.

Cryptsetup on the other is Linux go-to for full disk encryption. It comes preinstalled on the major Linux distributions out there. It supports multiple encryption formats, including Bitlocker’s.

But our focus here will be on the LUKS(Linux Unified Key Setup) format, which is the standard in terms of Linux disk encryption.

Cryptsetup offers plenty of options when encrypting drives.

But before doing anything, let’s run a benchmark to test our computer’s encryption/decryption speed. This will tell us the best algorithm to use to encrypt our USB drive.

Let’s run cryptsetup benchmark in the terminal.

Here's the output:

We will ignore the first test and jump right to the second part. Here the aes-xts algorithm has the fastest encryption and decryption speed overall for our machine. We’ll go with the last one in the list with a Key size of 512-Bit. (it's highlighted in blue)

Here are the settings I use when encrypting:

sudo luksFormat (usb drive path) -c aes-xts-plain64 — key-size 512 — hash sha512 — iter-time 50000
Let’s explore each one of these parameters:

c: Ciphering algorithm used for encryption.
key-size: Key size used for encryption.
hash: Hashing method used on the passphrase.
iter-time: Number of milliseconds to process the passphrase.

Keep in mind that running that command will format and fully encrypt your drive so be careful and make sure to backup your files before doing anything and you’re good to go.

Oh! I almost forgot, always use long pass-phrases to encrypt. This will ensure that your drive doesn’t get brute-forced easily.

How I Built My First Online Business With Zero Dollars

Ali — Sat, 23 Sep 2023 14:22:15 +0000

Back in 2018, I was brainstorming ideas for a business.

I already knew how to code in Python and knew the data stack pretty well.

At that time I was playing around with the Open Exoplanet Catalog dataset trying to find habitable planets. (Yes, I really did lol)

Then a few weeks passed and I found out about a side hustle called "Phone Flipping". Which consists of buying phones for a low price and reselling them at a higher price on sites like eBay.

I then decided to dig more so I joined a couple phone resale groups on Facebook.

And I noticed something, a recurring question from phone resellers.

A lot of them were asking others for help when appraising and estimating phone prices.

Asking questions such as:

Hey! I just bought an iPhone, how much should I resell it for?

I found what I was looking for.

A problem, now I needed a solution.

I knew that eBay was the main marketplace where iPhones were getting sold in the US.

And since I was into data science and I knew web scraping, that's exactly what I did.

I started pulling phone pricing data every single day of the week and running analysis on it inside a Jupyter Notebook on my Thinkpad X230.

I built an MVP in a couple weeks and released it to the world. Everything was built using Python and It was running on a local Debian Linux server in my bedroom.

This is how it worked:

Pull pricing data from eBay using Scrapy
Store data in text files
Clean data and analyze
Plot data in graphs inside Jupyter Notebook
Generate PDF reports from Jupyter Notebook
Send PDF reports using Gmail's SMTP Mail server

I had a free and paid monthly plan which I was charging $40 per month.

My first client was a Sprint store manager, when I saw that $40 deposited on my Paypal account I was the happiest man alive.

I then started sharing the pricing charts on multiple Facebook Groups every Friday morning.

I ended up with hundreds of free and paid users.

Then in 2020, the pandemic happened and I lost all my paying customers.

I took down everything.

I couldn't sustain it, it was too overwhelming for me.

I lost my business but I gained a ton of experience.

Full article here: https://dev.to/thetanweerali/how-i-bootstrapped-my-saas-business-with-0-38dg