Building an async OSINT scanner: from a local script to a distributable tool

Joseph Avatiants — Sun, 15 Nov 2026 15:59:28 +0000

A while ago I shared my utility Web-Recon — an asynchronous scanner that probes
hundreds of domains at high speed and, when it hits a network anomaly or a dead
host, automatically pulls a deep WHOIS profile and stores everything in SQLite.

After that first post, the repo came alive: views went up, and people started
cloning the script into their Kali Linux setups. That gave me a bold idea:
"Why not apply to have Web-Recon included in the official Kali Linux
distribution, so it can be installed with a simple sudo apt install?"

I registered on the Offensive Security bug tracker (the team behind Kali) and
opened a ticket in the "New Tool Request" category. The response from the
repository admin (daniruiz) came a couple of weeks later, and it made me
completely rethink the project's architecture:

"Hi! Thanks for your submission. We physically can't package every infosec
tool in the world, so we give strict priority to utilities that have wider
adoption and community usage. Good luck with your project!"

Translated into engineering terms, it meant: "Your software looks like a cool
local script, but Kali Linux is an industrial-grade distribution. Turn the
script into a proper package, solve the WAF blocking problem, add automated
tests, show that the project is alive, and we'll come back to this
conversation."

Challenge accepted. Over the following weeks, Web-Recon evolved from v0.5.0 to
release candidate v0.6.2. In this article I'll walk through how I tamed the
async power, built a three-tier WAF evasion pipeline, and set up CI/CD
pipelines in the cloud.

Chapter 1: Taming async power (semaphores and delays)

In its first iteration, Web-Recon ran on a reactive asyncio + aiohttp
core. The throughput was enormous: the engine threw hundreds of network
requests at once. But in real pentesting, this led to a disaster. When trying
to mass-probe a large corporate perimeter, the tool produced such a dense
barrage of simultaneous connections that it turned into an unintentional DoS
attack.

The result? Weak web resources went down, and advanced Web Application
Firewalls (WAFs) like Cloudflare instantly triggered on the abnormal rate and
permanently banned the scanner's IP address, returning batches of 403 Forbidden errors.

To make the network assault controlled, I had to introduce hardware-level
concurrency control — asyncio.Semaphore. Now, no matter how many thousands of
domains are in the target file, the semaphore acts as a strict turnstile: only
a fixed number of requests (the -c flag, default 50) are in flight at any
given time.

But limiting thread count alone isn't enough — WAFs can detect a stable
packet rate. To break that pattern, I added non-blocking delays based on
asyncio.sleep() (the -d flag). Unlike the standard time.sleep(), which
freezes the entire OS thread and hangs the UI, an async pause gently puts a
specific task to sleep, returning CPU time to the Event Loop. As a result, the
Rich progress bar keeps spinning smoothly while requests fly out with
carefully irregular pauses, mimicking organic traffic:

parser.add_argument("-d", "--delay", type=float,help="Delay in seconds between checks (default: 0.5)",default=0.5)

Chapter 2: The great disguise (cascading User-Agent rotation)

The second mine the scanner hit was static HTTP request headers. Initially, the code had a single hardcoded headers dictionary posing as Chrome 120 on Windows. But when the server's defense sees 100 requests arriving in one second, each carrying an identical User-Agent string down to the last character, the protection system instantly triggers a block.

In v0.6.2 I removed the global static headers entirely and designed a three-tier browser fingerprint masking pipeline based on cascading fallbacks:

Tier 1 (Custom file): A professional pentester can feed the tool their own pre-warmed list of fingerprints via the -A / --UserAgents flag. The script reads it in isolation and loads it into memory:

if args.UserAgents:
    if os.path.exists(args.UserAgents):
        if not args.quiet:
            console.print(f"[bold green]► Loading custom User-Agents from file: {args.UserAgents}[/bold green]")

        with open(args.UserAgents, 'r', encoding='utf-8') as file:
            for line in file:
                if line.strip():
                    loaded_agents.append(line.strip())
    else:
        console.print(f"[bold red]✖ Error: User-Agents file '{args.UserAgents}' does not exist.[/bold red]")
        sys.exit(1)

Tier 2 (Automatic live update): If the flag is empty, Web-Recon doesn't use stale lists. It makes a quick background request to a continuously updated raw GitHub repository (currently a placeholder URL), pulling the freshest desktop browser strings as of this morning. No heavy HTML parsers like BeautifulSoup — just a plain text stream:

else:
    if not args.quiet:
        console.print("[bold cyan]► Fetching fresh User-Agents from online database...[/bold cyan]")

    raw_site_url = "https://githubusercontent.com"

    try:
        r_response = requests.get(raw_site_url, timeout=4, headers=headers)
        if r_response.status_code == 200:
            for line in r_response.text.splitlines():
                if line.strip().startswith("Mozilla/5.0"):
                    loaded_agents.append(line.strip())
    except:
        pass

Tier 3 (Built-in shield): If scanning happens in an isolated air-gapped lab with no internet, the try/except block gracefully catches the network failure and deploys an embedded fallback list of stable Chrome, Firefox, Edge, and Safari strings:

if not loaded_agents:
    if not args.quiet:
        console.print("[bold yellow]⚠️ Internet offline. Activating built-in hardcoded User-Agent fallback shield.[/bold yellow]")

    loaded_agents = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15",
        "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0"
    ]

How it works in the network loop: when the async futures generator packs a fan of requests, the utility uses random.choice() to pull a random User-Agent for each individual link. Inside check_single_link_async, headers are compiled locally right before calling session.get().

From the target server's perspective, it looks as if a crowd of completely different people on different devices and operating systems hit the site at the same time.

Chapter 3: Engineering according to Debian Policy

To turn a hobbyist script into an adult open-source product that you wouldn't be ashamed to show to distribution maintainers, the codebase had to be restructured from scratch:

Src-layout migration: recon.py moved deep into src/web_recon/. The repository root was cleared of clutter.
pyproject.toml manifest: All the old setup.py crutches were fully deprecated. The project now builds via the modern PEP 517/621 standard. The manifest declares automatic dependency management (rich, aiohttp, aiosqlite, python-whois) and registers a system entry point ([project.scripts]).
Birth of the system command: Thanks to this packaging, users no longer need to sit in a specific folder and type python recon.py. After installation, the tool registers a system alias. Type one word anywhere in the Kali terminal — web-recon — and the utility takes off.
Cloud CI/CD pipeline: In the tests/ folder I added automated tests based on pytest-asyncio (I'll admit, the recon_test file was partly vibe-coded). They check SQLite transaction atomicity, [] table escaping against SQL injection, and argparse CLI flag collisions. A GitHub Actions workflow (.github/workflows/tests.yml) is integrated into the repo. Now on every git push, GitHub's servers spin up a free Linux VM, install Python 3.12 and 3.13, set up the environment, run the tests, and hang out a green PASSED stability marker.

Conclusion: one-click installation directly from source

Yes, the Kali Linux curators have temporarily postponed packaging due to a lack of "media weight." But that's a completely normal working stage. Most top infosec tools start their journey as independent open-source projects distributed directly through repositories.

The project is fully stabilized, migrated to src-layout, and ready for local builds. You can deploy Web-Recon in your Kali Linux as a full system command right now.

To do so, just clone the repository and install it using pipx (recommended for CLI tools on modern Debian/Kali systems):

# Clone the official repository
git clone https://github.com/joavat-2009/Web-Recon
cd Web-Recon

# Install pipx if you don't have it
sudo apt update && sudo apt install pipx -y
pipx ensurepath

# Install Web-Recon in an isolated environment and expose the `web-recon` command
pipx install .

Why pipx instead of pip install? Modern Debian/Kali/Ubuntu systems block system-wide pip install (PEP 668) to protect system packages. pipx creates an isolated virtual environment for the tool while making the web-recon command available globally — exactly what you want for a CLI utility. Using pip install --break-system-packages is strongly discouraged as it can break your system Python.

Alternatively, you can skip building files from the main repository page entirely: starting with v0.6.0, the Releases branch contains pre-built .whl files, and it will do so for every version. You can install them via pipx install <path-to-whl>.

After that, the package manager will distribute the dependencies, register the entry point, and you'll be able to run the utility from any terminal folder with a short system command: web-recon --help.

A small call to work together (I'm not forcing anyone, I respect the community's choice, whatever it may be): friends, the Kali maintainers explicitly said they're waiting for community interest in the project. If you're into async OSINT perimeter scanning, stealth, and automation — drop by the repository, test the stealth mode of fan-out rotation, open Issues, and, of course, leave the coveted Stars ⭐! Let's build a fast, convenient, and cool recon tool together.

Project link: https://github.com/joavat-2009/Web-Recon

Clean code and a safe perimeter to everyone! I'd be glad to hear your feedback and criticism in the comments.

DEV Community: Joseph Avatiants

Building an async OSINT scanner: from a local script to a distributable tool

Chapter 1: Taming async power (semaphores and delays)

Chapter 2: The great disguise (cascading User-Agent rotation)

Chapter 3: Engineering according to Debian Policy

Conclusion: one-click installation directly from source