DEV Community: Rafael Romero

From Just a Scanner to a Smart Agent: How I Improved my SEO Prospecting Tool 🐍

Rafael Romero — Fri, 28 Nov 2025 16:16:15 +0000

I recently built a prospecting agent with Python to find local businesses on Google’s lower-ranked pages and pitch them SEO services.

The initial version was... promising but flawed.

It tried to pitch Indeed.com because they didn't have a local phone number. It told Ford Dealerships their site was "down" because their firewall blocked my bot. It sent robotic emails starting with "Fail: H1 Missing" ... not exactly a charming opener.

I realized that to make this tool useful, I needed to move from a simple scraper to a true agent. Here is the breakdown of how I refactored the code to filter noise, crawl for contacts, and use GenAI to write personalized campaigns.

Step 1: Filtering the Noise (The "No-Go" List)

The first problem with scraping generic keywords is that half the results aren't businesses, they are directories, job boards, and government sites. My script was wasting resources auditing ZipRecruiter and Texas.gov.

The Fix:
I made the clean_and_deduplicate function even more robust with a strict blocklist. I expanded the existing blocklist significantly. We categorized domains into "Job Boards," "Government," "Social Media," and "National Brands" (like Penske) that wouldn't hire a local agency anyway.

# We filter these out before we even attempt an audit
DIRECTORY_DOMAINS = [
    'indeed', 'glassdoor', 'ziprecruiter', # Job Boards
    '.gov', 'texas.gov', 'fmcsa',          # Gov sites
    'yelp', 'yellowpages', 'bbb.org',      # Directories
    'penske', 'uhaul', 'ford.com'          # National Brands
]

def is_directory(url):
    if any(domain in url.lower() for domain in DIRECTORY_DOMAINS):
        return True
    return False

Result: The list went from ~200 "leads" to ~70 actual local businesses.

Step 2: Smarter On-Page Auditing

My original script checked for H1 tags using exact string matching. If the keyword was diesel mechanic and the H1 was Best Diesel Mechanic in Texas, the script marked it as a FAIL.

The Fix: Fuzzy Logic
I switched to token-based set matching. If the H1 contains a significant percentage of the target keywords (over 50%), it passes.

# Breaking strings into sets of words for flexible matching
required_words = set(keyword.lower().split())
found_words = set(h1_text.lower().split())

# Calculate intersection
matches = required_words.intersection(found_words)
match_percentage = len(matches) / len(required_words)

# If >50% overlap, it's a Pass.
if match_percentage >= 0.5:
    audit_data['H1_Audit_Result'] = "Pass"

Step 3: Distinguishing "Broken" from "Blocked"

Originally, if a site returned a 403 Forbidden, my script flagged it as "Actionable: Server Error." Pitching a client saying "Your site is down" when it's actually just secure is a great way to look incompetent.

The Fix: Handling Firewalls
I updated the requests logic to explicitly catch 403 and 406 errors and mark them as SKIP. Now, the agent only flags genuine connection errors (like 500 or SSLError) as actionable leads.

except RequestException as e:
    # If the server explicitly blocked us (Firewall/WAF), it's not a lead.
    if response.status_code in [403, 406, 429, 503]:
        audit_data['Error_Status'] = "Blocked" 
        return audit_data  # Stop processing, we will filter this out later

    # Real connection errors (DNS failure, Timeout) are actual leads 
    # We want to pitch "Site Health" services to these.
    audit_data['Error_Status'] = f"Error: {e.__class__.__name__}"

Step 4: The "Gap Analysis"

This was the strategic game-changer. A site with a missing H1 tag isn't necessarily a good lead. But a business with 50 five-star reviews and a missing H1 tag? That is a gold mine.

I integrated a secondary API call to fetch the Google Business Profile (GBP) ratings for every prospect to identify "Hidden Gems": businesses with great real-world reputations but poor digital presence.

# We categorize the lead before generating the pitch
is_gbp_strong = gbp_rating >= 4.0 and gbp_reviews >= 10
is_gbp_missing = gbp_rating == 0

# Strategy A: Strong GBP + Weak Site = "Your site hurts your reputation"
# Strategy B: No GBP + Weak Site = "You are invisible"

Step 5: The "Missing Link" - The Crawler

At this point, I had great prospects, but I was missing the most important piece of data: The Email Address. Many local businesses don't put their email in the Header; they hide it on the "Contact Us" page.

The Fix: The Spider Logic
I upgraded the agent to act like a human user:

Scan Home: Look for mailto: links or regex matches.
Heuristic Scoring: If no email is found, scan all links and score them. /contact-us gets 100 points, /about gets 30 points.
The Hop: The agent navigates to the highest-scoring URL and scrapes that page.

def find_best_contact_url(soup, base_url):
    # Heuristic Scoring Logic
    score = 0
    if 'contact' in url_path: score += 100
    if 'contact' in link_text: score += 50
    if link_is_in_footer: score += 10

    # Returns the URL with the highest score to crawl next
    return best_candidate

This logic alone saved ~40% of leads that would have otherwise been discarded as "No Contact Info."

Step 6: From Template to GenAI (Gemini Integration)

Finally, I tackled the outreach itself. My previous email template was rigid and impersonal. I wanted a 3-email sequence that felt human.

The Fix: Google Gemini 2.5 Flash
I integrated the Gemini API (which is proven to be fast and cost-efficient). Instead of using a fixed string, I feed the Audit Data + GBP Data into a prompt.

The AI generates a 3-stage campaign:

Email 1: The Hook (Referencing their specific Reputation vs. Site Gap).
Email 2: The Value (Educational content about the error found).
Email 3: The Breakup (Professional closing).

# Feeding the Gap Strategy into the LLM
prompt = f"""
    PROSPECT: {company}, Rating: {gbp_rating} stars.
    ISSUES: H1: {h1_status}, NAP: {nap_status}

    STRATEGY:
    1. If rating > 4.0, praise reputation but warn about site errors.
    2. Explain WHY {h1_status} kills rankings.
    3. Gentle breakup.

    OUTPUT FORMAT: JSON {{ "subject_1": "...", "body_1": "..." }}
"""

model = genai.GenerativeModel('gemini-2.5-flash')
response = model.generate_content(prompt)

The Result

The agent now runs autonomously. It scans SERPs, filters junk, crawls for emails across multiple pages, and uses LLMs to write custom campaigns.

The Metrics:

Raw Scrape: 200 URLs
After Cleaning Directories: 70 Businesses
Actionable Leads (With Emails): ~30 High-Quality Prospects

Key Takeaway: When developing an Agent or any tool in general, Iteration is King. You have to be able to know what you currently have and what's missing to reach that optimal output. In my case, the difference between "just a script" and an "agent" is the ability to handle imperfection, hopping pages when data is missing, understanding context, and generating dynamic output. This project has become something I look forward to working on and the most exciting part is that there's still room to grow.

🔗 Check out the Code:
You can find the full source code and contribute to the project on GitHub:
https://github.com/Rafa-romero-dev/seo-agent

A special thank you to the Dev.to team for featuring my previous article in the Top 7 Featured Dev Posts of the Week!

What do you think I should focus on next? What could use some refinement? Let me know in the comments!

Beyond Page One: Building a Highly Robust SEO Lead Generation Agent with Python and SerpApi🤖

Rafael Romero — Fri, 14 Nov 2025 20:38:47 +0000

Introduction

As a SEO consultant or agency owner, finding actionable leads is a constant hustle. The obvious Page 1 opportunities are highly competitive so the real opportunity is on Google Page 2 and beyond, where basic SEO mistakes by local businesses are common.

This post describes how I created a production-ready Python agent (depending on your definition) that:

Scrapes targeted SERP pages (Page 2+)
Performs instant on-page audits (H1, local NAP, etc.)
Generates a personalized sales pitch for every failure

The project focuses on three pillars of production code: Efficiency, Precision, and Robustness.

Phase 1 — Core Logic: Efficiency & Data Gathering

Scaling Google searches is the primary challenge. I used SerpApi to avoid scraping blocks and programmatically target long-tail local queries defined in serp_config.py (a combination of CITIES and KEYWORDS). I tried making a free version with open source tools but it was a huge failure, so SerpApi really did the heavy lifting here.

Key takeaway: target Page 2+ instead of Page 1. Use CLI arguments to control which pages to scan and iterate the start parameter.

# The SerpApi parameters in serpapi_extractor
params = {
    # ... other parameters ...
    "start": current_rank,  # page offset (0=Page 1, 10=Page 2, etc.)
    "num": 10
}

# Iterate from start_page to end_page (increase by 10 per page)
for current_rank in range(start_index, (end_page * 10), 10):
    # ... API call ...

Phase 2 — From Raw Data to Actionable Targets: Precision

After the data extraction we are presented with two main problems:

Duplicate URLs (same company ranking for multiple queries)
Irrelevant directories (e.g., Yelp, YellowPages)

So how did we handle these cases:

Deduplication

Convert raw results to a DataFrame
Normalize URLs (strip http://, https://, www.)
Keep the entry with the best (lowest) rank

Filtering directories

Exclude known directory domains to avoid wasted audits and false positives.

# Directory filter example (from __main__ section)
directory_domains = [
    'yelp.com', 'google.com/maps', 'yellowpages.com',
    'facebook.com', 'linkedin.com'
]

def is_directory(url):
    return any(domain in url.lower() for domain in directory_domains)

cleaned_df['Is_Directory'] = cleaned_df['URL'].apply(is_directory)
cleaned_df_filtered = cleaned_df[cleaned_df['Is_Directory'] == False].drop(columns=['Is_Directory'])

Phase 3 — The Critical Audit: Robustness & Personalization

Direct audits are fragile: sites block bots or return transient errors. Initial runs produced many leads caused by ConnectionError or HTTP 403 (technical failures, not SEO misses). Two enhancements fixed this:

CLI automation
- Replace input() with argparse so the script can run non-interactively (e.g., python agent.py 2 5).
Retry mechanism
- Retry transient request errors up to max_retries and record real audit failures only after retries fail.

# Retry snippet from run_on_page_audit
for attempt in range(max_retries):
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()  # raises HTTPError for 4xx/5xx
        break  # success

    except requests.exceptions.RequestException as e:
        # Retry logic...
        if attempt == max_retries - 1:
            audit_data['H1_Status'] = f"Error: Request Failed ({e.__class__.__name__})"
            return audit_data

Sales Pitch Generation

The generate_sales_pitch function turns audit failures into tailored outreach messages. It targets common failures like:

Missing H1
Missing NAP (Name, Address, Phone)
Server errors / blocked pages

Each CSV row (audited URL) becomes a qualified sales opportunity with a targeted pitch.

Conclusion and Next Steps

This agent provides a powerful foundation for automated SEO lead generation. It is highly efficient in its use of paid API credits, precise in its targeting, and robust enough to handle the realities of web scraping. The output is a highly qualified list of prospects ready for immediate outreach.

As a Frontend Developer primarily focused on JavaScript, I’m thrilled with how this project turned out! Python is a huge, yet rewarding, undiscovered land of possibilities for me. Collaborative tools like Gemini were essential, providing a step-by-step path to bring this script to an acceptable, production-ready state.

I believe open sourcing this kind of tool benefits the whole developer community. You can find the full, production-ready code, including the agent.py and instructions on how to set up serp_config.py, on GitHub: Seo Agent on Github

I'd love to hear your experiences and ideas!

Have you used automation for lead generation? What other on-page checks do you find most effective for a quick audit? Or is there a tool that you use that I can take inspiration from?

Let me know what improvements should I make! Drop a comment below or open an Issue on the repo to share your thoughts and suggestions.

The Frontend Challenge 4 - Glam Up My Markup: Recreation

Rafael Romero — Mon, 05 Aug 2024 03:21:59 +0000

This is a submission for Frontend Challenge v24.07.24, Glam Up My Markup: Recreation

What I Built

After a little break I came back for this challenge, I really liked the premise and I tried to create a design that was smooth and dynamic, this is a sports page after all. Is not as responsive and accessible as I wanted it to be, but this is the best I can do with my current time constrains and all the difficult situations on my country.

Of course I always try to adhere to the challenge rules but this time I went all out with the animations, those are the star of the show, specially the scroll animation.

Demo

For this demo I recommend to always see it in the full page view, specially in a new tab (please) and of course it will be functional in desktop (the best experience) and in mobile (the most common way).

You can see the page on a new tab clicking here

Journey

I can keep improving the code with some dry principles, specially with the aforementioned scroll animation, I repeated a lot of logic in the code but I did try my best to keep it as clean and clear as possible. The design process for this submission was my favorite part, it was not easy to choose a good color scheme, but I'm glad I landed on that red, grey and cream combo and I hope you find it pretty to look at as well.

Finding some "real" cricket images was easy with my new knowledge of the Commons Creative License, and I found some good icons in a tool I discovered in a recent DEV post (I'll try to find the post to credit it in the comments). But I also made some assets like the cricket ball pattern in the schedule section and of course, the cover image on this post.

And that's all for this one. If you're participating in the challenge, I hope you have as much satisfaction from your work as I do from mine. If not, perhaps the upcoming challenge will be your time to shine.

And as always, if you have some feedback you can write a comment or send me a DM. English is my second language, so please be nice to me ❣️ Thank you for reading up to this point, and until the next time, bye!

One Byte Explainer - The Nibble

Rafael Romero — Fri, 14 Jun 2024 15:55:31 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Explainer

Half a byte. Is a small unit of information used for more efficient use of memory & processing power when resources are limited.

Additional Context

You could call this one a One Nibble Explainer

The Frontend Challenge v2: Earth Day Edition🌎

Rafael Romero — Mon, 29 Apr 2024 03:33:19 +0000

This is a submission for Frontend Challenge v24.04.17, Glam Up My Markup: Earth Day Celebration Landing Page

What I Built

For this challenge, I endeavored to build a distinctive design, consistently prioritizing responsiveness and exploring an unexpected yet vibrant color scheme. As with the first challenge, I strived to adhere to the guidelines as closely as possible. The comprehensiveness of my design this time around might be somewhat daunting, but I am confident that it is engaging (and hopefully resonates with you).

This marks my second venture into the Frontend Challenges. Despite grappling with time management for my side projects (and kinda losing), I find immense enjoyment in these challenges that inspire creativity and encourage thinking outside the box.

Demo

The demo is (again) as responsive as I could do it. I think the earth's image could be better handled (any ideas please comment down below). I tried to take into consideration the main ways to see the demo in Codepen: In the 3 different default "Change view" options and the Live View in a different page. Always considering the mobile view, which I think is the most common. But the Live View (or full page view) in Desktop is my favorite.

Journey

As with all projects, I required a concept to guide my efforts. This time, the project was relatively straightforward, being a landing page. Naturally, the Earth theme opened up a plethora of opportunities for intriguing animations to enhance the design. However, I didn’t want to simply add animations and consider the job done. My rationale is that when you encounter an aesthetically pleasing landing page, you appreciate it, but then what? You consume the content and move on, it becomes forgettable. Therefore, I aimed to create something that would elicit a “wow, what a fun idea” reaction from people. They would explore the page, perhaps experience an “aha!” moment, and be pleasantly surprised by a little something extra.

With an idea in mind, I set out to create a mini-game. However, crafting something as complex as Space Invaders was out of the question (given my professional procrastination tendencies). Instead, I envisioned something swift and vibrant: an Easter egg hunt. At the bottom of the page, I incorporated a message that reads, "Can you make a difference on this page? Find the 5 'green' actions. Hint: Some icons are clickable!" Now, those who have merely skimmed the landing page from top to bottom have a reason to revisit and discover these clickable elements. Alternatively, if you're sufficiently curious (or have a mouse), you'll notice the cursor transforming into a pointer, indicating that something is about to happen.

So, what happens next? I was brimming with ideas, many of which were too complex to execute in a single day. However, I was determined to implement a color transition, something that would signify a transformation on the "planet" or page. We start with a vibrant planet, adorned with clear blue skies and lush green hills. Yet, pollution taints this vibrancy, the air turns toxic and loses its clarity, and the once vibrant green becomes dull and lifeless. Implementing these changes on the page while maintaining sufficient contrast was a challenge in itself. However, I believe I've made a commendable effort in articulating my concept here, so at least some of you reading this post can comprehend why I opted for these colors and why they undergo a transition.

But wait...

Why do I persistently refer to it as a game? As the text indicates, you're tasked with identifying all 5 "green" actions. Similar to switching off a lightbulb, you can alter the state of most icons on the page. While most are conspicuously visible, one is cleverly concealed. Once you've successfully adjusted all of them to the correct state, a surprise awaits you at the top of the page.

That's about it! There isn't much to elaborate on this time, as I didn't incorporate a multitude of features. However, there's ample room for enhancement. Landing pages are the most prevalent type of page online, and my submission could benefit from additional accessibility features and animations. I may revisit this project in the future to further refine it, adding more features. While a theme switch may not be particularly relevant for this project, the option for different languages could be a valuable addition. Anyway...

If you're participating in the challenge, I hope you have as much satisfaction from your work as I do from mine. If not, perhaps the upcoming challenge will be the turning point for you. Thank you for reading up to this point, and until the next time, bye!

If you have some feedback you can write a comment or send a DM. English is my second language so please be nice to me ❣️

The Frontend Challenge: Glammed Up Camp Activities

Rafael Romero — Sun, 31 Mar 2024 22:11:13 +0000

This is a submission for DEV Challenge v24.03.20, Glam Up My Markup: Camp Activities

What I Built

For this challenge, I tried to create a project that is dynamic, responsive, and accessible, all within my current abilities and adhering to the challenge rules. Although there may be some areas where improvements could be made, I believe my work is comprehensive and engaging. The process of crafting this submission was truly enjoyable and fun for me and I hope you find something interesting in it.

Demo

The demo itself is as responsive as I could do it. I tried to take into consideration the main ways to see the demo in Codepen: In the 3 different default "Change view" options and the Live View in a different page. I even considered the mobile view.

Journey

It all started with an idea, I wanted to add dynamic images on the form, to show the different camp activities, that way filling up the form wouldn't be so boring and you could better think of the option you want the most before submitting.

To achieve that, I started to search for free pics and resources, until I stumbled upon Storyset which -to my surprise- had images for all the activities so I could have the same style of images in the project.

Then I started thinking about the overall style of the project. While thinking about the project I was watching the Mashle season 2 opening and if you know what I'm talking about you would understand why I chose the yellow color palette. Of course, because this is a Dev.to challenge, I wanted to do a minimalist "note" style form. Hopefully, my design represents this style decision.

When I started adding the images in the project I realized that adding a lot of images would be difficult as the template can't be really manipulated (now I know this isn't true) so as a solution, I did a multi-step form, that way I can organize the info that's shown on screen. But that came with its own challenges.

I tried to make the code as clean and explicit as possible (with a lot of comments) but essentially, I needed to add new elements like the images, some buttons to control the steps, steps indicators and a random div to add some style. I also needed to add and remove (conditionally) a lot of attributes and classes, to make the form reactive and responsive.

After all the styling was on place and worked in the tab functionality I then started to make some accessibility changes, for example, I added focus classes, and manipulated the tab index to make keyboard navigation as clear as possible. Sadly, I'm very new to accessibility and I am not sure I did the best job in it. Nevertheless, I tried making it accessible and I think trying and learning from mistakes is better that not doing anything for fear that it comes up wrong.

Responsiveness was a difficult step, because I am using a lot of absolute positioning which I think is not the best option but is the easier. Like I said in the demo portion, I tried to take into consideration the default page views and different combinations of width and height. I hope you can see a version that looks good and don't judge me for the failed resolutions 🥲

Finally, the project was finished!

Until it wasn't...

Ok let me explain, this is my first challenge and big post I make for dev.to in the 6 YEARS I have in this platform. Seeing that I still had some hours to submit the project, I wanted to have a Plus Ultra moment and add some more features, form validation and dark mode.

It was actually fairly simple to add them considering that dark mode is one of the most common features in virtually all modern web projects and the validation was in the page that I was referencing when doing the multi step form. After adding this, and testing some more, checking all the styles and functionalities where in place, I now consider this project a huge success.

I really don't care too much about winning the challenge, I know there are thousands of people with insane skills and lots of experience, but I wanted to feel satisfied with my contribution, that I did all I could do, to show off my skills and most of all, remind myself that I can do amazing stuff when I put my heart into it.

Finally, if you are participating in the challenge I hope you feel the same and if not, maybe the next challenge will be your moment. Thanks for reading until here, and I have no idea how to end this so... Goodbye.

If you have some feedback you can write a comment or send a DM. English is my second language and this is my first "technical" post so please be nice to me ❣️

My Hacktoberfest 2023 Pledge

Rafael Romero — Wed, 11 Oct 2023 14:56:22 +0000

Hi I'm Rafael Romero, a Computer Engineer and a Frontend Developer from Venezuela. I always introduce myself the same way because I'm not very known in any tech community, I am a 5 year club member of dev.to and I really enjoy reading a lot of articles and seeing the wins of the week. There are a lot of amazing people in Dev and I've always wanted to be part of the community in a more significant way, write blogs and share experiences. That's why I joined Dev in the first place and I hope to write some posts before this year's end.

I really love Hacktoberfest. My first one was in 2019 and it was --in part-- the reason why I got a job in 2020 and since then I've always tried to complete the challenge and I always try to up the complexity of my contributions. In 2020 and 2021 I was full of work in October but I successfully returned to the challenge last year, I made my first project with Next.js and overall I felt really good with my contributions even if they are small.

This year, I've had some trouble starting my open source journey but I will start today, even if I start with something simple like some minor UI changes. My goal is to keep contributing after the Hacktoberfest so I want to find projects I think are cool or innovative and if able, help with frontend related issues so I can apply my knowledge in different projects, learn from different systems and code bases and help others.

Finally, congrats to everyone that already completed the challenge as of today, cheers to those that are half way there and for all of you that haven't made up your mind or are having some difficulties on the way, keep going, you can do it. Happy coding!

Which terminal do you use?

Rafael Romero — Sat, 08 Jun 2019 01:53:08 +0000

Ok so I'm used to use the windows command prompt to do almost anything in my laptop, and git Bash for git related stuff, after seeing hyper and zsh on some posts and videos, it sparked my curiosity, maybe I need to level up my terminal game.

So, I ask you people of Dev.to, which one is your favorite terminal and why, how do you configure yours to be your master shell? 👀