DEV Community: Zero Filter Diary

How to Automate Content Research Using Python and APIs (Step-by-Step)

Zero Filter Diary — Thu, 02 Jul 2026 06:08:10 +0000

I used to spend ten hours every week doing content research manually. Checking competitor blogs. Scanning Reddit threads. Copying and pasting search results into a spreadsheet. Trying to spot patterns in an ocean of unstructured text.

It was exhausting, slow, and completely unnecessary. Once I learned to automate this with Python and a few affordable APIs, I cut that ten-hour grind down to under thirty minutes. Here is the exact system I built, what it costs, and how you can replicate it yourself.

The Quick Answer

To automate content research with Python, combine a search API like Serper to pull structured Google search data, BeautifulSoup or requests-html to parse page content, and an LLM API like Gemini to synthesize insights into actionable content briefs. Connect these three components in a sequential Python pipeline and you have a fully automated research agent that runs in minutes instead of hours.

What I Actually Built

I needed a system that could do three things automatically:

First, find what real people are asking about any topic across Reddit, Quora, and Google search. Second, identify what my top competitors have written about that topic and where the gaps are. Third, summarize everything into a clean content brief I can use to write or generate an article.

I built this using Python with three core components: the Serper API for search data, BeautifulSoup for page parsing, and the Google Gemini API for synthesis. Total monthly cost: about twelve dollars.

I document the full working version of this system — including the Flask web interface and WordPress publishing integration — at https://zerofilterdiary.com

Step-by-Step Build Guide

Step 1: Install the Required Libraries

pip install requests beautifulsoup4 python-dotenv google-generativeai

Step 2: Set Up Your API Keys

Create a .env file in your project root:

SERPER_API_KEY=your_serper_key_here
GEMINI_API_KEY=your_gemini_key_here

Step 3: Search for Real Discussions Using Serper API

import requests
import os
from dotenv import load_dotenv

load_dotenv()

def search_topic(query, num_results=5):
    url = "https://google.serper.dev/search"
    headers = {
        "X-API-KEY": os.environ["SERPER_API_KEY"],
        "Content-Type": "application/json"
    }
    payload = {"q": query, "num": num_results}
    response = requests.post(url, headers=headers, json=payload)
    return response.json().get("organic", [])

# Search Reddit, Quora, and X separately
reddit_results = search_topic("python automation content research site:reddit.com")
quora_results = search_topic("python automation content research site:quora.com")

Step 4: Parse Page Content with BeautifulSoup

from bs4 import BeautifulSoup

def extract_text(url):
    try:
        headers = {"User-Agent": "Mozilla/5.0"}
        response = requests.get(url, headers=headers, timeout=8)
        soup = BeautifulSoup(response.text, "html.parser")
        # Remove scripts and styles
        for tag in soup(["script", "style", "nav", "footer"]):
            tag.decompose()
        return soup.get_text(separator=" ", strip=True)[:3000]
    except Exception as e:
        return f"Could not fetch: {e}"

Step 5: Synthesize with Gemini AI

import google.generativeai as genai

genai.configure(api_key=os.environ["GEMINI_API_KEY"])
model = genai.GenerativeModel("gemini-1.5-flash")

def generate_content_brief(topic, research_data):
    combined = "\n\n".join([
        f"Source: {item['title']}\nSnippet: {item['snippet']}"
        for item in research_data
    ])
    prompt = f"""Based on this research about '{topic}':

{combined}

Generate a content brief with:
1. Main angle to take
2. Key questions to answer
3. Suggested H2 headings
4. LSI keywords to include
"""
    response = model.generate_content(prompt)
    return response.text

Step 6: Wire It All Together

def run_research_pipeline(topic):
    print(f"Researching: {topic}")

    # Gather data from multiple sources
    all_results = []
    for site in ["site:reddit.com", "site:quora.com", ""]:
        results = search_topic(f"{topic} {site}", num_results=3)
        all_results.extend(results)

    print(f"Found {len(all_results)} sources")

    # Generate content brief
    brief = generate_content_brief(topic, all_results)
    print("\n--- CONTENT BRIEF ---")
    print(brief)
    return brief

if __name__ == "__main__":
    topic = input("Enter your topic: ")
    run_research_pipeline(topic)

Run this and in under 60 seconds you have a complete content brief backed by real search data.

My Real Results

I ran this pipeline across 30 different content research tasks and compared it to my old manual process:

The automated pipeline reviewed three times more sources in one tenth of the time. And because it runs identically every time, there is no "off day" where I miss something important because I was tired.

What Actually Works (And What Doesn't)

Use official APIs before scraping. Always check if a platform has a public REST API. Serper for Google, Reddit's official API for Reddit. Stable, legal, and never gets your IP banned.
Master async/await for speed. If you are querying multiple sites, running them sequentially is slow. Use asyncio to fire all requests in parallel.
Always parse HTML before sending to an LLM. Never dump raw HTML into an AI model. Strip it with BeautifulSoup first. Raw HTML wastes tokens and causes hallucinations.
Do not hardcode CSS selectors. Website layouts change constantly. Target stable elements like article tags, h1/h2 tags, and paragraph text rather than brittle nested class names.

What does not work: trying to scrape Google search results directly. They block you within minutes. Use Serper API — it costs fractions of a cent per query and gives you clean structured JSON.

Common Mistakes to Avoid

Underestimating IP bans
Running your scraper from your home IP across dozens of sites will get you blocked fast. For any project involving more than ten pages, use a dedicated scraping API or proxy rotation service.

Throwing raw HTML at AI models
This was my most expensive early mistake. Raw HTML bloats your token count massively and confuses the model. Always extract clean text with BeautifulSoup before passing anything to an LLM.

No data validation
Websites are messy. Some pages return empty titles, broken links, or missing snippets. If your script does not handle these gracefully with try-except blocks, it will crash mid-run and lose all progress.

Frequently Asked Questions

Is Python the best language for web scraping and API automation?
Yes. Python's ecosystem — BeautifulSoup, Scrapy, Requests, Pandas — is the industry standard for data collection and parsing. No other language has the same combination of simplicity and power for this type of work.

How do I handle dynamic JavaScript-heavy pages?
Use requests-html for simple dynamic rendering, or Playwright/Selenium for complex pages that require login or user interaction. Pair with a proxy-backed scraping API to avoid bot detection.

What are free alternatives to paid SEO research tools?
Build your own stack: Serper API for search data ($50 buys thousands of queries), BeautifulSoup for parsing (free), and Gemini API for synthesis (very cheap). This combination replaces tools that cost hundreds per month.

What to Do Next

Start small. Write a ten-line Python script that fetches the titles and snippets from one search query using Serper API. Get that working first. Then add BeautifulSoup parsing. Then add Gemini synthesis.

Build it in layers. Each layer is useful on its own, and each one makes the whole system more powerful.

The full production version of this pipeline — with Flask UI, multi-source research, and WordPress publishing — is documented at https://zerofilterdiary.com

How to Build an AI Blog Writing Agent with Python (Step-by-Step)

Zero Filter Diary — Tue, 30 Jun 2026 08:26:47 +0000

How to Build an AI Blog Writing Agent with Python (Step-by-Step)

I was staring at my screen at 2:00 AM, downing my third cold brew, trying to write five SEO-optimized articles for this blog while balancing a full-time gig and my sanity. That is when I realized I was doing mindless assembly-line work. I did not want to write generic AI fluff, but I also did not have twenty hours a week to spend on web research and structural formatting. So, being a developer who refuses to do repetitive manual labor, I decided to figure out how to build an AI blog writing agent with Python. I wanted a custom Python automation script that did not just spit out generic ChatGPT paragraphs, but actually researched real-time data, structured an outline, wrote deep content, and saved it directly as a Markdown file. Here is the exact unfiltered truth of how I built my own digital writing assistant, what it cost me, and how you can write your own code to get your life back.

The Quick Answer

To build an AI blog writing agent with Python, you need to initialize an LLM orchestrator using frameworks like LangGraph or CrewAI, define your system prompts, and connect essential API tools like Tavily for live web search and the Gemini API or OpenAI API for generation. Implementing asynchronous Python (asyncio) allows you to handle network wait times in parallel, compiling the state graph to generate highly structured, research-backed Markdown articles automatically.

What I Actually Did

I set aside a Saturday, locked myself in my home office, and decided to build this from scratch. I did not want to use complex, heavy frameworks that abstract everything away to the point where you cannot debug the code. Instead, I decided to use LangGraph for state management because it gives you absolute control over the workflow. I chose the Gemini API (specifically the Gemini 1.5 Flash model) because its massive context window and rock-bottom pricing make it perfect for digesting long research documents without breaking the bank. For web search, I hooked up the Tavily Search API, which is built specifically for LLM tool calling.

Here is the step-by-step breakdown of how I set up my environment and wrote the code.

Step 1: Setting Up the Local Environment

First, I set up a dedicated virtual environment to keep my dependencies clean:

python -m venv ai_agent_env
source ai_agent_env/bin/activate
pip install langgraph langchain-google-genai tavily-python python-dotenv

Then I created a .env file to store API keys:

GEMINI_API_KEY=your_gemini_api_key_here
TAVILY_API_KEY=your_tavily_api_key_here

Step 2: Defining the Writing State and Tools

The core of any LangGraph AI agent is its state — a Python class that defines what data gets passed from one node to the next:

import os
from typing import TypedDict
from dotenv import load_dotenv
from tavily import TavilyClient

load_dotenv()
tavily = TavilyClient(api_key=os.environ["TAVILY_API_KEY"])

class AgentState(TypedDict):
    topic: str
    research_notes: str
    outline: str
    draft: str
    file_path: str

Step 3: Building the Agent Nodes

I set up three distinct nodes — Researcher, Outliner, and Writer:

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", google_api_key=os.environ["GEMINI_API_KEY"])

def research_node(state):
    search_results = tavily.search(query=state["topic"], max_results=3)
    context = "\n".join([r["content"] for r in search_results["results"]])
    response = llm.invoke(f"Analyze these results about '{state['topic']}':\n\n{context}\n\nProvide research notes.")
    state["research_notes"] = response.content
    return state

def outline_node(state):
    response = llm.invoke(f"Create a structured blog outline for '{state['topic']}' based on:\n{state['research_notes']}")
    state["outline"] = response.content
    return state

def write_draft_node(state):
    response = llm.invoke(f"Write a full blog post using this outline:\n{state['outline']}\n\nAnd these notes:\n{state['research_notes']}")
    with open(f"{state['topic'].replace(' ','_')}.md", "w") as f:
        f.write(response.content)
    state["draft"] = response.content
    return state

Step 4: Compiling and Running the Agent

from langgraph.graph import StateGraph, END

workflow = StateGraph(AgentState)
workflow.add_node("research", research_node)
workflow.add_node("outline", outline_node)
workflow.add_node("write_draft", write_draft_node)
workflow.set_entry_point("research")
workflow.add_edge("research", "outline")
workflow.add_edge("outline", "write_draft")
workflow.add_edge("write_draft", END)
app = workflow.compile()

app.invoke({"topic": "How to Build an AI Blog Writing Agent with Python"})

I ran this script in my terminal, and in less than two minutes, a fully researched Markdown draft appeared in my project directory.

My Real Results

I spent two weeks testing single-agent vs multi-agent architectures. Here is the raw data:

My wallet practically begged me to stick with Gemini. Generating a deep, multi-agent researched post for less than two cents is an absolute game-changer.

What Actually Works (And What Doesn't)

Asynchronous Python (asyncio) is mandatory for scaling. Agents spend 95% of their time waiting on network APIs.
Gemini 1.5 Flash is the cost-efficiency king. Do not waste budget on GPT-4o for initial research parsing.
Direct API calls beat massive frameworks for simple tasks. Only use LangGraph when you need complex loops or memory.
Markdown file generation is superior to direct CMS publishing. Always write locally first, review, then upload.

I actually built a full AEO-optimized blog writing agent that does all of this automatically. You can read how it works here: https://zerofilterdiary.com

Common Mistakes to Avoid

Neglecting to Limit Search Query Tokens
Keep web searches limited to the top 3 relevant sources and extract short, summarized snippets only. Feeding raw HTML dumps into the LLM costs ten times more in tokens.
Relying on Single-Prompt Draft Generation
Asking an LLM to write a 2,500-word article in one prompt always fails. Design a workflow that writes each H2 section individually then compiles them into one file.
Forgetting Try-Except Blocks on Tool Calling
Web search APIs fail and LLM endpoints hit rate limits. Wrap every external API call in a try-except block or your entire pipeline will crash mid-run.

Frequently Asked Questions

Can I build an AI agent from scratch with zero coding experience?
Yes, using visual tools like n8n or Flowise. However, Python gives you infinite customization, complex file system integration, and total control over your state machine.

What is the cheapest LLM API for running blog writing agents?
Google Gemini 1.5 Flash. It offers a 1-million-token context window at roughly $0.075 per million input tokens — significantly cheaper than GPT-4o-mini or Claude 3 Haiku.

Why does my AI agent fail to write long articles?
Standard LLM endpoints have restricted output token limits (around 4,096 tokens). Design a modular workflow that writes section by section and compiles the file iteratively.

What to Do Next

Get a free Gemini API key from Google AI Studio, sign up for a free Tavily developer key, copy the three-node Python script above, and run it locally. Once you see a fully researched Markdown file appear in under sixty seconds, you will never go back to manual writing again.

If you want to see a complete, production-ready version of this system with Flask web interface, WordPress publishing, and AEO optimization built in, I documented the full project at https://zerofilterdiary.com