Building a Multi-Agent AI for Company LinkedIn Pages - Part 3: Building the Research Agent

#rag #agents #ai #vectordatabase

In the previous article, we built an Orchestrator Agent that can consistently classify a topic and return structured output that every downstream agent can rely on.

But classification alone isn't enough. LLMs shouldn't be limited to only their training knowledge. Every other agent built later depends on the factual information collected by the Research Agent.

Before we write a single line of code, let's first understand what the Research Agent takes as input and what it returns as output.

The Research Agent takes topic as input and searches the web, filters the results and scrapes the webpages. It splits the scraped text into chunks and embeds those chunks as vectors and stores them in ChromaDB.

When the user searches for a topic, that topic is converted into an embedding. ChromaDB then compares this query vector with every stored chunk and returns the chunks whose vectors are closest in semantic meaning.

Topic
   │
   ▼
Search Web
   │
   ▼
Filter Results
   │
   ▼
Scrape Pages
   │
   ▼
Chunk Content
   │
   ▼
Generate Embeddings
   │
   ▼
Store in ChromaDB
   │
   ▼
Semantic Search
   │
   ▼
Relevant Context

I'll use Serper to query Google Search and retrieve the top organic results. For each result, I only keep the title and URL.

Make sure to read through Requests and read until Response Content and BeautifulSoup only the Quick Start section, before we start writing the code.

load_dotenv()

def search_web(query:str) -> list[dict]:
    url = "https://google.serper.dev/search"
    api_key = os.getenv("SERPER_API_KEY")
    headers = {
        "X-API-KEY": api_key,
        "Content-Type": "application/json"
    }

    body = {
        "q": query
    }       
    r = requests.post(url, headers=headers, json=body)
    r_json = r.json()
    results = r_json["organic"]
    return [{"title": result["title"], "link": result["link"]} for result in results]

While building this agent, I came across a concept called Retrieval-Augmented Generation (RAG).

Think of it like an open-book exam.

Normally, an LLM answers from memory. With RAG, it first opens the right pages of a book (or knowledge base), reads the relevant information, and then answers your question based on both its knowledge and what it just retrieved.

That's exactly what our Research Agent is doing. To make that retrieval reliable, we first need to decide which webpages are worth indexing.

I prefer to omit social media sites as most of these pages are difficult to scrape consistently or don't expose their primary content as plain HTML, so I excluded them from the first version.

So, the is_scrapeable() function ensures the URL doesn't contain any of the restricted domains defined in SKIP_DOMAINS

SKIP_DOMAINS = ["instagram.com", "facebook.com", "reddit.com", "youtube.com", "x.com"]

def is_scrapeable(url: str) -> bool:
    return not any(domain in url for domain in SKIP_DOMAINS)

Now that we know which page to scrape and which one to avoid, let's write the scrape_page() function which takes a URL, downloads the webpage, and uses BeautifulSoup to extract all the text inside paragraph tags. It returns this combined text.

def scrape_page(url:str) -> str:
    try:
        response = requests.get(url, timeout=10)
    except requests.exceptions.RequestException as e:
        print("Failed to fetch page:", e)
        return ""
    soup = BeautifulSoup(response.text, "html.parser")
    if response:
        paras = soup.find_all("p")          
        return " ".join([p.get_text() for p in paras])
    else:
        return ""

Once we are done scraping the sites, we don't embed the whole page, we break them down into chunks. A webpage usually contains multiple ideas. Embedding the entire page into a single vector dilutes its meaning. Chunking lets each embedding represent a smaller, more focused piece of information, making retrieval much more accurate.

chunks = [" ".join(words[i:i+200]) for i in range(0, len(words), 200)]

Now we have to write the embed_and_store() function that converts each chunk into a dense vector using SentenceTransformer. Then save the text chunks, vectors and a unique ID into the chromaDB collection.

Make sure to go through ChromaDB Embedding Functions and ChromaDB Clients only the Persistent Client section before writing any code.

model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.PersistentClient(path="data/chroma")
collection = client.get_or_create_collection("research")

def embed_and_store(text: str, url: str) -> None:
    words = text.split()
    chunks = [" ".join(words[i:i+200]) for i in range(0, len(words), 200)]
    embeddings = model.encode(chunks)
    collection.add(
        documents = chunks,
        embeddings = embeddings.tolist(),
        ids = [f"{url}_chunk_{i}" for i in range(len(chunks))]
    )

Now we write the query_search() function which converts the user's original topic into a vector. ChromaDB compares the query embedding against every stored embedding and returns the chunks with the highest semantic similarity.

def query_research(topic: str, n_results: int = 3) -> list[str]:
    query = model.encode([topic]).tolist() 
    if n_results > collection.count():
        n_results = min(n_results, collection.count()) 
    res = collection.query(query_embeddings=query, n_results=n_results)
    return res["documents"][0]

Finally, we write the main research_agent() function which ties everything together:

Searches Google for the topic.
Loops through the search results.
Checks if the link is scrapeable.
Scrapes the text from the webpage.
Embeds and stores the text in the database.
Stops scraping once it has successfully processed 3 pages to save time and resources.
Runs a search on the database it just built to return the text chunks most relevant to the original topic.

For this project, scraping three high-quality articles provided enough context while keeping latency and API usage low.

Google Search
       │
       ▼
 Page 1   Page 2   Page 3
      │      │      │
      ▼      ▼      ▼
   Scrape Text
      │
      ▼
 Chunk Text
      │
      ▼
 Generate Embeddings
      │
      ▼
 Store in ChromaDB
      │
      ▼
 User Query
      │
      ▼
 Semantic Search
      │
      ▼
 Top 3 Chunks

This is the retrieval stage of our RAG pipeline. Instead of just searching Google, the agent reads the top articles, splits them into semantic chunks, stores them in a searchable vector database, and returns only the passages most relevant to the user's query.

def research_agent(topic: str) -> list[str]:
    results = search_web(topic)
    pages_scraped = 0
    for result in results:
        if is_scrapeable(result["link"]):
            text = scrape_page(result["link"])
            if text:
                embed_and_store(text, result["link"])
                pages_scraped += 1
                if pages_scraped == 3:
                    break
    return query_research(topic)