DEV Community

Cover image for Building a Multi-Agent AI Market Research Tool with CrewAI & Groq
Syed Kaifuddin
Syed Kaifuddin

Posted on

Building a Multi-Agent AI Market Research Tool with CrewAI & Groq

#ai #llm #opensource #agents

I recently built a multi-agent AI system that takes a product idea and generates a full market research report using 5 specialized AI agents — all running on Groq's free tier. No credit card. No paid API.

But the journey wasn't smooth. I hit 7 different errors before it worked. This article covers both — what I built and how I fixed everything that broke.

🧠 What I Built

Market Research Crew — a multi-agent AI pipeline where 5 autonomous agents collaborate sequentially to research any product idea:

  • 📊 Market Research Specialist — industry size, trends, opportunities
  • 🕵️ Competitive Intelligence Analyst — competitors, pricing, market share
  • 👥 Customer Insights Researcher — personas, pain points, needs
  • 🗺️ Product Strategy Advisor — positioning, feature roadmap
  • 📈 Business Analyst — synthesizes everything into recommendations

The user types a product idea into a Streamlit UI, hits run, and:

  • The UI updates in real time showing each agent's status
  • The IDE terminal streams live output — agent thoughts, tool calls, completions
  • 5 markdown reports are generated and displayed in tabs

GitHub: https://github.com/syed-kaif07/market-research-crew

🛠️ Tech Stack

Tech Role
CrewAI Multi-agent orchestration framework
Groq Free LLM API (insanely fast)
LLaMA 3.3 70B The language model powering all agents
Streamlit Web UI
Python 3.13 Language
uv Fast package manager

🏗️ Architecture

The most interesting part of this project is how the UI and terminal work simultaneously.

User types idea in Streamlit
         │
         ▼
streamlit_app.py
  └── subprocess.Popen(stdout=None)  ← key: inherits parent terminal
         │
         ▼
main.py
  ├── Prints colored agent banners to terminal
  ├── Hooks _TaskTracker into CrewAI's task_callback
  └── Calls crew.kickoff()
         │
         ▼
CrewAI runs 5 agents sequentially
  └── Each agent writes output to output/*.md
         │
streamlit_app.py polls output/*.md every 4 seconds
  └── Updates agent cards: QUEUED → RUNNING → DONE
Enter fullscreen mode Exit fullscreen mode

The magic is one line in streamlit_app.py:

proc = subprocess.Popen(
    [python_exe, main_script, "--product-idea", product_idea],
    stdout=None,   # ← inherits parent terminal = live output
    stderr=None,   # ← errors visible too
)
Enter fullscreen mode Exit fullscreen mode

By setting stdout=None, the child process inherits the parent's terminal handles. So everything CrewAI prints — agent thoughts, tool calls, our colored banners — all stream live to the IDE terminal while Streamlit polls for completed output files.

🤖 How CrewAI Works

CrewAI uses decorators to define agents and tasks cleanly:

@CrewBase
class MarketResearchCrew():
    agents_config = "config/agents.yaml"
    tasks_config  = "config/tasks.yaml"

    @agent
    def market_research_specialist(self) -> Agent:
        return Agent(
            config=self.agents_config["market_research_specialist"],
            llm=llm
        )

    @task
    def competitive_intelligence_task(self) -> Task:
        return Task(
            config=self.tasks_config["competitive_intelligence_task"],
            context=[self.market_research_task()]  # gets Agent 1's output
        )

    @crew
    def crew(self) -> Crew:
        return Crew(
            agents=self.agents,
            tasks=self.tasks,
            process=Process.sequential,  # one by one
            verbose=True,                # prints agent thoughts
            max_rpm=3,                   # respects free tier limits
        )
Enter fullscreen mode Exit fullscreen mode

Agent configs live in YAML files — clean separation of concerns:

# agents.yaml
market_research_specialist:
  role: Market Research Specialist
  goal: Analyze market size, trends, and opportunities for {product_idea}
  backstory: You are an expert market researcher with 10 years of experience...
Enter fullscreen mode Exit fullscreen mode

🖥️ Live Terminal Logging

To show live agent progress in the terminal, I hooked into CrewAI's task_callback:

class _TaskTracker:
    def __init__(self):
        self.done        = 0
        self.agent_start = time.monotonic()

    def on_task_complete(self, task_output):
        # Called automatically by CrewAI after every task
        self.done += 1
        elapsed = time.monotonic() - self.agent_start
        _agent_done(self.done, elapsed)      # prints ✓ AGENT X COMPLETE
        if self.done < len(AGENTS):
            _agent_start(self.done + 1)      # prints >> AGENT X+1 STARTING

# Attach to crew before kickoff
crew_obj.task_callback = tracker.on_task_complete
Enter fullscreen mode Exit fullscreen mode

The result in terminal looks like this:

=================================================================
        MARKET RESEARCH CREW - AGENT PIPELINE
        Powered by CrewAI x Groq x LLaMA 3.3 70B
=================================================================

  Research Topic: future of Gen AI in health sector

  AGENT PIPELINE QUEUE:
  1. 📊  Market Research Specialist        [ QUEUED ]
  2. 🕵️  Competitive Intelligence Analyst  [ QUEUED ]
  ...

-----------------------------------------------------------------
  >> AGENT 1/5 STARTING  📊  Market Research Specialist
  Time: 14:21:27
-----------------------------------------------------------------

  ✓ AGENT 1/5 COMPLETE  📊  Market Research Specialist
  Time taken: 43.2s
  Progress: [█░░░░] 1/5
Enter fullscreen mode Exit fullscreen mode

🐛 The Errors (The Real Story)

Here's where it gets interesting. Nothing worked on the first try.

Error 1 — AttributeError: st.session_state has no attribute "start_time"

Why it happened: Streamlit reruns the entire script on every user interaction. If a session_state key isn't initialized upfront, accessing it on a fresh rerun throws AttributeError.

Fix: Initialize ALL session state keys at the top of the script before any logic runs:

_DEFAULTS = {
    "running":      False,
    "completed":    False,
    "product_idea": "",
    "start_time":   None,   # ← was missing
    "process":      None,
}
for _key, _val in _DEFAULTS.items():
    if _key not in st.session_state:
        st.session_state[_key] = _val
Enter fullscreen mode Exit fullscreen mode

Rule: Always initialize session state with a defaults dict. Never access a key before setting it.

Error 2 — ImportError: Fallback to LiteLLM is not available

Why it happened: pyproject.toml had crewai[tools]==1.9.3 hardcoded. That old version of crewai couldn't find LiteLLM properly.

Fix: Update pyproject.toml:

# Before
"crewai[tools]==1.9.3"

# After
"crewai[tools,litellm]>=1.10.1b1"
Enter fullscreen mode Exit fullscreen mode

Then sync:

uv sync --upgrade --prerelease=allow
Enter fullscreen mode Exit fullscreen mode

Error 3 — openai version conflict

Why it happened: litellm needed openai>=2.8.0 but crewai 1.9.3 needed openai==1.83.0. They couldn't coexist.

Fix: Upgrading crewai to v1.10+ resolved this — the newer version aligned with litellm's openai requirement.

Error 4 — uv sync keeps rolling back versions

Why it happened: uv respects pyproject.toml strictly. Even after manually installing newer packages, uv sync would revert everything back to the pinned versions.

Fix: The pyproject.toml pin is the source of truth. Fix it there first, then sync.

Error 5 — TOML parse error 

missing comma between array elements, expected `,`
Enter fullscreen mode Exit fullscreen mode

Why it happened: Added "crewai[tools,litellm]>=1.10.1b1" without a trailing comma in the dependencies array.

Fix: TOML arrays need a comma after every element including the last one before ].

Error 6 — No solution found: crewai[tools]>=1.10.1 unsatisfiable

Why it happened: crewai 1.10.1 stable doesn't exist yet — only 1.10.1b1 (beta).

Fix: Use >=1.10.1b1 and add --prerelease=allow to uv commands.

Error 7 — git push rejected 

error: src refspec main does not match any
Enter fullscreen mode Exit fullscreen mode

Why it happened: I was running git commands from inside src/market_research_crew/ instead of the project root.

Fix:

cd ../..   # go to project root first
git push origin main
Enter fullscreen mode Exit fullscreen mode

💡 Key Lessons

1. pyproject.toml is the source of truth
When using uv, always fix version pins in pyproject.toml first. Manual pip installs get overridden on next uv sync.

2. Streamlit session state must be initialized upfront
Use a _DEFAULTS dict pattern. Any key you access must be initialized before Streamlit reruns.

3. stdout=None enables live terminal streaming
When spawning subprocesses, stdout=None inherits the parent terminal — much better for debugging than redirecting to a file.

4. CrewAI's task_callback is powerful
Hooking into task_callback lets you track exactly when each agent completes without modifying CrewAI internals.

5. Dependency conflicts need a root cause fix
Installing packages manually is a band-aid. The real fix is always the version constraint in your project config file.

🚀 Try It Yourself 

git clone https://github.com/syed-kaif07/market-research-crew.git
cd market-research-crew

pip install uv
uv sync --prerelease=allow

# Add your free Groq API key to .env
# GROQ_API_KEY=your_key_here
# MODEL=groq/llama-3.3-70b-versatile

python -m streamlit run src/market_research_crew/streamlit_app.py
Enter fullscreen mode Exit fullscreen mode

Get a free Groq API key at console.groq.com — no credit card needed.

🔮 What's Next

  • Add PDF export for the full report
  • Add web search tools so agents pull live data
  • Let users choose between different LLMs from the UI
  • Build a second crew for content writing or SEO

If you found this useful or have questions, drop a comment below. And if you're working on something similar with CrewAI, I'd love to see it!

Built by Syed Kaifuddin

Top comments (0)