How I Built an Autonomous AI Tool Engine with MCP, Llama 3.3, and Automated Hugging Face Pipelines

#buildinpublic #ai #nextjs #mcp

Most AI tool directories suffer from two issues: they are statically hardcoded, and they break when you search by intent instead of exact keywords.

I built AI Tool Hunter V2 to solve this—turning a basic directory into an automated discovery engine, open-source dataset pipeline, and Model Context Protocol (MCP) server.

Here is a complete breakdown of the system architecture and the engineering trade-offs made along the way.

1. The Autonomous Ingestion Pipeline
Rather than manually curating links, a daily GitHub Actions workflow (harvest.yml) triggers an ingestion script:

Live Web Scan: Tavily API queries web sources for newly launched software.
Intent & Metadata Extraction: Raw payload is passed to Llama 3.3-70B via Groq. The model parses pricing structures, feature sets, and assigns intent-based categories.
Deduplication & Storage: Clean records are verified against existing database entries and stored in Supabase (PostgreSQL with RLS).

2. The Open-Source Data Flywheel
To share this corpus with the open-source community, the daily cron pipeline doesn't stop at database insertion.
It formats the updated database into a structured corpus and uses huggingface_hub to push a nightly commit directly to Hugging Face (AI-Tools-Corpus-2026).

3. Exposing the System via MCP (Model Context Protocol)
Modern developer tools live in the IDE. I built a dedicated MCP server hosted on Render so external LLMs (in Cursor, VS Code, or Claude Desktop) can call the search index natively.

Tech Stack