DEV Community

Danil Shutsky
Danil Shutsky

Posted on

AI Factory: Automating AI Agent Project Setup

#ai #opensource #llm #programming

I'll tell you about a tool I created to speed up project configuration for effective work with AI agents (in my case, Claude Code). I worked with SpecKit for a long time - a great project . It helped me a lot and gave me understanding of working with LLMs. But over time, I started noticing drawbacks that became increasingly frustrating.

The Problem

Before starting work with an AI agent on a project, my standard scenario looked like this:

1. Work Organization

  • Topic research
  • Planning (I worked with SpecKit): explaining project structure to the agent, describing the stack and work rules, creating plans and checking them

2. Skills Setup

  • Search for suitable skills on skills.sh
  • Copy to .claude/skills/
  • If needed skill doesn't exist - write from scratch (well, generate to be precise)

3. MCP Server Configuration

  • Open .claude/settings.local.json and configure MCP servers
  • Add environment variables

Result: 30-60 minutes of setup before starting real work (I mean vibe-coding). Gradually, I started automating processes and this evolved into the AI Factory project.

What AI Factory Solves

Setup Automation:

  • ai-factory init → interactively configures everything in 2-3 minutes
  • Scans existing project (package.json, composer.json, requirements.txt)
  • Determines stack and downloads appropriate skills from skills.sh
  • Generates missing skills for your project
  • Configures MCP servers by choice

Result: from 30-60 minutes of setup to 5-10 minutes.

Unified Context:

  • .ai-factory/DESCRIPTION.md - always up-to-date project specification for agent context - no need to repeat explanations about what we're working on in each chat

Structured Workflow:

  • Clear commands: /task for quick tasks, /feature for big features, /fix for bugs
  • Automatic plans and checkpoints
  • Conventional commits out of the box

Learning System:

  • Each fix creates a patch with problem description
  • Next tasks account for past mistakes
  • /evolve improves skills based on project experience

AI Factory

The idea is simple: minimal setup, advantage of using specifications for LLM context and code quality with MCP+sub-agents with skills. And work comfort - everything you need is at hand!

GitHub

What prompted its creation? Experience working with SpecKit and OpenSpec. Both tools are good, but in my opinion have drawbacks.

Experience with SpecKit and OpenSpec

I actively used both tools and encountered specific problems:

SpecKit (from GitHub):

  • Excessive documentation: generates hundreds of lines of specifications, plans and checklists. For small tasks this is overhead
  • Rigid workflow: hard to skip steps like testing, even when it's not needed
  • Context problems: many tokens spent on work
  • Refactoring complexity: when you need to quickly fix a bug, you have to go through the whole cycle /specify → /plan → /tasks → /implement

OpenSpec (simpler):

  • Validation bugs: often shows validation errors even when everything is correct. openspec validate and openspec show return contradictory results
  • Control problems: AI sometimes ignores workflow and starts implementation without /openspec:apply command
  • Complexity for large projects: unclear how to apply for existing large codebases

Common problem for both:

  • Require manual setup for each project
  • Require preliminary topic research
  • Insufficient automation of skills generation and MCP connection

AI Factory takes these moments into account and solves them.

Getting Started 

npm install -g ai-factory
ai-factory init
Enter fullscreen mode Exit fullscreen mode

During initialization:

  • Interactive questions about project and agent
  • Automatic MCP server setup
  • Creating .ai-factory.json config

For a new project - will ask about stack. For existing - will analyze code itself and pick what's needed.

Main Commands

Command Usage Branch? Plan?
/ai-factory.task Quick tasks No .ai-factory/plan.md
/ai-factory.feature Big features Yes .ai-factory/features/<name>.md
/ai-factory.fix Bugs and errors No No
/ai-factory.implement Execute plan - -
/ai-factory.evolve Improve skills - -

Workflow

For small tasks:

/ai-factory.task → plan → /ai-factory.implement → done
Enter fullscreen mode Exit fullscreen mode

For features:

/ai-factory.feature → branch + plan → /ai-factory.implement → commits → done
Enter fullscreen mode Exit fullscreen mode

For bugs:

/ai-factory.fix → fix + logging + patch → done
Enter fullscreen mode Exit fullscreen mode

Key Features

1. /ai-factory - Context Setup

Analyzes project, determines stack, picks skills from skills.sh or generates new ones, configures necessary MCP.

2. Feature and Task Planning

/ai-factory.feature - for large tasks with branch and full plan.
/ai-factory.task - for quick changes without branch.

Both analyze requirements, study codebase, create tasks with dependencies.

3. /ai-factory.implement - Executing Created Plan 

/ai-factory.implement 5      # work on task #5 if we want to do something specific
Enter fullscreen mode Exit fullscreen mode

Before starting, agent will read all patches from .ai-factory/patches/ - learns from past mistakes.

4. /ai-factory.fix - Quick Fixes

Use when it's clear the problem is small. Agent studies problem, applies fix with logging, creates patch for self-learning. No plans - immediate solution and learning.

5. /ai-factory.evolve - Skills Self-Learning 

/ai-factory.evolve           # All skills
/ai-factory.evolve fix       # Specific skill
Enter fullscreen mode Exit fullscreen mode

Analyzes all patches, finds error patterns, improves skills with your approval.

6. /ai-factory.skill-generator

Learning mode - pass URL to generate skills from documentation:

/ai-factory.skill-generator https://fastapi.tiangolo.com/tutorial/
Enter fullscreen mode Exit fullscreen mode

Studies sources, enriches through search, generates full-fledged custom skills.

Self-Learning System

Each fix creates a patch - a document that helps avoid similar errors in the future.

Learning cycle:

/fix → bug → fix → patch → 
next /fix or /implement → reads patches → better code
Enter fullscreen mode Exit fullscreen mode

Patch structure:

  • Problem and cause
  • Applied solution
  • Prevention recommendations
  • Search tags

Periodically run /ai-factory.evolve - the tool will analyze accumulated patches and improve skills for your project.

Project Structure 

project/
├── .claude/skills/          # Skills
├── .ai-factory/
│   ├── DESCRIPTION.md       # Specification
│   ├── PLAN.md             # Current plan
│   ├── features/           # Feature plans
│   ├── patches/            # Learning patches
│   └── evolutions/         # Improvement logs
└── .ai-factory.json        # Config
Enter fullscreen mode Exit fullscreen mode

MCP Servers

GitHub, Postgres, Filesystem are supported. Configuration in .claude/settings.local.json.

Best Practices

  • Logging: DEBUG/INFO/WARN/ERROR levels, control via LOG_LEVEL
  • Commits: checkpoints every 3-5 tasks, conventional commits format
  • Tests: always asked before plan, not added without consent

Tool Development

After active use over several days, I made two releases (v1.1 and v1.2):

v1.1: improved Skill Generator, learning mode

v1.2: patch system, /ai-factory.evolve, feature organization

v1.3. Security First!

Critically important update - protection system against prompt-injection. When downloading skills from skills.sh, we trust external sources with access to your project. Now each skill undergoes mandatory scanning for 10 threat categories - from injections and data leaks to social engineering. Two-level verification (regex + LLM intent analysis) filters out malicious code. /skill-generator scan allows manual file checking.

Also added new skill /ai-factory.improve for refining already created plans - when you understand the plan is good but something is missing.

Of course, I used AI Factory to develop AI Factory).

Conclusion

AI Factory solves specific AI development problems:

✅ Automatic project setup

✅ Spec-driven approach with control

✅ Self-learning from mistakes

✅ Simple workflow

Perfect If You:

  • Tired of setup before each project
  • Want structured approach when working with AI
  • Work on real projects
  • Have experience with Claude Code or similar tools

About the Future

I now have a team of assistants - experts in different areas. I'm doing projects I wouldn't have taken on before due to lack of stack knowledge.

This is a tool. Developer experience matters - often need to correct proposed solutions. AI is an assistant for specialists, not a replacement for developers.

Links

The library is free. Contributions and feedback are welcome.

Do you have experience with spec-driven approach? Share in the comments.

Top comments (0)