DEV Community: rosidotidev

Spec-Driven Development with GitHub Spec Kit

rosidotidev — Mon, 12 Oct 2026 21:20:36 +0000

As a software architect and senior software engineer, I find myself in the middle of a change that I can only describe as epochal. The way we think about software development and enterprise architecture is shifting, and so are the roles that humans and "intelligent systems" play today and will play tomorrow.

My angle is Spec-Driven Development (SDD) with GitHub Spec Kit, seen from where I work every day: software architecture. In this first article I stay on the theory, with a small example of SDD to make it concrete. In the next ones I build a real application with Spec Kit. GitHub Spec Kit is not the only option: alternatives include Kiro, OpenSpec and BMAD. For now, though, GitHub Spec Kit is the one that best fits my preferences and the way I like to work. This is a personal choice, not a claim that one approach is universally better than the others. I wrote this article to share my view and to set a personal starting point that I can come back to (and probably correct) while things evolve. BMAD is one I may explore in a future article, or perhaps a short series of its own.

What Is Spec-Driven Development?

Let's start from the problem. Ask an AI agent to "add user registration to my app" and it has to guess: which fields, which password rules, is the email verified, how are accounts stored? The result is often plausible-looking code resting on dozens of unstated assumptions, many of them wrong. This is what people call vibe coding.

To me, SDD is the answer. It is an approach where a detailed, human- or AI-validated specification becomes the single source of truth before any code is written. Instead of documentation as an afterthought, the process is inverted: the specification is an executable blueprint that drives the AI coding agents.

Not every SDD approach keeps the spec alive for the same amount of time. I've found a classification featured in numerous articles to be incredibly helpful to distinguish three levels of rigor:

Spec-first: a well-thought spec is written before coding to guide the initial implementation. Once the code exists, the spec may be discarded or drift: its value is the initial clarity, which makes it a good fit for AI-assisted first builds and prototypes.
Spec-anchored: the spec is kept and evolves with the software, through automated tests and contracts. A change in behavior updates both, and tests derived from the spec (BDD scenarios, or OpenAPI with contract testing) fail when they drift. It suits most production systems well.
Spec-as-source: the spec is the main source file and agents generate or update code directly from its changes. Humans edit only the spec and never the generated code, so drift is eliminated by design. It needs mature, trusted generation tooling and today is practical mainly where generation is well defined, such as server stubs from OpenAPI.

My rule of thumb: use the minimum level of rigor that removes ambiguity in your context. Spec-first for AI-assisted initial development, spec-anchored for long-lived production systems, spec-as-source only when the generation tooling is mature and trusted.

The Specification as a Contract

This is the idea I want at the center of this series.

In a code-centric flow, requirements documents drift and design diagrams rot, so the code ends up being the de facto truth. Asking "what should this function do?" means reading the implementation. SDD flips the relationship: the specification becomes the authoritative description that humans and machines use to build and maintain the system. The spec declares the intent, the code realizes it.

I think three properties turn a specification into a contract and not just documentation:

Each phase constrains the next. Specify answers what, plan answers how, then come tasks, implementation and validation. Each phase produces an artifact that guides the following one, creating a chain of accountability from intent to implementation.
The spec stays the authority. When validation finds a gap, you either fix the code or revise the spec. Either way, the specification remains the reference point.
It is checked, not just read. Traditional design documents are read by developers who then write code that hopefully matches them. A contract is meant to be verified: if the code diverges, something should fail.

Contracts are not new in software. API specifications such as OpenAPI have long played this role: once the API contract is agreed, frontend and backend teams can work in parallel, and any implementation that matches the contract is valid by definition. SDD brings the same logic to the whole build process.

Mental Model: the spec declares the intent, the code is an implementation detail. And I would be careful with false confidence: a passing check only proves that the code matches the spec. If the spec is wrong, the code will faithfully implement the wrong thing.

GitHub Spec Kit in a Nutshell

Spec Kit is the open source toolkit from GitHub for spec-driven development. You interact with it through commands in your coding agent, and it plugs into many of them (Copilot, Claude, Gemini, Codex and others). It can also be extended with extensions, presets, workflows and bundles.

The core process is:

Specify -> Plan -> Tasks -> Implement -> Converge

The constitution, the project-wide principles and constraints, is defined once per project. Each phase produces a Markdown artifact that feeds the next one. For production features, three optional quality gates are added.

clarify: Asks targeted questions about anything underspecified and folds your answers back into the spec, so you're not planning on top of ambiguity. Run it before planning, optionally with a focus area.
checklist: Generates a custom quality checklist These custom checklists are reviewer-owned
analyze: Reports conflicts, gaps, and ambiguities across spec.md, plan.md, and tasks.md

A Simple Example: a Todo List

Theory is easier with a concrete case. Take the simplest idea I can think of, a todo list, and look at what goes where. The example is mine, written just to show the separation. The syntax is the real one from the Spec Kit docs, in GitHub Copilot's default skills mode (/speckit-*). You type these commands in the chat of your coding agent, not in the terminal. Other agents may invoke them in a different way, so check the docs for your setup.

Step 1: Constitution (project-wide principles, written once):

/speckit-constitution Todo List is an adaptive web application.
Every API change starts from the REST contract.
Business logic lives in the backend, never in the UI.
Every feature comes with automated tests.
The UI must work on both phone and desktop.

This creates or updates the project constitution: the guiding principles against which every later phase is evaluated. A few lines from the file look like this:

<prj_root>/.specify/memory/constitution.md

## Core Principles

### I. Adaptive Todo Application
The product MUST be an adaptive web application for managing todo items.
...
### II. REST Contract First
Every API change MUST begin with an update to the REST contract.
...
## Governance
Amendments MUST be reviewed and recorded in the constitution before dependent work proceeds.

Spec Kit also keeps dependent templates in sync. Create the file once at the beginning of the project, then update it whenever your principles change.

Step 2: Specify (what and why, no technology):

/speckit-specify Develop a todo list application where a user can add a task,
mark it as done, delete it and filter tasks by status.
A task needs a non-empty title.
Completed tasks stay visible until they are deleted.
Out of scope for now: sharing tasks between users and notifications.

The generated spec contains four user stories: Add a Task, Mark a Task Done, Delete a Task, and Filter Tasks by Status. Its built-in requirements checklist has 16 items, all checked in this example. A few excerpts from the generated files:

<prj_root>/specs/001-todo-list/spec.md

## User Scenarios & Testing
### User Story 1 - Add a Task (Priority: P1)
...
## Requirements
- **FR-002**: The system MUST reject an empty or whitespace-only title...
...
## Success Criteria
- **SC-004**: Users can complete the add, complete, delete, and filter journeys on both phone and desktop form factors.

<prj_root>/specs/001-todo-list/checklists/requirements.md

## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows

The spec focuses on the what and why: user-facing behavior, goals, and outcomes, not the tech stack. The built-in checklist reviews the spec's quality and completeness before planning.

Step 3: Plan (how, the technology belongs here):

/speckit-plan Use an Angular web application, a REST API and a backend component (Python)
that holds the business logic, on top of a relational database (SQLite).
Data model: Task(id, title, status, createdAt).
Expose create, read, update and delete endpoints on /tasks.

This creates the implementation plan and four supporting design artifacts under <prj_root>/specs/001-todo-list/. The excerpts below are copied from those files.

plan.md (Most important file for technical requirements):

<prj_root>/specs/001-todo-list/plan.md
## Summary

Deliver an adaptive todo-list web application for adding, completing, filtering, and deleting tasks.
The Angular client uses a contract-first REST API; a Python FastAPI service owns validation and
business rules and persists tasks to a file-backed SQLite database through Python's standard
`sqlite3` module.

research.md (technical discovery file):

<prj_root>/specs/001-todo-list/research.md
### SQLite from Python

- **Decision**: Use Python 3.12's built-in `sqlite3` module with one connection per request, created
  and closed on the request's worker thread. Keep each mutation in a short transaction and commit
  only after all statements succeed; roll back on failure.

data-model.md (file used for data model):

<prj_root>/specs/001-todo-list/data-model.md
## Task

| Field | Type | Required | Rules |
|---|---|---:|---|
| `id` | UUID | Yes | Generated by the backend; unique and immutable. |
| `title` | String | Yes | Must contain at least one non-whitespace character. Valid submitted values are preserved. |

contracts/openapi.json (REST interface spec):

<prj_root>/specs/001-todo-list/contracts/openapi.json
  "paths": {
    "/tasks": {
      "get": {
        "operationId": "listTasks",
        "summary": "List tasks, optionally filtered by status",

quickstart.md (setup frontend and backend):

<prj_root>/specs/001-todo-list/quickstart.md
## Start the Backend

From the repository root in PowerShell:

```powershell
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e "backend[test]"
```

Step 4: Tasks (break the work down):

/speckit-tasks

No arguments here: the command reads the design artifacts and creates <prj_root>/specs/001-todo-list/tasks.md. This file contains 40 tasks: Setup (3), Foundational (8), Add a Task (7), Mark a Task Done (6), Delete a Task (6), Filter Tasks by Status (7), and Polish (3). Each story has a goal, an independent test, and its own test and implementation tasks. In this example, the tests are included because the constitution requires automated tests for every feature.

The file defines the task format and traceability labels:

<prj_root>/specs/001-todo-list/tasks.md
## Format: `[ID] [P?] [Story] Description`
- **[P]** marks work that touches separate files and has no dependency on an incomplete task.
- **[Story]** identifies the user story for story-phase tasks; setup, foundation, and polish tasks have no story label.
- Every task names the exact file or files to create or modify.

Here is its User Story 1 section. The [X] markers shown are the current state, after /speckit-implement completed the tasks:

## Phase 3: User Story 1 - Add a Task (Priority: P1, MVP)
**Goal**: Create a task from a non-empty title and show it in the All task list.
**Independent Test**: Submit a valid title and confirm exactly one active task with the submitted title is returned and displayed; empty and whitespace-only submissions create no task and show a title-required message.
### Tests for User Story 1
- [X] T012 [P] [US1] Add POST `/tasks` and default GET `/tasks` request/response contract tests, including 201, Task response fields, and 422 Problem Details, in `backend/tests/contract/test_create_task_contract.py`.

Parallel work and phase dependencies are also recorded in the same file:

### Parallel Opportunities
- Setup tasks T001-T003 touch separate backend, frontend, and root configuration files.
- Within each story, its three test tasks use separate contract, integration, and UI test files and can be authored in parallel before story implementation.

Before implementation, /speckit-analyze can check the consistency of spec, plan, and tasks; it is read-only and reports conflicts and gaps.

Step 5: Implement (build it):

/speckit-implement

This is the command that writes the code. It executes the tasks in tasks.md, running each phase in dependency order and respecting the parallel markers. Before starting, it reads the state of the checklists as a gate: if some items are unchecked it asks before proceeding, and it never changes the checklist markers. For a small feature like our todo list you can run it once and build everything. For a larger one, you can scope each run with an argument (for example only the Setup and Foundational phases), validate the result and then continue.

If you catch yourself writing "use PostgreSQL" in the specification, move it to the plan. This is the whole point: the spec says what, the plan says how. And it is exactly what makes me curious about changing only the plan later.

Step 6: Converge (check that nothing was missed):

/speckit-converge

This assesses the codebase against the spec, the plan and the tasks to confirm that nothing was missed. It is append-only: it never edits or deletes code, and its only possible write is adding tasks to tasks.md. You run it only after implement has run on the current tasks.md. It first prints a summary of the findings graded by severity, then ends in one of two ways:

Converged: no gaps found. tasks.md stays unchanged and you can move on to the review or the pull request.
Tasks appended: gaps found. The new tasks are added under a Convergence section of tasks.md, you run implement again to complete them and then converge once more, until it reports converged.

The Role of the Human

SDD does not take people out of the loop: I think it moves the place where their judgment is applied. Accepting generated changes with minimal resistance is just vibe coding with more files. Machines take over the mechanical work: generating code, keeping artifacts consistent, detecting drift. Humans keep authority over intent, domain meaning, risk tolerance and the legal and ethical constraints that no execution trace can reveal. Those constraints have to be written explicitly into the specification, which is where human judgment is encoded and then enforced.

In practice, for me this means explicit approval boundaries: a breaking schema change, a policy shift, a refactoring proposed by the AI or a compatibility downgrade should never pass without a person's decision. Spec Kit points in the same direction. Every phase produces an artifact you review before moving on, and in its own agentic SDLC tests and scans stay deterministic automation while merge decisions stay with the maintainers.

For the engineer, I believe, this changes where value comes from: no longer the speed of writing lines of code, but the quality of the specifications they refine and validate, as an orchestrator and validator who defines the what upstream and approves the how at the boundaries.

Mental Model: bounded autonomy, not full automation. Enforcement is automated, meaning stays human.

Takeaways

If I have to summarize my starting point in a few lines:

The specification is a contract. Each phase constrains the next, the spec stays the authority, and it is checked and not just read.
GitHub Spec Kit gives you the workflow: Specify -> Plan -> Tasks -> Implement -> Converge, with clarify, checklist and analyze as quality gates.
Humans move upstream, and their responsibility grows. People decide specifications, guidelines and architectures: engineers and architects evolve from writing code to owning what the system must do and why.
Keep a reality check. Less context drift, fewer hallucinations and no technical debt from chaotic prompting are the declared goals of SDD, not guaranteed results.

In the next articles I move from theory to practice: an adaptive Angular web application that talks over REST to a backend component holding the business logic, on top of a relational database, specified and built with Spec Kit. Then I challenge myself with a port of the backend to another technology, changing only the plan, to see how far "the same specification, different stack" really holds. The REST contract, the tests, the data model and the Angular application are my fixed points: if they stay untouched, the specification survived the move. The code will be on my GitHub (rosidotidev).

What has your experience with Spec-Driven Development been like? I'd love to hear your thoughts, questions, or feedback in the comments.

Github copilota spec

rosidotidev — Sat, 03 Oct 2026 06:42:49 +0000

Karpathy's LLM Wiki? No Code with Claude or Github Copilot!

rosidotidev — Sat, 23 May 2026 15:22:49 +0000

I first encountered Andrej Karpathy's LLM Wiki gist and realized it captures something essential about how we should manage knowledge in the AI era: a structured, human-curated knowledge base that grows from what you feed it, not from what an LLM hallucinates. The problem is that most implementations I found required heavy coding. I decided to build a completely no-code version that works seamlessly with both Claude Code and GitHub Copilot's .github/copilot-instructions.md system.

The result is llm-wiki-nocode: a production-ready knowledge management system where you ingest documents, ask questions, and maintain a living wiki, all through natural language commands, zero Python required.

What Is an LLM Wiki?

Before diving into the implementation, let me explain the concept. An LLM Wiki is a structured knowledge base that:

Ingests raw documents and extracts entities (tools, people, companies), concepts (patterns, ideas), and relationships
Retrieves facts exclusively from stored knowledge (no training data, no hallucinations)
Maintains integrity through automated health checks and human review
Compounds knowledge as you validate and refine auto-generated answers

The key principle is simple: everything the wiki knows comes from what you feed it. Not from the LLM's training data. Not from the web. Only from your sources.

This solves a real problem. When you ask an LLM a question about your proprietary system, it often confidently makes things up. A wiki-based approach forces it to admit "I found nothing" if the knowledge isn't there. No pollution. No false authority.

Why No-Code?

Most LLM Wiki implementations assume you have a Python environment, dependency management, vector databases, and patience for debugging integrations. That's the wrong barrier to entry.

I'm also building a production Python version using the Microsoft Agent Framework, structured around agentic workflows, knowledge graph construction, and semantic retrieval. But that's a separate project with its own architecture and tradeoffs. It will be the subject of a future article.

This project takes the opposite approach: zero dependencies, zero installation, pure natural language. No friction. Immediate usability.

What if you could ingest a document, ask questions, and maintain your knowledge base entirely through natural language? No terminals. No pip install. No YAML configuration. Just commands that read like English:

/wiki-ingest

/wiki-query What are the key architectural patterns?

/wiki-lint

This is what llm-wiki-nocode delivers. The system is designed to be human-operable: someone who has never written a line of code can build and maintain their own knowledge base.

The Three Core Operations

The wiki's entire interface consists of three commands. Everything else follows from these three:

1. `/wiki-ingest`, Build Your Knowledge Base

Ingest is where documents become knowledge. You point it at a raw file (Markdown, plain text, whatever), and it extracts entities, the named things like tools, frameworks, companies, people, and concepts, the abstract ideas like patterns, methodologies, principles. It detects relationships between them, merges new information with existing wiki pages, flags contradictions for your review, and updates the index and audit log in one pass.

You can pass a file path, use scan for auto-ingest, or --RESET-ALL if you want to start fresh. The output is new and updated pages in wiki/, plus a detailed log of what changed.

2. `/wiki-query`, Ask Questions Without Hallucination

Query is the retrieval side. You ask a question in natural language, and the system reads 3–8 relevant wiki pages and synthesizes an answer exclusively from that content. No external knowledge. No training data pollution. This is the core promise: every claim in the answer is cited with a wikilink showing exactly where it came from. The answer is saved to questions_pending/ for your review before it gets integrated as a synthesis page in the wiki.

3. `/wiki-lint`, Maintain Health

Lint is your wiki's health check. It runs in two phases: first, a deterministic pass that catches broken links, orphaned pages, missing frontmatter, and empty sections. Then a semantic phase that looks for contradictions, stale claims, missing pages, and unreferenced entities. The output is a detailed report with suggested fixes, saved to lint_pending/ for you to review and approve.

The Directory Structure

Understanding the layout is key to understanding how the wiki works:

wiki/                      ← Living knowledge base
├── index.md               ← Catalog of all pages
├── log.md                 ← Operation audit trail
├── sources/               ← One page per ingested document
├── entities/              ← Named things: tools, frameworks, companies
├── concepts/              ← Abstract ideas: patterns, methodologies
└── synthesis/             ← Approved answers from your questions

raw/                       ← Documents awaiting ingestion

questions_pending/         ← Auto-generated answers (your review)
questions_approved/        ← Answers you've validated

lint_pending/              ← Wiki health reports (your review)
lint_approved/             ← Fixes you've approved

docs/                      ← Specifications and guides

The key insight: wiki/ is the single source of truth. Everything else is a queue waiting for a human decision. raw/ is input. questions_pending/ and lint_pending/ are decisions waiting to be made. Once you approve something, it moves to _approved/ and gets integrated back into wiki/.

Three Page Types

The wiki uses three distinct page types to keep knowledge organized.

Sources are created when you ingest a document, one page per source file. Each source page contains the title, author, date, raw content, and a list of all entities and concepts extracted from that document. This is your audit trail: you know exactly which document something came from.

Entities are named, identifiable things. A tool like Jira or Kubernetes. A framework like React or MAF. A company. A person. Each entity page lists its definition, where it appears (which sources and concepts reference it), and related entities. This creates a web of interconnected knowledge.

Concepts are abstract ideas and patterns. A methodology like microservices or CQRS. A principle like DRY or YAGNI. A design pattern. Each concept page includes an explanation, examples (always with source citations), and connections to related concepts and entities. Concepts are how you see patterns across your sources.

How It Works: The Human-Gated Loop

This is the part that makes llm-wiki-nocode different from other implementations. The system never makes 100 auto-decisions and leaves you to inherit the consequences. Instead, it follows a human-gated approval loop.

The system generates, ingests a document, answers a question, finds health issues. Then you review what was generated, looking at the _pending/ folders. You decide which auto-generated content is good enough to integrate, and which needs refinement or rejection. Once you approve, the content moves to _approved/ and gets integrated back into the living wiki.

This loop is non-negotiable. The system assists; you decide. You never wake up one morning and discover the wiki made a thousand bad decisions overnight.

Automated generation    →    Human review    →    Integration
         ↓                        ↓                      ↓
   @wiki-querier          questions_pending/      wiki/synthesis/
   @wiki-linter           lint_pending/           wiki pages updated

Getting Started: A Practical Example

The repo includes three ready-made documents in the bak/ directory for testing. There's afw-instructions.md, technical documentation about Microsoft Agent Framework best practices. mfa_agents_howto.md is a how-to guide for building agents. And ARCH_BOOKING_PLATFORM.md is a simulated system architecture document. These examples show how the wiki handles diverse source material.

To test the wiki, copy one of these files to raw/ and run ingest:

cp bak/ARCH_BOOKING_PLATFORM.md raw/
/wiki-ingest

The ingestor will parse the architecture doc, extract entities like services, technologies, and teams, pull out concepts like "microservices", "event-driven", and "CQRS", create pages for each in wiki/entities/ and wiki/concepts/, and log every change in wiki/log.md.

Next, ask it a question:

/wiki-query What are the core services in the booking platform?

The querier will search the wiki for relevant pages, answer using only what's there, cite sources with wikilinks, and save the answer for your review.

Finally, run a health check:

/wiki-lint

The linter will check for broken links, orphaned pages, contradictions, and suggest fixes. You review the report, decide which fixes to apply, and the wiki improves itself.

The Configuration Layer: `.claude/` and `.github/`

This is a critical design decision that deserves explanation.

`.claude/` Directory, Claude Code Configuration

The .claude/ folder contains the complete configuration for Claude Code:

.claude/
├── CLAUDE.md                  ← Project instructions and behavioral constraints
├── settings.json              ← Command registration and harness configuration
└── commands/
    ├── wiki-ingest.md         ← /wiki-ingest command workflow definition
    ├── wiki-query.md          ← /wiki-query command workflow definition
    └── wiki-lint.md           ← /wiki-lint command workflow definition

How it works: CLAUDE.md is the main system prompt that defines the wiki's identity and available commands. settings.json registers each command and maps it to a workflow. Each file in commands/ is a complete specification, it tells Claude exactly how to perform that operation: what to read, what to extract, what format to use, what to validate.

When you type /wiki-ingest in Claude Code, it reads commands/wiki-ingest.md and executes that workflow.

`.github/` Directory, GitHub Copilot Configuration

Similarly, .github/ contains GitHub Copilot-specific instructions:

.github/
├── prompts/
│   ├── wiki-ingest.prompt.md   ← LLM prompt for ingest command
│   ├── wiki-query.prompt.md    ← LLM prompt for query command
│   └── wiki-lint.prompt.md     ← LLM prompt for lint command
├── instructions/
│   └── wiki-schema.instructions.md  ← Shared page format and schema documentation
└── agents/
    ├── wiki-ingestor.agent.md  ← Ingestor agent definition
    ├── wiki-querier.agent.md   ← Querier agent definition
    └── wiki-linter.agent.md    ← Linter agent definition

GitHub Copilot reads these files automatically when you open the repo. The prompts/ folder contains the LLM instructions (analogous to Claude's commands/). The agents/ folder defines how each agent behaves. The instructions/ folder holds shared schema documentation so both Claude Code and Copilot understand the page format identically.

Mental Model: Think of .claude/ and .github/ as platform-specific adapters. The core logic (what to extract, how to format, what to validate) is shared through wiki-schema.instructions.md. But the way you invoke commands, the syntax, and how each LLM system structures its workflows differs between Claude Code and GitHub Copilot.

This dual-support approach means:

If you use Claude Code: Open the project, type /wiki-ingest, Claude reads .claude/commands/wiki-ingest.md
If you use GitHub Copilot: The repo integrates automatically, Copilot reads .github/prompts/wiki-ingest.prompt.md and uses .github/agents/wiki-ingestor.agent.md
If you use neither: You can still read all the files and understand exactly what each operation does

No core logic is duplicated. Both versions implement the same three operations with identical semantics, they just speak different dialects.

Key Design Choices

Working through the full implementation revealed several important patterns worth sharing.

The wiki uses retrieval-only queries, not fine-tuning. It doesn't train or modify the LLM. Instead, it uses semantic search to find relevant pages and asks the LLM to synthesize an answer from only those pages. This is both simpler to implement and more reliable than RAG with fine-tuning.

Entities and concepts are fundamentally different, and treating them that way matters. A tool like Jira or PostgreSQL is an entity, a concrete thing you can point to. A pattern like CQRS or event sourcing is a concept, an idea that can apply in many contexts. When the wiki understands this distinction, you can ask "What concepts relate to Kubernetes?" and get a meaningful, connected answer.

Human review before integration is non-negotiable. A single document ingest can generate hundreds of pages. If you didn't review them, you'd have garbage in your wiki. The _pending/ folders force a decision point. It's cheap human attention spent upfront that saves you from debugging corrupted data later.

The audit trail is mandatory. Every operation, every ingest, query, lint, is logged with timestamps, the files it touched, and what changed. This is how you answer "When did I ingest that architecture doc?" and "What exactly did that ingest change?" questions months later when your memory is fuzzy.

Finally, the wiki uses plain Markdown with simple YAML frontmatter. No proprietary format. No database. Everything is human-readable, editable, and version-controlled. You can throw it in Git, diff it, and understand what changed without needing special tools.

What's Not Included (Yet)

This is a no-code system for knowledge ingestion and retrieval. It doesn't include vector databases or semantic search, you don't need them at small scale, and simple keyword plus fuzzy matching works fine for document lookup. If you scale large, that's a future architectural decision you can make then.

The wiki doesn't fine-tune or train the model. It works with any LLM that supports Claude Code or Copilot integrations, so you're not locked into a single backend.

Web ingestion isn't built in. You ingest documents from files. If you want to ingest web pages, you'd scrape them to Markdown first and place them in raw/.

Multi-user collaboration is deferred. The system assumes a single human curator making review and approval decisions. Role-based approval and conflict resolution for teams are possible, but not yet implemented.

None of these are impossible, they're just outside the current scope, waiting for a real use case to drive them.

Takeaways

Building llm-wiki-nocode clarified something important: the most powerful knowledge systems are simple systems that humans control. The wiki doesn't make decisions. It generates, you review, you decide.

If you're drowning in documentation that lives in Slack threads, scattered Markdown files, or worse, in everyone's heads, this pattern offers a way out. Feed documents to the wiki. Ask questions. Maintain a single source of truth. All without leaving your editor.

The code is on GitHub (rosidotidev/llm-wiki-nocode). The specifications are in docs/. Start with a simple document, ingest it, ask a question, and see what happens.

What would you use a no-code LLM wiki for? Is knowledge management the bottleneck in your projects, or is something else? I'd love to hear your thoughts in the comments.

Microsoft Agent Framework: From Agents to Workflows, Five Iterations of a DOCX-to-Markdown Pipeline

rosidotidev — Sun, 26 Apr 2026 12:35:17 +0000

In my previous article, I built a multi-agent pipeline that reads a Markdown backlog and provisions Epics and Stories on Jira. That project used two agents collaborating sequentially, wired together with plain Python orchestration.

This time I wanted to understand something specific: does the Microsoft Agent Framework (MFA) have a construct for defining execution graphs, similar to what LangGraph offers with its StateGraph or CrewAI with its Flow? In other words, can you define nodes, edges, and let the framework handle execution order, state propagation, and event streaming? To find out, I needed a use case that was practical enough to be reusable, complex enough to stress the graph model, and different enough from the Jira pipeline to force me into new territory.

I landed on converting Word documents into rich Markdown files with AI-generated image descriptions. The kind of structured, image-heavy documents (functional specifications, architecture documents, technical guides) that are notoriously hard to index in a RAG system because the images carry as much information as the text. A useful asset in itself, and a good bench for a multi-step pipeline.

MFA Workflows in 30 Seconds

The answer to my question turned out to be yes. The framework provides a Workflow system built on three primitives:

Executor: a processing node. You subclass it, implement a @handler method, and it receives typed input and produces output. An executor can wrap an agent, call an API directly, or run pure Python, the framework doesn't care.
WorkflowBuilder: the graph constructor. You declare a start node, then chain .add_edge(source, target) calls to define the execution path.
Edges: directed connections between executors. Data flows along edges via ctx.send_message(), while ctx.set_state() / ctx.get_state() provide a shared store accessible by any node in the graph.

When you call workflow.run(input, stream=True), the framework walks the graph and emits events (executor_invoked, executor_completed, output) that give you real-time visibility into each step.

With these primitives in hand, I could start building.

The idea behind the use case is simple: if you can turn every image into a detailed textual description embedded directly in the Markdown, a future RAG pipeline can "see" diagrams, screenshots, and flowcharts without needing a multimodal model at query time. The image content becomes searchable text.

What started as a straightforward agent-based pipeline evolved through five iterations into something quite different: a workflow-based, LLM-minimal architecture where Python does 90% of the work and the LLM is called only for what it's genuinely good at, describing images.

All the code is available on GitHub (MSFTAgentSample).

The Use Case

Given a folder of .docx files, the pipeline should:

Extract all text, tables, and images from each document, preserving the logical order and tracking where each image appears in the text flow.
Describe every extracted image using GPT-4o vision, producing thorough, structured descriptions that capture every visible element.
Assemble a clean Markdown document where headings, paragraphs, tables, and images are rendered correctly, with each image followed by its AI-generated description.
Write the final .md files to disk, ready for indexing or publishing.

The output looks like this:

## 6. Functional Requirements

The system shall provide a task management interface...

![Image](./images/img-003-a1b2c3d4.png)
**[Image Description]:** This is a screenshot of a UI mockup showing
a task management dashboard with three columns: To Do, In Progress,
and Done. Each column contains card-like elements with task titles...
**[End Image Description]**

The Starting Point: Agents Only (v1)

The first version followed the same pattern as the Jira pipeline: sequential agents, no workflow. Three agents collaborating in a chain:

DocExtractorAgent → ImageAnalystAgent → ContentAssemblerAgent

Each agent wraps a specific capability:

DocExtractorAgent calls a @tool function that uses unstructured and python-docx to extract text segments and images from .docx files. The output is a structured JSON with text_segments (typed: Title, NarrativeText, Table, ListItem, image) and an images array with file paths and surrounding context.
ImageAnalystAgent receives the extraction JSON, collects all image references, and calls a describe_images tool that sends each image to GPT-4o-mini vision. The vision prompt is thorough: it asks for image type identification, element-by-element description, all visible text transcription, relationships and flow, and layout details.
ContentAssemblerAgent takes the extraction data plus the image descriptions and produces the final Markdown.

The orchestration is plain Python, same as before:

response1 = await extractor_agent.run(f'input_dir="{INPUT_DIR}", output_dir="{OUTPUT_DIR}"')
response2 = await analyst_agent.run(response1.text)
response3 = await assembler_agent.run(
    f"extraction:\n{response1.text}\n\ndescriptions:\n{response2.text}\n\noutput_dir=\"{OUTPUT_DIR}\""
)

This worked, but several problems were immediately visible.

What Broke and Why

The LLM forgot to close tags. The ContentAssemblerAgent was instructed to wrap each image description in **[Image Description]:** and **[End Image Description]** tags. But on long documents, the LLM would sometimes omit the closing tag, breaking downstream parsing.

No observability. With plain agent.run() calls, you only see the final output. You don't know which agent is running, how long each step took, or where a failure occurred.

Sequential image analysis was slow. Each image was sent to GPT-4o-mini one at a time. A document with 10 images meant 10 sequential API calls, each taking 3-5 seconds.

Scalability was questionable. The ContentAssemblerAgent received the entire extraction JSON in its prompt. For a 50-page document, that JSON could exceed the model's context window, causing truncation or quality degradation.

These issues led me to explore the Workflow feature of the Microsoft Agent Framework.

Introducing Workflows (v2)

With the workflow primitives I described above, the first step was straightforward: wrap each agent inside an executor. Here's what a minimal executor looks like:

from agent_framework import Executor, WorkflowContext, handler

class MyExecutor(Executor):

    def __init__(self):
        super().__init__(id="my_executor")

    @handler
    async def handle(self, input_data: str, ctx: WorkflowContext[str]) -> None:
        result = do_something(input_data)
        await ctx.send_message(result)  # forward to next executor

Wire them up, and you get a graph with event streaming out of the box:

workflow = (
    WorkflowBuilder(start_executor=executor_a)
    .add_edge(executor_a, executor_b)
    .add_edge(executor_b, executor_c)
    .build()
)

async for event in workflow.run(input_data, stream=True):
    if event.type == "executor_invoked":
        print(f"Starting {event.executor_id}...")
    elif event.type == "executor_completed":
        print(f"Done: {event.executor_id}")
    elif event.type == "output":
        print(f"Result: {event.data}")

For v2, the graph was:

DocExtractorExecutor ──▶ ImageAnalystExecutor ──▶ ContentAssemblerExecutor

One thing worth noting: besides edge-based data flow (ctx.send_message), executors can also use shared state (ctx.set_state / ctx.get_state). This lets a node store data that any downstream node can read directly, without it having to pass through intermediate nodes.

For example, the DocExtractor saves the extraction result to state so that the ContentAssembler (two steps later) can read it directly, without the data having to flow through the ImageAnalyst:

class DocExtractorExecutor(Executor):

    @handler
    async def handle(self, prompt: str, ctx: WorkflowContext[str]) -> None:
        response = await self._agent.run(prompt)
        ctx.set_state("extraction", response.text)  # shared state
        await ctx.send_message(response.text)        # edge to next executor

Same functionality as v1, but now with event streaming and a clear graph structure.

The Template + Filler Pattern (v3)

The closing-tag problem from v1 persisted in v2, because the ContentAssembler was still an LLM generating the full Markdown including image descriptions. LLMs are inherently unreliable for repetitive structural tasks like "always close every tag."

The solution was to split the assembly into two steps:

TemplateAssembler (LLM), generates the Markdown structure with  HTML comment placeholders instead of actual descriptions.
TemplateFiller (pure Python), reads the image descriptions from state and replaces each placeholder with the formatted description using a regex.

The workflow became a 4-node graph:

DocExtractor ──▶ ImageAnalystStateful ──▶ TemplateAssembler ──▶ TemplateFiller

The TemplateFiller is completely deterministic:

_PLACEHOLDER_RE = re.compile(r"<!--IMG:([\w.\-]+)-->")

def _replace(match: re.Match) -> str:
    fname = match.group(1)
    replacement = desc_map.get(fname)
    if replacement is None:
        return match.group(0)  # leave placeholder as-is
    return replacement

filled = _PLACEHOLDER_RE.sub(_replace, content)

No LLM, no token cost, no randomness. The closing tag problem disappeared because the Python code always writes both the opening and closing tags.

This was the turning point. It made me ask: where else am I using an LLM for something Python can do better?

Scaling Up (v4): Parallel Vision + Chunked Assembly

With the architecture cleaned up, two scaling bottlenecks remained:

Image analysis was sequential. The ImageAnalyst agent called the describe_images tool, which internally looped through images one at a time. For N images, that's N sequential API calls.

The fix: a new ParallelImageAnalystExecutor that bypasses the agent entirely and calls AsyncOpenAI directly with asyncio.gather, bounded by a semaphore:

class ParallelImageAnalystExecutor(Executor):

    def __init__(self):
        super().__init__(id="parallel_image_analyst")

    @handler
    async def handle(self, extraction_json: str, ctx: WorkflowContext[str]) -> None:
        aclient = AsyncOpenAI()
        semaphore = asyncio.Semaphore(MAX_CONCURRENT)

        tasks = [
            _describe_image_async(aclient, img["path"], img.get("context", ""), semaphore)
            for img in all_images
        ]
        results = await asyncio.gather(*tasks)

        result_json = json.dumps({"images": list(results)}, ensure_ascii=False)
        ctx.set_state("descriptions", result_json)
        await ctx.send_message(result_json)

With MAX_CONCURRENT=5 (configurable via environment variable), 10 images now take ~5 seconds instead of ~40.

Template assembly could exceed context window. The TemplateAssembler received the entire document extraction in a single prompt. For large documents, this meant prompt truncation or degraded output quality.

The fix: a ChunkedTemplateAssemblerExecutor that splits the document's text_segments at heading boundaries (using the Title type from unstructured), processes each chunk with a separate LLM call, and concatenates the results:

def _chunk_segments(text_segments: list[dict]) -> list[list[dict]]:
    chunks: list[list[dict]] = []
    current: list[dict] = []

    for seg in text_segments:
        if seg.get("type") in {"Title", "Header"} and current:
            chunks.append(current)
            current = []
        current.append(seg)

    if current:
        chunks.append(current)
    return chunks

Each LLM call sees only one section (~5-15 paragraphs), keeping token usage low and quality consistent regardless of document length.

Going LLM-Minimal (v5): The Final Architecture

At this point I stepped back and asked: do I actually need an LLM for any of the remaining steps?

The DocExtractor

The DocExtractor agent received a prompt, called the extract_docx_files tool, and returned the tool's JSON output unchanged. The agent was a pure passthrough, it added zero value over calling the Python function directly.

The replacement is trivial:

from afw_core.tools.docx_extractor import _extract_all_docx

class DocExtractorDirectExecutor(Executor):

    def __init__(self, input_dir: str, output_dir: str):
        super().__init__(id="doc_extractor_direct")
        self._input_dir = input_dir
        self._output_dir = output_dir

    @handler
    async def handle(self, prompt: str, ctx: WorkflowContext[str]) -> None:
        results = _extract_all_docx(
            Path(self._input_dir), Path(self._output_dir),
        )
        result_json = json.dumps(results, ensure_ascii=False)
        ctx.set_state("extraction", result_json)
        await ctx.send_message(result_json)

One LLM call eliminated.

The TemplateAssembler

This is where I had to think harder. The LLM was converting text_segments JSON into Markdown. The segments already had type information:

{"type": "Title", "text": "6.1 Functional Overview"}
{"type": "NarrativeText", "text": "The system shall provide..."}
{"type": "Table", "text": "| Col1 | Col2 |..."}
{"type": "image", "text": "[IMAGE: img-003-a1b2c3d4.png]"}
{"type": "ListItem", "text": "Enable a user to capture a task in less than 10 seconds"}

I investigated what metadata unstructured provides. It turned out that Title elements have a category_depth field (0 for top-level headings, 1 for sub-headings), and heading numbering patterns (6.1, 6.1.2) can be used to infer deeper nesting.

The entire Markdown assembly became a for loop with type-based rules:

def _segments_to_markdown(text_segments: list[dict]) -> str:
    lines: list[str] = []
    prev_type: str | None = None

    for seg in text_segments:
        seg_type = seg.get("type", "")
        text = seg.get("text", "").strip()
        if not text:
            continue

        if prev_type == "ListItem" and seg_type != "ListItem":
            lines.append("")

        if seg_type == "Title":
            prefix = _heading_prefix(text)  # ## / ### / #### based on numbering
            lines.append(f"{prefix} {text}")

        elif seg_type == "ListItem":
            lines.append(f"- {text}")

        elif seg_type == "Table":
            lines.append(text)  # already Markdown from extractor

        elif seg_type == "image":
            fname = _extract_filename(text)
            lines.append(f"![Image](./images/{fname})")
            lines.append(f"<!--IMG:{fname}-->")

        else:  # NarrativeText, Text, etc.
            lines.append(text)

        prev_type = seg_type

    return "\n".join(lines)

Deterministic, instantaneous, zero cost. And the heading hierarchy is actually more consistent than what the LLM produced, because it follows strict rules rather than "inferring" structure.

Many more LLM calls eliminated.

The Final Workflow Graph

DocExtractorDirect ──▶ ParallelImageAnalyst ──▶ DeterministicAssembler ──▶ TemplateFillerMulti
   (Python)               (async Vision)             (Python)                  (Python)

LLM usage: only the N parallel vision calls for image descriptions. Everything else is pure Python.

The Workflow Builder

The wiring is clean and concise:

from agent_framework import WorkflowBuilder

def build_doc_ingest_tpl_multi_opt_workflow(input_dir: str, output_dir: str):
    extractor = DocExtractorDirectExecutor(input_dir, output_dir)
    analyst = ParallelImageAnalystExecutor()
    assembler = DeterministicAssemblerExecutor()
    filler = TemplateFillerMultiExecutor()

    return (
        WorkflowBuilder(start_executor=extractor)
        .add_edge(extractor, analyst)
        .add_edge(analyst, assembler)
        .add_edge(assembler, filler)
        .build()
    )

Notice: no client or options parameter. Since no executor uses an agent, there's no LLM client to inject. The ParallelImageAnalyst creates its own AsyncOpenAI client internally because it needs vision, not chat.

The Entry Point

The entry point consumes the workflow's event stream:

async def main():
    workflow = build_doc_ingest_tpl_multi_opt_workflow(INPUT_DIR, OUTPUT_DIR)

    async for event in workflow.run(
        f'input_dir="{INPUT_DIR}", output_dir="{OUTPUT_DIR}"',
        stream=True,
    ):
        if event.type == "executor_invoked":
            label = {
                "doc_extractor_direct": "📖 Extracting documents (direct)",
                "parallel_image_analyst": "🔍 Analysing images (parallel)",
                "deterministic_assembler": "📝 Assembling Markdown (deterministic)",
                "template_filler_multi": "🔧 Filling image descriptions",
            }.get(event.executor_id, event.executor_id)
            print(f"\n{label}...")

        elif event.type == "executor_completed":
            print(f"   ✓ {event.executor_id} completed")

        elif event.type == "output":
            assembled = AssembledDocumentList.model_validate_json(event.data)
            for doc in assembled.documents:
                print(f"   ✅ {doc.source_file} → {doc.output_path}")

Evolution Summary

Version	Entry Point	Architecture	LLM Calls	Key Change
v1	`main_doc_ingest.py`	Sequential agents	3 per file	Baseline
v2	`main_doc_ingest_wfl.py`	Workflow (3 executors)	3 per file	Graph-based, event streaming
v3	`main_doc_ingest_wfl_tpl.py`	Workflow (4 executors)	2 + N images	Template + filler pattern
v4	`main_doc_ingest_wfl_tpl_multi.py`	Workflow (4 executors)	N parallel + M chunks	Parallel vision, chunked assembly
v5	`main_doc_ingest_wfl_tpl_multi_opt.py`	Workflow (4 executors)	N parallel only	LLM-minimal: Python does the rest

The cost reduction from v1 to v5 is dramatic. For a document with 5 images and 10 sections, v1 makes ~8 LLM calls (3 agents). V5 makes exactly 5 (one per image, in parallel). The extraction and assembly are free.

Key Lessons Learned

Executors don't have to wrap agents. The framework's Executor class is a general-purpose processing unit. It can wrap an agent, call an API directly, or run pure Python logic. The best executors are often the ones with no LLM at all.

The template + filler pattern is widely applicable. Whenever you need an LLM to generate structured output with repetitive elements (tags, placeholders, formatting), consider splitting the task: let the LLM generate the structure, and let Python fill in the repetitive parts deterministically.

asyncio.gather with a semaphore is the right pattern for parallel API calls. It's simple, it respects rate limits, and it keeps you in control of concurrency. No need for complex job queues or thread pools.

unstructured gives you more metadata than you think. Before deciding to use an LLM for heading hierarchy, I checked what unstructured already provides. It turned out it has category_depth, parent_id, and typed elements (Title, ListItem, NarrativeText, Table), enough to build a complete Markdown document without any AI involvement.

Workflow versioning via file suffixes works well. Each iteration is a separate file (doc_ingest.py, doc_ingest_tpl.py, doc_ingest_tpl_multi.py, doc_ingest_tpl_multi_opt.py) with its own entry point. Previous versions are preserved for comparison and rollback. No git archaeology needed.

Conclusion: Takeaways

There is an old saying: "When all you have is a hammer, everything looks like a nail." Working with LLMs can feel exactly like that. You have this incredibly powerful tool that can understand language, generate text, analyze images, and it becomes tempting to route every step through it. Extraction? Let the LLM handle it. Assembly? The LLM can figure out the Markdown. Heading hierarchy? Surely the LLM knows best.

The single biggest takeaway from this project is that the best LLM call is the one you don't make. Before building v5, I assumed the extraction and assembly steps "needed" an LLM because they involved understanding document structure. They didn't. The extraction was already a Python tool behind an agent wrapper. The assembly was a deterministic mapping from typed segments to Markdown. The LLM added cost, latency, and randomness without adding value.

Stripping away those unnecessary calls didn't just save tokens. It made the pipeline faster, more predictable, and easier to debug. When something goes wrong in a pure Python executor, you get a stack trace, not a hallucination.

The MFA workflow model made this evolution natural. Because each step is an independent executor wired into a graph, I could replace an agent-based node with a pure Python node without touching anything else. The framework doesn't push you toward LLMs, it gives you the scaffolding and lets you decide what goes inside each node. That turned out to be exactly the right level of abstraction.

The pattern of wrapping agents inside executors and passing data through shared state also proved valuable for reuse. The same agent can be plugged into different workflows without modification, because the executor handles the wiring and the state handles the context. This, combined with a clean, well-defined workflow API, made it straightforward to evolve the solution incrementally, from a naive agent chain all the way to an aggressively optimized pipeline, without ever having to rewrite from scratch.

If the first article was about discovering the framework's building blocks, this one was about learning when not to use them. Use the LLM where it genuinely shines, like describing images. For everything else, write Python.

The generated Markdown files, with their embedded image descriptions, are ready for a RAG pipeline. A simple chunking strategy (split on headings, keep image descriptions with their context) would make both text and image content searchable through standard text embeddings, no multimodal retrieval model required.

All the code is available on GitHub (MSFTAgentSample). The PIPELINE_DOCX_TO_MD.md file documents all five variants with setup instructions and usage commands.

Microsoft Agent Framework: From Zero to Multi-Agent Pipeline

rosidotidev — Sun, 19 Apr 2026 09:53:25 +0000

I have some background with other agent frameworks like CrewAI and LangGraph, so when Microsoft released the Agent Framework, a lightweight Python package for building AI agents with native MCP (Model Context Protocol) support, I was curious to give it a try. I decided to build something practical: a pipeline that reads a product backlog from a Markdown file and automatically creates Epics and Stories on Jira. I chose this specific use case because I had already implemented it with CrewAI, so I was familiar with the configuration setup and could focus on comparing the frameworks rather than figuring out the integration details from scratch.

As reported in the official documentation, the Microsoft Agent Framework is the direct successor of both Semantic Kernel and AutoGen, created by the same Microsoft teams. It combines AutoGen's simple abstractions for single- and multi-agent patterns with Semantic Kernel's enterprise-grade features like session-based state management, type safety, telemetry, and extensive model support. On top of that, the Microsoft Agent Framework introduces workflows for explicit control over multi-agent execution paths and a robust state management system for long-running and human-in-the-loop scenarios.

What I found was a framework that favors simplicity and explicitness. You write Python functions, you wire them together, and you stay in control of the flow. In this article, I walk through the incremental approach I followed, from an "hello world" agent to a fully modular multi-agent pipeline.

You can find all the code shown in this post on this GitHub repo (MSFTAgentSample).

What I Used So Far

I have only scratched the surface of the framework, but here are the building blocks I worked with in this project:

Agent: the core class. You give it a name, instructions, a chat client, and a list of tools. It runs autonomously, deciding which tools to call and when to stop.
OpenAIChatClient: one of the available LLM providers. The framework integrates with most major LLMs, but for simplicity I used OpenAI since I still had some tokens to spend :-).
MCPStdioTool: a bridge to any MCP server. Point it at a command and it auto-discovers all available tools via the MCP protocol.
@tool: a decorator to turn any Python function into a tool the agent can invoke.

There is certainly more to explore, but these four primitives were enough to build a fully working multi-agent pipeline.

Step 1: Hello World, One Agent, No Tools

The very first thing I did was verify that the framework works. The simplest possible setup: one agent, one LLM client, one hardcoded query, no tools at all.

from __future__ import annotations
import asyncio
import os

from agent_framework import Agent
from agent_framework.openai import OpenAIChatClient, OpenAIChatOptions
from dotenv import load_dotenv


async def run_manager_agent(query: str) -> None:
    client: OpenAIChatClient = OpenAIChatClient(
        api_key=os.environ.get("OPENAI_API_KEY", ""),
        model=os.environ.get("OPENAI_CHAT_MODEL", "gpt-4o-mini"),
    )

    agent: Agent[OpenAIChatOptions] = Agent(
        client=client,
        name="ManagerAgent",
        instructions=(
            "You are a manager agent. Answer the user's query "
            "as accurately and concisely as possible."
        )
    )

    result = await agent.run(query)
    print(result.text)


def main() -> None:
    load_dotenv()
    query = "What is a Large Language Model and how does it work?"
    asyncio.run(run_manager_agent(query))


if __name__ == "__main__":
    main()

Mental Model: This is the equivalent of a "print hello world" in the agent framework world. You create a client, create an agent, call agent.run(), and print the result. Everything is async, so you need asyncio.run() as the entry point. The .env file provides the API key and model name via python-dotenv.

Notice how explicit everything is. There is no magic configuration, no auto-discovery. You pass the API key, you choose the model, you write the instructions. The agent's identity is fully defined by a single instructions string.

Step 2: Adding an MCP Tool (Jira)

Once the basics worked, the next step was connecting the agent to the real world. The Microsoft Agent Framework has first-class support for MCP (Model Context Protocol), which is the standard for exposing tools to AI agents. The mcp-atlassian package provides a full MCP server for Jira and Confluence.

import asyncio
import os
from dotenv import load_dotenv
from agent_framework import Agent, MCPStdioTool
from agent_framework.openai import OpenAIChatClient, OpenAIChatOptions

load_dotenv()

async def main():
    # MCP Proxy: auto-discovers all Jira tools via MCP protocol
    jira_proxy = MCPStdioTool(
        name="jira_server",
        command="pipenv",
        args=["run", "mcp-atlassian"],
        env={
            "JIRA_URL": os.getenv("JIRA_URL"),
            "JIRA_USERNAME": os.getenv("JIRA_USERNAME"),
            "JIRA_API_TOKEN": os.getenv("JIRA_API_TOKEN"),
        },
    )

    client = OpenAIChatClient(
        api_key=os.environ.get("OPENAI_API_KEY", ""),
        model=os.environ.get("OPENAI_CHAT_MODEL", "gpt-4o-mini"),
    )
    options = OpenAIChatOptions(temperature=0.0)

    jira_agent = Agent(
        name="JiraManagerAgent",
        instructions=(
            "You are a professional Project Management Assistant. "
            "You have direct access to Jira via integrated tools. "
            "Your goal is to help users manage tickets, track progress, "
            "and create issues."
        ),
        client=client,
        default_options=options,
        tools=[jira_proxy],
    )

    print("Jira Manager Agent is online...")

    user_query = """
        IMPORTANT: Execute each step below ONE AT A TIME.
        Step 1: Create an epic in the SARI project called 'Shopping List'.
        Step 2: Create a story: 'Story 1: Shopping List CRUD Angular UI' 
                and set the epic as parent.
        Step 3: Create a story: 'Story 2: Shopping List CRUD Angular 
                in memory mocked service' and set the epic as parent.
        Create issues one at a time, never in parallel.
    """

    try:
        response = await jira_agent.run(user_query)
        print("\nAgent Response:")
        print(response.text)
    finally:
        await jira_proxy.close()

if __name__ == "__main__":
    asyncio.run(main())

The key piece here is MCPStdioTool. You point it at a command (pipenv run mcp-atlassian), pass the necessary environment variables, and the framework auto-discovers every tool the MCP server exposes: jira_create_issue, jira_search, jira_get_issue, jira_link_to_epic, and many more. The agent sees all of them and decides which ones to call based on your query.

A Hard Lesson: Parallel Tool Calls

This step is where I hit my first real problem. When asked to create an epic and two stories, the agent would sometimes send multiple jira_create_issue calls in parallel. The second call would fail with a cryptic error: expected 'key' property to be a string. After adding debug logging and investigating, I discovered that the MCP server cannot handle parallel tool calls reliably.

The fix was surprisingly simple: tell the agent explicitly in its instructions to "Create issues ONE AT A TIME, never in parallel." This is a pattern I now apply consistently. If your MCP server doesn't handle concurrency well, just instruct the agent accordingly. It respects the instruction.

Step 3: Two-Agent Pipeline (Monolithic)

With the Jira integration working, I wanted to build something more structured: a pipeline with two agents collaborating sequentially. The idea was simple:

BacklogReaderAgent reads a Markdown backlog file from disk
JiraExecutorAgent takes the backlog content and creates all issues on Jira

To give agents the ability to read and write files, I used the @tool decorator to create custom function tools:

from typing import Annotated
from agent_framework import tool

@tool
def read_file(file_name: Annotated[str, "Name of the file to read"]) -> str:
    """Read and return the contents of a file from the input directory."""
    path = os.path.join("input", file_name)
    with open(path, "r", encoding="utf-8") as f:
        return f.read()

@tool
def write_file(content: Annotated[str, "The content to write"]) -> str:
    """Write content to a timestamped file in the output directory."""
    os.makedirs("output", exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M")
    file_name = f"execution_result_{timestamp}.md"
    path = os.path.join("output", file_name)
    with open(path, "w", encoding="utf-8") as f:
        f.write(content)
    return f"File written: {path}"

The Annotated[str, "description"] syntax is how you document parameters for the agent. The framework reads these annotations and exposes them as part of the tool schema, so the LLM knows what to pass.

Then, the two agents and the orchestration logic, all in one file:

    # Agent 1: reads the backlog file
    reader_agent = Agent(
        name="BacklogReaderAgent",
        instructions=(
            "You are a backlog reader assistant. "
            "When asked, use the read_file tool to read a markdown file. "
            "Return the full contents of the file as-is."
        ),
        client=client,
        default_options=options,
        tools=[read_file],
    )

    # Agent 2: executes the backlog on Jira
    executor_agent = Agent(
        name="JiraExecutorAgent",
        instructions=(
            "You are a Jira execution assistant. "
            "Create issues ONE AT A TIME, never in parallel. "
            "After all operations, write a summary using write_file."
        ),
        client=client,
        default_options=options,
        tools=[jira_proxy, write_file],
    )

    # Orchestration: sequential pipeline
    read_response = await reader_agent.run(
        f"Read the file '{backlog_file}' and return its contents."
    )
    backlog_content = read_response.text

    exec_response = await executor_agent.run(
        f"Execute the following backlog on Jira.\n\n{backlog_content}"
    )

Mental Model: Notice that the orchestration is plain Python. There is no pipeline abstraction, no DAG. You call agent.run(), get the result, and pass it to the next agent. You are the orchestrator.

The backlog file is a simple Markdown document placed in the input/ directory:

# Weather Dashboard Backlog - SARI Project

## Epic: Weather Dashboard
- Type: Epic
- Description: Real-time weather dashboard application.

### Stories

- **Story 1: City Search and Autocomplete**
  - Type: Story
  - Description: Implement a search bar with autocomplete...

- **Story 2: Current Weather Display**
  - Type: Story
  - Description: Show current weather conditions...

The agent reads this, understands the structure, and creates the epic first, then each story linked to the epic as parent. All on Jira, automatically.

Step 4: Modular Pipeline with Pydantic Validation

The monolithic version worked perfectly, but everything was in one file. For a production-ready layout, I refactored the code into a well-structured directory:

MSFTAgentSample/
├── afw_core/
│   ├── agents/
│   │   ├── backlog_reader.py
│   │   └── jira_executor.py
│   ├── tools/
│   │   ├── file_reader.py
│   │   └── file_writer.py
│   ├── mcps/
│   │   └── jira.py
│   ├── llms/
│   │   └── openai.py
│   └── models/
│       └── backlog.py
│
├── input/
│   └── backlog.md
├── output/
├── main_backlog_from_md_std.py
└── .env

Each module has a single responsibility and exposes a factory function. For example, the agent definitions:

# afw_core/agents/backlog_reader.py
from agent_framework import Agent

def create_agent(client, options, tools):
    return Agent(
        name="BacklogReaderAgent",
        instructions=(
            "You are a backlog reader assistant. "
            "When asked, use the read_file tool to read a markdown file. "
            "After reading, respond with ONLY a JSON object matching this schema: "
            '{"epic_count": <int>, "story_count": <int>, "description": <string>}.'
        ),
        client=client,
        default_options=options,
        tools=tools,
    )

# afw_core/agents/jira_executor.py
from agent_framework import Agent

def create_agent(client, options, tools):
    return Agent(
        name="JiraExecutorAgent",
        instructions=(
            "You are a Jira execution assistant. "
            "Create issues ONE AT A TIME, never in parallel. "
            "When linking stories to an epic, first create the epic, "
            "then create each story and set the epic as parent. "
            "After all operations, write a summary using write_file."
        ),
        client=client,
        default_options=options,
        tools=tools,
    )

The convention I adopted is: name and instructions are hardcoded inside the factory function (they are intrinsic to the agent's identity), while client, options, and tools are always injected from outside (they are infrastructure concerns). This separation keeps agent definitions clean and reusable.

Pydantic for Structured Output

A key improvement in the modular version was adding Pydantic validation between the two agents. Instead of passing raw text from the reader to the executor, I defined a model:

# afw_core/models/backlog.py
from pydantic import BaseModel

class BacklogOutput(BaseModel):
    epic_count: int
    story_count: int
    description: str

The reader agent is instructed to return JSON matching this schema. The main script validates it:

from afw_core.models.backlog import BacklogOutput

read_response = await reader_agent.run(
    f"Read the file '{backlog_file}' and return its contents."
)
backlog = BacklogOutput.model_validate_json(read_response.text)
print(f"Backlog loaded: {backlog.epic_count} epic(s), {backlog.story_count} stories")

If the agent returns malformed JSON, Pydantic throws a validation error immediately, rather than letting corrupted data propagate to the executor agent. This is a simple but effective pattern for inter-agent data contracts.

The Entry Point

The modular entry point becomes clean orchestration logic with no implementation details:

# main_backlog_from_md_std.py
import asyncio
import os
from dotenv import load_dotenv

from afw_core.llms.openai import create_client
from afw_core.mcps.jira import create_proxy
from afw_core.tools.file_reader import read_file
from afw_core.tools.file_writer import write_file
from afw_core.agents.backlog_reader import create_agent as create_reader_agent
from afw_core.agents.jira_executor import create_agent as create_executor_agent
from afw_core.models.backlog import BacklogOutput

load_dotenv()

async def main():
    client, options = create_client(
        api_key=os.environ.get("OPENAI_API_KEY", ""),
        model=os.environ.get("OPENAI_CHAT_MODEL", "gpt-4o-mini"),
    )
    jira_proxy = create_proxy()

    reader_agent = create_reader_agent(client=client, options=options, tools=[read_file])
    executor_agent = create_executor_agent(client=client, options=options, tools=[jira_proxy, write_file])

    # Step 1: Read and validate
    read_response = await reader_agent.run("Read the file 'backlog.md' and return its contents.")
    backlog = BacklogOutput.model_validate_json(read_response.text)

    # Step 2: Execute on Jira
    try:
        exec_response = await executor_agent.run(
            f"Execute the following backlog on Jira.\n\n{backlog.description}"
        )
        print(exec_response.text)
    finally:
        await jira_proxy.close()

if __name__ == "__main__":
    asyncio.run(main())

How you can see, the entry point reads like a recipe: create the infrastructure, create the agents, run them in sequence, handle cleanup. All the complexity lives in the modules under afw_core/.

The Pipeline in Action

Here is what the pipeline produces when you run it against a real Jira project:

Jira board after execution: the Epic and all linked Stories created automatically.

Console output: the full agent execution log showing each tool call and result.

Execution report: the summary file generated by the executor agent in the output/ directory.

Key Lessons Learned

Working through these four steps, several patterns emerged that are worth sharing:

MCP tools don't handle parallelism well. When the LLM sends multiple tool calls in a single response, the MCP server may fail. The workaround is simple: add "ONE AT A TIME" to the agent's instructions. The agent respects this.

The framework's error handling has a hidden default. The max_consecutive_errors_per_request parameter defaults to 3. If an agent hits 3 consecutive tool errors, it stops retrying. This is defined in agent_framework._tools and caught me off guard initially. Knowing this default helps you debug "why did it stop?" scenarios.

No __init__.py needed. Python's implicit namespace packages work fine. The key is choosing a unique directory name (afw_core) that doesn't collide with installed packages. I initially tried naming directories agents/, tools/, mcp/, but these collided with the framework's own modules. Renaming to afw_core/agents/ solved everything.

A well-defined directory structure makes a real difference. Applying a clear project layout (afw_core/ with separate modules for agents, tools, MCP proxies, LLM clients, and models) greatly simplifies working with the framework. It keeps things organized and makes the codebase easy to extend as you add more agents and integrations.

The biggest gap today: no native tools. This is, in my opinion, the framework's main weakness right now. Other frameworks like LangChain/LangGraph and CrewAI ship with a rich ecosystem of built-in tools (web search, PDF readers, database connectors, vector stores, and many more). With the Microsoft Agent Framework, you either build every tool yourself with @tool or rely on MCP servers. For simple use cases that's fine, but for projects that need quick access to common integrations, the lack of native tools is a significant disadvantage that other frameworks still handle much better.

Pydantic validation between agents is cheap insurance. It adds minimal overhead and catches data corruption early. Especially useful when the first agent's output is the second agent's input.

Agent instructions are powerful. You have a single instructions string that gives you full freedom to express exactly what you need, including operational constraints like "never call tools in parallel."

Conclusion: Takeaways

The Microsoft Agent Framework is a solid entry point into the world of AI agent development. Its explicit, code-first approach means there are very few surprises: what you write is what gets executed. The MCP integration is first-class and makes it trivial to connect agents to external services like Jira, Confluence, or GitHub.

The incremental approach I followed, from a single agent with no tools to a modular multi-agent pipeline, worked well as a learning strategy. Each step introduced exactly one new concept, making it easy to debug when things went wrong.

If you are starting with AI agents and want a framework with minimal abstraction, the Microsoft Agent Framework is worth a try. The codebase in this article serves as a progressive tutorial you can follow step by step.

All the code is available on GitHub (MSFTAgentSample).

In-Depth Comparison: Workflow Control with LangGraph and CrewAI

rosidotidev — Fri, 01 Aug 2025 15:47:07 +0000

(Originally published on my blog: https://roby73.hashnode.dev/in-depth-comparison-workflow-control-with-langgraph-and-crewai)
Choosing between CrewAI and LangGraph for multi-agent orchestration isn’t trivial, especially for developers balancing rapid prototyping with future scalability.

Personally, I gravitate toward CrewAI for its straightforward setup and developer-friendly abstractions. It lets you build production-ready agent flows in minutes, staying focused on business logic rather than complex orchestration architecture. That ease of use and quick iteration cadence make CrewAI a great fit for many projects.

That said, LangGraph has quickly become the go-to framework for large, stateful projects in production, thanks to its expressive graph-based DSL and wide adoption among major tech players. It offers granular control over state, dependencies, and memory, which is often necessary for complex or long-running workflows.

Absolute Control Over Agent Steps: LangGraph vs. CrewAI

One of LangGraph’s undeniable strengths is the absolute control it offers over every single step executed by agents in the workflow. Thanks to its explicit graph-based architecture, you precisely define nodes, edges, and state transitions. This granular orchestration lets you monitor, intervene, and debug the entire flow of execution with full visibility.

In contrast, CrewAI’s abstraction via agents and tasks simplifies setup and speeds development, but sometimes this comes at the cost of less direct visibility and control over internal step execution. The orchestration logic is more implicit and "black-boxed" inside the Crew abstractions, which can make fine-grained intervention more challenging.

That said, with the introduction of Flows in CrewAI—explicitly defined workflows coordinating agents and tasks—you can reclaim a similar level of control and transparency over execution. The Flows feature lets you model sequences, conditions, and branching explicitly, reducing the "black box" effect while still keeping CrewAI’s ease of use.

The Java Verifier examples below provide a clear demonstration of this: while LangGraph exposes every node and edge explicitly, CrewAI Flows allow similarly controlled stepwise execution and monitoring inside a more developer-friendly environment.

Example: The Java Verifier

To illustrate the difference in abstraction and workflow style between CrewAI and LangGraph, here is a minimal “Verifier” agent implementation in both frameworks. The goal of this verifier is to validate, correct, and optimize small Java code snippets. The workflow is designed with a loop: it first checks the code for syntax errors. If an error is found, it attempts to correct the code and then re-enters the verification step. This "verify-correct" loop continues until the code is deemed syntactically correct. Only then does the workflow proceed to the final step of optimizing the code for readability and performance. You can find all the code shown in this post on this Github repo (Java-Verifier-LangGraph-CrewAI)

1. CrewAI: Java Verifier with Flow

(This image has been generated using the functionality at the end of the provided Python code)

CrewAI Flows allow you to define a multi-agent workflow in a class-based structure that feels familiar to developers. You declare agents and then use decorators (@start, @router, @listen) to map out the flow. This structure keeps the logic contained and easy to read. For this particular example, we chose to orchestrate the individual agents directly within the Flow rather than using a complete Crew. This approach gives us more direct control over each agent's step, which was ideal for this straightforward, cyclical process. It's important to note that in more complex scenarios, CrewAI's flexibility allows you to orchestrate entire Crews of agents, leveraging their collective intelligence for a different level of abstraction.

from typing import Any, Dict, List
from dotenv import load_dotenv
from pydantic import BaseModel, Field

from crewai.agent import Agent
from crewai.flow.flow import Flow, listen, start, router

load_dotenv()

class CodeValidationState(BaseModel):
    code: str = ""
    syntax_status: str = ""

class JavaCodeValidationFlow(Flow[CodeValidationState]):

    analyzer_agent = Agent(
        role="Java code syntax analyzer",
        goal="Receive Java code and reply exactly with 'correct' or 'incorrect' based on syntax.",
        backstory="You are a meticulous and precise code quality tool."
    )

    corrector_agent = Agent(
        role="Java code corrector",
        goal="Receive syntactically incorrect Java code and return corrected code with valid syntax only. Return only Java code with no explanation and no markdown",
        backstory="You are an expert Java programmer who fixes code errors.",
        verbose=True
    )

    optimizer_agent = Agent(
        role="Java code optimizer",
        goal="Receive syntactically correct Java code and optimize it for readability and performance without changing logic. Return only Java compilable code with no explanation and no markdown",
        backstory="You are a senior developer specializing in code optimization."
    )

    @start("try_again")
    def start_analysis(self):
        """Starts the workflow, sends the code to the analyzer."""
        print(f"Starting code analysis...\n {self.state.code}")
        result = self.analyzer_agent.kickoff(self.state.code)
        syntax_status = result.raw
        self.state.syntax_status = syntax_status
        print(f"Analyzer result: {syntax_status}")

    @router(start_analysis)
    def conditional_next_step(self):
        """Decides whether to proceed with correction or optimization."""
        syntax_status = self.state.syntax_status
        print(f"conditional_next_step {syntax_status}")
        if syntax_status=="correct":
            return "correct"
        else:
            return "incorrect"

    @router("incorrect")
    def correct_code_step(self):
        """Performs code correction."""
        result = self.corrector_agent.kickoff(self.state.code)
        print(f"Corrector: code corrected.\n {result.raw}")
        self.state.code = result.raw
        return "try_again"

    @listen("correct")
    def optimize_code_step(self):
        """Performs code optimization."""
        result = self.optimizer_agent.kickoff(self.state.code)
        self.state.code = result.raw
        print("Optimizer: optimization finished.")
        print("\nWorkflow complete. Final optimized code:")
        print("---" * 20)
        print(self.state.code)
        print("---" * 20)

# Example usage of the workflow
def run_code_validation_flow():
    # Initial Java code with a syntax error (missing semicolon)
    initial_java_code = """
    public class Test {
        public static void main(String[] args) {
            System.out.println("Hello, World!"
        }
    }
    """
    flow = JavaCodeValidationFlow()

    print("Starting CrewAI Java code validation flow...\n")
    # Start the workflow and get the final result
    flow.kickoff(inputs={"code": initial_java_code})
    return flow

if __name__ == "__main__":
    flow=run_code_validation_flow()
    flow.plot('my_flow_plot')

Mental Model: In CrewAI, you first define the roles and goals of your agents. Then, you use a Flow to orchestrate them through a series of steps. The flow itself feels like a regular class with methods, where decorators handle the routing logic. This keeps the logic contained and aligned with a developer's class-based thinking. Notice that here we are using default LLM used by CrewAI that at the minute is “gpt-4o-mini“.

2. LangGraph: Java Verifier with Graph

(This image has been generated using the functionality at the end of the provided Python code)

LangGraph's approach is more explicit and foundational. You define a graph with nodes and edges, where each node is a function. The state of the workflow is passed explicitly between nodes, and conditional edges are used to manage the flow.

import os
from langgraph.graph import StateGraph, END
from dotenv import load_dotenv
from typing import TypedDict, Annotated  # Import TypedDict and Annotated
from langgraph.types import Command
from langchain_openai import OpenAI
from IPython.display import Image, display
# Ensure your OpenAI API Key is set in environment variable OPENAI_API_KEY
load_dotenv()
# Define your state schema
class CodeState(TypedDict):
    code: str
    is_correct: bool
class JavaVerifierGraph(StateGraph):
    def __init__(self):
        # Initialize LangChain OpenAI wrapper (reads OPENAI_API_KEY from env)
        self.llm = OpenAI(model="gpt-4o-mini", temperature=0)
        workflow = StateGraph(CodeState)
        workflow.add_node("start_analysis_node", self.start_analysis_node)
        workflow.add_node("optimize_code_node", self.optimize_code_node)
        workflow.add_node("correct_code_node", self.correct_code_node)

        # Define entry and transitions
        workflow.set_entry_point("start_analysis_node")
        # Conditional edges from analyzer node
        workflow.add_conditional_edges("start_analysis_node",
                                       self.conditional_next_node,
                                       {"optimize_code_node": "optimize_code_node",
                                        "correct_code_node": "correct_code_node"})
        # Cycle back from corrector to analyzer
        workflow.add_edge("correct_code_node", "start_analysis_node")
        self.graph = workflow.compile()



    def llm_check_code_syntax(self,code: str) -> bool:
        prompt = (
            "Check if the following Java code is syntactically correct. "
            "Reply ONLY with 'correct' or 'incorrect'.\n\n"
            f"Java code:\n{code}\n\nAnswer:"
        )
        result = self.llm(prompt)
        return result.strip().lower() == "correct"

    def llm_correct_code(self,code: str) -> str:
        prompt = (
            "Correct the following Java code to be syntactically valid. "
            "Provide only the corrected code without explanations.\n\n"
            f"Java code:\n{code}\n\nCorrected Java code:"
        )
        return self.llm(prompt).strip()

    def llm_optimize_code(self,code: str) -> str:
        prompt = (
            "Optimize the following syntactically correct Java code "
            "for readability and performance. Return only Java code with no explanation and no markdown.\n\n"
            f"Java code:\n{code}\n\nOptimized Java code:"
        )
        return self.llm(prompt).strip()

    # The analyzer node sets 'is_correct' in state to guide conditional branching,
    # no explicit goto inside the node function.
    def start_analysis_node(self,state:CodeState):
        code = state["code"]
        print("Analyzer: Checking syntax via LLM...")
        is_correct = self.llm_check_code_syntax(code)
        print(f"Analyzer: Code is {'correct' if is_correct else 'incorrect'}")
        return Command(update={"is_correct": is_correct})

    # Optimizer node - performs optimization and ends the workflow.
    def optimize_code_node(self,state:CodeState):
        code = state["code"]
        print("Optimizer: Optimizing code via LLM...")
        optimized_code = self.llm_optimize_code(code)
        print("Optimizer: Optimization complete. Ending workflow.")
        return Command(update={"code": optimized_code}, goto=END)

    # Corrector node - fixes syntax errors; no goto, flow cycles via explicit edge.
    def correct_code_node(self,state:CodeState):
        code = state["code"]
        print("Corrector: Correcting code via LLM...")
        corrected_code = self.llm_correct_code(code)
        print("Corrector: Correction complete. Returning to analyzer.")
        return Command(update={"code": corrected_code})

    # Branching function to decide next node after analyzer based on 'is_correct'.
    def conditional_next_node(self,state:CodeState):
        return "optimize_code_node" if state.get("is_correct") else "correct_code_node"

    def run_workflow(self,code):
        initial_state = {
            "code": code,
            "is_correct": True  # 'analyzer' will set this value at the beginning
        }

        print("Starting the execution of the workflow with invoke()...\n")

        # Run the workflow and get the final state
        final_state = self.graph.invoke(initial_state)

        print("\nWorkflow completed!")
        print("-" * 50)
        print("Final Java code (corrected and optimized):\n")
        print(final_state["code"])
        print("-" * 50)
        print(f"Final 'is_correct' state: {final_state['is_correct']}")

    def get_graph(self):
        return self.graph.get_graph()


def run_code_validation_flow():

    initial_code = """
    public class Test {
        public static void main(String[] args) {
            System.out.println("Hello World!")
        }
    }
    """

    initial_state = {
        "code": initial_code,
        "is_correct": True  # 'analyzer' will set this value at the beginning
    }

    print("Starting the execution of the workflow with invoke()...\n")

    graph=JavaVerifierGraph()
    final_state = graph.run_workflow(initial_state)


    return graph

if __name__ == "__main__":
    graph=run_code_validation_flow()
    # This command generates the PNG data as bytes, which works outside of Jupyter
    mermaid_png_bytes = graph.get_graph().draw_mermaid_png()

    # Specify the filename for the output image
    output_filename = "langgraph_workflow.png"

    try:
        # Open the file in binary write mode ('wb')
        with open(output_filename, "wb") as f:
            # Write the image bytes to the file
            f.write(mermaid_png_bytes)
        print(f"Diagram sevaed on {output_filename}")
    except Exception as e:
        print(f"Error during diagram generation: {e}")

Mental Model: With LangGraph, you build the workflow by explicitly defining a state schema, followed by individual nodes (functions). You then connect these nodes with edges. The orchestration logic is literally a graph you construct, giving you a powerful, albeit more complex, low-level tool for state management.

Conclusion

I appreciate CrewAI’s lean, quick-to-launch workflows and developer experience, which speed up prototyping and smaller scope projects. The Flow feature adds a layer of explicit control that bridges the gap with more low-level frameworks. Meanwhile, LangGraph stands out for heavyweight, scalable AI pipelines in enterprise-grade systems with a need for full-control graph orchestration.

To gain a more complete understanding of each framework's workflow implementation capabilities, other critical aspects must be considered. These include state management, the ability to handle state persistence (saving and resuming a workflow), and the ease of implementing a Human-in-the-Loop (HITL) scenario. These factors are crucial for designing robust, long-running processes that require durability or external intervention.

For the time being, CrewAI fits my development style best, but I watch LangGraph’s evolution with interest as it appears to be solidifying its place as the go-to framework for complex AI agent orchestration at scale.

Crewai

rosidotidev — Wed, 30 Jul 2025 18:47:56 +0000

Spring.ai

rosidotidev — Sun, 27 Jul 2025 13:19:05 +0000

crew

rosidotidev — Sun, 20 Jul 2025 19:40:07 +0000

CrewAI meets RAG: built-in and custom solutions

rosidotidev — Sun, 20 Jul 2025 12:55:25 +0000

(Originally published on my blog: https://roby73.hashnode.dev/crewai-meets-rag-built-in-and-custom-solutions)

Generative AI offers immense possibilities, however to truly be effective in real-world applications, it needs to overcome key hurdles: ensuring accuracy and integrating with private, up-to-date company information. A standard Large Language Model (LLM) on its own often isn't enough for tasks demanding specific facts and deep domain knowledge. LLMs generate responses based solely on their training data. If they lack a direct answer, they might still attempt one, sometimes leading to convincing, yet fabricated, outputs (often referred to as hallucinations). For serious applications that demand reliable, grounded results, a smarter approach is essential.

Understanding RAG: The secret to accurate AI

This is where Retrieval Augmented Generation (RAG) comes in. Think of RAG as giving the LLM a set of specific notes to read before it answers a question. This makes its responses accurate and based on real data. The RAG process can be broadly divided into two main phases: Ingestion and Retrieval, both crucial for augmenting the LLM's knowledge.

RAG Ingestion Phase

This initial phase focuses on preparing your knowledge base.

Document Loading: Documents from various sources (PDFs, text files, database schemas, etc.) are loaded into the system.

Transformation (Chunking): Loaded documents are often too large to be processed all at once. They are broken down into smaller, manageable "chunks" or segments. This process might also involve cleaning or pre-processing the text.

Embedding: Each text chunk is then converted into a numerical representation called an embedding. This transformation is performed by an embedding model (e.g., a Sentence Transformer or a model like OpenAI's text-embedding-3-small). Embeddings capture the semantic meaning of the text, allowing for comparisons based on context rather than just keywords.

Persistence (Vector Database): These embeddings, along with references back to their original text chunks, are stored in a Vector Database (Vector DB). A Vector DB is optimized for efficiently storing and querying high-dimensional vectors, enabling fast similarity searches.

RAG Retrieval Phase

This phase occurs when a user submits a query to the Generative AI application.

Query Embedding: The user's input query is also converted into an embedding using the same embedding model that was used during the Ingestion phase. This ensures consistency in the vector space.

Similarity Search: The query embedding is then used to perform a similarity search within the Vector DB. The goal is to find the "closest" or most semantically similar text chunks (and their corresponding original content) to the user's query.

Contextual Augmentation: The relevant text chunks retrieved from the Vector DB serve as precise, up-to-date information. This retrieved context is then used to enrich the original user query, forming a new, more informed prompt.

LLM Interrogation: Finally, this augmented prompt (containing both the user's original request and the relevant retrieved context) is fed to the Large Language Model. The LLM then generates a factual and grounded response, drawing directly from the specific knowledge extracted from your knowledge base, significantly reducing hallucinations and increasing reliability.

Why RAG is a Game-Changer

RAG isn't just about accuracy; it solves critical problems:

Precision and Reliability: RAG ensures the LLM's answers are factually correct and you can trace them back to your trusted data. This significantly reduces the chance of hallucinations.

Cost Optimization: Giving the LLM the right context with RAG means you don't need super long, complex prompts. Getting data (like for embeddings) is usually much cheaper than writing huge prompts or paying for more LLM tokens.

Overcoming Token Limits: All LLMs have a maximum amount of text they can handle at once (their context window). RAG smartly gets around this by only giving the most relevant bits of information, instead of trying to stuff an entire knowledge base into the prompt. This lets LLMs work with much bigger and more complex data sets.

CrewAI: Orchestrating Smart Agents for flexible RAG

CrewAI is a powerful Python framework. It lets you create and manage teams of autonomous AI agents. With CrewAI, you can define specialized agents (like a "Researcher" or an "Analyst") with specific roles, goals, and tools. CrewAI is key for RAG because it allows these agents to work together and share tasks. This makes the RAG process strong and efficient. The "Tools" are the crucial link, connecting your agents to all your different knowledge sources. Flexibility is key: CrewAI lets you use both built-in RAG features that come with the framework (like PDFSearchTool and Knowledge) and custom tools you build yourself. This means you can connect to any specific data source you need, which is essential for real-world projects.

Implementing RAG with CrewAI: Three Practical Approaches

Here's how you can use CrewAI to implement RAG, showing its versatility.

Native RAG Tools

This is how CrewAI handles RAG natively using its built-in tools. The PDFSearchTool is a prime example, designed for semantic searches directly within PDF content. A CrewAI agent can use this out-of-the-box tool to efficiently find and retrieve specific passages based on a search query within a PDF document. By default, it uses OpenAI for both embeddings (to understand meaning) and summarization, though this can be customized. This automates knowledge extraction from your existing PDF archives directly within the CrewAI ecosystem.

Knowledge for Document-Based RAG

CrewAI offers a native "Knowledge" feature that allows agents and crews to directly access and utilize various external information sources. This acts as a built-in reference library for your agents. You can provide knowledge sources in formats like raw strings, .txt, .pdf, .csv, .xlsx, and .json documents by placing them in specified directories. CrewAI handles the storage (using ChromaDB by default) and embedding automatically. This approach empowers agents with direct access to a curated document set, ensuring their responses are grounded in your specific data.

Custom tool that uses Vector Search (FAISS in our case)

FAISS (Facebook AI Similarity Search) is great for very fast searches based on meaning, across huge sets of data (vector embeddings). Here, a custom tool lets CrewAI agents query a FAISS index. They find information that's semantically similar and feed it to the LLM. This provides fast and scalable semantic search for large, changing knowledge bases.

Real-World Example: Demonstrating RAG Flexibility with Multi-Crew Setup

In a real-world project, we implemented a system to analyze PDF documents. This setup, whose code is available in this repository: CrewAI-RAG-Sample, effectively demonstrated CrewAI's versatility.

├── db
│   ├── 455d85ec-a7f9-4c9c-82d8-d3567d9263aa
│   └── chroma.sqlite3
├── Easy_recipes.pdf
├── knowledge
│   └── Easy_recipes.pdf
├── LICENSE
├── main.py
├── Pipfile
├── Pipfile.lock
├── ragcrew
│   ├── config
│   │   ├── agents.yaml
│   │   └── tasks.yaml
│   ├── faiss_rag_crew.py
│   ├── pdf_knowledge_crew.py
│   ├── tool_rag_crew.py
│   └── tools
│       └── custom_tool.py
├── README.md
├── report.md
└── setup.txt

We achieved this by employing three distinct crews, executed one after the other, to reach the same analytical goal. What made each crew unique was the specific RAG approach it utilized, while the core agents and tasks remained consistent. This allowed for a direct comparison of RAG strategies in action.

Each of these three crews was designed with the same set of specialized agents, for instance, a PDF Researcher and a Content Analyst. Similarly, the tasks they performed were identical across all crews: first, Information Retrieval to locate and extract relevant data from the PDF based on a user query; then, Content Refinement to process and summarize the retrieved information for clarity and conciseness; and finally, Output Generation to format and save the final, refined content into a Markdown (.md) file.

The key differentiator lay in how each crew performed its Information research task declared within the tasks.yaml file. This task relies on pdf_researcher Agent.

research_task:
  description: >
    Search for specific information requested in the {topic}.
  expected_output: >
    Most relevant information about {topic},
  agent: pdf_researcher

content_task:
  description: >
    Review the context you got.
    Make sure the report is detailed and contains any and all relevant information.
  expected_output: >
    A fully fledged report with the main topics. The language to use is the language of the request "{topic}",

Let’s see how the three crews manage RAG different approaches.

ToolRagCrew: This crew leveraged CrewAI's out-of-the-box PDFSearchTool for its RAG mechanism. Agents directly queried PDF documents, showcasing a straightforward, native approach to unstructured document RAG. Here a Python code snippet for tool_rag_crew.py

    . . . 
    from crewai_tools import PDFSearchTool
    . . .
    @CrewBase
    class ToolRagCrew:
        agents: List[BaseAgent]
        tasks: List[Task]
        . . .
        def getPDFRagTool(self):
            if self.pdf_tool is None:
                self.pdf_tool = PDFSearchTool(pdf=self.pdf_path,
                                              #chunker=dict(chunk_size=2000,chunk_overlap=50)
                                              )
            return self.pdf_tool
        @agent
        def pdf_researcher(self) -> Agent:
            return Agent(
                config=self.agents_config['pdf_researcher'],
                verbose=True,
                tools=[self.getPDFRagTool()],
            )
        . . .

How you can see, a built in PDFSearchTool is configured as a tool for the PDF Researcher Agent

PDFKnowledgeCrew: Here, the crew utilized CrewAI's native "Knowledge" feature. PDF documents were pre-loaded into the agents' knowledge base by placing them in a designated directory, allowing agents to access and integrate this curated information during their tasks. Here a Python code snippet for pdf_knowledge_crew.py

    . . . 
    from crewai.knowledge.source.pdf_knowledge_source import PDFKnowledgeSource
    . . .
    @CrewBase
    class PDFKnowledgeCrew:
        agents: List[BaseAgent]
        tasks: List[Task]
        . . .
        def get_knowledge_sources(self):
            if self.pdf_source is None:
                self.pdf_source = PDFKnowledgeSource(file_paths=self.pdf_paths)
            return self.pdf_source

        @agent
        def pdf_researcher(self) -> Agent:
            return Agent(
                config=self.agents_config['pdf_researcher'],  # type: ignore[index]
                verbose=True,
                knowledge_sources=[self.get_knowledge_sources()]
            )
        . . .

How you can see, it is very simple to use Knowkedge feature with Crew. You have to use the attribute knowledge_sources within the Agent declaration. It accepts an array of instances of PDFKnowledgeSource. That’s it!

FAISSRagCrew: For this crew, a custom tool integrated with FAISS was implemented. This represented a more scalable solution for large datasets, where agents used this custom tool to perform semantic searches on vector embeddings derived from the PDF content. I’ve used FAISS (Facebook AI Similarity Search) because this library is very useful for similarity search and clustering of vectors. I also wanted to share how CrewAI framework is “opened” for integration with 3rd party libraries and tools. Here a Python code snippet for faiss_rag_crew.py

    . . .
    import pdfplumber
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain_community.vectorstores import FAISS
    from openai import OpenAI
    from typing import List
    from langchain_openai import OpenAIEmbeddings  
    from ragcrew.tools.custom_tool import PDFFAISSTool

    @CrewBase
    class FAISSRagCrew:

        agents: List[BaseAgent]
        tasks: List[Task]

        def __init__(self,pdf_path):
            self.pdf_path=pdf_path
            self.vector_store= None
            self.client=OpenAI()
            self.search_tool=None

        def get_openai_embedding(self,text):
            response = self.client.embeddings.create(
                input=text,
                model="text-embedding-3-small"
            )
            return response.data[0].embedding

        def load_pdf(self):
            with pdfplumber.open(self.pdf_path) as pdf:
                return " ".join(page.extract_text() for page in pdf.pages)

        def prepare_rag(self,pdf_text):
            text_splitter = RecursiveCharacterTextSplitter(
                chunk_size=2000,
                chunk_overlap=200
            )
            chunks = text_splitter.split_text(pdf_text)
            return FAISS.from_texts(chunks, OpenAIEmbeddings())

        def initFAISS(self):
            pdf_text = self.load_pdf()
            self.vector_store = self.prepare_rag(pdf_text)
            self.search_tool = PDFFAISSTool(self.vector_store)

        @agent
        def pdf_researcher(self) -> Agent:
            self.initFAISS()
            return Agent(
                config=self.agents_config['pdf_researcher'],  # type: ignore[index]
                verbose=True,
                tools=[self.search_tool]
            )
        . . .

How you can see, within the Agent declaration the initFAISS() method is invoked. It executes the ingestion phase of the RAG process: loading of the PDF file, persisting embedded data into the vector DB, and initialization of the PDFFAISSTool which is a custom tool defined in the project.

This multi-crew sample effectively highlights CrewAI's ability to maintain consistent operational logic (same agents, same tasks) while allowing for flexible and interchangeable RAG strategies.

Conclusion: Takeaways of the reading

In this article, we looked at different ways to set up Retrieval-Augmented Generation (RAG) within a CrewAI agent system. We saw how each method: using a special PDFSearchTool, the built-in Knowledge feature of CrewAI, and making a Custom Tool with FAISS, offers different ways to help our agents find and use outside information. We learned that all three methods aim to give Large Language Models (LLMs) the right information to create accurate answers and avoid making things up. But they are different in their purpose, how the agent uses the data, and how much control you have. What is clear is that CrewAI is a capable framework for building systems with many agents. Alongside other agent frameworks like LangChain, AutoGen, or LlamaIndex, CrewAI provides a valuable alternative. It is easy to use and focuses on how agents work together, what their jobs are, and what they want to achieve.

Trading

rosidotidev — Fri, 21 Mar 2025 22:53:12 +0000

Simplifying backlog management on Jira with CrewAI

rosidotidev — Sun, 19 Jan 2025 21:29:19 +0000

One of the key challenges in the Agile development lifecycle is defining User Stories with the right level of detail. During refinement phases, stories can range from being overly generic and high-level to fully detailed and actionable. This variability often leads to inefficiencies and inconsistencies in backlog management.

By leveraging a workflow of Generative AI agents capable of interacting with Jira's APIs (tools), we can automate backlog creation and refinement. These agents process requirements, enrich them with best practices, and interact directly with Jira to generate and organize backlog items. The workflow is designed to:

Break down requirements into structured Epics and User Stories.
Suggests stories in a structured and formalized format.
Seamlessly interface with Jira to update or query backlog items dynamically.

In this article we see how CrewAI framework can be used to reach this goal, it is a cutting-edge platform designed to orchestrate autonomous AI agents, enabling them to collaborate effectively on complex tasks.

Key components of CrewAI include:

Agents: Autonomous units with specific roles, expertise, and responsibilities. Each agent can perform tasks, utilize tools, and collaborate with other agents to achieve shared objectives.
Tasks Defined objectives assigned to agents, which can be executed sequentially or in parallel. Tasks are the building blocks of the workflow, guiding agents on what needs to be accomplished.
Tools Integrated functionalities that agents can employ to interact with external systems, such as APIs. For instance, agents can use tools to interface with Jira's API, enabling operations like creating issues or retrieving data.
Crews A crew in crewAI represents a collaborative group of agents working together to achieve a set of tasks. Each crew defines the strategy for task execution, agent collaboration, and the overall workflow.

By defining these components, CrewAI facilitates the creation of AI teams where each agent has specific roles, tools, and goals, working together to accomplish complex tasks. This structured approach ensures efficient collaboration and effective task execution within the framework. The basic idea is to define the 4 main capabilities using a friendly and basic UI.

1. Detailed Backlog Creation from Requirements

CrewAI agents process requirement descriptions and generate tasks enriched with security considerations, architectural patterns, and testing protocols. Whether a requirement is sparse or well-documented, the system produces actionable Epics and User Stories, aligning with best practices and reducing manual effort.

2. Collaboration with Product Owners

Tasks are automatically grouped into Epics and linked to User Stories with pre-defined acceptance criteria. Product Owners can easily review, adjust, and reprioritize tasks, ensuring alignment with project goals while reducing time spent on refinement.

3. Automated Jira Integration

The system integrates seamlessly with Jira, creating and organizing Epics and User Stories directly within the tool. This eliminates manual data entry, ensuring consistent and efficient backlog management.

4. Advanced Search and Filtering

With natural language queries and advanced filters (e.g., by priority, status, or assignee), the system can quickly navigate large backlogs, locate tasks, and identify gaps or dependencies to optimize task management.

Implementation overview

In this github project there is a basic implementation of these capabilities. This is the project structure

├── agents
│   ├── agile.py
│   └── reporter.py
├── tasks
│   ├── agile.py
│   └── reporter.py
├── tools
│   ├── easy_jira_wrapper.py
│   ├── jira_cleaner_util.py
│   ├── jira_epic_creator_tool.py
│   ├── jira_reader_tool.py
│   └── jira_story_creator_tool.py
├── crews.py
├── LICENSE
├── main.py
├── main_web_gradio.py
├── Pipfile
├── Pipfile.lock
├── README.md
├── setup.txt
├── statememnts_repo.py
└──web_app_gradio.py

main.py: Backend entry point for CrewAI workflows.
main_web_gradio.py: Entry point for the Gradio-based web interface.
web_app_gradio.py: Logic for the web app, handling user inputs and outputs.
crews.py: Defines Crews, collections of agents and tasks.
statememnts_repo.py: Repository for prompts, queries, or agent instructions.
README.md: Documentation for installation and usage.
LICENSE: Project licensing information.
Pipfile & Pipfile.lock: Dependency management with pipenv.
setup.txt: Instructions for initializing the project.

`agents`

Defines AI agents specialized in specific tasks:

agile.py: Agents for Agile processes like backlog refinement.
reporter.py: Agents for generating Jira-related reports.

`tasks`

Specifies objectives for agents:

agile.py: Tasks for creating Epics and User Stories.
reporter.py: Tasks for generating reports.

`tools`

Utilities for Jira integration:

easy_jira_wrapper.py: Simplified wrapper for Jira API operations.
jira_cleaner_util.py: Tool for cleaning outdated issues.
jira_epic_creator_tool.py: Automates Epic creation.
jira_reader_tool.py: Reads and filters Jira issues.
jira_story_creator_tool.py: Creates and links User Stories. ## User Interface

The user interface is designed for simplicity and functionality:

Input Area: Located in the top-left corner, this text area allows users to input their requirements or queries for the system.
Output Area: Positioned at the bottom, this section displays the system's response, including generated backlogs or Jira issue details.
Dropdown Menu: Found in the top-right corner, the dropdown provides the following options:

Create a Backlog and Jira User Stories on Jira

The user can enter a requirement (even just a few words) into the input area, then the system processes the input, generates a backlog, and saves it in Jira as an Epic with linked User Stories.

The output includes the Type, Key, Summary, and Description of all Jira Stories in Markdown format.
Create a Backlog

This option allows the user to generate a backlog based on the input requirement. The user can review and modify the backlog before submitting it to Jira with the next choice.
Create Jira User Stories on Jira

Use this option to submit a pre-defined or previously created backlog to Jira, where it will be saved as Epics and Stories.
Retrieve Jira Issues

This feature allows users to query Jira and retrieve existing issues based on specific criteria.

The interface provides a streamlined experience, ensuring users can efficiently manage backlogs and interact with Jira. Below an example of what can be created on Jira with this tool.

Conclusions

This project serves as a concrete example of how Agentic Generative AI can support a specific phase of the software development lifecycle. By automating the backlog creation and refinement process, it showcases the potential of AI agents to reduce manual effort, enhance consistency, and streamline Agile workflows.

CrewAI proves to be a strong competitor in the growing landscape of agentic frameworks, standing alongside platforms like LangGraph, LlamaIndex, AutoGen, and many others. Its flexibility in defining agents, tasks, and tools makes it a valuable choice for building advanced AI-driven workflows.

That said, the project is currently in a prototypal state and has room for improvement in several key areas:

Integration of a Retrieval-Augmented Generation (RAG) approach: To enhance contextual accuracy by combining AI outputs with real-time data retrieval.
Improved Input and Output Typing with Pydantic: Strengthening data validation and consistency across the workflow.
Agent and Task Abstraction: Introducing support for YAML-based configuration files to simplify the definition and customization of agents and tasks.
Expanded Jira Interactions: Adding more diverse operations to interact with Jira, such as advanced queries, bulk updates, and custom workflows.

This prototype highlights the immense potential of CrewAI and similar frameworks, demonstrating how they can revolutionize software development processes with intelligent, automated workflows.

DEV Community: rosidotidev

Spec-Driven Development with GitHub Spec Kit

What Is Spec-Driven Development?

The Specification as a Contract

GitHub Spec Kit in a Nutshell

A Simple Example: a Todo List

The Role of the Human

Takeaways

Github copilota spec

Karpathy's LLM Wiki? No Code with Claude or Github Copilot!

What Is an LLM Wiki?

Why No-Code?

The Three Core Operations

1. /wiki-ingest, Build Your Knowledge Base

2. /wiki-query, Ask Questions Without Hallucination

3. /wiki-lint, Maintain Health

The Directory Structure

Three Page Types

How It Works: The Human-Gated Loop

Getting Started: A Practical Example

The Configuration Layer: .claude/ and .github/

.claude/ Directory, Claude Code Configuration

.github/ Directory, GitHub Copilot Configuration

Key Design Choices

What's Not Included (Yet)

Takeaways

Microsoft Agent Framework: From Agents to Workflows, Five Iterations of a DOCX-to-Markdown Pipeline

MFA Workflows in 30 Seconds

The Use Case

The Starting Point: Agents Only (v1)

What Broke and Why

Introducing Workflows (v2)

The Template + Filler Pattern (v3)

Scaling Up (v4): Parallel Vision + Chunked Assembly

Going LLM-Minimal (v5): The Final Architecture

The DocExtractor

The TemplateAssembler

The Final Workflow Graph

The Workflow Builder

The Entry Point

Evolution Summary

Key Lessons Learned

Conclusion: Takeaways

Microsoft Agent Framework: From Zero to Multi-Agent Pipeline

What I Used So Far

Step 1: Hello World, One Agent, No Tools

Step 2: Adding an MCP Tool (Jira)

A Hard Lesson: Parallel Tool Calls

Step 3: Two-Agent Pipeline (Monolithic)

Step 4: Modular Pipeline with Pydantic Validation

Pydantic for Structured Output

The Entry Point

The Pipeline in Action

Key Lessons Learned

Conclusion: Takeaways

In-Depth Comparison: Workflow Control with LangGraph and CrewAI

Absolute Control Over Agent Steps: LangGraph vs. CrewAI

Example: The Java Verifier

1. CrewAI: Java Verifier with Flow

2. LangGraph: Java Verifier with Graph

Conclusion

Crewai

Spring.ai

crew

CrewAI meets RAG: built-in and custom solutions

Understanding RAG: The secret to accurate AI

RAG Ingestion Phase

RAG Retrieval Phase

Why RAG is a Game-Changer

CrewAI: Orchestrating Smart Agents for flexible RAG

Implementing RAG with CrewAI: Three Practical Approaches

Native RAG Tools

Knowledge for Document-Based RAG

Custom tool that uses Vector Search (FAISS in our case)

Real-World Example: Demonstrating RAG Flexibility with Multi-Crew Setup

Conclusion: Takeaways of the reading

Trading

Simplifying backlog management on Jira with CrewAI

Implementation overview

agents

1. `/wiki-ingest`, Build Your Knowledge Base

2. `/wiki-query`, Ask Questions Without Hallucination

3. `/wiki-lint`, Maintain Health

The Configuration Layer: `.claude/` and `.github/`

`.claude/` Directory, Claude Code Configuration

`.github/` Directory, GitHub Copilot Configuration

`agents`

`tasks`

`tools`