Introduction
We built over hundreds of AI agents under the new "agents as a service" model in the last several months. This document shares 26 key takeaways that we had to learn the hard way - lessons that cost us dissatisfied clients, time, and money, so you don't have to repeat our mistakes.
Agent Fundamentals
1. AI Agents Are Not Your Employees
- Agents are neither automations nor employees
- Automations: Every step is hardcoded with exact sequence known in advance
- Employees: Have more autonomy than agents
- Instead of thinking about agents in terms of roles, think in terms of SOPs (Standard Operating Procedures)
- Typically one agent can handle one SOP well, while one employee handles 5+ SOPs
2. Start From Well-Documented Processes
- SOPs (Standard Operating Procedures) make training agents significantly simpler
- Well-documented processes contain most of what you need to train agents
- Find onboarding materials and SOPs first before collecting data manually
3. Business Owners Will Never Build Their Own Agents
- Even with agents that build other agents, business owners will still need specialists
- Similar to how no-code tools created no-code developers, not the end of developers
- AI agent platforms will spike demand for AI agent developers
- Determining which agents to build is harder than building them
4. Business Owners Have No Idea Which Agents They Need
- ~50% of initial client ideas are not the most valuable agents for their business
- Consulting is a huge part of the service
- Start by mapping customer journeys to identify automation opportunities
- Don't assume client ideas are the best - use them as feedback only
5. You Don't Need 20+ Agents
- Building too many agents makes systems more complex
- More agents means:
- Harder maintenance
- More complex debugging
- Increased costs
- Longer response times
- Start with the smallest possible agent, then add more as needed
6. Data With Actions Deliver Results
- GIGO (Garbage In, Garbage Out) applies to AI agents
- The biggest impact comes from combining data with relevant actions
- Combining knowledge (e.g., Facebook marketing strategies) with actions (Facebook API control) achieves significantly better results
- Scrape both internal and external sources to enhance agent performance
Agent Development
7. Prompt Engineering Is an Art
- Prompt engineering is already a real job
- As models become larger and smarter, prompt engineering becomes more important
- Tips for effective prompts:
- Provide examples - one example is worth a thousand words
- Order matters - arrange prompts with most important parts at the end
- Iterate and test constantly
8. Integrations Are Just as Important as Functionality
- Don't over-focus on agent capabilities at the expense of integration
- If it's not convenient for users, even powerful agents won't provide value
- Integrate agents into the same systems employees use daily
9. Agent Reliability Has Been Solved
- If an agent isn't reliable, it's a developer problem, not an agent problem
- Pydantic (data validation library) can validate all agent inputs/outputs
- With proper validation logic, agents can't take harmful actions
- Pydantic allows building agents for any use case
10. Tools Are the Most Important Component
- The three most important components: instructions, knowledge, and actions
- 70% of work goes into building actions (tools)
- Standard chatbots generate value through responses
- Agents generate value through actions - they should do things, not just advise
11. No More Than 4-6 Tools Per Agent
- More than 4-6 tools causes hallucination
- Agents start to confuse which tools to use or the proper sequence
- If an agent starts hallucinating, split it into multiple agents
12. Model Costs Don't Matter
- If your use case makes sense, you'll almost always make tremendous ROI
- Example: Process reduced from $300 and 3 days to $1-2 and 20 minutes
13. Clients Don't Care About Which Model You Use
- Open source models aren't necessary
- Businesses care about value, not the model providing it
- For strict data privacy, use Azure OpenAI (runs models in private Azure instances)
- OpenAI remains the provider of choice due to developer experience
14. Don't Automate Until Value Has Been Established
- Don't automate businesses that don't exist yet
- First establish the process manually to ensure it works
- Development costs are the main concern, not model costs
- Hire someone on platforms like Upwork to test processes before automating
15. Don't Think About Use Cases, Think About ROI
- ROI formula: (Rate × Hours - Operational Costs) ÷ Development Costs
- Rate × Hours = Employee hourly rate × Total process hours
- Operational costs = Model costs + Server costs (typically negligible)
- Example: $50/hour × 10 hours/week with $5,000 development cost = 5.6× ROI after one year
16. Agent Development Is an Iterative Process
- Like data science competitions, testing the most variations often wins
- Try different architectures and compare side by side
- Build multiple variations when agents underperform to find what works best
17. Use Divide and Conquer Approach
- Break complex problems into manageable tasks
- Deliver solutions incrementally:
- Find an agent that can work independently
- Build and deliver that agent first
- Only proceed after client confirmation
- Automate by departments first before connecting systems
Scaling and Deployment
18. Evals Are a Big Deal (But Only for Big Companies)
- Evals track agent KPIs and performance over time
- Benefits for large companies:
- Helps eliminate competition
- Continuously improves solutions
- New clients benefit from previous solutions
- Enables future self-improvement
- SMBs may not need evals due to lower request volume
- Evals provide the last 20% of performance optimization
19. There Are Two Types of Agents
- Pure agents: Fully agentic systems
- Workflows: Processes with predetermined steps but agentic execution
- Steps and sequence are fixed
- Individual steps have agentic capabilities
- Example: Lead research with specific search patterns
20. Agents Need to Be Adaptable on Feedback
- Agents must interact with their environment and receive feedback
- Add tools that allow agents to analyze their results
- Example: Don't just build database update capability; add ability to read records to verify task completion
21. Don't Build Around Limitations
- Models continuously improve - don't optimize for current limitations
- Example: Complex systems built to avoid context token limits became obsolete when 128k context models were released
- Avoid building obvious general use cases that major AI companies might develop themselves
22. Deploying Agents Is Harder Than Building Them
- Building an agent might take 2-3 days
- Deploying and integrating it takes another 3 days
- Consider specialized platforms for agent deployment
23. Waterfall Projects Don't Work
- Agentic projects are too agile for fixed scopes
- Start with one-time fees but transition to subscription models
- Work as a partner, not just an outsourced team
- The goal is to automate business processes, not just build agents
24. Include a Human in the Loop for Mission-Critical Agents
- Some agents have no margin for error
- Add human review steps for high-stakes actions
- Example: Having clients review marketing campaigns in Notion
- Remove human review only after consistent approval and fine-tuning
25. 2025 Is the Year of Vertical AI Agents
- Vertical agents specialize in specific use cases for specific industries
- Benefits:
- Easier to scale
- Higher pricing potential
- Better problem-solving for specific businesses
- Start with horizontal agents, then identify patterns to create vertical agents
26. Agents Don't Replace People, They Help Businesses Scale
- Business owners typically don't fire people after automation
- Automation helps business owners think bigger and scale faster
- Employees can focus on higher-level tasks they enjoy
- Ultimately leads to a new age of abundance and prosperity
Top comments (0)