DEV Community: Chris Wood

AI Agents are shipping code faster than we can test

Chris Wood — Mon, 09 Feb 2026 17:43:34 +0000

Something shifted in January. Teams came back from the holidays and started using agents for real work. Claude Code, Cursor, Codex. Assigning tickets, letting them run in the background, reviewing the PR when it's done.

The code is good. Often better than what a human would write. But everyone's hitting the same wall: the verification process hasn't caught up.

The accountability gap

When a human submits a PR, there's an implicit expectation: they ran the code. They saw it work. If the change touches something user-facing, most teams expect a screenshot or video. Not just as proof of what changed, but as proof that someone was in the loop. They might have noticed something off in testing and fixed it before submitting. They thought about edge cases. Tried different states.

There's accountability baked in. If something breaks, someone should have caught it.

With agents, that layer is gone. Not because agents are bad at code. They're often better. But no one was in the loop. The agent did exactly what you asked. It just didn't think to check what else might have been affected.

You need more verification with agent PRs, not less. You want to see everything that changed, not just what the agent intended to change. You want to know the code ran and worked. And because there's no one else accountable, you're on the hook. If something breaks, it's on you. So you check it yourself.

But the volume makes that impossible. You could carefully verify a few human PRs a day. When you're getting 10+ from agents, something has to give.

The ceiling

Teams want to give agents bigger tasks. Refactors across multiple files. Design system updates. The kind of broad changes that are tedious for humans but perfect for agents.

But without verification, they can't. The risk is too high. One engineer I talked to put it this way: they're hitting a ceiling on how much they can use agents. They want to ramp up, not pull back. But the verification problem is blocking them.

So they're stuck giving agents small, safe tasks. Change this copy. Fix this one component. Things where the blast radius is limited and you can eyeball the diff.

The promise of agents is right there. Verification is what's stopping teams from grabbing it.

What I'm building

I've been working on qckfx, a regression testing tool for mobile. Record a flow, replay it deterministically, see exactly what changed.

Right now it runs locally. When you're working with Claude Code or Cursor, verification should happen in the development loop, not after. Agent makes changes, runs the test, catches regressions before they ever hit review.

A few things make it deterministic and trustworthy:

Network responses recorded and replayed. Tests don't flake on API timing or non-deterministic data like search results, recommendations, or AI outputs.
Dates stubbed, so time-dependent views don't break every run.
Disk state copied, so you're not fighting login flows and test data setup.

Your agent runs tests through MCP and sees what actually changed: screenshots diffed against baseline, network requests compared, logs captured.

It's free. Download for Mac

CI and team sync are on the roadmap. But local is the foundation. That's where you catch things before they become someone else's problem.

How are you solving this?

I'm building qckfx because I couldn't find a better way. If you've found something that works, I want to hear about it.

Are you doing manual QA on agent PRs? Writing more tests? Slowing down the agent volume?

If you want to try qckfx, I'm around to help: chris.wood@qckfx.com.

The AI Dev Tool Lottery: Why Building Your Own Tools Beats Playing the Odds

Chris Wood — Wed, 07 May 2025 20:14:42 +0000

Third-party AI developer tools often feel like playing the lottery - input your prompt and hope it works. Building your own tools gives you the control and visibility to transform unpredictable gambling into reliable engineering.

This post was originally shared on https://qckfx.com/blog/the-ai-dev-tool-lottery-why-building-your-own-tool-beats-playing-the-odds

In the rapidly evolving landscape of AI-powered developer tools, a frustrating pattern has emerged. You input your prompt, click submit, and find yourself crossing your fingers. Will it produce the code you need? Will it understand your problem correctly? Or will you need to try again and again, burning through your token budget while hoping for that winning ticket?

The Third-Party AI Tool Lottery

We've all heard the stories about teams excited by new autonomous coding assistants, only to discover that reliability becomes their biggest challenge. One day the tool solves a complex bug in seconds; the next day it stumbles on a simple function refactor.

What makes this situation particularly maddening is the black box nature of these solutions. When a third-party AI developer tool fails, you have no visibility into the system prompts being used, the underlying models powering the tool, the reasoning behind its actions, or the development decisions that shaped its capabilities.

This opacity transforms what should be a deterministic development process into something that feels more like playing the lottery. And when the tool inevitably fails, teams have nowhere to turn for answers. The discourse around these proprietary agents remains limited compared to the open discussions surrounding base LLMs, leaving developers without clear paths to improve from 70% effectiveness to the 99% they need for production use.

The result? Promising tools get shelved, not because they never work, but because teams cannot consistently trust them.

Taking Control of Your AI Development Stack

Building your own AI developer tools fundamentally changes this dynamic. When you control the entire stack, you gain complete visibility into prompts, models, and tools. You can capture failed runs and convert them into future evaluations. You have the freedom to experiment with different approaches for specific use cases and maintain control over the environments where your AI tools operate.

Building your own tools enables iterative improvement. With the right infrastructure, you can systematically flag problematic results, experiment with different prompts and model configurations, and even automate the improvement process. Tools like dspy demonstrate the potential for automated prompt refinement that adapts to your specific needs.

Measuring What Matters to Your Team

When building in-house AI developer tools, you define success on your terms. For a bug-fixing tool, this might mean tracking the number of bugs fixed correctly in one attempt versus multiple attempts. You might care about the steps required to implement a fix or the number of files modified when resolving issues. Cost efficiency per bug resolution could be critical for your team, as might whether fixes include runnable tests. You could evaluate the accuracy of the model's reasoning about bug causes or track quality metrics from code reviews.

Some metrics require human evaluation—but for code, these processes already exist through standard code review practices. The key advantage is that you determine which metrics matter most for your specific workflows and team needs.

Shifting the Mindset from Consumption to Ownership

The true transformation happens when teams stop treating AI as another third-party tool and start viewing it as an integral part of their development infrastructure. These AI systems differ fundamentally from traditional dev tools. They aren't undifferentiated infrastructure but highly opinionated assistants that directly influence your codebase and application functionality.

By building your own tools, your team can create AI systems that align with your existing workflows and respect established code patterns. These custom tools embody your team's engineering philosophy and automate work while maintaining your team's taste and style. The AI becomes an extension of your team rather than an external force pushing unfamiliar patterns or approaches.

The Diminishing Investment Barrier

With the right infrastructure layer—like what we're building at qckfx—the initial investment required to create custom AI developer tools has drastically decreased. Creating and deploying an internal dev agent can be as simple as authoring a prompt and configuring your preferred model and toolset. The platform handles the complex parts: monitoring, session recording, budget limits, debugging, and more.

This democratization of AI tool development means teams can spend less time wrestling with infrastructure and more time refining the capabilities that matter most to their specific development challenges.

Start Building, Stop Gambling

AI tooling represents the future of software development, but trusting exclusively in third-party black boxes means accepting unpredictable outcomes. By building your own AI tools—or at least using platforms that provide transparency and configurability—you transform the lottery into a system you can understand, measure, and improve.

AI agents have grown increasingly modular, with standards like Model Context Protocol (MCP) making it easier to share capabilities across agents and companies. This modularity addresses concerns about reinventing the wheel, allowing teams to focus on the unique aspects of their development workflow.

The teams that adopt and build AI development best practices today will hold a significant advantage tomorrow. The question facing development teams isn't whether to integrate AI into their development process but whether they'll continue playing the lottery or build a system they can count on.

Are you ready to stop gambling with your development process?

AI Coding Agents: The Building Blocks of Tomorrow's Software Development Lifecycle

Chris Wood — Tue, 06 May 2025 19:41:17 +0000

In the rapidly evolving landscape of software development, a transformative shift is underway. AI coding agents are emerging as fundamental building blocks that will reshape how we conceive, build, and maintain software. This transformation promises to redefine the entire Software Development Lifecycle (SDLC) in profound ways.

The Power of Generalizability and Customizability

What makes AI coding agents so revolutionary is their remarkable combination of generalizability and customizability. Modern large language models (LLMs) now possess the capability to handle almost every aspect of the development process with minimal specialized tooling. Given access to your codebase and contextual information, these agents can create design documents, enrich bug reports, deduplicate issues, write code, fix deployments, resolve merge conflicts, repair failing tests, and review code—all within the same fundamental architecture.

Their customizability amplifies this power. Through Model Context Protocol (MCP), teams can plug in existing tools from their stack—bug tracking systems, deployment pipelines, previous design documents, Figma designs, and even web browsers for live debugging. This flexibility extends to model selection as well, allowing teams to allocate their AI budget based on the complexity of specific tasks. More capable models command higher costs but aren't necessary for every aspect of development.

This adaptability enables each team to design their SDLC automation in alignment with their specific business needs and existing external dependencies.

Addressing the Pain Points of Traditional Development

The traditional software development lifecycle has always been labor-intensive and fragmented. Issue triage, bug reproduction, code writing, code review, deployment, debugging, feature flag management, A/B test monitoring, documentation writing, QA testing, and optimization—all these elements historically required massive team efforts, with knowledge scattered across different specialists and functions.

AI agents can now fully or partially automate many of these tasks. Perhaps more importantly, they make context-sharing seamless since everything becomes structured data. This streamlines development, making it faster and more cost-effective to build professional-grade software.

This shift introduces a new layer of complexity: monitoring, debugging, and assembling this SDLC supply chain becomes the new challenge. Since every team has unique dependencies, budget constraints, and development philosophies, no one-size-fits-all solution will emerge. While frameworks may help handle the more tedious aspects, each engineering team will likely invest considerable time iterating on their own SDLC automation pipelines.

Recent Breakthroughs Enable New Possibilities

Recent advances have turned theoretical promises into practical reality. With the arrival of models like Claude 3.7 Sonnet, Gemini Pro 2.5, and O3, we've crossed a critical threshold. These models possess the capabilities required for sophisticated self-directed automation.

More engineering teams now adopt these advanced models and allow them to operate for longer periods without human intervention. However, fully automated setups introduce challenges around monitoring, control, debugging, rollback mechanisms, deployment safety, and cost management. These guardrails will become essential as companies embrace this new paradigm.

Tangible Benefits: Quality, Speed, and Cost

The benefits of integrating AI coding agents into the SDLC are substantial and measurable. When properly instructed, AI often writes better code than most humans and excels at identifying and fixing bugs. This translates to fewer defects, faster shipping cadence, quicker bug fixes, and better-documented code.

Engineering costs associated with projects will likely decrease as leaner teams accomplish more and organizations optimize their AI budgets. Many development teams currently maintain extensive backlogs of nice-to-have features, technical debt, and minor bugs—all limited by human bandwidth and prioritization constraints. With AI, the limiting factor becomes budget, but many smaller tasks may cost only a few dollars from concept to deployment.

The results include healthier codebases, more polished code, potentially more expansive feature sets, and even more personalized customer offerings as development costs fall and specialized features become economically viable for smaller market segments.

The Evolving Role of Human Developers

Human developers will evolve to become orchestrators of SDLC automation. They'll oversee AI agents, balancing budget conservation with efficient outcomes. For simpler tasks where models may be overpowered, developers might fine-tune or optimize to reduce costs. Conversely, for complex tasks where models might be underpowered, the pragmatic approach often involves running models multiple times to select the best output—accepting higher costs for better results. This compute-scalable strategy naturally benefits from ongoing model improvements without requiring significant code rewrites.

Developers will also tackle the challenge of balancing general-purpose agents with specialized ones tailored to specific parts of the codebase. Some components will always require more specialized knowledge that wouldn't make sense to apply universally. This necessitates "router" agents or rules to determine which agent handles which tasks—essentially programming at a higher level of abstraction.

Every agent modification will likely require backtesting against historical data and controlled rollouts via A/B testing. Humans will monitor these processes, or perhaps, for the most ambitious teams, build meta-agents to help manage the lower-level agents.

Timeline for Adoption

We stand on the verge of rapid disruption. Software developers have consistently been early adopters of AI technologies, and significant research and funding currently flow into this field. Now that models have reached sufficient capability levels, the next year will see pioneering companies seriously exploring these approaches.

Within three years, AI-powered SDLC will likely emerge as an established best practice among leading teams, with late adopters scrambling to implement similar systems to remain competitive.

Building the Infrastructure for Tomorrow's SDLC

At qckfx, we focus on building the essential infrastructure needed for SDLC automation. This includes granular billing systems, intuitive agent design and deployment tools, lightweight agent frameworks, comprehensive monitoring, debugging tools, and starter agents to help teams get up and running quickly.

Our initial focus centers on bug-fixing agents designed to integrate with GitHub, allowing AI to take the first pass at issues reported by users, QA teams, or internal testers before routing them to human engineers. However, we recognize that the same framework applies across virtually all SDLC tasks.

We firmly believe these tools should be built internally rather than outsourced to third parties. As we've noted in a recent post, the SDLC represents the soul of an engineering team, and maintaining control over this process remains critical for long-term success.

Our approach at qckfx reflects this philosophy. We've designed our solution to be highly modular and almost entirely open source. Our agent SDK is open source, and the entire React server for running agents in synchronous (debug) mode is open-sourced as well. This debug environment functions similarly to Claude Code, but with the key advantages of integration with our asynchronous agents and compatibility with any LLM provider. We remain flexible regarding data storage and LLM cost management, with options for self-hosting an LLM proxy for teams that prioritize security. This empowers engineering teams to maintain ownership of their SDLC while leveraging our infrastructure for the complex parts.

Conclusion

The integration of AI coding agents into the software development lifecycle represents a fundamental reimagining of how software is created. By embracing these intelligent building blocks, development teams can achieve unprecedented levels of efficiency, quality, and innovation.

The question now focuses on how quickly organizations will adapt to this new reality—and how effectively they'll implement the monitoring, control, and optimization systems needed to harness its full potential.

Ready to transform your software development lifecycle? Sign up for our beta program or email chris.wood@qckfx.com to discuss how AI coding agents can revolutionize your engineering processes.

Why Engineering Teams Should Build Their Own AI Coding Agents

Chris Wood — Tue, 06 May 2025 00:25:50 +0000

Originally posted at https://qckfx.com/blog/why-engineering-teams-should-build-their-own-ai-coding-agents

The Engineering Landscape of 2028

Imagine walking into an engineering department three years from now. The highest-performing teams will be defined by the proprietary AI systems they've built, customized for their specific codebases, architectural patterns, and business domains. These teams will wield AI as a strategic differentiator, not merely consume it.

This is where our industry is clearly headed. As AI increasingly becomes the primary author of code across industries, the strategic advantage will shift from who can write the best code to who can best direct, customize, and optimize AI systems to write that code.

The best engineering teams won't achieve these results by simply adopting off-the-shelf AI coding tools. They'll need something more fundamental: AI systems that embody their unique approaches to software development. To understand why building your own AI coding agents is essential, we need to examine the critical limitations of today's commercial offerings.

The Generalization Problem

Today's commercial AI coding tools suffer from a fundamental limitation: they're designed to serve everyone rather than excel for anyone. This is the classic trade-off between specialized and generalized solutions that appears across industries. When a capability becomes a competitive advantage, leading companies bring it in-house rather than relying on vendors who serve them alongside their competitors.

These general-purpose tools must make compromises to accommodate diverse users. Their features, performance characteristics, and pricing models all reflect the needs of the average user rather than your team's specific requirements. The vendors build businesses around broad appeal, not targeted excellence.

This creates a dangerous dependency. Your engineering velocity becomes hostage to the decisions of vendors whose incentives diverge from yours. Pricing pressure forces vendors to use less capable models than optimal for your specific needs. They invest heavily in elaborate scaffolding — representing the bulk of their development costs — to extract more performance from cheaper models, allowing them to maintain higher profit margins while delivering suboptimal results. These complex infrastructures built around today's models actively constrain tomorrow's capabilities. Generalized solutions take precedence over domain-specific optimization, further limiting the potential benefit to your unique codebase and challenges.

The evidence speaks volumes. Anthropic's Claude Code—a relatively simple implementation leveraging the full capabilities of their Sonnet 3.7 model—outperforms more complex commercial offerings like Cursor and Windsurf, while narrowing the gap with even the most sophisticated solutions. Unlike many competitors who optimize for low cost, Claude Code offers a usage-based pricing model that allows teams to invest more for better performance. This approach recognizes what sophisticated engineering teams already know: the value of superior results often justifies higher costs. This demonstrates how the right model with minimal scaffolding often beats over-engineered solutions that make compromises to serve diverse customer bases at artificially constrained price points.

Engineering Ethos as Competitive Advantage

How you write, review, and test code reflects your engineering ethos. For technology companies, code isn't just a means to an end—it is the product. The processes, decisions, and values that shape this code constitute your core competency. This fundamental activity demands direct ownership, not outsourcing.

Consider AI coding agents as a new form of compiler—not the traditional kind we're accustomed to like LLVM, but something far more profound. While traditional compilers translate code to machine instructions with a focus on efficiency and speed, AI coding agents convert high-level plans and designs into functional code. Their value stems not from optimization metrics, but from how they fill the gaps in your plans and inject "taste" into the implementation.

This taste—embodied in subtle decisions about patterns, trade-offs, and style—becomes a direct extension of your engineering team's spirit and values. By building your own AI coding agents, you create a system that encodes your architectural preferences, quality standards, and technical philosophies directly into the code generation process. The subtle preferences in how errors are handled, interfaces designed, or performance optimized all emerge from your team's collective wisdom, amplified through customized AI.

When you master system prompts that align with your codebase patterns, you transform general AI capabilities into specialized tools that understand your context. By feeding outputs from coding agents into review agents, you create a coherent development cycle that mirrors your team's approach to quality. Through the Model-Context Protocol (MCP), you incorporate your documentation, error patterns, and integration points into the AI's knowledge base. Even budget allocation becomes an expression of priorities—directing premium model processing toward critical components while economizing elsewhere.

In a world increasingly dominated by AI-generated code, engineering leaders must recognize that their competitive edge lies not in writing code manually, but in how they direct these new compilers. In-house AI coding agents become the primary channel through which your engineering ethos expresses itself. The resulting code embodies not just functional requirements but the spirit of your team and product—those intangible qualities that differentiate exceptional software from merely functional solutions. For an engineering-driven organization, outsourcing this capability would be akin to outsourcing your identity.

The Challenge of Building Your Own

Building in-house AI coding systems introduces real challenges. The operational overhead cannot be ignored: you'll need deployment and maintenance systems, evaluation frameworks for A/B testing your agents, monitoring tools to track performance, billing integration for cost management, and role-based access controls to govern usage patterns.

These operational concerns create significant friction. Most teams view these as the "un-fun" parts of development—the necessary but uninspiring infrastructure that drives hesitation to build rather than buy. You'll need systems for debugging agent behavior, ways to hop in and examine chat sessions when things go wrong, and audit trails to track decisions and outputs.

Yet the strategic importance of custom AI coding agents demands overcoming these challenges. While the infrastructure and frameworks for internal agent deployment aren't fully mature today, pioneering teams like ours are actively building these foundations. We envision a future where deploying custom AI coding agents becomes as seamless as deploying microservices is today—where the operational complexity fades into the background, letting the strategic value take center stage.

The Future Belongs to the Builders

The coming AI revolution in software engineering will not be defined by who adopts AI coding tools first, but by who understands their true strategic significance. As the technology matures, the distinction between those who merely consume AI capabilities and those who shape them will become increasingly stark.

The most forward-thinking engineering organizations already recognize that their competitive edge lies not in using the same commercial AI tools as everyone else, but in crafting AI systems that embody their unique engineering philosophy. These teams understand that in a world where AI increasingly writes the code, the crucial differentiator becomes how you shape the AI's approach to that code.

This vision requires courage—the willingness to invest in capabilities that many currently view as commodities. It requires foresight—the ability to see beyond the immediate convenience of off-the-shelf solutions to the long-term strategic implications of ceding control over how your code is written. Most importantly, it requires conviction that your engineering ethos matters enough to be worth preserving and amplifying through custom AI systems.

The transformation won't happen overnight, but the trajectory is clear. Three years from now, the leaders in every sector will not be distinguished by which commercial AI coding platforms they've licensed, but by how effectively they've integrated AI into their engineering culture and processes. The code these systems produce will be more than functional—it will be a direct expression of each organization's unique approach to problem-solving, quality, and craft.

This is not merely about building better tools. It's about ensuring that as AI assumes an ever-larger role in software creation, the resulting code still bears the unmistakable signature of your team's values and vision. The future of engineering excellence lies not in surrendering to the homogenizing influence of general-purpose AI, but in bending AI to express and amplify what makes your engineering culture exceptional.

Note: This post starts a conversation about the strategic future of AI in engineering teams. If you're navigating these decisions and would like to discuss further, I welcome connecting to share insights and experiences. Contact me at chris.wood@qckfx.com.