DEV Community: Shifa Mohammadi

I stopped trusting my AI chatbot's answers. Turns out that's everyone's #1 complaint.

Shifa Mohammadi — Mon, 14 Sep 2026 11:13:00 +0000

A few months ago I caught myself double-checking every SQL join and every summary my chatbot gave me before using it in anything real. Not because it was obviously wrong — it was confidently, plausibly wrong often enough that I stopped trusting it by default. Now I verify everything, which eats into the time savings that made me want to use it in the first place.

Turns out that's the norm, not a me-problem. G2's analysis of 2,950+ verified AI chatbot reviews found that accuracy and hallucination is the #1 buyer complaint, showing up in 13.7% of negative mentions — ranking above cost, which says a lot given how much AI pricing complaints get in the discourse. One reviewer described exactly the failure mode I keep hitting:

"It hallucinates with confidence. I've had it generate DAX formulas and SQL joins that looked perfectly fine but were logically wrong. For a data analyst, blindly trusting the output can lead to bad reports and wrong business decisions. Always verify."

Here's the part that surprised me: the four vendors G2 surveyed (Maven AGI, Assembled, Letter AI, Apollo.io) — who build chatbots and agents on the exact same foundation models the rest of us use directly — report production failure rates as low as 1%. Same models, wildly different reliability. The difference isn't the model. It's the system wrapped around it: all four describe agent-like architectures with multi-step orchestration, grounding responses in external data instead of trusting the model's memory, running continuous eval pipelines, and updating prompts and logic at least weekly (Apollo.io does it daily). When something's going to fail, they've built the system to catch it and hand off to a human before the user ever sees it.

One vendor put it in a way that stuck with me:

"Autonomy without trustworthy fallback logic isn't a feature; it's a liability."

That's Abby Schervish, content lead at Maven AGI. Another vendor pushed back on the idea that agents are just automatically more reliable than chatbots:

"The hardest part of building agents for production isn't reasoning... The narrative undersells how much of agent quality is plumbing rather than intelligence."

So the reliability gap I'm feeling isn't really a chatbot-vs-agent problem, and it's not really a model problem either — it's an engineering gap. I don't have an eval pipeline running against my own usage. I don't have retrieval grounding responses in a source of truth. I'm just trusting raw model output and manually catching the failures myself, which is exactly the "trust layer" work the report says buyers are stuck doing on their own.

For what it's worth, the value is still real even with the trust issues — 30.7% of buyers cite ease of use and 21.7% cite time savings as what they like most about these tools, hallucination complaints and all. One reviewer summed up the upside: it cut their report-writing time from 3-4 hours down to 30-40 minutes with a few modifications. That tracks with my experience too, on the days it doesn't confidently make something up.

Curious if anyone here has actually built real verification/grounding into their own chatbot usage instead of just eyeballing outputs — feels like the obvious next step but I haven't done it yet. Full report here if you want the data: https://learn.g2.com/ai-chatbot

We're a 2-person team living off AI image generation. The 2026 roadmap isn't built for us.

Shifa Mohammadi — Mon, 14 Sep 2026 11:10:07 +0000

It's just the two of us, and AI image generation is doing the job we'd otherwise need a designer for — product mockups, social assets, ad creative, all of it. The tools are fast, cheap, and good enough. What's not great is the credit system: some months we blow through our plan mid-month because a launch needed more iterations than expected, and there's no good way to predict that in advance.

Turns out we're the majority, not the exception. G2's 2026 State of AI Image Generation report analyzed 2,111 verified reviews and surveyed 7 vendors (getimg.ai, NightCafe, Lapis, Recraft, Bria AI, Mage, 1minAI), and found that 70% of reviewers in the past 12 months are individual users or small businesses. Enterprise buyers make up just 14% of the category.

And yet look at where the vendors say they're actually investing for 2026: workflow automation and personalization at scale, cited by 6 out of 7 vendors as their top priority. Those are enterprise problems — integrations, brand consistency, governance — not "help me not blow my credit budget" problems.

The ROI data explains why vendors might not feel urgency here. Small businesses that reported a payback timeline hit ROI in under 6 months 80% of the time; individual users hit it 67% of the time. The tools are already paying for themselves fast, so there's less pressure to fix the pricing model that's causing friction. Meanwhile the #1 frustration in reviews isn't a missing feature — it's credit and pricing limits. One reviewer put it exactly the way we'd put it:

"The cost of credits and the rate at which you use up credits is prohibitive for most small businesses."

To be fair to the vendors, the enterprise bet isn't crazy. Bria AI's CEO frames it as the necessary next phase:

"The next phase of visual AI isn't about better pixels — it's about making generation controllable, predictable, and safe enough to embed inside enterprise workflows."

That's probably right for where the category eventually goes. It just means the buyers actually driving adoption right now — us included — are subsidizing a roadmap aimed at someone else's budget and someone else's problems.

If you're running lean and living on these tools day to day, curious whether you're feeling the same credit-anxiety, or if I'm overthinking a pricing model that works fine at your scale. Full report here if you want the numbers: https://learn.g2.com/state-of-ai-image-generation

We deployed a voice AI agent in a week. Proving its ROI took months.

Shifa Mohammadi — Mon, 14 Sep 2026 10:40:25 +0000

We rolled out a voice AI agent for our support line and had it live in about a week — the vendor demo was smooth, setup was mostly config, no big engineering lift. Then leadership asked the obvious question: "so what's it actually saving us?" That question took us months to answer well, and it wasn't because the agent wasn't working. It was because nobody had set up a way to measure it before turning it on.

Apparently that's extremely common. G2's 2026 State of AI Voice Assistants report surveyed 17 vendors and analyzed 1,419 verified reviews, and found the exact same gap at scale: 76% of users report significant or transformational operational ROI from their voice AI, but measuring that ROI ranks as the #3 challenge in the category. Organizations know the value is real. Proving it internally is a separate, harder problem.

The adoption numbers explain why so many teams end up in this spot. 88% of platforms reported rapidly increased adoption over the past 12 months, and 65% describe the category as being in "rapid adoption across industries" already. This isn't an early, experimental category anymore — it's being deployed fast, often by teams without a dedicated analytics or ops function backing it up.

Interestingly, the report found that companies with fewer than 10 employees are the single largest reviewer segment, while enterprise organizations account for just 59 reviews combined. Voice AI didn't spread top-down from big companies with formal ROI frameworks. It spread bottom-up, through small teams that didn't have a procurement process slowing them down — which also means fewer of them had a measurement plan going in.

"We're past the tipping point. Voice AI agents aren't experimental anymore. They're how modern businesses communicate with customers at scale. The companies investing now are building a durable competitive advantage, and the ones waiting are falling behind every month."

That's Evie Wang, Co-Founder & CMO at Retell AI. Another vendor exec framed the real bottleneck bluntly:

"The value comes when you combine strong language capabilities with connected data and systems — so you're not just understanding intent, you're resolving it end-to-end."

That tracks with what we saw internally. Getting the agent to answer calls was the easy part. Connecting it cleanly to the systems that would let us attribute deflected tickets, saved agent-hours, and faster resolution times to the agent specifically — that took real integration work nobody had scoped upfront.

None of this means voice AI is overhyped. The category averages a 9.26/10 recommendation score, and 93.1% of the 1,419 reviewers rate their experience 4 stars or higher — only 2% are negative. Buyers aren't disappointed with the software. They're stuck defending a win they can feel but can't fully quantify yet.

If you're deploying or evaluating a voice AI agent right now, save yourself the months we spent: build the ROI measurement plan before go-live, not after. Full report here if you want the data: https://learn.g2.com/ai-voice-assistant

"Low-code ML" promised speed. Our deployment took 5 months.

Shifa Mohammadi — Mon, 14 Sep 2026 10:07:31 +0000

We picked a low-code ML platform specifically because we didn't have a dedicated data science team and needed something fast. "No code, faster time to value" was basically the pitch on every vendor call. Five months later we finally had a model in production, and most of that time had nothing to do with the model itself — it went into wiring the platform into our existing data pipeline and getting IT, security, and the business team to agree on what "done" looked like.

Turns out that's not a us problem. G2's State of Low-Code Machine Learning in 2026 report analyzed 3,400+ verified ML platform reviews across five categories, and found that low-code ML platforms average 4.5 months to go live — the slowest deployment time of any machine learning category G2 tracks. Slower than full data science platforms. Slower than MLOps platforms. Slower than data labeling tools by 2.6x. The category sold on the promise of speed is, empirically, the slowest one to ship.

If you're at an enterprise it's worse: enterprise buyers average 5.47 months to go live versus 2.75 months for small businesses, and even after deploying, enterprises only put 35.5% of their licensed seats to actual use, against 49% at small businesses. A lot of adopted-in-name-only tooling out there.

The vendors G2 surveyed (Pecan AI, Acodis, Minitab, Kili Technology) all landed on the same root cause, and it isn't model quality:

"The real obstacle vendors are not talking about is the implementation work. Everyone expects automation to happen in a few clicks. But implementation takes time, on one side to fine-tune models, on the other to fit the model usage within a much bigger process."

That's Philippe Cayrol, Chief Revenue and Strategy Officer at Acodis. Another vendor exec put it more bluntly about enterprise deployments specifically:

"Currently, the biggest hurdle is the need for 3-4 teams to collaborate. For smaller organizations, this obstacle can be overcome. For larger organizations, it requires consultants, project managers, Forward Deployed Engineers, and very, very strong executive sponsorship to bring an idea all the way to a model into production."

The report also pokes a hole in the "no-code means no data scientist needed" pitch. When G2 asked the vendors how confident they were that non-technical users successfully deploy production-ready models without help, the average answer was 3.25 out of 5. In practice, a technical owner still builds and integrates the pipeline; the business side defines requirements and validates outputs, not the other way around.

None of this means low-code ML tools are bad — reviewers actually rate them highest on "Meets Requirements" (8.78/10) of any attribute measured. The tools do what they say. The gap is entirely in setup and admin, where the same reviewers score them lowest (8.44 and 8.43 respectively). If you're evaluating one of these platforms, budget your timeline around data readiness and integration work, not around how fast the demo looked.

Worth a full read if you're scoping a low-code ML rollout: https://learn.g2.com/low-code-machine-learning

We shipped an AI agent and it broke in ways nobody warned us about

Shifa Mohammadi — Mon, 14 Sep 2026 10:04:06 +0000

A few weeks ago we pushed our first "real" AI agent into production — nothing fancy, just something that was supposed to triage support tickets and pull context from three internal systems. It worked great in the demo. Then it hit real traffic and started doing weird things: retrying the same failed API call in a loop, silently dropping context between steps, occasionally hallucinating a customer's account status because an upstream call had timed out.

We assumed we'd built something broken. Turns out we're far from alone — G2's State of AI Agent Builders 2026 report surveyed 7 vendors and analyzed 770 verified reviews, and the pattern it found matches almost exactly what we hit.

The single most interesting stat: 6 out of 7 vendors named API and system integration failures as the most common cause of agent workflow failures — not model quality, not prompting, not reasoning. Integration. The plumbing.

Orchestration came up just as much. Vendors said their orchestration layer is doing an average of 3.4 out of 5 possible jobs — task routing, retries, timeout handling, escalation logic, governance — all bolted onto what looks like "just a loop that calls an LLM" from the outside. One quote from a SnapLogic engineer stuck with me:

"Individual agents are relatively straightforward to build. The hard engineering is in the orchestrator: routing decisions, conditional branching, parallelization, retry logic, timeout handling, and deciding when to escalate to a human vs. retry. That logic is bespoke, it's brittle, and it's almost never documented properly."

That's exactly the code nobody wants to own six months later.

The report also pushes back hard on the "plug and play" pitch. Models drift, edge cases pile up, and if you don't have monitoring in place, quality degrades quietly until someone notices in a support escalation. One vendor put it bluntly: teams don't invest in the eval infrastructure and continuous monitoring needed to catch regressions before customers do.

None of this means agents aren't worth building — the same report found 43.5% of buyers write meaningfully more about what they like than what they dislike, and integration capability itself ranks in their top 5 most-valued features. The takeaway isn't "don't do this," it's "the orchestrator and the integration layer are the actual product, budget your time accordingly."

If you're building or evaluating agent tooling right now, it's worth a read: https://learn.g2.com/state-of-ai-agent-builders-2026