DEV Community

Cover image for What AI coding agents install, and what they skip
Momen Elshamy
Momen Elshamy

Posted on Originally published at installrate.com

What AI coding agents install, and what they skip

I build something almost every day, and these days an AI agent writes most of the code. Again and again I watched it choose the tools for me: the login library, the database, the email API. I never named one. It typed npm install, and that was the decision.

I kept asking myself three things. Why this one? Would another model pick something else? And is there a better tool out there that I never see, because I'm stuck with the model's choice?

So I started measuring it. Here's what came out of 2,188 answers from 11 models.

How I measured it

  • 100 build tasks in 20 categories: auth, hosted Postgres, ORM, payments, email, hosting, file storage, job queues, cron, feature flags, error monitoring, analytics, logging, search, vector databases, LLM tracing, CMS, realtime, end-to-end tests and UI components. They read like real requests and never name a product. One of them: "I need a hosted Postgres database for my Next.js side project. Pick one, set it up and connect to it."
  • One short system prompt for every model: you're a coding agent in the user's project, make the technical choices yourself.
  • Each task twice per model, temperature 1.0, through the public APIs.
  • Run 1 (2 Nov): Claude Sonnet 5.5, GPT-5.6 Terra and Gemini 3.1 Pro. 600 answers.
  • Run 2 (16 Nov): 8 open-weight models: DeepSeek V4.1 Flash, Qwen3 Coder Next, GLM 5.3, Kimi K2, MiniMax M3.1 Flash, gpt-oss-120b, Devstral 2 and Gemma 4 31B. 1,588 answers.
  • The pick is what gets installed: the install command first, then package.json, then the imports. A product the answer only talks about doesn't count. 60 random answers per run were checked one by one against the pick.
  • Both runs cost under $10 in API calls.

Some categories are already decided

Bar chart: the product installed most in each of the 20 categories, out of 30 answers from Claude, GPT and Gemini. Sentry and Playwright 30, PostHog 28, Stripe 24, Neon 20, shadcn/ui 19, AWS S3 17, Drizzle 16, Resend 16.

Sentry for error monitoring and Playwright for end-to-end tests went into all 30 answers from Claude, GPT and Gemini. The open models mostly agree: Sentry in 76 of 80 answers, Playwright in 66 of 80.

In 9 of the 20 categories, one product won more than half of the big three's answers: Stripe for payments (24 of 30), Neon for Postgres (20), Drizzle over Prisma (16 to 5), Resend for email (16).

The bottom of the list matters too. Auth, feature flags and realtime have no clear winner, so there the pick still depends on the model and on how you ask.

Named isn't installed

The number I find most useful is the gap between being mentioned and being installed.

Chart: how many of the 30 answers named each product, and how many installed it. Supabase 4 to 0, Datadog 9 to 0, Cloudflare R2 6 to 0, Mailgun 4 to 0, Algolia 9 to 1, Postmark 9 to 4.

  • Supabase came up in 4 of the 30 hosted Postgres answers and was installed in none. Neon got 20. Claude, right after installing Neon's driver: "If you'd rather have auth and storage bundled too, Supabase is the alternative."
  • Datadog was named in 9 logging answers and installed in 0. The agents set up Grafana and Prometheus (12) or OpenTelemetry (10).
  • Cloudflare R2 was named 6 times, usually as "the same code with a different endpoint". The install was AWS S3 every time.
  • Mailgun was named 4 times and never installed. Resend got 16.

The open models make it even clearer. Across all 11 models, Datadog was named in 28 answers and installed in none. Mailgun: named 27 times, installed 0. R2: 14 and 0. (Supabase does better there: the open models installed it 6 times out of 80.)

A mention shows the agent knows the market. The install is the decision.

The models don't agree

Stacked bars: what Claude, GPT and Gemini installed for login and auth, 10 answers each. Claude mostly Auth.js, GPT wrote it by hand 4 times, Gemini mostly Clerk.

Auth is the clearest case. Claude reached for Auth.js. Gemini went to Clerk. GPT wrote its own login, with sessions and password hashing, in 4 of its 10 answers.

It goes beyond auth. Claude never installed Prisma; Gemini did 4 times out of 10. GPT kept vectors in Postgres with pgvector 6 times out of 10; Gemini never did. So "what AI recommends" depends on which AI.

The open models: same defaults, three real differences

Then I ran the same 100 tasks on 8 open-weight models. I expected a different world. It mostly isn't: they install the same first pick as Claude, GPT and Gemini in 17 of the 20 categories.

Chart: share of answers that installed each product first, Claude, GPT and Gemini against 8 open models. Unleash 20% to 31%, PostHog flags 33% to 16%, OpenTelemetry 33% to 51%, Grafana and Prometheus 40% to 30%, LangSmith 30% to 33%, Langfuse 33% to 22%, built by hand 9% to 14%.

The three where they don't:

  • Feature flags: Unleash first (31%). The big three go to PostHog (33%), which the open models pick only 16% of the time.
  • Logging: OpenTelemetry first (51%), not Grafana and Prometheus.
  • LLM tracing: LangSmith first (33%), Langfuse 22%. And in 38% of these answers they built the tracing themselves.

Two more patterns:

  • They build it by hand more often: 14% of all answers, against 9%. In feature flags, 28% of their answers rolled their own. In file storage, 41% installed no storage service at all; most just took the upload with Multer.
  • Single models have strong habits. Devstral 2 picked Cypress for end-to-end tests in 7 of 10 answers, where the big three never did. gpt-oss-120b sent email over plain SMTP with Nodemailer in 8 of 10. Gemma 4 picked LangSmith in 7 of 10. Qwen3 Coder and Kimi K2 went to OpenTelemetry in 8 of 10.

What I take from it

As a builder, I've stopped taking the default for granted. The model's first pick is often good. It isn't always the best tool for my case, so when it matters, I name the tool myself. And if you switch models, expect a slightly different stack from each.

If you make a dev tool, part of your funnel now runs through an agent that types the install command for the user. That reader doesn't see your landing page. It sees your package name, your docs and how short your setup is. Neon's driver goes in with one line, and the models describe Supabase as a bigger platform. That's true, and it's why Supabase ends up as the alternative and not the default.

The limits

  • These are API answers, not agents working inside a real repo with its own files and history. What's already installed would change things.
  • About 30 answers per category for the big three and 80 for the open models. Treat a gap of a few points as a tie.
  • Models change every few months, so every number carries its run date.

The data

Every prompt and every raw answer is public, free to reuse under CC BY 4.0:

If a number looks wrong to you, say so in the comments. I'd rather fix the data than defend it.

And I'm curious: what did your agent install for you lately that you didn't expect?

I wrote this with help from my AI assistant, which also runs the benchmark. Every number comes from the published data linked above.

Top comments (1)

Collapse
 
the_daily_flare profile image
The daily flare •

I think knowing what gets skipped tells me more than the install list itself.