At some point a senior engineer gets pulled into a vendor selection. Usually late, usually as "can you sit in on the technical call," and usually with the commercial decision already 80% made. If that is you, here is the useful framing: you are not there to evaluate the pitch. You are there to convert unverifiable claims into checkable artifacts.
Every question you ask in a sales conversation has a free answer. "Do you write tests?" Yes. "Do you do code review?" Of course. "Is your team senior?" Very. None of that is information — it is the only possible response, so it carries no signal.
Artifacts carry signal. Here is the list I actually use.
1. The CI pipeline config from a recent project
Ask for the actual file. A .github/workflows/*.yml, a .gitlab-ci.yml, a Jenkinsfile — redacted for secrets and client names, fine.
What you are reading for:
- Are tests a merge gate, or a job that runs and is allowed to fail? A red-but-mergeable pipeline is the single most common form of theatre in this category.
- Is there a lint and type-check stage, and does it block?
- Is there any security scanning — dependency audit, SAST, secret detection?
- Are migrations run and rolled back somewhere before production?
- How long does the pipeline take? A 45-minute pipeline shapes team behaviour whether anyone admits it or not.
A team that cannot produce this file in a day either does not have one or does not have access to their own past work. Both are informative.
2. An architecture decision record
One ADR from a real project. What you want to see is a decision where the team chose the less obvious option and wrote down why, including what they gave up.
The failure mode is a document that reads like a justification written after the fact, listing only advantages. Real ADRs have a "consequences" section that contains something the author is not happy about. If every ADR they have is uniformly positive, they are producing documentation as an artifact of process compliance rather than as a thinking tool, which tells you what the thinking is like.
If they cannot produce one at all, your future internal team inherits a system whose decisions exist only in the heads of people who will not be there.
3. A production incident timeline
Ask: what broke in production on your last engagement, how was it detected, how long did it take to resolve, and what changed afterwards?
The wrong answer is "nothing significant." Everything breaks. That answer means either the systems have no users or nobody is watching.
The right answer has a detection mechanism in it (an alert, not a customer email), a timeline with real numbers, and a follow-up change that was made to the system or the process. Bonus signal if they mention something they got wrong during the response.
This one question tells you more about engineering culture than the entire case study section of a proposal.
4. Their review policy for AI-generated code
This is now the highest-signal technical question available, because the industry has not converged on an answer yet and so the responses are genuinely differentiated.
Ask: what proportion of your production code is AI-assisted, and what is your review gate for it?
Teams who have engineered around this describe specific mechanisms — diff size limits so a reviewer can actually reason about a change, mandatory human review for anything touching authentication, payments, or personal data, coverage requirements before merge, and often a rule that generated code must be accompanied by tests the human wrote rather than tests the model wrote alongside it.
Teams who have not thought about it are enthusiastic and unspecific. That is the answer you are listening for, and it takes about thirty seconds to get.
The reason this matters technically: the volume of code arriving for review went up sharply while review capacity did not. Any team that has not adjusted its process for that is accumulating a review-quality deficit whether or not they can feel it yet.
5. Evidence of production AI, if you need AI features
Distinct from the above. Using a coding assistant well and shipping a system with a model in the request path are unrelated competencies.
If your roadmap includes AI features, ask for a live production system with usage numbers, then ask three follow-ups:
- What does your evaluation harness look like, and what regression did it catch?
- What is your cost per user session, and how did you get it there?
- What happens when the provider degrades — not goes down, degrades?
Teams with real production experience answer all three immediately because they have been hurt by all three. Teams without it pivot to describing a proof of concept. Serious LLM integration work is identifiable within about two minutes of questioning, and the same holds for agentic systems where you should additionally ask about action-level permissions, idempotency, and the rollback path.
6. Where the repos and cloud accounts will live
Not a technical question in the usual sense, but you are the person in the room who understands the consequences.
Repositories in your GitHub organisation from the first commit. Cloud accounts owned by your company with vendor engineers holding scoped IAM roles, not the reverse. CI configuration, monitoring dashboards, and third-party service accounts the same.
The version where the vendor "manages it for now and hands over at the end" is how handovers become renegotiations. Raise it early, because it is trivial to arrange at kickoff and genuinely painful to unwind eighteen months in.
7. The named team, in writing
Ask which specific engineers will be assigned and what they shipped most recently. Then ask for that to appear in the statement of work, with a clause requiring written approval to change it and a minimum commitment period for the technical lead.
Team substitution after signature is the most frequent complaint in this market. It is also entirely contractually preventable, and the fix costs nothing except the willingness to ask before signing rather than after.
None of this requires you to be the decision maker. It requires you to convert seven claims into seven artifacts, hand the folder back to whoever is deciding, and let the gaps speak.
The full buyer-side version — delivery models, current US rate bands, pricing structures, compliance expectations, and realistic twelve-month cost ranges — is here: Product Engineering Services Companies in USA: A 2026 Buyer's Guide.
Frequently Asked Questions
What if a vendor refuses to share a CI config?
A redacted pipeline file from any past project contains no client-identifying information, so refusal usually means it does not exist or is embarrassing. Offer to accept it with all names, secrets, and URLs stripped. If that is still refused, note it and move on.
Is a slow CI pipeline actually a problem?
Yes, indirectly. Pipelines over about fifteen minutes change behaviour — engineers batch changes into larger commits, skip local verification, and merge on optimism. You are reading the number as a proxy for how the team works day to day.
How do I evaluate AI code review policy if we do not use AI ourselves?
You are not evaluating the policy against your own practice; you are checking that one exists and is specific. Diff size limits, mandatory human review on sensitive paths, and coverage gates are the concrete markers. Absence of any policy means unreviewed volume is entering the codebase you will inherit.
What is the difference between using AI and building AI features?
Using AI is a development practice — code generation, refactoring, test scaffolding. Building AI features means owning retrieval quality, evaluation datasets, latency budgets, cost per session, and failure behaviour when a model provider degrades. Vendors deliberately blur the two; ask for a production system with users to separate them.
Should repositories really live in our org from day one?
Yes. It costs nothing at kickoff and it removes an entire category of leverage from any future dispute. The same applies to cloud accounts, CI configuration, monitoring, and third-party service accounts.
How many artifacts should I ask for before it looks adversarial?
All of them. Every serious firm has been asked before and can produce them within a few days. A vendor that treats routine technical due diligence as an insult has told you something useful about how they will respond to scrutiny during delivery.
Top comments (0)