DEV Community: Susmit Tyagi

When the LLM Gets It Wrong, Is the Model Really the Problem?

Susmit Tyagi — Thu, 29 Oct 2026 09:47:13 +0000

This article expands on the “Clarify the need and build a shared mental model” section of Before the First Line of Code: How AI Helps Turn Requirements into a Shared Design.

Have you ever wondered why an LLM does not return the result you expected?

Our first reaction is often to blame the model:

It did not understand the requirement.
It used the wrong design pattern.
It missed an authorization check.
It returned the wrong data.
It made unnecessary assumptions.

Sometimes the model is the problem. But often, the real issue is that important context was never provided.

When information is missing, the LLM must either stop and ask questions or make assumptions. Because LLMs are designed to provide useful answers, they often choose the second option.

The result may look technically convincing while being inconsistent with the organization’s architecture, security policies, API standards, or business rules.

The same thing would happen with a developer.

Imagine asking a developer who is new to the team:

Build an API that allows a store manager to retrieve customer orders.

The developer would need more information before writing code:

What are the organization’s API standards?
Is there a similar API to follow?
How is the caller identified?
How do we determine which stores the manager can access?
Which customer information may be returned?
How should large results be handled?
What performance and availability expectations apply?

We should not expect an LLM to know information that would also need to be explained to a developer.

The goal is therefore to give the LLM the same domain, engineering, security, and operational context that a developer would need.

Build a small context pack

A context pack is a focused collection of sources needed to understand a change. It should link to authoritative information rather than copy large amounts of documentation into the prompt.

What the context pack might tell us

The context pack may establish that:

The API must follow the organization’s existing standards.
The caller’s identity and organization come from a trusted source.
The caller must be authorized for the requested store.
Only permitted order and customer information may be returned.
Large results must be broken into manageable groups.
Customer and payment information must follow the organization’s privacy rules.
The service must follow existing monitoring and error-handling practices.

The pack may also reveal missing information:

What response-time target applies?
What is the expected volume for a high-traffic store?
What should happen when a required supporting service is unavailable?
Is audit recording required?

Missing information should remain visible. The LLM should not silently invent an answer.

Ask the LLM to explain its understanding

Before asking for code, ask the LLM to interpret the requirement using the context pack.

For example:

Using the supplied context pack, explain your understanding of the store-order retrieval requirement. Reference the source behind each important conclusion. Separate evidence, inferences, assumptions, and unknowns. Identify missing information that could change the scope, security model, API design, or implementation. Do not generate code until this interpretation has been reviewed.

This gives the team an opportunity to correct misunderstandings before they become implementation decisions.

What a good response should look like

A useful response might say:

Understanding

An authenticated store manager should be able to retrieve permitted customer orders for a store they are authorized to manage.

The API should follow the organization’s existing standards for structure, pagination, errors, security, and monitoring.

Only the information required for store operations should be returned.

Evidence

The API must follow the organization’s governance standards.
The caller’s identity must come from the approved identity system.
Store access must be verified using the established authorization process.
Similar store-facing APIs provide an approved implementation pattern.

Each conclusion should reference the source that supports it.

Inferences

The new API should probably reuse the design and authorization approach used by similar store-facing APIs.
The order search should be limited by both the caller’s organization and authorized store access.
Large order volumes will probably require pagination.
Existing logging, monitoring, and testing patterns should be reused.

These conclusions appear reasonable but still require review.

The LLM should recommend resolving the important questions before implementation.

Why this improves the result

This approach helps the LLM:

Follow organizational standards
Reuse approved design patterns
Respect security and privacy boundaries
Avoid making hidden business decisions
Identify missing requirements
Explain how it reached its conclusions
Produce an implementation that can be reviewed against evidence

It also makes review easier. The team can correct a short interpretation before reviewing hundreds of lines of generated code.

The principle is simple:

Before concluding that the LLM failed, check whether it had the context required to succeed.

An LLM should receive the same domain, engineering, security, and operational context that we would provide to any developer asked to implement the change.

From AI-Assisted Coding to Spec-Driven Development

Susmit Tyagi — Sun, 25 Oct 2026 13:03:56 +0000

Ten mindset changes for development teams

This article is part of my series on development using AI. Earlier articles explore making estimates more useful and moving from requirements to a shared design.

AI-assisted development can create an illusion of speed. Code appears quickly, but incorrect assumptions, missing requirements, and operational concerns may surface later during review, testing, and deployment.

Spec-driven development brings these decisions forward. The team establishes the intended behaviour, scope, constraints, and acceptance criteria before asking AI to implement the change.

This involves a trade-off. Spending more time upfront may delay the first visible implementation, but it can reduce rework and support longer-term stability and quality.

That requires a mindset change across developers, product teams, reviewers, and engineering leaders. Clarification, design, and verification need to be recognized as part of delivery. The speed at which code is generated is only one part of the outcome.

The specification as a shared agreement

A specification gives the team and AI a common reference for implementation and review. It captures what the change should achieve, the boundaries it must respect, and the evidence needed to confirm that it works.

It does not have to describe the entire system or settle every future decision. Define enough detail for the next increment, make unresolved questions visible, and update the specification as the team learns.

10 mindset changes for development teams

1. Start with intent, not code

Clarify the problem, intended outcome, and scope before discussing implementation. Check that the proposed change addresses the actual need and fits the existing system.

2. Make ambiguity visible

Separate confirmed requirements, assumptions, and open questions. Ask AI to identify gaps rather than silently make decisions. Resolve questions that could materially affect the design or scope.

3. Treat the specification as a living agreement

Update the specification when investigation or implementation changes the team’s understanding. Record important decisions and their rationale so the specification, code, and tests remain aligned.

4. Specify the next safe increment

Define a manageable change with clear boundaries, relevant dependencies, and acceptance criteria. Avoid overengineering or designing features that are not yet needed.

5. Define correctness beyond functionality

Consider security, privacy, performance, reliability, and observability alongside functional behaviour. Include the constraints relevant to the change, including what should happen when something fails.

6. Set measurable expectations

Replace vague requirements such as “fast” or “scalable” with agreed targets and operating conditions. Define how those targets will be verified. AI can suggest questions, but the team must confirm the expectations.

7. Define evidence before implementation

Map important requirements to acceptance criteria and verification methods. These may include automated tests, performance measurements, configuration checks, or specialist reviews.

Check that the evidence will demonstrate the intended behaviour, rather than simply confirm how the code was written.

8. Review intent before reviewing code

Review the specification for missing requirements, conflicting constraints, unsupported assumptions, and unnecessary scope. Correcting these early can prevent the same misunderstanding from spreading across code and tests.

9. Delegate execution, not accountability

AI can assist with planning, coding, testing, and review. The team remains responsible for validating the evidence, weighing trade-offs, and approving the result.

A passing test run or a confident AI explanation is an input to that judgment.

10. Treat context as shared infrastructure

Maintain domain rules, architecture decisions, engineering standards, and operational constraints as reusable project context. Give important sources an owner and keep them current.

Each specification can reference this context and add the details specific to the change.

Make room for the upfront work

This approach needs support across the team. Product teams help clarify outcomes. Engineers identify dependencies and constraints. Reviewers examine the proposed behaviour before implementation. Leaders allow time for these decisions.

The first implementation may take longer to appear. The aim is to carry fewer unresolved assumptions into development and spend less time correcting them later.

AI can accelerate execution. A shared specification helps the team direct that execution and assess whether the result is ready.

Making Architecture Decisions Clear and Reusable

Susmit Tyagi — Wed, 07 Oct 2026 11:28:05 +0000

This is a series on Development using AI. Other articles include making estimates more useful, and the moving from requirements to design.

Architecture decisions shape how a system is built and maintained. Teams may discuss options, reach agreement, and move on — but later, people may not remember why one option was chosen, what alternatives were considered, or which trade-offs were accepted.

An Architecture Decision Record (ADR) preserves that reasoning. The ADR AI skill helps teams turn a decision discussion into a clear, reviewable record. It can organize the context, suggest questions and comparison criteria, structure alternatives, and draft the ADR. The team checks the content, makes the decision, and approves it.

How AI helps create an ADR

The AI skill guides the team through the decision and helps capture:

The problem or decision being addressed
The context and constraints that shape the choice
Alternatives and a consistent basis for comparing them
The recommendation or approved decision
Benefits, drawbacks, and expected consequences
Affected components, owners, implementation notes, and security considerations
Reviews or approvals that may still be needed

As the team provides information, the AI can identify gaps — such as an option that hasn’t been considered or a consequence that needs further investigation. It can help make the comparison more consistent, but it cannot know whether the options are complete or the trade-offs are acceptable without the team’s system and business context.

The skill also frames the ADR according to its status. A Proposed ADR presents a recommendation for discussion. An Approved ADR records a decision the team has made. It can flag decisions that may need Architecture Review Board (ARB) attention. These prompts and flags help guide review; they do not grant approval.

Why keep the decision in a record?

An ADR gives future team members a way to understand not only what was chosen, but why. It can reduce repeated debates, help reviewers assess whether a proposal fits the system’s needs, and make consequences easier to revisit when requirements or technology change.

AI makes it easier to structure and preserve the discussion. The record is only as reliable as the information the team provides and reviews, so people remain responsible for validating the reasoning and approving the decision.

From Requirement to Design with AI

Susmit Tyagi — Wed, 07 Oct 2026 11:19:53 +0000

A requirement explains what someone needs. It doesn’t always explain how the system should meet that need, which parts may be affected, or what the team should consider before implementation.

That gap can lead to different assumptions, missed design choices, or late questions about security and reliability. An AI assistant can help explore those questions early and carry useful context forward.

AI can organize information, trace relevant code, suggest questions, and compare options. The team checks the facts, adds system knowledge, and decides what to build.

1. Clarify the need and build a shared mental model

Start with the problem, intended outcome, scope, constraints, and open questions. Ask the assistant to explain its interpretation so the team can correct misunderstandings before discussing a solution.

The assistant needs context to do this well. Code, architecture documents, configuration, and tests can help explain the existing application. However, a static repository may not reveal the actual cloud deployment, load balancer settings, scaling behavior, security restrictions, or business expectations.

Even checked-in infrastructure configuration may differ from the deployed environment. An existing setting also does not establish a requirement: a configured timeout does not tell us what response time users need.

Connect the relevant sources

Build a small context pack covering:

System context: components, dependencies, data flows, and architectural principles.
Environment context: deployment, load balancing, scaling, and differences between environments.
Requirements and constraints: availability, performance, recovery, data handling, and security.
Change context: the desired outcome, scope, exclusions, and unresolved questions.

The context pack can link to existing sources rather than duplicate them. Important information should have an owner and an indication of when it was last verified.

When the assistant retrieves these sources, their contents become part of the input it uses to interpret the requirement. Access to a repository does not mean every relevant file has been read.

Make uncertainty visible

Ask the assistant to separate:

Evidence: what the available sources establish.
Inferences: what those findings may imply.
Unknowns: what still needs verification.

A useful instruction is:

Explain the current behavior and requested outcome using the available sources. Reference the evidence behind your conclusions. Separate facts, assumptions, and unknowns. Identify missing information that could change the scope. Flag related improvements separately.

The team reviews this interpretation and resolves important gaps. This creates a shared mental model that can be updated as the work progresses.

2. Identify the components that need to change

With the need clarified, trace it through the system. Consider applications, services, data stores, infrastructure, and external dependencies.

For each proposed component change, ask:

Why is this change needed?
What evidence supports it?
Which other components depend on it?
Can the existing architecture support the requirement?

AI can help locate relevant code and configuration and suggest dependencies to investigate. The team checks whether those connections are correct.

Avoid introducing new components without a clear reason. Proposed changes should fit the established architecture and engineering principles, or explain why an exception is justified.

The result should be a proposed set of changes with a reason for each one.

3. Map behavior and compare options

Describe how requests and data should move through the affected components.

Work through success, failure, and recovery paths. Consider what happens when a dependency is unavailable, a request is retried, access is denied, or an instance stops during processing.

AI can draft flows and compare approaches against the requirements. The team evaluates the benefits, costs, operational effort, and risks.

Diagrams can make these discussions easier, but each flow should remain consistent with the component changes and known constraints. Any behavior that depends on an unanswered question should be marked as provisional.

4. Review and refine the design

Review the proposal from the perspectives relevant to the change. These may include application engineering, platform operations, security, testing, and product.

Ask whether the design meets the intended outcome, handles important failure cases, stays within scope, and can be deployed and supported.

AI can help prepare review questions and identify inconsistencies between the requirement, component changes, and proposed flows. Reviewers supply the system knowledge and judgment needed to resolve them.

Record decisions, their rationale, and any remaining questions. If a review exposes a missing requirement or an incorrect assumption, revisit the earlier steps.

5. Prepare for implementation

Turn the reviewed design into information engineers can act on:

Agreed component changes and responsibilities.
Acceptance criteria and verification checks.
Dependencies and unresolved blockers.
Deployment, monitoring, and rollback considerations.

AI can help draft this material and check that tasks connect back to the requirement. The team confirms that it accurately reflects the agreed design.

Example: sharing session data

Suppose the requirement is:

Users should remain logged in when requests move between application servers.

The assistant finds in-memory session storage in the repository. It records this as evidence and asks whether authentication depends on that storage. It also asks whether production runs multiple instances, whether session affinity is enabled, and whether sessions must survive instance termination.

The team confirms that login depends on session data and that instances can be removed during scaling. Security guidance identifies restrictions on the data that may be stored.

With that context, the team evaluates shared session storage. The assistant helps identify changes to session configuration, data serialization, deployment configuration, and tests.

The proposed flow covers both normal requests and storage failures. The team decides what users should experience during an outage and reviews security, availability, and operational implications before approving the approach.

The requirement started the discussion. Repository evidence, environment information, and team decisions made the design specific.

Let estimates and context evolve

An initial estimate may contain uncertainty about the existing system or the work required. Investigation and review can change that understanding, so estimates should be revisited when scope or assumptions change.

The process may follow this path:

Requirement → Initial estimate → Shared context → Component changes and flows → Design review → Implementation

Teams can bring reviews forward or revisit earlier steps. The context pack should evolve with the decisions, so later work builds on verified information.

AI helps keep that information connected. People remain responsible for validating it, making decisions, and approving the design.

Why Software Estimates Go Wrong — and How to Make Them More Honest

Susmit Tyagi — Tue, 06 Oct 2026 10:41:40 +0000

A software estimate is often requested before the team knows enough to give a confident answer.

The requirement may be incomplete. The affected parts of the system may be unclear. Another team may need to provide support. Yet someone still needs an estimate for planning.

Estimation has traditionally depended heavily on experienced developers. They carry context about the system, remember similar work, and know where surprises tend to appear. But that knowledge can be hard to share. Two people may give different estimates because they remember different projects or make different assumptions.

A single number can hide this uncertainty. A range, along with its assumptions and risks, gives the team a more honest starting point.

Estimate the uncertainty, not just the tasks

Consider a hypothetical example: a team plans to add a reporting feature to an existing application.

The visible work might include building a report and displaying it to users. But the estimate could change depending on questions such as:

Is the required data available and reliable?
Which users should be allowed to see the report?
How much data will the report need to process?
Will it need to update in real time?
What testing, security review, or changes to existing systems are needed?

Until the team investigates these questions, a precise estimate may be misleading. A useful estimate shows a range and explains what could move the work toward either end.

Use a method that fits the evidence

Different estimation techniques suit different situations:

Analogy compares the work with relevant past projects.
T-shirt sizing gives an early sense of scale when detailed planning is premature.
Three-point estimation considers optimistic, realistic, and pessimistic outcomes.
Expert judgment brings in specialist knowledge when the work is unusual or data is limited.
Parametric estimation applies a reliable measure or rate when one is available.

The technique should fit the evidence. A close comparison may make analogy useful. A major unknown may call for a wider range and further investigation.

No method removes uncertainty. The aim is to make the basis for an estimate clear enough that others can question and improve it.

Where AI can help

An AI estimation assistant can help structure the discussion. It can gather information about the requirement, scope, team, constraints, dependencies, and unknowns. If historical estimates and lessons are provided, it can use them to inform its recommendation — for example, by considering past accuracy, recurring risks, or which techniques worked well for similar tasks.

AI can also help break the work into areas the team can review and make assumptions and open questions visible. Its recommendations depend on the quality of the current information and available history. The team needs to check whether the suggested approach fits the work and the system.

Experienced developers remain essential. They validate technical assumptions, add context that may not be recorded, and challenge estimates that do not match what they know.

Treat the estimate as a forecast

A useful estimate should make clear:

What work is included and excluded
Which assumptions it depends on
What risks or dependencies could change the effort
How confident the team is, and why
What information could narrow the range

As the team learns more, it should revisit the estimate. In the reporting example, investigating the data source might reveal that it is ready to use — or that it needs cleanup first. Either result could change the range.

After delivery, comparing estimated and actual effort can reveal useful patterns. Perhaps data preparation is often overlooked, or a familiar type of work is more predictable than expected. Recording why estimates differed helps the team improve future assumptions. The goal is shared learning, not scoring individuals.

From estimate to design

Estimation can inform design by surfacing likely scope, dependencies, and unknowns:

Requirement → Past learnings → Estimate → Design → Further reviews

The path is flexible. Design may reveal new work and prompt the team to update its estimate. An estimate is not a commitment that freezes the plan; it reflects what the team knows at that point.

A good estimate doesn’t promise exactly how long work will take. It helps the team explain what it knows, what it doesn’t, and what it should learn next.