DEV Community: Akash Thakur

What Happens When Users Don't Behave as Expected?

Akash Thakur — Fri, 09 Oct 2026 07:08:37 +0000

In the previous article, we looked at why functional testing alone is not enough for AI applications.
Traditional testing usually starts with a simple assumption: developers know how users are expected to interact with the application, so they can build test cases around those scenarios.
AI applications make that assumption harder to maintain.
Users can interact with an AI system in ways that developers never explicitly planned for. They can ask unexpected questions, change context, combine instructions, or simply explore what the system is capable of doing.
The application may still respond.
And that response can reveal behavior that was never covered by the original test cases.

*Users Don't Always Follow the Test Cases
*
When developing an application, engineers typically define expected use cases.
For an AI assistant, this might include tasks such as answering questions, summarizing information, generating content, or helping users complete a workflow.
Testing these scenarios is important.
However, real users are not limited to predefined test cases.
A user might:

Rephrase a request in an unexpected way.
Ask multiple unrelated questions in the same conversation.
Provide incomplete or ambiguous instructions.
Change their request based on the previous response.
Try to influence how the AI interprets an instruction.
Continue interacting with the system after receiving an unexpected response.

These interactions may still be technically valid inputs.
The application may accept them, the API may return a successful response, and there may be no software error.
Yet the AI's behavior can still be different from what the development team intended.
This is one of the fundamental differences between testing traditional application logic and testing AI behavior.

*Unexpected Input Doesn't Always Mean an Error
*
In conventional software, unexpected input often produces a predictable outcome.
The application may reject the input, return a validation message, or follow a predefined error-handling path.
AI systems are different because they are designed to interpret natural language.
A request that was never included in the original test plan can still be understood and answered.
Consider an internal AI assistant designed to answer questions about company policies.
A functional test might verify that the assistant correctly answers:
"What is the company's leave policy?"
But users could interact with the same assistant in many other ways.
They might ask the question indirectly, combine it with another request, provide misleading context, or repeatedly modify their instructions.
The system may continue generating responses even though those interactions were never explicitly tested.
This means a successful API response does not necessarily mean the AI behaved correctly.
From an engineering perspective, the important question becomes:
Did the AI produce an acceptable response under conditions that were not explicitly defined during development?

*The Behavior Is Part of the System
*
One of the challenges with AI applications is that the model's behavior becomes part of the application's overall behavior.
The application code may remain unchanged while the resulting AI response changes because of differences in:

User input
Conversation context
Prompt instructions
Retrieved information
Model configuration
Model versions
Other surrounding application components

This makes AI testing more than a matter of checking individual functions.
Engineers also need to understand how the system behaves when those conditions change.
For example, an AI assistant may perform correctly during a controlled test with a predefined prompt.
But changing the wording, adding additional context, or continuing the conversation can produce a different response.
The challenge is not necessarily that the system is broken.
The challenge is that its behavioral boundaries may not be fully understood.

*Why Unexpected Behavior Is Difficult to Test
*
Testing a small number of expected scenarios manually is relatively straightforward.
Testing every possible way a user could interact with an AI system is not.
Natural language creates an enormous number of possible inputs.
Two requests can communicate the same intention using completely different wording. A single conversation can also evolve based on previous responses, creating additional combinations that are difficult to predict in advance.
This makes exhaustive testing impractical.
It also creates another engineering problem: how do we decide which unexpected behaviors are important enough to test?
Teams need a systematic way to challenge AI applications rather than simply trying random prompts and hoping to discover problems.
The testing process needs to consider different types of user behavior, evaluate the resulting responses, and determine whether the observed behavior is acceptable.
That shift—from testing only predefined scenarios to deliberately exploring unexpected behavior—is an important step toward more robust AI testing.

*From Expected Behavior to Adversarial Thinking
*
Unexpected user behavior does not always mean malicious intent.
Sometimes users simply ask questions differently from how developers expected.
But AI systems can also be deliberately challenged.
A person may intentionally try to push an AI application beyond its intended behavior and observe how the system responds.
This introduces an important concept in AI security testing: adversarial thinking.
Instead of asking only:
"Does the application work as designed?"
Engineers can also ask:
"What happens if someone deliberately tries to make it behave differently?"
This mindset changes the way AI applications are tested.
The objective is not simply to find software bugs. It is to understand the limits of the AI system and identify behaviors that may require additional controls, safeguards, or improvements.
And as AI applications become larger and more widely deployed, doing this manually becomes increasingly difficult.
That leads to the next engineering challenge:
How can teams test a growing number of AI interactions systematically and at scale?
That is where the need for structured and repeatable AI security testing becomes more apparent.

Conclusion

AI applications operate in an environment where users have enormous freedom in how they communicate with the system.
Developers can define expected workflows, but they cannot realistically predict every possible interaction.
An unexpected input may not cause an application error. Instead, the AI may respond in a way that exposes behavior the development team never considered.
For this reason, trustworthy AI requires engineers to think beyond predefined test cases.
Testing needs to explore not only whether the application works, but also how it behaves when users interact with it in unexpected or challenging ways.
This shift in thinking is an important foundation for AI Red Teaming.
In the next article, we'll look at the engineering challenge of testing AI behavior at scale—and why manual testing becomes increasingly difficult as AI applications grow.

*About This Series
*
Building Trustworthy AI: An Engineer's Journey into AI Red Teaming is a technical series by Nyuway exploring the engineering challenges behind testing, evaluating, and securing AI applications.
The series starts with the fundamentals of AI testing and gradually moves toward adversarial testing, AI Red Teaming, scalable security testing, findings, continuous evaluation, and practical implementation.

Learn more about Nyuway: https://nyuway.ai/
Explore ARTP: https://nyuway.ai/artp
Book a demo: https://nyuway.ai/contact-us
Contact: contact@nyuway.ai

AI Red Teaming • AI Security • LLM Security • Generative AI • Software Testing • Nyuway • ARTP

Why AI Applications Need More Than Functional Testing

Akash Thakur — Wed, 16 Sep 2026 08:53:29 +0000

Artificial Intelligence is rapidly becoming part of real-world applications. Developers are using Large Language Models (LLMs) for customer support, internal assistants, knowledge systems, automation, and many other workflows.
But AI applications introduce a challenge that traditional software testing doesn't completely solve:

An AI application can pass functional tests and still behave unexpectedly.
That makes testing AI more than just checking whether the application works.

Functional Testing Isn't Enough
Traditional software applications generally have well-defined inputs, processing logic, and expected outputs. Developers can create test cases around those expectations and verify whether the application produces the correct result.
For example, if an application is designed to calculate an invoice total, a test can provide a known set of values and verify the expected total.
AI applications are different.
An AI-powered application may involve a user input being processed by application logic and then passed to an LLM, which generates the final response. The response can be influenced by the user's input, conversation context, system instructions, retrieved information, and the behavior of the underlying model.
Because of this, the same application can produce different responses when the input or context changes.
A functional test might confirm that an AI assistant can successfully summarize a document and return a reasonable summary.
However, that test alone doesn't tell us how the assistant will behave when the user provides an unexpected or deliberately challenging input.
This creates a gap between testing whether the application works and understanding how the AI behaves.

The Happy Path Problem
Most application tests are designed around expected user behavior.
Developers define what the user should do, what the application should return, and whether the workflow completes successfully.
But real users don't always follow the happy path.
They may:
Ask unexpected questions.
Provide unusual or ambiguous instructions.
Combine multiple requests into a single interaction.
Provide information that the application wasn't specifically designed to handle.
Try to push the system beyond its intended behavior.
For traditional applications, unexpected input may result in an error, validation failure, or rejected request.
For AI applications, the system may still generate a response.
And that response may not always be what the developer expected.
This creates an important testing question:
What happens when the input is valid from the application's perspective, but intentionally designed to challenge the AI?
Functional testing alone cannot provide a complete answer to that question.

Testing AI Behavior
This doesn't mean functional testing should be replaced.
Functional testing remains essential for validating APIs, application logic, integrations, authentication, data processing, and expected workflows.
AI applications simply require additional testing dimensions.
Developers need to think beyond whether a feature produces the expected output and consider how the AI behaves across different inputs, contexts, and interaction patterns.
This is particularly important because AI behavior is not always deterministic in the same way as traditional application logic.
A small change in the prompt, conversation context, retrieved information, model version, or application configuration can affect the resulting response.
As a result, testing an AI application is not only about verifying individual features. It is also about understanding the boundaries and behavior of the AI system itself.

Why This Matters to Developers
When an AI application is small and experimental, manually testing a few interactions may be enough to identify obvious problems.
As the application grows, that approach becomes increasingly difficult to maintain.
More users introduce more types of inputs.
More integrations introduce more possible contexts.
More models and model updates can introduce behavioral changes.
And more business-critical use cases increase the impact of unexpected AI behavior.
For developers and engineering teams, this creates a practical challenge:
How can AI behavior be tested systematically instead of relying only on manual experimentation?
This is where a different approach to testing begins to become important—one that intentionally considers unexpected and challenging user behavior rather than focusing only on predefined test cases.
That approach leads into the broader discipline of AI Red Teaming, which we will explore in the next stages of this series.

Conclusion
AI has changed how modern applications behave, and testing practices need to evolve with it.
Functional testing can tell us whether an application performs its intended functions, but it doesn't necessarily tell us how the AI will behave when users provide unexpected inputs or interact with the system in ways developers did not anticipate.
For trustworthy AI, testing needs to move beyond the happy path.
The goal is not to replace traditional software testing, but to extend it so that teams can better understand, evaluate, and improve AI behavior.
Understanding this testing gap is the first step toward building AI applications that are more secure, reliable, and resilient.
In the next article, we'll explore what happens when users don't behave as expected—and why unexpected user behavior creates a unique testing challenge for AI applications.

About This Series
Building Trustworthy AI: An Engineer's Journey into AI Red Teaming is a technical series by Nyuway exploring the engineering challenges behind testing, evaluating, and securing AI applications.
The series starts with the fundamentals of AI testing and gradually moves toward adversarial testing, AI Red Teaming, scalable security testing, findings, continuous evaluation, and practical implementation.

Learn more about Nyuway: https://nyuway.ai/
Explore ARTP: https://nyuway.ai/artp
Book a demo: https://nyuway.ai/contact-us
Contact: contact@nyuway.ai

AI Red Teaming • AI Security • LLM Security • Generative AI • Software Testing • Nyuway • ARTP