I build plenty of apps due to the nature of my job. I use coding agents extensively. I decide what gets built and review what the agent writes. Tests are a big part of my development process as it ensures that whatever my agents build is up to standard.
Recently, I was working on a game that allows users to work as a bartender. In this game, you are expected to take orders, deliver them, and charge your customers. When the agent built the feature for charging the customers, it actually implemented a false-positive test: the test passed, but it shouldn’t have. In the application, orders arrive through an Amazon Simple Queue Service (Amazon SQS) queue. One of its tests simulated a crashed worker by charging an order and then forgetting to delete the message, forcing the message to be redelivered. The agent used the receipt handle as its key to prevent double-charging. Because SQS generates a new receipt handle for each delivery of the message, the check didn't recognize the order. This means the customer would have paid twice.
How could we have prevented this? Mocking an AWS service’s response works well for the happy path. But this example shows the dangers of assuming exactly what the service will return. Every rule the service enforces must first be in the test author's head, and the mock never tells you what you left out.
To cover what you are missing, you can run the tests on the actual cloud. But this means having an infrastructure to test against. Sometimes we run thousands of tests on different branches, but that incurs costs. On top of that, integration tests are sensitive to concurrent tests modifying shared resources. An alternative is something we call emulation. An emulator acts like the service you are calling, allowing you to run your app against your target APIs on your machine. Emulators can be used for a range of use cases, from emulating the behavior of an Android phone to emulating cloud APIs.
Today we announced the AWS Emulator, a local emulator that lets you test your AWS applications on your own machine, with no network access and no additional service fee. Your code sends the same requests to AWS Emulator instead of the cloud. The emulator checks each request, stores the data, and answers the way the service would. The rules live in the emulator, so a test can run into rules its author never thought about. It needs no deployment and no AWS account, so you can run it on every change.
When I ran the same tests for my bartender game against AWS Emulator, the order was re-delivered after its visibility timeout (the time Amazon SQS hides a message that's being handled), with a new receipt handle (the ID Amazon SQS gives each delivery). The test failed, and the agent automatically implemented a new solution that used the order id to prevent double charging.
How emulation got here, and where it fits
Running AWS services on your own machine isn't new. AWS has shipped local emulators for more than a decade. DynamoDB Local came out in 2013. The AWS Serverless Application Model (AWS SAM) CLI runs AWS Lambda functions and Amazon API Gateway routes on your machine. AWS Step Functions, AWS Glue, and the Lambda runtime each have their own local tools.
A broader ecosystem has also evolved toward helping developers emulate cloud behavior. AWS partner LocalStack has helped many developers test AWS applications locally for years. How well any emulator does the job depends on its fidelity, meaning how closely it answers like the real service. Waldemar Hummer, co-founder and CTO of LocalStack, sees the next step as a shared one:
"At LocalStack, we believe that cloud development should be fast, local, and accessible to every developer. AWS's investment in local cloud emulation is a testament to the role of local cloud development for the AWS community. We believe there's a significant opportunity to move the ecosystem forward together, from improving emulator fidelity and establishing open testing standards to integrating emulators of all kinds into the development experience our users rely on. LocalStack is excited to work with AWS to build on this vision of a unified emulator ecosystem."
We wanted to contribute to this ecosystem. Our AI agents have generated and are constantly evolving a suite of fidelity tests that they use to autonomously improve AWS Emulator’s fidelity. We’ve made both these fidelity tests and AWS Emulator’s source code available on GitHub as a public domain dedication under CC0 1.0 license. We did this so that anyone can use them to learn from how we approached the problem or integrate them directly into their product.
Why emulators are so important for agents
With the increase usage of AI coding agents, it is increasingly important for agents to be able to test end-to-end locally. This is why one of our primary focuses is building products like AWS Emulator that help customers give their agents more autonomy when building software.
Every copy of AWS Emulator is isolated, with its own port, keys, and state. Several agents can each run their own copy without sharing a test account, and none of them needs credentials for your AWS account. This enables you to run multiple agents in parallel, each implementing and testing different features.
We used similar techniques as we built AWS Emulator itself. Agents verified their implementations locally, so they could build more autonomously and with higher quality. As a result, we delivered faster, and customers got the tools they needed to innovate faster.
A quote from James Greenfield, VP of AWS Platform Experience, sheds more light on this commitment:
"Our customers have been asking for high fidelity emulators. We've focused on some of our most popular services such as DynamoDB, but this has left customers without options for some of our other services. AI has enabled us to move much more quickly. Because building an emulator is particularly well suited to agentic development, we've been able to use thousands of agents to deliver a high-fidelity emulator that covers the breadth of AWS. We think customers will also love that any discovered issue is fixed in a matter of hours."
Conclusion
An emulator checks your code against the service's own rules, including the ones you never thought of checking. In my project, that was the difference between a passing test and a customer paying twice. Coding agents widen the gap, because the agent that writes your code also writes your tests, including any mocks. Keep the mocks for the logic that you control. However, when your code depends on how an AWS service behaves, run it against an emulator.
To get started, see the AWS Emulator documentation and the AWS Emulator GitHub repository. Also ask your coding agent to install the Emulator skill to check whether your application can run fully against the emulator. Run your tests against the emulator, and if one of your tests catches something your mocks missed, tell me about it in the comments.
Top comments (0)