DEV Community

Naman Verma
Naman Verma

Posted on

“Real Developers Test in Production.” But Should They?

#go #beginners #tutorial #testing

test-in-prod-meme

You've probably heard this joke before. It pokes fun at developers who skip testing, push code straight to production, and let real users discover the bugs.

But there's a reason the joke exists. Software that works on a developer's machine can behave very differently when real requests, databases, and multiple services get involved.

A broken update endpoint could corrupt customer data. An unhandled error could take down a critical feature. A bad deployment could turn a routine release into hours of debugging, incident response, and frustrated users.

And fixing the code is only part of the problem. Someone has to identify what broke, reproduce the issue, assess its impact, roll back or deploy a fix, and verify that the same bug doesn't return. Depending on the organization, that work can involve developers, QA engineers, SREs, and incident responders.

So, should developers test in production?

Their are different types of Production testing which are done like A/B testing, Canary Releases, Feature Flags, Shadow Testing, Smoke testing etc.

But these testings are done for a specific purpose and that is to test the application in a production environment. But these testings are not enough to catch all the bugs that our existing tests miss before they affect real users.

Let's build a small application, break it deliberately, explore the testing techniques that help prevent these failures, and see how Keploy can turn recorded API interactions into reusable regression tests so we dont have to write the test cases and guess what scenarios to test.

Sample Application Setup

We have a sample Bookmark application with Go, Echo and PostgreSQL. It has all the CRUD operations for bookmarks along with a simple UI to interact with the application.

You need:

Clone and run this sample application locally using:

git clone https://github.com/Namanv0509/bookmark.git && cd bookmark

go mod download
Enter fullscreen mode Exit fullscreen mode

To start the application run:

docker compose up --build -d
Enter fullscreen mode Exit fullscreen mode

Check the health of endpoint:

curl http://localhost:8080/health
Enter fullscreen mode Exit fullscreen mode

Run the following command to stop it:

docker compose down
Enter fullscreen mode Exit fullscreen mode

Let's Break Something First

We'll use bookmark, a simple bookmark manager built with Go, Echo, and PostgreSQL.

Users can save bookmarks, add notes and tags, view their saved links, update them, and delete them. The application has a small browser interface and a REST API.

Start the application locally and Open http://localhost:8080 and create a bookmark. You can also interact with the API directly.

save-bookmark

curl -i -X POST http://localhost:8080/links \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.google.com/",
    "title": "Google Search",
    "notes": "Sounds like a good search engine. It helps me find good blogs like this.",
    "tags": ["Search Engine"]
  }'
Enter fullscreen mode Exit fullscreen mode

The application should save the bookmark and return its details.

Now let's imagine we're making a small improvement to the update endpoint. We change its SQL query, test that the application starts, and deploy the new version without checking whether the update behavior still works.

Here's the mistake we introduce:

UPDATE links
SET url = $1,
    title = $3,
    notes = $2,
    tags = $4,
    updated_at = NOW()
WHERE id = $5;
Enter fullscreen mode Exit fullscreen mode

The title and notes parameters are swapped. We intended to assign the title from $2 and the notes from $3, but accidentally did the opposite.

Let's send an update request with distinct values to see the effect.

bug-bookmark

curl -i -X PUT http://localhost:8080/links/1 \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.google.com/",
    "title": "Title",
    "notes": "Notes",
    "tags": ["updated"]
  }'
Enter fullscreen mode Exit fullscreen mode

The endpoint still return a successful response. The application is running and everything looks fine. Yet the bookmark's title can now become notes.

We've introduced a bug without necessarily causing a crash.

Imagine the same mistake in a production application. Users might save incorrect information, downstream services might consume the corrupted data, and the problem might go unnoticed until someone reports it.

Let's imagine a different mistake: a database connection failure that causes requests to time out. That could make a feature unavailable, potentially affecting every user who relies on it.

These are different failure modes, but they reveal the same problem: a successful deployment doesn't guarantee correct behavior.

What Does “Testing in Production” Actually Mean?

The phrase gets used to describe several different practices, but they aren't interchangeable.

Skipping tests and deploying directly

This is the reckless interpretation of the joke: code goes straight to production without sufficient verification.

It might appear faster initially, but failures can become expensive. Developers may have to investigate unfamiliar production states, QA teams may need to reproduce issues, and operations teams may have to mitigate an outage.

Testing in a staging environment

Staging

A staging environment is a separate environment intended to resemble production. Teams can deploy a release candidate there and run tests before exposing it to users.

Staging is useful, but it isn't a perfect copy of production. Data volumes, configurations, integrations, and traffic patterns can differ.

Testing in production deliberately

Some teams perform controlled tests in production because certain conditions are difficult to reproduce elsewhere.

For example:

  • Canary releases send a small portion of real traffic to a new version before a wider rollout.
  • Feature flags let teams enable or disable features for selected users and gradually expand access.
  • A/B testing compares different versions of a feature with different user groups to measure outcomes such as engagement or conversion.
  • Shadow testing mirrors real requests to a candidate version so its behavior can be evaluated without using its responses for normal user-facing results.results.

The Different Layers of Software Testing

Different tests answer different questions. A practical testing strategy combines them rather than relying on a single tool.

Type Question it answers bookmark example
Unit testing Does this piece of logic work? Does URL validation reject ftp:// URLs?
Integration testing Do components work together? Can the API save a bookmark in PostgreSQL?
API testing Does the HTTP interface behave correctly? Does an invalid request return the expected status and response?
End-to-end testing Does a complete user workflow work? Can a user create a bookmark through the browser and see it in the list?
Regression testing Did a change break existing behavior? Does updating a bookmark still preserve its title and notes?
Load testing How does the application behave under traffic? Can the list endpoint handle concurrent requests?

These categories can overlap. For example, an API test that exercises the actual database is also an integration test. Regression testing describes the goal of detecting unintended changes, not a particular test level.

Let's look at what each approach gives us.

Unit Testing: Check the Small Pieces First

A unit test checks a small, isolated piece of code.

For our bookmark application, we want to verify that the validation logic accepts valid HTTP URLs, rejects unsupported schemes, and requires a title. The test cases are already available in main_test.go.

Run the following commands from your project directory:

go test ./...
go test -v ./...
go test -cover ./...
Enter fullscreen mode Exit fullscreen mode

The first command runs the tests, the second displays individual test results, and the third reports code coverage.

You should see output similar to this:

=== RUN   TestValidateInput
=== RUN   TestValidateInput/valid_input
=== RUN   TestValidateInput/missing_title
=== RUN   TestValidateInput/unsupported_scheme
=== RUN   TestValidateInput/missing_URL
--- PASS: TestValidateInput (0.00s)
PASS
ok      linkvault       0.110s
Enter fullscreen mode Exit fullscreen mode

All four test cases pass, confirming that our validation logic handles these inputs as expected. However, our overall code coverage is only 7% because we've tested just a small part of the application.

Unit tests help us verify individual pieces of logic, but they don't tell us whether our HTTP handlers and SQL queries work correctly together.

For that, we need to test how the application interacts with its dependencies.

Dependencies: What Does Your Application Rely On?

A dependency is something your code relies on to perform its job.

bookmark has dependencies at several levels:

  • Echo handles HTTP routing and middleware.
  • A PostgreSQL driver lets Go communicate with the database.
  • PostgreSQL stores bookmark data.
  • The frontend depends on the API returning data in the expected format.

Some dependencies are libraries. Others are external systems that must be running and correctly configured.

This distinction matters when testing. Our validation function can run without PostgreSQL, but creating a bookmark requires a database connection, a valid schema, and a working SQL query.

When a test fails, we want to know whether our own logic is wrong or a dependency is unavailable or behaving differently than expected.

One way to investigate this is to test with the real dependency. Another is to replace it with a test double.

Integration and API Testing: Bring the Pieces Together

Let's return to the running application.

An integration test can send a request to bookmark, let the actual handler execute its SQL query, and verify that PostgreSQL stores the expected data.

An API test focuses on the behavior exposed through HTTP. We might check that:

  • Creating a valid bookmark returns the expected response.
  • An invalid URL produces a validation error.
  • Retrieving an unknown ID returns the appropriate not-found response.
  • Updating a bookmark changes its fields correctly.
  • Deleting a bookmark removes it from subsequent results.

For example:

curl -i -X POST http://localhost:8080/links \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://keploy.io",
    "title": "Keploy",
    "notes": "API testing tutorial",
    "tags": ["testing"]
  }'
Enter fullscreen mode Exit fullscreen mode

Then inspect the result:

curl http://localhost:8080/links
Enter fullscreen mode Exit fullscreen mode

These requests exercise more of the real application than our isolated unit tests.

The problem is that manually repeating these requests after every code change quickly becomes tedious. We also need to ensure the same scenarios are tested consistently.

Regression Testing

How do we know a code change hasn't broken behavior that already worked?

That's where regression testing comes in. It helps us verify that existing functionality continues to behave as expected after changes to the code.

Their are severl tools we can use for it like:

For our demo, we'll use Keploy to record API interactions and replay them after code changes. This lets us catch unexpected changes in application behavior without manually writing every test case.

Keploy can also mock supported dependency interactions during replay, reducing the need to run every dependency.

1. Record the API interactions

Start Keploy's recording workflow:

keploy record -c "docker compose up --build" \
  --container-name "linkvault-app" \
  --build-delay 60
Enter fullscreen mode Exit fullscreen mode

Your Keploy logs should look something like this in keploy-logs.txt

🐰 Keploy: 2026-10-22T12:00:59.071063+05:30   [34mINFO[0m   Starting Keploy {"version": "3.8.79", "apiServerURL": "https://api.keploy.io", "inDocker": false}
🐰 Keploy: 2026-10-22T12:01:01.555659+05:30   [34mINFO[0m   AppName in config (bookmark) does not match current directory name (linkvault). using current directory name as appName
🐰 Keploy: 2026-10-22T12:01:01.556065+05:30   [34mINFO[0m   agent replay posture resolved   {"replaying": false, "cmd": "record", "mode": "", "daemonset": false}
🐰 Keploy: 2026-10-22T12:01:01.577954+05:30   [34mINFO[0m   Running application using a temporary Keploy-generated Docker Compose file (will be cleaned up automatically)   {"cmd": "docker compose -f ./docker-compose-tmp.yaml up --abort-on-container-exit --exit-code-from app --build", "composePath": "docker-compose-tmp.yaml"}
🐰 Keploy: 2026-10-22T12:01:01.578072+05:30   [34mINFO[0m   Waiting for keploy-agent to be ready for docker compose...  {"Agent-uri": "http://localhost:61321/agent"}
🐰 Keploy: 2026-10-22T12:01:01.654583+05:30   [34mINFO[0m   Starting Application:   {"executing_cmd": "/bin/sh -c docker compose -f ./docker-compose-tmp.yaml up --abort-on-container-exit --exit-code-from app --build"}
🐰 Keploy: 2026-10-22T12:01:03.650059+05:30   [34mINFO[0m   Keploy agent is ready to record test cases and mocks.
Enter fullscreen mode Exit fullscreen mode

With recording active, open http://localhost:8080 and exercise the application:

  • Create a bookmark.
  • Retrieve the bookmark list and an individual bookmark.
  • Update its title and notes.
  • Delete a bookmark.

update-bookmark

We just performed an update to bookmark

🐰 Keploy: 2026-10-22T12:20:36.086862+05:30   [34mINFO[0m   🟠 Keploy has captured test cases for the user's application. {"path": "/Users/namanverma/Downloads/linkvault/keploy/test-set-0/tests", "testcase name": "post-links-1"}
🐰 Keploy: 2026-10-22T12:20:36.089196+05:30   [34mINFO[0m   🟠 Keploy has captured test cases for the user's application. {"path": "/Users/namanverma/Downloads/linkvault/keploy/test-set-0/tests", "testcase name": "get-links-1"}
🐰 Keploy: 2026-10-22T12:30:01.330101+05:30   [34mINFO[0m   🟠 Keploy has captured test cases for the user's application. {"path": "/Users/namanverma/Downloads/linkvault/keploy/test-set-0/tests", "testcase name": "post-links-2"}
🐰 Keploy: 2026-10-22T12:30:01.333557+05:30   [34mINFO[0m   🟠 Keploy has captured test cases for the user's application. {"path": "/Users/namanverma/Downloads/linkvault/keploy/test-set-0/tests", "testcase name": "get-links-2"}
Enter fullscreen mode Exit fullscreen mode

Use distinct values for the title and notes so that incorrect updates are easier to detect.

Stop recording with Ctrl+C. Keploy captures the interactions and generates test cases according to the configured setup. keploy can be configured to mask senstitive information and handle noise.

2. Introduce a regression

Let's deliberately introduce a bug in the update handler's HTTP response.

In main.go, locate the end of updateLink().
Just before return c.JSON(200, link) add the following line:

// Intentional regression: swap title and notes in the response.
link.Title, link.Notes = link.Notes, link.Title
return c.JSON(200, link)
Enter fullscreen mode Exit fullscreen mode

This simulates a response-mapping bug: the API returns the title as the notes and the notes as the title.

3. Replay the captured tests

With the regression in place, run:

keploy test -c "docker compose up --build" \
--container-name "linkvault-app" \
--build-delay 60 \
--delay 10
Enter fullscreen mode Exit fullscreen mode

Heads up: non-deterministic fields. Fields like created_at, updated_at (set by NOW()), and auto-generated IDs change on every run, keploy detects them and mark them as noise fields.

Keploy replays the captured requests against the rebuilt application and compares the resulting HTTP responses with the recorded responses.

Because the values differ, Keploy should flag the response mismatch, provided those fields are included in its response comparison.

and for sure you can see that in the test summary as well

 <=========================================> 
  COMPLETE TESTRUN SUMMARY. 
        Total tests: 2
        Total test passed: 1
        Total test failed: 1
        Total time taken: "28.14 s"

        Test Suite Name         Total Test      Passed          Failed          Time Taken      Failed Testcases

        "test-set-0"            2               1               1               "28.14 s"       
<=========================================>
Enter fullscreen mode Exit fullscreen mode

Your summary can vary depending on what and how many requests you have sent and your endpoint configs

We never had to write the test cases and we could catch the regression by replaying our traffic recorded previously. We can automate this testing with CI/CD, we will cover that in next article. 🐰

Conclusion

There's no single testing technique that catches every bug. Understanding when to use each approach helps us build more reliable software and catch problems before they reach production.

In the upcoming articles, we'll dive deeper into different types of testing, how they work, and when to use them.

If you'd like to explore further, here are two useful resources:

Until next time, see you in the next one!

Top comments (0)