DEV Community: Mehrad Sadeghi

Automated Testing in Backend Development: A Complete Guide from Unit Tests to Fuzz Testing

Mehrad Sadeghi — Sun, 27 Sep 2026 13:42:38 +0000

If you had to change one of your critical services right now, how confident are you that the change wouldn't silently break something else in your codebase ?

And harder still — if the only way to be sure was to manually test the entire system by hand, what would you do ?

This is exactly where automated tests earn their keep.

Automated tests aren't just a tool for catching bugs. In any serious project, they're a core part of keeping the system safe to change. Tests are what let you refactor code, swap out dependencies, optimize a database, or even redesign a service's architecture — without manually re-verifying the entire system from scratch every single time.

But here's the catch: not all tests are equal.

There are many kinds of tests — Unit, Integration, Feature, Contract, Fuzz, End-to-End, and more — and each one solves a different problem. A test that's extremely valuable in one project might just be maintenance overhead in another.

What Is Automated Testing, and Why Does It Matter in Backend Development ?

In simple terms, an automated test is a program that runs part of your software, compares the actual result to the expected one, and fails the test if there's a mismatch.

For example, imagine a service that calculates the final price of an order:

subtotal = 1,000,000
discount = 10%
shipping = 50,000

If we expect the final price to equal 950,000, we can turn that expectation into a test. From then on, every time the pricing logic changes, the test automatically verifies whether the expected behavior still holds.

Automated tests are extremely useful for catching regressions, documenting system behavior, and building confidence when changing code. They can also be plugged directly into CI/CD pipelines.

The goal isn't to test every single line of code — it's to cover the system's most important risks at a reasonable cost.

Types of Tests in Backend Development

To really understand test types, you first need to define their scope — which specific part of the system a given test touches. That gives us:

Unit Test
Integration Test
Feature / Functional Test
Contract Test
End-to-End Test
Characterization Test

Alongside these, there are techniques focused more on how inputs are generated or how strong your test suite is:

Property-Based Test
Fuzz Testing
Mutation Testing

And finally, tests that check the system's performance and quality:

Performance / Load / Stress Test
Smoke Test
Regression Test

These categories aren't fully independent of each other. A Feature Test can also be an Integration Test at the same time, and fuzzing can be applied to a single Unit Test.

So what matters more than a test's name is understanding exactly what risk it covers.

What Is a Unit Test ?

Unit testing is the smallest and fastest level of testing. In a unit test, you typically isolate a specific unit of behavior and verify that, given a specific input, it produces the expected result.

That "unit" can be:

A function
A method
A class
Or a small piece of domain logic

For example, take:

calculateDiscount(price, customerType)

We could write tests like:

Customer = Regular
Price = 1,000,000
Expected Discount = 0

and:

Customer = Premium
Price = 1,000,000
Expected Discount = 100,000

In a unit test, external dependencies like the database, file system, or network are never brought in directly — instead they're replaced with test doubles such as mocks, stubs, or fakes (we'll cover each of these later in the article).

Microsoft's definition of a unit test treats it as a test of a specific component or method, and explicitly places infrastructure pieces — database, file system, network — outside the normal scope of a unit test.

Advantages of Unit Tests

Very fast
Low execution cost
CI-friendly
Pinpoints the exact location of a bug
Great for refactoring
Lets you have a large number of tests

The Downside of Unit Tests

Every component might work correctly in isolation, but still break when combined.

For example:

OrderService
      ↓
Repository
      ↓
PostgreSQL

OrderService's logic might be completely correct, but the query it sends could still be wrong. A unit test alone won't catch that.

That's where Integration Testing comes in.

What Is an Integration Test ?

Integration testing checks whether two or more parts of your application — or one part and an external dependency — can actually work together. Common examples in backend systems include:

Application + PostgreSQL
Application + Redis
Application + Message Broker
Application + File System
Application + External API
Repository + Database

Say your repository does something like:

CreateOrder(order)

In a unit test, you'd probably mock the database.

But an integration test can:

Spin up a real PostgreSQL instance
Connect the application to it
Save an order
Read it back
Verify the result

At this point, you're no longer just testing your code's logic — you're verifying how components actually interact. Integration tests generally have a larger scope than unit tests and can pull in real infrastructure.

Advantages of Integration Tests

Tests more realistic behavior
Catches database-related issues
Catches serialization issues
Verifies configuration
Gives more confidence than unit tests alone

Disadvantages of Integration Tests

Slower
More complex setup
(Probably) requires real infrastructure
Harder to debug
Failures can have many possible causes

So it doesn't make sense to write an integration test for every single piece of business logic.

What Is a Feature Test (or Functional Test) ?

A Feature Test checks a more complete capability of the system, from the perspective of its expected behavior. In a backend, this test can start from the API itself. For example, a feature around placing an order:

POST /orders

Input:

{
  "productId": 123,
  "quantity": 2
}

And we expect the response:

201 Created

{
  "orderId": 456,
  "status": "created"
}

At this point, it no longer matters how many classes exist inside OrderService.

We're testing the feature's behavior: "Can a user create a valid order ?"

That's exactly why a Feature Test differs from a Unit Test.

A Unit Test asks: does this class or function work correctly ?

A Feature Test asks: does this specific feature of the system behave as expected ?

That's why Feature Tests are so valuable in API development.

How Is a Feature Test Different from an Integration Test ?

These two concepts can overlap. For instance, a single API test might flow through:

HTTP Request
   ↓
Controller
   ↓
Service
   ↓
Repository
   ↓
Database

In that case, you've tested both the feature and the integration between its components.

So these aren't necessarily two fully separate layers. A practical way to decide is to ask:

What risk does this test actually cover ?

If the goal is to verify that a feature works correctly from the API's point of view, call it a Feature/Functional Test. If the goal is to verify the repository's connection to PostgreSQL, call it an Integration Test.

What Is a Contract Test ?

Imagine the Order service sends a request to the Payment service:

POST /payments

and expects the response to look like:

{
  "paymentId": "123",
  "status": "paid"
}

Now imagine the Payment team changes that response without telling you:

{
  "id": "123",
  "state": "completed"
}

Your unit tests might still be green. But the integration between the two services is broken.

That's where Contract Testing becomes valuable.

A Contract Test verifies that both sides of an integration agree on a specific contract. For HTTP communication, that contract can define the expected request and response shapes — and the same idea applies to message-based integrations too.

One well-known approach is Consumer-Driven Contract Testing. In this model, the consumer defines its expected needs and interactions as a contract, and the provider verifies that contract.

The Benefit of Contract Testing

A Contract Test can reduce a chunk of integration risk between services without running the entire system.

This is especially useful when:

Different teams work on different services
Services are deployed independently
APIs change frequently
You have a microservices architecture

It's important to note that Contract Testing doesn't fully replace other test types. Instead, it's designed for one specific problem:

Do the two services still agree on the expected interface ?

What Is an End-to-End (E2E) Test ?

An End-to-End Test, or E2E, checks a scenario's entire path from start to finish.

For example, in an online store:

Login
 ↓
Add Product
 ↓
Create Order
 ↓
Payment
 ↓
Order Confirmation

E2E tests run this entire path. In backend development, even without a UI, we can run this kind of scenario purely through the API:

POST /login
POST /cart
POST /orders
POST /payments
GET /orders/{id}

And finally verify:

Order Status = Paid

The Benefit of E2E Tests

They most closely resemble the system's real-world behavior. That's why, for critical business scenarios, they can create a very high level of confidence.

The Main Challenge with E2E Tests

They're expensive.

E2E tests are usually:

Slower
Harder to debug when they fail, with more complex failure modes
Dependent on a broader environment
More demanding to maintain

That's why it doesn't make sense to test every system state end-to-end.

What Is a Characterization Test ?

Here we get to one of the most important techniques for legacy projects. Imagine you inherit a project that's been in production for 10 years, but the project looks like this:

Test Coverage = Low
Documentation = Incomplete
Code = Complex

The project manager says: "Just make one small change here."

The problem ? You don't actually know what the existing code does.

Something that looks like a bug to you might actually be behavior the system has relied on for years — behavior clients now depend on.

This is where Characterization Tests come in.

A Characterization Test isn't meant to say:

"This behavior is correct."

It says:

"This is the system's current behavior."

This approach matters a lot when working with legacy code, and alongside techniques like introducing a seam, it makes it possible to change old systems gradually and safely.

A seam is a point in the code where you can change or substitute a dependency in order to test or alter behavior. For example, if a service depends directly on a database, inserting an interface between them creates a seam — one that lets you swap the real database for a mock or fake during testing.

An Example of a Characterization Test

Say you have an old function:

calculateShipping(order)

and there's no existing test for it. You don't know what result this input produces:

Order = 1,000,000
City = Tehran
Weight = 3kg
Customer = VIP

Instead of guessing what the result "should" be, you run the function and observe the actual output. Say the output is 120,000. Now you turn that exact behavior into a test:

calculateShipping(order)
Expected = 120,000

From this point on, if a refactor changes the result to 150,000, the test fails. In other words: before you fully understand the code, you've already captured its existing behavior.

When Is a Characterization Test Valuable ?

Specifically when:

The project is legacy
Test coverage is low
Documentation is incomplete
A major refactor is coming up
An old system is about to be migrated

Important note: a Characterization Test doesn't necessarily claim the current behavior is correct. You might later discover that the captured behavior is actually a bug. At that point, you can follow this path:

Current Behavior
      ↓
Characterization Test
      ↓
Understanding
      ↓
Specification
      ↓
     Fix
      ↓
Normal Test

So a Characterization Test is really more of a safety net for changing an unfamiliar system.

What Is Property-Based Testing ?

Up to this point, most of the tests we've discussed followed this shape:

Input → Expected Output

For example:

Input: 10
Expected: 20

These are example-based tests — we manually specify the inputs ourselves. But sometimes, instead of checking specific examples, we can define a property of the system instead. Say we have this function:

sort(items)

Instead of only writing:

sort([3,1,2]) == [1,2,3]

we can say: the output must be sorted, and it must contain the same number of elements as the input.

Now a property-based testing framework can generate a large number of inputs and check the property against all of them. Libraries like Hypothesis and fast-check are built for exactly this kind of scenario.

For example, it might generate:

[]
[1]
[3,1]
[10,5,8,2]
[random data...]

and check, for all of them:

isSorted(result) == true
length(result) == length(input)

When Is Property-Based Testing Useful ?

When:

The number of possible input states is large
Edge cases are hard to find manually
You have a clear invariant or property
The algorithm or domain rule is complex
You want to check more examples at a lower cost

For instance, in financial systems, you might have a property like:

The sum of all transactions must preserve the balance

Or in serialization:

decode(encode(x)) == x

These kinds of properties can check a huge range of inputs at once.

What Is Fuzz Testing ?

Fuzz Testing (or fuzzing) is a testing technique where you run a program against unexpected, malformed, unusual, or automatically generated inputs, in order to find bugs, crashes, vulnerabilities, or unexpected behavior. OWASP also describes fuzzing as a technique for finding bugs, vulnerabilities, and unexpected behavior through unexpected or corrupted input.

Say we have this API:

POST /users

and we expect it to work correctly with input like:

{
  "name": "Ali",
  "age": 30,
  "email": "ali@example.com"
}

In a normal test, we might send inputs like:

age = 30
age = 0
age = -1

But fuzzing can generate far more varied inputs, such as:

age = -999999
age = 999999999
name = ""
name = an extremely long string
email = ""
email = malformed
unusual Unicode
truncated JSON
oversized JSON
null
nested objects
unexpected arrays

The goal isn't to prove exactly what the output should be — it's to verify that the system behaves safely and predictably in the face of unexpected input.

What's the Difference Between Fuzz Testing and Property-Based Testing ?

These two concepts are closely related, and some tools even combine them — but it's better not to treat them as identical.

In Property-Based Testing, the main question is:

Does a specific property hold across a large number of inputs ?

For example, sort(list) should always return a sorted list for any valid list.

In Fuzz Testing, the question is:

If we feed the system a large number of unexpected or malformed inputs, does it crash, expose a security issue, or behave unexpectedly ?

Put simply:

Property-Based Testing
        ↓
Generate Inputs
        ↓
Check Properties

versus:

Fuzz Testing
        ↓
Generate Unexpected Inputs
        ↓
Look for Failures / Crashes / Vulnerabilities

That said, the line between the two isn't perfectly sharp. Some property-based testing tools use fuzzing techniques, and some fuzzers also work on structured inputs with defined properties. Even Hypothesis's own documentation notes the close relationship between the two.

Where Is Fuzz Testing Useful in Backend Development ?

Fuzzing can be very effective on things like:

API input:

JSON
Query parameters
Path parameters
Headers

Parsers:

JSON parser
XML parser
CSV parser
Custom protocols

Authentication and authorization:

Unexpected input for sessions, tokens, or headers

File uploads:

Empty file
Huge file
Malformed file
Unexpected format

Search and filtering:

Very long strings
Special characters
Unicode
Unexpected combinations

Serialization / deserialization:

Input designed to trigger exceptions, incorrect
behavior, or abnormal resource consumption

Trade-offs of Fuzz Testing

Its advantages include:

Finding unexpected edge cases
Discovering crashes
Uncovering unknown behaviors
Great for complex inputs
Can run long-term in CI or dedicated fuzzing environments

And its drawbacks include:

Designing a good fuzzer can be time-consuming
Not all random inputs are equally valuable
Analyzing failures can be difficult
For some domains, it's hard to define a clearly "correct" output
Running fuzzing at scale can consume significant resources

So fuzzing is usually not a replacement for unit or integration tests — it's a complementary layer for catching a different class of problems.

What Is Mutation Testing ?

If all of your tests are green, how do you actually know your tests are capable of catching bugs ?

Mutation Testing exists to answer exactly that question.

In mutation testing, a tool deliberately introduces small changes into your production code to see whether your tests can detect them. For example, it might change:

if price > 100

into:

if price >= 100

or replace:

with:

Then your test suite runs.

If the tests fail:

Mutation = Killed

That means the tests successfully caught the change.

If the tests still pass:

Mutation = Survived

That can be a sign of a weakness in your test suite.

So Mutation Testing is less a type of test for your production code, and more a method for evaluating how strong your test suite actually is.

Performance Testing, Load Testing, and Stress Testing

Sometimes the question is: does the system stay stable at 10,000 requests per second ?

This is where we enter the world of Performance Testing.

Load Testing

Checks how the system performs under a specific, expected load. For example:

1,000 concurrent users
100 requests/sec
Response Time < 300ms

The goal of a Load Test is to check system behavior under expected load. Microsoft also defines Load Testing as checking a system's ability to handle a specific load — such as a given number of concurrent users and how the system responds to them.

Stress Testing

Here you push the load beyond expected capacity to see how the system behaves:

100 req/s
200 req/s
500 req/s
1000 req/s
...

And you check:

At what point does the system degrade ?
Does it crash ?
Does it recover ?
Does the queue grow unbounded ?
Does the error rate increase ?

These tests matter a lot for high-traffic backends.

What Is a Smoke Test ?

A Smoke Test is a quick, high-level test that checks whether the system starts up correctly and its core capabilities work.

For example, right after a deploy:

GET /health

or:

POST /login

If these basic paths fail, there's usually no point running more complex tests.

Smoke tests are typically short and fast, and can run right after deployment.

What Is a Regression Test ?

A Regression Test means re-running previous tests after a change, to make sure existing capabilities still work correctly. When you find a bug, you can write a test that reproduces that exact bug.

For example, say this bug:

Order with quantity = 0

used to result in the order being created successfully.

After the fix:

quantity = 0 → 400 Bad Request

You write a test for this behavior. From then on, if someone reintroduces the same bug through a different change, the test will fail.

The difference between Regression Testing and other test types is that it isn't defined independently by scope — its focus is more about purpose and when it's run. For example, say you've already tested the CreateOrder capability and everything worked. Now you make a new change — say, updating the discount logic. If you re-run the CreateOrder test to confirm that this new change hasn't broken previous behavior, that's a Regression Test.

The key point is that a single test can serve multiple purposes at once. A unit test can also be a regression test, if it's kept around specifically to prevent an old bug from resurfacing.

The Test Pyramid

A simple way to think about a project's tests is the Test Pyramid.

The core idea is to have tests at several levels:

Unit Test → small, fast tests
Integration/Feature Test → testing multiple parts together
E2E Test → testing the whole system end-to-end

The usual recommendation is to have many small, fast tests, and fewer large ones. That's because a unit test might run in a few milliseconds, while an E2E test has to exercise multiple parts of the system and usually costs more in time and resources.

But that doesn't mean, say, 70% of a project's tests must be unit tests. The right proportion of each test type depends on the project itself and its specific risks.

So instead of asking, "What percentage of tests should be unit tests ?"

It's better to ask: "What's the simplest, fastest test that can give us confidence this part works correctly ?"

For example, if you have a simple calculation, a unit test is probably enough. But if you need to confirm that the API, database, and several services work correctly together, you need an integration or E2E test.

So the Test Pyramid isn't a fixed rule — it's just a simple guide for choosing and balancing your tests.

Comparing Different Test Types

The most important distinction between test types is the question each one is really asking:

Unit: Is this logic correct ?

Integration: Do these two components work correctly together ?

Feature: Does this capability work as expected ?

Contract: Do the two services still honor their shared agreement ?

E2E: Does the entire real-world scenario work correctly from start to finish ?

Characterization: What does the current system actually do ?

Property-Based: Does a rule hold across a wide range of inputs ?

Fuzz Testing: Can unexpected input push the system into problematic behavior ?

Mutation: Can my tests actually detect incorrect changes ?

Load Test: How does the system behave under load ?

A Real-World Scenario: From Bug to Test

Imagine that, in an online store, users can occasionally place an order with a negative quantity:

{
  "productId": 10,
  "quantity": -2
}

and the system creates the order anyway.

Step 1: Reproduce the Bug

First, we write a test that defines the expected behavior:

quantity = -2

Expected: 400 Bad Request

The test fails initially. That's a good thing — it means we've captured the bug as a reproducible test.

Step 2: Find the Bug's Location

We trace this path:

API
 ↓
Controller
 ↓
OrderService
 ↓
Validator

It turns out the validator isn't checking the quantity value at all.

Step 3: Add a Unit Test

We write a small test for this rule:

quantity = -2
Expected = Validation Error

Now this test captures the failure more precisely, at the business-rule level.

Step 4: Write a Feature Test

Next, we write a Feature Test:

POST /orders
quantity = -2

Expected: 400

This test confirms the rule is properly enforced along the real API path as well.

Step 5: Fix

We fix the validator:

quantity > 0

Step 6: Run the Test Suite

Now:

Unit Tests ✓
Feature Tests ✓
Integration ✓

If everything's green, we have much more confidence that both the rule and the API path are correctly fixed.

Step 7: Prevent Regression

From now on, any change to order validation that reintroduces quantity = -2 will make the test fail.

This is exactly where automated testing shows its real value:

The test didn't just find the current bug — it also reduced the cost of that same bug happening again.

What Happens If We Fuzz This Same API ?

Let's take it one step further. We found the quantity = -2 bug using a specific test.

But is that the only problematic case ?

These might also produce interesting behavior:

quantity = -999999999
quantity = 0
quantity = 2147483647
quantity = null
quantity = "2"
quantity = []
quantity = {}

In a fuzz test, we don't have to manually define every possible input case — tools can generate a wide variety of unexpected inputs to check that the system behaves correctly across all of them. For example:

For every generated request:

System must not crash
AND
invalid quantity must not create an order

Now, if the fuzzer finds a specific input that triggers an exception or unexpected behavior, that exact input can be turned into a fixed, reproducible test.

This is a very practical workflow:

Fuzzing
   ↓
Unexpected Input
   ↓
Bug Found
   ↓
Minimize / Reproduce
   ↓
Regression Test
   ↓
Permanent Safety Net

In Property-Based Testing, when a specific input causes a test to fail, property-based frameworks can shrink that input down to the smallest example that still triggers the failure. This is called shrinking.

How to Write Tests That You Won't Hate Them ?

Writing a test is easy. Writing a maintainable test is harder.

A few key principles:

1. Test behavior, not implementation

Instead of a test depending on this kind of internal structure:

OrderService
 → Method A
 → Method B
 → Method C

it's better to check observable behavior as much as possible. If you refactor the implementation, tests shouldn't break for no reason. Microsoft's own unit testing guidance also emphasizes readability and resilience against unnecessary changes.

2. Tests should be readable

If you have to read a test's implementation just to understand what the test does, the test is probably too complicated. A name like:

CreateOrder_WithInvalidQuantity_ReturnsBadRequest

is usually more meaningful than something like:

ShouldCallValidatorThenRepositoryAnd...

A test's name should describe the scenario.

3. Take edge cases seriously

A test that only checks the happy path isn't enough. For instance, testing:

quantity = 1

is fine — but what happens with these ?

quantity = 0
quantity = -1
quantity = MAX_INT
quantity = null

Edge cases are exactly where bugs hide.

Does 100% Code Coverage Mean the System Is Fully Tested ?

This is one of the most common misconceptions in automated testing. Say:

Code Coverage = 100%

Does that mean the system is bug-free ?

No. Coverage only tells you how much of your code ran while your tests were executing — not whether the important behaviors were actually tested correctly. For example, a test might simply do this:

function calculatePrice() {
    ...
}

and just call the function without asserting anything meaningful about the result. In that case, coverage looks high, but confidence in correct behavior stays low. So it's better to treat coverage as a signal, not a final goal.

The more important question is whether the system's core business rules and failure modes have actually been tested.

What Is a Test Double ?

A Test Double means using a temporary stand-in for a real dependency, used specifically during testing.

The common types are:

1. Stub: returns a specific piece of fixed data:

getUser() → returns User(id=10)

A stub is essentially data a test can't run correctly without.

2. Mock: used to verify invocations. For example:

PaymentService.charge()

must be called exactly once, or the test fails.

3. Fake: a simpler but still realistic implementation of a dependency. For example:

InMemoryUserRepository

used in place of a real database.

4. Spy: records information about a dependency's real behavior during a test run. For instance, letting EmailService run as normal during a test, then checking how many times sendEmail was called and with what parameters.

Worth noting: these terms don't always mean exactly the same thing across different frameworks and sources. So what matters more than the label is understanding what role a given test double actually plays in your tests.

What Test Should You Write, and Where ?

For quick decision-making, you can use this rule of thumb:

If you have simple business logic:

Unit Test:

Discount calculation
Tax calculation
Permission rules
Validation rules
Pricing

If you're interacting with a database or infrastructure:

Integration Test:

Repository
Database
Redis
Message broker
File system

If you want to verify an API capability:

Feature / Functional Test:

POST /orders
POST /login
GET /profile

If two microservices communicate with each other:

Contract Test:

Order Service ↔ Payment Service

If the whole business flow matters:

E2E Test:

Login → Order → Payment → Confirmation

If you're dealing with untested legacy code:

Characterization Test:

Current Behavior → Capture → Refactor Safely

If the input space is very large:

Property-Based Test:

Generate Many Inputs
        ↓
Check Invariant

If unexpected or malformed inputs matter:

Fuzz Testing:

Generate Unexpected Inputs
        ↓
Execute System
        ↓
Find Crash / Failure / Unexpected Behavior

If you want to know the system's capacity:

Load / Performance Test:

100 → 500 → 1000 req/s

What Does a Professional Test Suite Look Like ?

A professional backend probably has something structured like this:

tests/
│
├── unit/
│   ├── pricing/
│   ├── validation/
│   └── authorization/
│
├── integration/
│   ├── database/
│   ├── redis/
│   └── messaging/
│
├── feature/
│   ├── orders/
│   ├── users/
│   └── payments/
│
├── contract/
│   └── payment-service/
│
├── e2e/
│   └── checkout/
│
├── characterization/
│   └── legacy-order/
│
├── property/
│   └── pricing/
│
└── fuzz/
    ├── api/
    └── parsers/

This structure is just one example — you don't need to implement it exactly this way. What matters is that the team knows exactly what risk each test exists to cover.

Feedback Loop Matters More Than Test Count

One common mistake is measuring a test suite's success by the number of tests it has. For example, 5,000 tests sounds impressive. But if running them takes 45 minutes, developers probably won't run them constantly. On the other hand, 1,500 tests that run in 30 seconds might deliver far more value throughout the development cycle. That's exactly why one of the most important traits of a professional test suite is fast feedback.

The lower the scope at which you can write an appropriate test, the faster your feedback tends to be, and the easier failures are to diagnose.

The Arrange/Act/Assert Pattern

A very common structure for writing tests is the Arrange/Act/Assert pattern. This structure improves readability in small tests and even in higher-level ones. It works like this:

Arrange: set up the data and dependencies
Act: run the code you want to test
Assert: verify the result

For example:

Arrange:

price = 1,000,000
customer = Premium

Act:

result = calculateDiscount(price, customer)

Assert:

result == 100,000

Choose Tests Based on Risk, Not Habit

The most important thing to take away from this article isn't that unit tests are good or E2E tests are bad — that view is overly simplistic. The right question isn't which type of test is "the best." It's: for a specific risk, which test gives you enough confidence at the lowest cost ?

Wrapping Up

Automated Testing isn't just about writing a handful of unit tests to boost code coverage. A professional test suite should be a balanced collection of tests, each covering a different kind of risk.

Unit Tests are for small, fast logic. Integration Tests are for the connections between units. Feature Tests are for the system's capabilities. Contract Tests are for the boundaries between microservices. E2E Tests are for critical real-world scenarios. And Characterization Tests are for understanding and safely working with legacy code.

Alongside these, Property-Based Testing and Fuzz Testing offer different ways to uncover edge cases and unexpected behavior, and Mutation Testing helps you understand just how good your test suite actually is at catching incorrect changes.

A good test isn't one that runs the most code — it's one that covers the most important risk at the lowest possible cost.

Once this mindset becomes part of how you design a test suite, tests stop being a side cost and become part of your architecture and development process.

One Final Question

If tomorrow you had to refactor a legacy service that has almost no tests, what's the very first test you'd write ? What has your experience been like on real-world projects ?

The Outbox Pattern Explained: A Complete Guide to Reliable Messaging in Distributed Systems

Mehrad Sadeghi — Tue, 22 Sep 2026 05:54:39 +0000

Image by Abhinav Thakur

What happens if your database successfully commits a transaction, but your Message Broker becomes unavailable at exactly the same moment ?

Imagine a user places an order in an online store, and the order is successfully stored in the database:

Order #123 → Created

A few milliseconds later, your service tries to send an OrderCreated event to Kafka or RabbitMQ, and suddenly:

The network goes down.
The Broker becomes unavailable.
The application crashes.
A timeout occurs.

So what happens ?

The database says:

The order has been created.

But the other services say:

We have no idea about this order!

This is one of the classic problems in distributed systems and Microservices architecture: the Dual Write Problem.

This is where the Outbox Pattern comes in.

The Core Idea Behind the Outbox Pattern

The idea behind the Outbox Pattern is surprisingly simple.

Instead of writing to the database and Message Broker at the same time, we store the business data change and the corresponding message/event in the same database transaction.

We then delegate message publishing to a separate process.

This small change in architecture can make a significant difference to system reliability.

What Is the Outbox Pattern ?

The Outbox Pattern, or more precisely the Transactional Outbox Pattern, is a design pattern for reliably sending messages and events in distributed systems.

Instead of having the application directly publish an event to a Message Broker after modifying the database, the application first stores the event in an Outbox Table in the same database.

The most important point is this:

The business data change and the event stored in the Outbox must happen within the same database transaction.

A separate process then reads events from the Outbox and sends them to the Message Broker.

At a high level:

Application
      │
      │ Transaction
      ▼
┌───────────────────────┐
│       Database        │
│                       │
│  Orders               │
│  Outbox Events        │
└───────────┬───────────┘
            │
            │ Async
            ▼
      Outbox Publisher
            │
            ▼
      Message Broker
            │
       ┌────┴────┐
       ▼         ▼
   Service A  Service B

Why Do We Need the Outbox Pattern ?

To understand the Outbox Pattern, we first need to understand the problem it solves.

Suppose we have an Order Service that needs to perform two operations:

Store the order in the database.
Send the OrderCreated event to Kafka.

Our code might initially look like this:

BEGIN TRANSACTION
INSERT INTO orders (...)
COMMIT

publish(OrderCreated)

At first glance, this seems reasonable.

But there is a dangerous gap between COMMIT and publish().

What happens if the database commits successfully but the application crashes before publishing the event ?

Database
   │
   └── Order Created ✅

Message Broker
   │
   └── OrderCreated ❌

Now the system is in an inconsistent state:

The internal business state has changed.
The corresponding event was never published.

AWS also describes Transactional Outbox as a solution to the Dual Write problem: a situation where one logical operation needs to modify both a database and a messaging system, while failure in one of them can leave the systems inconsistent.

What Is the Dual Write Problem ?

A Dual Write occurs when one logical operation needs to be writtern into two independent systems.

For example:

Database
    +
Kafka

Or:

Database
    +
RabbitMQ

Or even:

Database
    +
External API

The problem is that we usually don't have a shared transaction between these two systems.

For example:

1. UPDATE database ✅
2. SEND message    ❌

Or the opposite:

1. SEND message    ✅
2. UPDATE database ❌

In the first case, the event is lost.

In the second case, an event has been published even though the corresponding business state does not actually exist.

Why Isn't a Normal Database Transaction Enough ?

You might say:

Why don't we simply put message publishing inside the transaction ?

The problem is that the database and Message Broker are two independent resources. To make the entire operation atomic, we would need some form of distributed transaction.

One classic approach is Two-Phase Commit, or 2PC.

In 2PC, a coordinator asks all participants:

Are you ready to commit your changes ?

Only when all participants agree, the coordinator instruct them to perform the final commit.

If one participant has a problem, the entire operation can be stopped.

This approach is theoretically attractive, but modern distributed architectures can introduce problems such as:

Complexity
Latency
Coupling
Failure handling

In addition, not every database or Message Broker supports distributed transactions in the same way.

The Outbox Pattern asks a different question:

Do we really need the database and Broker to participate in one transaction ?

The answer is:

No.

Instead, we can reliably record the critical operation in a local database transaction and defer message publishing until afterward.

How Does the Outbox Pattern Solve the Dual Write Problem ?

Instead of:

Database → Commit
Broker   → Publish

we use:

Database
├── Business Data
└── Outbox Event

For example:

BEGIN;
INSERT INTO orders (id, customer_id, status) VALUES (123, 456, 'CREATED');
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload) VALUES ('event-789', 'Order', 123, 'OrderCreated', '{...}');
COMMIT;

If the transaction succeeds:

Order        ✅
Outbox Event ✅

If the transaction fails:

Order        ❌
Outbox Event ❌

The database guarantees that both changes are committed or rolled back together.

This is the key idea behind the Outbox Pattern.

What Is an Outbox Table ?

An Outbox is usually just a normal database table.

For example:

CREATE TABLE outbox (
    id              UUID PRIMARY KEY,
    aggregate_type  VARCHAR(100),
    aggregate_id    VARCHAR(100),
    event_type      VARCHAR(200),
    payload         JSONB,
    created_at      TIMESTAMP,
    processed_at    TIMESTAMP NULL
);

The actual schema can vary depending on your system's requirements.

Common fields include:

1. `id`

A unique identifier for the event.

This ID is extremely important for Deduplication and Idempotency.

2. `aggregate_id`

For example:

order_id = 123

This field can be important for maintaining event ordering in many architectures.

3. `event_type`

For example:

OrderCreated
OrderPaid
OrderCancelled

4. `payload`

The actual event data:

{
  "orderId": 123,
  "customerId": 456,
  "totalAmount": 250
}

5. `created_at`

The time at which the event was created.

6. `processed_at`

With a polling-based implementation, this field can be used to identify events that have already been processed.

How Does the Complete Outbox Flow Work ?

Let's look at the complete flow:

Client
   │
   ▼
Order Service
   │
   │ BEGIN TRANSACTION
   ├────────────────────────┐
   │                        │
   ▼                        ▼
Orders Table          Outbox Table
   │                        │
   └──────────┬─────────────┘
              │
            COMMIT
              │
              ▼
       Outbox Publisher
              │
              ▼
        Message Broker
              │
       ┌──────┴──────┐
       ▼             ▼
 Inventory       Notification
 Service           Service

Step by step:

The Client sends a request to create an Order.
The Order Service starts a Transaction.
The Order is stored in the database.
The corresponding Event is stored in the Outbox.
The Transaction is committed.
An Outbox Publisher reads the event from the Outbox.
The Event is sent to the Message Broker.
Different Consumers receive the Event.
After successful publishing, the Outbox Event is cleaned up or marked as processed.

An important detail is that the application does not need to wait for the Broker for the original request to succeed.

This means event publishing can happen asynchronously.

A Real-World Example: Creating an Order

Suppose a user creates order 1001.

Step 1: Create the Order

POST /orders

The application starts a transaction.

Step 2: Store the Order

orders

id: 1001
status: CREATED

Step 3: Store the Event

Inside the same transaction:

outbox

id: event-abc
event_type: OrderCreated
aggregate_id: 1001

Step 4: Commit

If the commit succeeds:

Order  → persisted
Event  → persisted

Now, even if the application crashes immediately afterward, the event has not been lost.

Step 5: Publish the Event

The Publisher eventually sends:

Kafka → OrderCreated

Step 6: Consumers Process the Event

For example:

Inventory Service:

OrderCreated
      ↓
Reserve Stock

Notification Service:

OrderCreated
      ↓
Send Confirmation Email

Analytics Service:

OrderCreated
      ↓
Record Conversion

This is where the Outbox Pattern becomes especially useful for Event-Driven Architecture.

Does the Outbox Pattern Prevent Duplicate Messages ?

No.

This is one of the most important things to understand about the Outbox Pattern.

The Outbox Pattern can ensure that an event is not lost, but it does not necessarily guarantee that the event will be published exactly once.

Consider this sequence:

The Publisher reads the event.
The Publisher sends the event to Kafka.
Kafka successfully receives the message.
The Publisher crashes before marking the event as processed.

After the Publisher restarts:

Publisher
   ↓
"This event is still unprocessed"
   ↓
Publish Again

As a result, OrderCreated may reach the Consumer twice.

This behavior is normal and is associated with At-Least-Once Delivery.

AWS's Transactional Outbox guidance also discusses duplicate messages and the need for idempotent Consumers.

Idempotency: The Other Half of the Story

If you use the Outbox Pattern, you need to ask:

What happens if the same event reaches a Consumer twice ?

For example:

PaymentCompleted

Suppose the Consumer processes the event twice and each time performs:

$100 → Charge Customer

The customer could potentially be charged twice.

The Consumer therefore needs a way to detect duplicate events.

One common approach is to maintain a table such as:

processed_events

event_id
--------
event-123
event-456

Before processing:

Does event-123 exist ?

If it exists:

Ignore

If it doesn't:

Process
Insert event_id

Ideally, these operations should also happen inside a transaction.

This makes the Outbox Pattern and the Idempotent Consumer Pattern complementary solutions.

Does Outbox Mean Exactly-Once Delivery ?

This is where terminology matters.

You sometimes hear:

The Outbox Pattern provides Exactly-Once Delivery.

That statement is an oversimplification.

The Outbox Pattern by itself does not provide an exactly-once guarantee.

A more realistic architecture looks like this:

Producer
   ↓
Outbox
   ↓
At-Least-Once Publish
   ↓
Idempotent Consumer

This combination can provide much more reliable behavior in terms of the final business state.

However, if the Consumer calls an external API, guaranteeing exactly-once behavior across the entire chain becomes much more complicated.

The important point is:

Exactly-once is not simply a property of one component; it is an end-to-end property.

How Is an Outbox Event Published to the Broker ?

There are two common approaches:

Approach 1: Polling Publisher

With polling, a worker periodically queries the Outbox table.

For example:

SELECT * FROM outbox WHERE processed_at IS NULL ORDER BY created_at LIMIT 100;

The workflow is:

Read
 ↓
Publish
 ↓
Mark as processed

The biggest advantage of polling is simplicity.

You don't need your system to work directly with the database's Transaction Log. Almost any database that supports queries can be used to build this type of architecture.

However, polling has a latency trade-off.

For example, if you poll every five seconds, an event created immediately after a polling cycle might have to wait almost five seconds before being detected.

If you poll more frequently:

100ms
50ms
10ms

you increase the load on the database.

So you have a trade-off:

Polling Frequency
       ↕
    Latency
       ↕
Database Load

Approach 2: Change Data Capture (CDC)

With Change Data Capture (CDC), instead of repeatedly querying the Outbox table, we track changes in the database's Transaction Log.

For example:

Application
     ↓
Database
     ↓
Transaction Log
     ↓
CDC Connector
     ↓
Message Broker

Different databases may use transaction logs such as:

WAL
Binlog
Redo Log

Tools such as Debezium can capture changes to an Outbox Table from the Transaction Log and transform them into events.

Debezium also provides a dedicated Outbox Event Router for this scenario.

Advantages of CDC

Lower latency
Less continuous polling
Suitable for high throughput
Better alignment with Transaction Log changes

Disadvantages of CDC

More architectural complexity
More infrastructure and monitoring requirements
Dependency on database capabilities
Higher operational complexity

Therefore, CDC is not necessarily better than polling. It is simply more appropriate for certain scales and latency requirements.

Polling or CDC: Which Should You Choose ?

If your system is still relatively new, polling can be a straightforward approach.

If event volume is high and low latency is particularly important, CDC can become a more suitable option.

The choice ultimately depends on factors such as:

Event volume
Latency requirements
Database capabilities
Operational complexity
Infrastructure maturity

Does the Outbox Pattern Preserve Event Ordering ?

Not automatically, and not in every architecture.

Suppose we have:

OrderCreated
OrderPaid
OrderShipped

A Consumer may need to receive these events in exactly this order.

If it receives:

OrderShipped
OrderCreated
OrderPaid

the business logic could behave incorrectly.

Therefore, ordering needs to be designed intentionally.

For example:

aggregate_id = order-123

You can then route events belonging to the same Aggregate to the same Partition in the Message Broker.

With Kafka, a Partition Key can help ensure that events for a particular Aggregate are placed in the same Partition and processed in order within that Partition.

The important distinction is:

The Outbox Pattern and Message Ordering are related, but they are separate problems.

What Happens to the Outbox Table Over Time ?

Suppose your system generates:

1,000 events / second

That's approximately:

86,400,000 events / day

If you never delete or archive processed events, the Outbox table will grow rapidly.

This can lead to:

Increased database size
Larger database indexes
Heavier backups
Slower queries
Higher storage costs

Therefore, an Outbox needs a Lifecycle Strategy.

For example:

Pending
   ↓
Published
   ↓
Retention Period
   ↓
Delete / Archive

Three Common Strategies

1. Delete

Delete events after they have been successfully published.

This is simple and inexpensive, but it makes historical replay more difficult.

2. Archive

Move events to another storage system after a certain period.

This can be more useful for:

Auditing
Debugging
Historical analysis

3. Partitioning

Partition the table based on time.

For example:

outbox_2026_09_18
outbox_2026_09_19
outbox_2026_09_20

Old partitions can then be deleted or archived efficiently.

Advantages of the Outbox Pattern

1. Preventing Lost Events

The most important advantage of the Outbox Pattern is that when the Business Transaction commits, the Event is also stored.

Therefore, if the Publisher crashes immediately after the commit, the Event is not lost.

2. No Need for 2PC

The database and Broker do not need to participate in a shared Distributed Transaction.

3. Separation of Business Operations and Message Publishing

The Business Request does not need to wait for the Broker.

4. Suitable for Event-Driven Architecture

The Outbox Pattern is particularly useful for communication between Microservices.

5. Easier Retries

If the Broker is temporarily unavailable:

Outbox
   ↓
Retry
   ↓
Retry
   ↓
Success

The Event remains safely stored in the database.

Disadvantages of the Outbox Pattern

The Outbox Pattern solves one class of reliability problems, but it also introduces additional responsibilities.

1. Increased Complexity

Instead of having only a database, you may now have:

Database
+
Outbox
+
Publisher
+
Broker
+
Retry
+
Monitoring
+
Cleanup

2. Duplicate Events

The Publisher may send an Event more than once.

Therefore, Consumers should be designed to be idempotent.

3. Eventual Consistency

An Event does not necessarily reach every service immediately after the database transaction commits.

For a short period, you might have:

Order Service     → CREATED
Inventory Service → still knows nothing

This delay needs to be acceptable from a business perspective.

4. Outbox Growth

Without a proper lifecycle strategy, the Outbox can become a very large table.

5. New Failure Modes

You now need to handle problems such as:

Publisher Crash
Broker Down
Database Down
Poison Message
Retry Storm
Duplicate Event
Outbox Backlog

The Outbox Pattern therefore does not eliminate reliability problems.

Instead, it changes the types of failures your system needs to handle.

Outbox Pattern vs. 2PC

The Outbox Pattern takes a fundamentally different approach from distributed transactions.

Instead of trying to create one global transaction, it keeps the transaction within the boundaries of a single database.

Database Transaction
├── Business Data
└── Outbox Event

Message publishing happens afterward.

The goal is not to make the database and Message Broker participate in the same transaction, but to make the database operation itself reliable and then handle message delivery separately.

Outbox vs. Direct Event Publishing

The simple approach is:

UPDATE DB
   ↓
PUBLISH EVENT

Advantages

Simple
Low cost
Low latency

Disadvantages

Dual Write
Potential Lost Event
Failure Window

The Outbox approach is:

UPDATE DB
   +
INSERT OUTBOX
   ↓
COMMIT
   ↓
PUBLISH ASYNC

Advantages

Higher reliability
Easier retries
No Lost Event in common failures between Commit and Publish

Costs

Increased complexity
Eventual Consistency
Duplicate handling
Operational overhead

Therefore, if an Event is not actually critical, the Outbox Pattern is not necessarily required.

For example, if Events are used only for non-critical analytics and losing a small percentage of them is acceptable, keeping the architecture simple may be more valuable.

Is the Outbox Pattern Only for Microservices ?

No.

A common misconception is that the Outbox Pattern is exclusively a Microservices pattern.

Even in a Monolithic Architecture, you may need to perform an external operation after changing the database.

For example:

Order Created
     ↓
Call a Webhook

Or:

User Registered
     ↓
Send Email

If the Email Provider is temporarily unavailable, you may not want to fail the user's registration.

Instead, you can store:

User
+
Email Event

inside the same transaction and send the email asynchronously afterward.

So the Outbox Pattern is better understood as a reliable solution for Asynchronous Messaging rather than something limited to Microservices.

When Is the Outbox Pattern Worth Using ?

The Outbox Pattern is particularly valuable when:

The Event is important to the business.
Losing an Event is unacceptable.
The database and Broker are independent systems.
The architecture is Event-Driven.
Services communicate asynchronously.
Retry and reliability are important.
Immediate consistency between services is not required.

When Isn't the Outbox Pattern Necessary ?

If an Event is completely non-critical, adding an Outbox may introduce unnecessary complexity.

For example:

User clicked button

Suppose this Event is used only for approximate analytics.

If losing some Events is acceptable, direct publishing—or even an independent analytics system—may be a simpler solution.

The right architecture depends on the business consequences of failure.

Technical Considerations When Implementing Outbox

1. Use a Unique Event ID

Every Event should have a stable identifier:

event_id = UUID

This ID is essential for Deduplication.

2. Design Consumers to Be Idempotent

Suppose:

OrderPaid

is delivered twice.

The result should not be:

Payment = Payment × 2

The Consumer needs to recognize that the Event has already been processed.

3. Index the Outbox

If you use polling, queries such as:

WHERE processed_at IS NULL ORDER BY created_at

should be supported by appropriate indexes based on your workload and access patterns.

4. Consider Batch Processing

Instead of:

1 Event → 1 Query

you may be better off processing:

100 Events → 1 Batch

The AWS Batch Processing guidance provides useful background on batch processing.

However, very large batches can also increase:

Lock duration
Memory usage
Retry costs

So the batch size should be chosen carefully.

5. Use Retry with Backoff

If the Broker is unavailable, you should not continuously retry every few milliseconds.

For example:

1s
2s
4s
8s
16s
...

Use exponential backoff together with appropriate retry limits and policies.

6. Don't Forget Dead-Letter Handling

Sometimes an Event will consistently fail.

If you retry indefinitely:

Event
 ↓
Fail
 ↓
Retry
 ↓
Fail
 ↓
Retry
 ↓
...

you can create a Retry Storm.

A problematic Event therefore needs an explicit failure-handling strategy, such as a Dead Letter mechanism.

An Important Note About Monitoring

One of the most useful Outbox metrics is the number of Events waiting to be published.

For example:

Outbox Pending Events = 12

might be completely normal.

But:

Outbox Pending Events = 2,000,000

could indicate that the Publisher has fallen significantly behind.

Useful metrics can include:

Outbox Backlog
Publish Latency
Publish Failure Rate
Retry Count
Oldest Unprocessed Event Age
Consumer Lag
Dead Letter Count

One particularly valuable metric is:

Oldest Unprocessed Event Age

For example:

Age = 45 minutes

Even if the total number of pending Events is relatively small, an old unprocessed Event could indicate an operational problem.

The Outbox Pattern and Eventual Consistency

The Outbox Pattern forces us to accept an important reality of distributed systems:

Not everything needs to be immediately consistent.

For example:

T0:
Order Created

T0 + 20ms:
Outbox Published

T0 + 50ms:
Inventory Updated

T0 + 100ms:
Notification Sent

For a short period, the system might look like this:

Order Service     → CREATED
Inventory Service → Previous State

This is not necessarily a bug.

If your business requirements allow this delay, Eventual Consistency can be a perfectly reasonable trade-off.

In Event-Driven Architectures, this is one of the fundamental design considerations.

Duplication, ordering, and idempotency all need to be deliberately designed.

Final Thoughts

The Outbox Pattern is a solution to one of the classic problems in distributed systems: the Dual Write Problem.

Instead of having an application simultaneously modify:

Database
+
Message Broker

we store the business state change and its corresponding Event in a single local database transaction:

Database
├── Business Data
└── Outbox Event

A separate process then transfers the Event to the Message Broker.

The main benefit is straightforward:

If the original transaction commits, the corresponding Event is also persisted and can be retried later.

But the Outbox Pattern is not the end of the story.

You still need to deal with:

Duplicate messages
Idempotency
Ordering
Eventual Consistency
Retries
Outbox Growth
Monitoring
Cleanup

In other words, the Outbox Pattern does not eliminate distributed-system complexity.

It moves that complexity into areas where it can be managed more explicitly.

A Question for Discussion

If losing an Event is unacceptable for your system, but a few seconds of Eventual Consistency is tolerable, would you introduce the additional complexity of the Outbox Pattern, retries, and idempotency ?

And at what point would you decide that architectural simplicity is more valuable than higher messaging reliability ?

Circuit Breaker Pattern Explained: How to Prevent Cascading Failures in Distributed Systems

Mehrad Sadeghi — Wed, 16 Sep 2026 13:05:24 +0000

Circuit Breaker Pattern

What happens when one service fails, but your entire system keeps calling it anyway ?

Imagine your payment service is down. Normally, every purchase request follows a path like this:

User
 ↓
Order Service
 ↓
Payment Service
 ↓
Bank

Now the Payment Service is unavailable — or perhaps it’s simply extremely slow. A request that normally takes 200 ms now takes 10 seconds. If your system continues sending requests to it, what happens ?

Request queues grow.
Threads and connections remain occupied.
Timeouts increase.
Latency goes up.

And eventually, services that were perfectly healthy can start failing too.

In other words, one failing dependency can potentially bring down an entire system.

This is where the Circuit Breaker pattern comes in.

Instead of continuously calling a dependency that is already failing, the system temporarily stops sending requests to it and fails fast.

The goal isn’t to fix the broken service. The goal is to prevent its failure from spreading.

What Is the Circuit Breaker Pattern ?

The Circuit Breaker is a resilience pattern commonly used in distributed systems to protect services from repeatedly calling an unhealthy dependency.

The idea comes from electrical circuit breakers. When electrical current becomes dangerously high, a circuit breaker interrupts the circuit to prevent further damage.

Software can apply a similar idea:

Healthy dependency
↓
Requests flow
↓
Dependency starts failing
↓
Failure threshold reached
↓
Circuit opens
↓
Requests fail fast

A Circuit Breaker doesn’t repair the dependency.

Instead, it can:

Stop unnecessary requests to a failing service
Protect your own application’s resources
Reduce additional load on the unhealthy dependency
Help prevent cascading failures
Give the dependency time to recover
Allow the system to degrade gracefully when possible

A useful way to think about it is:

Circuit Breaker doesn’t eliminate failure. It prevents failure from becoming a larger system-wide failure.

Why Do Distributed Systems Need Circuit Breakers ?

In a simple monolithic application, you might call a function like:

calculatePrice()

If the function fails, handling the failure is usually relatively straightforward. But distributed systems are different.

A call to another service crosses a network boundary:

Service A
↓
Service B
↓
Service C
↓
External APIService A
↓
Service B
↓
Service C
↓
External API

Unlike a local function call, a remote call can:

Time out
Lose its connection
Become extremely slow
Return a 5xx error
Be rate-limited
Become completely unavailable

And the more important problem is that failure can propagate between services.

For example:

Payment Service
↓
starts failing
↓
Order Service waits
↓
Threads become occupied
↓
Requests start queuing
↓
Latency increases
↓
Error rate increases
↓
The entire service comes under pressure

This is a cascading failure.

The original problem may have started in one service, but the resulting resource exhaustion can spread to otherwise healthy parts of the system.

Circuit Breaker is one of the patterns that can help interrupt this chain.

Reliability, High Availability, and Circuit Breakers

Two concepts often appear in discussions about distributed systems:

Reliability: is about a system behaving dependably in the presence of failures and recovering appropriately when failures occur.

High Availability: is about keeping a system accessible to users even when parts of its infrastructure or dependencies experience problems.

A Circuit Breaker does not make an external service permanently available.

Instead, it can help isolate its failure:

Dependency Failure
↓
Circuit Breaker
↓
Failure Isolation
↓
Graceful Degradation
↓
System remains responsive

This is where the idea of failure isolation becomes important.

Instead of allowing one dependency’s failure to spread across the entire system, we try to limit its blast radius.

How Does a Circuit Breaker Work ?

A Circuit Breaker is commonly modeled as a state machine with three primary states:

Closed
Open
Half-Open

CLOSED - Everything Is Normal

When the Circuit Breaker is CLOSED, requests are allowed to reach the dependency.

Client
↓
Circuit Breaker
↓
Payment Service

The Circuit Breaker monitors the results of those requests. For example:

100 requests
95 successes
5 failures

If the failure rate exceeds the configured threshold, the circuit transitions to OPEN.

But an important detail is that a failure threshold doesn’t necessarily mean:

Five requests failed consecutively.

The threshold can be based on different measurements, such as:

Number of failures
Failure percentage
Number of timeouts
Failures within a time window
A combination of multiple metrics

For example:

50 failures in the last 100 requests

or:

Failure Rate > 30% during the last 30 seconds

The right threshold depends on the actual behavior of the dependency and the reliability objectives of your system. There is no universal magic number.

OPEN - Stop Calling the Failing Service

When the Circuit Breaker transitions to OPEN, its behavior changes completely.

Before:

Request
↓
Circuit Breaker
↓
Payment Service

After:

Request
↓
Circuit Breaker
↓
FAIL FAST

The request never reaches the Payment Service. This is the central idea behind the Circuit Breaker pattern. If we already have strong evidence that a dependency is unhealthy, why should every new request wait for another network timeout ?

Suppose the Payment Service has a 10-second timeout. Now imagine 1,000 concurrent requests all waiting for that timeout:

1,000 requests × 10 seconds

A significant amount of your application’s resources can remain occupied simply waiting for a dependency that is already failing.

With an open circuit:

Request
↓
Circuit = OPEN
↓
Immediate failure

The system can reject the request much earlier.

That can help:

Free threads
Preserve connections
Reduce resource consumption
Keep latency more predictable
Prevent additional traffic from reaching the unhealthy dependency

Sometimes failing quickly is better than failing slowly.

But the circuit should not remain open forever. If it did, the system would never discover that the dependency has recovered.

After a configured period, the Circuit Breaker can transition to HALF-OPEN.

HALF-OPEN - Has the Service Recovered ?

This is one of the most subtle parts of the pattern.

Suppose the Payment Service was unavailable, but now appears to be healthy again. Should we immediately send all traffic back to it ?

Not necessarily. The service may have recovered only partially. For example, it might have just restarted and currently be capable of handling only a small amount of traffic.

Instead of immediately allowing thousands of requests through, the Circuit Breaker can allow a limited number of test requests.
For example:

HALF-OPEN
↓
5 test requests
↓
Are they successful ?

If the requests succeed:

HALF-OPEN
↓
Successful requests
↓
CLOSED

The circuit returns to normal operation.

If the test requests fail:

HALF-OPEN
↓
Failure
↓
OPEN

The circuit opens again.

Why Does HALF-OPEN Allow Only a Few Requests ?

Because service recovery doesn’t necessarily mean full recovery. Imagine a Payment Service that was down and has just come back online. It might currently be capable of processing only:

20 requests / second

If your system suddenly sends:

10,000 requests

you could create another overload:

Recovery
↓
Traffic spike
↓
Overload
↓
Failure again

In other words, your system can accidentally overwhelm the dependency immediately after it recovers.

Half-Open provides a controlled way to test recovery:

Instead of:

0 requests
↓
10,000 requests

we can do:

0 requests
↓
A few test requests
↓
Evaluate results
↓
Gradually increase traffic

The exact number of test requests is implementation- and system-dependent. There is no universal value that works for every architecture.

A Real-World Scenario: Payment Service Failure

Let’s put everything together. Imagine an online store:

User
↓
Order Service
↓
Payment Service
↓
Bank Gateway

Under normal conditions:

Order Service
↓
Circuit Breaker
↓
Payment Service
↓
Bank

Everything is working normally.

Step 1: The Failure Starts

The Bank Gateway begins experiencing problems. The Payment Service starts timing out:

Payment Request
↓
Timeout

The Circuit Breaker records these failures.

Step 2: The Failure Threshold Is Reached

Suppose our policy is:

100 requests
40 failures

and our configured rule is:

Failure Rate > 30% → OPEN

The threshold has been exceeded. The Circuit Breaker opens.

Step 3: New Requests Fail Fast

Now a new request arrives:

User
↓
Order Service
↓
Circuit = OPEN
↓
Immediate Failure

The Order Service no longer calls the Payment Service.

This prevents the application from repeatedly waiting for a dependency that is already known to be unhealthy.

Step 4: The System Degrades Gracefully

Once the circuit is open, the system still has to decide what the user should experience. For example:

Payment is temporarily unavailable. Please try again in a few minutes.

Or the application might have another valid strategy:

Keep the order in a PENDING state
Put the operation into a queue
Switch to another payment provider
Return non-critical information from a cache

This is where fallback becomes important.

What Is a Fallback?

A fallback answers a simple question:

If the primary path fails, do we have a meaningful alternative ?

For example:

Primary Payment Provider
↓
FAILED
↓
Secondary Payment Provider

But a fallback is not always possible. For a payment operation, you cannot simply return Payment successful, when the payment was never actually processed.

The fallback must therefore be consistent with the business logic. A fallback is not a fake success. It is an alternative behavior that is safe and meaningful for that particular operation.

Circuit Breaker ≠ Retry

These two concepts are often confused. They solve different problems.

Retry says:

The request failed. Maybe trying again will succeed.

Circuit Breaker says:

This dependency has been failing repeatedly. Stop sending requests to it for now.

Retry can be useful for transient failures, such as:

Temporary network glitch
Temporary timeout
Temporary throttling

But if the dependency is genuinely down, excessive retries can make the situation worse.

How Retries Can Cause a Cascading Failure

Suppose you have 1,000 requests and every failed request is retried three times. You might end up with:

1,000 original requests + 3,000 retries = 4,000 requests

That’s exactly what you don’t want when the dependency is already overloaded.

Now imagine:

A → B → C

Service C fails. Service B retries its requests to C. Service A retries its requests to B.

The result can look like this:

C fails
↓
B retries
↓
A retries
↓
More traffic
↓
C becomes even more overloaded
↓
More failures
↓
Cascading failure

This can lead to a retry storm.

So should we eliminate retries ?

No. The problem isn’t retrying itself. The problem is uncontrolled retries. A more resilient design might combine:

Request
↓
Timeout
↓
Retry
↓
Exponential Backoff + Jitter
↓
Circuit Breaker

The important point is that retries should generally be limited to failures where another attempt has a reasonable chance of succeeding.

For example, retrying a transient 503 Service Unavailable may make sense in some systems.

Retrying a 401 Unauthorized generally does not solve the underlying problem.

Exponential Backoff and Jitter

Imagine 10,000 requests fail at approximately the same time. If every client retries exactly one second later:

1 second
↓
10,000 requests

You’ve created another traffic spike.

With exponential backoff, the delay between retries increases:

1s
2s
4s
8s
…

This helps spread retries over a longer period.

But there’s still a problem. If every client follows exactly the same schedule, they can still retry at roughly the same moments. That’s where jitter helps.
Instead of:

1s
2s
4s
8s

different clients might retry at slightly different times:

1.2s
1.8s
2.4s
3.1s
…

The goal is to spread retry traffic instead of allowing thousands of clients to synchronize their retries.

Idempotency: The Problem You Must Consider When Retrying

Retries introduce another important problem: duplicate side effects.
Imagine:

POST /payments

The request reaches the Payment Service. The payment is successfully processed. But the response is lost because of a network failure. What does the client see?

Timeout

From the client’s perspective, it doesn’t know whether the payment succeeded. So it retries:

POST /payments

Now you potentially have:

Request #1 → Payment successful
↓
Response lost
↓
Retry
↓
Request #2 → Payment successful again

You may have charged the customer twice. This is where idempotency becomes critical. For example, the client could send an idempotency key:

Idempotency-Key: ABC123

The Payment Service can use that key to recognize that the same logical operation has already been processed.

Conceptually:

Retry
↓
Same Idempotency Key
↓
Same Logical Operation
↓
No Duplicate Side Effect

Therefore, whenever you introduce retries, ask:

Is this operation actually safe to retry ?

Important Circuit Breaker Configuration Parameters

Implementing a Circuit Breaker isn’t simply a matter of adding an if statement. Several parameters need to be defined.

1. Failure Threshold

How many failures should cause the circuit to open ?
For example:

Failure Rate > 50%

A threshold that is too low may cause the circuit to open because of temporary failures.

A threshold that is too high may allow the dependency to cause significant damage before the circuit reacts.

2. Recovery Timeout

Once the circuit is open, how long should we wait before testing the dependency again ?

For example:

30 seconds

A timeout that is too short can produce this cycle:

OPEN
↓
HALF-OPEN
↓
Failure
↓
OPEN
↓
HALF-OPEN
↓
…

On the other hand, a timeout that is too long may prevent requests from reaching a dependency even after it has recovered.

3. Number of HALF-OPEN Requests

When the circuit enters HALF-OPEN, how many requests should be allowed through ?

For example:

1 request

or:

5 requests

or a limited percentage of traffic.

Allowing more requests can provide more information about recovery, but it also creates more load on the recovering dependency.

4. Which Failures Should Open the Circuit ?

Not every error should necessarily be treated as evidence that a dependency is unhealthy.

For example:

Timeout → likely relevant
503 → likely relevant
Connection error → likely relevant
401 → probably not
400 → probably not

If every 4xx error contributes to the Circuit Breaker threshold, you could incorrectly conclude that the entire dependency is unavailable.

Circuit Breaker Has a Cost Too

No resilience pattern is free. Circuit Breaker provides important benefits, but it also introduces complexity.

Potential benefits

Helps prevent cascading failures
Reduces load on an unhealthy dependency
Enables fail-fast behavior
Protects application resources
Supports graceful degradation
Improves system resilience
Enables controlled recovery

Potential costs

More application complexity
Additional monitoring requirements
Threshold tuning
State management
Fallback design
More difficult debugging
False positives
False negatives

For example, if your threshold is too sensitive:

Temporary failure
↓
Circuit opens
↓
Requests rejected

even though the dependency wasn’t actually down. If the threshold is too permissive:

Dependency is failing
↓
Circuit remains CLOSED
↓
More failures
↓
More resource consumption

So configuring a Circuit Breaker is a trade-off, not a search for one perfect number.

Where Should the Circuit Breaker Live ?

A common placement is close to the remote call:

Order Service
↓
Circuit Breaker
↓
Payment Service

This keeps the decision close to the dependency being protected.

However, Circuit Breaking can also be implemented at other layers, depending on the architecture:

Application layer
API Gateway
Service Mesh
Sidecar

A Service Mesh for example, can provide resilience features without requiring every application to implement the same logic. But that introduces another trade-off. Moving more resilience behavior into infrastructure can simplify application code, while potentially making debugging and understanding system behavior more complicated.

Do You Always Need a Circuit Breaker ?

No.

Circuit Breaker is not automatically appropriate for every operation.

For example:

function calculateTax()

If this is a local, fast operation with no meaningful remote dependency, adding a Circuit Breaker may simply introduce unnecessary complexity.

Circuit Breakers become more interesting when dealing with things such as:

Remote APIs
External providers
Database dependencies
Microservices
Third-party services

The key question isn’t:

Do we use microservices ?

The better question is:

What happens to our system when this dependency becomes slow, unavailable, or unreliable ?

Circuit Breaker Alone Is Not Resilience

One common misconception is Circuit Breaker = Resilience. It isn’t.

Circuit Breaker is only one tool in a broader resilience strategy.

A more complete design might look like:

     ┌─────────────┐
     │   Timeout   │
     └──────┬──────┘
            ↓
     ┌─────────────┐
     │    Retry    │
     └──────┬──────┘
            ↓
┌───────────────────────┐
│ Exponential Backoff   │
│       + Jitter        │
└───────────┬───────────┘
            ↓
    ┌─────────────┐
    │   Circuit   │
    │   Breaker   │
    └──────┬──────┘
           ↓
    ┌─────────────┐
    │  Fallback   │
    └─────────────┘

Depending on the system, other patterns and mechanisms may also be relevant:

Resilience usually comes from combining the right mechanisms, not from adding one pattern everywhere.

How Do You Know Your Circuit Breaker Is Working ?

A Circuit Breaker without observability is difficult to operate. At a minimum, you should be able to monitor metrics such as:

Circuit state
Failure rate
Success rate
Timeout rate
Number of circuit openings
Half-Open attempts
Recovery time
Fallback rate
Request latency

State transitions can also be important monitoring events.
For example:

Circuit: CLOSED → OPEN

This may indicate a significant problem with an external dependency.

Without monitoring, you may know that requests are failing — but not know why the Circuit Breaker opened, how often it opens, or whether it is actually helping.

The Goal Isn’t to Eliminate Errors

This is one of the most important ideas behind the pattern. A Circuit Breaker does not necessarily reduce the number of errors users see. In fact, after a circuit opens, you may see more immediate failures. But those failures can be much more controlled.

Without a Circuit Breaker:

1,000 requests
↓
1,000 timeouts
↓
10 seconds each
↓
Thread exhaustion
↓
Entire service becomes unhealthy

With a Circuit Breaker:

1,000 requests
↓
Circuit OPEN
↓
1,000 fast failures
↓
Dependency protected
↓
Main service remains responsive

In both cases, there are failures. But the second scenario prevents those failures from consuming resources indefinitely. That’s the key distinction.

The goal of a Circuit Breaker isn’t to make failure disappear. It’s to turn an uncontrolled failure into a controlled one.

And that is one of the fundamental ideas behind resilient distributed systems.

The State Machine in One Picture

Let’s summarize the state transitions:

CLOSED: Requests are allowed through.

CLOSED
↓
Requests flow normally

If failures exceed the configured threshold:

CLOSED → OPEN

OPEN: Requests fail fast without calling the dependency.

OPEN
↓
Fail Fast

After the configured recovery timeout:

OPEN → HALF-OPEN

HALF-OPEN: A limited number of test requests are allowed.

If recovery succeeds:

HALF-OPEN → CLOSED

If the dependency fails again:

HALF-OPEN → OPEN

So the complete lifecycle is:

Image by geeksforgeeks

The Bigger Lesson

It’s useful to remember the three states, But understanding why they exist is much more important.

When a dependency is failing, continuing to send more requests isn’t necessarily resilience. Sometimes the most resilient thing your system can do is stop making the problem worse.

Circuit Breaker, together with carefully designed timeouts, limited retries, exponential backoff, jitter, fallback strategies, and idempotency, can help prevent a localized dependency failure from turning into a much larger outage.

The central idea is simple:

When a dependency is unhealthy, protect your system first — and give the dependency a controlled chance to recover.

Final Takeaway

The Circuit Breaker pattern is a resilience mechanism for systems that depend on remote services or external resources where failures, timeouts, and temporary unavailability are possible.

Its three primary states are:

CLOSED
↓
Requests flow normally

OPEN
↓
Requests fail fast

HALF-OPEN
↓
A limited number of requests test recovery

The most important lesson isn’t memorizing these states. It’s understanding the philosophy behind them:

A resilient system doesn’t just know how to handle failure. It knows when to stop making a failure worse.

A Final Thought Experiment

Imagine your Payment Service is completely down. Your system receives 10,000 payment requests per minute. Which behavior makes more sense ?

Option 1: Retry every request three times.
Option 2: Open the Circuit Breaker after the failure rate crosses the configured threshold and fail fast.
Option 3: Use a combination of:

The interesting part isn’t memorizing which pattern to use. The real engineering challenge is understanding how these mechanisms interact under failure. That’s where resilient distributed-system design begins.

MySQL Overselling: Why SELECT Isn't Enough and How SELECT ... FOR UPDATE Solves It

Mehrad Sadeghi — Sun, 13 Sep 2026 14:51:03 +0000

Imagine you’re running an online store, there’s exactly one item left in stock. Two customers click Buy at almost exactly the same time. Both requests reach your application. Both transactions ask the database:

SELECT stock FROM products WHERE id = 10;

And both get:

stock = 1

Now both customers believe the product is available.

So what happens next ? Welcome to the overselling problem.

The Naive Implementation

A simple purchase flow might look like this:

START TRANSACTION;
SELECT stock FROM products WHERE id = 10;
-- Application checks:
-- Is stock > 0 ?
UPDATE products SET stock = stock - 1 WHERE id = 10;
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

At first glance, this seems perfectly reasonable. But there’s a race condition.

Imagine two transactions:

Transaction A              Transaction B

SELECT stock               SELECT stock
     |                           |
     v                           v
   stock=1                    stock=1
     |                           |
     v                           v
"Available!"                 "Available!"

Both transactions read the same value before either one has completed the purchase. The problem isn’t necessarily that MySQL is broken. The problem is that our read and decision are not protected as one atomic operation.

Why Doesn’t REPEATABLE READ Solve This ?

This is an important question. InnoDB’s default isolation level is REPEATABLE READ. So you might think:

“If I’m using REPEATABLE READ, shouldn’t MySQL prevent this ?”

Not necessarily.

A normal:

SELECT ...

is a consistent, nonlocking read.

Under REPEATABLE READ, it can read from the transaction's consistent snapshot.

Isolation determines what your transaction sees. It doesn’t automatically mean:

“Nobody else can modify the row after I read it.”

That’s a completely different requirement.

If your business operation is:

“Read this row, verify a condition, and then modify it.”

you often need a locking read.

Enter SELECT … FOR UPDATE

MySQL provides:

SELECT ... FOR UPDATE

For example:

START TRANSACTION;
SELECT stock FROM products WHERE id = 10 FOR UPDATE;

This is not just a normal read. It is a locking read.

MySQL/InnoDB locks the records returned by the query, and another transaction attempting to acquire a conflicting lock on the same records has to wait until the first transaction commits or rolls back.

Now our two transactions look very different.

Transaction A Gets There First

Suppose Transaction A executes:

START TRANSACTION;
SELECT stock FROM products WHERE id = 10 FOR UPDATE;

The database finds:

stock = 1

and locks the relevant record.

Transaction B now tries:

SELECT stock FROM products WHERE id = 10 FOR UPDATE;

But Transaction A is already holding the conflicting lock.

So Transaction B waits.

Transaction A                 Transaction B
FOR UPDATE
     |
     v
 stock = 1
     |
  LOCK ROW
     |
     |                       FOR UPDATE
     |                            |
     |                            v
     |                         WAIT...
     |
 UPDATE stock = 0
     |
 INSERT ORDER
     |
   COMMIT
     |
  UNLOCK
                                |
                                v
                          FOR UPDATE succeeds

Now Transaction B gets its turn.

The important part is that Transaction B doesn’t get to make its decision before Transaction A finishes.

The Correct Purchase Flow

A safer implementation looks like this:

START TRANSACTION;
SELECT stock FROM products WHERE id = 10 FOR UPDATE;

Then the application checks the returned value:

if stock > 0:
    continue purchase
else:
    reject purchase

If stock is available:

UPDATE products SET stock = stock - 1 WHERE id = 10;
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

If the product is already sold out:

ROLLBACK;

The important property is that the check and subsequent modification happen while the relevant row is locked.

The Key Idea: Lock Before You Decide

This is the mental model worth remembering:

Unsafe:

READ
  ↓
CHECK
  ↓
UPDATE

The problem is that another transaction can interfere between the read and the update.

Safer:

LOCK + READ
     ↓
   CHECK
     ↓
   UPDATE
     ↓
   COMMIT

The lock protects the critical section.

But There’s Another Option

For a simple inventory decrement, you don’t always need to read the row first. You can make the condition part of the update itself:

UPDATE products SET stock = stock - 1 WHERE id = 10 AND stock > 0;

Then check how many rows were affected.

If:

affected_rows = 1

the purchase can proceed.

If:

affected_rows = 0

there wasn’t enough stock.

This approach can be extremely useful because the business condition is enforced directly by the database operation.

For example:

START TRANSACTION;
UPDATE products SET stock = stock - 1 WHERE id = 10 AND stock > 0;
-- If one row was updated:
-- create the order
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

In a real implementation, the application should only insert the order when the update actually succeeded.

So When Should You Use SELECT … FOR UPDATE ?

SELECT ... FOR UPDATE is particularly useful when your business logic needs to:

Read the current state of a row.
Make a decision based on that state.
Modify that same data.
Keep another transaction from changing it between those steps.

Common examples include:

inventory reservation
seat reservation
wallet balance updates
account transfers
job claiming
resource allocation
order processing

For example:

SELECT balance FROM accounts WHERE id = 123 FOR UPDATE;

Then:

Check balance
     ↓
Calculate new balance
     ↓
UPDATE account
     ↓
COMMIT

The lock protects the critical decision.

A Common Misunderstanding

One common misconception is:

“FOR UPDATE locks the entire table."

That’s not generally how InnoDB works.

InnoDB uses row-level locking, although the exact locks acquired depend on the query, indexes, search conditions, and isolation level. For some range queries, InnoDB can also use gap locks or next-key locks.

For example, a unique-index lookup such as:

SELECT stock FROM products WHERE id = 10 FOR UPDATE;

can lock the matching index record.

Range-based locking is more complicated.

For example:

SELECT * FROM products WHERE price BETWEEN 100 AND 200 FOR UPDATE;

may involve range-related locking behavior depending on the indexes and isolation level.

This is one reason understanding indexes is important when reasoning about MySQL concurrency.

Don’t Forget the Transaction

FOR UPDATE makes sense inside a transaction.

For example:

START TRANSACTION;
SELECT stock FROM products WHERE id = 10 FOR UPDATE;
UPDATE products SET stock = stock - 1 WHERE id = 10;
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

The lock is part of the transaction’s concurrency control. You don’t want to acquire a lock, perform a tiny operation, release it, and then perform the critical business operation later. The entire critical section should be designed intentionally.

What About Deadlocks ?

Locks solve one class of concurrency problems, but they introduce another possibility: deadlocks.

Imagine:

Transaction A                 Transaction B
locks Row 1                   locks Row 2
     |                              |
     v                              v
tries Row 2                    tries Row 1
     |                              |
     +---------- WAIT <-------------+

Now both transactions are waiting for each other.

InnoDB detects deadlocks and rolls back one of the transactions so that the other can continue.

This means production applications using transactions and locks should generally be prepared to retry transactions when appropriate.

Locking isn’t something you simply add without considering transaction boundaries, lock ordering, indexes, and failure handling.

The Real Lesson

The important lesson isn’t simply:

“Use SELECT ... FOR UPDATE."

The deeper lesson is:

Concurrency bugs happen when a business decision depends on data that can change between the read and the write.

You need to identify that critical section and choose an appropriate concurrency-control strategy.

Sometimes that’s:

SELECT ... FOR UPDATE

Sometimes it’s an atomic conditional update:

UPDATE ... WHERE stock > 0;

Sometimes it’s an optimistic concurrency strategy. And sometimes a database constraint is the best solution. The correct choice depends on the business operation.

Final Takeaway

When building a system that handles concurrent requests, don’t ask only:

“Which isolation level should I use ?”

Also ask:

“What happens if two transactions execute these exact statements at the same time ?”

That’s the question that exposes race conditions.

For an inventory operation, this:

SELECT stock FROM products WHERE id = 10;

and this:

SELECT stock FROM products WHERE id = 10 FOR UPDATE;

are not equivalent.

The first reads. The second reads with an intention to modify and acquires a lock.

Once you understand that distinction, MySQL transaction isolation becomes much easier to reason about.

And more importantly, you can start designing systems that remain correct even when thousands of users click Buy at exactly the same time.

Two Transactions, One Row: What Does MySQL Actually Let You See ?

Mehrad Sadeghi — Sun, 13 Sep 2026 14:47:01 +0000

Two users, One database row, Both reading and writing at almost exactly the same time. Who sees what ?

Can one transaction see another transaction’s uncommitted changes ?
Can the same query return different results inside a single transaction ?
And why does SELECT sometimes give you a snapshot, while SELECT ... FOR UPDATE
gives you something completely different ?

If you’ve ever wondered what MySQL actually does when transactions run concurrently, this is where it gets interesting.

InnoDB gives you four transaction isolation levels:

READ UNCOMMITTED
READ COMMITTED
REPEATABLE READ
SERIALIZABLE

Most developers know their names, fewer can predict what will actually happen when two transactions execute at the same time.

That’s what we’re going to do here.

No abstract definitions first. No memorizing a table. We’ll make the transactions collide and see what MySQL does.

Why Do We Need Transactions ?

Consider a typical e-commerce purchase. When a customer buys a product, several database operations may need to happen together:

Check the product inventory.
Decrease the inventory.
Create the order.
Record the payment.
Update the order status.

What happens if the inventory is successfully decreased but creating the order fails ?

You could end up with a database where the product’s inventory has decreased, but no corresponding order exists.

That’s exactly the kind of problem transactions are designed to prevent.

A transaction lets us treat multiple database operations as a single logical unit:

Either all of the operations succeed, or the changes are rolled back.

For example:

START TRANSACTION;
UPDATE products SET stock = stock - 1 WHERE id = 10;
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

If something goes wrong:

ROLLBACK;

Transactions are one of the foundations of ACID, and the I in ACID stands for Isolation. Isolation is where things get interesting.

What Is a Transaction Isolation Level ?

Imagine two transactions running at approximately the same time:

Transaction A                Transaction B
      |                            |
      |---- read data ------------>|
      |                            |
      |                       update data
      |                            |
      |<--- what can A see ? ------|

Both transactions may be reading and modifying the same data. The isolation level defines the rules for what each transaction is allowed to see while other transactions are running.

Those rules can dramatically change the behavior of your application.

For example:

Can you see another transaction’s uncommitted changes ?
Can the same query return a different value later ?
Do you keep seeing the same snapshot ?
Does your read lock the row ?
Can another transactions modify the row while you’re making a decision ?

Let’s find out.

1. READ UNCOMMITTED - When Uncommitted Data Becomes Visible

READ UNCOMMITTED is the least restrictive isolation level. A transaction can potentially see changes made by another transaction before those changes have been committed. This is known as a dirty read.

Suppose the database contains:

stock = 10

Transaction A starts:

START TRANSACTION;
UPDATE products SET stock = 0 WHERE id = 10;

But Transaction A hasn’t committed yet.

Now Transaction B executes:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

Transaction B may see:

Wait. That 0 hasn't actually been committed.

Now Transaction A rolls everything back:

ROLLBACK;

The actual value goes back to:

Transaction B just observed a value that never became part of the committed database state. That’s a dirty read.

Why is this dangerous ?

Imagine applying the same idea to:

payments
bank balances
inventory
orders
seat reservations

You could make a business decision based on data that ultimately disappears. A simple way to remember it:

READ UNCOMMITTED: you get more concurrency by giving up consistency.

2. READ COMMITTED - Only See What Has Been Committed

READ COMMITTED takes a more conservative approach. A transaction doesn’t see another transaction’s uncommitted changes through a normal consistent read. But there’s a catch.

The data you see can change while your transaction is still running.

Suppose:

Initial stock = 10

Transaction A starts:

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

It sees:

Now Transaction B changes the value:

START TRANSACTION;
UPDATE products SET stock = 5 WHERE id = 10;
COMMIT;

Transaction A executes the same query again:

SELECT stock FROM products WHERE id = 10;

This time:

The same transaction saw 10 and later 5. That’s a Non-Repeatable Read.

But why this happens ? Because under READ COMMITTED, each consistent read can establish its own fresh snapshot.

When is READ COMMITTED useful ?

It’s a good fit for systems where seeing relatively fresh committed data is more important than maintaining one consistent snapshot throughout the entire transaction.

It also changes InnoDB’s locking behavior. For locking READ, UPDATE, and DELETE, InnoDB generally uses record locks rather than gap locks, except where gap locking is needed for foreign-key and duplicate-key checks.

A simple way to remember it:

READ COMMITTED: every consistent read sees committed data as of that read, so a later read can see changes committed by other transactions.

3. REPEATABLE READ - Your Transaction Gets a Snapshot

Now we reach InnoDB’s default isolation level: REPEATABLE READ.

Suppose:

stock = 10

Transaction A starts:

SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

It sees:

Now Transaction B changes the value:

START TRANSACTION;
UPDATE products SET stock = 5 WHERE id = 10;
COMMIT;

Transaction A runs the same query again:

SELECT stock FROM products WHERE id = 10;

And for a normal, non-locking consistent read, it can still see 10, not 5.

Why is that ? Because consistent reads in a REPEATABLE READ transaction use the transaction's snapshot.

So from Transaction A’s perspective, the database can effectively look like:

Transaction A’s view:

stock = 10

even though the current committed value is now:

stock = 5

4. SERIALIZABLE - Make Concurrency Behave More Like Sequential Execution

SERIALIZABLE is the strictest of the four isolation levels. Its goal is to provide behavior that is closer to transactions executing one after another rather than freely interleaving.

Conceptually:

Transaction A
     |
     | read/write
     |
     v
Transaction B
     |
     | waits
     v
Transaction A commits
     |
     v
Transaction B continues

In InnoDB, when auto-commit is disabled, plain SELECT statements under SERIALIZABLE are implicitly converted to locking reads using FOR SHARE.

This provides stronger consistency guarantees, but the additional locking can reduce concurrency.

So while SERIALIZABLE sounds like the safest choice, it isn't automatically the best choice. Stronger isolation comes with a cost.

A simple way to remember it:

SERIALIZABLE: strongest isolation, but concurrency becomes more expensive.

Comparing the Four Levels

Here’s the quick mental model:

But don’t treat this table as the whole story.

InnoDB’s behavior also involves:

The interesting part isn’t memorizing the table. It’s being able to predict what happens when two transactions collide.

The Bigger Lesson

Here’s the part that matters most in real applications:

Choosing an isolation level doesn’t automatically make your concurrent code correct.

Isolation level determines the visibility and concurrency rules of transactions, But your SQL statements determine how you interact with the data.

Compare:

SELECT stock FROM products WHERE id = 10;

with:

SELECT stock FROM products WHERE id = 10 FOR UPDATE;

They may look almost identical, but they are not.

The first is a consistent read.

The second is a locking read.

And when two requests hit your application at almost exactly the same time, that difference can determine whether your system behaves correctly.

The Question You Should Ask

When debugging or designing concurrent database code, don’t ask only:

“Which isolation level am I using ?”

Ask this instead:

“What happens if two transactions execute these exact statements at the same time ?”

That’s the question that exposes race conditions.

Once you start thinking in terms of concurrent transactions instead of isolated SQL statements, MySQL’s transaction model becomes much easier to understand.

Final Takeaway

Transaction isolation isn’t just a list of four configuration values. It’s a set of rules governing what your transactions can see, when they can see it, and how they interact with other transactions.

If you remember only one thing from this article, remember this:

The real test of your database design isn’t what happens when one transaction runs. It’s what happens when two transactions run at the same time.

And when those two transactions are fighting over the last item in stock, the difference between a normal SELECT and SELECT ... FOR UPDATE suddenly becomes very important.

That’s where we’re going next.

Next: MySQL Overselling - Why SELECT Isn't Enough and How SELECT ... FOR UPDATE Solves It

Clean Code Made Simple - Part 3

Mehrad Sadeghi — Mon, 15 Aug 2022 17:28:54 +0000

Further to my last posts Part 1 and Part 2 of Clean Code Made Simple,
in this post I'm going to continue introducing techniques from Robert C. Martin’s Clean Code book.

#11 Law of Demeter

As wikipedia says:

The Law of Demeter (LoD) or principle of least knowledge is a design guideline for developing software, particularly object-oriented programs. can be succinctly summarized in each of the following ways:

Each unit should have only limited knowledge about other units: only units "closely" related to the current unit.

Each unit should only talk to its friends; don't talk to strangers.

Only talk to your immediate friends.

Now that we have a basic understanding of Demeter's law, in order to prevent breaking it, which one of the following structures do you think is better?

$ctx->getAbsolutePathOfScratchDirectoryOption();

$ctx->getScratchDirectoryOption().getAbsolutePath()

well non is good. If ctx is an object, we should be telling it to do something; we should not be asking it about its internals.

Consider far from code about (in that module) we needed the outputDir like below:

$outFile = $outputDir + "/" + $className->replace('.', '/') + ".class";
$fout = new FileOutputStream($outFile);
$bos = new BufferedOutputStream($fout);

So, what if we told the ctx object to do this:

$bos = $ctx->createScratchFileStream($classFileName);

That seems like a reasonable thing for an object to do! This allows ctx to hide its internals and prevents the current function from having to violate the Law of Demeter by navigating through objects it shouldn’t know about.

#12 Standard Pattern Names

By using the standard pattern names, such as COMMAND or VISITOR, in the names of the classes that implement those patterns, you can succinctly describe your design to other developers.

#13 Doing One Thing

How do you think we can figure out a function is doing more than “one thing”?

The answer is that if you can extract another function from it with a name that is not merely a restatement of its implementation, Then you would know it's doing more than one thing.

To do so, we need to make sure that the statements within our function are all at the same level of abstraction. for example
getHtml() is at a very high level of abstraction,
$pagePathName = PathParser::render($pagePath); is at intermediate abstraction,
and something like ->append("\n") is at low level of abstraction

#14 Base Class and Derivatives

The most common reason for partitioning concepts into base and derivative classes is so that the higher level base class concepts can be independent of the lower level derivative class concepts. Therefore, when we see base classes mentioning the names of their derivatives, we suspect a problem. In general, base classes should know nothing about their derivatives.

There are exceptions to this rule, of course. Sometimes the number of derivatives is strictly fixed, and the base class has code that selects between the derivatives. We see this a lot in finite state machine implementations. However, in that case the derivatives and base class are strongly coupled and always deploy together. In the general case we want to be able to deploy derivatives and bases separately.

#15 Naming

Methods should have verb or verb phrase names like postPayment, deletePage, or save.
Accessors, mutators, and predicates should be named for their value and prefixed with get,
set, and is.

$name = employee->getName();
$customer->setName("mike");
if ($paycheck->isPosted())...

Okay. This is it for this part. You can join my Telegram channel to get notified of the latest posts. Also, you can follow me on Twitter and LinkedIn.

Clean Code Made Simple - Part 2

Mehrad Sadeghi — Thu, 11 Aug 2022 16:49:00 +0000

Further to my last post Clean Code Made Simple - Part 1, in this post I'm going to continue introducing techniques from Robert C. Martin’s Clean Code book.

#6 Hiding Data

Which one of below code snippets do you think is better designed?

interface Vehicle {
    public function getFuelTankCapacityInGallons();
    public function getGallonsOfGasoline();
}

interface Vehicle {
    public function getPercentFuelRemaining();
}

In both of the above cases the second option is preferable. We do not want to expose the details of our data as much as possible. Rather we want to express our data in abstract terms. Serious thought needs to be put into the best way to represent the data that an object contains. The worst option is to blithely add getters and setters.

So in any situation where it's suitable to hide the data of an object, You may do it.

#7 Simple Design Rules

Kent Beck introduced these 4 rules as basic criteria to design programs better:

Runs all the tests
Contains no duplication
Expresses the intent of the programmer
Minimizes the number of classes and methods

In an effort to make our classes and methods small, we might create too many tiny classes and methods. we should also keep our function and class counts low. this is the forth rule of simple design rules by kent which is more important than other ones.

#8 Intention Revealing Functions

how do you think this code snippet can get better ?

public function getFlaggedCells() {

    $flaggedCells = new List();

    foreach ($cells as $cell) {
        if ($cell[STATUS_VALUE] == 1) {
            $flaggedCells->add($cell);
        }
    }

    return $flaggedCells;
}

We can write an intention-revealing function (call it isFlagged) to hide the magic numbers. It results in a new version of the function:

public function getFlaggedCells() {

    $flaggedCells = new List();

    foreach ($cells as $cell) {
        if ($cell->isFlagged())
            $flaggedCells->add($cell);

    return $flaggedCells;
}

#9 Writing Tests Leads to Better Designs

Systems that aren’t testable aren’t verifiable. Arguably, a system that cannot be verified should never be deployed. Fortunately, making our systems testable pushes us toward a design where our classes are small and single purpose. Writing tests leads to better designs.

#10 Separation of Concerns

What would you recommend the direction of arrows should be, to separate construction from use ?

Main             Application



Builder

Directions should be as follow:

            run
  Main  --------->  Application
   |
   | build and construct
  \|/
Builder

It simply means that the construction of our application should be separate from using it. These two steps are fundamentally separate and decoupling them gives us a better design and structure.

Okay. This is it for this part. You can join my Telegram channel to get notified of the latest posts. Also, you can follow me on Twitter and LinkedIn.

Clean Code Made Simple - Part 1

Mehrad Sadeghi — Sun, 07 Aug 2022 08:13:00 +0000

I’ve read Robert C. Martin’s Clean Code book a couple of years ago. It’s a great book, Especially for those who are in the junior years of their career. It helps you become a more mature software developer/engineer and write quality code more often.

I aggregated the tips, tricks, and practices I’ve learned from this book and will publish them in multiple parts.

Note that there are quotes and code examples that I’ve used from the original book in this series.

So without further ado, Let’s start.

#1 Short Functions Are Better

What uncle bob says about a function’s length is that the smaller a function is, The better.
He suggests that the block of code inside an if statement, else statement, and while statement should be just one line. Also, the indent level of a function should not be more than one or two.

It’s a good practice if you could do it. It’s not always possible to follow this rule strictly, But try to do it as much as possible.

#2 Introduce Instance Variables When Possible

Passing instance variables instead of primitive parameters into a function is a good idea when it’s proper. But when it’s proper?

Consider a large function with many variables declared within it. Let’s say you want to extract one small part of that function into a separate function. However, the code you want to extract uses four of the variables declared in the function.

This is a situation where passing an instance variable may be a good idea.

class Sample 
{ 
  public function __construct() 
  {

  }

  public function render() 
  {
   // doing some logic
   $this->getHtml($var1, $var2, $var3, $var4);
  }

  private function getHtml($var1, $var2, $var3, $var4) 
  {
   // doing some logic with vars
  }
}

Will become:

class Sample 
{
  private $var;

  public function __construct(VarThing $var) 
  {
   $this->var = $var;  
  }

  public function render() 
  {
   // doing some logic
   $this->getHtml();
  }

  private function getHtml() 
  {
   // $this->var is accessible here
  }
}

#3 The BUILD-OPERATE-CHECK Pattern

You’ve probably heard of Arrange-Act-Assert or AAA pattern before, Which is similar to the Build-Operate-Check pattern.

How do you think we can improve the following code?

public function testGetPageHieratchyAsXml()
{
    crawler()->addPage($root, PathParser()->parse("PageOne"));
    crawler()->addPage($root, 
       PathParser()->parse("PageOne.ChildOne"));
    crawler()->addPage($root, PathParser()->parse("PageTwo"));

    request()->setResource("root");
    request()->addInput("type", "pages");

    $responder = new SerializedPageResponder();
    $response = $responder->makeResponse(
        new FitNesseContext($root), $request
    );

    $xml = $response->getContent();

    assertEquals("text/xml", $response->getContentType());
    assertSubString("<name>PageOne</name>", $xml);
    assertSubString("<name>PageTwo</name>", $xml);
    assertSubString("<name>ChildOne</name>", $xml);
}

public function testGetPageHieratchyAsXmlDoesntContainSymbolicLinks()
{

    $pageOne = crawler()->addPage($root, 
       PathParser()->parse("PageOne"));

    crawler()->addPage($root, 
       PathParser()->parse("PageOne.ChildOne"));
    crawler()->addPage($root, PathParser()->parse("PageTwo"));

    $data = $pageOne->getData();

    $properties = $data->getProperties();
    $symLinks = $properties->set(SymbolicPage::PROPERTY_NAME);

    $symLinks.set("SymPage", "PageTwo");
    $pageOne.commit($data);
    request()->setResource("root");
    request()->addInput("type", "pages");

    $responder = new SerializedPageResponder();
    $response = $responder->makeResponse(
        new FitNesseContext($root), $request);

    $xml = $response->getContent();

    assertEquals("text/xml", $response->getContentType());
    assertSubString("<name>PageOne</name>", $xml);
    assertSubString("<name>PageTwo</name>", $xml);
    assertSubString("<name>ChildOne</name>", $xml);
    assertNotSubString("SymPage", $xml);
}

public function testGetDataAsHtml()
{
    crawler()->addPage($root, PathParser()
           ->parse("TestPageOne"), "test page");
    request()->setResource("TestPageOne");
    request()->addInput("type", "data");
    $responder = new SerializedPageResponder();
    $response = $responder->makeResponse(
        new FitNesseContext($root), $request);
    $xml = $response->getContent();
    assertEquals("text/xml", $response->getContentType());
    assertSubString("test page", $xml);
    assertSubString("<Test", $xml);
}

This is how we can improve it:

public function testGetPageHierarchyAsXml()
{
    makePages("PageOne", "PageOne.ChildOne", "PageTwo");
    submitRequest("root", "type:pages");
    assertResponseIsXML();
    assertResponseContains("<name>PageOne</name>", "<name>PageTwo</name>", "<name>ChildOne</name>");
}

public function testSymbolicLinksAreNotInXmlPageHierarchy() 
{
    $page = makePage("PageOne");
    makePages("PageOne.ChildOne", "PageTwo");
    addLinkTo($page, "PageTwo", "SymPage");
    submitRequest("root", "type:pages");
    assertResponseIsXML();
    assertResponseContains("<name>PageOne</name>", "<name>PageTwo</name>", "<name>ChildOne</name>");
    assertResponseDoesNotContain("SymPage");
}

public function testGetDataAsXml()
{
    makePageWithContent("TestPageOne", "test page");
    submitRequest("TestPageOne", "type:data");
    assertResponseIsXML();
    assertResponseContains("test page", "<Test");
}

The benefit of this pattern is that the vast majority of annoying details have been eliminated. The tests get right to the point and use only the data types and functions that they truly need. Anyone who reads these tests should be able to understand what they do very quickly, without being misled or overwhelmed by details.

#4 Single Responsibility and Error Handling

The best way to describe this is to quote directly from the book:

Functions should do one thing. Error handing is one thing. Thus, a function that handles errors should do nothing else. This implies that if the keyword try exists in a function, it should be the very first word in the function and that there should be nothing after the catch/finally.

#5 Better Error Handling

how do you think this piece of code can get better shaped?

$port = new ACMEPort(12);
try {
    $port->open();
} catch (DeviceResponseException $e) {
        reportPortError($e);
        logger()->log("Device response exception", $e);
    } catch (ATM1212UnlockedException $e) {
        reportPortError($e);
        logger()->log("Unlock exception", $e);
    } catch (GMXError $e) {
        reportPortError($e);
        logger()->log("Device response exception");
    } finally {
        /// ...
}

We can simplify our code considerably by wrapping the API that we are calling and making sure that it returns a common exception type:

$port = new LocalPort(12);

try {
    $port->open();
} catch (PortDeviceFailure $e) {
    reportError($e);
    logger()->log($e->getMessage(), $e);
} finally {
    // ...
}
class LocalPort {

    private $innerPort;

    public function __construct(int $portNumber) {
        $this->innerPort = new ACMEPort($portNumber);
    }

    public function open() {
        try {
            $this->innerPort->open();
        } catch (DeviceResponseException $e) {
            throw new PortDeviceFailure($e);
        } catch (ATM1212UnlockedException $e) {
            throw new PortDeviceFailure($e);
        } catch (GMXError $e) {
            throw new PortDeviceFailure($e);
        }
    }

    // ...
}

Wrappers like the one we defined for ACMEPort can be very useful.

Okay. This is it for this part. I’ll continue posting the next parts soon. You can join my Telegram channel to get notified of the latest posts. Also, you can follow me on Twitter and LinkedIn.

A Few Words on Webhooks

Mehrad Sadeghi — Mon, 01 Aug 2022 18:58:00 +0000

In 2006 during working on a startup, Jeff Lindsay wrote about an architectural pattern he called “webhooks”. Since then almost all web developers have heard this term.

I think webhooks are mostly based on the Observer design pattern. Observer is a pattern where an observer is observing (looking) for an event to be fired. The important part here is that the observer is not polling anything, It registers itself to the origin to get notified when the desired event happens.
Polling is another method where the destination is checking for new changes in origin constantly, As opposed to what the observer pattern does.

Webhooks are very similar to observers. A webhook is basically an endpoint that should be notified (called) when an event is dispatched. This approach is better than the “Polling” method in most cases. It is faster and uses less resource.

So by webhooks, Applications can use a so-called “hook”, To send automated messages to each other when something (an event) happens. For example, When you buy a product online, You receive an email or an SMS. What happens here is that the email or SMS provider provides an endpoint (in this context a webhook) and the store calls this endpoint when for example an order is placed.

You can learn more about webhooks in this crash course on youtube.

Laravel CRUD Generator

Mehrad Sadeghi — Tue, 22 Sep 2020 15:56:00 +0000

Laravel CRUD Generator is a simple and light-weight wrapper on Laravel's default controller generator, Which makes creating a controller with a pre-defined CRUD like a breeze.

You can define your validation rules in an interactive way that crud-generator provides. Also it is compatible with Laravel 5.x, 6.x, 7.x, 8.x and 9.x.

There is demo available here which is showing how it's done :)

Installation

$ composer require mehradsadeghi/laravel-crud-generator

Usage

It works based on your $fillable property of the target model.

If you would like to use $guarded instead of $fillable, It is supported too. In that case you'll need to have an existing Schema (table), Then the crud generator will automatically figures out your fillables.

As an example:

$ php artisan make:crud UserController --model=User

If you would like to have validaiton rules too, You may enter the --validation option:

$ php artisan make:crud UserController --model=User --validation

Customizing Default Stubs

It is possible to publish default stubs and change them in a way that fulfills your requirements.

In order to do this:

$ php artisan crud:publish

The published stubs will be located within stubs/crud directory in the root of your application. Any changes you make to these stubs will be reflected when you generate crud.

Feel free to take a look at crud-generator repository.

How To Filter Your Eloquent Queries Based on URL Query String in Laravel Like a Pro

Mehrad Sadeghi — Fri, 31 Jul 2020 17:50:00 +0000

Describing the Problem

You have probably faced the situation where you needed to filter your query based on given parameters in url query-string and after developing the logics, You've had a code-base like what we can in the above image.

This approach works, But it's not a good practice.

When the number of parameters starts to grow, The number of these kind of if statements also grows and your code gets huge and hard to maintain.

Also it's against the Open/Closed principal of SOLID principles, Because when you have a new parameter, You need to get into your existing code and add a new logic (which may breaks the existing implementations).

So we have to design a way to make our filters logics separated from each other and apply them into the final query, which is the whole idea behind the package I wrote for Laravel.

Laravel Filter Query String

First you need to install the package:

$ composer require mehradsadeghi/laravel-filter-querystring

Then you should use the FilterQueryString trait in your model, And define $filters property which can be consist of available filters or your custom filters.

use Mehradsadeghi\FilterQueryString\FilterQueryString;

class User extends Model
{
    use FilterQueryString;

    protected $filters = [];

    ...
}

Then you need to use filter() method in your eloquent query. For example:

User::select('name')->filter()->get();

Available Methods

Sort
Comparisons
In
Like
Where clause

For the purpose of explaining each method, Imagine we have such data in our users table:

id	name	email	username	age	created_at
1	mehrad	mehrad@example.com	mehrad123	20	2020-09-01
2	reza	reza@example.com	reza123	20	2020-10-01
3	hossein	hossein@example.com	hossein123	22	2020-11-01
4	dariush	dariush@example.com	dariush123	22	2020-12-01

And assume our query is something like this:

User::filter()->get();

Sort

Sort is the equivalent to order by sql statement which can be used flexible in FilterQueryString:

Conventions:

?sort=field
?sort=field,sort_type
?sort[0]=field1&sort[1]=field2
?sort[0]=field1&sort[1]=field2,sort_type
?sort[0]=field1,sort_type&sort[1]=field2,sort_type

In User.php

protected $filters = ['sort'];

Single sort:

https://example.com?sort=created_at

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01
reza	reza@example.com	reza123	20	2020-10-01
hossein	hossein@example.com	hossein123	22	2020-11-01
dariush	dariush@example.com	dariush123	22	2020-12-01

Note that when you're not defining sort_type, It'll be asc by default.

Multiple sorts:

https://example.com?sort[0]=age,desc&sort[1]=created_at,desc

Output:

name	email	username	age	created_at
dariush	dariush@example.com	dariush123	22	2020-12-01
hossein	hossein@example.com	hossein123	22	2020-11-01
reza	reza@example.com	reza123	20	2020-10-01
mehrad	mehrad@example.com	mehrad123	20	2020-09-01

Comparisons

Comparisons are consist of 6 filters:

greater
greater_or_equal
less
less_or_equal
between
not_between

Conventions:

?greater=field,value
?greater_or_equal=field,value
?less=field,value
?less_or_equal=field,value
?between=field,value1,value2
?not_between=field,value1,value2

In User.php

protected $filters = [
    'greater',
    'greater_or_equal',
    'less',
    'less_or_equal',
    'between',
    'not_between'
];

Example of greater:

https://example.com?greater=age,20

Output:

name	email	username	age	created_at
hossein	hossein@example.com	hossein123	22	2020-11-01
dariush	dariush@example.com	dariush123	22	2020-12-01

Example of not_between:

https://example.com?not_between=age,21,30

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01
reza	reza@example.com	reza123	20	2020-10-01

In

In clause is the equivalent to where in sql statement.

Convention:

?in=field,value1,value2

In User.php

protected $filters = ['in'];

Example:

https://example.com?in=name,mehrad,reza

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01
reza	reza@example.com	reza123	20	2020-10-01

Like

Like clause is the equivalent to like '%value%' sql statement.

Conventions:

?like=field,value
?like[0]=field1,value1&like[1]=field2,value2

In User.php

protected $filters = ['like'];

Single like:

https://example.com?like=name,meh

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01

Multiple likes:

https://example.com?like[0]=name,meh&like[1]=username,dar

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01
dariush	dariush@example.com	dariush123	22	2020-12-01

Where Clause (default filter)

Generally when your query string parameters are not one of previous available methods, It'll get filtered by the default filter which is the where sql statement. It's the proper filter when you need to directly filter one of your table's columns.

Conventions:

?field=value
?field1=value&field2=value
?field1[0]=value1&field1[1]=value2
?field1[0]=value1&field1[1]=value2&field2[0]=value1&field2[1]=value2

Assuming we want to filter name, username and age database columns, In User.php

protected $filters = ['name', 'username', 'age'];

Example:

https://example.com?name=mehrad

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01

Example:

https://example.com?age=22&username=dariush123

Output:

name	email	username	age	created_at
dariush	dariush@example.com	dariush123	22	2020-12-01

Example:

https://example.com?name[0]=mehrad&name[1]=dariush

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01
dariush	dariush@example.com	dariush123	22	2020-12-01

Example:

https://example.com?name[0]=mehrad&name[1]=dariush&username[0]=mehrad123&username[1]=reza1234

Output:

name	email	username	age	created_at
mehrad	mehrad@example.com	mehrad123	20	2020-09-01

Custom Filters

By custom filters you can define your own methods as filters. This helps with the Open/Closed of SOLID principles, Hence each time a new filter is needed, you don't have to edit previous filters and you can just write a separate method for it.

Let's create a custom filter. Assuming you want to create a filter named all_except which retrieves all users except the one that is specified:

In User.php

protected $filters = ['all_except'];

public function all_except($query, $value) {
    $query->where('name', '!=', $value);
}

To test our newly added filter:

https://example.com?all_except=mehrad

Output:

name	email	username	age	created_at
reza	reza@example.com	reza123	20	2020-10-01
hossein	hossein@example.com	hossein123	22	2020-11-01
dariush	dariush@example.com	dariush123	22	2020-12-01

Note that your custom defined filters have the most priority which means you can even override available filters.

For example lets change in filter in a way that only accepts 3 values:

In User.php

protected $filters = ['in'];

public function in($query, $value) {

    $exploded = explode(',', $value);

    if(count($exploded) != 4) {
        // throwing an exception or whatever you like to do
    }

    $field = array_shift($exploded);

    $query->whereIn($field, $exploded);
}

Another good example for custom filters are when you don't want to expose your database table's column name. For example assume we don't want to expose that we have a column named username in users table:

In User.php

protected $filters = ['by'];

public function by($query, $value) {
    $query->where('username', $value);
}

https://example.com?by=dariush123

Output:

name	email	username	age	created_at
dariush	dariush@example.com	dariush123	22	2020-12-01

Minor Tip

In order to prevent your model to get messy or populated with filter methods, You can create a trait for it and put everything about filters inside the trait.

You can also take a look at Laravel Filter Query String GitHub repo.

DEV Community: Mehrad Sadeghi

Automated Testing in Backend Development: A Complete Guide from Unit Tests to Fuzz Testing

What Is Automated Testing, and Why Does It Matter in Backend Development ?

Types of Tests in Backend Development

What Is a Unit Test ?

Advantages of Unit Tests

The Downside of Unit Tests

What Is an Integration Test ?

Advantages of Integration Tests

Disadvantages of Integration Tests

What Is a Feature Test (or Functional Test) ?

How Is a Feature Test Different from an Integration Test ?

What Is a Contract Test ?

The Benefit of Contract Testing

What Is an End-to-End (E2E) Test ?

The Benefit of E2E Tests

The Main Challenge with E2E Tests

What Is a Characterization Test ?

An Example of a Characterization Test

When Is a Characterization Test Valuable ?

What Is Property-Based Testing ?

When Is Property-Based Testing Useful ?

What Is Fuzz Testing ?

What's the Difference Between Fuzz Testing and Property-Based Testing ?

Where Is Fuzz Testing Useful in Backend Development ?

Trade-offs of Fuzz Testing

What Is Mutation Testing ?

Performance Testing, Load Testing, and Stress Testing

Load Testing

Stress Testing

What Is a Smoke Test ?

What Is a Regression Test ?

The Test Pyramid

Comparing Different Test Types

A Real-World Scenario: From Bug to Test

Step 1: Reproduce the Bug

Step 2: Find the Bug's Location

Step 3: Add a Unit Test

Step 4: Write a Feature Test

Step 5: Fix

Step 6: Run the Test Suite

Step 7: Prevent Regression

What Happens If We Fuzz This Same API ?

How to Write Tests That You Won't Hate Them ?

Does 100% Code Coverage Mean the System Is Fully Tested ?

What Is a Test Double ?

What Test Should You Write, and Where ?

What Does a Professional Test Suite Look Like ?

Feedback Loop Matters More Than Test Count

The Arrange/Act/Assert Pattern

Choose Tests Based on Risk, Not Habit

Wrapping Up

One Final Question

Further Reading

The Outbox Pattern Explained: A Complete Guide to Reliable Messaging in Distributed Systems

The Core Idea Behind the Outbox Pattern

What Is the Outbox Pattern ?

Why Do We Need the Outbox Pattern ?

What Is the Dual Write Problem ?

Why Isn't a Normal Database Transaction Enough ?

How Does the Outbox Pattern Solve the Dual Write Problem ?

What Is an Outbox Table ?

1. id

2. aggregate_id

3. event_type

4. payload

5. created_at

6. processed_at

How Does the Complete Outbox Flow Work ?

A Real-World Example: Creating an Order

Step 1: Create the Order

Step 2: Store the Order

Step 3: Store the Event

Step 4: Commit

Step 5: Publish the Event

Step 6: Consumers Process the Event

Does the Outbox Pattern Prevent Duplicate Messages ?

Idempotency: The Other Half of the Story

Does Outbox Mean Exactly-Once Delivery ?

How Is an Outbox Event Published to the Broker ?

1. `id`

2. `aggregate_id`

3. `event_type`

4. `payload`

5. `created_at`

6. `processed_at`