DEV Community: Sheikh Shahzaman

Why We Switched from Direct API Calls to Kafka and What Broke Along the Way

Sheikh Shahzaman — Wed, 15 Apr 2026 01:45:38 +0000

TL;DR: We migrated 10+ microservices from direct HTTP calls to Kafka event-driven communication. Reliability improved massively but the migration was harder than expected. Here are the real lessons including the mistakes.

Our system started as a monolith. Then we split it into microservices. The services talked to each other using direct HTTP calls. Service A would POST to Service B which would POST to Service C. It worked fine when we had 3 services.

Then we had 10.

The Day Everything Cascaded

One Tuesday morning our notification service crashed because of a memory leak. No big deal right? Restart it and move on.

But the order service was calling the notification service directly during checkout. When notification service was down the order endpoint started timing out. Users could not place orders. The billing service was also calling notification service to confirm payment receipts. Billing started failing too.

One crashed service took down three other services because they were all directly dependent on it.

That was the day we decided to move to event-driven architecture.

How We Set It Up

The concept is simple. Instead of Service A calling Service B directly Service A publishes an event to Kafka. Service B listens for that event and processes it on its own time.

// Before: Direct coupling
class OrderService {
    public function complete($order) {
        $order->markComplete();
        Http::post('billing-service/invoice', $order->toArray());
        Http::post('notification-service/email', $order->toArray());
        Http::post('analytics-service/track', $order->toArray());
    }
}

// After: Event-driven
class OrderService {
    public function complete($order) {
        $order->markComplete();
        KafkaProducer::publish('order.completed', [
            'order_id' => $order->id,
            'tenant_id' => $order->tenant_id,
            'total' => $order->total,
            'completed_at' => now()->toIso8601String(),
        ]);
    }
}

The order service does not know or care who listens to that event. Billing creates an invoice. Notifications send an email. Analytics tracks a metric. Each service subscribes to the event independently.

What Broke During Migration

I wish I could say the migration was smooth. It was not.

Problem 1: Event ordering. We assumed events would arrive in the order they were published. They mostly did. But when we had high throughput some consumers processed events out of order. An "order.updated" event arrived before "order.created" and the consumer crashed because the order did not exist yet.

The fix was adding an event version number and having consumers check if they had already processed a newer version before applying changes.

Problem 2: Duplicate events. Kafka guarantees at-least-once delivery. That means consumers can receive the same event twice. We had a bug where a payment was processed twice because the consumer was not idempotent.

The fix was adding a unique event ID and checking if we had already processed that ID before taking action.

class InvoiceConsumer {
    public function handle($event) {
        if (ProcessedEvent::where('event_id', $event['id'])->exists()) {
            return;
        }

        $invoice = Invoice::createFromOrder($event['order_id']);

        ProcessedEvent::create(['event_id' => $event['id']]);
    }
}

Problem 3: Debugging was harder. With direct API calls you could trace a request from start to finish in one log. With events the flow is split across multiple services and multiple time periods. Finding out why an invoice was not created required checking logs in three different services.

We solved this by adding a correlation ID to every event. When the order service publishes an event it includes a unique request ID. Every downstream consumer includes that same ID in their logs. Now you can search for one ID and see the entire flow across all services.

The Patterns That Saved Us

Dead letter queue. When a consumer fails to process an event after 3 retries it goes to a dead letter topic. We have a dashboard that shows failed events and lets us replay them after fixing the bug.

Schema registry. We define the structure of every event in a shared schema. If a producer tries to publish an event that does not match the schema it fails at publish time not at consume time. This prevented so many bugs.

Consumer lag monitoring. We track how far behind each consumer is. If the notification consumer falls 10,000 events behind we get an alert. This caught performance issues before users noticed them.

The Results

After 3 months on event-driven architecture:

Zero cascading failures. One service going down does not affect any other service.
We can deploy services independently without coordinating with other teams.
Adding a new consumer takes 30 minutes instead of modifying 5 different services.
Event replay lets us reprocess historical data when we add new features.

What I Would Do Differently

I would have implemented idempotency from day one not after the duplicate payment bug. Every consumer should be idempotent by default.

I would have invested in better tooling earlier. A good event viewer that shows the flow of events across services would have saved weeks of debugging time.

And I would not have migrated everything at once. We tried to move all 10 services in one sprint. It should have been gradual. Start with the least critical services and work toward the most critical.

Event-driven architecture is powerful but it adds complexity. If you have 3 services that rarely fail direct API calls are probably fine. If you have 10+ services and reliability matters events are worth the investment.

Have you migrated from direct API calls to event-driven architecture? What surprised you the most?

How defer() in Laravel 11 Made Our API Responses 3x Faster Without Touching a Single Queue

Sheikh Shahzaman — Sun, 29 Mar 2026 12:17:24 +0000

TL;DR: Laravel 11 introduced defer() which runs code after the HTTP response is sent to the user. No queues, no job classes, no workers. Just wrap your fire-and-forget logic in defer() and your API becomes instantly faster.

I spent two days last year trying to figure out why our order API endpoint was taking 1.2 seconds to respond. The order itself was being created in about 80ms. So where was the rest of the time going?

Turns out we were sending a confirmation email, tracking an analytics event, syncing inventory with a third-party service, and clearing a cache key. All of that happened synchronously before the response was sent back to the user.

The user did not care about any of those things completing before they saw their order confirmation. They just wanted to know their order went through.

The Old Way: Queues for Everything

The typical advice is to push these tasks onto a queue. Create a job class, dispatch it, run a queue worker, set up monitoring, handle failed jobs. For a large application with complex background processing that makes sense.

But for simple fire-and-forget tasks like sending an email or tracking an event? That is a lot of infrastructure for something that should be simple.

We had 14 different job classes in our application. Eight of them were single-method classes that just did one small thing. Each one had its own file, its own test, and its own entry in the failed jobs table. It felt like overkill.

Enter defer()

Laravel 11 added defer() and it changed how I think about background tasks. Here is how it works:

Route::post('/order', function () {
    $order = Order::create($data);

    defer(fn() => Mail::send(new OrderConfirmation($order)));
    defer(fn() => Analytics::track('order_placed', $order));
    defer(fn() => InventorySync::push($order));
    defer(fn() => Cache::forget("user:{$order->user_id}:cart"));

    return response()->json($order);
});

The response goes back to the user immediately after the order is created. Then Laravel runs all the deferred callbacks after the response is sent. The user never waits for them.

No job classes. No queue workers. No Redis or database queue driver. Just a closure that runs after the response.

When To Use defer() vs Queues

This is the part most articles get wrong. They either say "use defer for everything" or "always use queues." The reality is more nuanced.

Use defer() when the task is simple and does not need retry logic. Sending a notification email, tracking an analytics event, clearing a cache, logging an activity. If it fails you do not need to retry it automatically.

Use queues when the task is complex or needs reliability guarantees. Processing a payment, generating a large PDF, syncing thousands of records with an external API. If it fails you need to know about it and retry it.

I made the mistake of using defer() for a webhook delivery early on. The webhook target was unreliable and about 10% of deliveries failed silently. There was no retry mechanism, no failed job record, no way to know it happened. I moved that back to a queue with 3 retries and the problem was solved.

Real Numbers

After refactoring our order endpoint to use defer() for non-critical tasks:

Response time dropped from 1.2 seconds to 280ms
User-perceived performance improved dramatically
We deleted 8 single-method job classes
Queue worker load decreased because fewer jobs were being dispatched

The email still sends. The analytics event still tracks. The cache still clears. The user just does not wait for any of it.

A Pattern I Use in Every Project

I created a simple middleware that defers common request-level tasks:

class DeferCommonTasks
{
    public function handle($request, Closure $next)
    {
        $response = $next($request);

        defer(fn() => ActivityLog::record($request));
        defer(fn() => MetricsCollector::trackRequest($request, $response));

        return $response;
    }
}

Every request automatically logs activity and tracks metrics without adding any latency to the response. The middleware runs once and every endpoint benefits from it.

What I Would Do Differently

If I started over I would establish a clear rule from day one: if the task does not need retry logic and takes less than 5 seconds, use defer(). If it needs retries or takes longer, use a queue. Having that rule early would have prevented the 14 unnecessary job classes we ended up with.

I also would not use defer() inside database transactions. If the transaction rolls back the deferred callback still runs because it executes after the response. This caused a bug where we were sending order confirmation emails for orders that failed to save. I learned that the hard way.

The Bottom Line

defer() is not a replacement for queues. It is a replacement for the dozens of tiny job classes that exist only because you needed something to run "after" the response.

One line of code. No infrastructure changes. Measurably faster responses.

If you are on Laravel 11 or later and you are not using defer() yet you are probably making your users wait for things they do not need to wait for.

What is the simplest performance win you have found in your Laravel applications?

How defer() in Laravel 11 Made Our API Responses 3x Faster Without Touching a Single Queue

Sheikh Shahzaman — Sun, 29 Mar 2026 01:19:09 +0000

TL;DR: Laravel 11 introduced defer() which runs code after the HTTP response is sent to the user. No queues, no job classes, no workers. Just wrap your fire-and-forget logic in defer() and your API becomes instantly faster.

The user did not care about any of those things completing before they saw their order confirmation. They just wanted to know their order went through.

The Old Way: Queues for Everything

But for simple fire-and-forget tasks like sending an email or tracking an event? That is a lot of infrastructure for something that should be simple.

Enter defer()

Laravel 11 added defer() and it changed how I think about background tasks. Here is how it works:

Route::post('/order', function () {
    $order = Order::create($data);

    defer(fn() => Mail::send(new OrderConfirmation($order)));
    defer(fn() => Analytics::track('order_placed', $order));
    defer(fn() => InventorySync::push($order));
    defer(fn() => Cache::forget("user:{$order->user_id}:cart"));

    return response()->json($order);
});

The response goes back to the user immediately after the order is created. Then Laravel runs all the deferred callbacks after the response is sent. The user never waits for them.

No job classes. No queue workers. No Redis or database queue driver. Just a closure that runs after the response.

When To Use defer() vs Queues

This is the part most articles get wrong. They either say "use defer for everything" or "always use queues." The reality is more nuanced.

Real Numbers

After refactoring our order endpoint to use defer() for non-critical tasks:

Response time dropped from 1.2 seconds to 280ms
User-perceived performance improved dramatically
We deleted 8 single-method job classes
Queue worker load decreased because fewer jobs were being dispatched

The email still sends. The analytics event still tracks. The cache still clears. The user just does not wait for any of it.

A Pattern I Use in Every Project

I created a simple middleware that defers common request-level tasks:

class DeferCommonTasks
{
    public function handle($request, Closure $next)
    {
        $response = $next($request);

        defer(fn() => ActivityLog::record($request));
        defer(fn() => MetricsCollector::trackRequest($request, $response));

        return $response;
    }
}

Every request automatically logs activity and tracks metrics without adding any latency to the response. The middleware runs once and every endpoint benefits from it.

What I Would Do Differently

The Bottom Line

defer() is not a replacement for queues. It is a replacement for the dozens of tiny job classes that exist only because you needed something to run "after" the response.

One line of code. No infrastructure changes. Measurably faster responses.

If you are on Laravel 11 or later and you are not using defer() yet you are probably making your users wait for things they do not need to wait for.

What is the simplest performance win you have found in your Laravel applications?