DEV Community: Aasim Ghaffar

Stop Using Offset for Pagination: Switching to Cursor-Based Filtering for Massive Datasets

Aasim Ghaffar — Thu, 09 Jul 2026 00:54:41 +0000

When building APIs, data tables, or infinite scroll feeds, standard pagination is usually handled via SQL offsets:

SELECT * FROM activity_logs ORDER BY id DESC LIMIT 20 OFFSET 500000;

While this works beautifully on small datasets, it contains a massive, hidden performance trap. As your database grows to millions of rows, pages 1, 2, and 3 will load in milliseconds, but page 25,000 will take seconds to load, spiking your database CPU to 100%.

The Trap: Why OFFSET Destroys Database Performance
Relational database engines cannot jump directly to a specific row offset. To give you rows 500,001 to 500,020, the database engine must actually read through the first 500,000 rows from disk, sort them in memory, discard them, and then return only the 20 rows you requested.

As the offset number gets larger, the query gets slower. This is called a Linear Scan Limitation ($O(N)$ complexity).

The Solution: Cursor-Based Pagination ($O(1)$ Complexity)Instead of telling the database how many rows to skip, you tell the database exactly where to start looking based on a unique identifier (a cursor) from the previous page's payload.

The Old Offset Way:
-- Database scans 500,000 rows just to throw them away
SELECT * FROM activity_logs ORDER BY id DESC LIMIT 20 OFFSET 500000;

-- Database uses primary index to jump instantly to the exact row ID
SELECT * FROM activity_logs WHERE id < 145023 ORDER BY id DESC LIMIT 20;

Because your id field is indexed, the database utilizes a direct pointer lookup to locate the target row instantly, completely ignoring the preceding millions of rows. It doesn't matter if you are loading page 2 or page 50,000—the query execution time remains constant ($O(1)$).

Implementation in Application Code
When returning a collection payload to your frontend, you simply include the last item's ID as the next_cursor pointer:

{
"data": [
{ "id": 145025, "event": "user_login" },
{ "id": 145024, "event": "file_upload" },
{ "id": 145023, "event": "user_logout" }
],
"meta": {
"next_cursor": 145023,
"has_more": true
}
}

The frontend reads next_cursor and appends it to the query parameters (?cursor=145023) for the next page request.

Conclusion
If you are architecting large enterprise systems, application performance comes down to minimizing disk I/O operations. Swapping out standard numerical offset pagination for deterministic, index-driven cursors is one of the easiest ways to scale your read-heavy data pipelines smoothly.

How We Built an Atomic Circuit Breaker to Survive Third-Party API Outages at Scale

Aasim Ghaffar — Thu, 09 Jul 2026 00:49:50 +0000

Every modern backend relies on third-party APIs for things like payment gateways, SMS notifications, or shipping calculations. But what happens when that external API slows down or goes completely dark?

If your application keeps blindly hitting an offline API, your incoming request threads will hang waiting for a network timeout. Within minutes, your server's thread pool is completely exhausted, causing a cascading failure that brings your entire platform down just because one minor external service is having an outage.

To prevent this cascading failure, you need to implement a distributed Circuit Breaker Pattern.

The Strategy: Fail Fast Using Redis State Tracking
A Circuit Breaker acts exactly like an electrical circuit breaker in your home. It intercepts external API calls and monitors their failure rates. If the failure rate crosses a specific threshold, the circuit trips open, instantly blocking all subsequent calls to that API and returning a graceful fallback response without wasting any server resources.

Here is how to implement a clean, atomic Circuit Breaker using a Redis-backed abstraction:

namespace App\Services;

use Illuminate\Support\Facades\Redis;
use Exception;

class ResilientExternalProcessor
{
private const CIRCUIT_KEY = 'circuit:external_api:state';
private const FAILURE_COUNT_KEY = 'circuit:external_api:failures';
private const FAILURE_THRESHOLD = 5; // Trip after 5 failures
private const TIMEOUT_WINDOW = 60; // Stay open for 60 seconds

public function executeSecureCall(callable $apiCall)
{
    // 1. Check if the circuit is currently "OPEN" (Tripped)
    if (Redis::get(self::CIRCUIT_KEY) === 'OPEN') {
        // Circuit is open; fail fast immediately to save server resources
        return $this->getFallbackData();
    }

    try {
        // 2. Attempt the actual network call
        $response = $apiCall();

        // Success! Reset failure tracking
        Redis::del(self::FAILURE_COUNT_KEY);
        return $response;

    } catch (Exception $e) {
        // 3. Handle failure and track it atomitcally
        $failures = Redis::incr(self::FAILURE_COUNT_KEY);

        if ($failures >= self::FAILURE_THRESHOLD) {
            // Trip the circuit to "OPEN" for 60 seconds
            Redis::setex(self::CIRCUIT_KEY, self::TIMEOUT_WINDOW, 'OPEN');
            Log::emergency("External API failure threshold reached. Circuit tripped OPEN.");
        }

        return $this->getFallbackData();
    }
}

private function getFallbackData(): array
{
    // Return cached or localized data instead of throwing a 500 error
    return ['status' => 'fallback_mode', 'data' => []];
}

}

Why This Protects Your Infrastructure
By wrapping fragile external network boundaries in this structure, an API outage at your payment or SMS provider will never cause your app to go down. Your server detects the failures, trips the circuit in memory, and handles traffic gracefully until the third-party service recovers.

Defeating Database Lock Contention in High-Concurrency APIs

Aasim Ghaffar — Wed, 08 Jul 2026 22:22:46 +0000

When thousands of users try to read or update the same state simultaneously—such as real-time cross-platform mobile sync streams—server crashes are rarely caused by language execution speed. Instead, they are almost always caused by database lock contention.

If multiple incoming web threads hit your relational database (like MySQL or PostgreSQL) to run updates on the same row at the exact same millisecond, the database creates a lock queue. As the queue grows, connection pools exhaust, and your entire API goes down.

To fix this, you must intercept high-frequency read/write spikes using an intermediate cache-aside layer before the data ever touches your permanent storage engine.

Here is a clean implementation pattern using a lightweight repository structure:

namespace App\Repositories;

use Illuminate\Support\Facades\Cache;
use App\Models\UserSyncState;

class ConcurrencySyncRepository
{
/**
* Intercept and read from cache first to eliminate DB read pressure.
*/
public function getLatestState(int $userId): array
{
return Cache::remember("user:sync:{$userId}", 60, function () use ($userId) {
return UserSyncState::where('user_id', $userId)->firstOrFail()->toArray();
});
}

/**
 * Update volatile state in-memory instantly, then queue the DB write.
 */
public function updateStateAsynchronously(int $userId, array $newData): void
{
    // 1. Instantly update memory layer to keep client sync accurate
    Cache::put("user:sync:{$userId}", $newData, 60);

    // 2. Dispatch a low-priority background job to write to the DB safely
    // dispatch(new PersistSyncStateToDatabase($userId, $newData));
}

}

Key Architectural Advantages:
Sub-Millisecond Read Latency: By reading directly from Redis or Memcached memory layers, database read operations drop to virtually zero during high traffic.

Eliminating Write Deadlocks: Deferring the heavy database UPDATE query to an asynchronous background worker pool turns random traffic spikes into a flat, predictable line of orderly database executions.

Smooth State Synchronization: Cross-platform mobile clients can sync mutations dozens of times per second without causing application-wide lag or resource starvation.

By building a structured, cache-intercepted sync engine, you turn chaotic database resource bottlenecks into an incredibly smooth, highly parallel data pipeline.

I’m a Software Engineer & Data Scientist (MSc) with 9+ years of experience specializing in high-concurrency Laravel systems and open-source utility design.

The Right Way to Handle AI API Timeouts in Laravel

Aasim Ghaffar — Wed, 08 Jul 2026 22:20:54 +0000

When integrating third-party AI or LLM APIs into your application, running them synchronously inside your standard web controllers is an architectural trap. If the external API takes 10+ seconds to respond, your server's worker pool can exhaust its memory constraints rapidly.

To solve this, you must shift external execution tracks into decoupled, asynchronous background processes. Here is a lean, production-ready Laravel Job setup designed to safely handle external API latency:

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;

class ProcessAIPipeline implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable;

// Retry settings to survive external API timeouts safely
public $tries = 3;
public $backoff = [15, 45, 90];

protected array $data;

public function __construct(array $data)
{
    $this->data = $data;
}

public function handle(AIEngineService $ai): void
{
    // Off-thread processing protects your application core
    $response = $ai->generate($this->data['prompt']);
}

}

Key Architectural Advantages:
Immediate Client Release: Your controller handles rapid validation, dispatches the job, and instantly returns an HTTP 202 Accepted status to the client application.

Exponential Backoff Protection: If the AI provider hits a rate limit or unexpected downtime, your worker handles retries gracefully (15s, 45s, 90s) instead of throwing unhandled 500 errors.

Decoupled Extensibility: By shifting this logic off the HTTP thread, you can easily wrap this implementation into reusable, open-source vendor tools.

By building decoupled pipelines, you protect your infrastructure from connection timeouts and keep your core ecosystem incredibly resilient.

I’m a Software Engineer & Data Scientist (MSc) with 9+ years of experience specializing in high-concurrency Laravel systems and open-source utility design.