DEV Community

Cover image for Laravel chunk() Silently Skips Half Your Rows (+3 Eloquent Mistakes on Big Tables)
Muiduzzaman Lipu for Refactorly

Posted on

Laravel chunk() Silently Skips Half Your Rows (+3 Eloquent Mistakes on Big Tables)

This Laravel job was supposed to send 1,000 orders. It sent 500. There was no error and no exception, and it skipped exactly half.

Below is why, plus three more Eloquent mistakes that only show up when your tables get big. Everything was measured on Laravel 13 (PHP 8.5, SQLite).

Video version:

1. chunk() + update skips rows

Order::where('status', 'pending')
    ->chunk(100, function ($orders) {
        foreach ($orders as $order) {
            $order->update(['status' => 'sent']);
        }
    });
Enter fullscreen mode Exit fullscreen mode

It looks perfect: process pending orders 100 at a time and mark each one as sent. Here's what happened with 1,000 pending orders:

sent: 500   still pending: 500   ❌
Enter fullscreen mode Exit fullscreen mode

The query log shows why:

select * from "orders" where "status" = ? order by "orders"."id" asc limit 100 offset 0
select * from "orders" where "status" = ? order by "orders"."id" asc limit 100 offset 100
select * from "orders" where "status" = ? order by "orders"."id" asc limit 100 offset 200
...
Enter fullscreen mode Exit fullscreen mode

chunk() pages with OFFSET. After the first batch is updated, those 100 orders are no longer pending, so the result set shrinks by 100. The next query, offset 100, then jumps over 100 orders that were never processed. Every page skips a page.

The fix: chunkById()

Order::where('status', 'pending')
    ->chunkById(100, fn ($orders) =>
        $orders->each->update(['status' => 'sent']));
Enter fullscreen mode Exit fullscreen mode
select * from "orders" where "status" = ? and "id" > ? order by "id" asc limit 100
Enter fullscreen mode Exit fullscreen mode

Instead of an offset, chunkById() remembers the last id and asks for id > last_id. Updated rows can't shift the pages anymore, and all 1,000 / 1,000 get sent.

Rule: if you update the column you're filtering on, use chunkById() (or lazyById()). Laravel's docs warn about this, but it's easy to miss. It's the same OFFSET problem I covered in keyset pagination.

2. ->get()->count() loads every row just to count it

On 100,000 rows:

// ❌ 760–950 ms, +152 MB
$total = Order::where('status', 'pending')->get()->count();

// ✅ 14 ms, +2 MB
$total = Order::where('status', 'pending')->count();
Enter fullscreen mode Exit fullscreen mode

get() builds 100,000 Eloquent models in PHP, and then counts the collection. Calling count() on the query runs SELECT COUNT(*) in the database instead, which is ~60× faster and uses almost no memory. The same goes for sum(), max(), avg() and exists().

3. all() loads the whole table into memory

// ❌ peak +152 MB on 100k rows
foreach (Order::all() as $order) { /* export, report... */ }

// ✅ peak +4 MB
foreach (Order::lazy() as $order) { /* export, report... */ }
Enter fullscreen mode Exit fullscreen mode

lazy() looks like a normal loop, but behind the scenes it fetches 1,000 rows at a time (101 queries for 100k rows), so memory stays flat. On a million rows, all() will hit your memory limit and crash the job, while lazy() won't.

If you update rows inside that loop, use lazyById(), for the same reason as #1.

4. update() in a loop: 10,001 queries

// ❌ 10,001 queries (16 s in my SQLite test)
$orders = Order::where('status', 'pending')->get();
foreach ($orders as $order) {
    $order->update(['status' => 'sent']);
}

// ✅ 1 query (23 ms)
Order::where('status', 'pending')->update(['status' => 'sent']);
Enter fullscreen mode Exit fullscreen mode

When every row gets the same change, one mass update does it in a single UPDATE ... WHERE query. The exact time depends on your database and disk, but 10,001 round trips will always lose to one.

⚠️ One catch: a mass update does not fire model events or observers (updating, updated, saved...). If you rely on those, keep the per-model update, but use chunkById().

Cheat sheet

❌ Mistake ✅ Fix
chunk() while updating the filter column chunkById()
->get()->count() ->count()
all() on big tables lazy() / lazyById()
update() in a loop one mass update() (no model events!)

When in doubt, turn on DB::enableQueryLog() and look at what actually runs.

Go search your codebase for chunk( followed by an update. 👀


I post one tested coding tip every day on YouTube at Refactorly. The 30-second version of this one is here.

Top comments (1)

Collapse
 
sunnatulla_dev profile image
Sunnatulla •

Two ways chunkById() still bites, both checked on illuminate/database 13.42 with SQLite.

An ungrouped orWhere turns it into an endless loop. Laravel appends and "id" > ? to the last condition only:

where "status" = ? or "status" = ? and "id" > ? order by "id" asc limit 100
Enter fullscreen mode Exit fullscreen mode

So with ->where('status', 'pending')->orWhere('status', 'failed') and a callback that touches some other column (a notified_at flag, say), the pending rows come back on every page. I capped it at 30 callbacks and the same rows had been processed 30 times. Wrap them in a closure, ->where(fn ($q) => $q->where(...)->orWhere(...)), and it's 10 callbacks for 1,000 rows. The docs mention this one, but only in a small note.

The quieter one: an orderBy() already on the query stays in front of the id sort. ->orderBy('priority', 'desc')->chunkById(100, ...) ran order by "priority" desc, "id" asc. The first page ended on id 1000, so the next query was id > 1000, and the job finished after 100 of 1,000 rows with no error. Same silent-skip result as #1, just from the other direction.