DEV Community: php

Implementing JWT Refresh Token Rotation for Video API Clients

ahmet gedik — Tue, 14 Jul 2026 21:00:01 +0000

Last month a partner app shipped one of our access tokens into a debug log that got scraped off a public CI artifact. On most APIs that is a slow-motion disaster: the token stays valid until it expires, and if you issued long-lived tokens to keep client developers from re-authenticating every few minutes, "until it expires" can mean weeks. At TrendVidStream we serve a streaming-discovery API to clients in 8 regions, and a leaked token means someone can pull our ranked feeds, exhaust per-region rate limits, and poison the recommendation telemetry we build rankings on. Short-lived access tokens combined with refresh-token rotation is the fix — one of the few security controls that actually shrinks the attack surface without making the client developer's life miserable. This post is the exact scheme we run in production: the SQLite schema, the PHP 8.4 server, concurrency-safe Python and Go clients, and the traps nobody warns you about.

The problem a token that never dies

Every token scheme is a negotiation between two pressures that pull in opposite directions:

Short expiry is safer. A stolen token is only useful for the seconds left on its clock. But if access tokens expire in 10 minutes and that is all you issue, clients re-authenticate constantly and your login endpoint becomes the hottest path in the system.
Long expiry is convenient. One token that lasts two weeks means the client authenticates once and forgets about it. But now a leaked token is a two-week skeleton key, and you have no way to tell a legitimate long-lived session from a stolen one.

Rotation resolves the tension by splitting the job across two tokens with completely different lifetimes and storage models. The access token is short-lived and disposable; the refresh token is long-lived, single-use, and traceable. You get the convenience of long sessions and the blast radius of short ones.

What rotation actually means

People say "rotation" and mean three subtly different things, so here is the precise contract we implement:

Access token: a stateless HS256 JWT with a 10-minute expiry. We never store it server-side. Any edge node validates it with the shared secret and a kid header, so a request to our ap-south node never has to phone home to us-east.
Refresh token: a long opaque random string (not a JWT), stored only as a SHA-256 hash, single-use, valid for 14 days.
Rotation: every time a refresh token is redeemed, it is marked used and a brand-new refresh token is minted in return. The old one is dead the instant it is used.
Reuse detection: refresh tokens carry a family id shared across the whole lineage. If a token that was already used shows up again, that is either a bug or an attacker replaying a stolen token — so we revoke the entire family and force a real re-authentication.

That last point is what makes rotation worth the effort. Without reuse detection you have rotation for cosmetic reasons. With it, a leaked refresh token self-destructs the moment either the attacker or the real client uses it a second time, and both parties get locked out — which is exactly what you want, because it turns a silent compromise into a loud, detectable event.

The SQLite schema

We run SQLite with FTS5 for the discovery data and a separate auth.db for tokens. One table does the whole job. The family column is the lineage id; used_at being NULL is the definition of "still live".

CREATE TABLE refresh_tokens (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    token_hash TEXT    NOT NULL UNIQUE,   -- sha256 of the opaque token
    client_id  INTEGER NOT NULL,
    region     TEXT    NOT NULL,          -- us-east, eu-west, ap-south, ...
    family     TEXT    NOT NULL,          -- lineage id, shared across rotations
    issued_at  INTEGER NOT NULL,
    expires_at INTEGER NOT NULL,
    used_at    INTEGER,                   -- set on rotation; NULL = still live
    revoked    INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_rt_family  ON refresh_tokens(family);
CREATE INDEX idx_rt_expires ON refresh_tokens(expires_at);

We store the SHA-256 of the token, never the token itself. A read-only leak of auth.db — say, a stray copy pulled over FTP — hands an attacker a table of useless hashes rather than a table of live credentials. The UNIQUE constraint on token_hash also gives us a cheap collision guarantee for free.

Issuing and rotating on the server (PHP 8.4)

Here is the core of the server. A tiny self-contained Jwt helper signs the access token, and AuthService handles issuance, rotation, and reuse detection. It is runnable as-is against a PDO SQLite handle — no external dependencies.

<?php
declare(strict_types=1);

final class Jwt
{
    public function __construct(
        private string $secret,
        private string $kid,
    ) {}

    public function sign(array $claims): string
    {
        $b64 = static fn(array $x): string => rtrim(strtr(
            base64_encode(json_encode($x, JSON_UNESCAPED_SLASHES)),
            '+/', '-_'), '=');
        $head = $b64(['alg' => 'HS256', 'typ' => 'JWT', 'kid' => $this->kid]);
        $body = $b64($claims);
        $sig  = hash_hmac('sha256', "$head.$body", $this->secret, true);
        $tail = rtrim(strtr(base64_encode($sig), '+/', '-_'), '=');
        return "$head.$body.$tail";
    }
}

final class AuthException extends \RuntimeException {}

final class AuthService
{
    public function __construct(
        private PDO $db,
        private Jwt $jwt,
        private int $accessTtl  = 600,       // 10 minutes
        private int $refreshTtl = 1209600,   // 14 days
    ) {}

    /** Issue a fresh pair and start a new token family. */
    public function issue(int $clientId, string $region): array
    {
        $family = bin2hex(random_bytes(16));
        return $this->mint($clientId, $region, $family);
    }

    /** Exchange a refresh token for a new pair, rotating the old one out. */
    public function refresh(string $presented): array
    {
        $hash = hash('sha256', $presented);
        $stmt = $this->db->prepare(
            'SELECT id, client_id, region, family, used_at, revoked, expires_at
               FROM refresh_tokens WHERE token_hash = :h'
        );
        $stmt->execute([':h' => $hash]);
        $row = $stmt->fetch(PDO::FETCH_ASSOC);

        if ($row === false) {
            throw new AuthException('unknown_refresh_token');
        }
        if ((int) $row['revoked'] === 1) {
            throw new AuthException('family_revoked');
        }
        if ((int) $row['expires_at'] < time()) {
            throw new AuthException('expired_refresh_token');
        }
        // Reuse detection: an already-rotated token came back. Burn the family.
        if ($row['used_at'] !== null) {
            $this->revokeFamily($row['family']);
            throw new AuthException('refresh_reuse_detected');
        }

        $this->db->beginTransaction();
        try {
            $mark = $this->db->prepare(
                'UPDATE refresh_tokens SET used_at = :now
                   WHERE id = :id AND used_at IS NULL'
            );
            $mark->execute([':now' => time(), ':id' => $row['id']]);
            if ($mark->rowCount() === 0) {
                // Lost the race: a concurrent refresh already consumed this row.
                $this->db->rollBack();
                $this->revokeFamily($row['family']);
                throw new AuthException('refresh_reuse_detected');
            }
            $pair = $this->mint(
                (int) $row['client_id'], $row['region'], $row['family']
            );
            $this->db->commit();
            return $pair;
        } catch (\Throwable $e) {
            if ($this->db->inTransaction()) {
                $this->db->rollBack();
            }
            throw $e;
        }
    }

    private function mint(int $clientId, string $region, string $family): array
    {
        $refresh = bin2hex(random_bytes(32));
        $now = time();
        $ins = $this->db->prepare(
            'INSERT INTO refresh_tokens
               (token_hash, client_id, region, family, issued_at, expires_at)
             VALUES (:h, :c, :r, :f, :i, :e)'
        );
        $ins->execute([
            ':h' => hash('sha256', $refresh),
            ':c' => $clientId,
            ':r' => $region,
            ':f' => $family,
            ':i' => $now,
            ':e' => $now + $this->refreshTtl,
        ]);

        $access = $this->jwt->sign([
            'sub'    => $clientId,
            'region' => $region,
            'scope'  => 'discovery.read',
            'iat'    => $now,
            'exp'    => $now + $this->accessTtl,
        ]);

        return [
            'access_token'  => $access,
            'refresh_token' => $refresh,
            'expires_in'    => $this->accessTtl,
        ];
    }

    private function revokeFamily(string $family): void
    {
        $this->db->prepare(
            'UPDATE refresh_tokens
                SET revoked = 1, used_at = COALESCE(used_at, :now)
              WHERE family = :f'
        )->execute([':now' => time(), ':f' => $family]);
    }
}

The important detail is the conditional UPDATE ... WHERE id = :id AND used_at IS NULL. Two requests can read the same live row before either writes; the WHERE used_at IS NULL clause means only one UPDATE can affect a row, and rowCount() tells the loser it lost. The loser does not silently retry — it treats the collision as reuse and revokes the family, because a legitimate client never fires two refreshes for the same token on purpose. (We soften that with a small grace window in production; see the gotchas.)

The client side surviving rotation (Python)

Rotation moves complexity onto the client: the refresh token changes on every use, so a client that forgets to persist the new one locks itself out permanently on the next call. This is by far the most common integration bug we see. Here is a TokenStore that does it correctly, with a skew buffer and a single 401-triggered retry.

import threading
import time
import requests


class TokenStore:
    def __init__(self, base_url: str, refresh_token: str):
        self._base = base_url.rstrip("/")
        self._refresh = refresh_token
        self._access = None
        self._exp = 0.0
        self._lock = threading.Lock()

    def _rotate(self) -> None:
        r = requests.post(
            f"{self._base}/auth/refresh",
            json={"refresh_token": self._refresh},
            timeout=10,
        )
        if r.status_code == 401:
            raise RuntimeError("refresh rejected - full re-authentication needed")
        r.raise_for_status()
        data = r.json()
        self._access = data["access_token"]
        self._refresh = data["refresh_token"]          # keep the rotated token!
        self._persist(self._refresh)                   # and durably store it
        self._exp = time.time() + data["expires_in"] - 30  # 30s skew buffer

    def _persist(self, refresh: str) -> None:
        # Write to disk/keychain BEFORE using the new access token, so a crash
        # never leaves you holding a refresh token the server already retired.
        with open(".refresh_token", "w") as fh:
            fh.write(refresh)

    def token(self) -> str:
        with self._lock:
            if not self._access or time.time() >= self._exp:
                self._rotate()
            return self._access

    def get(self, path: str):
        headers = {"Authorization": f"Bearer {self.token()}"}
        resp = requests.get(f"{self._base}{path}", headers=headers, timeout=10)
        if resp.status_code == 401:
            with self._lock:
                self._exp = 0.0            # force a rotate on the next token()
            headers = {"Authorization": f"Bearer {self.token()}"}
            resp = requests.get(f"{self._base}{path}", headers=headers, timeout=10)
        resp.raise_for_status()
        return resp.json()

The ordering in _rotate is deliberate: persist the new refresh token to durable storage before you rely on the new access token. If the process crashes between receiving the response and writing it down, you must recover with the token the server considers current — otherwise you replay the old one and trip your own reuse detection.

Making refresh concurrency-safe (Go)

Go clients tend to be the ones that break rotation, because they fan out dozens of goroutines that all hit a 401 at the same instant and all try to refresh. Without coordination that is N concurrent refreshes of the same token, and every one after the first is a reuse-detection hit that revokes the family. A single mutex around the rotate-or-return decision collapses the stampede into exactly one refresh.

package videoclient

import (
    "encoding/json"
    "errors"
    "net/http"
    "strings"
    "sync"
    "time"
)

type Tokens struct {
    mu      sync.Mutex
    baseURL string
    access  string
    refresh string
    expires time.Time
    http    *http.Client
}

func (t *Tokens) rotate() error {
    body, _ := json.Marshal(map[string]string{"refresh_token": t.refresh})
    resp, err := t.http.Post(
        t.baseURL+"/auth/refresh", "application/json", strings.NewReader(string(body)))
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode == http.StatusUnauthorized {
        return errors.New("refresh rejected: full re-authentication needed")
    }
    var out struct {
        Access  string `json:"access_token"`
        Refresh string `json:"refresh_token"`
        In      int    `json:"expires_in"`
    }
    if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
        return err
    }
    t.access = out.Access
    t.refresh = out.Refresh // rotation: adopt the new refresh token
    t.expires = time.Now().Add(time.Duration(out.In-30) * time.Second)
    return nil
}

// Token returns a valid access token, rotating at most once even when
// dozens of goroutines call it simultaneously.
func (t *Tokens) Token() (string, error) {
    t.mu.Lock()
    defer t.mu.Unlock()
    if t.access == "" || time.Now().After(t.expires) {
        if err := t.rotate(); err != nil {
            return "", err
        }
    }
    return t.access, nil
}

Because the freshness check and the rotate live inside the same lock, the first goroutine through does the network round-trip while the rest block; when they acquire the lock the token is already fresh and they return immediately. One refresh, one rotation, zero false reuse hits.

Cleaning up across regions with cron

Each edge node runs its own auth.db, and each has a cron job that prunes dead rows and surfaces reuse-detection events for alerting. We keep used rows around for 24 hours so that a replay still has something to trip on — delete them too eagerly and a stolen token that comes back later looks like an unknown token instead of a reused one, and you lose the family-revocation signal.

#!/usr/bin/env php
<?php
declare(strict_types=1);

// Runs from the per-region cron on each edge node.
$db = new PDO('sqlite:' . __DIR__ . '/data/auth.db');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db->exec('PRAGMA busy_timeout = 5000');   // wait, don't fail, under write locks

$now = time();

// 1. Purge expired tokens and used tokens older than 24h.
$purge = $db->prepare(
    'DELETE FROM refresh_tokens
      WHERE expires_at < :now
         OR (used_at IS NOT NULL AND used_at < :cutoff)'
);
$purge->execute([':now' => $now, ':cutoff' => $now - 86400]);

// 2. Report families force-revoked in the last hour - these are the
//    reuse-detection hits worth paging on.
$hits = $db->query(
    'SELECT region, COUNT(DISTINCT family) AS families
       FROM refresh_tokens
      WHERE revoked = 1 AND used_at > ' . ($now - 3600) . '
      GROUP BY region'
)->fetchAll(PDO::FETCH_ASSOC);

fwrite(STDERR, sprintf(
    "auth-cleanup purged=%d reuse_families=%s\n",
    $purge->rowCount(), json_encode($hits)
));

Gotchas we hit in production

None of these are in the RFCs. All of them cost us a debugging session:

Never FTP-sync auth.db. Our deploys are FTP-based, and the first version of the deploy script happily overwrote each node's live token database with a stale local copy, silently invalidating every active session on push. Exclude data/ from the sync and let each region own its own token store. Code deploys; state does not.
Persist the rotated refresh token before using the new access token. This is the number-one client bug. If you use the new access token and crash before writing down the new refresh token, your next refresh replays a retired token and revokes your own family.
Add a small reuse grace window. A flaky network can make a client resend the exact same refresh request after a timeout, even though the server already processed the first one. Punishing that with immediate family revocation generates false lockouts. We allow the same rotated response to be returned once within a 10-second idempotency window (keyed on the presented hash) before we treat a repeat as a genuine attack.
Budget for clock skew. Access tokens are validated against exp on eight independent nodes whose clocks drift. Give validation a few seconds of leeway and have clients refresh 30 seconds early, as the samples above do, so a slightly-fast client clock never presents an already-expired token.
Mind SQLite write contention. Under a refresh burst, concurrent writers will collide. PRAGMA busy_timeout turns an instant SQLITE_BUSY failure into a short wait, and keeping the auth store per-region rather than shared keeps write volume low enough that a single-writer database is never the bottleneck.
Rotate signing keys with kid. The access token header carries a kid so you can introduce a new HMAC secret, sign with it, and still validate tokens minted under the old key until they age out — no flag-day, no mass invalidation.

Conclusion

Refresh-token rotation is not complicated, but it is unforgiving: the security payoff comes entirely from single-use tokens plus reuse detection, and both of those turn ordinary client sloppiness — a dropped persist, a goroutine stampede, a network retry — into a lockout unless you handle it deliberately. Get the server contract right first (mark-then-mint inside one transaction, revoke the whole family on replay), then make every client persist the rotated token before it trusts the new access token, and collapse concurrent refreshes behind a single lock. Do that and a leaked token stops being a two-week skeleton key and becomes a self-destructing, loudly-alerting event — which, for an API that ranks video across eight regions, is exactly the trade you want. Steal the schema and the client patterns above; they are the same ones running in production for us today.

EU Withdrawal Requests: A Magento 2 Module for Secure Compliance Flows

KodeXpo — Tue, 14 Jul 2026 20:01:01 +0000

Magento stores that sell to EU customers need a withdrawal flow that is secure, auditable, and clear for both customers and support teams.

The module is built to support merchants implementing the consumer right of withdrawal under Directive 2011/83/EU (CELEX 32011L0083). The directive establishes the 14-day withdrawal period and allows withdrawal by an unequivocal statement, which is what this module operationalizes in Magento Open Source.

The problem is that many stores handle this with a contact form, a loose email thread, or a custom one-off workflow that exposes order data and creates confusion later. That is not enough when you need ownership verification, request tracking, manual review, and store-specific timing rules.

Why This Module Exists

EU withdrawal requests are easy to get wrong.

A good implementation needs to balance:

customer experience
guest verification
request integrity
admin review
configurable compliance timing
safe email handling
abuse resistance

This module is designed to cover all of that without exposing order data publicly or relying on brittle manual steps.

What It Does

KodeXpo_ReturnCompliance adds a secure withdrawal flow for Magento 2 stores:

customer account withdrawal requests
guest withdrawal requests with secure verification links
manual admin review with approve / reject handling
configurable withdrawal window logic
configurable anchor offsets for delivery timing
resend throttling and confirmation attempt limits
configurable email templates and sender identity
optional guest reCAPTCHA support
translated storefront and admin copy

How the Flow Works

Customer Flow

A logged-in customer can open an order in My Account and submit a withdrawal request directly from the order view page.

The module then:

checks whether the order is still eligible
stores the request
shows a confirmation screen
sends the request into the admin review queue

Guest Flow

Guest users must verify ownership before the request can proceed.

The guest flow uses:

order number
email address on the order
secure tokenized confirmation link
throttled resend behavior
max attempt protection

This keeps the flow usable without exposing the order or verification details in the URL.

Admin Flow

The admin side gives support teams a clear request view:

request status
order reference
customer email
review note
reviewed by
reviewed at
approval / rejection decision

When a request is reviewed, the module also appends an order comment so the decision is visible in the order history.

Why This Is Better Than a Simple Form

A simple form is not enough for a real withdrawal workflow.

This module gives you:

request tracking instead of a one-off submission
secure guest ownership verification
timing rules you can tune per store
manual review before a decision is finalized
email templates that can be localized and branded
throttling that reduces abuse
admin visibility that does not depend on tribal knowledge

In other words: it makes the process explicit instead of silent.

Security and Abuse Protection

Security was a core requirement.

The module avoids the common mistakes:

no raw customer email or verification code exposed in the guest URL
opaque verification tokens instead of predictable identifiers
resend throttling
limited confirmation attempts
configurable guest verification retention
form key protection where applicable
optional reCAPTCHA for guest entry points

The goal is not only to make the flow work, but to make it hard to abuse.

Configuration

The module is configurable from:

Stores > Configuration > KodeXpo > EU Return Compliance

Typical settings include:

enable / disable module
withdrawal window
anchor order priority
AWB, shipment, invoice, and order offsets
guest verification TTL
resend delay
max attempts
email template selection
from identity
reCAPTCHA support

The module also uses Magento’s own email and configuration conventions, so it fits into a normal store setup instead of introducing a parallel system.

Compatibility

The module is designed for:

Magento Open Source 2.4.6 through 2.4.9
Adobe Commerce 2.4.6 through 2.4.9
PHP 8.1 through 8.5

Localization

The module includes translations for a wide set of locales, including:

en_US
ro_RO
de_DE
fr_FR
es_ES
it_IT
pt_PT
nl_NL
pl_PL
cs_CZ
da_DK
sv_SE
fi_FI
nb_NO
el_GR
hu_HU
sk_SK
sl_SI
hr_HR
bg_BG
lt_LT
lv_LV
et_EE

Installation

Composer

composer require kodexpo/module-return-compliance
bin/magento module:enable KodeXpo_ReturnCompliance
bin/magento setup:upgrade
bin/magento cache:clean

If needed:

bin/magento setup:di:compile
bin/magento setup:static-content:deploy en_US -f

Manual

Copy the module to:

app/code/KodeXpo/ReturnCompliance

Then run:

bin/magento module:enable KodeXpo_ReturnCompliance
bin/magento setup:upgrade
bin/magento cache:clean

Limitations

This module is intentionally focused on the withdrawal workflow.

It does not:

auto-approve requests
automatically correct order state
replace legal review
bypass store-specific eligibility rules

It is meant to make the compliance flow secure and operational, not to remove the need for review.

Closing

If your store needs a withdrawal flow that is secure, configurable, and visible to admins, this module gives you a practical base to build on.

The important part is not just supporting EU withdrawal requests. It is doing it in a way that is safe, auditable, and maintainable inside Magento.

Seven things production taught me that no tutorial did

Varun Markanday — Tue, 14 Jul 2026 16:57:56 +0000

I spent the last few months building out a shipping integration on top of Shippo, and along the way, I kept running into the same lesson: the "quick" way to write Laravel code almost always works — right up until it doesn't. This isn't a theory post. Every pattern below came from something that actually broke, or almost broke, in this codebase.

Here are seven things I'd tell a past version of myself before writing another shipment controller.

1. Stop passing raw arrays to third-party APIs.
The ticket says "wire up Shippo shipment creation." The fastest way to close it is to grab a few fields off the order, throw them into an array, and fire it at the HTTP client. It works on the first try. It works in the demo. So what's wrong with it?

Nothing — until six months from now, when a new dev who's never opened Shippo's docs calls your method, forgets that parcels need a distance_unit key, and finds out the hard way via a 422 in staging. The array approach doesn't fail where the mistake happens. It fails at the network boundary, with a stack trace pointing at your HTTP client instead of the line that actually got it wrong.

My first instinct was "Shippo owns this shape, so a DTO is just re-describing their API." That's backwards. The DTO isn't there to constrain Shippo — it's there to constrain me. It makes it structurally impossible to send Shippo something malformed, and it gives the next person a single file that answers "what does a shipment request actually need?"

phpfinal readonly class ShippoParcel
{
public function __construct(
public float $lengthCm,
public float $widthCm,
public float $heightCm,
public float $weightKg,
) {}

public function toPayload(): array
{
    return [
        'length' => (string) $this->lengthCm,
        'width' => (string) $this->widthCm,
        'height' => (string) $this->heightCm,
        'distance_unit' => 'cm',
        'weight' => (string) $this->weightKg,
        'mass_unit' => 'kg',
    ];
}

}

toPayload() is doing something specific: it's the one place in the codebase that knows Shippo wants distance_unit and mass_unit as separate string keys instead of floats. That's a Shippo quirk, not a domain concept, and it deserves to live exactly once — at the edge — instead of being copy-pasted into every place that builds a parcel array.

Do the same thing on the response side. If a successful transaction contains a tracking_number buried three levels deep, don't make every consumer memorize that. Wrap it in a response DTO and throw a domain exception right there if a field you depend on is missing, instead of discovering it three calls later when something null-derefs.

2. Contextual binding: stop putting environment checks inside your classes.

Most of the time, one global container binding is exactly right. Don't reach for contextual binding before you need it — that's its own flavor of overengineering. But there's a specific smell that tells you when you've outgrown it: a service class that checks config() or does an instanceof check to decide its own behavior. That class just took on a second job — deciding which implementation it should be — and that job belongs to the container.

Here's a real one from this project: staging needs to hit Shippo's actual sandbox API so QA can test the full flow. But the internal support tool — the one an ops person uses to reprint a lost label — needs live credentials even when it's deployed next to staging, because a support agent reprinting a real label needs a real tracking number. An APP_ENV check can't express that; both consumers live in the same deployment.

php$this->app->when(QaShipmentController::class)
->needs(ShippoClient::class)
->give(SandboxShippoHttpClient::class);

$this->app->when(SupportLabelController::class)
->needs(ShippoClient::class)
->give(ShippoHttpClient::class);

Both controllers just type-hint ShippoClient. Neither has an if branch. Neither can drift onto the wrong credentials because someone forgot an env check.

One thing I'd flag: when you use the closure form of give() to resolve something like a region-specific storage driver, make the unmatched case throw instead of silently falling back to a default region.

php$this->app->when(LabelArchiver::class)
->needs(StorageDriverInterface::class)
->give(function ($app) {
return match (config('services.region')) {
'eu' => $app->make(S3EuStorageDriver::class),
'us' => $app->make(S3UsStorageDriver::class),
default => throw new RuntimeException('Unconfigured region: no storage driver bound.'),
};
});

A misconfigured deployment failing loudly at boot is a five-minute fix. The same misconfiguration that silently archives EU customer data to a US bucket for three weeks is a very different conversation with legal.

3. API versioning: the database doesn't get an opinion

The tempting shortcut when a mobile endpoint needs to change: just change it, ship a new app version, tell stragglers to update. It's less work than standing up parallel routes and resources for what might be a one-field difference.

Here's why that doesn't survive contact with mobile clients: you can't force an update. A web deploy hits everyone within minutes. A phone with your app from fourteen months ago, sitting in a drawer half the year, hits your API tomorrow with the same confidence as someone on today's build. You can't patch that client — you can only decide, in advance, how long you're willing to keep talking to it.

The rule I hold myself to now: version at the boundary, never inside business logic. The second a version check shows up inside a service class, that check is going to outlive the version it was written for.

phpfinal class ShipmentController extends V1\ShipmentController
{
protected function resource(Shipment $shipment): JsonResource
{
return ShipmentResource::make($shipment);
}
}

If v2 only changes the response shape, don't fork the whole controller — fork the resource and let v2 inherit the rest. Duplicating the entire controller feels safer in the moment, but now every bug fix has to be applied twice by someone who has to remember both files exist.

And tell clients a version is dying using something machine-readable, not a changelog nobody on the client side reads:

php$response->headers->set('Deprecation', 'true');
$response->headers->set('Sunset', 'Wed, 01 Oct 2026 00:00:00 GMT');

Then actually count requests against the old version after your sunset date. That count is the only honest answer to "can we turn this off yet" — not a guess from support tickets.

The part that actually burns teams isn't the routing layer; it's migrations. Add new columns as nullable, write to both old and new during the transition, confirm traffic on the old version has actually dropped to zero, and only then drop the legacy column in a separate deploy. "Low" traffic isn't zero traffic — one enterprise customer's warehouse scanner still hitting v1 nightly is enough to turn a routine cleanup into an on-call incident.

4. Real-time dashboards: broadcasting demos beautifully and then lying to you

Before reaching for WebSockets at all — just poll. setInterval, a fetch call, done. For a lot of internal dashboards, that's genuinely the right answer, not a lesser one. Broadcasting earns its complexity when both the update volume and the number of watchers climb.

What the "add a trait, save a model, watch it update live" demo doesn't show you: broadcasts that silently never arrive, events that land out of order, and clients that missed a whole window of updates because their laptop slept for two minutes.

A shipment tracking dashboard hits all three because "label purchased" and "in transit" webhooks can land milliseconds apart, and normal network jitter is enough to flip that order by the time it reaches a browser. The fix is a version field that the client can compare against:

phppublic function broadcastWith(string $event): array
{
return match ($event) {
'updated' => [
'id' => $this->id,
'status' => $this->status,
'version' => $this->updated_at->getTimestampMs(),
],
default => [],
};
}
jsEcho.private(shipments.${shipmentId}).listen('.ShipmentUpdated', (e) => {
if (e.version <= (lastSeenVersion[e.id] || 0)) return; // stale, ignore
lastSeenVersion[e.id] = e.version;
updateDashboard(e.status, e.trackingNumber);
});

The one everybody skips: reconnects. A dropped socket has zero memory of what it missed — there's no replay buffer. On reconnect, refetch the current state from a plain REST endpoint before trusting the next broadcast. If you skip this, you'll eventually get a bug report that reads "the dashboard says pending but it actually shipped hours ago," and it'll take you a minute to realize it's not a broadcasting bug at all — it's a missing REST call.

5. N+1 queries hide behind conditionals, not in obvious loops

Nobody writes an obvious N+1 loop and misses it in review. The ones that ship are nested two or three levels deep inside a branch that only fires for a subset of rows:

php$shipments = Shipment::with('customer')->get();
foreach ($shipments as $shipment) {
if ($shipment->customer->isHighVolume()) {
// N+1 fires here — parcels were never eager loaded
$shipment->parcels->each(fn ($p) => $p->carrier->name);
}
}

with('customer') only reaches one level deep. This is invisible in review unless someone mentally traces every branch two relations deep, and invisible in local dev because your ten-seeded shipments run ten queries instead of ten thousand. It shows up as connection pool exhaustion in production, during your busiest hour — not as a comment in a PR.

The actual fix isn't "remember to eager load" — memory isn't an engineering control. Make the framework refuse to ship the mistake:

phpModel::preventLazyLoading(! $this->app->isProduction());
Model::handleLazyLoadingViolationUsing(function (Model $model, string $relation): void {
logger()->warning("Attempted to lazy load [{$relation}] on model [" . get_class($model) . "].");
});

Throw in local and staging, where a broken deploy costs nothing. Log-and-allow in production, where throwing on a lazy load would turn a performance bug into an outage on the spot. And enable this in your test suite too — a test that seeds exactly one shipment with one parcel will never trip an N+1, because there's nothing to multiply.

For the fix itself, reach for loadMissing() instead of sprinkling with() everywhere defensively. It costs nothing if the caller already did the right thing, and acts as a safety net at every controller/job boundary if they didn't.

6. Cache::flush() is a sledgehammer with a hair trigger

A carrier's negotiated rate changes, something needs invalidating, and Cache::flush() is sitting right there — one line, guaranteed correct, impossible to get subtly wrong. It also wipes every unrelated cached value in your app and triggers a stampede as every concurrent request misses at once and hits the database simultaneously.

Cache tags scope the blast radius to exactly what changed:

phpCache::tags(['rates', "carrier:{$carrierId}"])->put(
"rate.{$originZip}.{$destZip}", $rateQuote, now()->addHours(6)
);

Cache::tags(["carrier:{$carrierId}"])->flush();

Wire the flush to the model lifecycle so it's correct by construction instead of by every developer remembering to add an invalidation line next to every write:

phpprotected static function booted(): void
{
static::saved(function (CarrierAccount $account): void {
if ($account->wasChanged('negotiated_rate_table')) {
Cache::tags(["carrier:{$account->id}"])->flush();
}
});
}

Two things worth knowing before you build around this: cache tags require Redis or Memcached — the file and database drivers throw at runtime, not fail gracefully. And flush after your transaction commits, not before. Flush too early, and a request landing in that window reads the pre-update row, caches it, and now you're serving stale data for the full TTL with nothing telling you it happened.

Laravel 13 also adds Cache::touch(), which is worth knowing about for the more common case: a key is still valid, and you just want to push the TTL out without paying to rewrite the payload.

7. Model::all() works fine until your table grows

It's one call, it reads clean, and it's the answer in every tutorial. On a few hundred rows, it's genuinely correct. On a table with hundreds of thousands of open shipments, it's an OOM before your loop runs a single iteration — and the failure has nothing to do with your loop logic. It happens at the query level.

The choice comes down to two questions: does your loop mutate the column you're filtering on, and is each iteration doing slow I/O?

Read-only, fast → chunk()
Mutates the filtered column → chunkById() — pages by primary key instead of offset, so a row changing status mid-loop can't silently shift what the next page fetches (this is the one that bites people: chunk() under mutation doesn't error, it just quietly skips rows)
Slow outbound I/O → lazyById() — same generator ergonomics as cursor(), but doesn't hold a database connection open for the full duration of every outbound API call

phpShipment::where('status', 'in_transit')
->lazyById(500)
->each(function (Shipment $shipment): void {
$shipment->refreshTrackingStatus();
$shipment->save();
});

Get the second question wrong, and you trade an OOM for silently under-processing your table — which is arguably worse, because nothing throws to tell you it happened.

None of these is exotic. They're all things Laravel gives you out of the box. The theme across all seven, if there is one, is the same: local dev and a small seed dataset will hide every one of these problems from you, and production will find every single one at the worst possible time. Building the guardrail in — a DTO, a container binding, a lazy-loading exception, a version header — beats trying to remember not to make the mistake.

If you've hit a different version of any of these, I'd genuinely like to hear it — drop it in the comments.

You can't find a bug you never named

Robin Dhiman — Tue, 14 Jul 2026 15:22:39 +0000

There's a post going around arguing that every project should ship an invariants.md. I agree. What convinced me wasn't a blog post, though. It was a SELECT statement.

The query that every test approved

In a review of a split-settlement module for a multi-vendor marketplace, a resource model built a query that read base_shipping_refunded off sales_creditmemo. That column isn't on the credit memo. It's on sales_order. The first real refund in production would have thrown.

The unit tests were green. All of them. The resource model was mocked, and a mock returns whatever you told it to return, including a value for a column that has never existed in any database anywhere.

Static analysis didn't care either. The column name is a string.

So the code cleared every gate we had, and the bug was still sitting in the diff, in plain sight, unremarked. Not because anyone was sloppy. Because nothing in the project stated the rule it broke.

A finding is a violated rule

Here's the reframe that changed how I review code.

A bug isn't a smell or an ugliness. A bug is a specific statement about the system that is supposed to be true and isn't. "The settlement legs sum to what the customer paid." "The same webhook delivered twice moves money once." "Every column a query names exists on the table it names."

Which means: you cannot find a violation of a rule you never wrote down. A reviewer reading a diff cold is doing two jobs at once. Inferring what the system is supposed to guarantee, then checking whether this code breaks it. The first job is the hard one, and it gets redone, badly, by every person who opens the file.

Write the rules down once and reviewing collapses into a smaller task: take each statement, try to break it.

What they actually look like

Not architecture. Not "the code should be clean." Falsifiable statements about state:

# invariants.md (settlement)

CONSERVATION
- sum(settlement legs on an order) == the amount the customer actually paid.
- No path creates money. No path destroys it. A refund reduces legs; it never mints one.

IDEMPOTENCY
- Any gateway callback may be delivered twice. Processing it twice moves money once.
- Every write is keyed by the gateway's own reference, unique-constrained in the schema.

ORDERING
- A payout release refuses while any refund on that order is open.
- A leg never goes from settled back to pending.

ISOLATION
- One vendor's failed leg does not block another vendor's payout on the same order.

SCHEMA
- Every column a query names exists on the table it names. Verified against db_schema.xml,
  not against a mock.

Each line is testable, arguable, and either true or false of a given code path. That last one exists purely because of the SELECT above. An invariants file is where you keep the scar tissue.

Point the review at the list

The list earns its keep at review time, and it earns it twice over when the reviewer is a model.

"Look for bugs in this diff" produces plausible findings, and you spend the afternoon disproving them. "Prove none of these six statements can be broken by this diff, and give me the exact inputs that break the one that can" produces something you can act on. The rules give the review a shape, and they give you grounds to reject a finding that isn't anchored to one.

They also make disagreement useful. On that same review, two reviewers reached opposite conclusions about whether a credit memo could over-refund tax. Neither position was checkable against a vibe. Both were checkable against the invariant, and the invariant was checkable against the framework source. Reading the core class that registers a credit-memo item settled it in ten minutes: the refunded amount it tracks is tax-exclusive, so the over-refund was real.

A model's finding is a lead. The invariant tells you which lead is worth chasing. The source tells you whether it's true.

Turn the ones you can into assertions

Some invariants become queries. Conservation is the obvious one:

SELECT o.entity_id,
       o.base_grand_total,
       SUM(l.base_amount) AS legs
FROM   sales_order o
JOIN   vendor_settlement_leg l ON l.order_id = o.entity_id
GROUP  BY o.entity_id, o.base_grand_total
HAVING ABS(o.base_grand_total - SUM(l.base_amount)) > 0.01;

Zero rows means the money conserves. Run it nightly against production and a paragraph of intent has become an alarm.

Idempotency usually becomes a unique key rather than a test, which is stronger. If the gateway reference is unique in db_schema.xml, a redelivered webhook cannot insert a second leg no matter what the application layer does. Enforce it where it can't be forgotten.

Others stay prose. "One vendor's failure doesn't freeze another vendor's payout" was one we couldn't fully hold. The credit-memo freeze in that module is order-wide, so a dispute on one vendor's item does hold the rest. We shipped it anyway as a documented trade-off, and it got to be a decision instead of an argument only because it was written down as a named rule with a known blast radius.

What this doesn't fix

An invariants file doesn't tell you the truth. It tells you what to check.

The schema rule is the clearest case. Writing "every column exists" changes nothing by itself. It becomes real when someone opens db_schema.xml and reads the table definition instead of trusting a green test. Mocked tests do not validate columns. They never have. That's the gate most reviews skip, and it costs about four minutes.

The habit

Before the next module goes to review, write the file. Six to ten lines. What must always be true, stated specifically enough to be wrong.

Then review against it, and treat anything you find that isn't on the list as evidence the list is incomplete.

The bug I opened with was caught by reading a schema file, not by being clever. That's usually how it goes.

Source / further reading: https://twitter.com/PiccoGabriele/status/2076876444760957440

The Specification Pattern in Laravel: Refactoring Complex Business Rules

CodeCraft Diary — Tue, 14 Jul 2026 14:43:00 +0000

As our Laravel applications grow, we inevitably face a common architectural dilemma: where do we put complex business logic that dictates whether an entity meets certain criteria?

You start simple. A user is allowed to purchase a premium item if they are active. Easy—you write a quick Eloquent scope. But three months later, the business team changes the rules. Now, a user can only buy that item if they are active, have a verified email, have spent at least $500 in the last 90 days, and are not flagged for suspicious activity.

Suddenly, your code becomes littered with long chains of Eloquent scopes in your queries, and worse, identical blocks of if-else statements in your PHP services to check the exact same logic for already-loaded models.

This is where the codebase starts to smell. We are violating the Single Responsibility Principle, duplicating rules, and creating a maintenance nightmare. To fix this, developers often reach for complex design patterns that end up overengineering the solution, violating the KISS (Keep It Simple, Stupid) principle.

But there is a elegant, highly practical pattern that solves this perfectly without breaking a sweat: The Specification Pattern.

Previous article in refactoring category: https://codecraftdiary.com/2026/06/15/laravel-event-driven-architecture/

The Real-World Problem: The "Premium Customer" Mess

Let’s look at a concrete example. Imagine an e-commerce platform where we need to determine if a customer qualifies for a VIP Discount.

The business rules for a VIP customer are:

The account must be active.
The user must have a verified email.
They must have made at least 5 orders in total.
They must not be on our fraud blacklist.

The Naive Laravel Approach

Typically, you see developers solving this in two ways. First, they write a massive query in a controller or service:

// In a Controller or Service
$vipUsers = User::query()
    ->where('is_active', true)
    ->whereNotNull('email_verified_at')
    ->whereHas('orders', function ($query) {
        $query->where('status', 'completed');
    }, '>=', 5)
    ->whereDoesntHave('flags', function ($query) {
        $query->where('type', 'fraud');
    })
    ->get();

This works fine for fetching data from the database. But what happens when you already have a User instance in memory—say, inside a Job or an Event Listener—and you need to check if this specific user qualifies?

You end up duplicating the logic in pure PHP:

public function qualifiesForVipDiscount(User $user): bool
{
    return $user->is_active 
        && $user->email_verified_at !== null
        && $user->orders()->where('status', 'completed')->count() >= 5
        && !$user->flags()->where('type', 'fraud')->exists();
}

Why this breaks the KISS principle:

Duplication: If the fraud rule changes, you have to find and rewrite both the SQL/Eloquent logic and the in-memory PHP logic.
Model Bloat: Shoving this into the User model makes it a "God Object" over time.
Hidden Rules: Your business logic is trapped inside technical database implementation details.

Enter the Specification Pattern

The Specification Pattern solves this by turning a business rule into a first-class citizen—a single, isolated, reusable class.

A strict specification pattern can sometimes get overly complex with abstract syntax trees and custom expression builders. To keep it KISS-compliant, we are going to build a pragmatic version tailored for modern PHP 8.5+ and Laravel.

Let's define a simple interface for our specifications:

namespace App\Specifications;

use Illuminate\Database\Eloquent\Builder;

interface Specification
{
    /**
     * Check if a given object satisfies the specification in-memory.
     */
    public function isSatisfiedBy(mixed $candidate): bool;

    /**
     * Apply the specification directly to an Eloquent query builder.
     */
    public function apply(Builder $query): void;
}

Implementing the Specification

Let's build our IsVipCustomer specification. Instead of writing abstract logic, we put our exact four business rules inside this class.

namespace App\Specifications;

use App\Models\User;
use Illuminate\Database\Eloquent\Builder;

class IsVipCustomer implements Specification
{
    public function isSatisfiedBy(mixed $candidate): bool
    {
        if (!$candidate instanceof User) {
            return false;
        }

        return $candidate->is_active
            && $candidate->email_verified_at !== null
            && $candidate->orders->where('status', 'completed')->count() >= 5
            && !$candidate->flags->contains('type', 'fraud');
    }

    public function apply(Builder $query): void
    {
        $query->where('is_active', true)
            ->whereNotNull('email_verified_at')
            ->whereHas('orders', function ($q) {
                $q->where('status', 'completed');
            }, '>=', 5)
            ->whereDoesntHave('flags', function ($q) {
                $q->where('type', 'fraud');
            });
    }
}

Notice how clean this is. The technical details of how we determine a VIP customer are encapsulated in exactly one file.

Making It Truly Useful: Combining Specifications

Where this pattern completely demolishes standard refactoring approaches is when business requirements change or combine. What if we want to find users who are VIP Customers AND are Located in the EU (for tax or shipping promotions)?

Instead of writing a third massive scope, we can create a simple composition layer. Let's create an AndSpecification:

namespace App\Specifications;

use Illuminate\Database\Eloquent\Builder;

class AndSpecification implements Specification
{
    private array $specifications;

    public function __construct(Specification ...$specifications)
    {
        $this->specifications = $specifications;
    }

    public function isSatisfiedBy(mixed $candidate): bool
    {
        foreach ($this->specifications as $spec) {
            if (!$spec->isSatisfiedBy($candidate)) {
                return false;
            }
        }
        return true;
    }

    public function apply(Builder $query): void
    {
        foreach ($this->specifications as $spec) {
            $spec->apply($query);
        }
    }
}

Putting It Into Practice (Where It Pays Off)

Let's look at how this simplifies our daily Laravel tasks. We'll look at a Controller (database querying) and a Service/Job (in-memory validation).

Scenario A: Querying the Database

Inside your Admin Dashboard controller, you need to list all VIP customers from the EU to send them a special newsletter.

namespace App\Http\Controllers;

use App\Models\User;
use App\Specifications\AndSpecification;
use App\Specifications\IsVipCustomer;
use App\Specifications\IsLocatedInEU;

class VipNewsletterController extends Controller
{
    public function __invoke()
    {
        // Define the composite rule
        $vipInEuSpec = new AndSpecification(
            new IsVipCustomer(),
            new IsLocatedInEU()
        );

        // Apply it directly to the query
        $query = User::query();
        $vipInEuSpec->apply($query);

        $users = $query->paginate(30);

        return view('admin.vip-newsletter', compact('users'));
    }
}

Scenario B: In-Memory Domain Validation

Now imagine a completely different part of the app. A user clicks "Claim VIP Reward". The user is already logged in, so we already have the $user instance loaded in memory. We don't want to hit the database with complex heavy joins again if we can avoid it.

namespace App\Http\Controllers;

use App\Specifications\IsVipCustomer;
use Illuminate\Http\Request;

class RewardController extends Controller
{
    public function claim(Request $request, IsVipCustomer $vipSpec)
    {
        $user = $request->user();

        // Check the exact same business rule in-memory
        if (!$vipSpec->isSatisfiedBy($user)) {
            return response()->json([
                'error' => 'You do not qualify for this reward.'
            ], 403);
        }

        // Process reward...
        return response()->json(['success' => 'Reward claimed!']);
    }
}

Why This is NOT Overengineering

If you look at the code above, a skeptic might say: "Why did we create three classes when a couple of Eloquent scopes could do the trick?"

Here is why this approach honors the KISS principle when dealing with real complexity:

The "Single Source of Truth" rule: If the marketing team changes the definition of a VIP customer to require 10 orders instead of 5, you change one number in one class. Your controllers, jobs, and models remain completely untouched.
Database vs. Memory consistency: Standard Eloquent scopes cannot be run on a regular PHP collection or an already-loaded model without triggering fresh SQL queries. The Specification Pattern bridges this gap natively.
Impeccable Testability: Testing complex business logic mixed with controllers is painful. Testing a Specification class is a pure unit test. You pass a mock user, assert true or false, and you're done in milliseconds.

public function test_user_with_insufficient_orders_is_not_vip()
{
    $user = new User(['is_active' => true, 'email_verified_at' => now()]);
    $user->setRelation('orders', collect()); // 0 orders
    $user->setRelation('flags', collect());

    $spec = new IsVipCustomer();

    $this->assertFalse($spec->isSatisfiedBy($user));
}

Summary: When to Use It

Don't use this pattern for simple things like User::where('role', 'admin'). That would be overengineering.

Reach for the Specification Pattern when:

A business rule is dynamic and changes often based on business requirements.
You need to use the exact same rule for both database filtering and in-memory validation.
You need to combine multiple business rules dynamically depending on the context (e.g., matching different combinations of criteria for user segmentation).

By encapsulating volatile business logic into isolated specifications, you keep your models thin, your controllers clean, and your application incredibly adaptive to future changes. It’s Clean Architecture at its most practical level.

Testing Laravel Events and Listeners: Ensuring Reliable Asynchronous Workflows

CodeCraft Diary — Tue, 14 Jul 2026 13:48:00 +0000

In modern Laravel applications, events and listeners are the glue that holds our complex business logic together. They allow us to decouple our code, keeping controllers thin and services focused. However, as applications scale, this "decoupling" can become a testing nightmare.

We’ve all been there. You trigger an OrderPlaced event. It’s supposed to send an email, update the inventory, and notify the warehouse. One day, you realize the email never sent because an exception in the inventory listener swallowed the entire process. If you aren’t testing your events properly, you aren’t just missing code coverage—you are leaving your business workflows to chance.

In this guide, we’ll move beyond basic Event::fake() assertions and explore how to build a robust testing strategy for event-driven Laravel applications.

Previous article in this category: https://codecraftdiary.com/2026/06/22/tdd-in-laravel/

The Problem with "Testing Too Much"

When developers start testing events, the default instinct is to reach for Event::fake(). It’s easy, it’s fast, and it makes the test pass.

public function test_order_is_placed(): void
{
    Event::fake();

    $this->post('/checkout', [...]);

    Event::assertDispatched(OrderPlaced::class);
}

This test tells us one thing: Did we trigger the event? It tells us absolutely nothing about whether the listeners actually work or if they communicate correctly with each other. If you rely solely on faking, your test suite becomes a "smoke screen" that passes even when your underlying infrastructure is broken.

1. Unit Testing Listeners in Isolation

The best way to ensure reliability is to treat your Listeners like any other service class. A listener should have one job. If it has complex logic, extract it into a dedicated Action or Service class and test that.

// app/Listeners/SendOrderConfirmation.php
public function handle(OrderPlaced $event): void
{
    // Don't put business logic here!
    $this->mailer->sendConfirmation($event->order);
}

By keeping the listener thin, your unit test for SendOrderConfirmation becomes trivial. You can mock the Mailer service and simply verify the handle() method is called correctly. This is fast, deterministic, and catches bugs in the communication layer without overhead.

2. Integration Testing: The "End-to-End" Event Flow

When you want to verify that the OrderPlaced event actually triggers the SendOrderConfirmation listener and writes the correct data to the database, you need an integration test.

Crucially, do not fake the event here. Instead, allow the event to dispatch and verify the side effects.

public function test_order_placed_event_triggers_side_effects(): void
{
    // 1. Arrange: Setup user and cart
    $order = Order::factory()->create();

    // 2. Act: Trigger the event directly
    OrderPlaced::dispatch($order);

    // 3. Assert: Check side effects, not just the dispatch
    $this->assertDatabaseHas('emails', [
        'order_id' => $order->id,
        'status' => 'sent'
    ]);
}

This approach proves that your Service Provider is correctly registered and that the binding between the Event and Listener is active.

3. Handling Asynchronous Queues

The biggest trap is the mix of synchronous and asynchronous listeners. If your listener implements ShouldQueue, Event::fake() will prevent the job from ever being pushed to the queue.

To test queued listeners, use the Bus and Queue facades in tandem:

public function test_order_placed_event_queues_notification(): void
{
    Queue::fake();

    Event::dispatch(new OrderPlaced($order));

    Queue::assertPushed(SendOrderConfirmation::class, function ($job) use ($order) {
        return $job->order->id === $order->id;
    });
}

Pro Tip: If you are testing a complex flow, don't forget to test the failure state. Use Queue::assertPushed to verify that the retry logic or failure handling (like failed() methods) is configured for your mission-critical jobs.

4. Avoiding "The Silent Failure"

One common mistake is neglecting what happens when a listener fails. If your application relies on a chain of events, you must test the "atomic" nature of the flow.

If you are using Laravel 11/12+ features, ensure you are testing your custom shouldDiscoverEvents logic if you use event discovery. Hidden logic is the enemy of maintainable tests. If a developer adds a new listener to a directory, does your test suite automatically include it? By writing explicit integration tests for critical flows, you ensure that even "magically" discovered events are held accountable.

Testing Checklist for Events

To keep your test suite maintainable as your project grows:

Logic Extraction: If a listener is more than 5 lines, move the logic to an Action class.
Fake Sparingly: Use Event::fake() only when you specifically want to verify the triggering of an event, not its outcome.
Verify Side Effects: Always write at least one integration test that lets the event fire "for real" to ensure the pipeline is wired up correctly.
Queue Awareness: Always distinguish between testing the dispatch and testing the queued job execution.

Автоматические купоны ко дню рождения в 1С-Битрикс: batching, cron и защита от дублей

Андрей Викулов (VProger) — Tue, 14 Jul 2026 13:21:58 +0000

Зачем нужны автоматические купоны ко дню рождения в e-commerce

Поздравление покупателя с днём рождения — один из самых конверсионных инструментов в email-маркетинге интернет-магазина. По данным исследований Experian, письма с купонами ко дню рождения открывают в 2,5 раза чаще, чем обычные промо-рассылки, а их конверсия в покупку выше в несколько раз. Причина проста: человек уже лоялен к магазину, он ждёт внимания, и персонализированное предложение воспринимается не как спам, а как подарок.

Проблема возникает сразу, как только база покупателей вырастает до нескольких тысяч человек. Ручная выдача купонов через панель администратора — это десятки минут ежедневной рутины. Маркетолог должен зайти в Битрикс, найти всех именинников, создать купон, скопировать код и отправить письмо. На практике это либо не делается вовсе, либо делается нерегулярно — и маркетинговый инструмент теряет эффективность.

В этой статье я покажу законченное продакшн-решение, которое автоматизирует весь цикл: ежедневный cron обходит всех активных пользователей, находит именинников через 5 дней, создаёт одноразовый купон через DiscountCouponTable и отправляет письмо через почтовое событие Битрикс. Предусмотрена защита от повторной выдачи купона в одном году, корректная обработка 29 февраля и ручной режим для единичного выпуска по ID или email пользователя.

Как работает система: общая схема автоматической выдачи купонов

Весь процесс выглядит следующим образом:

cron (birthday_coupon_cron.php)
│
▼
sendCouponsInBatches()
│
├─── resolveDiscount() ←── DiscountTable::getList() по ID 125
│ │ или fallback по NAME / DISCOUNT_VALUE
│ ▼
│ [discount array]
│
├─── processBatch() ←── loadBirthdayUsersBatch() (lastUserId пагинация)
│ │
│ ├─ parseBirthday() → месяц + день из PERSONAL_BIRTHDAY
│ ├─ resolveNextBirthdayDate() → ближайшая дата ДР
│ ├─ DAYS_BEFORE_BIRTHDAY === 5 → пропуск всех остальных
│ ├─ hasCouponForYear() → дубль за год? пропуск
│ ├─ createCoupon() → DiscountCouponTable::add()
│ └─ sendCouponEmail() → CEvent::Send() + fallback mail()
│
└─── log('done', [...]) → /local/logs/birthday_coupon_sender.log

Система даёт три ключевые гарантии:

Не дублировать. Перед созданием купона метод hasCouponForYear() проверяет, не был ли уже выдан купон этому пользователю в текущем году. Проверка идёт по строке-маркеру в поле DESCRIPTION таблицы купонов — никаких дополнительных таблиц.
Не пропускать 29 февраля. Метод buildBirthdayDate() обрабатывает ситуацию, когда в невисокосный год нельзя создать дату 29 февраля: в таком случае купон выдаётся 28 февраля, и пользователь не остаётся без поздравления.
Не падать при отсутствии правила скидки. Метод resolveDiscount() имеет три уровня поиска: по фиксированному ID, затем по точному совпадению процента, затем по ключевым словам в имени. Если ничего не найдено — процесс завершается с информативным JSONL-сообщением в лог, но не бросает исключение.

Архитектура класса BirthdayCouponSender в 1С-Битрикс

Класс объявлен как final — это намеренное решение. Логика выдачи купонов — специализированная и сплочённая; наследование здесь создало бы только риски нарушить инварианты (например, переопределить hasCouponForYear() и сломать защиту от дублей). final явно сигнализирует коллегам и IDE: этот класс не предназначен для расширения.

Все методы статические, потому что класс не хранит изменяемого состояния между вызовами. Единственный исключение — статическое свойство $lastResolveDiscountError, которое накапливает диагностическое сообщение о причине неудачного поиска скидки и передаётся наружу через публичные методы. Это позволяет избежать исключений для штатной ситуации «скидка не настроена».

Публичный API класса минимален и осознан:

Метод	Что делает	Доступ
`sendCouponsInBatches(batchSize, maxBatches, sendEmail)`	Батчевый обход всех активных пользователей	публичный
`issueCouponForUser(userId, sendEmail)`	Ручной выпуск купона на одного пользователя	публичный
`ensureAgentRegistered()`	Регистрация CAgent (альтернатива cron)	публичный
`runAgent()`	Обёртка для вызова из CAgent	публичный
`sendCoupons()`	Алиас для `sendCouponsInBatches()` с параметрами по умолчанию	публичный
`resolveDiscount()`	Поиск правила скидки (три уровня)	приватный
`processBatch()`	Обработка одной пачки пользователей	приватный
`loadBirthdayUsersBatch()`	Запрос пользователей через UserTable	приватный
`hasCouponForYear()`	Проверка дубля через DESCRIPTION	приватный
`createCoupon()`	Создание записи в DiscountCouponTable	приватный
`sendCouponEmail()`	Отправка через CEvent::Send + fallback	приватный
`parseBirthday()`	Парсинг PERSONAL_BIRTHDAY в 7 форматах	приватный
`resolveNextBirthdayDate()`	Вычисление ближайшей даты ДР	приватный
`buildBirthdayDate()`	Создание DateTimeImmutable с учётом 29 фев	приватный
`buildCouponMarker()`	Генерация строки-маркера для DESCRIPTION	приватный
`log()`	JSONL-запись в файл	приватный

Шаг 1: подключение класса в init.php

Класс BirthdayCouponSender размещается в /local/php_interface/lib/Local/User/BirthdayCouponSender.php и подключается через /local/php_interface/init.php:

use Local\User\BirthdayCouponSender;
$birthdayCouponSenderPath = __DIR__. '/lib/Local/User/BirthdayCouponSender.php';
if (file_exists($birthdayCouponSenderPath)) {
require_once $birthdayCouponSenderPath;
}

Обратите внимание на file_exists перед require_once. В init.php ошибка подключения файла приводит к фатальной ошибке PHP и падению всего сайта — поэтому проверка существования файла обязательна. require_once вместо require гарантирует, что класс не будет подключён дважды при возможном повторном включении init.php.

Если вы используете автозагрузку через Composer или собственный PSR-4 загрузчик в Битрикс, эту секцию init.php можно пропустить — класс будет найден автоматически по пространству имён LocalUser.

Шаг 2: батчевый cron для автоматической рассылки купонов ко дню рождения

Основной сценарий запуска — ежедневный cron. Скрипт размещается в /local/cli/birthday_coupon_cron.php:

<?php
if (PHP_SAPI !== 'cli') {
http_response_code(403);
exit('Forbidden');
}
$_SERVER['DOCUMENT_ROOT'] = realpath( __DIR__. '/../..');
$DOCUMENT_ROOT = $_SERVER['DOCUMENT_ROOT'];
define('NO_KEEP_STATISTIC', true);
define('NOT_CHECK_PERMISSIONS', true);
define('BX_NO_ACCELERATOR_RESET', true);
require $_SERVER['DOCUMENT_ROOT'] . '/bitrix/modules/main/include/prolog_before.php';
use Local\User\BirthdayCouponSender;
$batchSize = BirthdayCouponSender::DEFAULT_BATCH_SIZE;
$maxBatches = 0;
$sendEmail = true;
foreach (array_slice($argv, 1) as $argument) {
if (preg_match('/^--batch=(\d+)$/', (string)$argument, $matches)) {
$batchSize = max(1, (int)$matches[1]);
continue;
}
if (preg_match('/^--max-batches=(\d+)$/', (string)$argument, $matches)) {
$maxBatches = max(0, (int)$matches[1]);
continue;
}
if ((string)$argument === '--no-email') {
$sendEmail = false;
}
}
$result = BirthdayCouponSender::sendCouponsInBatches($batchSize, $maxBatches, $sendEmail);
echo json_encode($result, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT) . PHP_EOL;
exit(($result['success'] ?? false) ? 0 : 1);

Почему PHP_SAPI !== 'cli'. Скрипт использует $argv, ручное задание DOCUMENT_ROOT и prolog_before.php с неполной инициализацией сессий. Запуск через браузер привёл бы к ошибкам и потенциально к несанкционированной выдаче купонов. Проверка PHP_SAPI на первой строке — стандартная практика для CLI-скриптов Битрикс.

Константы NO_KEEP_STATISTIC и BX_NO_ACCELERATOR_RESET отключают запись статистики посещений и сброс кэша акселератора. Без них cron-скрипт создаёт «мусорные» записи в статистике и может непреднамеренно инвалидировать кэши.

Флаги командной строки:

--batch=N — размер пачки пользователей (по умолчанию 500). Уменьшайте при ограничении памяти.
--max-batches=N — максимальное число итераций. По умолчанию 0 — обходить всех пользователей. Задайте 10–20, если cron ограничен по времени выполнения.
--no-email — создать купоны без отправки писем (полезно для тестирования).

Запись в crontab:

0 8 * * * /usr/bin/php /var/www/bitrix/local/cli/birthday_coupon_cron.php >> /var/log/birthday_coupon_cron.log 2>&1

Запуск в 8:00 каждый день. Путь к PHP и к документ-руту замените на актуальные для вашего сервера. Вывод JSON-результата сохраняется в отдельный системный лог — это удобно для мониторинга через tail -f.

Шаг 3: ручной выпуск купона для конкретного пользователя

Для поддержки и маркетинговых акций предусмотрен CLI-скрипт birthday_coupon_issue.php, который выдаёт купон на одного пользователя:

<?php
if (PHP_SAPI !== 'cli') {
http_response_code(403);
exit('Forbidden');
}
$_SERVER['DOCUMENT_ROOT'] = realpath( __DIR__. '/../..');
$DOCUMENT_ROOT = $_SERVER['DOCUMENT_ROOT'];
define('NO_KEEP_STATISTIC', true);
define('NOT_CHECK_PERMISSIONS', true);
define('BX_NO_ACCELERATOR_RESET', true);
require $_SERVER['DOCUMENT_ROOT'] . '/bitrix/modules/main/include/prolog_before.php';
use Bitrix\Main\UserTable;
use Local\User\BirthdayCouponSender;
$userId = 0;
$email = '';
$sendEmail = true;
foreach (array_slice($argv, 1) as $argument) {
if (preg_match('/^--user=(\d+)$/', (string)$argument, $matches)) {
$userId = (int)$matches[1];
continue;
}
if (preg_match('/^--email=(.+)$/', (string)$argument, $matches)) {
$email = trim((string)$matches[1]);
continue;
}
if ((string)$argument === '--no-email') {
$sendEmail = false;
}
}
if ($userId <= 0 && $email !== '') {
$user = UserTable::getList([
'select' => ['ID'],
'filter' => ['=EMAIL' => $email],
'limit' => 1,
])->fetch();
$userId = (int)($user['ID'] ?? 0);
}
if ($userId <= 0) {
echo json_encode([
'success' => false,
'message' => 'Передайте --user=ID или --email=user@example.com',
], JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT) . PHP_EOL;
exit(1);
}
$result = BirthdayCouponSender::issueCouponForUser($userId, $sendEmail);
echo json_encode($result, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT) . PHP_EOL;
exit(($result['success'] ?? false) ? 0 : 1);

Примеры запуска:

# По ID пользователя, с отправкой письма
php /var/www/bitrix/local/cli/birthday_coupon_issue.php --user=42
# По email, без отправки письма (только создать купон)
php /var/www/bitrix/local/cli/birthday_coupon_issue.php --email=user@example.com --no-email
# По ID без письма
php /var/www/bitrix/local/cli/birthday_coupon_issue.php --user=42 --no-email

Скрипт возвращает JSON с полями success, coupon, birthday_date, active_to, email_sent. При успехе код завершения — 0, при ошибке — 1, что позволяет использовать скрипт в Makefile или CI-пайплайнах.

Защита от дублей: маркер в поле DESCRIPTION купона

Главная проблема ежедневного cron — повторная выдача купона пользователю, которому он уже был выдан в этом году. Наивное решение — проверять наличие купона по USER_ID и диапазону дат — ненадёжно: если даты хранятся в разных форматах или купон был продлён вручную, фильтр может дать ложноотрицательный результат.

В этом решении используется строка-маркер, которая записывается в поле DESCRIPTION каждого созданного купона:

birthday_coupon:user=42;year=2026

Метод hasCouponForYear() выполняет точный поиск по этой строке:

private static function hasCouponForYear(int $userId, int $year): bool
{
$existingCoupon = DiscountCouponTable::getList([
'select' => ['ID'],
'filter' => [
'=USER_ID' => $userId,
'=DESCRIPTION' => self::buildCouponMarker($userId, $year),
],
'limit' => 1,
])->fetch();
return is_array($existingCoupon);
}
private static function buildCouponMarker(int $userId, int $year): string
{
return 'birthday_coupon:user=' . $userId . ';year=' . $year;
}

Почему поле DESCRIPTION, а не отдельная таблица? Таблица купонов в Битрикс уже хранит все необходимые данные. Добавлять отдельную таблицу для маркера — это усложнение без выгоды: нужно создавать таблицу, следить за её миграциями, удалять устаревшие записи. Поле DESCRIPTION доступно из коробки, индексируется через фильтр D7 ORM и не требует никакой дополнительной инфраструктуры. Строка birthday_coupon:user=N;year=YYYY достаточно уникальна, чтобы исключить ложные совпадения с другими купонами, где поле DESCRIPTION тоже может быть заполнено.

Поиск правила скидки: три уровня fallback в resolveDiscount

Метод resolveDiscount() реализует устойчивый поиск правила скидки с тремя уровнями:

Уровень 1 — фиксированный ID. Сначала ищется правило с ID = 125 (константа DISCOUNT_ID). Это самый быстрый путь: один запрос к БД. Если правило найдено, но неактивно или в нём отключены купоны (USE_COUPONS !== 'Y') — метод возвращает null с диагностическим сообщением.

Уровень 2 — точное совпадение параметров. Если правило с ID 125 не найдено на текущем сайте, происходит поиск всех активных правил с купонами, где DISCOUNT_TYPE = 'P' (процентная скидка) и DISCOUNT_VALUE = 5.0. Среди найденных выбирается лучший кандидат через pickBestDiscount().

Уровень 3 — поиск по ключевым словам. Если второй уровень не дал результата, поиск расширяется до всех активных правил с купонами. pickBestDiscount() присваивает каждому правилу очки: +100 за каждое ключевое слово из DISCOUNT_NAME_HINTS (день рождения, birthday, ДР и другие) в поле NAME или XML_ID, +20 за совпадение процента. Правило с максимальным счётом становится победителем.

private const DISCOUNT_NAME_HINTS = [
'день рождения',
'деньрождения',
'birthday',
'birth day',
'др',
];

Почему важен fallback? На боевых проектах правила скидок нередко пересоздаются после импорта или миграции между стендами — ID меняется. Трёхуровневый поиск позволяет системе найти нужное правило даже после таких операций, без ручного вмешательства в код.

Парсинг PERSONAL_BIRTHDAY и проблема 29 февраля в Битрикс

Поле PERSONAL_BIRTHDAY хранится в таблице b_user как строка и не имеет жёстко заданного формата. На реальных проектах встречаются все возможные варианты в зависимости от истории импортов и форм регистрации. Метод parseBirthday() последовательно перебирает семь форматов через DateTimeImmutable::createFromFormat:

$formats = [
'd.m.Y', // 15.03.1990 — основной формат Битрикс
'd.m.Y H:i:s', // 15.03.1990 00:00:00 — с временем
'd.m.y', // 15.03.90 — двузначный год
'Y-m-d', // 1990-03-15 — ISO 8601
'Y-m-d H:i:s', // 1990-03-15 00:00:00
'd/m/Y', // 15/03/1990
'm/d/Y', // 03/15/1990 — американский формат
];

Если ни один формат не подошёл, делается попытка через strtotime() — это последний шанс для нестандартных строк вида «15 марта 1990».

Метод возвращает только месяц и день — год рождения намеренно игнорируется, чтобы не зависеть от возможных ошибок ввода (пользователи нередко указывают год неверно).

Проблема 29 февраля. Если пользователь родился 29 февраля, то в невисокосный год эту дату создать невозможно. Метод buildBirthdayDate() обрабатывает этот случай явно:

private static function buildBirthdayDate(int $year, int $month, int $day): ?\DateTimeImmutable
{
if (checkdate($month, $day, $year)) {
return new \DateTimeImmutable(sprintf('%04d-%02d-%02d', $year, $month, $day));
}
if ($month === 2 && $day === 29) {
return new \DateTimeImmutable(sprintf('%04d-02-28', $year));
}
return null;
}

В невисокосный год именинники 29 февраля получат купон 28 февраля — это наиболее справедливое поведение. В следующий високосный год checkdate(2, 29, year) вернёт true, и купон выйдет в правильный день.

Почтовое событие USER_BIRTHDAY_COUPON: настройка в Битрикс

Для отправки письма используется почтовое событие Битрикс. Его нужно создать в разделе «Настройки → Почта → Типы почтовых событий» с именем USER_BIRTHDAY_COUPON и добавить шаблон письма.

Метод sendCouponEmail() передаёт в событие следующие поля:

Поле	Значение	Описание
`EMAIL_TO`	email пользователя	Адрес получателя
`USER_NAME`	Имя Фамилия или login	Имя для обращения
`COUPON`	Код купона	Строка вида `BDAY-A1B2C3`
`DISCOUNT_PERCENT`	`5`	Процент скидки
`BIRTHDAY_DATE`	`dd.mm.YYYY`	Дата дня рождения
`COUPON_ACTIVE_TO`	`dd.mm.YYYY`	Последний день действия купона
`DAYS_BEFORE_BIRTHDAY`	`5`	За сколько дней выдаётся купон

В шаблоне письма обращайтесь к этим полям через синтаксис #COUPON#, #USER_NAME#, #COUPON_ACTIVE_TO#.

Fallback через mail(). Если CEvent::Send() вернула false (событие не настроено или почтовый транспорт недоступен), метод sendFallbackMail() отправляет упрощённое текстовое письмо напрямую через mail(). Это обеспечивает минимальную отказоустойчивость. В production рекомендуется всегда настраивать событие и шаблон — fallback предназначен только для экстренных ситуаций.

Логирование в JSONL-формате

Каждое действие системы записывается в файл /local/logs/birthday_coupon_sender.log в формате JSONL (JSON Lines — одна запись в строку):

{"ts":"2026-06-27T08:00:01+03:00","level":"sent","context":{"user_id":42,"email":"user@example.com","coupon":"BDAY-X1Y2Z3","birthday_date":"01.07.2026","active_to":"06.07.2026"}}
{"ts":"2026-06-27T08:00:01+03:00","level":"skip","context":{"reason":"birthday_coupon already exists for this year","user_id":15}}
{"ts":"2026-06-27T08:00:05+03:00","level":"done","context":{"success":true,"processed_users":1250,"matched_users":3,"sent_coupons":2,"batches":3,"discount_id":125}}

Уровни записей:

sent — купон создан и письмо отправлено
coupon_created_without_email — купон создан, письмо не отправлялось (флаг --no-email)
skip — пользователь пропущен (нет ДР, дубль за год, неактивен, не найдена скидка)
error — ошибка при создании купона или другая исключительная ситуация
done — итоговая запись после завершения всей батчевой обработки

Для просмотра последних ошибок:

grep '"level":"error"' /var/www/bitrix/local/logs/birthday_coupon_sender.log | tail -20

Для подсчёта купонов за день:

grep '"level":"sent"' /var/www/bitrix/local/logs/birthday_coupon_sender.log | grep "2026-06-27" | wc -l

Типичные ошибки при внедрении автоматических купонов в Битрикс

Купоны создаются повторно каждый год

Причина: при создании купона не записывается маркер в поле DESCRIPTION, или hasCouponForYear() ищет не по точному совпадению строки, а по LIKE.

Решение: убедитесь, что createCoupon() передаёт buildCouponMarker($userId, $year) в ключ DESCRIPTION при вызове DiscountCouponTable::add(). Фильтр '=DESCRIPTION' в hasCouponForYear() должен использовать оператор точного равенства (префикс =).

Скрипт не находит правило скидки

Причина: правило скидки создано не для текущего SITE_ID, или поле USE_COUPONS установлено в N, или правило неактивно.

Решение: проверьте лог — там будет конкретная причина отказа. Войдите в администраторскую панель, откройте правило с ID 125, убедитесь, что оно активно, привязано к нужному сайту и в нём включены купоны. Если правило было пересоздано с другим ID — обновите константу DISCOUNT_ID в классе.

Письмо не отправляется

Причина: почтовое событие USER_BIRTHDAY_COUPON не создано или для него нет активного шаблона.

Решение: создайте тип почтового события в разделе «Настройки → Почта → Типы почтовых событий» и добавьте шаблон с перечисленными выше полями. Проверьте раздел «Настройки → Почта → Очередь почтовых сообщений» — письмо может быть в очереди, а не отправлено. Также убедитесь, что настроен почтовый транспорт (SMTP или sendmail).

Cron падает с ошибкой «prolog_before.php: no such file»

Причина: переменная DOCUMENT_ROOT задаётся через realpath( __DIR__. '/../..'), но скрипт запускается из другой рабочей директории, и относительный путь не разрешается корректно.

Решение: используйте абсолютный путь к скрипту в crontab: php /var/www/html/local/cli/birthday_coupon_cron.php. Проверьте, что realpath( __DIR__. '/../..') на вашем сервере возвращает корень сайта, а не false. В случае проблем замените на явный абсолютный путь: $_SERVER['DOCUMENT_ROOT'] = '/var/www/html';.

Чеклист проверки после внедрения купонов ко дню рождения

Перед переводом в production пройдитесь по этим шести пунктам:

Правило скидки с ID 125 активно, привязано к нужному сайту, купоны включены (USE_COUPONS = Y)
Почтовое событие USER_BIRTHDAY_COUPON создано, шаблон активен, все поля в шаблоне написаны верно
Тестовый запуск --no-email завершился без ошибок: grep '"level":"error"' /local/logs/birthday_coupon_sender.log — пусто
Ручной выпуск birthday_coupon_issue.php --user=YOUR_TEST_ID создал купон и прислал письмо
В таблице купонов (Битрикс → Маркетинг → Купоны) появился купон с правильным DESCRIPTION, датами и привязкой к нужному правилу
Cron-задача зарегистрирована, проверьте crontab -l и убедитесь, что путь к PHP и к скрипту верны

Заключение: масштабируемость и дальнейшее развитие

Решение работает стабильно на базах с десятками тысяч пользователей: пагинация через lastUserId вместо OFFSET исключает деградацию производительности при росте таблицы. Размер пачки и лимит батчей позволяют точно подстроить потребление ресурсов под ограничения вашего хостинга. JSONL-лог совместим с Loki, Elastic и любыми другими агрегаторами логов без дополнительной обработки.

Возможные направления развития: расширить процент скидки в зависимости от сегмента покупателя, добавить SMS-канал через CEvent или сторонний провайдер, ввести настройку DAYS_BEFORE_BIRTHDAY через публичный конструктор вместо константы. Практическое руководство по скидкам и купонам (в т.ч. DiscountCouponTable::add()) — в официальной документации Битрикс: Комплекты, наборы и скидки. Полная справка по CEvent::Send — в разделе справочника по модулю main.

How Protobuf Cut Our Video Metadata Payloads by 71% Across Services

ahmet gedik — Tue, 14 Jul 2026 13:00:02 +0000

Our trend-detection pipeline was choking on JSON. Every two hours a Cloudflare Worker pulls a batch of trending videos, hands them to a Python enrichment job, which writes to a PHP 8.4 service backed by SQLite in WAL mode, which then fans the same records out to a ranking service. The video metadata record itself is small — an ID, a few timestamps, view counts, region codes, a language tag, some engagement ratios. But we were moving millions of them per day, and the JSON representation of that tiny record was costing us more than the data was worth.

The specific number that made me stop and rewrite the serialization layer: our average metadata record was 412 bytes as JSON and 119 bytes as Protobuf. That is a 71% reduction on the wire, before any gzip. At ViralVidVault we track viral video trends across the European market, and cross-region fan-out means the same record hops between three or four services before it lands. Multiply 293 saved bytes by tens of millions of records and the egress bill, the Worker CPU time, and the SQLite write amplification all move in the same direction.

This is a walkthrough of how we did it — the schema design, the PHP encoding path, the Python and Go interop, and the specific traps around evolving a schema without breaking a running fleet.

Why JSON Was the Wrong Default

JSON is the right default for a lot of things. Human-readable, self-describing, debuggable with curl. But for a fixed-shape record that machines produce and consume at high volume, every one of those virtues becomes a cost:

Field names ship on every record. "view_count": is 13 bytes repeated on every single message. Protobuf sends a field number — one byte for the common cases.
Numbers are text. 1483920 is 7 bytes as a JSON string of digits. As a protobuf varint it is 3 bytes, and a count of 200 is a single byte.
No native integer/enum types. Region codes like "DE", "FR", "GB" are strings in JSON. As a protobuf enum they are one byte each.
Parsing is expensive. JSON parsing means scanning for delimiters and allocating strings. Protobuf parsing is length-prefixed and reads fields by number.

The self-describing property is genuinely useful during development, so we kept a JSON debug endpoint. But the transport between services had no business paying that tax.

Designing the Schema

Here is the actual .proto definition we run, trimmed to the fields that matter for this article. The design choices are the interesting part, not the syntax.

syntax = "proto3";

package viralvidvault.metadata.v1;

enum Region {
  REGION_UNSPECIFIED = 0;
  REGION_DE = 1;
  REGION_FR = 2;
  REGION_GB = 3;
  REGION_ES = 4;
  REGION_IT = 5;
  REGION_NL = 6;
  REGION_PL = 7;
}

enum Platform {
  PLATFORM_UNSPECIFIED = 0;
  PLATFORM_YOUTUBE = 1;
  PLATFORM_TIKTOK = 2;
  PLATFORM_INSTAGRAM = 3;
}

message VideoMetadata {
  string video_id = 1;
  Platform platform = 2;
  Region region = 3;

  uint64 view_count = 4;
  uint32 like_count = 5;
  uint32 comment_count = 6;

  // Unix seconds. int64, not a timestamp string.
  int64 published_at = 7;
  int64 first_seen_at = 8;

  string language = 9;          // BCP-47, e.g. "de-DE"
  float velocity_score = 10;    // views/hour, our trend signal

  repeated string tag = 11;

  // Reserved for GDPR: never reuse these numbers.
  reserved 12, 13;
  reserved "uploader_email", "uploader_ip";
}

A few decisions worth calling out:

Field numbers 1–4 are the hot path. Protobuf encodes the field number and wire type into a single "tag" byte for field numbers 1–15. Numbers 16 and up need two bytes for the tag. So the fields present on every record — video_id, platform, region, view_count — got the low numbers. Rarely-populated fields can live above 15.
uint64 for view_count, uint32 for likes. A viral video can cross a billion views, so view_count needs 64 bits. Likes and comments realistically fit in 32. Right-sizing the integer type keeps the varint short for the common small values.
Enums for region and platform. This is where a big chunk of the savings lives. "de-DE" region strings became one-byte enums.
reserved for the fields we deliberately do not carry. We had two personally-identifying fields early in the design — uploader email and IP. GDPR data-minimization says we do not move those between services, so we removed them and reserved the numbers. Reserving means no future engineer can accidentally reintroduce a PII field on the same wire number that an old client might misread. That is a compliance guarantee enforced by the compiler.

Encoding in PHP 8.4

Our write service is PHP. The Google protobuf runtime for PHP is available as a C extension (fast) or a pure-PHP package (portable). On LiteSpeed with the extension installed, encoding is cheap. Here is the encode/decode path, using classes generated by protoc --php_out.

<?php
declare(strict_types=1);

use Viralvidvault\Metadata\V1\VideoMetadata;
use Viralvidvault\Metadata\V1\Region;
use Viralvidvault\Metadata\V1\Platform;

final class MetadataCodec
{
    /**
     * Build a protobuf message from an ingest row and return the
     * binary wire bytes ready to store or forward.
     */
    public function encode(array $row): string
    {
        $msg = new VideoMetadata();
        $msg->setVideoId($row['video_id']);
        $msg->setPlatform($this->platformEnum($row['platform']));
        $msg->setRegion($this->regionEnum($row['region']));
        $msg->setViewCount((int) $row['view_count']);
        $msg->setLikeCount((int) $row['like_count']);
        $msg->setCommentCount((int) $row['comment_count']);
        $msg->setPublishedAt((int) $row['published_at']);
        $msg->setFirstSeenAt($row['first_seen_at'] ?? time());
        $msg->setLanguage($row['language'] ?? '');
        $msg->setVelocityScore((float) ($row['velocity_score'] ?? 0.0));

        if (!empty($row['tags'])) {
            $msg->setTag($row['tags']); // repeated string
        }

        return $msg->serializeToString();
    }

    public function decode(string $bytes): VideoMetadata
    {
        $msg = new VideoMetadata();
        $msg->mergeFromString($bytes); // throws on malformed input
        return $msg;
    }

    private function regionEnum(string $code): int
    {
        return match (strtoupper($code)) {
            'DE' => Region::REGION_DE,
            'FR' => Region::REGION_FR,
            'GB' => Region::REGION_GB,
            'ES' => Region::REGION_ES,
            'IT' => Region::REGION_IT,
            'NL' => Region::REGION_NL,
            'PL' => Region::REGION_PL,
            default => Region::REGION_UNSPECIFIED,
        };
    }

    private function platformEnum(string $name): int
    {
        return match (strtolower($name)) {
            'youtube'   => Platform::PLATFORM_YOUTUBE,
            'tiktok'    => Platform::PLATFORM_TIKTOK,
            'instagram' => Platform::PLATFORM_INSTAGRAM,
            default     => Platform::PLATFORM_UNSPECIFIED,
        };
    }
}

The match expressions bridging our string ingest data to enums are the boring but important glue. Do this mapping in exactly one place. If you scatter string-to-enum conversion across the codebase you will eventually store REGION_UNSPECIFIED for a typo'd "de" and spend an afternoon finding out why a whole region dropped off your trend charts.

Storing Blobs in SQLite Without Losing Queryability

We run SQLite in WAL mode. The naive move is to store the whole protobuf blob in a BLOB column and call it done. That works for the transport payload, but it destroys your ability to run WHERE region = 'DE' ORDER BY velocity_score. Protobuf is opaque to SQL.

Our compromise: store the blob for fast fan-out, and denormalize the handful of columns you actually query alongside it. The blob is the source of truth for the full record; the columns are a query index.

<?php
declare(strict_types=1);

final class MetadataStore
{
    public function __construct(private \PDO $db) {}

    public function upsert(VideoMetadata $msg): void
    {
        // Denormalize only the query dimensions. The blob holds
        // the complete record for downstream services.
        $sql = <<<SQL
            INSERT INTO video_metadata
                (video_id, region, platform, velocity_score, published_at, payload)
            VALUES
                (:vid, :region, :platform, :velocity, :published, :payload)
            ON CONFLICT(video_id) DO UPDATE SET
                velocity_score = excluded.velocity_score,
                payload        = excluded.payload
            SQL;

        $stmt = $this->db->prepare($sql);
        $stmt->execute([
            ':vid'       => $msg->getVideoId(),
            ':region'    => $msg->getRegion(),      // stored as int enum
            ':platform'  => $msg->getPlatform(),
            ':velocity'  => $msg->getVelocityScore(),
            ':published' => $msg->getPublishedAt(),
            ':payload'   => $msg->serializeToString(), // BLOB column
        ]);
    }
}

The payoff of WAL mode here is that the enrichment writers and the read-heavy ranking queries do not block each other. The protobuf blob keeps the row narrow — storing a 119-byte blob instead of a 412-byte JSON text column measurably reduced page churn in our write-heavy tables, which matters when your database is a single file and every writer contends on it.

One caution: SQLite BLOBs are binary-safe, but if you ever pipe a row through something that assumes UTF-8 text (a careless CSV export, some ORMs), it will mangle the bytes. Keep the payload column strictly typed as BLOB and never SELECT payload into a text-handling code path.

Cross-Service Interop: Python Enrichment, Go Ranking

The whole point of protobuf is that the same schema generates types in every language, and the wire bytes are identical. Our enrichment step is Python. It reads the raw ingest, computes the velocity score, and emits protobuf bytes that the PHP service stores verbatim.

import time
from metadata.v1 import metadata_pb2 as pb


def enrich(raw: dict) -> bytes:
    msg = pb.VideoMetadata()
    msg.video_id = raw["video_id"]
    msg.platform = pb.PLATFORM_YOUTUBE
    msg.region = pb.Region.Value(f"REGION_{raw['region'].upper()}")
    msg.view_count = int(raw["view_count"])
    msg.like_count = int(raw.get("like_count", 0))
    msg.published_at = int(raw["published_at"])
    msg.first_seen_at = int(time.time())
    msg.language = raw.get("language", "")

    # Trend signal: views accrued per hour since we first saw it.
    age_hours = max((time.time() - msg.first_seen_at) / 3600.0, 1.0)
    msg.velocity_score = msg.view_count / age_hours

    msg.tag.extend(raw.get("tags", []))
    return msg.SerializeToString()


def parse(blob: bytes) -> pb.VideoMetadata:
    msg = pb.VideoMetadata()
    msg.ParseFromString(blob)  # raises DecodeError on garbage
    return msg

The ranking service is Go, chosen for its concurrency when scoring large batches. It reads the same blobs and never once has to agree with PHP or Python on a JSON field name — the schema is the contract.

package ranking

import (
    "sort"

    "google.golang.org/protobuf/proto"
    pb "viralvidvault/gen/metadata/v1"
)

// DecodeBatch turns stored blobs into typed messages, skipping
// any record that fails to parse rather than aborting the batch.
func DecodeBatch(blobs [][]byte) []*pb.VideoMetadata {
    out := make([]*pb.VideoMetadata, 0, len(blobs))
    for _, b := range blobs {
        var m pb.VideoMetadata
        if err := proto.Unmarshal(b, &m); err != nil {
            continue // one bad record must not sink the batch
        }
        out = append(out, &m)
    }
    return out
}

// TopByVelocity returns the n hottest videos for a region.
func TopByVelocity(msgs []*pb.VideoMetadata, region pb.Region, n int) []*pb.VideoMetadata {
    filtered := msgs[:0:0]
    for _, m := range msgs {
        if m.GetRegion() == region {
            filtered = append(filtered, m)
        }
    }
    sort.Slice(filtered, func(i, j int) bool {
        return filtered[i].GetVelocityScore() > filtered[j].GetVelocityScore()
    })
    if len(filtered) > n {
        filtered = filtered[:n]
    }
    return filtered
}

Three languages, one schema, byte-identical wire format. When we add a field, we regenerate the bindings for all three from the single .proto, and old services keep working because of how proto3 handles unknown fields — which is the next thing to get right.

Schema Evolution Without Breaking the Fleet

This is where teams get burned. You cannot deploy PHP, Python, and Go simultaneously across a running system. There is always a window where an old encoder talks to a new decoder or vice versa. Protobuf handles this gracefully only if you follow the rules:

Never change a field's number. The number is the identity. The name is cosmetic. Rename freely; renumber never.
Never change a field's type in an incompatible way. Going uint32 → uint64 is safe (both varints). Going int32 → string is a corruption bug waiting to happen.
New fields must be optional in spirit. In proto3 every scalar has a default. A new field added by the encoder is simply ignored by an old decoder — it lands in the unknown-fields set and is preserved on re-serialization. That last part matters: an old service that reads and re-writes a record will carry forward fields it does not understand, so it will not strip data added by a newer service.
Deleting a field means reserving its number. As shown with the GDPR fields above, reserved 12, 13; prevents reuse. Reusing a deleted number is the single most common way to silently corrupt data across a version skew.
Roll out decoders before encoders. Deploy the services that read the new field first. Once every reader understands field 14, start writing it. Reverse that order and your new field vanishes into unknown-fields on services that have not caught up.

We encode a schema version into a Cloudflare Worker header (X-Meta-Schema: v1) so we can spot version skew in logs during a rollout, but the wire compatibility does not depend on it — that header is an observability aid, not a gate.

What We Measured Afterward

Numbers from our own pipeline after the migration, averaged over a week of production traffic:

Payload size: 412 B → 119 B per record (−71%), before compression. After gzip the gap narrows but protobuf still wins by ~40% because there is less redundant text for the compressor to exploit.
Encode/decode CPU: the PHP protobuf C extension encoded roughly 3× faster than json_encode on our record shape, and decoding in Go was not even a measurable line item.
SQLite write throughput: narrower rows meant fewer page splits; our two-hourly ingest batch finished noticeably quicker under WAL, with less checkpoint pressure.
Cloudflare Worker egress: the fan-out between services carries a fraction of the bytes, which on a per-request-billed edge platform is a direct cost line.

The non-numeric win is the one I value most: the schema is now a real contract. When someone adds a field, the compiler forces every service to agree on its number and type. The class of bug where PHP sends viewCount and Go expects view_count is simply gone.

When Not to Reach for Protobuf

To be fair to JSON, protobuf is the wrong choice when:

The payload is consumed by a browser or a public API where human-readability and ubiquitous tooling matter more than bytes.
The data shape is genuinely dynamic — arbitrary user-defined keys — where a schema fights you instead of helping.
The volume is low enough that the bytes never show up on any bill or latency graph. Do not add a code-generation step to move a thousand records a day.

For internal, high-volume, fixed-shape machine-to-machine traffic — which is exactly what video metadata fan-out is — the trade lands firmly on protobuf's side.

Conclusion

The migration came down to a few concrete moves: put the hot-path fields on numbers 1–15, turn repeated string codes into enums, right-size the integer types, store the binary blob in SQLite alongside a thin set of denormalized query columns, and treat field numbers as immutable while reserving anything you delete. The 71% payload reduction was the headline, but the durable win was turning a loose JSON convention into a compiler-enforced contract shared by PHP, Python, and Go. If your services are shipping the same field names over and over on every record, that repetition is a bill you are paying — and protobuf is a well-worn way to stop paying it.

A Late Introduction to VonCMS

Mohd Hakim Johari — Tue, 14 Jul 2026 12:41:41 +0000

VonCMS has actually been open source for a while, and the current release is version 1.25.6. I already shared it on Reddit and a few other places, but somehow forgot to post about it here.

VonCMS is a publishing CMS built with React, Vite, PHP, and MySQL.

I started building it because I wanted something simple for publishers and bloggers who mainly want to write, publish, and manage their content without dealing with too many settings.

It is also designed to work on normal shared hosting, so running it does not require a VPS, a headless setup, or a complicated deployment process.

Some of the current features include:

Article and page publishing
Shared hosting support
Built-in SEO features
Media management
User and role management
React frontend
PHP and MySQL backend
Web-based installation

I have been working on VonCMS for almost a year, and I also use it on a live publishing website. That helps me find real issues and improve things based on actual usage.

That said, VonCMS is still a work in progress.

It may not be suitable for everyone yet, and there is still a long way to go. Some parts need better documentation, more testing, cleaner code, and more feedback from different hosting environments and use cases.

This is also one of the reasons I decided to make it open source.

I would be happy to have more people involved in the project. Contributions do not have to be large. You can help by:

Reporting bugs
Testing it on different hosting providers
Improving the documentation
Suggesting useful features
Reviewing the code
Submitting pull requests
Helping improve the user interface

The project is available under the GNU GPL v3 license.

Current version: 1.25.6

GitHub:

https://github.com/Vondereich/VonCMS

VonCMS is not perfect, but it is improving with every release. Hopefully, with more feedback and contributions, it can grow into something useful for more publishers and developers.

Manticore Speaks MySQL - So I Made It a Laravel Database Driver Instead of a Scout Engine

Renato Maldonado — Tue, 14 Jul 2026 12:21:59 +0000

The problem

I've been working with Manticore Search for about two years at EricaPRO, building the search layer for two financial data platforms. For the past year, that work has been a Laravel API.

Manticore was never the problem. It's fast and it's stable. The problem was the gap between Manticore and Laravel.

I had already built a package for this — laravel-manticore-search, a fluent builder over Manticore's HTTP/JSON API. It works, and it's still in use. But it's a client wrapper. Every feature had to be implemented manually. Every new filter or facet meant more custom architecture around the client, and none of the things Laravel gives you for free — models, migrations, pagination, casts — applied to it.

Scout doesn't close that gap either. Scout gives you search(). No full query builder, no migrations for your indexes, sync is your problem. That's not a criticism — Scout abstracts over engines with completely different APIs, so it exposes the lowest common denominator. It just wasn't what I needed.

What I needed was simple to describe and annoying to not have: something plug and play. Something Laravel way. Point Eloquent at Manticore and use Eloquent.

The insight

Manticore speaks the MySQL wire protocol. Out of the box, port 9306. You can connect to it with any MySQL client and run SQL.

I had been using that port for two years without thinking about what it meant for Laravel.

Because here's the thing: all of Eloquent — models, query builder, migrations, pagination, chunking — sits on top of a Connection and a Grammar. The grammar compiles builder calls into SQL for a specific dialect. That's the entire mechanism behind Laravel supporting MySQL, Postgres, SQLite and SQL Server with one codebase.

So the real question was never "how do I re-implement Eloquent on top of Manticore's client?" It was "how thin can a Manticore grammar be?"

If Manticore accepts MySQL-protocol connections and mostly-MySQL SQL, then a Laravel database driver — a custom connection plus a grammar handling Manticore's dialect quirks — gets you the whole framework. Not a re-implementation. Inheritance.

That's laravel-manticore-eloquent: Manticore registered as a first-class Laravel database connection.

What it looks like

composer require renatomaldonado/laravel-manticore-eloquent
php artisan vendor:publish --provider="ManticoreEloquent\ManticoreEloquentServiceProvider" --tag=config

A model that lives entirely in Manticore extends ManticoreModel. After that, it's just Eloquent:

use ManticoreEloquent\Eloquent\ManticoreModel;

class Article extends ManticoreModel
{
    protected $table = 'articles_rt';
    protected $guarded = [];
    protected $casts = ['published' => 'boolean', 'created_at' => 'datetime'];
}

Article::where('published', true)
    ->match('laravel')          // MATCH() — Manticore full-text
    ->orderBy('created_at', 'desc')
    ->paginate();

Article::find(10);
Article::create(['title' => 'Hello', 'body' => '…', 'published' => true]);

If your model already lives in a relational database and you only want to query it through Manticore, there's a trait instead:

class Company extends Model
{
    use HasManticore;

    public function searchableAs(): string
    {
        return 'companies_rt';
    }
}

Company::query()->where('id', 1)->first();          // relational DB
Company::manticore()->match('fintech')->paginate(); // Manticore

Migrations, but for a search engine

This is the part I missed the most before. Manticore real-time indexes are created with standard Laravel migrations — the schema grammar maps Laravel column types to Manticore types and adds the Manticore-only ones:

Schema::connection('manticore')->create('articles_rt', function (Blueprint $table) {
    $table->text('body');                        // full-text indexed
    $table->string('title');
    $table->integer('views');
    $table->boolean('published');
    $table->timestamp('created_at');
    $table->mva('tag_ids');                      // multi-value attribute
    $table->floatVector('embedding', dims: 384); // vector column for KNN

    $table->minInfixLen(2);                      // infix search
    $table->morphology('stem_en');
});

Your search schema is versioned next to the rest of your schema. php artisan migrate builds it.

Vector search with Eloquent

Manticore supports KNN natively, and since the driver just compiles SQL, it plugs straight into the builder:

Article::manticore()
    ->knn('embedding', $queryVector, k: 10)
    ->selectRaw('*, knn_dist() as distance')
    ->get();

Embedding search with paginate(), casts and everything else Eloquent gives you. No separate vector database in the stack.

The other Manticore-specific helpers cover the rest of the dialect: option() for ranker tuning, facet(), highlight(), replace() for full-document rewrites, and maxMatches() — raised automatically when you paginate past Manticore's default 1,000-row cap, so deep pagination just works.

In production

This isn't a weekend experiment. The driver runs in production in the API I maintain at EricaPRO — the first consumer is a compliance screening service that matches entity and person names against Manticore indexes before calling an external API.

Real usage looks like this:

EntityStakeholderPerson::match($match, '(name)')
    ->when(filled($country), fn ($q) => $q->where('EntityCountryISO', $country))
    ->limit(20)
    ->option('ranker', 'proximity_bm25')
    ->get();

The $match there is a quorum expression built per query — "john doe smith"/2 — so recall stays high on partial names. Different rankers per use case (bm25 for entities, proximity_bm25 for persons), all through the same builder.

Two patterns that fell out of production and might save you time:

A base model for your conventions. Our indexes are prefixed per project and suffixed per environment, and the underlying index is latin1, so a base model handles both once:

class BaseManticoreModel extends ManticoreModel
{
    public function __construct(array $attributes = [])
    {
        $prefix = config('app.project_prefix');
        $suffix = app()->isProduction() ? '' : 'test';
        $this->table = $prefix.$this->table.$suffix;

        parent::__construct($attributes);
    }

    public function getAttributeValue($key)
    {
        $value = parent::getAttributeValue($key);

        if (is_string($value) && ! mb_check_encoding($value, 'UTF-8')) {
            return mb_convert_encoding($value, 'UTF-8', 'ISO-8859-1');
        }

        return $value;
    }
}

Every concrete model is then two lines. That's the point of having real Eloquent models instead of a search client: framework patterns like this just apply.

The parts that don't map cleanly

A search engine wearing a MySQL costume is still a search engine. The seams:

No transactions, no row locks. Manticore doesn't have them. lockForUpdate() is a no-op. Don't rely on DB::transaction() semantics on this connection.
UPDATE vs REPLACE. Manticore's UPDATE only changes attribute (non-text) columns in place. Rewriting a full document — including text fields — needs REPLACE INTO, so the builder exposes an explicit replace() instead of pretending update() always works.
id is special. Manticore owns document ids. Migrations skip any id column you declare.
JOINs are limited. Cross-index JOINs in Manticore are restricted. Denormalize your indexes. Eager-loading relations that live on your relational connection works as usual.
Emulated prepares. Manticore's server-side prepared statement support is limited, so the driver forces PDO's emulated prepares — bindings, including MATCH(?), are interpolated client-side.

None of these are dealbreakers for search workloads. They're the cost of the approach, and they're documented so nobody discovers them in production.

Try it

PHP 8.1+, Laravel 10/11/12, 95% test coverage enforced in CI. The design notes are in docs/ARCHITECTURE.md if you want the exact SQL differences the grammar handles.

Repo: https://github.com/Renato27/laravel-manticore-eloquent

If you run it against a real workload, I want to hear what breaks. I'm especially interested in edge cases in the grammar — builder combinations that compile to SQL Manticore rejects. Issues and PRs are open.

Getting the public IP in PHP — no dependencies, no API key

Vix — Tue, 14 Jul 2026 09:46:36 +0000

Getting the public IP in PHP — no dependencies, no API key

PHP is still one of the most widely deployed server-side languages, running a significant share of the web's backend code. If you're building a PHP application that needs the public IP address — for geolocation, DDNS, diagnostics, or country detection — this article covers the common patterns using IPPubblico.org: free, no key, HTTPS, JSON and plain text endpoints.

Use case 1 — Your server's own public IP (one-liner)

The simplest case: a PHP script that needs to know its own public IP.

<?php
$ip = trim(file_get_contents('https://ipv4.ippubblico.org/'));
echo $ip; // 203.0.113.42

file_get_contents works if allow_url_fopen is enabled (it is by default on most servers). If not, use cURL (see below).

Use case 2 — With cURL (recommended for production)

file_get_contents has no timeout control and minimal error handling. For production code, cURL is better:

<?php
function getPublicIP(): ?string {
    $ch = curl_init('https://ipv4.ippubblico.org/');
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 5,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_SSL_VERIFYPEER => true,
    ]);
    $response = curl_exec($ch);
    $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);

    if ($response === false || $httpCode !== 200) {
        return null;
    }
    return trim($response);
}

$ip = getPublicIP();
echo $ip ?? 'Unavailable';

Use case 3 — Full geolocation data

When you need country, city, ISP and timezone in addition to the IP:

<?php
function getIPInfo(?string $ip = null): ?array {
    $url = 'https://ippubblico.org/?api=1';
    if ($ip !== null) {
        $url .= '&ip=' . urlencode($ip);
    }

    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 5,
        CURLOPT_SSL_VERIFYPEER => true,
    ]);
    $response = curl_exec($ch);
    $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);

    if ($response === false || $httpCode !== 200) {
        return null;
    }

    $data = json_decode($response, true);
    if (!$data || $data['status'] !== 'ok') {
        return null;
    }
    return $data;
}

$info = getIPInfo();
if ($info) {
    echo 'IP:       ' . $info['ip'] . PHP_EOL;
    echo 'Country:  ' . $info['geo']['country'] . PHP_EOL;
    echo 'City:     ' . $info['geo']['city'] . PHP_EOL;
    echo 'ISP:      ' . $info['isp'] . PHP_EOL;
    echo 'Timezone: ' . $info['timezone'] . PHP_EOL;
}

Use case 4 — Multilingual response

One of IPPubblico's distinct features: city, region and country names in the user's language. Add ?lang=XX:

<?php
function getIPInfoLocalized(string $lang = 'en'): ?array {
    $url = 'https://ippubblico.org/?api=1&lang=' . urlencode($lang);

    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 5,
        CURLOPT_SSL_VERIFYPEER => true,
    ]);
    $response = curl_exec($ch);
    curl_close($ch);

    $data = json_decode($response, true);
    return ($data && $data['status'] === 'ok') ? $data : null;
}

// Japanese response
$info = getIPInfoLocalized('ja');
echo $info['geo']['country']; // 日本
echo $info['geo']['city'];    // 東京

// Portuguese response
$info = getIPInfoLocalized('pt');
echo $info['geo']['country']; // Japão
echo $info['geo']['city'];    // Tóquio

Supported localized languages: de, es, fr, ja, pt, ru, zh.

Use case 5 — Client IP geolocation in a web application

Detecting where your visitor is connecting from — the most common PHP web use case:

<?php
function getClientIP(): string {
    // Handle proxies and load balancers
    $headers = [
        'HTTP_CF_CONNECTING_IP',    // Cloudflare
        'HTTP_X_FORWARDED_FOR',     // Standard proxy header
        'HTTP_X_REAL_IP',           // Nginx
        'HTTP_X_FORWARDED',
        'REMOTE_ADDR',              // Direct connection fallback
    ];

    foreach ($headers as $header) {
        if (!empty($_SERVER[$header])) {
            $ip = trim(explode(',', $_SERVER[$header])[0]);
            if (filter_var($ip, FILTER_VALIDATE_IP, FILTER_FLAG_NO_PRIV_RANGE | FILTER_FLAG_NO_RES_RANGE)) {
                return $ip;
            }
        }
    }
    return $_SERVER['REMOTE_ADDR'] ?? '0.0.0.0';
}

function getClientIPInfo(): ?array {
    $clientIP = getClientIP();
    return getIPInfo($clientIP);
}

// Usage in a web request
$info = getClientIPInfo();
$country = $info['geo']['country_code'] ?? 'US'; // fallback to US

Use case 6 — Country-based content in Laravel

A Laravel middleware that attaches country information to every request:

<?php
// app/Http/Middleware/DetectCountry.php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Cache;

class DetectCountry
{
    public function handle(Request $request, Closure $next)
    {
        $ip = $request->ip();

        // Cache by IP for 1 hour to avoid repeated API calls
        $countryCode = Cache::remember("country_{$ip}", 3600, function () use ($ip) {
            return $this->detectCountry($ip);
        });

        $request->attributes->set('country_code', $countryCode);

        return $next($request);
    }

    private function detectCountry(string $ip): string
    {
        $ch = curl_init("https://ippubblico.org/?api=1&ip=" . urlencode($ip));
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 3,
            CURLOPT_SSL_VERIFYPEER => true,
        ]);
        $response = curl_exec($ch);
        curl_close($ch);

        $data = json_decode($response, true);
        return $data['geo']['country_code'] ?? 'US';
    }
}

protected $middlewareGroups = [
    'web' => [
        // ...
        \App\Http\Middleware\DetectCountry::class,
    ],
];

Use in any controller:

public function index(Request $request)
{
    $country = $request->attributes->get('country_code', 'US');

    return view('home', [
        'currency' => $this->getCurrencyForCountry($country),
        'country'  => $country,
    ]);
}

Use case 7 — WordPress plugin snippet

Detecting country in WordPress without a plugin or paid API:

<?php
function ipp_get_visitor_country(): string {
    $ip = $_SERVER['REMOTE_ADDR'] ?? '';

    // Check transient cache first (24 hours)
    $cacheKey = 'ipp_country_' . md5($ip);
    $cached    = get_transient($cacheKey);
    if ($cached !== false) {
        return $cached;
    }

    $response = wp_remote_get(
        'https://ippubblico.org/?api=1&ip=' . urlencode($ip),
        ['timeout' => 5, 'sslverify' => true]
    );

    if (is_wp_error($response)) {
        return 'US'; // fallback
    }

    $data = json_decode(wp_remote_retrieve_body($response), true);
    $country = $data['geo']['country_code'] ?? 'US';

    set_transient($cacheKey, $country, DAY_IN_SECONDS);
    return $country;
}

// Usage in any template
$country = ipp_get_visitor_country();
if ($country === 'IT') {
    echo '<p>Benvenuto! Visita il nostro negozio italiano.</p>';
}

Use case 8 — DDNS updater script (CLI)

A PHP CLI script that checks if the server's public IP changed and updates a Cloudflare DNS record:

<?php
// Run with: php ddns_updater.php

define('CF_API_TOKEN',  getenv('CF_API_TOKEN'));
define('CF_ZONE_ID',    getenv('CF_ZONE_ID'));
define('CF_RECORD_ID',  getenv('CF_RECORD_ID'));
define('RECORD_NAME',   'home.yourdomain.com');
define('CACHE_FILE',    '/tmp/ddns_last_ip.txt');

function getCurrentIP(): ?string {
    $ch = curl_init('https://ipv4.ippubblico.org/');
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 10,
    ]);
    $ip = trim(curl_exec($ch));
    $ok = curl_getinfo($ch, CURLINFO_HTTP_CODE) === 200;
    curl_close($ch);
    return $ok ? $ip : null;
}

function getLastKnownIP(): ?string {
    return file_exists(CACHE_FILE) ? trim(file_get_contents(CACHE_FILE)) : null;
}

function updateCloudflare(string $ip): bool {
    $payload = json_encode([
        'type'    => 'A',
        'name'    => RECORD_NAME,
        'content' => $ip,
        'ttl'     => 60,
    ]);

    $ch = curl_init(sprintf(
        'https://api.cloudflare.com/client/v4/zones/%s/dns_records/%s',
        CF_ZONE_ID, CF_RECORD_ID
    ));
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_CUSTOMREQUEST  => 'PUT',
        CURLOPT_POSTFIELDS     => $payload,
        CURLOPT_HTTPHEADER     => [
            'Authorization: Bearer ' . CF_API_TOKEN,
            'Content-Type: application/json',
        ],
        CURLOPT_TIMEOUT        => 10,
    ]);
    $response = json_decode(curl_exec($ch), true);
    curl_close($ch);

    return $response['success'] ?? false;
}

// Main
$current = getCurrentIP();
if ($current === null) {
    echo '[ERROR] Failed to get public IP' . PHP_EOL;
    exit(1);
}

$last = getLastKnownIP();

if ($current === $last) {
    echo "[OK] IP unchanged: {$current}" . PHP_EOL;
    exit(0);
}

echo "[INFO] IP changed: {$last} → {$current}" . PHP_EOL;

if (updateCloudflare($current)) {
    file_put_contents(CACHE_FILE, $current);
    echo "[OK] DNS updated to {$current}" . PHP_EOL;
} else {
    echo '[ERROR] DNS update failed' . PHP_EOL;
    exit(1);
}

Add to cron:

*/5 * * * * /usr/bin/php /usr/local/bin/ddns_updater.php >> /var/log/ddns.log 2>&1

Handling rate limits

The API returns 429 Too Many Requests with a Retry-After header if the same IP sends too many requests in a short period:

<?php
function getPublicIPWithRetry(int $maxRetries = 2): ?string {
    for ($attempt = 0; $attempt <= $maxRetries; $attempt++) {
        $ch = curl_init('https://ipv4.ippubblico.org/');
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 5,
            CURLOPT_HEADER         => true,
        ]);
        $response = curl_exec($ch);
        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
        curl_close($ch);

        if ($httpCode === 200) {
            return trim(substr($response, $headerSize));
        }

        if ($httpCode === 429 && $attempt < $maxRetries) {
            $headers = substr($response, 0, $headerSize);
            preg_match('/Retry-After:\s*(\d+)/i', $headers, $matches);
            $wait = (int)($matches[1] ?? 10);
            sleep($wait);
            continue;
        }

        break;
    }
    return null;
}

Quick reference

Need	Endpoint	Response
IPv4 only	`https://ipv4.ippubblico.org/`	`203.0.113.42`
IPv6 only	`https://ipv6.ippubblico.org/`	`2001:db8::1` or `NONE`
Both protocols	`https://ippubblico.org/?text=1`	`IPv4: x\nIPv6: x`
Full geolocation	`https://ippubblico.org/?api=1`	JSON with country, city, ISP
Localized geo	`https://ippubblico.org/?api=1&lang=ja`	City/country in Japanese

Full documentation: ippubblico.org/docs.html
Code examples: github.com/ippubblico/examples

Conclusion

PHP's cURL extension covers every use case cleanly — plain text for scripts and DDNS tools, JSON for web applications, and all the caching and retry logic you need in a few dozen lines. The Laravel middleware and WordPress snippets are production-ready starting points that cache results correctly to avoid hitting the API on every request.

Using IPPubblico in a PHP project? Share your use case in the comments.

How to parse lots of PDFs and more into markdown, with Laravel

Peter Van Dijck — Tue, 14 Jul 2026 07:56:08 +0000

You want to ask questions about your own documents from a Laravel app. The AI SDK that shipped with Laravel 13 handles most of it: embeddings, vector queries, agents. What it doesn't handle is the part that comes first, where a user hands you a 40-page scanned PDF and you need clean text out of it.

That's the step this tutorial covers properly. The rest of the pipeline is here too, end to end: upload, parse, chunk, embed, search, answer.

What we're building

Users upload documents. We convert each one to Markdown, split it into chunks, embed the chunks into a pgvector column, and answer questions with an agent that searches those chunks. Postgres only, no separate vector database.

You'll need Laravel 13, Postgres with the pgvector extension available, and two packages:

composer require laravel/ai parseforartisans/laravel

Full disclosure: parseforartisans/laravel is my product, Parse for Artisans. It's a hosted parsing API with a Laravel-native SDK. If all your documents are digital PDFs with a real text layer, you can swap that piece for spatie/pdf-to-text and pay nothing; I'll point out where. The hosted API earns its keep when uploads are scans, .doc files, emails, or all of the above, which in my experience is what "users upload documents" actually means.

Migrations

Two tables: the document, and its chunks with an embedding each.

use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
Schema::ensureVectorExtensionExists();
Schema::create('documents', function (Blueprint $table) {
    $table->id();
    $table->string('source_path');
    $table->string('status')->default('pending');
    $table->timestamps();
});
Schema::create('chunks', function (Blueprint $table) {
    $table->id();
    $table->foreignId('document_id')->constrained()->cascadeOnDelete();
    $table->text('content');
    $table->vector('embedding', dimensions: 1536)->index();
    $table->timestamps();
});

The models are small. Chunk casts the vector to an array, and Document gets the relation the listener will use:

class Chunk extends Model
{
    protected $guarded = [];
    protected function casts(): array

{
        return ['embedding' => 'array'];
    }
}
class Document extends Model
{
    protected $guarded = [];
    public function chunks(): HasMany

{
        return $this->hasMany(Chunk::class);
    }
}

Upload and parse

Store the upload on your disk, then point the parser at the stored path. The SDK resolves paths like Storage does, so there's no temp-file juggling:

use App\Models\Document;
use Illuminate\Http\Request;
use ParseForArtisans\Facades\Parse;
public function store(Request $request)
{
    $request->validate([
        'document' => ['required', 'file', 'mimes:pdf,doc,docx,xlsx,pptx,msg,eml', 'max:51200'],
    ]);
    $path = $request->file('document')->store('uploads', 's3');
    $document = Document::create(['source_path' => $path]);
    Parse::disk('s3')->file($path)->for($document)->parse();
    return back()->with('status', 'Processing.');
}

Parsing is async. ->for($document) ties the job to your model, so when the result comes back you get your record handed to you instead of matching ids. Scanned PDFs are OCR'd automatically; you don't tell it the file type.

The spatie alternative: replace the Parse:: line with Pdf::getText(Storage::disk('s3')->path($path)) and do the chunking inline. Works well until the first scan or .docx shows up, because pdftotext reads text layers and nothing else.

Chunk and embed when parsing completes

The Markdown arrives in a ParseCompleted event. Listen for it, split the text, and embed all chunks in one call:

use Laravel\Ai\Embeddings;
use ParseForArtisans\Events\ParseCompleted;
public function handle(ParseCompleted $event): void
{
    $document = $event->request->parsable;   // your Document model, typed
    $markdown = $event->request->markdown();
    $chunks = $this->split($markdown);
    $embeddings = Embeddings::for($chunks)->generate()->embeddings;
    foreach ($chunks as $i => $content) {
        $document->chunks()->create([
            'content' => $content,
            'embedding' => $embeddings[$i],
        ]);
    }
    $document->update(['status' => 'ready']);
}

For splitting, start dumb. Markdown gives you natural seams, so split on headings, then cap the size:

/** @return string[] */
private function split(string $markdown, int $maxLength = 2000): array
{
    $sections = preg_split('/^(?=#{1,3} )/m', $markdown);
    return collect($sections)
        ->flatMap(fn ($s) => str_split(trim($s), $maxLength))
        ->filter(fn ($s) => strlen($s) > 50)
        ->values()
        ->all();
}

This is the crudest chunker that works, and clean Markdown input is what makes it viable. When your parser preserves headings and table structure, heading-based splits keep related content together. When your input is the word soup that OCR tools produce, no chunking strategy saves you. Garbage in, garbage embedded.

Ask questions

The AI SDK's agents can search your chunks with the built-in similarity search tool. Make an agent:

php artisan make:agent DocumentAssistant

namespace App\Ai\Agents;
use App\Models\Chunk;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasTools;
use Laravel\Ai\Promptable;
use Laravel\Ai\Tools\SimilaritySearch;
class DocumentAssistant implements Agent, HasTools
{
    use Promptable;
    public function instructions(): string

{
        return 'Answer using the document excerpts you retrieve. '
            .'If the excerpts do not contain the answer, say so.';
    }
    public function tools(): iterable

{
        return [
            SimilaritySearch::usingModel(Chunk::class, 'embedding', limit: 8),
        ];
    }
}

$response = (new DocumentAssistant)->prompt(
    'What termination notice period does the supplier contract require?'
);
echo $response;

The agent embeds the question, pulls the nearest chunks, and answers from them. If you'd rather control retrieval yourself, the query builder does it directly:

$chunks = Chunk::query()
    ->whereVectorSimilarTo('embedding', 'termination notice period')
    ->limit(8)
    ->get();

Passing a string makes the SDK generate the query embedding for you.

Scaling notes

Ten thousand files work the same way as one. Parse::files($paths) submits a batch, and each file fires its own ParseCompleted event, so the listener you already wrote is the whole pipeline. Queue the listener (implements ShouldQueue) so embedding runs off the request path.

One alternative before you build any of this: if you don't want to own chunking and a vector column at all, the AI SDK also supports provider vector stores. Stores::create(), add files, and give an agent the FileSearch tool. You give up control over chunking and your data lives with the provider, but it's less code. I like owning the Postgres side; on a quick prototype I'd take the store.

That's the pipeline: a controller, a listener, an agent. The reason it stays this short is that every step downstream of parsing gets to assume clean Markdown. That assumption is the entire game in document RAG, and it's decided at the step most tutorials skip.

Parse for Artisans is a document parsing API for Laravel that converts PDF, DOCX, scans, and 20+ formats to clean Markdown. There's a free tier of 15,000 pages a month.