Idempotent gateway sync: adopt on conflict, paginate for real, and never widen scope

TL;DR

• Syncing config into an API gateway (Kong here) is a converge operation, not an insert. Re-running it must be safe.
• Three bugs, one theme: adopt an existing record on a 409 UNIQUE instead of throwing; read every page of a list API, not just the first 100; never let a scoped plugin silently escape to global scope.
• All three are the same mistake wearing different clothes: treating a declarative sync like a one-shot create.

Sync is a converge, not a create

A gateway sync takes your desired state (routes, upstreams, plugins, consumers) and makes the gateway match it. The mental trap is writing that as "create each thing." Creates are one-shot; sync runs again tomorrow, and the day after. The correct model is: for each item, make the gateway's state equal to mine — whether that means create, adopt, or update.

Get that wrong and the second run is where it hurts.

Bug 1: adopt on 409, don't throw

Creating an upstream target that already exists returns 409 Conflict on a UNIQUE constraint. The naive code lets that bubble up and the whole sync aborts partway — leaving the gateway half-configured, which is worse than not syncing at all.

The fix: treat the conflict as "already there, adopt it" and carry on.

try {
    $target = $this->gateway->createUpstreamTarget($upstream, $payload);
} catch (ConflictException $e) {
    // 409 UNIQUE — the target exists. Adopt it instead of failing the run.
    $target = $this->gateway->findUpstreamTarget($upstream, $payload['target']);
}

The conflict isn't an error condition here — it's just the gateway telling you the desired state is already met. Converge means you shrug and move on.

Bug 2: paginate, or your diff lies

Here's a subtle one. To sync, you first read the gateway's current state and diff it against desired. If your read only fetches the first page (Kong defaults to 100 per page), then the moment you have 101 plugins, item 101 looks missing — so your diff decides to create it, and hits Bug 1's conflict, or worse, your cleanup logic decides an existing record is an orphan and deletes it.

A partial read doesn't just miss data — it actively corrupts every downstream decision.

private function getAll(string $path): array
{
    $items = [];
    $next = $path;

    while ($next !== null) {
        $res = $this->client->get($next);
        $items = array_merge($items, $res['data']);
        $next = $res['next'] ?? null; // follow the cursor to the end
    }

    return $items;
}

Symptom	Root cause
Sync tries to re-create records that exist	Read capped at page 1, diff thinks they're missing
Cleanup deletes live config as "orphans"	Same partial read, inverted consequence
Works in dev, breaks in prod	Dev has < 100 records; prod crosses the page boundary

That last row is the trap. It passes every test until real volume arrives.

Bug 3: scope is identity

A plugin scoped to a single route or service must never silently become a global plugin during sync. Scope isn't metadata you can drop — it's part of what the record is. A global plugin applies to all traffic; a scoped one applies to one route. Confusing them turns a targeted rate-limit into a gateway-wide one.

The fix is to carry scope through every comparison and write, and to assert it in tests so a refactor can't quietly flatten it:

it('keeps a route-scoped plugin scoped after sync', function () {
    $plugin = syncPlugin(scope: 'route', routeId: $route->id);

    expect($plugin->scope)->toBe('route')
        ->and($plugin->route_id)->toBe($route->id)
        ->and($plugin->is_global)->toBeFalse();
});

Takeaway

Idempotency isn't a nice-to-have for a sync — it's the whole job. Every bug today came from a create-shaped assumption: a conflict is a failure (no — it's convergence), a list fits on one page (no — follow the cursor), scope is optional (no — it's identity). Write the sync so running it twice is a no-op, and prod stops surprising you.