DEV Community

Hugo Bachmann
Hugo Bachmann

Posted on AI-assisted

When the database and the game disagree: idempotency and compensation for RCON in Laravel

During the closed beta of Agoralis, a player bought a property from the web app. In the game, the region became theirs. In the database, the money never left their account. We saw it right away, because every beta transaction was being followed closely, and fixed it by hand in Filament. It was the first time the web app and the game disagreed, and it would not be the last if we left the code as it was.

This article shows how we stopped that from happening. You will see the first version we shipped and why it could split in two, then the three pieces we added: an idempotency key so a double click never pays twice, a compensation step when the game refuses a command, and an audit trail that tells a moderator exactly what was done and what was undone. The code is Laravel 10, Livewire 3, Filament 3 and thedudeguy/rcon, but the pattern applies to any app that writes to its own database and then drives a system it does not control.

TL;DR

  • You cannot wrap MySQL and a game server in one transaction. Write the money and the list of commands to send in the same DB transaction (an outbox), then send the commands from a queued job.
  • Give every user action an idempotency key, created when the confirmation modal opens and stored under a unique index. A double click or a Livewire retry then hits the index instead of paying twice.
  • Sort your RCON commands into "safe to retry" (rg addowner: setting a state) and "not safe" (exp give: adding a quantity). Retry the first kind; for the second, read the game state before retrying.
  • When a command fails for good, do not delete the transfer: write a reverse transfer and log it. Money is corrected by new rows, never by erasing old ones.
  • Group the transfer, its commands and any compensation under one activity-log batch, so a moderator reads one story instead of five unrelated lines.

The context

Agoralis is an economic role-play Minecraft server: players have a job, a bank account, properties they buy, sell or rent, and XP they can trade. The game itself was written in Java by the Agoralis team. We built everything in the browser: the player web app (Laravel, Livewire), the moderation panel (Filament) and the link between the web app and the game. The full project story, with the timeline and the team is on our site (in French).

The link is RCON, Minecraft's remote console protocol. The web app opens a TCP connection, authenticates with a password, sends a text command such as webapi rg addowner <region> <player> and reads a text reply. There is no transaction, no acknowledgement beyond that reply, and no way to ask "did you already run this?".

Three constraints shaped the design:

  • The game server was being built at the same time as the web app. Commands, plugin replies and restarts changed during development. We could not ask the Java team for a transactional API.
  • Some commands only work when the player is online. exp give and exp take act on the player's in-game XP bar, so they fail if the player logged off between the click and the command.
  • A two-person team, about eight months of back-end work. Whatever we added had to stay readable by one developer and debuggable by volunteer moderators.

The first version

The first version did the obvious thing: write to MySQL, then send the commands. Here is the property purchase from app/Livewire/Map.php, trimmed:

// 1. Money moves: the WireTransfer model updates both balances in its `creating` hook
WireTransfer::create([
    'user_id' => auth()->user()->id,
    'amount' => $this->propertyToBuy->price,
    'label' => 'Achat de ' . $this->propertyToBuy->name,
    'origin_account_id' => auth()->user()->bankAccount->id,
    'destination_account_id' => $this->propertyToBuy->user->bankAccount->id,
]);

$server = new Server();

// 2. The game is updated, one command at a time
$server->sendCommand('webapi rg removemember ' . $this->propertyToBuy->slug . ' -a');
$server->sendCommand('webapi rg removeowner ' . $this->propertyToBuy->slug . ' ' . $this->propertyToBuy->user->mojang_username);
$server->sendCommand('webapi rg addowner ' . $this->propertyToBuy->slug . ' ' . auth()->user()->mojang_username);
// ... mail + tell to the seller

// 3. The property changes owner in the database
$this->propertyToBuy->user_id = auth()->user()->id;
$this->propertyToBuy->is_for_sale = false;
$this->propertyToBuy->save();
Enter fullscreen mode Exit fullscreen mode

The RCON wrapper, app/Service/Server.php, caught exceptions, logged them and called abort(503) (or returned null on the home page). Read the two together and every failure point is visible. If RCON fails at step 2, the request stops: the money has moved (step 1), the region is half-changed, and the property still belongs to the seller in MySQL (step 3 never ran). A double click on "Buy" runs the whole thing twice. And nothing records that the purchase was left half done, so moderators only learned about it from the player.

Options we considered

Option What it gives Why we did or didn't keep it
DB transaction around everything, RCON inside Rolls back MySQL if a command throws Rejected alone: it cannot roll back a command the game already ran, and holds row locks during network calls
Send RCON first, then write to MySQL No money lost if the game refuses Kept for XP deposits (exp take then credit). Wrong for purchases: the game changes before we know the buyer can pay
Distributed transaction / two-phase commit Atomicity across both systems Impossible: the game server has no prepare/commit step
Outbox + idempotency key + compensation DB and "commands to send" committed together; safe retries; a defined way back Chosen. Plain Laravel (migration, job, observer), no new infrastructure

Step 1: store the commands next to the money

The core idea is an outbox. Instead of sending RCON commands during the request, we write them to a table in the same transaction as the money. If the transaction commits, both the transfer and the "things the game must do" exist. If it rolls back, neither does. A queued job sends the commands later.

Two tables carry it. wire_transfers gains three columns: a unique idempotency_key, a batch_uuid, and reverses_id, which points a compensation at the transfer it cancels. A new game_commands table stores each command with its batch_uuid, its position in the sequence, a retry_safe flag (step 4), a status (pending, sent, failed, compensated), the number of attempts and the game's last_response. The batch_uuid is the thread that ties everything together: the transfer, its commands, a possible compensation and the activity-log entries all share it.

Step 2: make the money move inside a real transaction

The original WireTransfer model moved balances in its creating hook with a read-modify-write: $originAccount->balance -= $model->amount; $originAccount->save();. Two transfers from the same account at the same moment could both read 500 and both write 500 minus their amount. We kept the hook, because rent, tax and loan payouts rely on it, but made it atomic: inside DB::transaction(), it locks both bank accounts with lockForUpdate() (always in id order, to avoid deadlocks between A→B and B→A), checks the balance, then calls decrement() and increment(). Account 1, the state bank, may still go negative for loans.

The balance check used to live in the observer ($wireTransfer->originAccount->balance > $wireTransfer->amount), which runs after the money has moved. Moving it before the update, under the lock, is what makes it a real guard. When the hook runs inside an outer DB::transaction(), Laravel turns the inner one into a savepoint, so the whole purchase still commits or rolls back as one unit.

Step 3: an idempotency key per click

Livewire makes double submits easy: a double click, a slow network that makes the player click again, a request retried after a timeout. Each one used to create a new WireTransfer. We now create a key when the confirmation modal opens, and the purchase refuses to run twice with the same key.

In the Livewire component, the key is a #[Locked] public ?string $purchaseKey, set to Str::uuid() by the method that opens the modal and passed to the action on confirm. #[Locked] matters: without it a player could change the key from the browser console and replay a purchase. The key is created on the server and the client can only send it back.

The action does the rest. It looks for an existing transfer with the key, and relies on the unique index for the case where two requests arrive in the same millisecond:

// app/Actions/Economy/PurchaseProperty.php
public function handle(User $buyer, Property $property, string $idempotencyKey): WireTransfer
{
    if ($existing = WireTransfer::firstWhere('idempotency_key', $idempotencyKey)) {
        return $existing; // second click: same result, nothing moves
    }

    LogBatch::startBatch();
    $batchUuid = LogBatch::getUuid();

    try {
        $transfer = DB::transaction(function () use ($buyer, $property, $idempotencyKey, $batchUuid) {
            // Re-read the property under lock: it may have been sold a second ago
            $property = Property::whereKey($property->id)->lockForUpdate()->firstOrFail();

            if (! $property->is_for_sale || $property->user_id === $buyer->id) {
                throw new PropertyNotForSale($property);
            }

            $seller = $property->user;

            $transfer = WireTransfer::create([
                'idempotency_key' => $idempotencyKey,
                'batch_uuid' => $batchUuid,
                'user_id' => $buyer->id,
                'amount' => $property->price,
                'label' => 'Achat de ' . $property->name,
                'origin_account_id' => $buyer->bankAccount->id,
                'destination_account_id' => $seller->bankAccount->id,
            ]);

            $property->update(['user_id' => $buyer->id, 'is_for_sale' => false]);

            GameCommand::enqueue($transfer, [
                ["webapi rg removemember {$property->slug} -a", true],
                ["webapi rg removeowner {$property->slug} {$seller->mojang_username}", true],
                ["webapi rg addowner {$property->slug} {$buyer->mojang_username}", true],
                ["webapi mail {$seller->mojang_username} Votre propriété a été vendue.", true],
            ]);

            return $transfer;
        });
    } catch (UniqueConstraintViolationException) {
        // Lost the race against an identical request: return the winner's transfer
        return WireTransfer::where('idempotency_key', $idempotencyKey)->firstOrFail();
    } finally {
        LogBatch::endBatch();
    }

    SendGameCommands::dispatch($batchUuid)->afterCommit();

    return $transfer;
}
Enter fullscreen mode Exit fullscreen mode

Note that the property now changes owner in MySQL inside the transaction, together with the money. In the first version it changed last, after the RCON calls, so any RCON failure left the database saying the seller still owned a plot they had been paid for.

Step 4: send the commands, and know which ones are safe to retry

Retrying is only safe if running the command twice gives the same result as running it once. That is true for some of our commands and false for others, so each GameCommand row carries a retry_safe flag set by whoever enqueues it.

Command Effect Safe to retry?
rg addowner <region> <player> Sets a state: the player is an owner Yes, adding an owner twice changes nothing
rg removemember <region> -a Sets a state: no members Yes
perms add <uuid> <group> Sets a state: the player is in a group Yes
exp give <player> <n> / exp take Adds or removes a quantity No, twice means double XP
mail / tell Sends a message Not strictly, but a duplicate message is harmless, so we flag it retry-safe

The RCON wrapper also had to change. Server::sendCommand() used to swallow errors and call abort(503), which only makes sense in an HTTP request, not in a queued job. It now throws, and says whether the command may have reached the game:

// app/Service/Server.php
public function sendCommand(string $command): string
{
    // connect() returns false instead of throwing when the server is down
    if (! $this->rcon->connect()) {
        throw RconException::unreachable($command);          // never sent: safe to retry
    }

    try {
        $response = $this->rcon->sendCommand($command);
    } finally {
        $this->rcon->disconnect();
    }

    if ($response === false) {
        throw RconException::noResponse($command);            // sent, outcome unknown
    }

    // The plugin answers HTTP 200-style: errors come back as text
    foreach (config('app.rconErrorPatterns') as $pattern) {
        if (str_contains(mb_strtolower($response), $pattern)) {
            throw RconException::refused($command, $response); // the game said no
        }
    }

    return $response;
}
Enter fullscreen mode Exit fullscreen mode

The job walks through the pending commands in order and stops at the first one it cannot send:

// app/Jobs/SendGameCommands.php (trimmed)
class SendGameCommands implements ShouldQueue, ShouldBeUnique
{
    public int $tries = 5;
    public array $backoff = [10, 30, 60, 120];

    public function __construct(public string $batchUuid) {}

    public function uniqueId(): string
    {
        return $this->batchUuid; // never two workers on the same batch
    }

    public function handle(Server $server): void
    {
        LogBatch::setBatch($this->batchUuid);

        foreach (GameCommand::pendingFor($this->batchUuid) as $command) {
            if ($command->attempts > 0 && ! $command->retry_safe) {
                // We may already have run it: do not guess
                $this->fail(new RconException('Outcome unknown: ' . $command->command));
                return;
            }

            try {
                $command->markSent($server->sendCommand($command->command));
            } catch (RconException $e) {
                if (! $e->isUnreachable()) {
                    $command->increment('attempts'); // the command may have reached the game
                }

                if ($e->isRefused()) {
                    $this->fail($e); // the game will say no again: compensate now
                    return;
                }

                throw $e; // network issue: the queue retries with backoff
            }
        }
    }

    public function failed(Throwable $e): void
    {
        app(CompensateBatch::class)->handle($this->batchUuid, $e);
    }
}
Enter fullscreen mode Exit fullscreen mode

Two details are worth knowing. First, attempts only counts tries where the connection opened. An unreachable server means the command never left, so even exp give can be retried safely; a lost reply after the connection opened cannot. Second, ShouldBeUnique relies on a cache lock, so every worker must share the same cache store (Redis or the database), not the file driver on several servers.

Step 5: compensate instead of deleting

When a batch fails for good, the money must go back and the game must be put back as it was. The original code had a commented-out deleted() observer that created a reverse transfer and then deleted the original. We kept the first half and dropped the second: a transfer that happened stays in the history, and the correction is a new transfer that points at it.

// app/Actions/Economy/CompensateBatch.php (trimmed)
public function handle(string $batchUuid, Throwable $reason): void
{
    DB::transaction(function () use ($batchUuid, $reason) {
        $transfer = WireTransfer::where('batch_uuid', $batchUuid)
            ->whereNull('reverses_id')
            ->lockForUpdate()
            ->firstOrFail();

        if (WireTransfer::where('reverses_id', $transfer->id)->exists()) {
            return; // already compensated: failed() can run more than once
        }

        // The money goes back as a new transfer; the original stays in the history
        WireTransfer::create([
            'reverses_id' => $transfer->id,
            'batch_uuid' => $batchUuid,
            'amount' => $transfer->amount,
            'label' => 'Annulation : ' . $transfer->label,
            'origin_account_id' => $transfer->destination_account_id,
            'destination_account_id' => $transfer->origin_account_id,
            // ...
        ]);

        // Undo in the game only what was applied, in reverse order
        $undo = GameCommand::sentFor($batchUuid)->reverse()
            ->map(fn (GameCommand $c) => $c->inverse())
            ->filter();

        GameCommand::enqueue(null, $undo->values()->all(), batchUuid: $batchUuid);

        // ... give the property back to the seller, log the reason
    });

    SendGameCommands::dispatch($batchUuid)->afterCommit();
}
Enter fullscreen mode Exit fullscreen mode

inverse() is a small match on the command: rg addowner X P becomes rg removeowner X P, exp give P n becomes exp take P n, messages return null. The compensation can itself fail, of course. We do not compensate a compensation: the batch is flagged in Filament and a moderator takes it from there. That never happened on Agoralis.

The compensation cannot always be perfect. If the seller already spent the money, the reverse transfer pushes their account below zero. We accepted that: the bank already allowed overdrafts for loans, and an overdraft is visible and fixable, while a player paid for nothing is not.

Step 6: XP, where retries need a look at the game first

XP deposits (exp take in game, credit on the web account) already ran the game command first and wrote to MySQL second, which is the right order: if the game refuses, nothing has moved. The risky case is the other direction, withdrawing XP from the web account into the game with exp give, when the reply is lost. Did the player get the XP or not?

Retrying blindly can give it twice. Not retrying can lose it. So before retrying an exp give, the job asks the game: it runs webapi exp bal <player> and compares the result with the balance read before the first attempt plus the amount. If the player already has it, the command is marked sent without being resent.

balance_before is read with exp bal just before the first attempt and stored in a JSON context column. This check is not perfect: the player keeps playing and earning XP, so a balance above the target does not prove our command ran. We did not add a margin for that: in doubt, the command goes to a moderator, and we never hit the case in practice. This check runs before the retry_safe guard shown in step 4.

What broke during the beta

The desyncs showed up during the closed beta, before the public opening in August 2025. Moderators fixed them in both places: balances and owners in Filament, regions and XP with commands in the game console. Two problems actually hit us. Going back through the code to understand them, we found four more that had not caused an incident yet, but would have with more players.

When What happened Why What changed
Early beta RCON down, purchase "succeeded" thedudeguy/rcon does not throw when the server is unreachable: connect() returns false and sendCommand() returns false when not connected. Our try/catch never fired, the code carried on and nothing was logged Server::sendCommand() checks both return values and throws RconException
Beta Scheduled payouts crashed on RCON errors Server called Route::current()->uri to decide between null and abort(503). In an Artisan command (RentPayout, TaxPayout, CreditPayout) there is no route, so that line crashed instead Server throws; callers decide. HTTP concerns left the service
Code review A plugin refusal would read as success The game plugin replies with text, and errors (unknown player, unknown region) are just another text reply Error patterns in config, checked on every reply
Code review The same plot could be bought twice No lock on the property: two buyers could both read is_for_sale = true lockForUpdate() on the property inside the purchase transaction
Code review Each feature would fail differently Purchase wrote money, then RCON, then the property. Rental wrote the property, then RCON, then money One rule everywhere: DB state + outbox in one transaction, game after commit
Code review Balance checked after the fact The "enough money?" test lived in the observer, which runs after the creating hook has already moved the balances Check moved into the hook, under lock

The lesson we took from the first row: when you integrate a library that talks to an external system, read what it does on failure before writing the happy path. A wrapper that "catches exceptions" around a library that returns false is a wrapper that catches nothing.

Wiring it into Laravel

The first version created the RCON client with new Server() in about twenty places (Livewire components, Filament pages, observers, Artisan commands), which made it impossible to fake in tests. We bound Server in AppServiceProvider and inject it instead. The plugin's error replies live in config('app.rconErrorPatterns'). We had little documentation on what the plugin returns, so we started with a short list of obvious lower-case fragments such as unknown command and player not found, to extend as new replies show up.

The queue needs a real driver (we use database; volumes are low and there are no big peaks), a worker kept alive by Supervisor, and a cache store shared by workers for ShouldBeUnique. In Filament, a GameCommandResource lists failed and compensated batches, filtered by batch_uuid, so a moderator can see the transfer, each command with its reply, and the compensation on one screen.

Logging: one batch, one story

The project already used spatie/laravel-activitylog (through rmsramos/activitylog for the Filament UI), and its migrations already had a batch_uuid column we were not using. Opening a batch with LogBatch::startBatch() in the action, then LogBatch::setBatch($uuid) in the job and in the compensation, gives every log line the same UUID: the transfer created by the model, the property update, each RCON command sent, the reverse transfer.

Before, a moderator searching a player's complaint found "Achat de parcelle-12" in one place and an RCON error in laravel.log with no link between them. After, one filter on the batch shows the whole story in order.

Two rules we follow for these logs: they are append-only (nobody edits a transfer, a correction is a new row), and they say who acted: a player, a moderator or system for the job and the compensation.

Testing

The repository had only the Jetstream starter tests when we started this work. With Server in the container, the consistency rules become plain feature tests:

// tests/Feature/PurchasePropertyTest.php (Pest)
// phpunit.xml sets QUEUE_CONNECTION=sync, so the job runs right after the commit

it('refunds the buyer when the game refuses a command', function () {
    // Buyer starts with 1000, the plot costs 500
    [$buyer, $seller, $property] = propertyForSale(price: 500, buyerBalance: 1000);

    $server = Mockery::mock(Server::class);
    $server->shouldReceive('sendCommand')->andReturnUsing(function (string $command) use ($buyer) {
        if (str_contains($command, 'rg addowner') && str_contains($command, $buyer->mojang_username)) {
            throw RconException::refused($command, 'Player not found');
        }

        return 'ok';
    });
    app()->instance(Server::class, $server);

    app(PurchaseProperty::class)->handle($buyer, $property, (string) Str::uuid());

    expect($buyer->bankAccount->fresh()->balance)->toEqual(1000)
        ->and($property->fresh()->user_id)->toBe($seller->id);

    $this->assertDatabaseHas('wire_transfers', ['label' => 'Annulation : Achat de ' . $property->name]);
});
Enter fullscreen mode Exit fullscreen mode

The idempotency test is the same shape: call the action twice with the same key, assert one transfer and one debit.

What these tests do not cover is the real plugin. Its replies changed while the Java team built the server, so we completed the Pest suite with manual tests on the game server.

Security

  • The RCON password is a full admin console. Whoever has it can op themselves, give items, stop the server. Keep it in .env only, never in code or comments, and restrict the RCON port to the web server's IP at the firewall.
  • Every command is built from strings. Player names come from Mojang and are validated ([A-Za-z0-9_]{3,16}), region slugs come from our database, and amounts are cast to integers. Nothing typed freely by a player is ever concatenated into a command; free text (labels, messages) stays in MySQL.
  • The idempotency key is server-side. It is created in PHP and #[Locked] in Livewire, so a client cannot pick or reuse one.

Wrapping up

  • Two systems, no shared transaction: pick which one is the source of truth. For us it is MySQL. The game is brought in line with it, by commands sent after commit and, if needed, by commands that undo.
  • Idempotency is a property of each command, not of the integration. Setting an owner can be retried; adding XP cannot. Write it down per command, in the data.
  • Correct money with new rows. A reverse transfer that points at the original is easier to explain to a player and to audit than a deleted line.
  • Read the failure path of your client library first. Ours returned false where we expected exceptions, and that alone explained the silent desyncs.

Top comments (0)