DEV Community: Web Pioneer

Self-Hosting Your Ops Stack on One VPS: Metrics, Errors, CI and Backups Without a SaaS Bill

Web Pioneer — Mon, 20 Jul 2026 11:45:01 +0000

Small engineering teams reach a point where the monthly tooling bill starts to look strange next to the infrastructure bill. Metrics, dashboards, error tracking, CI, uptime monitoring and a business system can easily cost more per month than every server they run on.

Self-hosting is not automatically the answer — it moves cost from a subscription line into your team's time, and that trade is only good if the operational burden stays small. This article is about keeping it small: what genuinely fits on one modest VPS, how to size it, the failures that catch people, and where the line is.

What fits on one box

A single VPS with 8 vCPU and 16 GB of RAM comfortably runs the following, with headroom:

Component	Role	Realistic memory
Prometheus	Metrics collection and storage	1–2 GB, grows with cardinality
Grafana	Dashboards and alerting UI	200–400 MB
Blackbox exporter	External uptime and TLS expiry probes	The memory trap. Self-hosted error-tracking platforms are not a single service — they are a stack of web workers, background workers, a relay, a database, a cache and a queue, and the default compose files are tuned for a much larger machine than yours. This one component will consume more RAM than everything else combined. Budget for it first, or discover it when the kernel's OOM killer starts choosing victims at 3am.

Budget memory before you install anything

The single most common failure mode is not a service crashing — it is the OOM killer terminating whichever process happened to be largest, which is frequently your database. Two mitigations, both cheap:

1. Cap every container explicitly. Unbounded containers will expand until the host has nothing left:

# docker-compose.yml
services:
  worker:
    deploy:
      resources:
        limits:
          memory: 1g
    restart: unless-stopped

2. Add swap even on an SSD box. Swap is not a substitute for RAM, but it converts an instant OOM kill into a slow period you can alert on and react to:

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10   # prefer RAM, use swap as a cushion

Then alert on swap usage above zero. On a healthy box it stays at zero, so any sustained swap is an early warning that you are one deploy away from an outage.

Three failures that catch almost everyone

Failure 1: the firewall silently breaks certificate renewal

You harden the box, allow only 443 and SSH, and everything works. Sixty days later every service on the host goes untrusted simultaneously.

The HTTP-01 certificate challenge requires the certificate authority to reach port 80. If your firewall drops it, renewal fails — quietly, from a cron job nobody reads, for weeks. The first symptom is a browser warning on every subdomain at once.

ufw allow 80/tcp        # required for HTTP-01, even if you redirect everything to 443
ufw allow 443/tcp
certbot renew --force-renewal --dry-run

Two durable fixes: monitor certificate expiry as a metric rather than trusting the renewal job, and prefer DNS-01 challenges if you genuinely cannot open port 80. The blackbox exporter exposes probe_ssl_earliest_cert_expiry — alert at 21 days remaining and this class of incident disappears permanently.

Failure 2: stale resolvers make monitoring lie

A subtler one. Some hosting providers ship images preconfigured with their own recursive resolvers, and those resolvers can serve stale records long after you have changed DNS.

The consequence is specific and nasty: your uptime probes resolve a domain to an address that no longer serves it. The monitoring system reports the site as down while it is perfectly healthy — or, far worse, reports it as up while pointing at a decommissioned host. You end up debugging an application that has nothing wrong with it.

# /etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    eth0:
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

netplan apply
resolvectl status | grep 'DNS Servers'
dig +short your-domain.com @1.1.1.1

Rule: a monitoring host must resolve names the same way your users do. Pin it to public resolvers and verify after every image rebuild — provider defaults come back.Failure 3: backups the attacker can delete

The naive design has each server push its backups to a central repository, which means every server holds credentials that can write to — and therefore delete or encrypt — that repository. One compromised host destroys every backup you have. This is precisely how ransomware incidents escalate from bad to unrecoverable.

Invert it. The backup host pulls from each server over SSH using a key restricted to a read-only rsync command. The servers hold no backup credentials at all:

# on the backed-up host, ~/.ssh/authorized_keys
command="rsync --server --sender -vlogDtpre.iLsfxC . /path/to/data/",\
no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding ssh-ed25519 AAAA...

Then snapshot into a deduplicating repository with retention, and — the step people skip — verify:

restic backup /srv/pulled/hostname --tag nightly
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check --read-data-subset=5%   # actually read data, not just metadata

An unverified backup is a hypothesis. Schedule a real restore of one service to a scratch directory monthly and time it — that number is your true recovery objective, and it is always longer than anyone guesses.

Monitor the monitoring

A self-hosted observability stack has an obvious flaw: when the host dies, so does everything that would have told you. Close it with two cheap measures.

First, an external uptime check from a third party — a free tier is fine — that watches the monitoring host itself. Second, a dead-man's switch: your stack pings an external endpoint every few minutes, and that service alerts you when the pings stop. Inverting the signal catches total failure, which normal alerting cannot.

# pushes only if the local health check passes
*/5 * * * * curl -fsS --max-time 10 http://localhost:9090/-/healthy >/dev/null \
  && curl -fsS --max-time 10 https://your-deadman-endpoint/ping >/dev/null

Lock the perimeter

One box now holds metrics, error traces containing request payloads, CI credentials and possibly business data. Treat it accordingly:

Nothing internal on a public port. Bind services to 127.0.0.1 and expose only the reverse proxy. A dashboard tool reachable from the internet with default credentials is a well-known way to lose a server.
SSH keys only, password authentication disabled, plus fail2ban or equivalent.
Authentication in front of every UI, including the ones you assume nobody knows about. Metrics endpoints leak internal hostnames, versions and topology.
Secrets out of compose files. Environment files with restrictive modes, excluded from version control, rotated when anyone leaves.
Unattended security updates enabled, with reboot windows scheduled rather than deferred indefinitely.

What we do not recommend self-hosting

Being honest about the boundary is what makes the rest credible:

Do self-host	Do not
Metrics and dashboards — stable, low-maintenance, data grows predictably	Outbound email delivery — deliverability is a full-time reputation problem; use a provider
Error tracking — high SaaS cost per event, tolerable to operate	Authentication for customers — the blast radius of getting it wrong is unbounded
CI for your own repositories	Payments — compliance scope you do not want
Internal business systems	Your status page — it must survive your infrastructure failing
Uptime probing (as a second opinion)	Primary DNS — anycast networks are better and effectively free

The real cost calculation

The subscription saving is easy to compute and easy to overstate. The honest version includes:

The VPS itself, plus a second small instance if you want offsite backup separation.
Roughly half a day per month of routine maintenance — updates, disk pressure, retention tuning.
Occasional incident time when a component surprises you, front-loaded in the first two months.
The knowledge concentration risk: if exactly one person understands the stack, that is an availability problem. Write the runbook before you need it.

For a team already running its own servers and comfortable with Linux, this trade is usually clearly positive. For a team without that baseline, a managed service is genuinely the cheaper option once you price the time correctly.

Summary

Budget memory before installing anything; cap containers and add swap you alert on.
Open port 80 for HTTP-01, and monitor certificate expiry rather than trusting the renewal cron.
Pin your monitoring host to public DNS resolvers so probes see what users see.
Pull backups from a host the servers cannot write to, and verify restores on a schedule.
Watch the watcher with an external check and a dead-man's switch.
Keep email, customer auth, payments and your status page off the box.

We design, deploy and operate infrastructure like this for companies across Egypt and the Gulf — including the monitoring, backup verification and incident response that make it safe to run yourself. If you want a review of your current setup before it becomes an incident, our team is available.

FAQ

How much server do I need to self-host a monitoring and error-tracking stack?

For a small team, 8 vCPU and 16 GB of RAM is a comfortable starting point, and the memory matters far more than the cores. Self-hosted error-tracking platforms are the dominant consumer — they run as a group of web workers, background workers, a database, a cache and a queue, and typically need 4 to 6 GB on their own. Metrics, dashboards and probes together fit in about 2 GB. Add swap and cap every container so one runaway process cannot trigger the OOM killer.

Why did my Let's Encrypt certificates stop renewing after I enabled a firewall?

The HTTP-01 challenge requires the certificate authority to reach your server on port 80. If the firewall drops it, renewals fail silently from cron and you only find out roughly sixty days later when every service on the host goes untrusted at once. Allow 80/tcp even if you redirect all traffic to HTTPS, or switch to a DNS-01 challenge. Then monitor certificate expiry as a metric and alert with about three weeks of margin.

Should backups be pushed from servers or pulled by the backup host?

Pulled. If each server pushes, every server holds credentials that can delete or encrypt the backup repository, so a single compromised host can destroy all your backups. With a pull model the backup host connects outward using an SSH key restricted to a read-only command, and the servers hold no backup credentials at all. Add retention, and verify with periodic real restores rather than trusting that the job exited zero.

What should I never self-host?

Outbound email delivery, customer authentication, payment processing, primary DNS, and your status page. Email deliverability is an ongoing reputation problem that a provider solves better; auth and payments carry a blast radius and compliance scope that is not worth owning; anycast DNS from a provider is better and essentially free; and a status page hosted on your own infrastructure goes down exactly when you need it.

How do I get alerted when the monitoring server itself goes down?

Two independent mechanisms. Use an external uptime service to watch the monitoring host from outside your network, and add a dead-man's switch — have the stack ping an external endpoint on a schedule and configure that service to alert when the pings stop rather than when they fail. Inverting the signal is what catches total host failure, which internal alerting by definition cannot report.

Originally published at web-pioneer.com.

A Local WhatsApp Gateway for Laravel Notifications: Architecture, Trade-offs and Honest Limits

Web Pioneer — Mon, 20 Jul 2026 11:38:19 +0000

Every business application in this region eventually needs to send a WhatsApp message. Booking confirmations, OTPs, delivery updates, invoice reminders — the channel your customers actually read.

The engineering question is not "how do I send a WhatsApp message." It is "how do I keep that decision reversible." Messaging providers change, pricing changes, template approval rules change, and a project that hardcodes one provider's HTTP client across forty call sites will pay for it repeatedly.

This article describes the shape we use: WhatsApp delivery sits behind Laravel's own notification abstraction, with a swappable transport underneath, developed and tested against a small self-hosted gateway on localhost. It also states plainly where that gateway is not appropriate, because that part usually gets left out.

Start with the honest constraint

Read this before you build anything. There are two fundamentally different ways to send WhatsApp messages programmatically. The official WhatsApp Business Cloud API is a supported product with documented rate limits, template approval, and an account you can be held accountable to. Community libraries that drive the WhatsApp Web protocol from a linked device are unofficial. They are not covered by WhatsApp's terms for business messaging, the number can be restricted or banned, and there is no support path when that happens.

Our position: a local gateway is legitimate for development, integration testing, and internal notifications to your own team's numbers with their consent. For customer-facing production messaging, use the official Cloud API. The architecture below is designed so that switching between them is a config change, not a rewrite.

If a vendor tells you an unofficial gateway is a production-grade substitute for the Cloud API, they are transferring risk to you.

The seam: one interface, several transports

Laravel already gives you the right abstraction. A notification declares what it wants to send and through which channel; it should never know how bytes reach a phone.

Define a narrow contract:

<?php
namespace App\Messaging;

interface MessageTransport
{
    /**
     * @param  string $to       E.164 digits, no punctuation
     * @param  string $body     Plain text
     * @return string           Provider message id, for correlation
     * @throws TransportException
     */
    public function send(string $to, string $body): string;
}

Then bind whichever implementation the environment asks for:

// app/Providers/MessagingServiceProvider.php
public function register(): void
{
    $this->app->bind(MessageTransport::class, function ($app) {
        return match (config('messaging.driver')) {
            'cloud_api' => new CloudApiTransport(config('messaging.cloud_api')),
            'gateway'   => new LocalGatewayTransport(config('messaging.gateway')),
            'log'       => new LogTransport(),
            'null'      => new NullTransport(),
            default     => throw new \InvalidArgumentException('Unknown messaging driver'),
        };
    });
}

Four drivers, and the last two matter more than people expect. log writes the message to your application log instead of sending it — that is your local development default. null discards silently — that is your test suite default, so nothing escapes during CI.

Design rule: the default in a fresh .env.example should never be a driver that can reach a real phone. Make the safe option the one you get by forgetting to configure anything.

The notification channel

Wrap the transport in a custom channel so notifications stay declarative:

<?php
namespace App\Messaging;

use Illuminate\Notifications\Notification;

class WhatsAppChannel
{
    public function __construct(private MessageTransport $transport) {}

    public function send($notifiable, Notification $notification): void
    {
        $to = $notifiable->routeNotificationFor('whatsapp');
        if (! $to) {
            return;
        }

        $message = $notification->toWhatsApp($notifiable);
        $this->transport->send($this->normalise($to), $message);
    }

    private function normalise(string $number): string
    {
        return preg_replace('/\D+/', '', $number);   // digits only
    }
}

Call sites now look like this, and stay this way forever regardless of provider:

$booking->customer->notify(new BookingConfirmed($booking));

The local gateway

The gateway is a small Node service that owns a linked WhatsApp session and exposes a minimal HTTP surface. Three design decisions carry all the weight:

1. Bind to loopback only

The gateway holds credentials equivalent to a logged-in phone. It must never be reachable from the internet:

app.listen(3010, '127.0.0.1');   // not 0.0.0.0. ever.

Laravel talks to it over http://127.0.0.1:3010. If your app servers are separate machines, put it behind an authenticated internal tunnel — do not open the port.

2. Persist the session outside the process

Pairing a device is a manual, interactive step. If your session state lives only in memory, every restart, deploy, or crash forces someone to re-scan a code — which means the service will be unavailable exactly when nobody is watching. Persist auth state to disk, and back that directory up like any other credential store.

3. Run it under a supervisor with restart limits

# /etc/systemd/system/wa-gateway.service
[Unit]
Description=Local WhatsApp gateway
After=network-online.target

[Service]
Type=simple
User=wagateway
WorkingDirectory=/opt/wa-gateway
ExecStart=/usr/bin/node server.js
Restart=on-failure
RestartSec=10
StartLimitBurst=5
StartLimitIntervalSec=300
Environment=NODE_ENV=production
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/wa-gateway/session

[Install]
WantedBy=multi-user.target

StartLimitBurst matters. A gateway whose session has been invalidated will fail on every start; without a limit, systemd will restart it in a tight loop and hammer the upstream service, which is the fastest way to get a number flagged. Let it fail loudly after five attempts and alert on the unit state.

The transport implementation

<?php
namespace App\Messaging;

use Illuminate\Support\Facades\Http;

class LocalGatewayTransport implements MessageTransport
{
    public function __construct(private array $config) {}

    public function send(string $to, string $body): string
    {
        $response = Http::timeout($this->config['timeout'] ?? 10)
            ->retry(2, 500, throw: false)
            ->withToken($this->config['token'])
            ->post($this->config['url'] . '/send', [
                'to'      => $to,
                'message' => $body,
            ]);

        if ($response->failed()) {
            throw new TransportException(
                'Gateway rejected message: ' . $response->status() . ' ' . $response->body()
            );
        }

        return $response->json('id', '');
    }
}

Note the explicit timeout. A messaging gateway that hangs will hold a queue worker hostage; without a timeout the default can be far longer than you think, and a single unreachable service can drain your entire worker pool.

Queue it, and rate-limit it

Never send from inside a web request. Notifications should implement ShouldQueue so a slow or dead gateway degrades into a delayed message rather than a timed-out HTTP response.

More importantly, throttle deliberately. Messaging platforms treat sudden bursts from a new sender as abuse signals, and the penalty lands on your number:

// app/Notifications/BookingConfirmed.php
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\Middleware\RateLimited;

class BookingConfirmed extends Notification implements ShouldQueue
{
    public $tries = 3;
    public $backoff = [30, 120, 600];   // seconds, escalating

    public function middleware(): array
    {
        return [new RateLimited('whatsapp')];
    }

    public function via($notifiable): array
    {
        return [WhatsAppChannel::class];
    }
}

// AppServiceProvider::boot()
RateLimiter::for('whatsapp', fn () => Limit::perMinute(20));

Twenty per minute is a starting point, not a recommendation — tune it to your volume and warm a new sender up gradually rather than starting at full rate.

What to log, and what never to log

Log this	Never log this
Provider message id	Session tokens or auth state
Recipient in masked form (+2010****9705)	Full recipient numbers in plain text
Template or notification class name	Message bodies containing OTPs or personal data
Transport latency and status code	Anything you would not want in a support ticket

Messaging logs are among the highest-sensitivity data your application produces, and they are frequently the least protected. Mask on write, not on read.

Testing without sending anything

Because the transport is an interface, tests never touch the network:

public function test_customer_is_notified_on_confirmation(): void
{
    Notification::fake();

    $booking = Booking::factory()->create();
    $booking->confirm();

    Notification::assertSentTo(
        $booking->customer,
        BookingConfirmed::class,
        fn ($n, $channels) => in_array(WhatsAppChannel::class, $channels)
    );
}

And force the safe driver in your test configuration so a misconfigured environment variable can never leak a real message from CI:

<!-- phpunit.xml -->
<env name="MESSAGING_DRIVER" value="null" force="true"/>

The force="true" attribute is the important part — without it, an existing environment variable wins and your test suite starts messaging real customers. We treat that flag as mandatory for every side-effecting service in phpunit.xml.

Migrating to the official API later

When the project graduates to the Cloud API, the work is contained:

Implement CloudApiTransport against the same interface.
Add template management, because the Cloud API requires pre-approved templates for business-initiated messages outside the customer service window. This is the one place the abstraction leaks — plain-text sends become template sends with parameters, so design toWhatsApp() to return a structured object rather than a raw string if you know this migration is coming.
Flip MESSAGING_DRIVER.

Zero changes at any call site. That is the entire return on the indirection.

Summary

Put every messaging provider behind one narrow interface before you write the second call site.
Default to a non-sending driver; make reaching a real phone something you opt into explicitly.
A local gateway is a development and internal-notification tool. Customer-facing production messaging belongs on the official API.
Bind the gateway to loopback, persist its session, supervise it with restart limits, and alert on the unit.
Queue everything, rate-limit deliberately, time out aggressively, and mask recipients in logs.

We build and operate Laravel systems with WhatsApp, SMS, and email notification layers for businesses across Egypt and the Gulf. If you are designing this seam and want a review before it hardens into forty call sites, get in touch.

FAQ

Is it safe to use an unofficial WhatsApp library in production?

For customer-facing business messaging, no. Libraries that drive the WhatsApp Web protocol from a linked device fall outside WhatsApp's business messaging terms, and the number can be restricted or banned with no support path. They are reasonable for development, integration testing, and internal notifications to your own team's numbers with consent. Production customer messaging belongs on the official WhatsApp Business Cloud API.

Why route WhatsApp through Laravel's notification system instead of calling an API directly?

Because it keeps the provider decision reversible. A notification declares what to send and through which channel; the transport underneath is bound at runtime from configuration. When pricing, terms, or provider availability change, you implement one new class and flip an environment variable instead of editing every call site in the application.

How do I stop my test suite from sending real messages?

Bind a null transport as the default and force it in phpunit.xml with force="true" on the environment variable. Without that attribute an existing environment variable takes precedence and a developer machine or CI runner with production values configured will send real messages during tests. Use Notification::fake() in feature tests so assertions run without any transport being invoked at all.

What rate should I send WhatsApp messages at?

Lower than you think, and ramped gradually. Sudden bursts from a new sender look like abuse and the penalty lands on your number. Put a queue rate limiter in front of the channel, start conservatively, warm the sender up over days rather than launching at full volume, and make the limit a configuration value so you can lower it instantly during an incident.

What should a messaging gateway never do?

It should never bind to a public interface, never log message bodies or unmasked recipient numbers, never run without a supervisor restart limit, and never be called synchronously from a web request. The first exposes credentials equivalent to a logged-in phone, the second creates a high-sensitivity data leak, the third produces restart loops that get numbers flagged, and the fourth lets a hung gateway exhaust your workers.

Originally published at web-pioneer.com.

The Laravel 500 That Logged Nothing: When Your Web PHP Runs as a Different User

Web Pioneer — Mon, 20 Jul 2026 11:37:42 +0000

A contact form on a production Laravel site stopped working. Every submission returned the same response:

{"message": "Server Error"}

No stack trace. No Sentry event. And storage/logs/laravel.log had not been written to in eleven days — its modification timestamp was frozen on the exact day the form broke.

That frozen timestamp is the whole story, but it took a while to read it correctly. The instinct is to treat an empty log as "nothing happened." It is usually the opposite: something happened to the logger.

The symptom that misleads you

An empty log during an active incident has three plausible explanations, and engineers usually check them in the wrong order:

The code path never ran. Easy to disprove — the form returned a 500, so something executed.
The log level filtered it out. Also easy — an uncaught exception logs at error, which no sane config filters.
The logger itself failed. Rarely checked, because logging feels like infrastructure that cannot fail.

It was the third one. And it failed in the most inconvenient way available.

Root cause: two different users, one log file

On this stack, PHP runs under two identities depending on how it is invoked:

Context	Runs as	Can write to a 664 file owned by the site user?
CLI (php artisan, cron, deploy scripts)	the site's system user	Yes — it is the owner
Web requests (LSAPI / PHP-FPM pool)	nobody (uid 65534)	No — not owner, not in group

This split is common on shared and cPanel-style hosting, and it is invisible during development because everything on a laptop runs as one user. It stays invisible in production too — right up until a file that the web process must write is created by the CLI process.

Which is exactly what happened. A maintenance task run from the shell recreated laravel.log with mode 664 and the site user as owner. From that moment, every web request that tried to log threw:

UnexpectedValueException: The stream or file ".../storage/logs/laravel.log"
could not be opened in append mode: Permission denied

Why the exception was completely silent

The controller looked roughly like this — and this shape is everywhere in real codebases:

try {
    Log::info('Contact form submitted', $payload);
    ContactJob::dispatch($payload);
} catch (\Throwable $e) {
    Log::error('Contact form failed: ' . $e->getMessage());
    return response()->json(['message' => 'Server Error'], 500);
}

Trace it with a dead logger:

Log::info(...) throws UnexpectedValueException — permission denied.
Control jumps to the catch.
Log::error(...) throws the same exception again, this time from inside the catch block.
Nothing catches that one. It propagates to Laravel's handler, which also tries to log, which also fails.
The client gets a bare 500 with no context anywhere.

The general lesson: a catch block that logs is only as reliable as the logger. If logging is the thing that broke, your error handling amplifies the outage instead of reporting it — and it does so in a way that destroys the evidence.

Diagnosing it when the app cannot tell you anything

The usual reflex is php artisan tinker. That failed here for an unrelated but instructive reason: the CLI PHP binary on this server did not have the Redis extension loaded, while the web SAPI did. Any attempt to boot and handle a request from the CLI blew up on the session/cache driver before reaching the real bug.

Two environments, two extension sets, two user identities. The CLI simply could not reproduce a web request.

What worked was to stop trying to simulate the web context and instead run inside it — a temporary diagnostic script in the public directory that boots the kernel, handles the target route, and prints the exception object directly:

<?php // public/diag-temp.php — DELETE IMMEDIATELY AFTER USE
require __DIR__ . '/../vendor/autoload.php';
$app = require_once __DIR__ . '/../bootstrap/app.php';
$kernel = $app->make(Illuminate\Contracts\Http\Kernel::class);

$request = Illuminate\Http\Request::create('/send-contact', 'POST', [
    'name' => 'diag', 'email' => 'diag@example.com', 'message' => 'diag',
]);

$response = $kernel->handle($request);
var_dump($response->exception);   // the thing the log never got to record

Hit it with curl, read the real exception, delete the file. The permission error appeared on the first run.

Handle with care: a script like this bypasses your normal routing and middleware entry point. Give it an unguessable filename, run it once, and remove it in the same session. Never leave one on a production host.

The fix

Three changes, smallest first:

1. Rotate the poisoned file and recreate it writable by everyone that needs it. The old log had also grown to 280 MB, which is its own problem:

mv storage/logs/laravel.log storage/logs/laravel.log.backup-$(date +%F)
touch storage/logs/laravel.log
chmod 666 storage/logs/laravel.log

2. Make Monolog create future files with the right mode. Laravel's log channels accept a permission key, and without it you inherit whatever umask happened to be in effect:

// config/logging.php
'single' => [
    'driver'     => 'single',
    'path'       => storage_path('logs/laravel.log'),
    'level'      => env('LOG_LEVEL', 'debug'),
    'permission' => 0666,   // survives log rotation and CLI-created files
],

3. Stop the catch block from being able to throw. If the logger is the failure, the handler must degrade rather than escalate:

} catch (\Throwable $e) {
    try {
        Log::error('Contact form failed', ['exception' => $e]);
    } catch (\Throwable $loggingFailure) {
        error_log('Contact form failed: ' . $e->getMessage());   // syslog fallback
    }
    return response()->json(['message' => 'Server Error'], 500);
}

error_log() writes to the SAPI's own error log, which is owned by the web server and therefore always writable by it. It is not elegant, but it is the one channel that survives when your application logger does not.

The second bug, found only after logging worked

With the log alive again, the very next submission produced a real stack trace — and a completely different bug that had been hiding behind the first one:

ErrorException: Undefined array key "url"

A queued listener read $data['url'] from the payload. That field was optional and absent on some of the forms feeding the same endpoint. One ?? null fixed it.

This is the part worth internalising: a broken logger does not hide one bug, it hides all of them. Every failure during those eleven days collapsed into the same anonymous 500.

Two operational notes that cost extra time

Queue workers cache your code. After editing a listener, the running worker keeps executing the old class until restarted. Kill it and let your supervisor or cron restart it, then re-run the failed jobs with php artisan queue:retry all.
Anything the web process writes needs web-process-writable permissions. Logs, cache files, compiled views, uploaded media. If a deploy script or a shell session creates any of them, it must set ownership and mode explicitly, or the next web request inherits a file it cannot touch.

A checklist worth stealing

Check	Command	What you want to see
Who does web PHP run as?	<?php echo get_current_user(), ' / ', posix_getuid(); via a temp file	The same identity your writable paths expect
Is the log actually being written?	ls -l --time-style=full-iso storage/logs/	An mtime from minutes ago, not weeks
Can the web user append?	stat -c '%U %G %a' storage/logs/laravel.log	Mode that includes the web process
Does CLI match web?	php -m vs a web phpinfo()	Matching extension sets, or you cannot reproduce
Is the log unbounded?	du -sh storage/logs/	Rotation configured; not hundreds of MB

Add the monitor you wish you had

The cheapest possible detection for this class of failure is a scheduled check that the log file is still being touched. If your application logs anything at all on a normal day, an mtime older than a few hours is an incident:

# alert if laravel.log has not been written in 6 hours
find /path/to/storage/logs/laravel.log -mmin +360 \
  -exec echo "WARNING: laravel.log stale" \;

Wire that into whatever already pages you. Eleven days of silent failure is not a logging problem — it is a monitoring problem that a logging problem exposed.

Takeaways

An empty log during an outage means the logger failed until proven otherwise.
CLI PHP and web PHP are frequently different users with different extensions. Reproduce bugs in the context where they occur.
Never let a catch block throw. Wrap logging calls or fall back to error_log().
Set permission explicitly on file-based log channels.
Monitor the freshness of your log file, not just its contents.

We run this pattern across the Laravel applications we maintain, alongside a stale-log check on every host. If you are hardening a production Laravel deployment and want a second pair of eyes on the failure modes that never make it into your error tracker, our team is happy to help.

FAQ

Why does Laravel return a generic 500 with no log entry at all?

Almost always because the logger itself cannot write. If Monolog cannot open the log file in append mode it throws, and if your catch block also calls Log::error() it throws a second time from inside the catch, producing an uncaught exception with no recorded trace. Check the modification time on storage/logs/laravel.log first — a frozen timestamp during an active incident points straight at file permissions.

Why does my web request behave differently from php artisan tinker?

Web PHP and CLI PHP are usually separate SAPIs. They can run as different operating-system users, load different extension sets, and read different php.ini files. On many cPanel and shared hosts the web process runs as nobody while CLI runs as the site's own user. That means CLI cannot always reproduce a web-only failure, and a file created by CLI may be unwritable by the web process.

What permissions should storage/logs/laravel.log have?

The web process must be able to append to it. Where web PHP runs as a different user from the deploy user, that in practice means mode 666 on the file and 777 on the directory, or aligning ownership so the web user's group has write access. Also set the 'permission' key on the log channel in config/logging.php so Monolog recreates the file with the correct mode after rotation.

How do I debug a Laravel 500 when logging is broken?

Run inside the web context rather than trying to simulate it. Drop a temporary PHP file into your public directory that boots the kernel, handles the target request, and dumps $response->exception, then hit it with curl and delete it immediately. Give it an unguessable name and never leave it on the server.

How can I detect this failure before users report it?

Monitor the freshness of the log file, not just its contents. A scheduled job that alerts when laravel.log has not been modified in several hours catches silent logger failures. Pair it with an external uptime check that submits a real form or hits a health endpoint, so a broken write path surfaces as an alert rather than as weeks of missing leads.

Originally published at web-pioneer.com.