DEV Community: JP

DevOps Do's and Don'ts: What I Wish Someone Told Me 10 Years Ago

JP — Sat, 28 Feb 2026 09:55:23 +0000

DevOps Do's and Don'ts: What I Wish Someone Told Me 10 Years Ago

It's 2 AM. Phone's ringing. Production's down. Users are mad. CEO's asking questions. And somewhere in your git history is a deployment that passed every test but somehow broke everything.

Yeah. Been there. More times than I'd like to admit.

I've been doing DevOps for about 10 years now, mostly as a freelancer. That means I've seen a LOT of companies, a LOT of different setups, and honestly... a LOT of the same mistakes over and over again.

This isn't some theoretical best practices guide. This is stuff I learned by screwing up, fixing it at 3 AM, and promising myself I'd never do it again. (Spoiler: I did it again. We all do.)

Let's Be Honest About What DevOps Actually Is

Before we get into it - DevOps isn't a job title. I know, I know, my LinkedIn says "DevOps Engineer" too. But technically it's supposed to be a culture where devs and ops work together instead of hating each other.

In reality though? Companies hire us to:

Keep servers running
Build CI/CD pipelines
Deal with infrastructure
Be the person who gets called at 2 AM

So let's talk about doing that without losing your mind.

DO: Automate Stuff (But Don't Go Crazy)

Here's where everyone gets it wrong. They read about DevOps, get excited, and try to automate EVERYTHING in week one.

I watched a startup spend three weeks building this insane auto-scaling Kubernetes setup with service mesh, observability, the whole nine yards. For an app with like... 50 users. Meanwhile their devs were still SSH-ing into servers to deploy code manually.

That's backwards.

What you should automate first:

Deployments - seriously, if you're still deploying manually, stop reading this and go set up GitHub Actions or something. Even a basic pipeline will save you hours every week.
Environment setup - if it takes half a day to spin up a dev environment, you're wasting time. Automate that.
Backups - this should honestly be #1. Automate your backups. Test your backups. I've seen too many "oh shit we need to restore" moments where the backups were broken.
Monitoring alerts - can't fix what you don't know is broken.

What can wait:

That complex orchestration that runs once a month
Edge cases that happen twice a year
Stuff that keeps changing (wait till it stabilizes)

Real Talk

I helped that startup I mentioned earlier. We ripped out the fancy Kubernetes setup and went with:

Docker Compose (yes, really)
GitHub Actions for CI/CD
Basic health checks

Deploy time went from 30 minutes to 3 minutes. Setup took 2 days instead of 3 weeks. They're doing fine. When they actually need to scale, we'll scale. But you don't need Kubernetes for 50 users.

DON'T: Skip Monitoring Because "We'll Add It Later"

This is probably the mistake I see most often. "We'll add monitoring once we're live."

Then you go live. Everything SEEMS fine. Three days later you realize your database has been maxed out for 48 hours and you lost customer data.

Yeah.

Minimum viable monitoring:

Infrastructure stuff:

CPU, RAM, disk space
Network latency
Is the service even up

Application stuff:

Error rates
How fast things respond
Database connection pools
Queue lengths if you use background jobs

Business stuff:

User signups
Payment success rate
Whatever matters to your business

The Cron Job Blind Spot

Here's one that gets everyone: cron jobs.

People monitor their web servers, their databases, their APIs. But scheduled jobs? Those just run in the background, silent and unmonitored.

Until they don't run. And nobody notices for a week.

I've seen:

Invoice jobs that stopped for a week (customers were NOT happy)
Database backups failing for 3 days (found out when we needed to restore)
Data sync breaking silently

The fix is stupid simple. Make your cron job ping a URL when it's done. If it doesn't ping within the expected time, get an alert.

That's literally it. One curl at the end of your script. I built CronGuard because I was tired of this exact problem. But there are other tools too. Just... monitor your cron jobs.

DO: Put Everything in Git

If it's not in git, it doesn't exist.

I mean it. Everything should be reproducible from your repo.

Infrastructure as code:

# Don't click buttons in AWS console
# Write Terraform instead

resource "aws_instance" "app" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.micro"

  tags = {
    Name = "production-app"
  }
}

Configuration management:

# Don't SSH in and edit nginx.conf
# Use Ansible

- name: Configure Nginx
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
  notify: restart nginx

Database migrations:

# Don't run random SQL scripts
# Use proper migrations

npx prisma migrate deploy

Why This Actually Matters

Client of mine had a production server die last year. Hardware failure. Completely dead.

They had backups. They had monitoring. What they DIDN'T have was any record of how that server was configured.

Two years of manual tweaks. Undocumented changes. Some guy who left 6 months ago had set up "some stuff" but nobody knew what.

Took 3 days to get back online instead of 3 hours.

If everything was in git - Terraform configs, Ansible playbooks, whatever - they could've rebuilt that server in an hour.

DON'T: Treat Production Like a Playground

"I'll just quickly fix this in prod..."

Famous last words.

I've said them. You've probably said them. We've all said them. Sometimes it works out. Sometimes you take down production for 4 hours.

When something breaks in prod:

Is it on fire RIGHT NOW? If yes, do what you gotta do. If no, keep reading.
Can you reproduce it in staging?
Fix it in a branch
Test it
Code review
Deploy through your normal pipeline
Verify it worked
Write down what happened so it doesn't happen again

For actual emergencies:

Yeah okay sometimes you DO need to hotfix prod. When you do:

Write down EXACTLY what you changed
Tell your team what you did
Create a ticket to do it properly later
Add a test so it can't happen again

Feature Flags Are Your Friend

Want to deploy without the stress? Feature flags.

if (featureFlags.newCheckout) {
  return <NewCheckoutFlow />
} else {
  return <OldCheckoutFlow />
}

Deploy the new code. Turn it on for yourself. Then a few internal users. Then 10% of traffic. Then everyone.

Something breaks? Flip the switch back. No emergency deploys. No 3 AM rollbacks. Just... flip it off.

DO: Have a Rollback Plan (And Actually Test It)

Every deployment should be reversible. Every single one.

Database migrations need down migrations:

# Going up
ALTER TABLE users ADD COLUMN email_verified BOOLEAN;

# Going back down
ALTER TABLE users DROP COLUMN email_verified;

Keep your old docker images. Tag releases properly. Document how to rollback.

And here's the thing - TEST YOUR ROLLBACKS.

Don't wait till production is on fire to find out your rollback procedure doesn't work. Do it in staging. Time how long it takes. Fix the slow parts.

We do a quarterly drill where we deploy something, intentionally break it, and practice rolling back. Sounds paranoid but it's saved us multiple times.

DON'T: Skip Security Because It's Annoying

"We'll fix the security issues after launch."

No you won't.

Security isn't a feature you add later. It's a requirement.

Don't do this:

# Secrets in git
DATABASE_URL=postgresql://admin:password123@prod-db.com/myapp

Do this:

# Environment variables
DATABASE_URL=${DATABASE_URL}

Use AWS Secrets Manager, HashiCorp Vault, or at minimum just environment variables. But get secrets out of your code.

Don't give everything admin access:

// Bad
{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

// Good
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::my-bucket/*"
}

Least privilege. Always.

Use HTTPS everywhere. Even in dev. mkcert makes it easy to get trusted local certificates. In production, Let's Encrypt is free.

And monitor your SSL certificates so they don't expire. (Yeah I built CertGuard for this too. Expired certificates are embarrassing.)

Rate limit your APIs. Even internal ones. You'll thank me later.

Don't run containers as root. Just don't.

USER node

DO: Write Things Down

"The code is self-documenting" is a lie we tell ourselves.

Future you (or the poor person who replaces you) needs to understand how this works.

Document:

How services talk to each other
Database schemas
External dependencies
How to deploy
How to rollback
How to debug common issues
Where the logs are

Keep it close to the code. README files work great. Architecture Decision Records are good too. Just... write it down.

And keep it updated. Outdated docs are worse than no docs.

DON'T: Build For Scale You Don't Have

"We need Kubernetes from day one."

No you don't.

Most apps don't need Kubernetes. Most databases don't need sharding. Most APIs don't need Redis caching.

Scale when you have data showing it's actually a problem.

Not when you THINK it might be a problem someday. When your monitoring shows:

Response times above your SLA
Database queries timing out
Servers hitting resource limits

Start simple:

One server handles 10k users easily
SQLite handles millions of rows
You don't need a CDN till you have global users

I've watched startups spend 6 months building distributed systems for apps that never got past 100 users. That time could've been spent building features people actually wanted.

DO: Break Things (In Staging)

Staging should be where things break safely.

Kill services randomly. Simulate slow databases. Fill up the disk. Cut network connections.

See what breaks. Fix it. Repeat.

You don't need Netflix's fancy Chaos Monkey. Just manually break stuff and see what happens.

DON'T: Ignore Technical Debt

"We'll refactor this later."

You won't.

That hacky fix becomes critical infrastructure. That "temporary" workaround is still there 2 years later.

Managing tech debt:

Create tickets for known issues
Estimate the actual cost (time + risk)
Dedicate 20% of your sprint to fixing it
Don't let it pile up

And explain to stakeholders WHY you're "not building features." Show them the cost of NOT fixing the debt.

What It All Comes Down To

Strip away all the fancy tools and methodologies and it's really just:

Automate the painful stuff
Monitor what matters
Make it reproducible
Plan for failure
Secure by default
Write down the why
Scale when needed (not before)
Pay down your debt

Final Thoughts

DevOps isn't about having the coolest tools or the most complex setup. It's about:

Shipping faster
Breaking less
Recovering quicker when you do break
Actually sleeping at night

Start small. Automate ONE thing this week. Add ONE monitor. Write ONE runbook. Ship ONE improvement.

Do that every week and in a year you'll be way ahead of teams still "planning their DevOps transformation."

Trust me. I've been doing this for 10 years and I'm still learning. We all are. That's kind of the point.

About me: I'm Jean-Pierre, a freelance DevOps engineer in the Netherlands. I help teams build infrastructure that doesn't suck. More stuff on my blog at FreelyIT.nl.

Got DevOps war stories? Drop them in the comments. I want to hear what you've learned the hard way too.

Why Your SSL Certificate Will Expire at 3 AM on a Saturday (And How to Stop It)

JP — Thu, 19 Feb 2026 14:20:38 +0000

Let me tell you about the worst Saturday morning of my career.

3:17 AM. My phone explodes with alerts. The main production site is down. API is down. Mobile app can't connect. Customer support is getting flooded.

The culprit? An expired SSL certificate.

Not a server crash. Not a database failure. Not a DDoS attack. A certificate that expired at 3:00 AM on a Saturday morning, taking down a multi-million dollar e-commerce platform.

The certificate had been valid for a year. We knew it would expire. There were emails. There were reminders. But between team changes, inbox filters, and "I'll do it tomorrow" syndrome, it slipped through the cracks.

That incident cost the company six figures in lost revenue and taught me that SSL certificate monitoring isn't optional -- it's survival.

The Problem with SSL Certificates

SSL/TLS certificates are one of those things that work perfectly until they don't. And when they don't, they fail catastrophically:

Silent expiry -- No warning. Your site just stops working.
Browser warnings -- "This connection is not private" scares away customers
API breakage -- Mobile apps and integrations stop working
Zero grace period -- The second it expires, everything breaks

Even worse, modern infrastructure has dozens or hundreds of certificates:

Main domain (example.com)
WWW subdomain (www.example.com)
API endpoints (api.example.com)
Admin panels (admin.example.com)
CDN origins
Wildcard certificates (*.example.com)
Internal services
Load balancers
VPNs

Each one is a ticking time bomb.

Why Certificates Expire (And Why That's Good)

Before we dive into monitoring, let's understand why certificates expire:

Security -- Shorter validity periods limit damage if a private key is compromised
Cryptographic aging -- Today's "secure" algorithms become tomorrow's vulnerabilities
Ownership changes -- Domains change hands, organizations restructure

Certificate authorities (CAs) have been steadily reducing validity periods:

2015: 5 years maximum
2018: 2 years maximum (825 days)
2020: 1 year maximum (398 days)
2024: Proposals for 90 days or less

Let's Encrypt already issues 90-day certificates by default. The industry is moving toward automated, short-lived certificates -- which means monitoring becomes even more critical.

The Anatomy of an SSL Certificate

To monitor certificates effectively, you need to understand what you're monitoring:

# Check a certificate with OpenSSL
openssl s_client -connect example.com:443 -servername example.com < /dev/null 2>/dev/null | openssl x509 -noout -dates

# Output:
notBefore=Jan 15 00:00:00 2026 GMT
notAfter=Apr 15 23:59:59 2026 GMT

Key fields to monitor:

Subject -- Who the cert is issued to (CN=example.com)
Issuer -- Who issued it (Let's Encrypt, DigiCert, etc.)
Valid From (notBefore)
Valid Until (notAfter) -- THE critical field
Subject Alternative Names (SANs) -- Additional domains covered
Signature Algorithm -- Is it still secure?

The SAN Trap

Here's a gotcha that bit me: Subject Alternative Names (SANs).

Modern certificates often cover multiple domains:

CN=example.com
SAN=example.com, www.example.com, api.example.com

If you only check the primary domain but monitor from a different SAN, you might miss expiry warnings. Always verify the certificate covers the specific hostname you're checking.

Building a Certificate Monitor

The simplest version is a shell script:

#!/bin/bash
DOMAIN="example.com"
PORT=443
ALERT_DAYS=30

# Get expiry date
EXPIRY=$(echo | openssl s_client -servername "$DOMAIN" -connect "$DOMAIN:$PORT" 2>/dev/null | \
         openssl x509 -noout -enddate | cut -d= -f2)

# Convert to epoch timestamp
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( ($EXPIRY_EPOCH - $NOW_EPOCH) / 86400 ))

if [ $DAYS_LEFT -lt $ALERT_DAYS ]; then
    echo "Certificate for $DOMAIN expires in $DAYS_LEFT days!"
    # Send alert (email, Slack, etc.)
fi

Run this daily via cron:

0 9 * * * /usr/local/bin/check-cert.sh

But this has problems:

Only checks when the cron runs
No historical data
Doesn't handle multiple domains
No retry logic for network blips
Alert fatigue (you get the same alert every day)

Let's level up.

A Production-Grade Solution

Here's a Node.js version that handles edge cases:

const tls = require('tls');
const https = require('https');

async function checkCertificate(hostname, port = 443) {
    return new Promise((resolve, reject) => {
        const options = {
            host: hostname,
            port: port,
            servername: hostname, // SNI support
            rejectUnauthorized: false, // Check even invalid certs
        };

        const socket = tls.connect(options, () => {
            const cert = socket.getPeerCertificate();

            if (!cert || !cert.valid_to) {
                socket.end();
                return reject(new Error('No certificate found'));
            }

            const validTo = new Date(cert.valid_to);
            const validFrom = new Date(cert.valid_from);
            const now = new Date();
            const daysLeft = Math.floor((validTo - now) / (1000 * 60 * 60 * 24));

            const result = {
                hostname: hostname,
                subject: cert.subject,
                issuer: cert.issuer,
                validFrom: validFrom,
                validTo: validTo,
                daysLeft: daysLeft,
                isValid: now >= validFrom && now <= validTo,
                subjectAltNames: cert.subjectaltname,
                serialNumber: cert.serialNumber,
                fingerprint: cert.fingerprint
            };

            socket.end();
            resolve(result);
        });

        socket.on('error', (err) => {
            reject(err);
        });

        socket.setTimeout(10000, () => {
            socket.destroy();
            reject(new Error('Connection timeout'));
        });
    });
}

// Usage
checkCertificate('example.com')
    .then(cert => {
        console.log(`Certificate for ${cert.hostname}:`);
        console.log(`  Expires: ${cert.validTo}`);
        console.log(`  Days left: ${cert.daysLeft}`);
        console.log(`  Issuer: ${cert.issuer.O}`);

        if (cert.daysLeft < 30) {
            sendAlert(cert);
        }
    })
    .catch(err => console.error('Check failed:', err.message));

Python Alternative

import ssl
import socket
from datetime import datetime, timedelta

def check_certificate(hostname, port=443):
    context = ssl.create_default_context()

    with socket.create_connection((hostname, port), timeout=10) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert = ssock.getpeercert()

            # Parse expiry
            not_after = datetime.strptime(cert['notAfter'], '%b %d %H:%M:%S %Y %Z')
            days_left = (not_after - datetime.now()).days

            return {
                'hostname': hostname,
                'subject': dict(x[0] for x in cert['subject']),
                'issuer': dict(x[0] for x in cert['issuer']),
                'valid_to': not_after,
                'days_left': days_left,
                'serial_number': cert['serialNumber'],
                'san': cert.get('subjectAltName', [])
            }

# Usage
cert = check_certificate('example.com')
print(f"Certificate expires in {cert['days_left']} days")

if cert['days_left'] < 30:
    send_alert(cert)

Alert Thresholds: A Practical Strategy

Don't just alert at "30 days". Use multiple thresholds with escalating urgency:

60 days Informational (Slack channel, low priority)
30 days Warning (Email to team)
14 days Urgent (Email + Slack mention)
7 days Critical (SMS/PagerDuty to on-call)
3 days Emergency (Wake everyone up)

function getAlertLevel(daysLeft) {
    if (daysLeft <= 3) return 'EMERGENCY';
    if (daysLeft <= 7) return 'CRITICAL';
    if (daysLeft <= 14) return 'URGENT';
    if (daysLeft <= 30) return 'WARNING';
    if (daysLeft <= 60) return 'INFO';
    return 'OK';
}

function sendAlert(cert) {
    const level = getAlertLevel(cert.daysLeft);

    const message = `${level}: Certificate for ${cert.hostname} expires in ${cert.daysLeft} days (${cert.validTo})`;

    switch(level) {
        case 'EMERGENCY':
            sendSMS(message);
            sendEmail(message);
            sendSlack(message);
            break;
        case 'CRITICAL':
            sendEmail(message);
            sendSlack(message, '@channel');
            break;
        case 'URGENT':
            sendEmail(message);
            sendSlack(message);
            break;
        case 'WARNING':
            sendEmail(message);
            break;
        case 'INFO':
            sendSlack(message); // No email spam
            break;
    }
}

Edge Cases That Will Bite You

1. Certificate Chains

A valid server cert with an expired intermediate certificate will fail:

Root CA (trusted)
  Intermediate CA (EXPIRED!)
    Server Cert (valid)

Always check the entire chain:

const cert = socket.getPeerCertificate(true); // true = include chain
const chain = [cert];
let current = cert;

while (current.issuerCertificate && current !== current.issuerCertificate) {
    chain.push(current.issuerCertificate);
    current = current.issuerCertificate;
}

// Check each cert in the chain
chain.forEach((c, i) => {
    const validTo = new Date(c.valid_to);
    const daysLeft = Math.floor((validTo - Date.now()) / 86400000);
    console.log(`  [${i}] ${c.subject.CN} - ${daysLeft} days left`);
});

2. Wildcard Certificates

A wildcard cert (*.example.com) covers:

api.example.com
www.example.com
admin.example.com

But NOT:

example.com (root domain)
api.staging.example.com (nested subdomain)

Monitor both the wildcard AND the root domain separately.

3. SNI (Server Name Indication)

Modern servers host multiple domains on one IP. Without SNI, you might check the wrong certificate:

# Wrong (no SNI)
openssl s_client -connect 192.0.2.1:443

# Right (with SNI)
openssl s_client -connect 192.0.2.1:443 -servername example.com

Always specify -servername or the equivalent in your programming language.

4. Load Balancers and CDNs

You have:

Origin server cert (your server)
Load balancer cert (AWS ELB, Cloudflare, etc.)
CDN edge cert (Cloudflare, Fastly, etc.)

Monitor the certificate that users actually see, not just your origin. Check from outside your infrastructure.

5. Internal Services

Don't forget:

Database SSL connections
Redis TLS
Internal APIs
VPNs
LDAP/LDAPS
SMTP with STARTTLS

These often use self-signed or internal CA certs that expire quietly.

Let's Encrypt: Automation & Monitoring

Let's Encrypt changed the game with free, automated certificates. But automation doesn't mean "set and forget".

ACME Renewal Monitoring

Certbot (the Let's Encrypt client) auto-renews certs. But what if renewal fails?

# Check last renewal attempt
journalctl -u certbot.timer --since "7 days ago" | grep -i error

# Or parse certbot's logs
grep "Renewal failed" /var/log/letsencrypt/letsencrypt.log

Better: Monitor the certificate expiry and renewal success separately:

const fs = require('fs');
const path = require('path');

function checkLetsEncryptRenewal(domain) {
    const renewalConfig = `/etc/letsencrypt/renewal/${domain}.conf`;

    if (!fs.existsSync(renewalConfig)) {
        return { error: 'Renewal config not found' };
    }

    const stats = fs.statSync(renewalConfig);
    const daysSinceUpdate = (Date.now() - stats.mtime) / 86400000;

    // LE renews certs older than 60 days
    if (daysSinceUpdate > 60) {
        return { 
            warning: `Renewal config not updated in ${Math.floor(daysSinceUpdate)} days`
        };
    }

    return { ok: true };
}

Common Let's Encrypt Failures

HTTP-01 challenge blocked -- Firewall rules, CDN config
DNS-01 fails -- API rate limits, DNS propagation
Webroot permissions -- Certbot can't write to .well-known/acme-challenge/
Rate limits hit -- Too many renewals in a week

Monitor renewal logs and alert on failure patterns.

Dashboard: Making Data Actionable

Raw alerts are good. A dashboard is better. Track:

Certificates expiring soon (sorted by days left)
Recently renewed (confirm automation is working)
Issuer distribution (Are you still using that old CA?)
Expiry timeline (visual calendar of upcoming expirations)

Simple HTML dashboard:

async function generateDashboard(domains) {
    const checks = await Promise.all(
        domains.map(d => checkCertificate(d).catch(err => ({ 
            hostname: d, 
            error: err.message 
        })))
    );

    // Sort by days left (ascending)
    checks.sort((a, b) => (a.daysLeft || 999) - (b.daysLeft || 999));

    const html = `
    <h1>SSL Certificate Dashboard</h1>
    <table>
        <tr><th>Domain</th><th>Days Left</th><th>Expires</th><th>Issuer</th></tr>
        ${checks.map(c => `
            <tr class="${getRowClass(c.daysLeft)}">
                <td>${c.hostname}</td>
                <td>${c.daysLeft || 'ERROR'}</td>
                <td>${c.validTo?.toLocaleDateString() || '-'}</td>
                <td>${c.issuer?.O || '-'}</td>
            </tr>
        `).join('')}
    </table>
    `;

    return html;
}

function getRowClass(daysLeft) {
    if (!daysLeft) return 'error';
    if (daysLeft < 7) return 'critical';
    if (daysLeft < 14) return 'urgent';
    if (daysLeft < 30) return 'warning';
    return 'ok';
}

Pro Tips from Production

1. Check from Multiple Locations

Your cert might be valid from your office but expired on your CDN edge nodes:

const locations = [
    'us-east.probe.example.com',
    'eu-west.probe.example.com',
    'ap-southeast.probe.example.com'
];

for (const probe of locations) {
    const cert = await checkCertificateViaProxy(probe, 'example.com');
    // Compare results
}

2. Track Certificate Changes

Alert when certificates change unexpectedly (potential security issue):

const lastKnownFingerprint = db.get('cert_fingerprint', domain);
const currentFingerprint = cert.fingerprint;

if (lastKnownFingerprint && lastKnownFingerprint !== currentFingerprint) {
    sendSecurityAlert(`Certificate changed for ${domain}!`);
}

db.set('cert_fingerprint', domain, currentFingerprint);

3. Test Renewal Before It's Urgent

Don't wait until 7 days before expiry to test renewal. Run a dry-run monthly:

certbot renew --dry-run --quiet || echo "Renewal will fail!"

4. Document Your Renewal Process

When the cert expires at 3 AM and the person who set it up left the company 6 months ago, you need documentation:

# SSL Certificate Renewal - example.com

**Provider:** Let's Encrypt  
**Renewal method:** Certbot with HTTP-01 challenge  
**Auto-renewal:** Yes (certbot.timer systemd service)  
**Manual renewal:** `sudo certbot renew --force-renewal`  
**Webroot:** `/var/www/example.com/`  
**On-call:** ops@example.com  
**Last manual renewal:** 2026-01-15 by @john

Tools Worth Knowing

SSL Labs -- Free online checker (https://www.ssllabs.com/ssltest/)
testssl.sh -- Comprehensive SSL/TLS scanner
Certbot -- Let's Encrypt client
cert-manager -- Kubernetes cert automation
AWS Certificate Manager -- Managed certs for AWS

And of course, dedicated monitoring services exist (I'll leave product names out of this, but they're worth Googling).

The Post-Incident Checklist

After that 3 AM disaster, here's what we implemented:

Automated monitoring for all domains (check daily)
Multi-threshold alerts (60/30/14/7/3 days)
Multiple alert channels (email, Slack, SMS)
Dashboard showing all certs at a glance
Documentation for manual renewal
Automated renewal (Let's Encrypt)
Renewal monitoring (separate from expiry monitoring)
On-call runbook for cert emergencies
Monthly dry-run renewal tests
Certificate inventory (every cert, every service)

We've had zero certificate-related outages in the two years since.

Conclusion

SSL certificates will expire. It's not if, it's when. The question is: will you know about it 30 days in advance, or at 3 AM on a Saturday when everything is on fire?

Monitoring isn't hard:

Check expiry dates regularly (daily is fine)
Alert with escalating urgency
Automate renewal where possible
Monitor renewal success separately
Document everything

The 3 AM outage is 100% preventable. Don't let it happen to you.

Have you had a certificate disaster story? Share it in the comments. Misery loves company, and we all learn from each other's mistakes.

Building certificate monitoring? What challenges are you facing? Drop a comment and let's discuss.

The Hidden Costs of Skipping Code Reviews (And How Different Teams Handle Them)

JP — Tue, 17 Feb 2026 10:17:58 +0000

A practical look at code review approaches, what they catch, and why most teams still get it wrong.

Every developer has been there. You ship a feature, it works in staging, you deploy on a Friday afternoon, and by Monday morning you're dealing with a production incident that a basic code review would have caught in fifteen minutes.

Code reviews aren't just about catching bugs. They're about knowledge transfer, consistency, and -- increasingly important -- security. But not all code reviews are created equal. Different types catch different problems, and most teams pick one approach and call it a day.

Let's break down the main types of code review, what they're actually good at, and where they fall short.

1. The Classic Peer Review

The most common type. A colleague reads your code, leaves comments, you argue in the PR thread about variable naming for 45 minutes, and eventually someone approves it.

What it's good at:

Logic errors and edge cases
Readability and maintainability
Knowledge sharing between team members
Catching obvious architectural mistakes

Where it falls short:
Peer reviews are only as good as your team. If nobody on your team knows the OWASP Top 10 by heart, SQL injection vulnerabilities will sail through undetected. Same goes for subtle memory leaks, race conditions, or dependency vulnerabilities. Humans are great at reading business logic, but terrible at consistently catching security issues under deadline pressure.

There's also the familiarity problem: if you've been staring at the same codebase for two years, you stop seeing the things that are obviously wrong to a fresh set of eyes.

2. Automated Static Analysis

Tools like ESLint, SonarQube, ReSharper, or Semgrep run against your code and flag issues automatically. These are fantastic â€” and completely table stakes at this point.

What it's good at:

Code style and formatting consistency
Known anti-patterns
Basic security rules (hardcoded secrets, unsafe functions)
Cyclomatic complexity warnings

Where it falls short:
Static analysis tools generate a lot of noise. Teams learn to ignore the warnings, which defeats the purpose. They also can't understand context: a tool might flag a eval() call without understanding that it's intentionally sandboxed and reviewed.

More importantly, static analysis works at the file level. It doesn't understand your architecture, your deployment environment, or how your dependencies interact with each other.

3. Security-Focused Code Audits

This is where things get serious. Security audits go beyond "does this look clean" and ask "can this be exploited?"

A proper security audit covers:

OWASP Top 10 vulnerabilities:

Injection attacks (SQL, NoSQL, command injection)
Broken authentication
Sensitive data exposure
XML External Entities (XXE)
Broken access control
Security misconfiguration
Cross-Site Scripting (XSS)
Insecure deserialization
Using components with known vulnerabilities
Insufficient logging and monitoring

What makes security audits different is the adversarial mindset. You're not asking "does this work?" -- you're asking "how could someone break this?" That requires a completely different way of reading code.

The challenge: real security expertise is expensive and hard to find. Most teams schedule a security audit once a year (if that) and hope for the best between audits. Meanwhile, they're shipping new code every week.

4. Dependency Audits

This one gets overlooked constantly, and it's becoming one of the most critical review types.

Modern applications import hundreds of packages. Each one is a potential attack vector. The log4shell vulnerability from 2021 brought down major enterprises because of a transitive dependency nobody knew was there. The event-stream npm compromise injected malicious code into a popular package. This is not theoretical risk.

A proper dependency audit checks:

Known CVEs (Common Vulnerabilities and Exposures) in your dependencies
Outdated packages with security patches available
License compliance (GPL in a commercial product is a legal problem)
Transitive dependencies (what your packages depend on)
Abandoned packages with no maintenance

Most teams run npm audit or pip check occasionally and consider that sufficient. It's not. These tools only catch known vulnerabilities that have been formally reported. Zero-days and newly discovered issues require more comprehensive scanning.

5. Performance Reviews

Separate from security, performance reviews look at your code through a completely different lens.

What gets checked:

N+1 query problems (the classic database performance killer)
Unoptimized loops and algorithmic complexity
Memory leaks and resource cleanup
Unnecessary re-renders in frontend applications
Missing database indexes
Inefficient data structures

Performance issues are notoriously hard to catch in code review because they often only manifest at scale. Code that works fine with 100 users can grind to a halt at 10,000. A good performance review combines static code analysis with understanding of production load patterns.

6. Infrastructure and Container Reviews

As teams move to containerized deployments, a whole new category of review has emerged.

Docker images and Kubernetes configurations introduce their own class of vulnerabilities:

Running containers as root (bad idea)
Exposing unnecessary ports
Missing resource limits (hello, runaway processes)
Hardcoded secrets in Dockerfiles or environment variables
Overly permissive RBAC configurations
Images built from unverified base images

This is specialized knowledge that most developers don't have. And yet, most teams ship Kubernetes configs that have never been reviewed by someone who knows what they're looking at.

The Real Problem: Review Gaps

Here's the uncomfortable truth about most engineering teams: you're probably doing peer review reasonably well, you have some static analysis in your CI pipeline, and you've run npm audit recently. But you have large gaps in:

Systematic security auditing (not just occasional penetration tests)
Thorough dependency vulnerability scanning
Performance review at the architecture level
Infrastructure security review

These gaps exist for a predictable reason: expertise is expensive, time is scarce, and manual comprehensive reviews don't scale with shipping velocity.

Where AI Changes the Calculus

This is where the landscape is genuinely shifting. AI-powered code review tools can now perform comprehensive audits that would previously require a team of specialists.

One tool worth looking at is ScanMyCode -- it runs AI-powered audits covering all the categories above: code quality, security (OWASP Top 10), dependency vulnerabilities (CVE lookup and license compliance), performance bottlenecks, and Docker/K8s configuration. Results come back within 24 hours, at a fraction of the cost of a human consultant.

This doesn't replace human expertise -- but it closes the gap between expensive periodic audits and the continuous shipping reality that modern teams live in. Think of it as a comprehensive first pass that surfaces the issues worth a human's attention.

Building a Review Culture That Actually Works

The teams with the strongest review cultures share a few common traits:

They layer their reviews. Automated tools catch what they're good at. Peers review logic and context. Dedicated audits (manual or AI-assisted) catch what humans miss under deadline pressure.

They review infrastructure as code. Dockerfiles, Kubernetes manifests, and CI/CD configurations go through the same rigor as application code. Because they should.

They treat dependencies as first-class citizens. Regular dependency audits aren't a one-time event â€” they're part of the development lifecycle.

They create psychological safety. Code review culture dies when people feel attacked in PR threads. Comments should be about the code, not the author. "This query could cause an N+1 issue" is useful. "Why would you write it this way?" is not.

They follow up on findings. A code review that surfaces issues but doesn't result in fixes is just documentation of technical debt. The value is in the action that follows.

The Bottom Line

Code review is not a checkbox. It's a discipline that, done well, catches the kind of issues that make the news â€” and the ones that quietly drain performance and maintainability over years.

The teams that get it right don't rely on any single approach. They build layered review processes that match their risk profile, shipping velocity, and resource constraints. And increasingly, they're supplementing human review with AI-powered auditing to close the gaps that manual processes inevitably leave.

Start with whatever you're not doing. Add a dependency audit to your CI pipeline. Schedule a security review for your most critical service. Run your Docker configuration through something that actually understands container security.

The breach you prevent is the one you never hear about.

Have a codebase that needs a thorough once-over? ScanMyCode does comprehensive AI-powered audits covering security, performance, dependencies, and Docker/K8s -- starting from €39.

I Built a Cron Job Monitor Because Silence Kills Production

JP — Mon, 16 Feb 2026 14:38:40 +0000

I Built a Cron Job Monitor Because Silence Kills Production

Three months ago, my client's daily database backup hadn't run in 11 days. The cron job was still scheduled. No errors in the logs. The monitoring dashboard was green. Everything looked fine.

Until someone tried to restore from a backup that didn't exist.

That's when I learned the hard way: traditional monitoring is terrible at catching things that don't happen.

The Problem with Traditional Monitoring

Most monitoring tools are great at telling you when something breaks:

Server is down? Alert.
API returns 500? Alert.
Disk is full? Alert.

But what about when your nightly backup job silently stops running? Or your data sync task fails to start? Or your cleanup script never executes?

Silence.

Traditional monitoring watches for events. Cron jobs that don't run don't generate events. They just... don't happen. And you don't find out until it's too late.

The "Dead Man's Switch" Approach

After the backup incident, I started thinking differently about monitoring scheduled tasks. Instead of watching for failures, what if we watched for missing successes?

The concept is simple:

Your cron job pings an endpoint when it runs successfully
If the ping doesn't arrive within the expected window, you get alerted
Silence = failure

It's like a dead man's switch on a train. If the operator stops pressing the button (the "heartbeat"), the train stops. If your job stops checking in, you get alerted.

Building CronGuard

I built CronGuard to solve this for myself and my clients. The core idea is dead simple:

Every monitor gets a unique ping URL. Your job hits that URL when it completes. If we don't get a ping within the expected schedule, we alert you.

Here's what a basic integration looks like:

#!/bin/bash

# Your backup script
pg_dump mydb > backup.sql
tar -czf backup-$(date +%Y%m%d).tar.gz backup.sql
aws s3 cp backup-$(date +%Y%m%d).tar.gz s3://my-backups/

# Ping CronGuard when done
curl -fsS https://cronguard.app/api/ping/your-monitor-id

That's it. If the backup fails, the ping doesn't happen. If the cron job stops running, the ping doesn't happen. If the server dies, the ping doesn't happen.

In all cases: you get alerted.

The Technical Choices That Mattered

1. Keep the Ping Endpoint Stupid Simple

The ping endpoint is just an HTTP GET. No authentication required (the URL itself is the secret). No JSON body. No headers. Just:

curl https://cronguard.app/api/ping/abc123

Why? Because I wanted it to work from anywhere:

Bash scripts
Python scripts
Inside Docker containers
Lambda functions
GitHub Actions
Cron jobs on a Raspberry Pi

If you can make an HTTP request, you can monitor your job. No SDK. No dependencies. No authentication dance.

2. Grace Periods Are Critical

Here's a mistake I made early: treating cron schedules as exact.

If a job is scheduled for 03:00, and runs at 03:02, that's not a failure. Servers reboot. Tasks queue. Execution time varies.

CronGuard uses grace periods:

Daily job at 03:00? Alert if no ping by 04:00.
Hourly job? Alert if no ping after 70 minutes.
Every 5 minutes? Alert after 7 minutes.

This eliminated false positives and made the system actually useful.

3. The First Ping Problem

When you create a new monitor, you haven't sent your first ping yet. Should the system immediately alert that the job is "down"?

No. That's annoying.

Solution: monitors are in a "waiting" state until they receive their first ping. After that, the clock starts ticking.

4. Recovery Notifications Matter

Early version: alert when job stops checking in. Done.

Reality: you also want to know when it starts working again.

Now CronGuard sends recovery notifications too:

"Backup job missed check-in (expected by 04:00)"
"Backup job recovered (checked in at 04:15)"

This helps you confirm your fix actually worked.

Lessons from Running It in Production

Lesson 1: Cron Jobs Fail More Than You Think

After deploying CronGuard for my own infrastructure and a handful of clients, I learned that cron jobs are fragile.

Things I've seen cause silent failures:

Server rebooted, cron daemon didn't restart properly
Environment variables missing in cron context
Disk full, job can't write temp files
Database credentials rotated, job can't connect
Dependencies updated, script breaks
Path issues (/usr/local/bin not in cron's PATH)

None of these would trigger traditional monitoring. All of them stopped critical jobs from running.

Lesson 2: Most People Don't Monitor Their Cron Jobs At All

I thought everyone had sophisticated monitoring setups. Turns out, most developers and small teams just... hope their cron jobs work.

They schedule it once, see it run once, and assume it'll run forever. Until it doesn't.

Lesson 3: The Real Value Is Peace of Mind

The best feedback I got wasn't "this caught a bug." It was "I sleep better knowing I'll find out if something stops working."

That's the real value: confidence that silence won't kill production.

When Dead Man's Switch Monitoring Makes Sense

This approach isn't for everything. Here's when it works:

Scheduled tasks: backups, data syncs, cleanup jobs, report generation

Async workers: if you expect jobs to complete regularly

Periodic data ingestion: RSS feeds, API polling, scraping

Real-time services: use traditional uptime monitoring

Event-driven systems: if execution is unpredictable

Alternative Approaches (and Why I Didn't Use Them)

Option 1: Log parsing

Parse cron output for errors. Problem: no output = no detection.

Option 2: Process monitoring

Check if the process is running. Problem: cron spawns processes, they finish.

Option 3: File timestamps

Check modification time on output files. Problem: requires filesystem access, brittle.

Option 4: Traditional uptime monitoring

Ping the endpoint yourself. Problem: doesn't tell you if the job ran, just if the endpoint responds.

Dead man's switch monitoring is the only approach that directly answers: "Did my job complete successfully?"

Try It Yourself

I run CronGuard as a free service for basic monitoring (5 monitors, 5-minute checks). If you've got cron jobs, backups, or scheduled tasks you care about, give it a shot:

cronguard.app

You can literally start monitoring in 30 seconds:

Create a monitor
Copy the ping URL
Add curl -fsS <url> to the end of your script

That's it. Now you'll know if it stops working.

The Bottom Line

Traditional monitoring watches for things that happen. But some of the most critical failures are things that don't happen.

If you've got scheduled tasks keeping your infrastructure alive, you need to monitor for silence.

Because in production, silence kills.

Questions? Running into silent failures with your own infrastructure? Drop a comment. I'd love to hear your war stories about cron jobs that stopped working and how long it took to notice.

Built something similar? Using a different approach? Let me know. I'm always curious how other teams solve this.