DEV Community: Stephen Souza

Shopify fired the webhook. My server never processed it. Here's how I catch that now.

Stephen Souza — Wed, 27 May 2026 05:59:08 +0000

Shopify's webhook delivery is reliable. That's not the problem.

The problem is what happens after the webhook lands on your side.

Shopify fires orders/create. Your Flask endpoint receives it. Returns 200. Shopify marks it delivered. And somewhere between that 200 and the database write, something silently fails. Order never recorded. No error thrown. No alert fired. Customer emails you 3 hours later asking where their order confirmation is.

That's the webhook processing gap — and it's invisible to every standard monitoring tool because nothing failed by definition.

This is the Flask boilerplate I built to close it.

What this monitors

Order created — real-time alert with amount, items, customer
Order paid — payment confirmation with amount
Order cancelled — immediate warning with cancellation reason
Refund issued — instant alert with refund amount
Checkout started — abandoned checkout signal

Every event sends actual numeric amounts in the meta object. NotiLens ML reads these to learn your revenue baseline automatically — your normal Wednesday order volume, your typical refund rate, your checkout-to-order conversion pattern. When reality diverges from baseline, the anomaly alert fires. No thresholds to configure.

The broader concept

This boilerplate is Shopify-specific but the pattern applies to any webhook source — Stripe, GitHub, custom systems, internal services.

The concept is simple:

Receive the webhook
Verify the signature — confirms it's genuinely from the source
Send a notification immediately — you know it arrived and was processed
Send actual business metrics in the meta — enables ML baseline learning
Use force_send=True for critical events — bypasses ML, fires immediately regardless The gap this closes: webhook delivered on the sender's side, processed on your side, NotiLens knows both happened. If orders/create webhooks stop arriving during business hours — silence alert fires. That's the "webhook processing silently stopped" detection that Shopify's own dashboard will never show you.

Setup

Step 1 — Install dependencies

pip install flask notilens

Step 2 — Get credentials

Shopify webhook secret — Shopify Dashboard → Settings → Notifications → Webhooks → Signing secret
NotiLens token + secret — app.notilens.com → Create New Topic → Token + Secret

Step 3 — Configure

SHOPIFY_WEBHOOK_SECRET = "YOUR_SHOPIFY_WEBHOOK_SECRET"
NOTILENS_TOKEN         = "YOUR_NOTILENS_TOKEN"
NOTILENS_SECRET        = "YOUR_NOTILENS_SECRET"
APP_NAME               = "shopify-monitor"
PORT                   = 5000

Step 4 — Run

python notilens-shopify-webhook-monitor.py

Step 5 — Expose publicly

Local development:

ngrok http 5000

Production — deploy to Railway, Render, or any server. Your webhook endpoint:

POST https://your-domain.com/webhook/shopify

Step 6 — Register in Shopify

Shopify Dashboard → Settings → Notifications → Webhooks → Add webhook → paste your endpoint URL → select events.

The code — section by section

Signature verification

def verify_shopify_signature(payload: bytes, hmac_header: str) -> bool:
    """Verify webhook is genuinely from Shopify"""
    digest = hmac.new(
        SHOPIFY_WEBHOOK_SECRET.encode("utf-8"),
        payload,
        hashlib.sha256
    ).digest()
    computed = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(computed, hmac_header or "")

Every incoming webhook gets verified before processing. Shopify signs every webhook with your secret using HMAC-SHA256. If the signature doesn't match — 401, no processing, no NotiLens notification.

Never skip this. Any public endpoint without signature verification is open to spoofed webhooks.

Initialize NotiLens

nl = notilens.init(name=APP_NAME, token=NOTILENS_TOKEN, secret=NOTILENS_SECRET)

One line. The nl object is used for all notifications throughout the handler.

Order created

if topic == "orders/create":
    total = price(data.get("total_price"), currency)
    customer = data.get("email") or data.get("contact_email", "guest")
    items = len(data.get("line_items", []))
    nl.notify(
        "order.created",
        f"New order — {total} · {items} item(s) · {customer}",
        force_send=False,  # route through ML — reduces noise on high volume
        meta={
            "order_id":     data.get("id"),
            "order_number": data.get("order_number"),
            "total":        float(data.get("total_price", 0)),
            "currency":     currency,
            "items":        items,
            "customer":     customer,
            "is_new":       data.get("customer", {}).get("orders_count", 1) == 1,
        },
        tags="shopify,order",
    )

force_send=False — on high volume stores, every order firing an immediate push notification becomes noise fast. With force_send=False, NotiLens routes the notification through ML. Normal order arrives — logged, no push. Anomaly detected — push fires. This is how you get alerted when order volume drops 80% from your Wednesday baseline without getting pinged on every single order.

is_new flag — tracks whether this is a first-time customer. Useful for conversion monitoring.

Refund and cancellation — force send

elif topic == "refunds/create":
    nl.notify(
        "refund.created",
        f"Refund issued — {amt}",
        level="warning",
        meta={
            "order_id":       data.get("order_id"),
            "refund_id":      data.get("id"),
            "amount":         total_refund,
            "currency":       currency,
            "items_refunded": len(refund_lines),
        },
        tags="shopify,refund",
    )

No force_send flag here — defaults to True for warning level events. Refund issued and order cancelled are things you want to know about immediately, not filtered through ML. Push fires regardless of baseline.

Checkout started — abandoned checkout signal

elif topic == "checkouts/create":
    nl.notify(
        "checkout.created",
        f"Checkout started — {total} · {customer}",
        force_send=False,
        meta={
            "checkout_id": data.get("id"),
            "total":       float(data.get("total_price", 0)),
            "currency":    currency,
            "customer":    customer,
            "items":       len(data.get("line_items", [])),
        },
        tags="shopify,checkout",
    )

Tracking checkouts/create alongside orders/create gives you a checkout-to-order signal. NotiLens learns the normal ratio between checkouts started and orders completed. When that ratio changes significantly — checkout volume stays the same but orders drop — that's the silent failure signal. Payment processing broke somewhere between checkout and order confirmation.

This is the broken flow detection concept applied to e-commerce — not monitoring individual events in isolation but the relationship between them.

The webhook handler — full flow

@app.route("/webhook/shopify", methods=["POST"])
def shopify_webhook():
    payload    = request.data
    hmac_header = request.headers.get("X-Shopify-Hmac-Sha256", "")
    topic      = request.headers.get("X-Shopify-Topic", "")

    if not verify_shopify_signature(payload, hmac_header):
        return jsonify({"error": "Invalid signature"}), 401

    data     = request.get_json()
    currency = data.get("currency", "USD")

    # ... event handlers ...

    return jsonify({"success": True}), 200

Always return 200 after successful processing. Returning non-200 causes Shopify to retry delivery — which creates duplicate processing if your actual handler succeeded. The 200 confirms receipt. NotiLens confirms what happened after receipt.

What the monitoring layer adds

Without this boilerplate — Shopify fires webhooks, your endpoint processes them, you find out something went wrong when a customer emails you.

With this boilerplate:

Every event processed — NotiLens logs it. You have a record of every order, payment, refund, and checkout that arrived and was processed. Not just what Shopify sent — what your backend actually handled.

Silence detection — if orders/create webhooks stop arriving during your normal business hours, NotiLens fires a silence alert. Your backend might be healthy. Your Shopify webhook registration might have expired. The endpoint URL might have changed after a deployment. NotiLens catches the gap.

Revenue anomaly detection — NotiLens ML learns your normal order volume and average order value. Sudden drop in order frequency, spike in refund volume, checkout-to-order ratio collapsing — anomaly alert fires before you check the end-of-day report.

Refund and cancellation spikes — immediate push notification. Not filtered. Not batched. Fires the moment it happens.

Adapting for other webhook sources

The same pattern works for any webhook source. Replace the Shopify signature verification with the equivalent for your source:

Stripe:

import stripe
event = stripe.Webhook.construct_event(
    payload, sig_header, stripe_webhook_secret
)

GitHub:

signature = "sha256=" + hmac.new(
    secret.encode(), payload, hashlib.sha256
).hexdigest()

Custom internal service:

# Use a shared secret in the request header
token = request.headers.get("X-Webhook-Token")
if token != WEBHOOK_SECRET:
    return 401

The NotiLens notification pattern — nl.notify() with numeric meta values — works the same regardless of source. The ML baseline learning works on whatever numeric values you send. Order amounts, payment values, record counts, response times — all become signals NotiLens learns from.

Full boilerplate

Copy the complete script from the gist:

👉 gist.github.com/notilens/703cd96c1d08ec441f1f102d8b001249

Why this pattern matters

Shopify's webhook logs show delivery. Your server logs show receipt. Neither shows you whether the business logic that runs after receipt actually worked correctly.

That gap — between webhook delivered and business outcome confirmed — is where silent failures live. A webhook monitor that only tracks delivery is watching the wrong thing.

The pattern this boilerplate implements watches the outcome: order processed, payment recorded, refund logged, checkout started. When those outcomes stop arriving — that's the alert.

notilens.com — 7-day free trial, no credit card required.

Server health monitoring in pure bash - CPU, memory, disk, and process alerts with no dependencies

Stephen Souza — Tue, 26 May 2026 10:28:43 +0000

Most server monitoring setups involve installing an agent, configuring a daemon, opening ports, and maintaining a separate process on every server you run.

That's a lot of moving parts for something that should be simple — check if the server is healthy, send the numbers somewhere useful, alert if something goes wrong.

Here's a lighter approach. Pure bash. No agents. No installs. Just curl and a cron job.

What this monitors

CPU usage — actual percentage, sent as a numeric value
Memory usage — actual percentage, sent as a numeric value
Disk usage — used percentage and free GB, both sent as numeric values
Process status — whether nginx, postgres, redis, or any process you care about is running Every metric is sent as an actual number, not a string. This matters for ML anomaly detection — NotiLens uses these values to learn your server's normal baseline automatically and alert when something drifts. No thresholds to configure.

How it works

The script runs every 5 minutes via cron. Each run:

Collects CPU, memory, and disk metrics using standard Linux tools
Sends all metrics in one payload to NotiLens via curl webhook
Checks each process in your list
Sends a healthy ping for running processes
Sends an urgent, actionable alert for any process that's down — repeats every 5 minutes until acknowledged No dependencies beyond curl, which ships with every Linux server.

Setup

Step 1 — Get your credentials

Create a free account at notilens.com
Go to app.notilens.com → Custom Integration → create a topic
Copy your Topic ID and Secret Key

Step 2 — Download the script

curl -o notilens-server-health-monitor.sh \
  https://gist.githubusercontent.com/notilens/f6414a8d58f80bd50fb9e04fd59b4eec/raw/notilens-server-health-monitor.sh
chmod +x notilens-server-health-monitor.sh

Or copy the full script from the gist directly:
👉 gist.github.com/notilens/f6414a8d58f80bd50fb9e04fd59b4eec

Step 3 — Configure

Edit the config section at the top:

NOTILENS_TOPIC_ID="YOUR_TOPIC_ID"     # from app.notilens.com → Custom Integration
NOTILENS_SECRET_KEY="YOUR_SECRET_KEY" # from app.notilens.com → Custom Integration
SERVER_NAME="prod-server-1"           # shown in all NotiLens alerts
PROCESSES=("nginx" "postgres" "redis") # processes to check

Step 4 — Add to crontab

crontab -e

Add this line to run every 5 minutes:

*/5 * * * * /bin/bash /path/to/notilens-server-health-monitor.sh >> /var/log/notilens-health.log 2>&1

The script — section by section

Collect metrics

CPU=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1 | cut -d',' -f1)
MEM=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100}')
DISK_USED=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
DISK_FREE_GB=$(df -BG / | tail -1 | awk '{print $4}' | tr -d 'G')

Standard Linux tools — top, free, df. No additional packages needed.

Send metrics payload

curl -s -X POST "$WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -H "X-NOTILENS-KEY: $NOTILENS_SECRET_KEY" \
  -d "{
    \"title\": \"$SERVER_NAME — Health Check\",
    \"message\": \"CPU: ${CPU}% | Memory: ${MEM}% | Disk: ${DISK_USED}% used (${DISK_FREE_GB}GB free)\",
    \"type\": \"info\",
    \"event\": \"server.health\",
    \"tags\": \"$SERVER_NAME\",
    \"meta\": {
      \"server\": \"$SERVER_NAME\",
      \"cpu_percent\": $CPU,
      \"memory_percent\": $MEM,
      \"disk_used_percent\": $DISK_USED,
      \"disk_free_gb\": $DISK_FREE_GB
    }
  }"

The key detail — cpu_percent, memory_percent, disk_used_percent, and disk_free_gb are sent as actual numbers in the meta object, not strings. NotiLens ML reads these numeric values to build your server's baseline over time.

After a few days of runs, NotiLens knows what normal CPU looks like on a Tuesday afternoon vs a Sunday night. Anomaly detection fires automatically when values drift from that learned baseline — no manual threshold needed.

Process check — healthy

if pgrep -x "$PROCESS_NAME" > /dev/null; then
  curl -s -X POST "$WEBHOOK_URL" \
    -d "{
      \"title\": \"$SERVER_NAME — $PROCESS_NAME\",
      \"message\": \"$PROCESS_NAME is running\",
      \"type\": \"info\",
      \"event\": \"server.process.up\",
      \"meta\": {
        \"server\": \"$SERVER_NAME\",
        \"process\": \"$PROCESS_NAME\",
        \"status\": 1
      }
    }"

Sends a status ping for every healthy process. NotiLens tracks the frequency — if a process that normally pings every 5 minutes goes quiet, that's a silence alert.

Process check — down

else
  curl -s -X POST "$WEBHOOK_URL" \
    -d "{
      \"title\": \"$SERVER_NAME — $PROCESS_NAME DOWN\",
      \"message\": \"$PROCESS_NAME is not running on $SERVER_NAME\",
      \"type\": \"urgent\",
      \"event\": \"server.process.down\",
      \"meta\": {
        \"server\": \"$SERVER_NAME\",
        \"process\": \"$PROCESS_NAME\",
        \"status\": 0
      },
      \"is_actionable\": true,
      \"force_send\": true
    }"
fi

Two important flags here:

is_actionable: true — NotiLens resends this alert every 5 minutes until someone acknowledges it. A process down alert that fires once and gets missed is useless. This keeps firing until the on-call engineer ACKs it.

force_send: true — bypasses ML filtering and fires immediately. For a process down event you don't want the ML deciding whether this is anomalous enough to alert on. It fires regardless.

What you get in NotiLens

Every 5 minutes — a health check entry with CPU, memory, and disk values. NotiLens builds a baseline from these over time. When CPU spikes to 94% on a server that normally runs at 12% — that's the anomaly alert. When disk free drops below your historical normal — that's the drift alert.

Process down — immediate push notification to your phone. Repeats every 5 minutes until acknowledged. Escalation policy fires if nobody ACKs within your configured window.

No dashboard to watch. No thresholds to tune. No YAML.

Customisation

Monitor a different disk partition:

DISK_USED=$(df /data | tail -1 | awk '{print $5}' | tr -d '%')
DISK_FREE_GB=$(df -BG /data | tail -1 | awk '{print $4}' | tr -d 'G')

Add more processes:

PROCESSES=("nginx" "postgres" "redis" "your-app" "celery")

Change the check frequency:

Every minute instead of every 5:

* * * * * /bin/bash /path/to/notilens-server-health-monitor.sh >> /var/log/notilens-health.log 2>&1

Add network connectivity check:

if ping -c 1 8.8.8.8 &> /dev/null; then
  NETWORK_STATUS=1
else
  NETWORK_STATUS=0
fi

Add \"network_status\": $NETWORK_STATUS to the meta object.

Full script

Copy the complete boilerplate from the gist:

👉 gist.github.com/notilens/f6414a8d58f80bd50fb9e04fd59b4eec

Why pure bash

No agent to install means no agent to maintain, update, restart, or debug. No extra process running on your server consuming memory. No dependency on a language runtime. No version conflicts.

The script runs, sends the data, exits. Cron handles the scheduling. curl handles the delivery. That's the entire dependency tree.

Works on any Linux server. Tested on Ubuntu 20.04, 22.04, 24.04, Debian, and Amazon Linux.

notilens.com — 7-day free trial, no credit card required.

Silent failures in production - why conventional tools miss them and how NotiLens catches them

Stephen Souza — Wed, 20 May 2026 07:29:49 +0000

Your server is up. Your API is responding. Your error rate is zero.

And your business has been quietly bleeding for six hours.

This is the silent failure problem and it's more expensive than any crash you've ever had.

What is a silent failure?

A silent failure is a failure that produces no error signal.

No exception thrown. No status code changed. No alert fired. No dashboard turned red. The system is technically operating while something in the business logic has quietly stopped working.

Three characteristics define a silent failure:

No error signal — the failure doesn't produce an exception, a non-2xx response, or a log entry that reads like a problem. Everything looks clean.

Invisible onset — silent failures start at a specific moment but aren't noticed until hours or days later, when the damage has already compounded.

Business impact before technical detection — the first signal is usually a user complaint, a dropped metric in a weekly report, or an end-of-day revenue number that looks wrong.

Why silent failures cost more than crashes

A crash is visible. Something turns red. An alert fires. The fix starts within minutes.

The damage window of a crash is bounded — it ends when detection happens.

A silent failure is invisible. The damage window is unbounded — it ends when someone happens to check the right screen, a user complains, or a weekly report surfaces something unexpected.

The math:

$80 average order value
4 orders per hour

Crash detected in 5 minutes  → ~$27 lost
Silent failure for 6 hours   → $1,920 lost

Same root cause. Completely different damage profile.

And that's before refund processing, support tickets, trust damage, and engineering hours spent reconstructing what happened from logs that weren't designed to answer that question.

Fix time is bounded. Detection time isn't. That's where the real cost lives.

Where silent failures hide in production

Payment flows

payment.initiated fires. Stripe delivers the webhook. Your endpoint returns 200. Somewhere between delivery and database write, business logic fails silently. Stripe marks it delivered. Dashboard shows green. Revenue isn't recording.

Or the subtler version — payment.initiated fires and payment.completed never arrives. Each webhook delivers correctly. The sequence never finishes. No error. No alert.

Cron jobs

Two variants:

Variant 1 — job stops running entirely. Nightly invoice sync stopped running Thursday. Found out Monday. Four days of un-synced records, zero error thrown.

Variant 2 — job runs but processes nothing. The job runs at midnight. Starts. Exits clean. Processed zero records. From the outside — healthy run. From the inside — nothing happened.

Variant 2 is harder to catch because nothing failed by definition. The job completed successfully against zero records.

AI agent loops and ghost runs

AI agents don't crash — they drift, loop, stall, and consume resources while producing nothing.

A looping agent looks identical to a healthy agent doing legitimate multi-step research. Process running. Tool calls firing. Tokens accumulating. What's actually happening — the agent called the same tool 47 times, produced nothing, ran up $4.80 in tokens.

A ghost run is the agent equivalent of the cron job that processed zero records. The agent ran. Completed. Reported task done. Produced nothing meaningful.

Automation workflows

Your Zapier zap hasn't fired since Tuesday. Your n8n workflow silently stopped three days ago. Your Make scenario failed on one step and the entire workflow halted.

Automation failures live outside your main application stack. They don't throw exceptions. They don't affect uptime. They just stop.

Signup and onboarding flows

Your signup flow broke at 2am. Form submits. Confirmation email never sends. User lands on a broken onboarding step. Every component looks healthy. The sequence never completes.

No new signup in 4 hours on a Wednesday when your baseline is 12 per hour. Your monitoring has no alert for that. No vocabulary for absence.

Why conventional monitoring misses silent failures

Every tool in your observability stack was built to watch infrastructure. Silent failures are a business logic problem.

Uptime monitoring — watches whether your service responds. A silent failure doesn't affect uptime. The server is up. That's part of what makes it silent.

Error rate monitoring — watches for exceptions and non-2xx responses. Silent failures don't throw exceptions. Error rate stays clean.

APM tools — watch latency, throughput, error rates at the service level. No coverage for business logic correctness or event frequency.

Log monitoring — can surface silent failures but requires you to know what to look for before you can find it. Reactive, not proactive.

Threshold alerts — require you to define abnormal before the failure happens. You can't threshold what you haven't seen yet.

The fundamental gap: every conventional monitoring tool watches for something going wrong. Silent failures are defined by the absence of something going right.

How to detect silent failures with NotiLens

NotiLens is built specifically around the silent failure problem — business pulse monitoring that watches the things that should be happening and alerts the moment they aren't.

Before you start

Create a free account at notilens.com
Create a source in the dashboard — you'll get a token and secret
Install the SDK

pip install notilens

npm install @notilens/notilens

Initialize

Python

import notilens

nl = notilens.init(
    name="my-app",
    token="YOUR_TOKEN",
    secret="YOUR_SECRET"
)

# Or use environment variables: NOTILENS_TOKEN / NOTILENS_SECRET
nl = notilens.init(name="my-app")

Node.js

import { NotiLens } from '@notilens/notilens';

const nl = NotiLens.init('my-app', {
  token: 'YOUR_TOKEN',
  secret: 'YOUR_SECRET'
});

// Or use environment variables: NOTILENS_TOKEN / NOTILENS_SECRET
const nl = NotiLens.init('my-app');

1. Business event monitoring — silence detection for payments, signups, orders

Track events with nl.track(). NotiLens watches the frequency and fires a silence alert when they stop arriving.

Python

# Track payment events
nl.track("payment.completed", "Payment received", meta={"amount": 149.99})

# Track signups
nl.track("user.signup", "New user registered")

# Track orders
nl.track("order.placed", "Order #1234", meta={"amount": 89.00})

Node.js

// Track payment events
nl.track('payment.completed', 'Payment received', { meta: { amount: 149.99 } });

// Track signups
nl.track('user.signup', 'New user registered');

// Track orders
nl.track('order.placed', 'Order #1234', { meta: { amount: 89.00 } });

When payment.completed stops arriving for longer than your baseline window — silence alert fires. NotiLens learns your normal frequency automatically. No threshold to configure.

2. Cron job monitoring — records processed, time taken, missed runs

The most common silent failure — job runs, processes nothing, exits clean, looks healthy.

NotiLens tracks records processed and time taken alongside the heartbeat. Zero records on a job that normally touches 500 is the alert. Job that didn't ping at all — also the alert.

Python

import notilens

nl  = notilens.init(name="invoice-sync", token="TOKEN", secret="SECRET")
run = nl.task("nightly-sync")
run.start()

try:
    run.progress("Fetching invoices")
    records = process_invoices()

    run.metric("records", records)       # track records processed
    run.metric("duration_ms", 1240)      # track time taken

    run.complete(f"Processed {records} invoices")
except Exception as e:
    run.fail(str(e))

Node.js

import { NotiLens } from '@notilens/notilens';

const nl  = NotiLens.init('invoice-sync', { token: 'TOKEN', secret: 'SECRET' });
const run = nl.task('nightly-sync');
run.start();

try {
  run.progress('Fetching invoices');
  const records = await processInvoices();

  run.metric('records', records);        // track records processed
  run.metric('duration_ms', 1240);       // track time taken

  run.complete(`Processed ${records} invoices`);
} catch (err) {
  run.fail(err.message);
}

NotiLens ML learns your expected record volume and duration. A run that took 4x longer than usual — even with the same record count — is the alert.

3. AI agent loop detection — iteration count, token burns, duration anomalies

Call run.loop() on every agent iteration. NotiLens ML learns how many iterations your agent normally runs and alerts when a run is anomalously high with no run.complete() arriving.

Python

import notilens

nl  = notilens.init(
    name="research-agent",
    token="TOKEN",
    secret="SECRET",
    patch=True  # auto-instruments OpenAI, Anthropic, LangChain calls
)

run = nl.task("research")
run.start()

try:
    run.progress("Starting research")

    for i, step in enumerate(steps):
        run.loop(f"[{i+1}] Tool: {step.tool_name}")  # every iteration
        run.metric("tool_calls", 1)                   # accumulates

        result = agent.execute(step)
        run.metric("tokens", result.usage.total_tokens)
        run.metric("cost_usd", result.usage.cost)

    run.output_generated("Research complete")
    run.complete(f"Processed {len(steps)} steps")
except Exception as e:
    run.fail(str(e))

Node.js

import { NotiLens } from '@notilens/notilens';

const nl  = NotiLens.init('research-agent', { token: 'TOKEN', secret: 'SECRET' });
const run = nl.task('research');
run.start();

try {
  run.progress('Starting research');

  for (const [i, step] of steps.entries()) {
    run.loop(`[${i+1}] Tool: ${step.toolName}`);  // every iteration
    run.metric('tool_calls', 1);                   // accumulates

    const result = await agent.execute(step);
    run.metric('tokens', result.usage.totalTokens);
    run.metric('cost_usd', result.usage.cost);
  }

  run.outputGenerated('Research complete');
  run.complete(`Processed ${steps.length} steps`);
} catch (err) {
  run.fail(err.message);
}

When the alert fires you see: which agent, which task, current loop count vs baseline, token consumption vs baseline, last progress message before deviation started.

4. Agent stall detection — when agents pause on slow tools

For agents pausing on slow external APIs or tools:

Python

run.wait("Awaiting API response")
result = call_slow_external_api()   # if this stalls, Smart Silence Detection alerts
run.progress("API response received")

run.wait() is non-terminal — the run continues. NotiLens learns how long your agent normally spends between events and fires if the gap becomes anomalous.

5. Quick alerts — no task context needed

For simple one-off alerts — disk space, deployment events, server health:

Python

nl.notify("disk.space", "Only 1GB left", level="warning")
nl.notify("deploy.done", "Deployed to production",
    open_url="https://dashboard.example.com/deploys"
)

Node.js

nl.notify('disk.space.full', 'Only 1GB left', { level: 'warning' });
nl.notify('deploy.done', 'Deployed to production', {
  openUrl: 'https://dashboard.example.com/deploys'
});

6. CLI — for shell scripts and bash pipelines

No code changes needed. Register once, use anywhere:

notilens init --name my-app --token YOUR_TOKEN --secret YOUR_SECRET

# Simple notifications
notilens notify order.placed "Order #1234" --name my-app
notilens notify disk.space.full "Only 1GB left" --name my-app --type warning

# Full task lifecycle
notilens start    --name my-app --task nightly-sync
notilens progress "Fetching records" --name my-app --task nightly-sync
notilens metric   records=461 --name my-app --task nightly-sync
notilens complete "Done" --name my-app --task nightly-sync

ML anomaly detection — no thresholds to configure

Most monitoring requires you to know what's wrong before you can alert on it. NotiLens ML anomaly detection learns your baseline automatically.

Your Wednesday signup rate. Your typical payment volume. Your agent's normal run count. Your cron's expected record output. No configuration needed.

What it catches that threshold alerts miss:

Spike detection — payment volume 3x above your Wednesday baseline at 2pm. Could be fraud. Could be a campaign. Either way, worth knowing immediately.

Drop detection — signups down 80% from your Tuesday morning baseline. Server up. No errors. Just unusually quiet.

Drift detection — API response time slowly climbing from 120ms to 250ms over two weeks. No single data point crosses a threshold. The trend is the signal.

Broken flow detection — payment.initiated normally reaches payment.completed within 3 minutes. When the gap extends, alert fires before the window closes on recovery.

No YAML. No threshold tuning. Cold-start mode shows calibration progress so you know when the model is ready.

The detection gap calculation

Before you close this — run this for your own stack.

Take your most important revenue event. How many per hour during peak? What's your average transaction value? How long would a silent failure run before you'd notice?

events_per_hour × average_value × hours_until_detection = detection_gap_cost

Most teams, when they run that number for the first time, immediately understand why detection time matters more than fix time.

The fix is always bounded. The detection gap is where revenue bleeds.

Full silent failure monitoring checklist

Business events tracked with nl.track() — payments, signups, orders
run.start() fires when any task begins
run.complete() fires on successful completion
run.fail() fires on any unhandled exception
run.loop() called on every agent iteration
run.metric("records", n) tracks output volume on cron jobs
run.metric("tokens", n) tracks token usage on AI agents
run.metric("duration_ms", n) tracks time taken
run.wait() fires when agent pauses on slow external calls
ML anomaly detection active — no thresholds needed
On-call routing configured for silence alerts
Tested — confirmed NotiLens detects when expected events stop arriving

Summary

Silent failures are the expensive failures that conventional monitoring was never built to catch. They don't throw errors. They don't affect uptime. They just quietly stop working while every dashboard stays green.

Detection time is where revenue bleeds. Shrinking the detection gap from 6 hours to 60 seconds doesn't change how fast you fix things. It changes how much there is to fix.

notilens.com

If your AI agent looped 40 times last night, would you know?

Stephen Souza — Fri, 15 May 2026 14:04:10 +0000

Probably not. And that's the problem.

Your process was running. Tool calls were firing. No exceptions thrown. From every angle your stack can see — healthy agent doing legitimate work.

What was actually happening: the agent called web_search with "latest AI news". Got results. Called it again with "recent AI developments". Got similar results. Called it again. And again. 47 tool calls. Zero useful output. $4.80 in tokens. Nobody knew until the OpenAI invoice arrived.

This is the failure mode conventional monitoring was never built to catch.

Why your current monitoring misses it

Your uptime monitor sees: process running.
Your error monitor sees: no exceptions thrown.
Your log monitor sees: tool calls completing successfully.
Your OpenAI dashboard sees: token consumption — but no per-task breakdown.

Nothing in that stack identifies that the agent is calling the same tool repeatedly with no progress toward completion. A looping agent looks identical to a healthy agent doing legitimate multi-step research.

The signal isn't an error. It's a pattern — same tool, repeated calls, no convergence toward a completion event. Catching that requires monitoring that understands what the agent is supposed to be doing, not just whether it's technically running.

What actually causes loops

Ambiguous task objective — the agent can't determine if it has "enough" to complete the task. It keeps searching for a confidence threshold it can never reach.

Tool output that looks like progress but isn't — each result genuinely looks like partial progress. The agent never recognises the cycle.

Malformed tool responses — the agent retries with slightly different parameters, gets the same malformed response, retries again.

Context window pressure — as context fills with results, the agent loses track of what it already tried and starts repeating earlier tool calls.

max_iterations as the only safeguard — stops the loop but doesn't alert you, doesn't tell you how many times the agent looped, and doesn't fire until the budget is already spent. Circuit breaker, not a monitor.

Three loop signatures to watch

Signature 1 — Same tool, repeated calls

web_search → web_search → web_search ← loop detected

Same tool appears in 3 of the last 5 calls without a completion event.

Signature 2 — High iteration count, no completion

Tool calls: 18
Normal completion at: 4–6 tool calls
Status: no completion event
→ Loop detected

Signature 3 — Token budget anomaly

Tokens consumed: 42,000
Normal consumption: 3,000–6,000
Status: no completion event
→ Budget anomaly detected

Setting up loop detection with NotiLens

Install:

pip install notilens

npm install @notilens/notilens

The pattern is simple — call run.loop() on every agent iteration. NotiLens ML learns how many iterations your agent normally runs and alerts when a run is anomalously high with no run.complete() arriving. No threshold to configure. No detection logic to write.

Python

import notilens

nl  = notilens.init(name="ai-agent")
run = nl.task("research")
run.start()

for iteration in range(max_iterations):
    tool_name, tool_input = agent.decide_next_action()

    run.loop(f"[{iteration + 1}] Tool: {tool_name}")  # every iteration
    run.metric("tool_calls", 1)                        # accumulates

    result = execute_tool(tool_name, tool_input)

    if agent.is_done(result):
        break

run.metric("tokens", total_tokens_used)
run.complete("Task completed")

Node.js

import { NotiLens } from '@notilens/notilens';

const nl  = NotiLens.init('ai-agent');
const run = nl.task('research');
run.start();

for (let i = 0; i < maxIterations; i++) {
  const { toolName, toolInput } = agent.decideNextAction();

  run.loop(`[${i + 1}] Tool: ${toolName}`);  // every iteration
  run.metric('tool_calls', 1);               // accumulates

  const result = await executeTool(toolName, toolInput);

  if (agent.isDone(result)) break;
}

run.metric('tokens', totalTokensUsed);
run.complete('Task completed');

Token and cost tracking

Track token consumption per run — NotiLens uses this alongside iteration count to detect cost anomalies:

run.metric("tokens", tokens_used_this_call)
run.metric("cost_usd", round(tokens_used_this_call * 0.0000002, 6))

If a run consumes 5x more tokens than your baseline without completing, NotiLens flags it — even without a manually configured budget limit.

Agent stall detection

For agents pausing on slow tools or external APIs:

run.wait("Awaiting API response")
result = call_slow_external_api()
run.progress("API response received")

run.wait() is non-terminal — the run continues. NotiLens learns how long your agent normally spends between events and fires if the gap becomes anomalous.

What the alert looks like

✅ task.started      Research agent — task started
🔄 task.loop         [1] Tool: web_search
🔄 task.loop         [2] Tool: web_search
🔄 task.loop         [3] Tool: web_search
🔄 task.loop         [4] Tool: web_search
🔄 task.loop         [5] Tool: web_search
🔄 task.loop         [6] Tool: web_search
⚠️  Anomaly detected  Iteration count 6 — exceeds learned baseline (avg: 3.2)
                     No task.complete received
                     tool_calls: 6 | tokens: 8,420 | cost: $0.0017
→ Push notification fired

One line per iteration. NotiLens detected the pattern automatically.

Full agent monitoring checklist

run.loop() called on every agent iteration
run.start() fires when task begins
run.complete() fires on successful completion
run.fail() fires on any unhandled exception
run.error() fires on non-fatal tool errors
run.timeout() fires if agent exceeds SLA window
run.wait() fires when agent pauses on slow external call
run.metric("tool_calls", 1) accumulates per iteration
run.metric("tokens", ...) tracks token usage
run.metric("cost_usd", ...) tracks cost per run

The short answer

max_iterations stops the loop. NotiLens tells you it happened at call 6, not call 47 — while there's still something to investigate.

Works with LangChain, CrewAI, AutoGen, LlamaIndex, Pydantic AI, or any custom agent loop. No framework dependency.

notilens.com

Stop checking dashboards. Build the monitoring layer that checks for you.

Stephen Souza — Tue, 12 May 2026 08:00:32 +0000

Every developer has a version of this story.

You open your laptop in the morning. Something feels off.

You check
Stripe - green.
Check the server - up.
Check the error logs — clean.

Spend 20 minutes manually clicking through dashboards before realising signups stopped 6 hours ago.

That's not monitoring. That's detective work.

Dashboards are useful. The problem is knowing which one to open. Silent failure monitoring solves that — it watches for anomalies, silence, and broken flows across your stack, then tells you exactly where to look. You still investigate. You just stop guessing where to start.

What silent failures look like in production

Silent failures don't throw errors. They don't trigger uptime alerts. They just quietly cost you money until someone notices.

No new signup in 4 hours on a Wednesday when your baseline is 12 per hour. Payment.initiated that never reaches payment.completed. Nightly sync that ran at midnight, processed zero records, exited clean, no error thrown. AI agent that looped 47 times, burned token budget, produced nothing. Cron job that stopped running Thursday. Zapier workflow that hasn't fired since Tuesday.

None of these show up as red in any dashboard. All of them show up in revenue.

Before you start

Create an account at notilens.com
Create a source in the dashboard — you'll get a token and secret
Install the SDK

Install NotiLens

pip install notilens

npm install @notilens/notilens

Initialize

Python

import notilens

nl = notilens.init(
    name="my-app",
    token="YOUR_TOKEN",
    secret="YOUR_SECRET"
)

Node.js

import { NotiLens } from '@notilens/notilens';

const nl = NotiLens.init('my-app', {
  token: 'YOUR_TOKEN',
  secret: 'YOUR_SECRET'
});

Credentials are saved to ~/.notilens_config.json after first use. Use environment variables NOTILENS_TOKEN and NOTILENS_SECRET to avoid hardcoding.

Cron job monitoring — know when jobs run and what they actually did

The most expensive silent failure for most teams — cron job stops running, nobody knows for days. Or worse, it runs but processes nothing, exits clean, and looks healthy.

NotiLens catches both. Your job pings when it starts, passes records processed and time taken, and pings when it completes. Zero records on a job that normally touches 500 — that's the alert. Job that didn't ping at all — also the alert.

Python

import notilens

nl  = notilens.init(name="invoice-sync", token="TOKEN", secret="SECRET")
run = nl.task("nightly-sync")
run.start()

try:
    run.progress("Fetching invoices")
    records = process_invoices()

    run.metric("records", records)
    run.metric("duration_ms", 1240)

    run.complete(f"Processed {records} invoices")
except Exception as e:
    run.fail(str(e))

Node.js

import { NotiLens } from '@notilens/notilens';

const nl  = NotiLens.init('invoice-sync', { token: 'TOKEN', secret: 'SECRET' });
const run = nl.task('nightly-sync');
run.start();

try {
  run.progress('Fetching invoices');
  const records = await processInvoices();

  run.metric('records', records);
  run.metric('duration_ms', 1240);

  run.complete(`Processed ${records} invoices`);
} catch (err) {
  run.fail(err.message);
}

NotiLens fires a silence alert if the job doesn't ping within its expected window. You open the dashboard knowing exactly which job missed and when.

AI agent monitoring — loops, hangs, token burns, and human approval waits

AI agents fail differently from regular software. They don't crash — they drift, loop, stall, and consume resources while producing nothing. Standard error monitoring has no vocabulary for this.

NotiLens tracks the full agent lifecycle — start, progress, loop detection, output, completion. When something goes wrong you get an alert with context: which agent, which task, how many loops, what metrics looked like before it stalled.

Python

import notilens

nl  = notilens.init(
    name="outreach-agent",
    token="TOKEN",
    secret="SECRET",
    patch=True  # auto-instruments OpenAI, Anthropic, LangChain calls
)

run = nl.task("email-campaign")
run.start()

try:
    run.progress("Fetching leads")

    for i, lead in enumerate(leads):
        run.loop(f"Processing lead {i+1} of {len(leads)}")
        result = agent.process(lead)
        run.metric("tokens", result.usage.total_tokens)
        run.metric("cost", result.usage.cost)

    run.output_generated("Campaign emails ready")
    run.complete(f"Processed {len(leads)} leads")
except Exception as e:
    run.fail(str(e))

Node.js

import { NotiLens } from '@notilens/notilens';

const nl  = NotiLens.init('outreach-agent', { token: 'TOKEN', secret: 'SECRET' });
const run = nl.task('email-campaign');
run.start();

try {
  run.progress('Fetching leads');

  for (const [i, lead] of leads.entries()) {
    run.loop(`Processing lead ${i+1} of ${leads.length}`);
    const result = await agent.process(lead);
    run.metric('tokens', result.usage.totalTokens);
    run.metric('cost', result.usage.cost);
  }

  run.outputGenerated('Campaign emails ready');
  run.complete(`Processed ${leads.length} leads`);
} catch (err) {
  run.fail(err.message);
}

Human-in-the-loop — when your agent needs approval:

Python

run.input_required("Please confirm before sending emails")
run.input_approved("User confirmed")

Node.js

run.inputRequired('Please confirm before sending emails');
run.inputApproved('User confirmed');

Business event monitoring — payments, signups, orders

For business events that don't fit a task lifecycle — payment received, order placed, signup completed — use track. NotiLens watches the frequency of these events and fires a silence alert when they stop arriving.

Python

nl.track("payment.completed", "Payment received", meta={"amount": 149.99})
nl.track("user.signup", "New user registered")
nl.track("order.placed", "Order #1234", meta={"amount": 89.00})

Node.js

nl.track('payment.completed', 'Payment received', { meta: { amount: 149.99 } });
nl.track('user.signup', 'New user registered');
nl.track('order.placed', 'Order #1234', { meta: { amount: 89.00 } });

When payment.completed stops arriving for longer than your baseline window — silence alert fires. You open Stripe knowing exactly what to look for.

ML anomaly detection — no thresholds to configure

Most monitoring requires you to know what's wrong before you can alert on it. Set a threshold, define a rule, write the condition. The problem — you can't threshold what you haven't seen yet.

NotiLens learns your baseline automatically. Your Wednesday signup rate. Your typical payment volume. Your agent's normal run count. Your cron's expected record output. No configuration needed — it figures out what normal looks like and alerts when reality diverges from it.

What it catches that threshold alerts miss:

Spike detection — payment volume 3x above your Wednesday baseline at 2pm. Could be legitimate. Could be fraud. Either way, worth knowing.

Drop detection — signups down 80% from your Tuesday morning baseline. Server is up. No errors. Just unusually quiet.

Drift detection — API response time slowly climbing from 120ms to 250ms over two weeks. No single data point crosses a threshold. The trend is the signal.

Broken flow detection — payment.initiated normally reaches payment.completed within 3 minutes. When the gap extends, NotiLens catches it before the window closes on recovery.

No YAML. No threshold tuning. No false alerts during warm-up — NotiLens shows calibration progress in cold-start mode so you know when the model is ready.

The alert you get isn't "value exceeded X". It's "this is genuinely abnormal for your business at this time of day on this day of week."

Quick alerts — no task context needed

Python

nl.notify("disk.space", "Only 1GB left", level="warning")
nl.notify("deploy.done", "Deployed to production",
    open_url="https://dashboard.example.com/deploys"
)

Node.js

nl.notify('disk.space', 'Only 1GB left', { level: 'warning' });
nl.notify('deploy.done', 'Deployed to production', {
  openUrl: 'https://dashboard.example.com/deploys'
});

CLI — for shell scripts and bash pipelines

No code changes needed. Register once, use anywhere:

notilens init --name my-app --token YOUR_TOKEN --secret YOUR_SECRET

notilens notify order.placed "Order #1234" --name my-app
notilens notify disk.space "Disk 7.5GB" --name my-app --meta size=7.5

notilens start    --name my-app --task nightly-sync
notilens progress "Fetching records" --name my-app --task nightly-sync
notilens metric   records=461 --name my-app --task nightly-sync
notilens complete "Done" --name my-app --task nightly-sync

What you get

One feed across your entire stack. Cron jobs, AI agents, payments, signups, orders, servers — all in one place with context. When an alert fires, you know which dashboard to open, what to look for, and roughly when it started.

No more guessing. No more morning dashboard checks hoping something is obviously wrong.

The dashboards are still there. The monitoring layer just tells you which one matters right now.

Beyond Uptime: The Complete Monitoring Stack for SaaS Builders

Stephen Souza — Thu, 07 May 2026 04:36:41 +0000

Your uptime monitor says green.

Your server is responding. CPU is normal. No errors in the logs.

But signups stopped 4 hours ago. Nobody noticed.

That's the gap most monitoring stacks have and it's the gap that costs the most.

This is the monitoring stack we run at NotiLens, built for SaaS teams who don't have a dedicated DevOps engineer watching dashboards all day.

The problem with uptime-only monitoring

Traditional monitoring answers one question: is the server on?

What it doesn't answer:

Are users actually signing up?
Are payments completing — not just initiating?
Are cron jobs processing records — not just running?
Are AI agents producing output — not just executing?

These are business-layer failures. Infrastructure monitoring completely misses them.

Here's how to cover both layers.

Layer 1: Revenue monitoring

Stripe webhooks

Stripe webhook failures are the silent killer most SaaS builders don't monitor. Your endpoint can return 200s while silently failing to process events — subscriptions go stale, payment failures go unhandled, refunds queue up.

NotiLens monitors Stripe from two angles simultaneously.

Signal 1 — Stripe sends directly to NotiLens:
Configure a NotiLens webhook endpoint in your Stripe dashboard alongside your existing endpoint. NotiLens receives the raw event.

Signal 2 — Your backend confirms processing:

app.post('/webhooks/stripe', async (req, res) => {
  const event = stripe.webhooks.constructEvent(
    req.body,
    req.headers['stripe-signature'],
    process.env.STRIPE_WEBHOOK_SECRET
  );

  // Your existing processing logic
  await handleStripeEvent(event);

  // Confirm to NotiLens that processing completed
  await notilens.track("stripe.webhook.processed", {
    type: event.type,
    customerId: event.data.object.customer
  });

  res.json({ received: true });
});

What ML detects:

Stripe sent the webhook ✓ but your backend never confirmed processing ✗ → broken flow alert
Both signals arrived but volume dropped below normal baseline → silence alert
Sudden spike in webhook volume → anomaly alert

The gap between Signal 1 and Signal 2 is where most payment failures hide.

→ Stripe webhook monitoring
→ Stripe payment failure alerts

Shopify orders

Configure Shopify to send webhook events directly to NotiLens:

Go to your Shopify Admin → Settings → Notifications
Scroll to Webhooks → click Create webhook
Select event: Order creation and Order payment
Paste your NotiLens Shopify webhook URL
Set format to JSON → Save NotiLens watches incoming order volume against your baseline. If orders go abnormally quiet for your time of day — silence alert fires. No manual threshold needed.

→ Shopify order monitoring
→ Shopify silent order drop alerts

Layer 2: Silence monitoring

This is the most important layer — and the one nobody talks about.

Silence monitoring answers: is anything actually happening?

Your server can be perfectly healthy while:

No new users have signed up in 6 hours
No new orders have come in since midnight
A background job ran but processed zero records
An API is responding but returning empty results

None of these trigger a server alert. All of them are serious.

// Track every signup
await notilens.track("user.signup.completed", {
  userId: user.id,
  plan: user.plan
});

// Track every activated user
await notilens.track("user.activated", {
  userId: user.id
});

NotiLens learns your baseline — how many signups per hour is normal at 2am on a Tuesday — and alerts you when it drops significantly below that. No manual threshold needed.

You can also detect broken flows — user.signup.completed fired but user.activated never followed within 30 minutes:

Layer 3: Infrastructure basics

Keep this minimal. What you actually need:

Server up/down — server downtime alerts
Server silence — server silence monitoring for when your server stops reporting entirely
API error rate spikes — API error rate monitoring

What you probably don't need yet: APM dashboards, distributed tracing, custom metrics pipelines.

Layer 4: Cron jobs and scheduled tasks

The problem isn't when a cron job crashes. It's when it runs successfully but does nothing.

Exit code 0. Zero records processed. No alert.

Fix: heartbeat monitoring. Your job sends a ping on successful completion. If the ping doesn't arrive in the expected window — alert fires.

// At the end of your cron job
const result = await processBillingRecords();

await notilens.track("billing.sync.job", {
  recordsProcessed: result.count,
  duration: result.durationMs
});

NotiLens ML detects two anomalies beyond just "did it run?":

recordsProcessed consistently 0 — job ran but did nothing
durationMs spikes above normal baseline — job is taking significantly longer than usual, often the first sign of a database or dependency issue before it becomes an outage

→ Cron job failure monitoring

Three jobs to instrument first:

Billing sync
Email delivery
Data cleanup / reporting

Layer 5: Developer activity

Use the official NotiLens GitHub Action — no curl needed:

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to production
        run: ./deploy.sh

      - name: Notify deploy success
        if: success()
        uses: notilens/notify-action@v1
        with:
          token:    ${{ secrets.NOTILENS_TOKEN }}
          secret:   ${{ secrets.NOTILENS_SECRET }}
          event:    task.completed
          message:  "Deployed to production — ${{ github.ref_name }}"
          tags:     deploy,production
          open_url: https://myapp.com

      - name: Notify deploy failure
        if: failure()
        uses: notilens/notify-action@v1
        with:
          token:   ${{ secrets.NOTILENS_TOKEN }}
          secret:  ${{ secrets.NOTILENS_SECRET }}
          event:   task.failed
          message: "Production deployment failed — ${{github.ref_name }}"

The action automatically includes repo, branch, commit, actor, and a direct link to the workflow run — no extra config needed.

→ GitHub CI/CD alerts

Know immediately when a deployment breaks. Don't find out because something stopped working in production.

Layer 6: AI agents and automations

AI agents fail in ways traditional monitoring completely misses:

Silent no-output — runs, completes, exits 0, produces nothing.
Infinite loops — keeps retrying the same step, token costs climb silently.
Stuck tool calls — waiting for a response that never comes.

// Track agent lifecycle
await nl.start('Agent run started', { task: 'report-agent' });

// Track token usage — ML detects anomalous spikes (loops)
await nl.metric({ tokens: response.usage.total_tokens }, { task: 'report-agent' });

// On completion
await nl.complete('Agent completed', { task: 'report-agent' });

// On loop/timeout detection
await nl.timeout('Agent exceeded expected duration', { task: 'report-agent' });

NotiLens detects when token usage spikes above your normal baseline — catches infinite loops before your API bill does.

→ AI agent monitoring

For no-code automation platforms:
→ Zapier workflow failure alerts
→ n8n automation monitoring
→ Make.com automation monitoring

The setup order

Don't try to instrument everything at once.

Week 1 — Revenue first:

Stripe webhook tracking
Payment failure alerts
Shopify order silence (if applicable)

Week 2 — Business health:

Signup silence alert
Server up/down
One critical cron job heartbeat

Week 3 — Operations:

API error rate
GitHub CI/CD failures
Second cron job

Week 4+ — AI and automation:

Agent monitoring
Zapier/n8n/Make workflow monitoring

Start with what touches revenue. Work outward from there.

The full SDK install

npm install @notilens/notilens

import { NotiLens } from '@notilens/notilens';

const nl = NotiLens.init('my-app', { token: 'YOUR_TOKEN', secret: 'YOUR_SECRET' });


// Track a business event
await nl.track('event.name', 'Event description', { meta: { ...metadata } });

// Task lifecycle
await nl.start('Job started', { task: 'job-name' });
await nl.complete('Job done', { task: 'job-name' });
await nl.fail('Job failed', { task: 'job-name' });

// Metrics
await nl.metric({ records: 1500, durationMs: 3200 }, { task: 'job-name' });

Full docs at notilens.com/doc

SDK support

NotiLens has official SDKs for most stacks — no HTTP wiring needed:

# Node.js
npm install @notilens/notilens

# Python
pip install notilens

# PHP
composer require notilens/notilens

# Go
go get github.com/notilens/sdk-go

# Rust
cargo add notilens

# Ruby
gem install notilens

Java and Kotlin available via Maven and Gradle. Shell/CLI also supported — useful for bash scripts and cron jobs with no code changes needed.

Full SDK docs at notilens.com/doc/sdk

The honest truth

You can't watch everything. Nobody on your team can.

But you can instrument the things that matter — revenue, user activity, scheduled jobs, agents — and let a system watch them for you.

The goal isn't a dashboard someone checks every morning. The goal is confidence that if something goes quiet or breaks, the right person finds out before your users do.

That's the only monitoring that matters at this stage.

NotiLens covers everything in this stack — silence detection, webhook monitoring, cron heartbeats, AI agent oversight, and automation monitoring. 7-day free trial, no credit card required.

We're giving eligible founders, small teams, and startups 3 months free in exchange for honest feedback — reach out directly if that's interesting.

Our SaaS stopped getting signups at 2am. No alerts fired. Here's why.

Stephen Souza — Tue, 05 May 2026 12:02:37 +0000

It was a Tuesday morning. I opened my laptop, checked the dashboard, and noticed something off.

No signups since 2:17am.

Not one. For almost six hours.

My first thought: slow day. Maybe people just weren't signing up. It happens.

My second thought, ten minutes later after staring at the chart: this has never happened before.

Everything was "up"

Here's the part that still bothers me.

My uptime monitor? Green.
Server health? Normal.
Error logs? Clean.
SSL cert? Valid.
API response time? Fine.

By every traditional measure, my product was working perfectly. Which meant none of my alerts fired. Not a single one.

But my signup flow was completely broken.

What actually happened

Somewhere around 2am, something in our signup flow started returning responses that looked like success — but weren't. Not a crash. Not an error. Just silent wrong behaviour. The kind that look like success to a health check but actually fail when a real user tries to sign up.

The signup form submitted. The spinner spun. Then nothing. No account created. No error shown to the user. Just a quiet dead end.

Users didn't get an error. They got silence. And they left.

For six hours.

I found out when a user emailed me saying they'd tried to sign up three times the night before and gave up.

That email hurt more than the outage itself.

The monitoring gap nobody talks about

Most monitoring tools are built around one question: "Is it up?"

Is the server responding? Is the endpoint returning 200? Is the cert valid?

These are good questions. But they're the wrong question for this failure.

The right question was: "Is anything actually happening?"

Specifically — are signups happening at the rate they normally do? Because on a normal Tuesday at 2am, even with low traffic, I get some signups. When that number drops to zero for six hours, something is wrong. Always.

But no tool I was using asked that question. They were all watching the pipes. Nobody was watching the water.

The difference between "up" and "working"

Your server can be up. Your API can respond. Your database can be connected.

And your business can still be silently bleeding.

This happens more than people admit:

Payment webhooks stop arriving — Stripe has a hiccup, webhooks queue and don't retry correctly. Your server never goes down. Your revenue processing stops.
Emails stop sending — your SMTP provider throttles you, but returns 200s. Onboarding emails never arrive. Users churn thinking you don't care.
Cron jobs silently skip — the job "runs" but processes zero records due to a config drift. No error. No alert. Your data pipeline is stale for days.
Signup flow breaks — exactly what happened to us. The form works. The backend "responds." Zero accounts created.
AI agents go quiet — agent is "running" but stops producing outputs. No exception, no crash. Task queue drains, nothing gets done.
AI agents loop infinitely — agent keeps retrying the same step, burning tokens and API credits silently. No alert. You see the bill at end of month.
AI agents get stuck — tool call hangs waiting for a response that never comes. The agent neither fails nor succeeds. Just... waits. Forever.

In every one of these cases, traditional monitoring sees nothing wrong. Because technically, nothing is wrong with the infrastructure. The failure is at the business event level — the things that actually matter.

What I wished I had

I didn't want another dashboard to stare at.

I wanted something to notice that signups had gone quiet — and tell me.

Not because I configured a threshold. Not because I manually set up an alert for "signups < 1 per hour." But because the system knew what normal looked like for my app at 2am on a Tuesday, and knew that zero signups for six hours was abnormal.

Basically: I wanted the monitoring equivalent of someone who's worked at my company long enough to say "hey, something feels off today."

What We built

After this incident — and two others like it in the same week (a CPU spike that correlated with a memory leak, and an anomalous jump in signups that turned out to be a bot wave) — We built NotiLens.

The core idea is what We call Smart Silence Detection.

Instead of asking "is the server up?", it asks "is your business behaving normally?"

It learns your baseline — what normal event volume looks like at each hour of the day, each day of the week. Then it alerts you when things go abnormally quiet. No manual threshold configuration needed. No spreadsheet of "expected events per hour." It just learns, and it watches.

When signups stop. When webhooks dry up. When your cron job runs but processes nothing. When orders drop to zero on a Saturday afternoon. When your AI agent stops producing outputs. When it starts looping through the same step burning your API credits. When it hangs waiting for a tool response that never comes.

That's the alert you need. Not "server down." But "your business just went silent."

There's a second pattern it catches that We didn't even plan for initially: broken flows.

Silence detection watches for events that stop happening altogether. Broken flow detection watches for events that start but never finish.

payment.initiated fired. But payment.completed never followed.

user.registered fired. But user.activated never followed.

order.placed fired. But order.fulfilled never followed.

Each individual event looks fine. No errors. No timeouts. The payment was "initiated" — technically true. But the money never moved.

This is where most revenue leaks actually happen. Not in crashes. In the gap between two events that should always travel together — but sometimes don't.

The practical setup (for the technically curious)

The way it works under the hood:

You send business events to NotiLens via a simple SDK call or webhook — signup.completed, payment.received, order.placed, whatever matters to your app.
NotiLens builds a rolling baseline of expected event frequency using ML — per event type, per hour of day, per day of week.
When observed frequency drops significantly below baseline for a sustained period, it fires an alert — push notification to your phone, Slack, email, whatever you have set up.
You also get anomaly detection in the other direction: sudden spikes (bot attacks, viral traffic, billing loops) are caught too.
For broken flows, you define the relationship between two events — payment.initiated should always be followed by payment.completed within X minutes. If it isn't, you get alerted immediately. No polling. No manual checks.

What I learned

A few things I'd tell myself before that Tuesday morning:

1. "No alerts" is not the same as "no problems."
Silence from your monitoring tools means your infrastructure is up. It says nothing about whether your business is working.

2. The failures that hurt most are the ones users experience silently.
A full server outage is obvious. Users tweet about it, you get flooded with emails, you know within minutes. A broken signup flow at 2am? You find out when a user emails you three days later — if they email at all. Most just leave.

3. Business events are first-class monitoring targets.
signup.completed, payment.received, user.activated — these deserve the same monitoring attention as CPU and memory. Maybe more.

4. Absence of data is data.
Zero signups for six hours is a signal. Treat it like one.

If any of this sounds familiar — if you've had that moment of "wait, when did this stop working?" — I'd love to hear your story in the comments.

And if you want to try NotiLens — if you're a solo founder, running a small team, building with AI agents, or just tired of juggling multiple systems with no single place watching whether your business is actually working — we're giving early users 3 months free in exchange for honest feedback. Just reach out directly or drop a comment below.