The call did not come from our on-call rotation. It came from a customer who noticed two unrelated degradations on their side and asked why we had not paged. We had not paged because Grafana OnCall had been silently dropping alerts for roughly 72 hours. Alertmanager's deliveries were still getting 200 OK, but the Celery task that turns a delivery into an incident failed on every one of them, so nothing new was being recorded at all, and every attempt to acknowledge or resolve the two incidents left over from the upgrade returned HTTP 500. The on-call engineer who first tried to clear one of them that morning had assumed the spinner was a UI bug and moved on. The thing meant to wake us up was the thing that was broken.
Problem signals:
- OnCall UI Acknowledge and Resolve buttons spin and time out with a generic 500
- New alerts from real degradations never appear at all, even though Alertmanager still gets 200 OK: the Celery alert-creation task fails and nothing is recorded
- OnCall pod logs show ORM errors referencing a column that does not exist in the table
- The Helm pre-upgrade migration job reported success but Postgres logs show a lock_timeout on one ALTER TABLE
- There is no Prometheus alert on OnCall's own API error rate, so the regression went undetected
72 hours of dropped alerts and two incidents stuck firing since the upgrade
The alerting platform was the incident
When we got on the bridge, OnCall's incident list would not render at all. The page came back 500, and so did the API behind it. The last state anyone had seen was two incidents in firing state, both written the night of the Helm upgrade three days earlier, both with zero acknowledgment events. Nothing had been escalated since, even though the runbook said any firing incident over 15 minutes old gets escalated, because nothing new had appeared. Nothing new had appeared because nothing new was being recorded: for roughly 72 hours OnCall's ingest endpoint had been answering Alertmanager with 200 OK and then losing the payload in the Celery worker that creates the incident, so every alert it sent was dropped silently. The two stale incidents were not absorbing anything. They were simply the last rows written before the upgrade.
The first thing we tried was the obvious one: clear those two incidents. Acknowledge returned a 500. So did Resolve, so did Snooze, and so did the list endpoint that was supposed to render them, whether we drove it from the UI or called it directly with curl. The web pods were up, the database was reachable, Redis was fine. Nothing in any dashboard suggested a problem, because nobody had built a dashboard that watched OnCall itself.
# note the header: the OnCall HTTP API takes the token raw, with no
# Bearer prefix. Prefixing it gets you a 403, not the 500 we were chasing.
$ curl -s -X POST -H "Authorization: $TOKEN" \
https://oncall.internal/api/v1/alert_groups/I8KZ.../acknowledge/
{"detail": "Internal server error"}
# from the oncall-engine pod. grep -i, and match traceback too: the
# Django traceback carries no uppercase ERROR token to match on.
$ kubectl logs deploy/oncall-engine -c engine --tail=50 | grep -iA2 -e error -e traceback
Traceback (most recent call last):
File "/etc/app/src/engine/apps/api/views/alert_group.py", line 412, in acknowledge
alert_group.acknowledge_by_user(request.user, action_source=ActionSource.WEB)
--
DatabaseError: column alerts_alertgroup.acknowledged_by_confirmation_phone does not exist
LINE 1: ...ledged_by_user_id", "alerts_alertgroup"."acknowledged_by_co...
The ORM was reaching for a column the table did not have.
A silent ALTER TABLE timeout the Helm hook never noticed
Why the migration job exited 0 with a half-finished schema
Our first guess was a bad release. The previous Helm upgrade had bumped OnCall by a minor version, and we assumed the new application code was looking at a field that genuinely had not shipped yet. That was half right. The running code did expect the column, but the schema change that was supposed to create it had never landed, and django_migrations on the OnCall database had no row for that migration at all. That absence is the actual signature of a half-applied schema, not the thing people go looking for. Django records a migration only after its operations return cleanly, so a cancelled statement leaves the row missing while the new pods carry on expecting the column. It is also why migrate would have retried the migration on the next run: with no row recorded, Django still considers it unapplied. That cuts both ways, and it is exactly why the hand-applied fix further down has to be reconciled with django_migrations afterwards using --fake, or the next run re-applies DDL that is already in the table. One note before the artifacts below: the column and trigger names here are anonymized from the client's environment, so do not match them against upstream Grafana OnCall field names.
The clue was in Postgres logs from three days earlier, exactly when the Helm pre-upgrade hook ran the migration job. One line, easy to miss, in the middle of dozens of normal statement logs:
2026-05-11 02:14:07 UTC ERROR: canceling statement due to lock timeout
2026-05-11 02:14:07 UTC STATEMENT: ALTER TABLE alerts_alertgroup
ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
2026-05-11 02:14:07 UTC LOG: duration: 30001.114 ms
alerts_alertgroup is one of the highest-write tables in OnCall. At 02:14 a backlog of inserts was holding row locks, the ALTER hit the lock_timeout we had set globally to 30 seconds (a sensible default we put in years ago to stop one bad migration from wedging the whole database), and Postgres killed the statement. Django did not swallow that. The DatabaseError propagated out of the schema editor, manage.py migrate aborted with a traceback, and the process exited non-zero. What swallowed it was our own job entrypoint: a shell wrapper that ran migrate alongside two other setup steps with no set -e, so the failing command's status was discarded and the script exited 0 on its last line. The Helm hook checked the job's exit code, saw 0, and let the upgrade proceed. ArgoCD synced. The new pods rolled. And from that moment, every code path that touched AlertGroup broke. Django names every concrete field of a model in its default SELECT and INSERT, so a single missing column takes out the ack and resolve path, the list endpoint, and the task that creates alert groups, all alike. The synchronous paths returned 500. Ingest did not: the integration endpoint resolves the AlertReceiveChannel by token, which lives in a different table that still had all its columns, hands the payload to a Celery task with apply_async, and returns 200 immediately. So Alertmanager got a 200 OK for all 72 hours; the AlertGroup row, the only thing that touches alerts_alertgroup and therefore the only thing that hits the missing column, is written inside the worker, and that is where the alert died. That is the diagnostic trap. alertmanager_notifications_failed_total stayed at zero and there were no delivery errors on Alertmanager's side at all, so anyone who checks Alertmanager first rules out the exact cause. The evidence lives in the oncall-celery worker logs and its task-failure counts, carrying the same DatabaseError, which is precisely why the drop was silent.
A plain retry would not have worked either, for a second and separate reason. That migration runs with atomic = False, so the operations that had already succeeded before the ALTER were not rolled back with it, and one of them had left a pgtrigger object behind on alerts_alertgroup, which we only found by querying pg_trigger directly. Django's migration would have blown up on replay trying to create a trigger that already existed. That leftover trigger did not cause the lock_timeout (an existing trigger does not block an ALTER TABLE; only conflicting locks held by other sessions do). It just meant the migration could not be re-run cleanly until we removed it, and the ALTER itself still needed a window where the write backlog was not holding the table.
Why we forward-fixed instead of rolling the Helm release back
Drop the trigger, add the column, then unstick the zombies
We considered rolling back to the previous OnCall version. It looked clean on paper: the old image did not need the missing column, so the schema would match again and acks would work. We talked ourselves out of it for two reasons. First, the new pods had been running for three days and had written data shaped for the new version, including new fields in adjacent tables. A rollback would have meant either accepting writes that the old code did not understand or restoring a 72-hour-old database snapshot, which would erase three days of incident history including the zombies we wanted to clean up. Second, a rollback would leave that migration still unapplied, so the next upgrade attempt would re-run the same ALTER against the same hot table and hit the same lock_timeout the same way. We would be back here in a week.
Forward-fix it was. The sequence had to be careful, because the table was still taking writes and we were going to ALTER it. ADD COLUMN and DROP TRIGGER both take an ACCESS EXCLUSIVE lock, and that conflicts with the ACCESS SHARE lock every plain SELECT takes, so leaving the web tier up would have done two bad things at once: kept a supply of conflicting lock holders sitting in front of us, and then queued every subsequent read behind our waiting ALTER, because Postgres hands out lock requests in order. So we picked a low-write window and drained both tiers, the Celery workers that wrote to alerts_alertgroup and the web pods serving the API, and took a short deliberate outage on the ack path rather than an unbounded stall on the hottest table in OnCall.
-- 1. confirm the column is genuinely missing
SELECT column_name FROM information_schema.columns
WHERE table_name = 'alerts_alertgroup'
AND column_name = 'acknowledged_by_confirmation_phone';
-- (0 rows)
-- 2. find the blocking trigger left over from the failed attempt
SELECT tgname FROM pg_trigger
WHERE tgrelid = 'alerts_alertgroup'::regclass
AND tgname LIKE 'pgtrigger_%';
-- 3. add_column.sql: drop it and ALTER in one transaction, with a
-- SHORT lock_timeout. Fail fast, never hold the queue open.
BEGIN;
SET LOCAL lock_timeout = '2s';
DROP TRIGGER IF EXISTS pgtrigger_oncall_protect_finished
ON alerts_alertgroup;
ALTER TABLE alerts_alertgroup
ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
COMMIT;
# 4. the waiting happens out here, not in the lock manager
for i in $(seq 1 30); do
psql -v ON_ERROR_STOP=1 -f add_column.sql && break
sleep 5
done
# 5. reconcile django_migrations with the DDL we just applied by hand,
# and reinstall the pgtrigger object step 3 dropped. Skip this and the
# next pre-upgrade hook re-runs the migration and dies on DuplicateColumn.
python manage.py migrate alerts 0043_alertgroup_ack_confirmation_phone --fake
python manage.py pgtrigger install alerts.AlertGroup
Short lock_timeout plus an outside retry loop, then a --fake so django_migrations matches the schema.
We did not change the global lock_timeout. Setting it LOCAL keeps the change inside this one transaction, and setting it short is the whole point: a two-second ceiling means a failed attempt lets go immediately instead of parking an ACCESS EXCLUSIVE request at the head of the queue with every read and write piling up behind it. A five-minute wait would have converted a fast failure into a five-minute freeze of the busiest table in the product. The retry loop does the waiting instead. The fourth attempt took the lock. Once the column existed, we brought the web pods and the Celery workers back and watched the engine pod logs. The 500s stopped within seconds, and the next Alertmanager delivery was actually recorded instead of dying in the worker.
Step 5 is the one that is easy to skip and expensive to skip. Applying the DDL by hand leaves django_migrations with no row for that migration, so the next pre-upgrade hook runs manage.py migrate, Django still sees the migration as unapplied, and it dies on psycopg2.errors.DuplicateColumn: column "acknowledged_by_confirmation_phone" of relation "alerts_alertgroup" already exists. Faking the row is what closes that loop. And because step 3 dropped the pgtrigger object the same migration was supposed to install, the trigger has to be reinstalled in the same pass, or the schema stays half-right in the other direction. Hand-applied DDL is not done until django_migrations agrees with it.
That left the two stale incidents. Acknowledging them was not enough. An acknowledged incident still sits in the firing state from OnCall's deduplication perspective, so now that ingest was working again, new alerts matching their labels would fold into them. We had to mark them resolved. We did it through the API first to make sure the lifecycle hooks fired and downstream integrations got the resolved webhook. That worked for one of the two. The API still refused the other for an unrelated reason: its integration had been deleted, so OnCall could not look up the routing to fire the webhook. For that single record we set resolved=TRUE and resolved_at to the current timestamp in the database directly, with a note in the incident's raw payload explaining the manual close.
We then fired a synthetic alert from Alertmanager and watched a new incident appear, ack it from the UI in under two seconds, resolve it, and confirm a follow-up alert created a fresh incident instead of folding into the resolved one. That was the real all-clear.
Meta-monitoring for the platform that does the monitoring
What we wired up so the next silent migration trips an alarm
The thing that kept us up afterward was not the migration. Migrations fail. Database locks happen. The thing that kept us up was that OnCall had been broken for three days and not one signal in our monitoring stack had told us. We had alerts on Prometheus being down, on Alertmanager being down, on Grafana being down, on every customer-facing service. We had nothing watching the incident management platform itself.
We added two rules the same week. The first is a straight error-rate alert on OnCall's API. If more than 1% of requests to /api/v1/ return 5xx for five minutes, page the platform team at critical severity. Five minutes is short enough that a real outage gets caught but long enough that a single bad deploy rolling does not page. We picked critical because if OnCall is degraded, nothing else paging matters; alerts get swallowed.
# --- Prometheus rules ---
groups:
- name: oncall-meta
rules:
- alert: OncallApiErrorRateHigh
expr: |
sum(rate(django_http_responses_total_by_status_total{job="oncall",status=~"5.."}[5m]))
/
sum(rate(django_http_responses_total_by_status_total{job="oncall"}[5m]))
> 0.01
for: 5m
labels:
severity: critical
service: oncall
annotations:
summary: "OnCall API returning >1% 5xx for 5m"
runbook: "https://internal/runbooks/oncall-api-errors"
# --- Loki ruler rules (loki-rules ConfigMap, NOT Prometheus) ---
# A job that exits 0 is Complete, so kube_job_status_failed never
# moves. The log stream is the only place the failure exists. Match it
# with |~ and (?i): |= is a case-sensitive literal substring filter and
# the Django traceback contains no uppercase ERROR token anywhere.
groups:
- name: oncall-meta-logs
rules:
- alert: OncallMigrationJobErrors
expr: |
count_over_time({namespace="oncall", app="migration"} |~ "(?i)traceback|error" [10m]) > 0
for: 1m
labels:
severity: critical
service: oncall
The second rule is a Loki ruler rule, not a Prometheus one: job status cannot see an exit-0 failure.
The second rule is the lesson from this specific incident. Helm trusts the exit code, and here the exit code was a lie told by our own wrapper script, not by Django. Kubernetes agreed with the lie: the Job was Complete, kube_job_status_failed stayed at zero, and no Prometheus rule built on job status could have caught it. The only place the truth lived was the job's log stream, which is why the rule runs in the Loki ruler and matches on the text of the failure itself, from any pod with the migration label in the oncall namespace, regardless of whether the job reported success. The match expression matters as much as the ruler does. LogQL's |= is a case-sensitive literal substring filter, and what we needed to catch was a Python traceback reading django.db.utils.OperationalError: canceling statement due to lock timeout, which contains no uppercase ERROR token at all. A |= "ERROR" rule would have matched nothing on the exact failure it was written for, which is the same silent no-op we were trying to alarm on in the first place. The same trap catches you at the keyboard, which is why the grep further up runs with -i and an alternated -e traceback: a case-sensitive grep ERROR on that pod returns nothing and reads as "the engine is clean." Use a case-insensitive regex filter instead, |~ "(?i)traceback|error", or the tighter |~ "(?i)(traceback \(most recent call last\)|django\.db\.utils\.\w*Error)", and verify it against a captured log from a real failing job before you trust it. If you copy it, also make sure your migration pods actually ship logs to Loki; the rule matches nothing otherwise. We have caught two real issues with it in the months since (neither as bad as this one, both worth knowing about within minutes instead of days).
The broader pattern, and one we now apply on every recovery engagement we run, is that any tool you depend on to notice problems needs an independent way to notice when that tool itself is the problem. We have written more about this category of failure in our migration recovery work, because the same shape appears in database cutovers, queue platform upgrades, and identity provider migrations: the system you rely on to tell you the truth is the system that has stopped telling the truth, and you only find out from a customer.
When acks are silently 500ing and you cannot tell what data is real
If your OnCall is doing this right now
The hard part of this incident is not the SQL. The hard part is making the call between forward-fix and rollback when your incident history, your zombie state, and your live alert routing are all entangled in a database that is currently being written to by application code that expects a schema it does not have. Roll back without a plan and you lose three days of incident records. Forward-fix without checking for leftover triggers and migration locks and your second attempt fails the same way as the first. Run an ALTER on a hot table with a long lock_timeout and the whole table queues behind you until it finishes or gives up.
We do these engagements every few weeks. Partial Django migrations on Grafana OnCall is the specific case we have now seen three times this year, twice from lock_timeout and once from a leftover pgtrigger object that made the migration fail on replay. Adjacent variants we have handled: Sentry post-deploy migrations that left a column nullable when the code expected NOT NULL, Mattermost upgrades where one index creation timed out, Keycloak realm migrations that completed on the primary but failed on a replica. The pattern is identical and the recovery sequence rhymes.
If your team is staring at a 500 on every ack and trying to decide whether to roll back the Helm release, book an infrastructure review with our team and we will be on a bridge with you the same day. We will help you confirm the schema delta, plan the forward-fix or the rollback with the data implications spelled out, and clean up the stuck incidents without losing the history you need for the postmortem.
Originally published at https://infraforge.agency/insights/grafana-oncall-stuck-incidents-partial-migration/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)