DEV Community: Wajahat Ali Abid

Automated On-Call Roster Alerting System

Wajahat Ali Abid — Wed, 22 Jul 2026 17:24:00 +0000

Overview

I built an automated on-call alerting system that reads team schedules from a central Google Sheet and sends Slack notifications tagging the appropriate on-call person every weekday. The system supports multiple scheduling models (weekday-based and sprint-based) and handles team-specific channel routing, all without requiring teams to understand the underlying automation.

Problem Statement

The challenge was to provide a flexible on-call notification system that:

Supports different scheduling preferences across teams (per-day, per-sprint)
Requires no technical knowledge from teams managing their own rosters
Avoids alert fatigue by not resending unchanged sprint assignments
Delivers notifications to both a central channel and team-specific channels
Operates autonomously with zero maintenance from the DevOps team once a new team onboards

Solution Architecture

The system uses a Google Sheet as the single source of truth and a scheduled Lambda function that reads it, resolves on-call assignments, and delivers Slack alerts.

Scheduling: EventBridge rules trigger the Lambda on weekdays at two different times for two modes
Data Source: Google Sheets API reads team tabs and reserved system tabs
Resolution Logic: Determines the on-call person based on the team's tab structure
State Management: A cache tab tracks current assignments to detect changes
Notification: Slack API delivers alerts with user mentions to appropriate channels

Implementation Details

Scheduling Modes

The system supports two modes, automatically detected by the column headers in each team's worksheet:

Problem Statement

The challenge was to provide a flexible on-call notification system that:

Supports different scheduling preferences across teams (per-day, per-sprint)
Requires no technical knowledge from teams managing their own rosters
Avoids alert fatigue by not resending unchanged sprint assignments
Delivers notifications to both a central channel and team-specific channels
Operates autonomously with zero maintenance from the DevOps team once a new team onboards

Solution Architecture

The system uses a Google Sheet as the single source of truth and a scheduled Lambda function that reads it, resolves on-call assignments, and delivers Slack alerts.

Scheduling: EventBridge rules trigger the Lambda on weekdays at two different times for two modes
Data Source: Google Sheets API reads team tabs and reserved system tabs
Resolution Logic: Determines the on-call person based on the team's tab structure
State Management: A cache tab tracks current assignments to detect changes
Notification: Slack API delivers alerts with user mentions to appropriate channels

Implementation Details

Scheduling Modes

The system supports two modes, automatically detected by the column headers in each team's worksheet.

Default Mode (12:00 UTC daily)

For teams that assign on-call per weekday. The system checks today's day of the week and looks up the matching person. Every weekday, an alert is sent regardless of whether the person changed.

Detection: Team tab has a "Day of week" column.

Sprint Mode (05:00 UTC daily)

For teams that assign on-call per sprint or date range. Each row has From/To dates. The system finds which range covers today, then checks if the person differs from the last alert sent. Only when the person changes does a new alert go out.

Detection: Team tab has both "From" and "To" columns.

This means teams never need to configure which mode they use — the system infers it from how they structure their sheet.

Technology Stack

Google Sheets API: Central data source (schedules, user mappings, team config)
AWS Lambda: Serverless execution of the alerting logic
EventBridge: Scheduled triggers for both modes
Slack API: Notification delivery with user mentions
Python: Core implementation language

Core Components

1. On-Call Job (`on_call_job.py`)

The main Lambda handler that orchestrates the entire flow:

Event Configuration:

{"id": "Default"}  // or {"id": "Sprint"}

Execution Flow:

Validates the job identifier and skips weekends
Opens the Google Sheet and reads all team tabs
For each team, detects scheduling mode from headers
Skips teams that don't match the current mode
Resolves the on-call person using mode-specific logic
For Sprint mode: compares against cache and skips if unchanged
Looks up the person's Slack user ID from the data tab
Sends alert to the central channel
Sends alert to dedicated team channel if configured
Updates the cache with the current assignment

Helper Class (_OnCallJobHelper):
Static methods encapsulating the resolution logic:

is_sprint_team(headers) — detects mode from column headers
resolve_default_on_call(...) — day-of-week or cache-based lookup
resolve_sprint_on_call(...) — date range matching
has_on_call_person_changed(...) — cache comparison for deduplication

Google Sheet Structure

Reserved System Tabs

Tab	Purpose
`_Data`	Maps team member names to Slack user IDs
`_Cache`	Tracks current on-call person per team (managed by automation)
`_Teams`	Team configuration (dedicated Slack channel)

Team Tabs — Default Mode

Day of week	Name	Backup
Monday	Alice Smith	Bob Jones
Tuesday	Charlie Brown

Team Tabs — Sprint Mode

From	To	Name	Backup
2026-07-14	2026-07-25	Alice Smith
2026-07-28	2026-08-08	Bob Jones

Alert Deduplication (Sprint Mode)

The key design challenge for sprint-based teams was avoiding daily alert spam. The solution:

On each run, resolve today's on-call person from the date ranges
Compare against the _Cache tab which stores the last alerted person
If unchanged, silently skip — no alert sent
If different (new sprint started), send alert and update cache

This means sprint teams only see a notification at the boundary between sprints.

Multi-Channel Alerting

Any team can configure a Dedicated Channel in the _Teams tab. When set, the alert is sent to both the central channel (for cross-team visibility) and the team's own channel (for focused notification).

Self-Service Onboarding

Teams onboard themselves without DevOps involvement:

Add members to _Data tab with Slack user IDs
Add team config to _Teams tab
Create a worksheet tab with (tbd) suffix while setting up
Populate the schedule in their preferred format
Remove (tbd) from the tab name — alerts begin automatically

The (tbd) convention acts as a feature flag at the data layer.

Design Decisions

Why Google Sheets?

Non-technical users can manage schedules without learning new tools
Different teams across the organization already use Sheets daily — no adoption friction
Built-in tab-level protection allows locking down each team's tab so only authorized members can edit their own roster, preventing accidental cross-team changes
Teams can self-service without submitting tickets to DevOps
Immediate visibility into who is on call (just open the sheet)
No deployment required to update schedules

Why Detect Mode from Headers Instead of Configuration?

Zero configuration burden on teams
Impossible to misconfigure — the structure IS the configuration
Adding a "From" and "To" column is the only step to switch modes
No separate config tab entry to keep in sync

Why Cache-Based Deduplication?

Sprint-based teams would get the same alert every single day otherwise
Polling the sheet daily is cheap; sending alerts daily is noisy
Cache comparison is simple (string equality on name) and reliable
Cache also serves as a "last known state" for operational visibility

Why Two Separate Schedules?

Sprint alerts at 05:00 UTC so teams know about changes before workday begins
Default alerts at 12:00 UTC (noon) as a mid-day reminder of who is on call
Separating them makes it easy to adjust timing per mode independently
Each invocation only processes relevant teams, keeping execution fast

Why Skip Weekends?

On-call coverage during weekends follows a different process (PagerDuty)
Sending alerts to empty offices creates noise and degrades trust in the system
Weekend coverage for critical systems uses separate escalation paths

Operational Considerations

Common Failure Modes

Name mismatch: Names in team tabs must exactly match _Data (case-sensitive)
Missing date range: Sprint teams with no row covering today get silently skipped
Empty Slack ID: Person gets mentioned as empty string in Slack (no notification)

Maintenance Requirements

Sprint teams must add future date ranges before current ones expire
New team members need entries in _Data with correct Slack user IDs
The _Cache tab should never be manually edited

Monitoring

Lambda execution logs capture which teams were processed and which were skipped
Warning-level log emitted when no sprint range covers today for a team
Error-level log for unknown job identifiers

Lessons Learned

Let the Data Be the Config: Detecting mode from sheet structure eliminated an entire category of misconfiguration errors
Alert Fatigue is Real: The cache-based deduplication for sprint mode was critical — without it, teams would have muted the channel within a week
Self-Service Scales: The (tbd) convention for in-progress teams meant zero DevOps tickets for onboarding after the first month
Non-Technical Users Need Guardrails, Not Documentation: Case-sensitive name matching is the #1 support issue — explicit validation in the code would be a worthwhile future addition
Separate Concerns by Time: Running Default and Sprint at different times simplified debugging and made it possible to adjust cadence per mode without code changes

Secure Docker Builds with Secrets & Multi-Platform CI

Wajahat Ali Abid — Fri, 10 Jul 2026 13:30:16 +0000

1. Concept: Build Secrets vs. Build Args

When building Docker images, you often need sensitive data (NuGet tokens, NPM keys, API credentials) to fetch dependencies.

Build Args (ARG): These are stored in the image's history. Running docker history <image> will reveal these values to anyone with access to the image. Never use these for secrets.
Docker Secrets (--secret): These are temporarily mounted into a specific RUN command. They exist only in memory or as a temporary file during that specific layer's execution. They are never committed to the image layers.

2. The Universal Dockerfile Template

This template uses a multi-stage build, a non-root user for security, and the Secret Mount syntax.

# --- Stage 1: Build ---
FROM mcr.microsoft.com/dotnet/sdk:10.0-bookworm-slim AS build-env
WORKDIR /app

# Non-sensitive configuration via ARGs
ARG PACKAGE_SOURCE_URL
ARG SOURCE_NAME
ARG SOURCE_USER

# Copy project files for restoration
COPY *.slnx ./
COPY src/**/*.csproj ./src/

# Use the Secret Mount to safely handle the token
# The id 'app_token' must match the CLI --secret id
RUN --mount=type=secret,id=app_token \
    TOKEN=$(cat /run/secrets/app_token) && \
    dotnet nuget add source "$PACKAGE_SOURCE_URL" \
    --name "$SOURCE_NAME" \
    --username "$SOURCE_USER" \
    --password "$TOKEN" \
    --store-password-in-clear-text && \
    dotnet restore

# Copy remaining source and publish
COPY . .
RUN dotnet publish -c Release -o /app/output --no-restore

# --- Stage 2: Runtime ---
FROM mcr.microsoft.com/dotnet/runtime:10.0-bookworm-slim
WORKDIR /app

# Security: Create a non-root user
# 'useradd' is used instead of 'adduser' for compatibility with 'slim' images
RUN useradd -m -d /app -s /bin/bash appuser

# Copy output and set ownership
COPY --from=build-env --chown=appuser:appuser /app/output .

# Optional: Install diagnostics tools
RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

USER appuser
ENTRYPOINT ["dotnet", "MyApplication.dll"]

3. CI/CD Integration

A. GitHub Actions Implementation

GitHub Actions requires you to explicitly map a Secret to an environment variable in the shell before Docker can consume it.

- name: Build Docker Image
  env:
    # Map the GitHub Secret to a shell environment variable
    READONLY_TOKEN: ${{ secrets.ORG_READONLY_TOKEN }}
    DOCKER_BUILDKIT: 1 # Ensure BuildKit is enabled
  run: |
    docker build \
      --secret id=app_token,env=READONLY_TOKEN \
      --build-arg PACKAGE_SOURCE_URL="${{ secrets.SOURCE_URL }}" \
      --build-arg SOURCE_NAME="InternalFeed" \
      --build-arg SOURCE_USER="${{ secrets.SOURCE_USER }}" \
      -t my-app:latest .

B. AWS CodeBuild Implementation

CodeBuild retrieves secrets from AWS Secrets Manager and populates them as environment variables automatically.

version: 0.2

env:
  # Pulls directly from AWS Secrets Manager
  secrets-manager:
    BUILD_TOKEN: "path/to/secret:key_name"

phases:
  build:
    commands:
      - export DOCKER_BUILDKIT=1
      - |
        docker build \
          --secret id=app_token,env=BUILD_TOKEN \
          --build-arg PACKAGE_SOURCE_URL="${SOURCE_URL}" \
          --build-arg SOURCE_NAME="InternalRepo" \
          --build-arg SOURCE_USER="${SOURCE_USER}" \
          -f Dockerfile \
          -t $REPOSITORY_URI:latest .

4. Critical Troubleshooting Guide

Issue	Root Cause	Solution
`adduser: not found`	Using a Debian "slim" image.	Use `useradd` (the low-level binary) instead of `adduser`.
`Both UserName and Password must be specified`	A variable passed to `dotnet nuget` is empty.	1. Ensure `ARG` is declared after the `FROM` line. 2. Wrap all vars in double quotes `"$VAR"`.
`Secret not found`	ID mismatch between CLI and Dockerfile.	Ensure `--secret id=XYZ` matches `RUN --mount=type=secret,id=XYZ`.
`Invalid Build Flag`	BuildKit is disabled.	Set `DOCKER_BUILDKIT=1` or use `docker buildx build`.

5. Best Practices Checklist

[ ] Multi-stage builds: Always separate the SDK (Build) from the Runtime (Execution) to minimize attack surface.
[ ] Non-root execution: Never run your application as root inside the container.
[ ] Layer Optimization: Copy .csproj or .slnx files and restore before copying the full source code to take advantage of layer caching.
[ ] Secret Hygiene: Never echo secrets to the console. To verify if a secret exists during debug, check its length: echo ${#TOKEN}.
[ ] Quotes: Always wrap shell variables in double quotes to prevent word-splitting if a password or URL contains special characters.

bash: No such file or directory

Wajahat Ali Abid — Sat, 30 Mar 2024 18:43:03 +0000

I am using KDE Neon as my daily driver on my laptop. Recently due to KDE Neon update 6.0, there have been many bugs, but one as annoying as the one I faced the other day.

After system update, my bash stopped working completely. Granted I use zsh as my shell, bash is required by many programs on Linux. But everytime, I opened any such program, say pyenv, it just wouldn't work and give following error

/usr/bin/env: ‘bash’: No such file or directory linux

I looked into /usr/bin and couldn't find bash there. I couldn't install bash using dpkg or apt. This was pretty perplexing for me. Here is how I resolved the error with bash.

First of all, I edited the file /var/lib/dpkg/info/bash.prerm and changed first line to #! /bin/sh from #! /bin/bash

#! /bin/sh
set -e  

case "$1" in  
   upgrade)  
       update-alternatives --remove builtins.7.gz \  
           /usr/share/man/man7/bash-builtins.7.gz  
       ;;  

   remove|deconfigure)  
       ;;  

   failed-upgrade)  
       ;;  
   *)  
       echo "prerm called with unknown argument \`$1'" >&2  
       exit 1  
       ;;  
esac

Next I looked into /var/cache/apt/archives/ and found the file bash_5.1-6ubuntu1.1_amd64.deb. If you cannot find this file, you can download from here
Next I ran the command

sudo dpkg --force-remove-reinstreq -i /var/cache/apt/archives/bash_5.1-6ubuntu1.1_amd64.deb

This helped me restore the bash and helped me run programs again.

Stream Windows Event Logs to Cloudwatch

Wajahat Ali Abid — Mon, 04 Mar 2024 19:29:18 +0000

I received a requirement for sending Windows event logs to Amazon Cloudwatch because we wanted to monitor user activity on various Windows servers in our environment. There can be various other use cases for this requirement, however we will focus on setting up Windows event logs to Cloudwatch in this article.

To achieve this, we first need to install Amazon Cloudwatch Agent and then configure the server to push logs to Cloudwatch Logs.

Install Cloudwatch Agent

There are a couple of ways you can install Amazon Cloudwatch Agent on your servers.

Configure Iam Role

Create a policy CloudwatchAgentPolicyForWindowsLogging with following body

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudwatchLogsStatement",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:us-west-2:123456789012:log-group:windows-event-viewer-logs",
                "arn:aws:logs:us-west-2:123456789012:log-group:windows-event-viewer-logs:log-stream:*"
            ]
        }
    ]
}

Attach this policy to any server you want to enable Cloudwatch Logs on.

Configure Cloudwatch Agent

Go to C:\Program Files\Amazon\AmazonCloudWatchAgent and create a file config.json

Add logs section to Amazon Cloudwatch Agent configuration file.

{ 
    ...,
    "logs": {
        "logs_collected": {
            "windows_events": {
                "collect_list": [
                    {
                        "event_format": "xml",
                        "event_levels": [
                            "VERBOSE",
                            "INFORMATION",
                            "WARNING",
                            "ERROR",
                            "CRITICAL"
                        ],
                        "event_name": "System",
                        "log_group_name": "windows-event-viewer-logs",
                        "log_stream_name": "{instance_id}/System",
                        "retention_in_days": 365
                    },
                    {
                        "event_format": "xml",
                        "event_levels": [
                            "VERBOSE",
                            "INFORMATION",
                            "WARNING",
                            "ERROR",
                            "CRITICAL"
                        ],
                        "event_name": "Security",
                        "log_group_name": "windows-event-viewer-logs",
                        "log_stream_name": "{instance_id}/Security",
                        "retention_in_days": 365
                    }
                ]
            }
        }
    }
}

Open Powershell and run the following command

> cd 'C:/Program Files/Amazon/AmazonCloudWatchAgent/'
> ./amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:config.json -s

Start the Amazon Cloudwatch Agent Service and after some time, you'll see log stream created in the log group windows-event-viewer-logs.

Verify

Open Cloudwatch Logs and open the log group windows-event-viewer-logs.

Enable ContainerLogV2 on AKS

Wajahat Ali Abid — Wed, 28 Feb 2024 22:09:43 +0000

ContainerLogV2 provides additional data over ContainerLog (which is v1). The additional data includes

ContainerName
PodName
PodNamespace

To enable ContainerLogV2 on an existing AKS cluster, you need to create a configmap. Here's how you do this

Download the template ConfigMap YAML file for ContainerLog
Set containerlog_schema_version = "v2" in the yaml file
Update the exclude_namespaces under log_collection_settings.stdout and log_collection_settings.stderr as per your use case
Save the changes
Connect to Kubernetes cluster using CLI

kubectx my-cluster

Run the following command to create the configmap.

kubectl apply -f container-azm-ms-agentconfig.yaml

Done

I hope this helps. Refer to Container insights log schema and Configure data collection in Container insights using ConfigMap for more insights.

Yet another implementation for Slack Commands

Wajahat Ali Abid — Mon, 19 Sep 2022 13:27:48 +0000

We're using slack for communication within our team. Using slack slash commands, we also handle common day-to-day tasks like triggering deployments on different servers based on requirements. For a long time, that remained our primary use case, so we had a single AWS Lambda function taking care of this job, however, with the growing team and business, we felt the need to handle several other tasks this way. Thus we created a few more lambda functions that handled different tasks but the behavior among different lambda functions wasn't consistent, since every lambda was maintained separately.

General Idea

We decided to create a single lambda function that will be solely used for handling all the slash command requirements. The commands will follow a similar structure as normal shell commands do. Some examples are as follows

Command	Description
help	List down all the slash commands with usage
help deploy	List down help text for deploy commands, including all the options it takes and flags
deploy	Deploy for a specific client

Some examples of slack slash commands are as follows (assuming our slash command is /example)

/example help
/example help deploy
/example help deploy --client test-client --branch master

We can limit users who can issue a command as well as channels where the command can be issued from by using a very simple configuration

{
    "command": "deploy"
    "users": [
        "example.user"
    ],
    "channels": [
        "channel_id"
    ]
}

Implementation

We implemented this solution in python with the help of some useful packages from the open source community. Some of these packages are

Fast Api: For the implementation of the APIs
Mangum: For the integration of the Aws Lambda and the Api Gateway with the FastApi
Python Jenkins: For interacting with the Jenkins server that handles deployments
Argparse: For handling argument parsing

The way our implementation worked was as follows

The User sends a command via slack
Lambda function verifies
- If the command exists
- If the user is allowed to run the command
- If the command is allowed to be run from the specified channel
- If the command has all the required parameters

and responds appropriately

Execute the command and send the response back to the slack channel
Offload the long running commands to the Ecs container (e.g, database backup or restore)

The general architecture of this solution is as follows

This helps us implement new slack commands really quickly. All the error handling is similar across the different commands, so we don't need to handle the errors separately for every new command.

DEV Community: Wajahat Ali Abid

Automated On-Call Roster Alerting System

Overview

Problem Statement

Solution Architecture

Implementation Details

Scheduling Modes

Problem Statement

Solution Architecture

Implementation Details

Scheduling Modes

Default Mode (12:00 UTC daily)

Sprint Mode (05:00 UTC daily)

Technology Stack

Core Components

1. On-Call Job (on_call_job.py)

Google Sheet Structure

Reserved System Tabs

Team Tabs — Default Mode

Team Tabs — Sprint Mode

Alert Deduplication (Sprint Mode)

Multi-Channel Alerting

Self-Service Onboarding

Design Decisions

Why Google Sheets?

Why Detect Mode from Headers Instead of Configuration?

Why Cache-Based Deduplication?

Why Two Separate Schedules?

Why Skip Weekends?

Operational Considerations

Common Failure Modes

Maintenance Requirements

Monitoring

Lessons Learned

Secure Docker Builds with Secrets & Multi-Platform CI

1. Concept: Build Secrets vs. Build Args

2. The Universal Dockerfile Template

3. CI/CD Integration

A. GitHub Actions Implementation

B. AWS CodeBuild Implementation

4. Critical Troubleshooting Guide

5. Best Practices Checklist

bash: No such file or directory

Stream Windows Event Logs to Cloudwatch

Install Cloudwatch Agent

Configure Iam Role

Configure Cloudwatch Agent

Verify

Enable ContainerLogV2 on AKS

Yet another implementation for Slack Commands

General Idea

Implementation

1. On-Call Job (`on_call_job.py`)