DEV Community

Cover image for System Design: How Do We Host an App That Runs a Job Every 5 Minutes?
Khushi Patel
Khushi Patel

Posted on

System Design: How Do We Host an App That Runs a Job Every 5 Minutes?

#devops #programming #cloud #automation

Imagine you're building an app that needs to perform a task every 5 minutes.

Examples:

  • Send reminder notifications
  • Check product expiry dates
  • Generate daily reports
  • Sync data from third-party APIs
  • Clean old records from the database

A common beginner question is:

If nobody opens my app, who will trigger these jobs every 5 minutes?

Let's break down how this works in production.

Option 1: Cron Job on a Server

The simplest solution is using a Cron Job.

A cron job is a scheduler built into Linux that can execute commands at specific intervals.

Example:

*/5 * * * * node reminderJob.js
Enter fullscreen mode Exit fullscreen mode

This means:

Every 5 minutes
Run reminderJob.js
Enter fullscreen mode Exit fullscreen mode

Architecture:

┌─────────────┐
│ Linux Server│
└──────┬──────┘
       │
       ▼
  Cron Scheduler
       │
       ▼
   Node.js Job
       │
       ▼
    Database
Enter fullscreen mode Exit fullscreen mode

Works well for:

  • Small projects
  • Internal tools
  • MVPs

Problem With a Single Server

What happens if the server crashes?

Server Down
     ↓
Cron Stops
     ↓
Jobs Missed
Enter fullscreen mode Exit fullscreen mode

For critical applications, this is unacceptable.

Imagine:

  • Payment settlement jobs
  • Order processing
  • OTP cleanup
  • Subscription renewals

Missing even one execution can create problems.

Option 2: Dedicated Background Worker

Large systems separate API servers and background workers.

Architecture:

        Users
          │
          ▼
    API Servers
          │
          ▼
      Database

          ▲
          │

    Worker Service
          │
          ▼
    Scheduled Jobs
Enter fullscreen mode Exit fullscreen mode

Benefits:

  • APIs remain fast
  • Jobs don't affect user requests
  • Easier to scale independently

Option 3: Kubernetes CronJobs

Modern cloud applications often run on Kubernetes.

Kubernetes provides a resource called CronJob.

Example:

schedule: "*/5 * * * *"
Enter fullscreen mode Exit fullscreen mode

Every 5 minutes Kubernetes automatically creates a container and runs the job.

Architecture:

      Kubernetes
           │
           ▼
      CronJob
           │
           ▼
    New Container
           │
           ▼
       Execute Job
Enter fullscreen mode Exit fullscreen mode

Advantages:

  • Automatic retries
  • Auto-healing
  • Easy deployment
  • Cloud-native

Preventing Duplicate Execution

Suppose you have 3 servers.

Server A
Server B
Server C
Enter fullscreen mode Exit fullscreen mode

If all three execute the same job:

Send Notification
Send Notification
Send Notification
Enter fullscreen mode Exit fullscreen mode

Users receive the same notification three times.

Not good.

Distributed Locking

To ensure only one server executes the job, companies use distributed locks.

Popular choices:

  • Redis Lock
  • Database Lock
  • ZooKeeper
  • etcd

Flow:

Worker Starts
      │
      ▼
Acquire Lock
      │
      ▼
Lock Success ?
   /       \
 Yes        No
  │          │
  ▼          ▼
Run Job    Exit
Enter fullscreen mode Exit fullscreen mode

Only one worker gets the lock.

How Big Companies Do It

Companies rarely depend on a single cron job.

Typical architecture:

                Scheduler
                    │
                    ▼
              Message Queue
                    │
        ┌───────────┼───────────┐
        ▼           ▼           ▼
     Worker 1    Worker 2    Worker 3
        │           │           │
        └───────────┼───────────┘
                    ▼
                 Database
Enter fullscreen mode Exit fullscreen mode

Popular technologies:

  • Cron
  • Kubernetes CronJobs
  • Redis
  • RabbitMQ
  • Apache Kafka
  • AWS EventBridge
  • Google Cloud Scheduler

The scheduler creates tasks.

Workers process those tasks.

This makes the system scalable and fault tolerant.

Example: Expiry Reminder App

Let's say you built an app that stores expiry dates.

Every 5 minutes:

  1. Scheduler triggers a job.
  2. Job finds products expiring soon.
  3. Notification tasks are added to a queue.
  4. Workers send notifications.
  5. Results are stored in the database.

Architecture:

Cron Scheduler
      │
      ▼
Expiry Checker
      │
      ▼
Database Query
      │
      ▼
Message Queue
      │
      ▼
Notification Workers
      │
      ▼
Firebase / APNs
Enter fullscreen mode Exit fullscreen mode

Key Takeaway

A scheduled task running every 5 minutes is usually not handled by user traffic.

Production systems use:

  • Cron Jobs
  • Background Workers
  • Kubernetes CronJobs
  • Message Queues
  • Distributed Locks

The bigger the scale, the more important reliability, retries, and duplicate prevention become.

Next time you receive a notification exactly on time, remember:

Somewhere in the cloud, a scheduler woke up, executed a job, and triggered that notification.

Top comments (0)