Building a Reliable Notification Service in .NET

A production-grade notification service in ASP.NET Core should deliver notifications through multiple channels, including Email, Push Notifications, WhatsApp, SMS, and Telegram. It should also handle failures, retries, background processing, duplicate messages, and high traffic without slowing down the main application.

For a platform like MindStick, a centralized notification service with a queue-based architecture is a strong starting point. It lets different modules request notifications through a common system while keeping delivery logic separate from business logic.

1. Recommended architecture

The central principle is that business modules request notifications, but dedicated workers deliver them. A comment service should not need to know how SMTP, Firebase, or WhatsApp APIs work.

2. Core components

Component	Responsibility
Notification API	Accept requests and validate inputs
Notification orchestrator	Decide recipients, channels, templates, and preferences
SQL Server	Store notification records, attempts, and delivery states
Hangfire	Schedule and execute background jobs
Message broker	Buffer and distribute high-volume workloads
Channel providers	Send email, push, SMS, WhatsApp, or Telegram messages
Retry manager	Retry temporary failures with backoff
Monitoring	Track failures, queue depth, latency, and delivery rates

For a first production version, ASP.NET Core + SQL Server + Hangfire can be sufficient. Introduce a broker such as RabbitMQ or Azure Service Bus when workload volume, throughput, or service isolation justifies the extra infrastructure.

3. How a notification flows through the system

Consider a MindStick user receiving a reply to a question they posted.

1. An event occurs. A user posts a reply to a question.
2. Create a notification request. The application records the intended recipient, event type, and relevant content identifiers.
3. Persist before delivery. Store the notification durably. Ideally, use a transactional outbox so the business change and notification event cannot silently get out of sync.
4. Process asynchronously. A Hangfire job or broker consumer retrieves the pending notification.
5. Select channels. Check user preferences and available contact details. The user may have enabled email and push but disabled WhatsApp.
6. Send through a provider. Render the correct template and call the channel adapter.
7. Record the outcome. Store the provider response, attempt count, timestamps, and resulting status.

The user-facing request should normally finish without waiting for the external provider to deliver the message.

4. Designing an extensible channel system

Use a common interface so new notification channels can be added without rewriting the orchestrator.

Conceptually, every provider should support:

• Accepting a notification and its recipient.
• Rendering or receiving a prepared message.
• Sending through its underlying provider API.
• Returning a normalized success, retryable failure, or permanent failure result.

For example, an IEmailNotificationProvider can encapsulate email delivery, while an IPushNotificationProvider encapsulates push delivery. A shared orchestration layer handles routing and status tracking.

Avoid putting all channel-specific code in one large service class with multiple if/else blocks. That becomes difficult to test and maintain as new providers are added.

5. Database design

A practical initial SQL Server schema can use the following tables.

Notifications

One logical notification request

• NotificationId — primary key
• UserId — recipient
• EventType — e.g. QuestionReply
• PayloadJson — event-specific data
• Priority — delivery priority
• Status — pending, processing, completed, failed
• CreatedAtUtc, ScheduledAtUtc

NotificationDeliveries

One record per channel delivery

• DeliveryId — primary key
• NotificationId — parent request
• Channel — Email, Push, WhatsApp, etc.
• Recipient — destination reference
• Status — pending, sent, failed
• ProviderMessageId — provider reference
• AttemptCount, NextAttemptAtUtc

NotificationAttempts

Detailed delivery history

• AttemptId — primary key
• DeliveryId — delivery reference
• AttemptedAtUtc — attempt timestamp
• Outcome — success, timeout, rejected, etc.
• ErrorCode, ErrorMessage

UserNotificationPreferences

Recipient-level configuration

• UserId
• NotificationType
• Channel
• IsEnabled

Two additional tables are valuable for more demanding workloads:

• OutboxMessages: stores events in the same database transaction as the business change.
• NotificationTemplates: stores versioned templates, locales, and channel-specific content.

Use foreign keys, appropriate indexes, and a retention policy. Avoid storing sensitive message content or provider credentials unnecessarily.

6. Reliable retries and failure handling

External providers can fail for many reasons: network timeouts, rate limits, expired credentials, invalid recipients, or temporary outages.

A reliable system classifies errors instead of retrying everything indiscriminately.

Failure	Recommended action
Network timeout	Retry with exponential backoff and jitter
HTTP 429 / rate limit	Respect Retry-After where provided
HTTP 5xx	Retry with bounded backoff
Invalid recipient	Mark as permanently failed
Invalid credentials	Alert operations and pause affected delivery
Provider rejects message permanently	Record the reason; do not blindly retry

A typical backoff policy might use delays of 5 seconds, 30 seconds, 2 minutes, and 10 minutes, subject to provider limits and the message's delivery deadline.

After the retry limit is reached, move the delivery into a terminal failure state or dead-letter queue for investigation.

Important: A successful HTTP/API response does not necessarily mean the message reached the recipient. For channels that support delivery-status webhooks, process those callbacks and distinguish accepted, sent, delivered, read, and failed statuses.

7. Preventing duplicate notifications

Duplicate delivery is one of the most important production concerns.

Suppose a worker sends a WhatsApp message successfully but crashes before saving the success status. The job may run again and send the message twice.

Use these controls:

• Assign each logical event a stable idempotency key.
• Enforce uniqueness at the database level for the relevant event and channel.
• Use provider-supported idempotency keys when available.
• Make job processing safe to repeat.
• Use atomic state transitions or leases so multiple workers do not intentionally process the same delivery concurrently.

A database uniqueness constraint helps prevent duplicate records, but it does not guarantee exactly-once delivery. If the provider cannot deduplicate requests, a timeout after sending can leave the outcome uncertain. Design for at-least-once processing and explicitly handle that ambiguity.

8. Hangfire versus a message broker

Both are useful, but they solve different operational needs.

Capability	Hangfire + SQL Server	Message broker + consumers
Scheduled jobs	Excellent	Depends on broker and scheduler
Delayed retries	Built in	Depends on configuration
Operational dashboard	Built in	Usually separate tooling
High-throughput distribution	Suitable for moderate workloads	Often preferable at scale
Multiple consumer services	Possible	Natural fit
Event buffering and backpressure	More limited	Strong fit
Operational complexity	Lower initially	Higher initially

For a MindStick implementation, I would begin with Hangfire and SQL Server if the expected traffic is moderate and you want simple deployment and administration.

Move toward a durable broker such as RabbitMQ or Azure Service Bus when you need independent consumers, higher throughput, workload isolation, or better buffering during provider outages. You can also use Hangfire for scheduling and a broker for delivery without forcing every task into the same mechanism.

9. Production checklist

Readiness checklist

1. Durability: Persist requests before dispatch; use a transactional outbox where needed.
2. Idempotency: Handle duplicate jobs and uncertain provider outcomes.
3. Retry policy: Classify errors, bound retries, and use backoff.
4. Rate limiting: Apply per-provider and per-recipient limits.
5. Security: Keep credentials in a secret store; protect personal data and webhook endpoints.
6. Observability: Track queue depth, delivery latency, retry counts, and permanent failures.
7. User preferences: Respect opt-outs, consent, quiet hours, and channel eligibility.
8. Graceful shutdown: Allow workers to finish or safely release in-progress work.
9. Testing: Test provider timeouts, outages, duplicate jobs, and database failures.

10. Recommended implementation roadmap

1. Foundation: Create the notification tables, application service, provider interface, and one email channel.
2. Background processing: Add Hangfire, retry policies, scheduling, and a delivery-status dashboard.
3. Reliability: Introduce the outbox pattern, idempotency, error classification, and concurrency controls.
4. Multiple channels: Add push, WhatsApp, SMS, and Telegram adapters as needed.
5. Operations: Add rate limiting, provider webhooks, metrics, alerts, and retention policies.
6. Scale: Introduce a message broker if measured workload or reliability requirements justify it.

For implementation references, Microsoft's documentation on hosted services in ASP.NET Core and background jobs in Hangfire are good starting points.

My recommendation: Keep the notification orchestration, delivery tracking, and channel providers separate. Use SQL Server as the durable record, Hangfire for initial background processing, and a transactional outbox for critical business events. This gives you a maintainable foundation that can grow without requiring a complete redesign.

Comments

[Abhinav Sharma]

The crash-after-sending example makes the duplicate problem clear. A useful test case to keep alongside the happy path, rather than treating every failed job as safe to retry.

[Ravi Vishwakarma]

Thanks! Exactly, that's why I'll consider "crash after send, before save state" as its own test case, not just a retry edge case. Simulating a worker kill immediately after the provider call and ensuring that only one message is sent is one of the most valuable tests for this type of service.

[Odinaka Robert Nnamani]

persist before you send is the whole trick. crash-after-send is where duplicates are born.