SMS Delivery Pipeline Project using Go #1

My first post on DEV! As I began this project, I wanted to find a place where I could gather feedback and connect with others, which brought me to this platform. I’d be happy if this project helps someone, and I look forward to all the learning opportunities that come from interacting with you all.

I previously worked on developing an SMS Delivery Platform, a project filled with common challenges in backend and distributed systems, including:

• High throughput
• Low latency
• Fault tolerance
• High concurrency
• Tenant fairness
• Strict rate limiting
• Just-in-time processing and more

Initially, I approached this project with questions such as, "𝙃𝙤𝙬 𝙛𝙖𝙨𝙩 𝙘𝙤𝙪𝙡𝙙 𝙄 𝙙𝙚𝙫𝙚𝙡𝙤𝙥 𝙩𝙝𝙞𝙨 𝙪𝙨𝙞𝙣𝙜 𝙘𝙪𝙧𝙧𝙚𝙣𝙩 𝘼𝙄 𝘾𝙤𝙙𝙞𝙣𝙜 𝘼𝙜𝙚𝙣𝙩𝙨?" and "𝘼𝙩 𝙬𝙝𝙖𝙩 𝙨𝙘𝙖𝙡𝙚 𝙬𝙤𝙪𝙡𝙙 '𝙫𝙞𝙖𝙗𝙡𝙚 𝙘𝙤𝙙𝙞𝙣𝙜'—𝙧𝙚𝙡𝙮𝙞𝙣𝙜 𝙚𝙣𝙩𝙞𝙧𝙚𝙡𝙮 𝙤𝙣 𝙖𝙜𝙚𝙣𝙩𝙨—𝙗𝙚𝙘𝙤𝙢𝙚 𝙞𝙢𝙥𝙤𝙨𝙨𝙞𝙗𝙡𝙚?" However, I soon realized that these questions were less meaningful, as the answers depend heavily on the specifications and how much of the business requirements are reproduced.

As I continued developing, I began to reflect on my earlier decisions: "𝙒𝙝𝙮 𝙙𝙞𝙙 𝙄 𝙘𝙝𝙤𝙤𝙨𝙚 𝙩𝙝𝙞𝙨 𝙨𝙥𝙚𝙘𝙞𝙛𝙞𝙘 𝙙𝙚𝙨𝙞𝙜𝙣 𝙗𝙖𝙘𝙠 𝙩𝙝𝙚𝙣? 𝙒𝙝𝙖𝙩 𝙬𝙚𝙧𝙚 𝙩𝙝𝙚 𝙩𝙧𝙖𝙙𝙚-𝙤𝙛𝙛𝙨?" and "𝙒𝙝𝙖𝙩 𝙬𝙤𝙪𝙡𝙙 𝙄 𝙙𝙤 𝙙𝙞𝙛𝙛𝙚𝙧𝙚𝙣𝙩𝙡𝙮 𝙞𝙛 𝙄 𝙘𝙤𝙪𝙡𝙙 𝙜𝙤 𝙗𝙖𝙘𝙠 𝙩𝙤 𝙩𝙝𝙖𝙩 𝙩𝙞𝙢𝙚?" Through this process, I discovered many "better ways" to tackle these challenges today.

The real interest lies in writing the code, considering the design and trade-offs, and addressing challenges step-by-step. To start, I created a minimal and naive Go implementation. This version does not utilize goroutines; it’s a monolith running as a single process that meets only the bare minimum functional requirements. While it cannot satisfy the non-functional requirements of an early-stage product, I believe it provides a clear understanding of the system's functionality.

Moving forward, I plan to evolve this code by setting specific challenges for each iteration and sharing my reflections on the design decisions and trade-offs involved.

// Start begins the background polling loop.
func (w *Worker) Start(ctx context.Context) {
    // poll every 5 seconds check for pending campaigns
    // TODO: what if the worker crashes? what if we have many campaigns?
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            log.Println("Worker context cancelled, stopping...")
            return
        case <-ticker.C:
            w.processNextCampaign(ctx)
        }
    }
}

func (w *Worker) processNextCampaign(ctx context.Context) {
    // Find the next available campaign
    // TOOD: what if company A has 1000 campaigns and company B has 1 campaign? Company A will monopolize the worker.
    campaign, err := w.db.FindPendingCampaign(ctx)
    if err != nil {
        log.Printf("Error checking for pending campaigns: %v\n", err)
        return
    }
    if campaign == nil {
        // No pending campaign found
        return
    }

    log.Printf("Found pending campaign %s, starting processing...\n", campaign.ID)

    // Fetch the CSV file using the storage port
    csvData, err := w.storage.FetchCSV(ctx, campaign.DestinationsFilePath)
    if err != nil {
        // TODO: what if temporary storage is down? or what if the csv file was accidentally deleted?
        log.Printf("Failed to fetch CSV for campaign %s: %v\n", campaign.ID, err)
        return
    }

    // Parse the CSV
    reader := csv.NewReader(bytes.NewReader(csvData))
    // TODO: what if the csv file is huge?
    records, err := reader.ReadAll()
    if err != nil {
        log.Printf("Failed to parse CSV for campaign %s: %v\n", campaign.ID, err)
        return
    }

    // Loop through records, create SMS messages, and send to the Carrier API
    targetMessage := campaign.TemplateBody
    for i, record := range records {
        if len(record) == 0 {
            continue
        }

        // Assuming the first column is the phone number
        phoneNumber := record[0]
        if phoneNumber == "phone_number" && i == 0 {
            // Skip header row if it exists
            continue
        }

        // to make this code readable, I'm not going to replace variables in the template body for now.

        // TODO: yes, CarrerAPI has rate limits. We need to respect that.
        err = w.carrier.SendSMS(ctx, phoneNumber, targetMessage)

        dispatch := &domain.SmsDispatch{
            SMSMessageID: uuid.New(),
            CampaignID:   campaign.ID,
            TenantID:     campaign.TenantID,
            PhoneNumber:  phoneNumber,
            DispatchedAt: time.Now(),
            IsSuccessful: err == nil,
        }

        if err != nil {
            log.Printf("Failed to send SMS to %s: %v\n", phoneNumber, err)
        } else {
            log.Printf("Successfully sent SMS to %s\n", phoneNumber)
        }

        // record the result
        // TODO: what if this worker crashes after sending some SMS of a campaign, but still have pending SMS to send?
        if saveErr := w.db.SaveSmsDispatch(ctx, dispatch); saveErr != nil {
            log.Printf("Failed to save SMS dispatch for %s: %v\n", phoneNumber, saveErr)
        }

        if err != nil {
            continue
        }
    }

    // Mark the campaign as completed in the DB
    err = w.db.MarkCampaignCompleted(ctx, campaign.ID)
    if err != nil {
        log.Printf("Failed to mark campaign %s as completed: %v\n", campaign.ID, err)
    } else {
        log.Printf("Campaign %s processing completed.\n", campaign.ID)
    }
}

pepnova-9 / message-dispatcher

📖 Overview

This repository demonstrates the evolutionary system design of an event-driven SMS delivery pipeline.

This project is inspired by a real-world SMS delivery system I previously worked on. I chose this specific domain because it serves as an excellent crucible for tackling the core challenges of modern backend and distributed systems. It naturally demands robust solutions for

• high throughput
• low latency
• fault tolerance
• high concurrency
• tenant fairness
• strict rate limiting
• just-in-time processing
• and more

Instead of starting with a complex distributed architecture, this project begins as a minimal, Go monolith. It progressively evolves into a decoupled system to address specific backend challenges as the hypothetical scale grows.

🎯 Core Service Specifications

This platform enables client businesses to reliably manage and deliver SMS communications for various use cases, such as marketing campaigns and appointment reminders. The actual physical delivery to end-user devices is handled by mobile carriers (e.g., Verizon, T-Mobile…

View on GitHub