DEV Community: Shreyas Yadav

Adding MapReduce to My Go Distributed File System

Shreyas Yadav — Tue, 29 Sep 2026 22:41:12 +0000

I had already built a small HDFS-style distributed file system in Go. (Part I) It has a controller for metadata, storage nodes that hold replicated chunks, and a client. All three talk over TCP with Protocol Buffers. This post is about the next step: running MapReduce jobs on the data stored in it.

Here's the design goal. The data is already split into chunks and spread across storage nodes, so the computation should go to the chunks, not the other way around.

This is a course project, so the repository is private. If you’re interested, please comment below, and I’ll grant you viewer access.

The pieces

MapReduce adds one new binary and extends two existing ones:

Computation manager (new): accepts jobs, schedules map and reduce tasks, tracks progress, and merges the final output. It's a separate process from the controller and asks the controller for chunk placements with the same GetFileRequest the client uses.
Storage nodes (extended): they now also run map tasks on their local chunks, receive shuffle data, and run reduce tasks.
Client (extended): new submitjob and jobstatus commands.

All of this uses the same framing as the file system: a length-prefixed protobuf Wrapper, with new message types like SubmitJobRequest, MapTaskRequest, ShuffleData, ReduceTaskRequest, and GetJobStatusResponse.

Jobs are just executables. A mapper reads lines from stdin and writes key<TAB>value lines to stdout. A reducer reads sorted key<TAB>value lines and writes its results. The repo has three example jobs: word count, a log analyzer that counts log lines per domain, and a second-stage report job that reads that output and produces the top 10 domains. The mapper binary has to exist at the same path on every storage node, because each node runs it with exec.Command(req.PluginPath).

Step 0: chunks that don't split lines

The original file system split files at fixed byte offsets. That works for storing files but breaks text processing: a chunk boundary can land in the middle of a line, and then no mapper sees that line whole.

So the client now checks the file type before chunking. It combines http.DetectContentType on the first 8 KB with the file extension. Text-like types (text/*, JSON, CSV, YAML, and a few others) go through planLineChunks, which only cuts between lines. A single line longer than the chunk size gets a chunk of its own. Everything else is still split at fixed byte offsets.

Each text chunk also records the line number it starts at. That number travels through StoreFileRequest.chunk_start_lines into ChunkPlacement.start_line. When a node runs a mapper, formatMapInput prefixes every line with its line number in the whole file (lineNumber<TAB>line). So a mapper sees the same line numbers it would see if it read the whole file in one pass.

I also sped up uploads while I was changing the client: storeChunksConcurrently now uploads with a pool of 8 workers (storeWorkers) instead of sending one chunk at a time.

Submitting a job

The client sends SubmitJobRequest with the mapper path, the input file, the number of reducers, and the reducer path. The manager fetches the placements for the input file, creates a JobState, assigns an ID, and starts dispatching. A job moves through queued, running, reducing (or merging for map-only jobs), and ends as complete or failed. The client polls with jobstatus.

Reduce tasks are dispatched before map tasks, so each reducer is already waiting when mapper output starts arriving. selectReducerNodes assigns reducers round-robin over the primary node of each chunk.

For each map task, pickNode picks the replica of that chunk with the fewest in-flight map tasks (tracked in nodeActiveTasks). Each map task runs on a node that already has the chunk on local disk, so no input data moves over the network.

The map side: spill, sort, merge

On the storage node, executeMapTask reads the chunk from local disk, verifies its SHA-256 checksum, formats the input, and starts the mapper process. Mapper output can be much larger than memory, so it isn't collected in a slice. spillMapperOutput reads stdout line by line, buffers up to mapSpillMaxLines (50,000) lines, sorts them, and writes each batch to a spill_NNNN.run file. Every line must contain a tab, or the task fails.

When the mapper exits, the sorted runs are merged with a k-way merge over a min-heap. The heap orders by key, then by the full line:

for h.Len() > 0 {
    item := heap.Pop(h).(lineHeapItem)
    if err := emit(item.reader.line); err != nil {
        return err
    }
    if err := item.reader.advance(); err != nil {
        if err == io.EOF {
            continue
        }
        return err
    }
    heap.Push(h, item)
}

emit does two things at once. It writes the line to the node's local map output file, and it partitions the pair to a reducer by hashing the key with FNV-1a:

emit := func(line string) error {
    if _, err := writer.WriteString(line); err != nil {
        return err
    }
    if err := writer.WriteByte('\n'); err != nil {
        return err
    }

    if len(senders) == 0 {
        return nil
    }

    key, value, err := parseKVLine(line)
    if err != nil {
        return err
    }
    reducerID := int(fnv32a(key) % uint32(len(senders)))
    return senders[reducerID].append(key, value)
}

Each shuffleSender holds one open TCP connection to its reducer's node. It batches pairs and sends a ShuffleData message every shuffleBatchMaxKVs (2,048) pairs. At the end it sends a final message with done: true. The merge output is sorted, so every batch a reducer receives is already sorted too.

If a job has zero reducers, there's no shuffle. Each map task uploads its output to the DFS as __mr_tmp_<job>_map_chunk_<i>.out, and the manager concatenates those files in chunk order.

The reduce side: waiting for every mapper

This part has a race to handle. A reducer needs to know how many mappers to wait for, and that number comes in ReduceTaskRequest.num_mappers. But shuffle data from a fast mapper can arrive at the node before the ReduceTaskRequest does.

reducerBuffer handles both orders. If shuffle data arrives first, the buffer is created with numMappers = -1 (unknown). Each non-empty batch is written straight to disk as its own sorted run file, and each done message increments a counter:

func (s *StorageNode) acceptShuffleData(sd *pb.ShuffleData) {
    buf := s.getOrCreateReducerBuffer(sd.JobId, sd.ReducerId, -1)

    buf.mu.Lock()
    defer buf.mu.Unlock()

    if len(sd.Pairs) > 0 {
        runPath := s.reducerRunPath(sd.JobId, sd.ReducerId, buf.nextRunID)
        buf.nextRunID++
        if err := s.writeShuffleRun(runPath, sd.Pairs); err != nil {
            if buf.err == nil {
                buf.err = err
            }
        } else {
            buf.runFiles = append(buf.runFiles, runPath)
        }
    }
    if sd.Done {
        buf.doneCount++
        if buf.numMappers >= 0 && buf.doneCount >= buf.numMappers {
            buf.once.Do(func() { close(buf.ready) })
        }
    }
}

When the ReduceTaskRequest arrives, getOrCreateReducerBuffer fills in numMappers and runs the same check, so the ready channel closes whichever message comes last. sync.Once makes sure it only closes once. executeReduceTask blocks on <-buf.ready.

After that, the reducer reuses mergeSortedRuns from the map side and streams the merged lines directly into the reducer process's stdin. The reducer sees every value for a key together, in sorted order, without the node ever loading the whole partition into memory. The reducer's stdout is uploaded to the DFS as __mr_tmp_<job>_reducer_<id>.out. Once every reducer reports success, the manager concatenates the partitions in reducer-ID order into <job>_output.out.

Handling failed and hung map tasks

A map task can fail in two ways, and the manager handles each one differently.

It reports failure. The node sends back MapTaskResponse{success: false}. The manager moves to the next replica in that chunk's placement and dispatches there right away. Once no replicas are left, the job fails.

It never answers. The node accepts the task and then dies or hangs. watchJobTimeout checks every 10 seconds for chunks that have been running longer than the timeout and re-dispatches them to the next replica. The timeout adapts to how fast tasks are actually finishing:

timeout := fallbackTimeout
if job.MapsTotal > 0 && float64(job.MapsDone)/float64(job.MapsTotal) >= threshold && len(job.completionTimes) > 0 {
    var total time.Duration
    for _, d := range job.completionTimes {
        total += d
    }
    avg := total / time.Duration(len(job.completionTimes))
    adaptiveTimeout := time.Duration(adaptiveMultiplier) * avg
    if adaptiveTimeout < minAdaptiveTimeout {
        adaptiveTimeout = minAdaptiveTimeout
    }
    if adaptiveTimeout < timeout {
        timeout = adaptiveTimeout
    }
}

Until 70% of map tasks have finished, the timeout is the fallback of 15 minutes. After that, it's twice the average completion time, but never less than 3 minutes and never more than 15. Early in a job there's no data about how long a task normally takes, so a fixed 3-minute timeout would kill slow but healthy tasks. With enough completions to estimate from, a stuck task gets retried much sooner than the 15-minute fallback. The most recent change to this code made the adaptation more conservative.

If the original task and its retry both finish, completedChunks counts the chunk only once.

What I'd change next

Some limitations I know about:

Shuffle data isn't tied to a map attempt. ShuffleData carries job_id and reducer_id, but no chunk index or attempt number. If a map task sends part of its shuffle and then fails, or if a timed-out task and its retry both finish, the reducer can receive duplicate pairs, and the extra done messages can make it start before the real last mapper finishes. Tagging each stream with its chunk index, and counting done messages per chunk, would fix this.
Retries can go back to the same node. Retries walk the placement list starting at index 1, whichever replica pickNode actually chose. If pickNode picked Nodes[1] and that task fails, the first retry goes to the same node again.
Reducers aren't retried. One failed reduce task fails the whole job.
The final merge happens in memory. The manager downloads every partition into one byte slice and uploads the result as a single chunk with replication factor 1. That limits output size to the manager's memory, and the output has no replicas.
Nothing gets cleaned up. Spill files, shuffle runs, __mr_tmp_ files in the DFS, and the per-job reducer buffers all stay around after a job finishes.
Job state lives in the manager's memory only. If the manager restarts, every job in flight is lost.

The ideas I learned the most from were external sorting with spill files and a heap merge, and handling shuffle data that shows up before the reduce task does. Scheduling map tasks on the nodes that already hold the chunk was the easy part, because the file system already tracked where every chunk lives.

Building an HDFS-Style Distributed File System in Go

Shreyas Yadav — Tue, 29 Sep 2026 22:23:51 +0000

I built a small distributed file system in Go, modeled loosely on HDFS. It has three binaries:

a controller that owns metadata: which nodes are alive and which nodes hold which chunks
storage nodes that keep chunk bytes on local disk
a client with store, get, getmeta, delete, ls, and nodes commands

Everything talks over plain TCP using Protocol Buffers. There is no gRPC and no external coordination service. This post walks through how the pieces fit together and the failure cases I had to handle.

The source code is on GitHub: Shreyas-Yadav/hdfs-lite.

The wire protocol: one envelope, length-prefixed

Every message type is a field in a single oneof called Wrapper, so each connection carries one kind of thing: a Wrapper. That covers RegisterNode, StoreChunkRequest, Heartbeat, ReplicateChunkRequest, CommitFileRequest, and the rest.

TCP is a byte stream, so each message needs a boundary. MessageHandler writes a 4-byte big-endian length followed by the protobuf body. On the read side, ReadMsg reads 4 bytes, then reads exactly that many bytes with io.ReadFull, then unmarshals. Callers just call Send(m), which type-switches on the message and wraps it. On the receive side they use typed helpers like ReceiveStoreChunkResponse(), built on a small generic receiveTyped[T], which return an error if a different message type shows up.

Each server accepts connections in a loop and starts one goroutine per connection. That goroutine loops over ReadMsg and dispatches on msg.Msg.(type).

Joining the cluster and staying alive

On startup, a storage node binds its listening port first and registers with the controller only after that, so it never announces an address it can't serve. The node ID is simply host:port. After registering, the node keeps the TCP connection to the controller open and sends a Heartbeat over it every 5 seconds.

Each heartbeat carries the node's free space (from syscall.Statfs), a count of requests handled, and a new_chunks list of chunks stored since the last beat. If a heartbeat fails, reconnectLoop tries to re-register. It waits 5 seconds first and doubles the wait after each failure. It stops doubling once the wait reaches the 60-second maxBackoff, so the longest wait in practice is 80 seconds.

On the controller, monitorNodes runs every monitorInterval (5s). Any node that hasn't sent a heartbeat within nodeFailureTimeout (15s, about three missed beats) is marked dead, and that triggers a replication repair pass. If a dead node later sends another heartbeat, the controller clears the dead flag and starts another repair pass.

Placing chunks

The client splits a file into fixed-size chunks and sends a StoreFileRequest. The replication factor defaults to 3. The controller rejects the request if the file already exists or if fewer live nodes are available than the replication factor requires.

selectNodes handles placement with an offset round-robin over the live nodes, sorted by ID. The node list for chunk i starts at index i, so the primary role rotates across nodes and the write load is spread. The first node in each placement is the primary and the rest are replicas. This scheme ignores free space. Free space only matters later, when the controller picks a node to repair onto.

Writing: pipeline replication

The client doesn't send each chunk three times. It sends each chunk once, to the primary, with the rest of the chain in Replicas and a SHA-256 checksum of the data. The primary:

checks that the checksum matches the received bytes
writes the chunk plus a .checksum sidecar file to disk
forwards the chunk to Replicas[0] and passes along only Replicas[1:]

Each hop waits for the next hop's ack before it acks upstream. So when the client gets ok from the primary, every node in the chain has already written the chunk. If a forward fails, the node deletes its own copy before reporting failure:

if len(req.Replicas) > 0 {
    next := req.Replicas[0]
    if err := forwardChunk(next.Address, req); err != nil {
        if rbErr := node.deleteChunk(req.Filename, req.ChunkIndex); rbErr != nil {
            log.Error("rollback failed for chunk %d of %s: %v", req.ChunkIndex, req.Filename, rbErr)
        }
        _ = handler.Send(&pb.StoreChunkResponse{
            Ok:      false,
            Message: fmt.Sprintf("pipeline forward to %s failed: %v", next.Address, err),
        })
        continue
    }
}

Committing: no half-written files in the namespace

When the controller receives a StoreFileRequest, it puts the placement plan in pendingFiles, not files. The client uploads every chunk and then sends CommitFileRequest on the same controller connection. Only then does commitFile move the record into files and write state to disk.

If the client dies partway through, its controller connection closes. A deferred block in the connection handler sees a pending filename that never got committed and calls abortPending. That removes the record and asks every planned node to delete its chunk. On storage nodes, deleting a chunk that doesn't exist counts as success, so cleaning up a partial upload is safe.

Only committed files are persisted, as JSON in controller_state.json. Node membership is not persisted. The controller rebuilds it as nodes register and send heartbeats.

Reading: parallel fetch, positional writes

getFile gets the placements from the controller, creates the output file, and truncates it to its full size up front. It then fetches chunks in parallel. A buffered channel caps the fetches at 8 goroutines at a time:

sem := make(chan struct{}, 8)
var wg sync.WaitGroup
var firstErr error
var errMu sync.Mutex

for _, placement := range placements {
    wg.Add(1)
    sem <- struct{}{}

    go func(p *pb.ChunkPlacement) {
        defer wg.Done()
        defer func() { <-sem }()

        data, err := fetchChunkWithFallback(filename, p)

Each goroutine writes its chunk with WriteAt at offset ChunkIndex * ChunkSize. Chunks can finish in any order, and the file is never assembled in memory. fetchChunkWithFallback tries each replica in placement order and checks the SHA-256 of what it receives. If the data was corrupted in transit, or any other error occurs, it moves on to the next replica.

Self-healing reads on the storage node

Checksums are also checked on disk. When a storage node serves a GetChunkRequest, it compares the chunk against its .checksum sidecar. If the chunk is missing or the checksum doesn't match, the node doesn't just return an error. It calls repairChunkFromReplicas, which fetches a good copy from another replica, stores that copy locally, and then serves it.

Replica A might fetch from B while B's copy is also bad, and B would then try to repair from A. GetChunkRequest.avoid_node_ids prevents that loop. Each node adds itself to the list before asking the next one:

avoidSet[s.nodeID] = true
nextAvoidNodeIDs := append(append([]string(nil), avoidNodeIDs...), s.nodeID)

for _, placement := range placements {
    if placement.ChunkIndex != chunkIndex {
        continue
    }
    for _, replica := range placement.Nodes {
        if replica == nil || replica.Address == "" || avoidSet[replica.NodeId] || replica.Address == s.address {
            continue
        }

        data, err := fetchReplicaChunk(filename, chunkIndex, replica.Address, nextAvoidNodeIDs)
        if err != nil {
            continue
        }

As a result, a client read also repairs a corrupted local copy.

Re-replication after a node dies

reconcileUnderReplicated brings chunks back up to their target replica count. It works in three phases so the controller mutex is never held during network calls:

Step A (under lock): Find every chunk with fewer live replicas than its target and queue the repair work: a healthy source, a replacement node, and the dead node to replace.
Step B (no lock): For each work item, send a ReplicateChunkRequest to the source storage node. The source verifies its own checksum and pushes the chunk straight to the destination. Chunk bytes never pass through the controller.
Step C (under lock): Swap the replacement into the placement and save state once at the end.

pickReplacementNode picks the live node with the most free space that doesn't already hold the chunk. If no spare node exists, the chunk stays marked under-replicated and a later pass retries it.

Why the 3-minute grace period

This one came from a real failure mode. When the controller restarts, it reloads every file's placements, but it doesn't know about any nodes yet. If it ran a repair pass right away, every chunk would look under-replicated, and the controller would start copying data around the cluster for no reason.

So the first pass waits for startupGracePeriod:

go controller.monitorNodes()
go func() {
    time.Sleep(startupGracePeriod)
    controller.reconcileUnderReplicated("controller startup")
}()

startupGracePeriod is 3 minutes. The repair triggers on node registration and on node recovery are also skipped during this window. That gives storage nodes, which retry on their own backoff, time to reconnect before the controller makes any replication decisions.

Deletes

The client sends DeleteFileRequest to the controller. The controller asks every replica to delete its chunk, then drops the metadata. If some replicas can't be reached, the delete still succeeds from the client's point of view, and the response message says how many replica deletes couldn't be confirmed.

What I'd change next

Some limitations I know about:

The controller is a single point of failure. Metadata goes to one local JSON file, and saveState writes it with os.WriteFile directly, not a write-to-temp-then-rename. A crash during the write could corrupt the state file.
Uploads are sequential per chunk. Reads run 8 at a time, but storeFile reads and pushes chunks one after another.
new_chunks is reported but barely used. Storage nodes send it in every heartbeat, but the controller only logs it. The controller never compares its placement map against what nodes actually have on disk.
Initial placement ignores capacity. Round-robin spreads load evenly, but a nearly full node gets as many new chunks as an empty one.
The liveness window is duplicated. Two functions hard-code 15*time.Second instead of using nodeFailureTimeout.

The ideas I learned the most from were pipeline replication with rollback, commit-after-upload, and checksum-verified reads that repair the bad copy. None of them took much code, but each one closed a failure case that I only understood after it had happened.

Implementing Strict Priority and Deficit Round Robin Schedulers in NS-3

Shreyas Yadav — Tue, 29 Sep 2026 21:10:11 +0000

Stack: NS-3 · C++ · nlohmann/json · Wireshark

When a router's outgoing link is slower than the traffic arriving at it, packets wait in a queue, and the router has to decide which packet goes next. That decision is what Quality of Service (QoS) is about. I implemented two classic answers to that question in NS-3: Strict Priority Queueing (SPQ) and Deficit Round Robin (DRR). Both plug into a small Differentiated Services (DiffServ) framework that I wrote, and both are configured from JSON.

This post covers the design, one classification bug that is easy to hit in NS-3, the results in Wireshark, and what I would do differently.

Code: github.com/Shreyas-Yadav/QoS-NS3

The Setup: Creating a Bottleneck

The topology is three nodes in a line:

Client ──(4 Mbps, 10 ms)──> Router ──(1 Mbps, 10 ms)──> Server

Each UDP client sends a 1000-byte packet every 2 ms, which is about 4 Mbps per flow. The router can only forward 1 Mbps, so its egress queue fills up quickly. That queue is where the scheduler runs. I replaced the router's default queue on the server-facing device with my own:

Ptr<PointToPointNetDevice> routerEgress =
    m_routerNode->GetDevice(1)->GetObject<PointToPointNetDevice>();
routerEgress->SetQueue(spq);  // or drr

I capture PCAP traces on both sides of the router (Pre_* and Post_*) so I can compare the traffic before and after scheduling.

Design

How a packet moves through classification, the SPQ or DRR scheduler, and out of the router.

Classification: Filters and Traffic Classes

Every packet must be placed in a traffic class before a scheduler can pick from it. The classification logic has three layers:

FilterElement checks a single field: source or destination IP, subnet mask, source or destination port, or protocol.
Filter combines elements with AND logic, so all of them must match.
TrafficClass holds a list of filters combined with OR logic, plus its own queue, packet limit, and priority or weight.

If no class matches a packet, it goes to the class marked Default. If there is no default class, the packet is dropped.

The Scheduler Interface

DiffServ is the base class. It handles enqueue (classify, then push to the matching class) and dequeue. Subclasses only implement one method:

virtual Ptr<const Packet> Schedule() const = 0;

Schedule() returns the packet that should go next, and the base class removes it from the correct queue. SPQ and DRR differ only in how they implement this one method.

JSON Configuration

The scheduler type and the traffic classes come from a config file:

{
  "name": "drr",
  "queues": [
    { "no": 1, "MaxPackets": 3000, "Weight": 10, "DestPort": 9000 },
    { "no": 2, "MaxPackets": 3000, "Weight": 20, "DestPort": 10000 },
    { "no": 3, "MaxPackets": 3000, "Weight": 30, "DestPort": 11000 }
  ]
}

A Gotcha: The PPP Header

My first version of the port filter matched nothing. At the point where a packet sits in a point-to-point device's queue, NS-3 has already added a PPP header in front of the IPv4 header. You have to remove that header before you can read anything else:

Ptr<Packet> copy = pkt->Copy();   // never modify the real packet

PppHeader ppp;
if (!copy->RemoveHeader(ppp)) return false;

Ipv4Header ip;
if (!copy->RemoveHeader(ip)) return false;

if (ip.GetProtocol() == 17) {     // UDP
    UdpHeader udp;
    copy->PeekHeader(udp);
    return udp.GetDestinationPort() == m_port;
}

Working on a copy matters too. RemoveHeader changes the packet, and the real packet still has to leave the router with all its headers.

Strict Priority Queueing

SPQ is simple: always serve the highest-priority queue that has packets. A lower priority number means a higher priority.

for (auto tc : classes) {
    if (!tc->IsEmpty() && tc->GetPriority() < highestPriority) {
        best = tc;
        highestPriority = tc->GetPriority();
    }
}
return best ? best->Peek() : nullptr;

Test scenario: the low-priority flow (port 10001) starts at 0 s. The high-priority flow (port 10000) starts at 15 s.

Before the router (4 Mbps link). Red: low priority (port 10001). Green: high priority (port 10000). The two flows share the link equally.

After the router (1 Mbps bottleneck). While the high-priority queue has packets, the low-priority flow gets nothing.

After the router, the low-priority flow uses the full link (about 120 packets/s) until 15 s. When the high-priority flow starts, the low-priority flow drops to zero. It gets no bandwidth until the high-priority queue is empty, and then it recovers.

This is exactly what SPQ promises, and it also shows SPQ's weakness: starvation. A heavy high-priority flow can block everything else indefinitely.

Deficit Round Robin

DRR shares bandwidth in proportion to weights instead. Each queue has a credit counter. On each visit, the queue earns credit equal to its weight. If its credit covers the size of the packet at the front, it sends that packet and pays for it. Otherwise, the scheduler moves on to the next queue.

while (true) {
    if (!queues[i]->IsEmpty()) {
        credit[i] += queues[i]->GetWeight();
        uint32_t size = queues[i]->Peek()->GetSize();
        if (size <= credit[i]) {
            credit[i] -= size;
            return queues[i]->Peek();
        }
    }
    i = (i + 1) % queues.size();
}

Peeking Must Not Spend Credit

NS-3 calls Peek() on a queue separately from Dequeue(). If Schedule() updated the real credit counters, a single peek would spend credit for a packet that was never sent.

To avoid this, Schedule() works on temporary state (m_tempCreditBalance, m_scheduledQueueIndex). Dequeue() copies that state into the real counters only after a packet actually leaves:

Ptr<Packet> DRR::Dequeue() {
    Ptr<Packet> p = DiffServ::Dequeue();
    if (p) {
        m_currentQueueIndex = m_scheduledQueueIndex;
        m_creditBalance = m_tempCreditBalance;
    }
    return p;
}

Test scenario: three flows with weights 10, 20, and 30, all starting at 0 s.

Before the router. All three flows arrive at the same rate.

After the router. Blue: weight 10. Teal: weight 20. Yellow: weight 30.

After the router, the flows settle at about 20, 40, and 60 packets/s, which is the 1:2:3 ratio of the weights.

The graph also shows DRR's other property. When the weight-30 flow's queue empties at about 22 s, its share is redistributed to the remaining flows. They move to about 40 and 80 packets/s, which keeps their 1:2 ratio. When the weight-20 queue empties at about 27 s, the last flow gets the full link. DRR never leaves the link idle while there are packets waiting.

What I Would Change

Looking back at the code, these are the things I would fix:

The quantum is smaller than a packet. The weights (10–30) are added as bytes of credit, but packets are about 1000 bytes. The DRR loop therefore goes around dozens of times before any queue can send. The ratio is still correct, but the loop wastes work. In textbook DRR, the quantum is at least one MTU, so a queue can send on every visit.
Credit is not reset when a queue empties. In standard DRR, an empty queue loses its leftover credit, so it can't save up credit and send a burst later. My implementation keeps the credit.
Queue limits count packets instead of bytes. A queue full of small packets and a queue full of large packets hold very different amounts of data.
Classification assumes IPv4 over PPP. Any other link layer or IPv6 would not match.
The config file has no schema validation. A typo in a field name silently falls back to a default value.
DiffServ does too much. Classification, scheduling, and queue storage all live in one class. Splitting them into a Classifier, a Scheduler, and a Queue Manager would make each part easier to test and replace.

Run It Yourself

Copy the source files into scratch/final-project/ in an NS-3 (3.36 or later) installation, install nlohmann-json3-dev, and run:

./ns3 run scratch/final-project/driver.cc -- scratch/final-project/spq_config.json
./ns3 run scratch/final-project/driver.cc -- scratch/final-project/drr_config.json

Then open the PCAP files in Wireshark, go to Statistics → I/O Graph, and add one line per flow with a filter such as udp.dstport == 10000.

The full code is on GitHub.

I Built a Nightly Pipeline That Deploys My App While I Sleep

Shreyas Yadav — Mon, 09 Mar 2026 04:07:07 +0000

Stack: Next.js 14 · Express.js · MySQL 8 · Docker · GitHub Actions · AWS EC2 · AWS ECR · AWS RDS · Route 53 · Nginx · Let's Encrypt

I built a Job Application Tracker, a full-stack SPA with GitHub OAuth login, and set up an automated nightly pipeline that builds Docker images, runs smoke tests on a temporary EC2, pushes verified images to ECR, and deploys to a persistent QA server, all without manual intervention. Here's exactly how I did it.

Architecture Overview
The Application: Job Application Tracker
Dockerizing the App
Local Development with Docker Compose
AWS Infrastructure Setup
GitHub Actions CI/CD Pipeline
Domain Name with Route 53 and SSL with Let's Encrypt
Nginx as a Reverse Proxy
Security Best Practices
Lessons Learned

1. Architecture Overview

The project is split into two repositories, a separation of concerns that keeps application code and infrastructure code independent:

Repo	Purpose
`job-application-tracker`	Application source code (frontend, backend, Dockerfiles, local docker-compose)
`job-application-tracker-infra`	Infrastructure: GitHub Actions workflows, Nginx config, smoke tests, prod docker-compose

High-Level Architecture

Developer pushes to source repo
         │
         ▼
  GitHub Actions (infra repo)
  Nightly at 2 AM UTC
         │
  ┌──────▼──────┐
  │  1. BUILD   │  Build Docker images with timestamp tag
  │             │  Push to AWS ECR
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │  2. SMOKE   │  Launch temporary EC2 (t3.micro)
  │    TEST     │  Run containers, execute curl tests
  │             │  Terminate EC2 (pass or fail)
  └──────┬──────┘
         │ (only if tests pass)
  ┌──────▼──────┐
  │  3. PROMOTE │  Retag timestamp → :latest in ECR
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │  4. DEPLOY  │  SSH-less deploy via AWS SSM
  │    TO QA    │  Pull :latest from ECR
  │             │  docker compose up on persistent EC2
  └─────────────┘
         │
  ┌──────▼──────┐
  │  QA EC2     │  Nginx (SSL/HTTPS)
  │  shri.      │  Frontend :3000 → /
  │  software   │  Backend  :5000 → /api/
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │  AWS RDS    │  MySQL 8 (persistent, managed)
  └─────────────┘

2. The Application: Job Application Tracker

The finished app running at shri.software with HTTPS enforced

The app lets users track their job applications (Applied → Interview → Offer / Rejected). Authentication is handled via GitHub OAuth using NextAuth.js, no username/password to manage.

Tech Stack

Frontend: Next.js 14.2 with React 18, Tailwind CSS, NextAuth.js
Backend: Express.js 4.18, JWT middleware
Database: MySQL 8 (local via Docker, production via AWS RDS)

Database Schema

-- Users created on first GitHub login
CREATE TABLE users (
  id                  INT AUTO_INCREMENT PRIMARY KEY,
  email               VARCHAR(255) UNIQUE NOT NULL,
  name                VARCHAR(255),
  avatar_url          TEXT,
  provider            VARCHAR(50),
  provider_account_id VARCHAR(255),
  UNIQUE KEY (provider, provider_account_id),
  created_at          TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at          TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

-- Each row belongs to one user (CASCADE delete keeps DB clean)
CREATE TABLE job_applications (
  id           INT AUTO_INCREMENT PRIMARY KEY,
  company      VARCHAR(255) NOT NULL,
  role         VARCHAR(255) NOT NULL,
  status       ENUM('applied','interview','offer','rejected') DEFAULT 'applied',
  date_applied DATE,
  user_id      INT NOT NULL,
  FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
  created_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

Dashboard view, filter by status, edit or delete entries inline

Backend API

Method	Route	Auth	Description
GET	`/health`	None	Health check for monitoring
POST	`/api/users/upsert`	None	Called by NextAuth on login
GET	`/api/applications`	JWT	List user's applications
POST	`/api/applications`	JWT	Create application
PUT	`/api/applications/:id`	JWT	Update application
DELETE	`/api/applications/:id`	JWT	Delete application

Authentication Flow

NextAuth.js handles the GitHub OAuth dance. The key insight: after OAuth completes, we generate a JWT signed with NEXTAUTH_SECRET that the frontend sends to the backend on every API call.

// frontend/src/app/api/auth/[...nextauth]/route.js (simplified)
export const authOptions = {
  providers: [GitHubProvider({ clientId, clientSecret })],
  callbacks: {
    async signIn({ user, account }) {
      // Store GitHub identity in our DB on first login
      await fetch(`${process.env.NEXT_PUBLIC_API_URL}/api/users/upsert`, {
        method: 'POST',
        body: JSON.stringify({
          email: user.email,
          name: user.name,
          provider: account.provider,
          provider_account_id: account.providerAccountId,
        }),
      });
      return true;
    },
    async session({ session, token }) {
      // Mint a JWT for API calls; embed it in the session
      session.backendToken = jwt.sign(
        { userId: token.userId, email: token.email },
        process.env.NEXTAUTH_SECRET,
        { expiresIn: '1h' }
      );
      return session;
    },
  },
};

The backend verifies this token on every protected route:

// backend/src/middleware/auth.js
const auth = (req, res, next) => {
  const token = req.headers.authorization?.split(' ')[1];
  if (!token) return res.status(401).json({ error: 'No token' });
  const decoded = jwt.verify(token, process.env.NEXTAUTH_SECRET);
  req.user = decoded; // { userId, email }
  next();
};

3. Dockerizing the App

Backend Dockerfile

FROM node:20-alpine

# Non-root user for security
RUN addgroup -S shri && adduser -S shri -G shri

WORKDIR /app
COPY package.json .
RUN npm install --production
COPY src/ ./src/

RUN chown -R shri:shri /app
USER shri

EXPOSE 5000
CMD ["node", "src/index.js"]

Key decisions:

Alpine base, minimal attack surface, smaller image (~50 MB vs ~900 MB for node:20)
Non-root user, if the container is compromised, the attacker can't escalate to root on the host
--production install, excludes dev dependencies from the final image

Frontend Dockerfile (Multi-Stage Build)

# --- Stage 1: Build ---
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# --- Stage 2: Runtime ---
FROM node:20-alpine
RUN addgroup -S shri && adduser -S shri -G shri
WORKDIR /app

# Copy only the artifacts needed to run
COPY --from=builder /app/.next ./.next
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./

RUN chown -R shri:shri /app
USER shri

EXPOSE 3000
CMD ["node_modules/.bin/next", "start"]

Multi-stage builds are critical for Next.js, the build stage pulls in all dev dependencies and compiles TypeScript/JSX. The final stage only contains the compiled output, shrinking the image dramatically.

4. Local Development with Docker Compose

The app running locally via Docker Compose at localhost:9000

The source repo's docker-compose.yml wires everything together for local development with a single command:

# docker-compose.yml
services:
  db:
    image: mysql:8
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - db_data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      retries: 5

  backend:
    build: ./backend
    ports:
      - "10000:5000"
    environment:
      DB_HOST: db
      DB_NAME: ${MYSQL_DATABASE}
      DB_USER: ${MYSQL_USER}
      DB_PASSWORD: ${MYSQL_PASSWORD}
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
    depends_on:
      db:
        condition: service_healthy  # Wait for MySQL, not just the container

  frontend:
    build: ./frontend
    ports:
      - "9000:3000"
    environment:
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
      NEXTAUTH_URL: http://localhost:9000
      GITHUB_CLIENT_ID: ${GITHUB_CLIENT_ID}
      GITHUB_CLIENT_SECRET: ${GITHUB_CLIENT_SECRET}
      NEXT_PUBLIC_API_URL: http://localhost:10000
    depends_on:
      - backend

volumes:
  db_data:

To get started locally:

# 1. Clone the source repo
git clone https://github.com/Shreyas-Yadav/job-application-tracker
cd job-application-tracker

# 2. Copy and fill in the env file
cp .env.example .env
# Edit .env with your GitHub OAuth credentials and secrets

# 3. Launch everything
docker compose up

# App available at:
# Frontend: http://localhost:9000
# Backend:  http://localhost:10000

All three containers (db, backend, frontend) starting up successfully

The depends_on with condition: service_healthy is important, the backend waits for MySQL to be truly ready (not just the container running) before starting.

5. AWS Infrastructure Setup

5.1 GitHub OAuth App

Go to GitHub → Settings → Developer Settings → OAuth Apps → New OAuth App:

Homepage URL: https://shri.software
Authorization callback URL: https://shri.software/api/auth/callback/github

Save the Client ID and Client Secret, these go into GitHub Actions secrets.

5.2 AWS ECR (Elastic Container Registry)

Create two private repositories to store Docker images:

Both ECR repositories created and ready to receive images from the CI pipeline

aws ecr create-repository --repository-name job-tracker-backend --region us-east-1
aws ecr create-repository --repository-name job-tracker-frontend --region us-east-1

5.3 AWS RDS (MySQL 8)

Create a MySQL 8 RDS instance in the AWS Console:

Engine: MySQL 8.0
Instance class: db.t3.micro (Free Tier eligible)
Storage: 20 GB gp2
Important: Place in the same VPC as your EC2 instances
Set a master username and password
Create a database: jobtracker
Note the endpoint, it looks like job-tracker-db.xxx.us-east-1.rds.amazonaws.com

RDS MySQL 8 instance showing "Available" status with the endpoint used by the backend

Why RDS instead of a containerized DB? Managed backups, automated patching, and persistence across EC2 restarts without dealing with EBS volumes.

5.4 QA EC2 Instance

Launch a persistent EC2 instance (Ubuntu 22.04, t3.micro):

# After SSH-ing in, install Docker and the SSM agent
sudo apt-get update
sudo apt-get install -y docker.io docker-compose-plugin

# Enable SSM agent (usually pre-installed on Ubuntu 22.04 AMIs)
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# Allow ubuntu user to run docker without sudo
sudo usermod -aG docker ubuntu

The persistent QA EC2 instance with LabRole attached, running Docker and Nginx

Attach the LabRole to this EC2 instance. In AWS Academy, LabRole is a pre-provisioned IAM role that already has the permissions needed for SSM, ECR, and EC2 operations, you don't create it yourself, it's provided by the lab environment.

5.5 IAM Credentials for GitHub Actions (AWS Academy)

AWS Academy accounts don't support OIDC federation or long-lived IAM users. Instead, you get temporary session credentials from the AWS Details panel in the Vocareum lab console. These rotate every session.

Store them as GitHub Secrets:

AWS_ACCESS_KEY_ID      → from AWS Details panel
AWS_SECRET_ACCESS_KEY  → from AWS Details panel
AWS_SESSION_TOKEN      → from AWS Details panel (required for temporary credentials)

Then configure credentials in your workflow using the static credential method:

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }}
    aws-region: us-east-1

Note: AWS Academy session credentials expire when the lab session ends (~4 hours). You'll need to update these three secrets each time you start a new lab session. This is a limitation of the Academy environment, in a real AWS account you would use OIDC to avoid static credentials entirely.

6. GitHub Actions CI/CD Pipeline

The infra repo contains five workflow files that chain together via workflow_call. Each does one thing well.

6.1 Nightly Orchestrator (`nightly.yml`)

This is the entry point, triggered on a schedule or manually:

# .github/workflows/nightly.yml
name: Nightly Build and Deploy

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC every day
  workflow_dispatch:        # Allow manual triggers

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      image_tag: ${{ steps.tag.outputs.image_tag }}
    steps:
      - id: tag
        run: echo "image_tag=$(date +%Y%m%d%H%M%S)" >> $GITHUB_OUTPUT

  build:
    needs: setup
    uses: ./.github/workflows/build.yml
    with:
      image_tag: ${{ needs.setup.outputs.image_tag }}
    secrets: inherit

  smoke-test:
    needs: [setup, build]
    uses: ./.github/workflows/smoke-test.yml
    with:
      image_tag: ${{ needs.setup.outputs.image_tag }}
    secrets: inherit

  promote:
    needs: [setup, smoke-test]  # Only runs if smoke test passes
    uses: ./.github/workflows/promote.yml
    with:
      image_tag: ${{ needs.setup.outputs.image_tag }}
    secrets: inherit

  deploy-qa:
    needs: promote
    uses: ./.github/workflows/deploy-qa.yml
    secrets: inherit

The timestamp tag (e.g., 20250307021530) makes every build uniquely identifiable. If a smoke test fails, the image stays tagged only with the timestamp, it's never promoted to :latest and never deployed.

6.2 Build and Push to ECR (`build.yml`)

# .github/workflows/build.yml
name: Build and Push to ECR

on:
  workflow_call:
    inputs:
      image_tag:
        required: true
        type: string

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout source code
        uses: actions/checkout@v4
        with:
          repository: Shreyas-Yadav/job-application-tracker
          token: ${{ secrets.SOURCE_REPO_TOKEN }}

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }}
          aws-region: us-east-1

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push backend
        run: |
          docker build -t ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/job-tracker-backend:${{ inputs.image_tag }} ./backend
          docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/job-tracker-backend:${{ inputs.image_tag }}

      - name: Build and push frontend
        run: |
          docker build -t ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/job-tracker-frontend:${{ inputs.image_tag }} ./frontend
          docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/job-tracker-frontend:${{ inputs.image_tag }}

6.3 Smoke Test on Temporary EC2 (`smoke-test.yml`)

This is the most interesting part. Instead of testing on the QA instance (risking breaking it), we launch a fresh, temporary EC2 for every test run:

# .github/workflows/smoke-test.yml
jobs:
  smoke-test:
    runs-on: ubuntu-latest
    steps:
      - name: Launch temporary EC2
        id: launch
        run: |
          INSTANCE_ID=$(aws ec2 run-instances \
            --image-id ${{ secrets.TEMP_EC2_AMI }} \
            --instance-type t3.micro \
            --subnet-id ${{ secrets.TEMP_EC2_SUBNET_ID }} \
            --security-group-ids ${{ secrets.TEMP_EC2_SG_ID }} \
            --iam-instance-profile Name=LabInstanceProfile \
            --query 'Instances[0].InstanceId' \
            --output text)
          echo "instance_id=$INSTANCE_ID" >> $GITHUB_OUTPUT

      - name: Wait for SSM agent
        run: |
          # Wait up to 4 minutes for the instance to boot and SSM to connect
          for i in {1..24}; do
            STATUS=$(aws ssm describe-instance-information \
              --filters "Key=InstanceIds,Values=${{ steps.launch.outputs.instance_id }}" \
              --query 'InstanceInformationList[0].PingStatus' \
              --output text 2>/dev/null || echo "None")
            [ "$STATUS" = "Online" ] && break
            sleep 10
          done

      - name: Run smoke tests via SSM
        run: |
          COMMAND_ID=$(aws ssm send-command \
            --instance-ids ${{ steps.launch.outputs.instance_id }} \
            --document-name "AWS-RunShellScript" \
            --parameters commands='[
              "aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com",
              "IMAGE_TAG=${{ inputs.image_tag }} docker compose -f /tmp/docker-compose.smoke.yml up -d",
              "bash /tmp/smoke-test.sh"
            ]' \
            --query 'Command.CommandId' --output text)

          # Poll until complete
          aws ssm wait command-executed \
            --command-id $COMMAND_ID \
            --instance-id ${{ steps.launch.outputs.instance_id }}

      - name: Terminate temporary EC2
        if: always()  # Clean up even if tests fail
        run: |
          aws ec2 terminate-instances \
            --instance-ids ${{ steps.launch.outputs.instance_id }}

The smoke-test.sh script runs three checks:

#!/bin/bash
BACKEND=${1:-"http://localhost:5000"}
FRONTEND=${2:-"http://localhost:3000"}

# Wait up to 2 minutes for backend to be ready
for i in {1..12}; do
  curl -sf "$BACKEND/health" > /dev/null && break
  echo "Waiting for backend... ($i/12)"
  sleep 10
done

# Test 1: Backend health check
curl -sf "$BACKEND/health" | grep -q '"status":"ok"' || { echo "FAIL: /health"; exit 1; }
echo "PASS: Backend /health returns ok"

# Test 2: Auth middleware is working (unauthenticated request → 401)
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BACKEND/api/applications")
[ "$STATUS" = "401" ] || { echo "FAIL: /api/applications should return 401, got $STATUS"; exit 1; }
echo "PASS: /api/applications correctly requires authentication"

# Test 3: Frontend is serving pages
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$FRONTEND")
[ "$STATUS" = "200" ] || { echo "FAIL: Frontend returned $STATUS"; exit 1; }
echo "PASS: Frontend is serving pages"

echo "All smoke tests passed!"

6.4 Promote Image (`promote.yml`)

Once smoke tests pass, we retag the timestamp image as :latest:

# .github/workflows/promote.yml
jobs:
  promote:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        repo: [job-tracker-backend, job-tracker-frontend]
    steps:
      - name: Retag image as latest
        run: |
          REGISTRY="${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com"
          # Fetch the manifest of the tested image
          MANIFEST=$(aws ecr batch-get-image \
            --repository-name ${{ matrix.repo }} \
            --image-ids imageTag=${{ inputs.image_tag }} \
            --query 'images[0].imageManifest' --output text)

          # Push the same manifest with the :latest tag
          aws ecr put-image \
            --repository-name ${{ matrix.repo }} \
            --image-tag latest \
            --image-manifest "$MANIFEST"

This approach (retagging the manifest) is instant, no re-pulling or re-pushing layers. The :latest image is byte-for-byte identical to the tested timestamp image.

6.5 Deploy to QA EC2 (`deploy-qa.yml`)

The final step deploys to the persistent QA server using AWS SSM, no SSH keys needed:

# .github/workflows/deploy-qa.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Check QA EC2 is running
        id: check
        run: |
          STATE=$(aws ec2 describe-instances \
            --instance-ids ${{ secrets.QA_EC2_INSTANCE_ID }} \
            --query 'Reservations[0].Instances[0].State.Name' \
            --output text)
          echo "state=$STATE" >> $GITHUB_OUTPUT

      - name: Sync config files
        if: steps.check.outputs.state == 'running'
        run: |
          # Base64-encode configs and decode them on the EC2 to avoid quoting issues
          COMPOSE_B64=$(base64 -w0 docker-compose.prod.yml)
          NGINX_B64=$(base64 -w0 nginx/nginx.conf)

          aws ssm send-command \
            --instance-ids ${{ secrets.QA_EC2_INSTANCE_ID }} \
            --document-name "AWS-RunShellScript" \
            --parameters commands="[
              \"echo $COMPOSE_B64 | base64 -d > /home/ubuntu/app/docker-compose.prod.yml\",
              \"echo $NGINX_B64 | base64 -d > /etc/nginx/sites-enabled/default\",
              \"nginx -t && systemctl reload nginx\"
            ]"

      - name: Deploy containers
        if: steps.check.outputs.state == 'running'
        run: |
          aws ssm send-command \
            --instance-ids ${{ secrets.QA_EC2_INSTANCE_ID }} \
            --document-name "AWS-RunShellScript" \
            --parameters commands="[
              \"aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com\",
              \"cd /home/ubuntu/app && docker compose -f docker-compose.prod.yml pull\",
              \"DB_HOST=${{ secrets.DB_HOST }} DB_NAME=${{ secrets.DB_NAME }} DB_USER=${{ secrets.DB_USER }} DB_PASSWORD=${{ secrets.DB_PASSWORD }} NEXTAUTH_SECRET=${{ secrets.NEXTAUTH_SECRET }} docker compose -f docker-compose.prod.yml up -d\",
              \"sleep 10 && curl -sf http://localhost:5000/health\"
            ]"

All 5 jobs in the nightly pipeline completing successfully in 5m 26s

The deploy-qa job expanded showing each SSM step: sync config files, deploy via docker compose, health check

7. Domain Name with Route 53 and SSL with Let's Encrypt

7.1 Get a Domain and Migrate to Route 53

Purchase a domain on Name.com (or any registrar)
Create a Hosted Zone in AWS Route 53 for your domain
Note the 4 NS (nameserver) records Route 53 provides
In Name.com's DNS settings, replace the default nameservers with Route 53's NS records
Wait 24-48 hours for propagation

7.2 Create DNS Records

In Route 53, create an A record pointing to your QA EC2's public IP:

Route 53 hosted zone for shri.software showing the A record, NS, and SOA records

Type: A
Name: shri.software
Value: <EC2 Public IP>
TTL: 300

Note: If you stop/start EC2 instances, the public IP changes. Consider using an Elastic IP for the QA instance to keep the IP stable.

7.3 Install Certbot and Get an SSL Certificate

SSH into your QA EC2 and run:

# Install Certbot with the Nginx plugin
sudo apt-get update
sudo apt-get install -y certbot python3-certbot-nginx

# Obtain a certificate (Certbot automatically configures Nginx)
sudo certbot --nginx -d shri.software

# Verify auto-renewal is configured
sudo systemctl status certbot.timer
sudo certbot renew --dry-run

Certbot will:

Verify domain ownership by placing a file at /.well-known/acme-challenge/
Download the certificate to /etc/letsencrypt/live/shri.software/
Modify your Nginx config to use the certificate

8. Nginx as a Reverse Proxy

Nginx sits in front of both services, routing traffic and terminating SSL:

# /etc/nginx/sites-enabled/default

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name shri.software;
    return 301 https://$host$request_uri;
}

# HTTPS server
server {
    listen 443 ssl;
    server_name shri.software;

    ssl_certificate     /etc/letsencrypt/live/shri.software/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/shri.software/privkey.pem;

    # NextAuth routes must go to the frontend (Next.js handles them)
    location /api/auth/ {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto https;
    }

    # All other /api/ routes go to Express backend
    location /api/ {
        proxy_pass http://localhost:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Everything else goes to Next.js frontend
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto https;
    }
}

The order of location blocks matters: /api/auth/ must be listed before /api/ because Nginx uses longest prefix matching.

9. Security Best Practices

Here's what I did deliberately for security:

Containers

Non-root users in every container (adduser -S shri)
Alpine base images, smaller surface area, fewer CVEs
--production npm install, dev tools never ship to production
Multi-stage builds, build tools (compilers, test runners) never end up in production images

Authentication

GitHub OAuth, no password storage, no credential database to protect
Short-lived JWTs (1 hour expiry) between frontend and backend
User-scoped queries, every DB query filters by user_id from the verified JWT; users cannot access each other's data

Infrastructure

No SSH keys, AWS SSM for all remote execution; port 22 is closed
LabRole on EC2, AWS Academy's pre-provisioned role grants SSM and ECR access without custom policy authoring
RDS in private subnet, database is not publicly accessible
Secrets in GitHub Secrets, never hardcoded in workflow files or checked into git
HTTPS only, Nginx enforces HTTP → HTTPS redirect at the server level

Secrets Management

All secrets stored in GitHub, values are never visible after being saved

# GitHub Secrets used in this project:
AWS_ACCOUNT_ID          # AWS account number
AWS_ACCESS_KEY_ID       # From AWS Academy lab details panel
AWS_SECRET_ACCESS_KEY   # From AWS Academy lab details panel
AWS_SESSION_TOKEN       # From AWS Academy lab details panel (rotates each session)
TEMP_EC2_AMI            # AMI ID for smoke test instances
TEMP_EC2_SUBNET_ID      # VPC subnet for temporary instances
TEMP_EC2_SG_ID          # Security group ID
QA_EC2_INSTANCE_ID      # Persistent QA EC2 instance ID
DB_HOST                 # RDS endpoint
DB_NAME / DB_USER / DB_PASSWORD
NEXTAUTH_SECRET         # Shared secret for JWT signing
GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET  # OAuth app credentials
SOURCE_REPO_TOKEN       # PAT to check out source repo from infra repo

Conclusion

The full pipeline, from a git push to a verified deployment on HTTPS, runs without any manual steps. The key architectural wins:

Two repos for clean separation of application vs. infrastructure concerns
Timestamp-tagged images with promotion to :latest only after passing tests
Ephemeral test infrastructure that is created and destroyed per pipeline run
SSM-based deployment with no SSH keys to manage
Route 53 + Let's Encrypt for production-grade DNS and SSL at zero cost

The live app is running at https://shri.software. The source code is split across:

Source repo: github.com/Shreyas-Yadav/job-application-tracker
Infra repo: github.com/Shreyas-Yadav/job-application-tracker-infra

Building High-Availability Web Apps on AWS: Auto Scaling, ALB, and Private Subnets

Shreyas Yadav — Tue, 14 Oct 2025 22:26:38 +0000

This guide, we’ll see how to deploy a production ready weather application on AWS with enterprise grade architecture. We’l' use Auto Scaling Groups and Application Load Balancers for high availability, public-private subnets for security isolation, and Amazon ECR for containerized deployments. To make it production complete, we'll configure a custom domain with Route 53 and enable HTTPS using AWS Certificate Manager. The result? A fully scalable, secure web application accessible via your own domain with SSL encryption exactly how professional applications should be deployed.

GitHub Repository: https://github.com/Shreyas-Yadav/weather-aws

Clone the above repo and continue.

Our weather application uses a multi-tier architecture on AWS that separates the frontend and backend for better security and scalability.

Step 1: Creating the VPC

What is a VPC?

A VPC (Virtual Private Cloud) is your isolated network in AWS. Think of it as your own private data center in the cloud where all your servers and resources will live securely.

Setting Up the VPC

Navigate to VPC Creation

Go to AWS Console → Search "VPC" → Click "Your VPCs" → Click "Create VPC"

Choose "VPC and more"

You'll see two options:

VPC only - Manual setup (tedious)
VPC and more - Automatic setup(recommended)

Select "VPC and more" - AWS will automatically create all networking components for you.

Basic Configuration

Name tag auto-generation: weather-app
This prefix will be added to all resources, making them easy to identify.

IPv4 CIDR block: 10.0.0.0/16
This is your VPC's IP address range (65,536 available IPs).

IPv6: Select "No IPv6 CIDR block"
Tenancy: Keep as "Default"

Availability Zones and Subnets

Number of Availability Zones: Select 2
This is crucial for high availability - if one data center fails, your app continues running in the other.

Number of public subnets: 2
These will host your frontend servers (internet-accessible)

Number of private subnets: 2
These will host your backend servers (hidden from internet)

Subnet CIDR blocks: Keep the defaults:

Public subnet 1a:  10.0.0.0/20
Public subnet 1b:  10.0.16.0/20
Private subnet 1a: 10.0.128.0/20
Private subnet 1b: 10.0.144.0/20

NAT Gateways

Select "1 per AZ" (creates 2 total)

What's a NAT Gateway?
It allows your private backend servers to access the internet (for API calls, updates) while staying hidden from incoming internet traffic.

Why 2 NAT Gateways?
High availability each private subnet gets its own NAT Gateway for redundancy.

Additional Settings

VPC endpoints: None
DNS options: Enable both

Enable DNS hostnames
Enable DNS resolution

Create VPC

Click "Create VPC" and wait 2-3 minutes.

AWS will create:

1 VPC
4 Subnets (2 public, 2 private)
1 Internet Gateway
2 NAT Gateways
Route tables

Step 2: Creating Security Groups

What are Security Groups?

Security Groups are virtual firewalls that control inbound and outbound traffic to your AWS resources.

We'll create 4 security groups to control traffic between components.

Security Group 1: Frontend ALB Security Group

Navigate: EC2 → Security Groups → Create security group

Name: weather-frontend-alb-sg
Description: Security group for Frontend Application Load Balancer
VPC: weather-app-vpc

Inbound Rules:

HTTP (80) from 0.0.0.0/0
HTTPS (443) from 0.0.0.0/0

Outbound Rules: All traffic (default)

Security Group 2: Frontend EC2 Security Group

Name: weather-frontend-sg
Description: Security group for Frontend EC2 instances
VPC: weather-app-vpc

Inbound Rules:

HTTP (80) from Custom → weather-frontend-alb-sg
SSH (22) from My IP (optional for debugging)

Outbound Rules: All traffic (default)

Security Group 3: Backend ALB Security Group

Name: weather-backend-alb-sg
Description: Security group for Backend Application Load Balancer
VPC: weather-app-vpc

Inbound Rules:

HTTP (80) from Custom → weather-frontend-sg

Outbound Rules: All traffic (default)

Security Group 4: Backend EC2 Security Group

Name: weather-backend-sg
Description: Security group for Backend EC2 instances
VPC: weather-app-vpc

Inbound Rules:

Custom TCP (3000) from Custom → weather-backend-alb-sg
SSH (22) from My IP (optional for debugging)

Outbound Rules: All traffic (default)

Traffic Flow

Internet → Frontend ALB (80,443) → Frontend EC2 (80) → Backend ALB (80) → Backend EC2 (3000)

Step 3: IAM Role for EC2 Instances.

Note for AWS Academy/Lab Users

If you're using AWS Academy Labs or AWS Learner Lab, you cannot create custom IAM roles. Instead, use the pre-configured role provided:

Role name: LabRole

This role already has the necessary permissions for ECR, Parameter Store, and other AWS services.

When creating EC2 instances or Launch Templates, select LabRole as the IAM instance profile.

For Regular AWS Accounts

If you're using a personal/company AWS account (not AWS Labs), follow these steps to create a custom IAM role:

Navigate: IAM → Roles → Create role

Step 1: Select Trusted Entity

Trusted entity type: AWS service
Use case: EC2

Click Next

Add Permissions

Search and select these 2 managed policies:

AmazonEC2ContainerRegistryReadOnly
AmazonSSMManagedInstanceCore

Click Next

Name and Create

Role name: weather-ec2-role
Description: IAM role for Weather App EC2 instances

Click Create role

Add Parameter Store Policy

Click on weather-ec2-role → Permissions tab
Click Add permissions → Create inline policy → JSON tab

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParameters"
      ],
      "Resource": "arn:aws:ssm:*:*:parameter/weather-app/*"
    }
  ]
}

Policy name: ParameterStoreReadPolicy
Click **Create

Step 4: Creating ECR Repositories

What is Amazon ECR?

Amazon Elastic Container Registry (ECR) is a Docker container registry where we'll store our frontend and backend Docker images. EC2 instances will pull images from here during deployment.

Create ECR Repositories

Navigate: Amazon ECR → Repositories → Create repository

Repository 1: Frontend

Repository name: weather-frontend
Image tag mutability: Mutable
Encryption: AES-256 (default)
Scan on push: Enable (optional)

Click Create repository

Repository 2: Backend

Repository name: weather-backend
Image tag mutability: Mutable
Encryption: AES-256 (default)
Scan on push: Enable (optional)

Click Create repository

Note Your Repository URIs

After creation, note down both repository URIs (format: account-id.dkr.ecr.region.amazonaws.com/repository-name):

Frontend URI: 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-frontend
Backend URI: 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-backend

You'll need these URIs to push Docker images and configure EC2 instances.

Step 5: Build and Push Docker Images to ECR

Prerequisites

Ensure you have installed:

AWS CLI (latest version)
Docker

Build Docker Images

Important: Build images compatible with EC2 architecture (Linux AMD64/ARM64).

Navigate to your project directory and build both images:

Build Backend:

docker buildx build --platform linux/amd64,linux/arm64 -t weather-backend --load ./backend

Build Frontend:

docker buildx build --platform linux/amd64,linux/arm64 -t weather-frontend --load ./frontend

Why --platform flag?

EC2 instances run Linux. This ensures your Docker images are compatible with EC2's operating system and CPU architecture.

Push Images to ECR

AWS provides ready-to-use push commands for each repository.

To view push commands: ECR → Repositories → Select repository → Click "View push commands"

Authenticate Docker to ECR

Replace us-east-1 with your region and 339712997278 with your AWS account ID:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 339712997278.dkr.ecr.us-east-1.amazonaws.com

Tag Frontend Image

docker tag weather-frontend:latest 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-frontend:latest

Push Frontend Image

docker push 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-frontend:latest

Tag Backend Image

docker tag weather-backend:latest 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-backend:latest

Push Backend Image

docker push 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-backend:latest

Verify Images in ECR

Go to Amazon ECR → Repositories

Check both repositories:

weather-frontend - Should show latest tag
weather-backend - Should show latest tag

—

Step 6: Creating Target Groups

What are Target Groups?

Target Groups are used by Application Load Balancers to route traffic to registered EC2 instances. They also perform health checks to ensure traffic only goes to healthy instances.

We'll create 2 target groups - one for frontend and one for backend.

Target Group 1: Backend Target Group

Navigate: EC2 → Target Groups → Create target group

Step 1: Basic Configuration

Target type: Instances
Target group name: weather-backend-tg
Protocol: HTTP
Port: 3000
VPC: weather-app-vpc
Protocol version: HTTP1

Step 2: Health Checks

Health check protocol: HTTP
Health check path: /health

Advanced health check settings:

Healthy threshold: 2
Unhealthy threshold: 3
Timeout: 5 seconds
Interval: 30 seconds
Success codes: 200

Step 3: Register Targets

Skip this step - instances will be added automatically by Auto Scaling Group.

Click Create target group

Target Group 2: Frontend Target Group

Step 1: Basic Configuration

Target type: Instances
Target group name: weather-frontend-tg
Protocol: HTTP
Port: 80
VPC: weather-app-vpc
Protocol version: HTTP1

Step 2: Health Checks

Health check protocol: HTTP
Health check path: /health

Advanced health check settings:

Healthy threshold: 2
Unhealthy threshold: 3
Timeout: 5 seconds
Interval: 30 seconds
Success codes: 200

Step 3: Register Targets

Skip this step - instances will be added automatically by Auto Scaling Group.

Click Create target group

Step 7: Creating Application Load Balancers

What is an Application Load Balancer?

An Application Load Balancer (ALB) distributes incoming traffic across multiple EC2 instances. We'll create two ALBs:

Frontend ALB (Internet-facing) - Receives traffic from users
Backend ALB (Internal) - Receives traffic from frontend instances

ALB 1: Backend Application Load Balancer

Navigate: EC2 → Load Balancers → Create load balancer → Application Load Balancer

Basic Configuration

Load balancer name: weather-backend-alb
Scheme: Internal
IP address type: IPv4

Network Mapping

VPC: weather-app-vpc
Availability Zones: Select both
  ☑ us-east-1a → Select private subnet
  ☑ us-east-1b → Select private subnet

Security Groups

Remove: default
Add: weather-backend-alb-sg

Listeners and Routing

Protocol: HTTP
Port: 80
Default action: Forward to weather-backend-tg

Click Create load balancer

Wait 2-3 minutes for the ALB to become active.

Important: Copy the Backend ALB DNS name - you'll need it later!

ALB 2: Frontend Application Load Balancer

Basic Configuration

Load balancer name: weather-frontend-alb
Scheme: Internet-facing
IP address type: IPv4

Network Mapping

VPC: weather-app-vpc
Availability Zones: Select both
  ☑ us-east-1a → Select public subnet
  ☑ us-east-1b → Select public subnet

Security Groups

Remove: default
Add: weather-frontend-alb-sg

Listeners and Routing

Protocol: HTTP
Port: 80
Default action: Forward to weather-frontend-tg

Click Create load balancer

Wait 2-3 minutes for the ALB to become active.

Verify Load Balancers

Go to EC2 → Load Balancers

Check both ALBs show State: Active

Note down both DNS names:

Frontend ALB DNS: weather-frontend-alb-xxxxx.us-east-1.elb.amazonaws.com
Backend ALB DNS: weather-backend-alb-xxxxx.us-east-1.elb.amazonaws.com

Step 8: Configure AWS Systems Manager Parameter Store

What is Parameter Store?

Parameter Store allows you to store configuration data and secrets centrally. EC2 instances will fetch these values at runtime, eliminating hardcoded configuration.

Create Parameters

Navigate: AWS Systems Manager → Parameter Store → Create parameter.

Create the following parameters:

Parameter 1: AWS Region

Name: /weather-app/aws-region
Type: String
Value: us-east-1 (or your region)

Parameter 2: Backend ALB Host (used by frontend to call an api)

Name: /weather-app/backend-host
Type: String
Value: <Your Backend ALB DNS without http://>
Example: internal-weather-backend-alb-xxxxx.us-east-1.elb.amazonaws.com

Parameter 3: Backend ALB Port (used by frontend to call an api)

Name: /weather-app/backend-port
Type: String
Value: 80

Parameter 4: Frontend ECR Registry

Name: /weather-app/ecr-registry-frontend
Type: String
Value: <Your account ID>.dkr.ecr.us-east-1.amazonaws.com/weather-frontend
Example: 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-frontend

Parameter 5: Backend ECR Registry

Name: /weather-app/ecr-registry-backend
Type: String
Value: <Your account ID>.dkr.ecr.us-east-1.amazonaws.com/weather-backend
Example: 339712997278.dkr.ecr.us-east-1.amazonaws.com/weather-backend

Parameter 6: OpenWeather API Key

Name: /weather-app/openweather-api-key
Type: String (or SecureString for encryption)
Value: <Your OpenWeatherMap API key>

Get your free API key from: https://openweathermap.org/api

Verify Parameters

All 6 parameters should now be visible in Parameter Store with the prefix /weather-app/

—

Step 9: Create Launch Templates

What is a Launch Template?

A Launch Template defines the configuration for EC2 instances that will be launched by Auto Scaling Groups. We'll create two templates - one for backend and one for frontend.

Launch Template 1: Backend Launch Template

Navigate: EC2 → Launch Templates → Create launch template

Launch Template Name and Description

Launch template name: weather-backend-lt
Template version description: Backend instances for weather app

Application and OS Images (AMI)

Quick Start: Ubuntu
Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
Architecture: 64-bit (x86)

Instance Type

Instance type: t3.micro (or t2.micro for free tier)

Key Pair

Key pair name: Select your existing key pair

Network Settings

Subnet: Don't include in launch template
Firewall (security groups): weather-backend-sg

Storage

Keep default: 8 GiB, gp3

Advanced Details

IAM instance profile:

LabRole (for AWS Academy users)
OR weather-ec2-role (for regular AWS accounts)

Metadata accessible: Enabled

User data:

Copy the backend user data script from the repository:

Backend User Data Script →

This script:

Installs Docker and AWS CLI
Fetches configuration from Parameter Store
Pulls Docker image from ECR
Starts the backend container

Click Create launch template

Launch Template 2: Frontend Launch Template

Follow the same steps with these differences:

Launch template name: weather-frontend-lt
Template version description: Frontend instances for weather app
Security group: weather-frontend-sg

User data:

Copy the frontend user data script from the repository:

Frontend User Data Script →

This script:

Installs Docker and AWS CLI
Fetches backend ALB DNS from Parameter Store
Pulls Docker image from ECR
Starts the frontend container with backend configuration

Click Create launch template

Step 10: Create Auto Scaling Groups

What is an Auto Scaling Group?

An Auto Scaling Group (ASG) automatically manages EC2 instances - launching new instances when traffic increases and terminating them when traffic decreases. We'll create two ASGs - one for backend and one for frontend.

Note Create ASG first for backend and then create for frontend.

Auto Scaling Group 1: Backend ASG

Navigate: EC2 → Auto Scaling Groups → Create Auto Scaling group

Step 1: Choose Launch Template

Auto Scaling group name: weather-backend-asg
Launch template: weather-backend-lt
Version: Default (1)

Click Next

Step 2: Choose Instance Launch Options

VPC: weather-app-vpc

Availability Zones and subnets:
  ☑ weather-app-subnet-private1-us-east-1a
  ☑ weather-app-subnet-private2-us-east-1b

Click Next

Step 3: Configure Advanced Options

Load balancing:
  ☑ Attach to an existing load balancer

Choose from your load balancer target groups:
  ☑ weather-backend-tg

Health checks:
  ☑ Turn on Elastic Load Balancing health checks
  Health check grace period: 300 seconds

Click Next

Step 4: Configure Group Size and Scaling

Group size:
  Desired capacity: 2
  Minimum capacity: 2
  Maximum capacity: 4

Scaling:
  ☑ Target tracking scaling policy

  Scaling policy name: backend-cpu-scaling
  Metric type: Average CPU utilization
  Target value: 70
  Instance warmup: 300 seconds

Click Next → Next → Create Auto Scaling group

Wait 5-10 minutes for instances to launch and become healthy.

Auto Scaling Group 2: Frontend ASG

Follow the same steps with these differences:

Step 1: Choose Launch Template

Auto Scaling group name: weather-frontend-asg
Launch template: weather-frontend-lt

Step 2: Network

Availability Zones and subnets:
  ☑ weather-app-subnet-public1-us-east-1a
  ☑ weather-app-subnet-public2-us-east-1b

Step 3: Load Balancing

Target group: weather-frontend-tg

Step 4: Group Size and Scaling

Desired capacity: 2
Minimum capacity: 2
Maximum capacity: 4

Scaling policy name: frontend-cpu-scaling
Metric type: Average CPU utilization
Target value: 70

Click Create Auto Scaling group

Wait 5-10 minutes for instances to launch and become healthy.

Verify Auto Scaling Groups

Check Instances:

Go to EC2 → Instances
You should see 4 running instances:
- 2 backend instances (in private subnets)
- 2 frontend instances (in public subnets)

Check Target Health:

Go to EC2 → Target Groups
Select weather-backend-tg → Targets tab
- Both targets should show Healthy
Select weather-frontend-tg → Targets tab
- Both targets should show Healthy

—--

Step 11: Testing Your Application

Test Application via Frontend ALB

Get Frontend ALB DNS:

Go to EC2 → Load Balancers
Select weather-frontend-alb
Copy the DNS name

Example: weather-frontend-alb-841040545.us-east-1.elb.amazonaws.com

Test in Browser:

Open your browser and navigate to:

http://weather-frontend-alb-XXXXXXX.us-east-1.elb.amazonaws.com

You should see:

Weather Application interface
Search box to enter city name
Real-time weather data when you search

Test the Complete Flow:

Enter a city name (e.g., "Mumbai", "London", "New York")
Click Search
Weather information should display:
- Current temperature
- Weather condition (Haze, Clear, etc.)
- Feels like temperature
- Humidity
- Wind speed
- Pressure

Common Issues:

If the application doesn't work:

Check Target Health:
- EC2 → Target Groups → Check both target groups show "Healthy"
Check Instance Logs:
- Connect to instances via Session Manager
- View logs: sudo docker logs weather-frontend or sudo docker logs weather-backend
Check Security Groups:
- Verify all security group rules are correct
- Frontend ALB should allow 80 from internet
- Backend ALB should allow 80 from frontend SG

—--

Step 12: Add Custom Domain and Enable HTTPS

Prerequisites

You need a registered domain name from any registrar (Namecheap, GoDaddy, etc.)

Part A: Create Hosted Zone in Route 53

Navigate: Route 53 → Hosted zones → Create hosted zone

Configuration:

Domain name: shri.software
Type: Public hosted zone

Click Create hosted zone

Update Nameservers at Your Domain Registrar:

Copy the 4 NS (nameserver) records from Route 53
Go to your domain registrar (Namecheap, GoDaddy, etc.)
Replace existing nameservers with the 4 AWS nameservers
Save changes

Wait 15-30 minutes for DNS propagation

Part B: Request SSL/TLS Certificate

Navigate: AWS Certificate Manager (ACM) → Request certificate
Step 1: Request Certificate
Certificate type: Request a public certificate
Click Next

Step 2: Domain Names
Fully qualified domain name: shri.software
Add another name: *.shri.software (optional, for subdomains)

Step 3: Validation Method
Validation method: DNS validation - recommended

Step 4: Key Algorithm
RSA 2048 (default)
Click Request

Part C: Validate Certificate

After requesting the certificate, ACM will show validation details.
Click "Create records in Route”

This automatically adds the required CNAME validation records to your hosted zone.
Wait 5-10 minutes for validation to complete.
Certificate Status: Will change from "Pending validation" to "Issued"

Part D: Create A Record for Your Domain

Navigate: Route 53 → Hosted zones → shri.software

Create A Record:
Record name: (leave empty for root domain)
Record type: A - Alias
Route traffic to: Alias to Application Load Balancer
Region: us-east-1
Load balancer: weather-frontend-alb

Click Create records

Part E: Add HTTPS Listener to Frontend ALB

Navigate: EC2 → Load Balancers → weather-frontend-alb
Add HTTPS Listener:

Click Listeners tab
Click Add listener

Protocol: HTTPS
Port: 443
Default action: Forward to weather-frontend-tg
Default SSL/TLS certificate: From ACM → Select your certificate (shri.software)

Click Add

Part F: Redirect HTTP to HTTPS

Edit HTTP:80 Listener:

Select HTTP:80 listener
Click Actions → Edit listener
Remove existing forward action
Add action → Redirect to...

Protocol: HTTPS
Port: 443
Status code: 301 - Permanently moved

Click Save changes

Verify Your Setup

Test in Browser:

Open: https://your_domain_name

You should see:
✅ Padlock icon (secure connection)
✅ Weather application loads
✅ HTTP automatically redirects to HTTPS

Congratulations! Your application is live.

Step 13: Cleanup and Delete Resources

It is essential to delete all resources created in this guide to avoid incurring unnecessary charges on your AWS account.

Cleanup Order

Auto Scaling Groups (ASGs):
Delete weather-frontend-asg and weather-backend-asg. (This action automatically terminates all running EC2 instances.)
Application Load Balancers (ALBs):
Delete weather-frontend-alb and weather-backend-alb.
Target Groups:
Delete weather-frontend-tg and weather-backend-tg.
Launch Templates:
Delete weather-frontend-lt and weather-backend-lt.
ECR Repositories:
Delete weather-frontend and weather-backend. (This deletes the stored Docker images.)
Route 53 & AWS Certificate Manager (ACM):
Delete the A Record pointing to the ALB.
Delete the ACM Certificate.
Delete the Hosted Zone in Route 53.
IAM Role & Security Groups:
Delete the IAM Role (weather-ec2-role).
Delete the 4 Security Groups (weather-frontend-alb-sg, weather-frontend-sg, etc.).
Systems Manager Parameters:
Delete all Parameters under the /weather-app/ path.
VPC:
Delete the VPC (weather-app-vpc). (This final step cleans up Subnets, Internet Gateway, and NAT Gateways.)

Conclusion and Thank You..!

Congratulations! You have successfully deployed a highly available, secure, and production-ready weather application on AWS using enterprise-grade architecture.

By completing this guide, you’ve mastered key AWS services including VPC, Auto Scaling Groups, Application Load Balancers, ECR, Route 53, and ACM. This foundational knowledge is crucial for deploying any modern, scalable web application.

Thank you for following along, and I hope this guide helps you on your cloud journey!

DEV Community: Shreyas Yadav

Adding MapReduce to My Go Distributed File System

The pieces

Step 0: chunks that don't split lines

Submitting a job

The map side: spill, sort, merge

The reduce side: waiting for every mapper

Handling failed and hung map tasks

What I'd change next

Building an HDFS-Style Distributed File System in Go

The wire protocol: one envelope, length-prefixed

Joining the cluster and staying alive

Placing chunks

Writing: pipeline replication

Committing: no half-written files in the namespace

Reading: parallel fetch, positional writes

Self-healing reads on the storage node

Re-replication after a node dies

Why the 3-minute grace period

Deletes

What I'd change next

Implementing Strict Priority and Deficit Round Robin Schedulers in NS-3

The Setup: Creating a Bottleneck

Design

Classification: Filters and Traffic Classes

The Scheduler Interface

JSON Configuration

A Gotcha: The PPP Header

Strict Priority Queueing

Deficit Round Robin

Peeking Must Not Spend Credit

What I Would Change

Run It Yourself

I Built a Nightly Pipeline That Deploys My App While I Sleep

Table of Contents

1. Architecture Overview

High-Level Architecture

2. The Application: Job Application Tracker

Tech Stack

Database Schema

Backend API

Authentication Flow

3. Dockerizing the App

Backend Dockerfile

Frontend Dockerfile (Multi-Stage Build)

4. Local Development with Docker Compose

5. AWS Infrastructure Setup

5.1 GitHub OAuth App

5.2 AWS ECR (Elastic Container Registry)

5.3 AWS RDS (MySQL 8)

5.4 QA EC2 Instance

5.5 IAM Credentials for GitHub Actions (AWS Academy)

6. GitHub Actions CI/CD Pipeline

6.1 Nightly Orchestrator (nightly.yml)

6.2 Build and Push to ECR (build.yml)

6.3 Smoke Test on Temporary EC2 (smoke-test.yml)

6.4 Promote Image (promote.yml)

6.5 Deploy to QA EC2 (deploy-qa.yml)

7. Domain Name with Route 53 and SSL with Let's Encrypt

7.1 Get a Domain and Migrate to Route 53

7.2 Create DNS Records

7.3 Install Certbot and Get an SSL Certificate

8. Nginx as a Reverse Proxy

9. Security Best Practices

Containers

Authentication

Infrastructure

Secrets Management

Conclusion

Building High-Availability Web Apps on AWS: Auto Scaling, ALB, and Private Subnets

Step 1: Creating the VPC

What is a VPC?

Setting Up the VPC

Step 2: Creating Security Groups

What are Security Groups?

Security Group 1: Frontend ALB Security Group

Security Group 2: Frontend EC2 Security Group

Security Group 3: Backend ALB Security Group

Security Group 4: Backend EC2 Security Group

Traffic Flow

6.1 Nightly Orchestrator (`nightly.yml`)

6.2 Build and Push to ECR (`build.yml`)

6.3 Smoke Test on Temporary EC2 (`smoke-test.yml`)

6.4 Promote Image (`promote.yml`)

6.5 Deploy to QA EC2 (`deploy-qa.yml`)