DEV Community

Cover image for Automated Private Server Backup Pipeline to S3 with a Serverless Audit Trail
Hyelngtil Isaac
Hyelngtil Isaac

Posted on

Automated Private Server Backup Pipeline to S3 with a Serverless Audit Trail

Project Summary

This project builds an automated, secure backup pipeline for an application server that has no public IP address and no ability to delete its own backups. A private EC2 instance generates application data continuously via a systemd service, and a cron-scheduled bash script archives and uploads that data to an S3 bucket every ten minutes. An S3 Gateway VPC Endpoint provides the private server with direct access to S3 over the AWS backbone, eliminating the ongoing cost and attack surface of a NAT Gateway. IAM is configured with strict least-privilege: the instance role permits only s3:PutObject, so neither a compromised server nor an operator mistake can erase existing backups. On the serverless side, a Lambda function fires on every S3 PUT event and writes an audit record to DynamoDB, creating an independent audit trail with zero human involvement.

Key tools: AWS VPC, EC2, S3, IAM, S3 Gateway VPC Endpoint, Lambda (Python), DynamoDB, systemd, cron, AWS CLI v2.

Architectural objective: Defence-in-depth, network isolation prevents lateral movement, least-privilege IAM limits blast radius, and a serverless audit trail ensures every backup is independently recorded without relying on the server that produced it.


How the Pipeline Works and Why Each Component Exists

Every ten minutes, cron on the private app server archives /var/app/data and uploads it to S3 through a Gateway VPC Endpoint. Each upload triggers a Lambda function that writes an audit record to DynamoDB.

Component Role Why it exists
VPC with public and private subnets Network boundary Separates the one internet-facing machine from the workload.
Bastion host SSH jump box The only entry point, locked to my IP. The app server has no public IP.
App server + systemd service Simulated workload Produces data continuously and survives reboots and crashes.
Cron + backup.sh Scheduler and archiver Automates backups with no human in the loop and logs each result locally.
S3 bucket Backup storage Durable storage with Block Public Access on.
S3 Gateway Endpoint Private path to S3 Lets the private server reach S3 without any internet route.
IAM role + PutObject-only policy Credentials and permissions Gives the server temporary credentials and the minimum access it needs.
Lambda + DynamoDB Audit trail Records every backup independently of the server that made it.
CloudWatch Logs Lambda observability Shows whether the audit function ran or failed.
AWS Budget Cost guardrail Warns me before a forgotten resource becomes a bill.
Temporary NAT Gateway One-time installer access Used only to install the AWS CLI, then deleted.

Why an IAM role instead of access keys

  • Temporary credentials: the role delivers short-lived credentials through the instance metadata service, and they rotate automatically. Access keys are long-lived until someone rotates them.
  • Nothing to leak: no key is ever written to disk, committed to Git or baked into an AMI.
  • Easy to revoke: detaching the role cuts access immediately.
  • Scoped to EC2: the trust policy lets only the EC2 service assume the role.

Why a Gateway VPC Endpoint instead of a NAT Gateway

NAT Gateway S3 Gateway Endpoint
Cost Hourly charge, per-GB processing charge and an Elastic IP No hourly or data-processing charge
Reach The entire internet S3 only (gateway endpoints exist only for S3 and DynamoDB)
Traffic path Out through the Internet Gateway Stays on the AWS network via a route-table entry
Security Gives a compromised server a route for data exfiltration The server still has no internet route
Role in this project Temporary, for installing the CLI Permanent

Prerequisites & Environment

Requirement Detail
AWS account (IAM User) With administrative IAM access (or scoped permissions to create VPC, EC2, S3, IAM, Lambda, DynamoDB, and CloudWatch resources)
AWS region All resources in a single region: eu-south-1
EC2 key pair An existing key pair in eu-south-1.
Local machine SSH client with agent forwarding support (ssh-add, ssh -A)
Public IP Current public IPv4 address for bastion host.
AMI Ubuntu Server 24.04 LTS (or latest LTS)
Budget Estimated cost under \$5 for a few hours of lab work if resources are torn down promptly. The NAT Gateway bills hourly, delete it as soon as software installation is complete

Initial state: A clean AWS account (or an account where no resources conflict with the names above). Your SSH keys .pem file are on your local machine. You have not yet created any of the resources.


Phase 1: Network Infrastructure

This phase creates the VPC, subnets, Internet Gateway, route tables, security groups, and both EC2 instances. By the end, you will have an SSH session on the private app server, reached through the bastion host.

Step 1: Create the VPC

Provision a VPC with a /16 CIDR block to house both the public and private subnets.

  1. Open the VPC Console → Your VPCs → Create VPC.
  2. Select VPC only (not "VPC and more" — building each component manually gives full control and understanding).
  3. Configure:
Field Value
Name tag backup-pipeline-vpc
IPv4 CIDR block 10.0.0.0/16
IPv6 CIDR block No IPv6 CIDR block
Tenancy Default

4. Click Create VPC.

Step 2: Create the public and private subnets

Create one subnet for the internet-facing bastion and NAT Gateway, and one for the application server, both in eu-south-1a.

The public subnet is the only subnet with a route to the Internet Gateway. Placing the bastion there lets you SSH in from outside AWS, while everything else stays private.

  1. Navigate to VPC Console → Subnets → Create subnet and select backup-pipeline-vpc.
  2. Add both subnets (use Add new subnet for the second):
Subnet name Availability Zone IPv4 CIDR block
public-subnet eu-south-1a 10.0.1.0/24
private-subnet eu-south-1a 10.0.2.0/24

3. Click Create subnet.

  1. Select public-subnet → Actions → Edit subnet settings → enable Auto-assign public IPv4 address → Save. Leave this off for private-subnet.

Step 3: Create and attach the Internet Gateway

Provision an Internet Gateway and attach it to the VPC to give the public subnet internet connectivity.

  1. Navigate to VPC Console → Internet Gateways → Create internet gateway.
  2. Name it backup-pipeline-igw.
  3. Click Create internet gateway.
  4. On the next screen, click Actions → Attach to VPC → select backup-pipeline-vpc → Attach internet gateway.

Step 4: Configure route tables

Create two route tables: one for the public subnet (with a default route to the IGW) and one for the private subnet (initially with only the local route; the S3 endpoint and temporary NAT route come later).

Public route table:

  1. Navigate to VPC Console → Route Tables. Create a new one:
    • Click Create route table → Name: public-rtb → VPC: backup-pipeline-vpc → Create route table.
  2. Select public-rtb → Routes tab → Edit routes → Add route:
Destination Target
0.0.0.0/0 Select the Internet Gateway (backup-pipeline-igw)

3. Click Save changes.
4. Go to the Subnet associations tab → Edit subnet associations → select public-subnet → Save associations.

Private route table:

  1. Click Create route table → Name: private-rtb → VPC: backup-pipeline-vpc → Create route table.
  2. Do not add any routes yet (only the local 10.0.0.0/16 route exists by default).
  3. Go to the Subnet associations tab → Edit subnet associations → select private-subnet → Save associations.

Step 5: Create security groups

Create two security groups: one for the bastion (SSH from your IP only) and one for the app server (SSH from the bastion's security group only).

Bastion security group:

  1. Navigate to VPC Console → Security Groups → Create security group.
  2. Configure:
Field Value
Security group name bastion-sg
Description SSH access from my IP only
VPC backup-pipeline-vpc

3. Inbound rules → Add rule:

Type Protocol Port Range Source
SSH TCP 22 <YOUR_PUBLIC_IP>/32

4. Leave outbound rules as the default (all traffic allowed).
5. Click Create security group.

App server security group:

  1. Click Create security group.
  2. Configure:
Field Value
Security group name app-server-sg
Description SSH from bastion only
VPC backup-pipeline-vpc

3. Inbound rules → Add rule:

Type Protocol Port Range Source
SSH TCP 22 Select Custom → type bastion-sg and select the security group

4. Leave outbound rules as the default (all traffic allowed).
5. Click Create security group.

Step 6: Launch the bastion host

Launch a small EC2 instance in the public subnet to serve as the SSH jump box.

  1. Navigate to EC2 Console → Instances → Launch instances.
  2. Configure:
Field Value
Name bastion
AMI Ubuntu Server 24.04 LTS (or latest LTS)
Instance type t2.micro/t3.micro (free-tier eligible)
Key pair <bastion KEY_PAIR_NAME>(create new)
Network settings → VPC backup-pipeline-vpc
Subnet public-subnet
Auto-assign public IP Enable (should be enabled by subnet setting)
Security group Select existing → bastion-sg

3. Leave storage as default (8 GiB gp2/gp3).
4. Click Launch instance.

Step 7: Launch the app server

Launch the private application server in the private subnet with no public IP address.

  1. Navigate to EC2 Console → Instances → Launch instances.
  2. Configure:
Field Value
Name app-server
AMI Ubuntu Server 24.04 LTS (same as bastion)
Instance type t3.micro
Key pair <app-server KEY_PAIR_NAME> (create new; key pair you will use agent forwarding)
Network settings → VPC backup-pipeline-vpc
Subnet private-subnet
Auto-assign public IP Disable
Security group Select existing → app-server-sg

3. Leave storage as default.
4. Click Launch instance.

Step 8: SSH through the bastion to the app server

Verify the network and security group configuration by establishing an SSH session on the app server via the bastion using agent forwarding.

On your local machine, run:

# Change key-pair permission
sudo chmod 400 <bastion KEY_PAIR_NAME>.pem
sudo chmod 400 <app-server KEY_PAIR_NAME>.pem

# Load your private key into the SSH agent
eval "$(ssh-agent -s)"
ssh-add <bastion KEY_PAIR_NAME>.pem
ssh-add <app-server KEY_PAIR_NAME>.pem

# SSH to the bastion with agent forwarding (-A)
ssh -A ubuntu@<BASTION_PUBLIC_IP>
Enter fullscreen mode Exit fullscreen mode

Once on the bastion, SSH to the app server using its private IP:

# From the bastion, hop to the app server
ssh ubuntu@<APP_SERVER_PRIVATE_IP>
Enter fullscreen mode Exit fullscreen mode

Expected result: You land on the app server's shell. Running hostname returns the app server's hostname. Running curl -m 5 https://example.com or curl -s ifconfig.me should time out or fail (confirming no public internet access.
Note: The NAT Gateway does not exist yet).

Verify you are on the private server:

# Confirm the private IP matches what the EC2 console shows
hostname -I
Enter fullscreen mode Exit fullscreen mode


Phase 2: NAT Gateway & Software Installation

The app server needs internet access exactly once to install the AWS CLI and unzip. A NAT Gateway provides that access temporarily. It will be deleted as soon as installation is complete, because NAT Gateways bill hourly and the app server must not retain internet access in the final architecture.

Step 9: Allocate an Elastic IP for the NAT Gateway

Allocate an Elastic IP address that the NAT Gateway will use as its public-facing address.

  1. Navigate to VPC Console → Elastic IPs → Allocate Elastic IP address.
  2. Leave all settings at their defaults (Amazon's pool of IPv4 addresses).
  3. Click Allocate.
  4. Note the Allocation ID (e.g. eipalloc-xxxxxxxx) — you will need it in the next step and when releasing the IP later.

Optionally, add a Name tag: nat-gw-eip.

Step 10: Create the NAT Gateway

Place a NAT Gateway in the public subnet so private-subnet instances can reach the internet for package installation.

  1. Navigate to VPC Console → NAT Gateways → Create NAT gateway.
  2. Configure:
Field Value
Name temp-nat-gw
Subnet public-subnet
Connectivity type Public
Elastic IP allocation ID Select the Elastic IP allocated in Step 9

3. Click Create NAT gateway.
4. Wait for the status to change from Pending to Available (typically 1–2 minutes).

Step 11: Add a default route to the private route table via the NAT Gateway

Point the private subnet's outbound internet traffic through the NAT Gateway.

  1. Navigate to VPC Console → Route Tables → select private-rtb.
  2. Go to the Routes tab → Edit routes → Add route:
Destination Target
0.0.0.0/0 Select the NAT Gateway (temp-nat-gw)

3. Click Save changes.

Step 12: Verify internet access from the app server

Confirm that the app server can now reach the internet through the NAT Gateway before installing software.

SSH to the app server (via the bastion, as in Phase 1) and run:

# Test outbound internet connectivity
curl -s -o /dev/null -w "%{http_code}" https://example.com
Enter fullscreen mode Exit fullscreen mode

It's expected that command returns 200. If it hangs or returns an error, verify that private-rtb has the NAT Gateway route and that the NAT Gateway status is **Available.

Step 13: Install unzip and AWS CLI v2 on the app server

Install the tools the app server needs to create and upload backups.

From the app server's SSH session:

# Update package lists and install unzip
sudo apt update && sudo apt install -y unzip

# Download the AWS CLI v2 installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# Extract and install
unzip awscliv2.zip
sudo ./aws/install

# Verify installation
aws --version
Enter fullscreen mode Exit fullscreen mode

Next, set the default region so every AWS CLI command targets the correct region without requiring --region each time:

# Set the default region (no access keys needed — the IAM role handles credentials)
aws configure set region <AWS_REGION>
Enter fullscreen mode Exit fullscreen mode

Clean up the installer files:

rm -rf awscliv2.zip aws/
Enter fullscreen mode Exit fullscreen mode

  • Remove the NAT Gateway immediately to stop the hourly billing and to enforce the final architecture where the app server has no internet access.

  • Release the Elastic IP that was allocated for the NAT Gateway to avoid charges on an unattached Elastic IP.

  • Confirm that the NAT Gateway route in the private-rtb is now a blackhole, proving the app server has lost internet access.

Step 14: Create the S3 Gateway VPC Endpoint

Create a Gateway Endpoint for Amazon S3 and attach it to the private route table so the app server can upload backups without traversing the public internet.

  1. Navigate to VPC Console → Endpoints → Create endpoint.
  2. Configure:
Field Value
Name tag s3-gateway-endpoint
Service category AWS services
Service Search for s3 → select the Gateway type entry (e.g. com.amazonaws.<AWS_REGION>.s3, Type: Gateway)
VPC backup-pipeline-vpc
Route tables Select private-rtb

3. Leave the policy as Full access.
4. Click Create endpoint.

Step 15: Verify the endpoint route and the VPC resource map

Confirm the endpoint took effect and the topology looks right.

  1. Navigate to VPC Console → Route Tables → private-rtb → Routes tab. A route for the S3 prefix list, targeting the endpoint, should now appear beside the local route.
  2. Navigate to VPC Console → Your VPCs → backup-pipeline-vpc → Resource map tab to see the whole topology.


Phase 3: S3 Bucket & IAM Configuration

This phase creates the backup destination bucket with all public access blocked, builds a least-privilege IAM role that permits only s3:PutObject, attaches it to the app server, and proves the restriction works by demonstrating a successful upload and a denied deletion.

Step 16: Create the S3 bucket with Block Public Access

Create a private S3 bucket that will store all backup archives.

  1. Navigate to S3 Console → Create bucket.
  2. Configure:
Field Value
Bucket name s3-backup-pipeline-bucket (select Account Regional namespace)
AWS Region eu-south-1 (same region as the VPC and endpoint)
Object Ownership ACLs disabled (recommended)
Block Public Access Block all public access — ensure all four checkboxes are checked
Bucket Versioning Disabled (noted as a production improvement in the article)
Default encryption SSE-S3 (default)

3. Leave all other settings at their defaults.
4. Click Create bucket.

Step 17: Create the IAM policy for the EC2 backup role

Define a policy that permits only s3:PutObject on the backups/* prefix in your bucket. No other S3 actions are allowed.

  1. Navigate to IAM Console → Policies → Create policy.
  2. Switch to the JSON tab and paste the following policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::s3-backup-pipeline-bucket-<ACCOUNT_ID>-eu-south-1-an/backups/*"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

3. Click Next.
4. Configure:

Field Value
Policy name ec2-backup-put-only
Description Allows PutObject only on s3 bucket

5. Click Create policy.

Step 18: Create the IAM role for the EC2 instance

Create an IAM role with an EC2 trust policy and attach the backup policy to it.

  1. Navigate to IAM Console → Roles → Create role.
  2. Configure the trust policy:
Field Value
Trusted entity type AWS service
Use case EC2

3. Click Next.
4. On the Add permissions page, search for ec2-backup-put-only and select it.
5. Click Next.
6. Configure:

Field Value
Role name ec2-backup-role
Description EC2 role for backup uploads, PutObject only

7. Click Create role.

Step 19: Attach the IAM role to the app server

Associate the IAM role with the running app server instance so it receives temporary credentials automatically.

  1. Navigate to EC2 Console → Instances → select app-server.
  2. Click Actions → Security → Modify IAM role.
  3. In the IAM role dropdown, select ec2-backup-role.
  4. Click Update IAM role.

Step 20: Test the IAM role: allowed upload, denied delete

Two tests show the policy does exactly what it should: uploading works, deleting doesn't.

The upload proves the whole chain works: the VPC Endpoint gives network access, the IAM role gives credentials, and the policy scopes access to the right bucket and prefix. The failed delete proves the role has no s3:DeleteObject.

SSH to the app server (via the bastion) and run:

# Create a test file and upload it to the backups/ prefix
echo "backup test $(date)" > /tmp/test-backup.txt
aws s3 cp /tmp/test-backup.txt s3://s3-backup-pipeline-bucket-<ACCOUNT_ID>-eu-south-1-an/backups/test-backup.txt
Enter fullscreen mode Exit fullscreen mode

Expected result: upload: ./test-backup.txt to s3://<BUCKET_NAME>/backups/test-backup.txt

Now try to delete it:

aws s3 rm s3://s3-backup-pipeline-bucket-<ACCOUNT_ID>-eu-south-1-an/backups/test-backup.txt
Enter fullscreen mode Exit fullscreen mode

The Access Denied error confirms the policy is working as intended. The role permits PutObject but denies all other S3 actions, including DeleteObject, GetObject, and ListBucket.

Finally, open S3 Console → Buckets → <BUCKET_NAME> → backups/ and confirm test-backup.txt is listed with a recent timestamp.


Phase 4: Application & Automation

This phase creates a systemd service that continuously produces application data, simulating a real workload. The service must start on boot and restart automatically if it crashes.

Step 21: Create and start the data generator service

Create the data directory, then a systemd unit that appends a timestamped line to /var/app/data/app.log every 30 seconds.

This service simulates a real application producing data continuously. Running it under systemd instead of a background script gives you boot persistence, automatic restarts and systemctl status for monitoring: Restart=always with RestartSec=5 brings it back five seconds after any exit, and WantedBy=multi-user.target starts it on every boot.

SSH to the app server (via the bastion) and run:

sudo mkdir -p /var/app/data

sudo tee /etc/systemd/system/data-generator.service > /dev/null << 'EOF'
[Unit]
Description=Application data generator — appends timestamped lines to /var/app/data/app.log
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c 'while true; do echo "$(date +\"%%Y-%%m-%%d %%H:%%M:%%S\") - Application event: data generated" >> /var/app/data/app.log; sleep 30; done'
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd, then enable the service for boot persistence and start it now
sudo systemctl daemon-reload
sudo systemctl enable --now data-generator.service
Enter fullscreen mode Exit fullscreen mode

Step 22: Verify the data generator is active

Confirm the service is running and producing data.

Check the service status:

sudo systemctl status data-generator.service
Enter fullscreen mode Exit fullscreen mode

Wait at least 60 seconds, then check the application log:

head -10 /var/app/data/app.log
Enter fullscreen mode Exit fullscreen mode

Step 23: Write the backup script

Create /usr/local/bin/backup.sh a bash script that archives /var/app/data, uploads the archive to S3, logs the result, and removes the local copy on success.

sudo tee /usr/local/bin/backup.sh > /dev/null << 'SCRIPT'
#!/bin/bash
set -euo pipefail

#---------------------------------------------------------
# Configuration
#---------------------------------------------------------
BUCKET="s3-backup-pipeline-bucket-<ACCOUNT_ID>-eu-south-1-an"
DATA_DIR="/var/app/data"
BACKUP_LOG="/var/log/backup.log"
TIMESTAMP="$(date +%Y-%m-%d-%H%M)"
ARCHIVE_NAME="backup-${TIMESTAMP}.tar.gz"
ARCHIVE_PATH="/tmp/${ARCHIVE_NAME}"
S3_DEST="s3://${BUCKET}/backups/${ARCHIVE_NAME}"

#---------------------------------------------------------
# Create the archive
#---------------------------------------------------------
/bin/tar -czf "${ARCHIVE_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"

#---------------------------------------------------------
# Upload to S3
#---------------------------------------------------------
if /usr/local/bin/aws s3 cp "${ARCHIVE_PATH}" "${S3_DEST}"; then
    echo "$(date +%Y-%m-%d\ %H:%M:%S) - SUCCESS: Uploaded ${ARCHIVE_NAME} to ${S3_DEST}" >> "${BACKUP_LOG}"
    # Remove the local archive only after a successful upload
    /bin/rm -f "${ARCHIVE_PATH}"
else
    echo "$(date +%Y-%m-%d\ %H:%M:%S) - FAILURE: Upload of ${ARCHIVE_NAME} to ${S3_DEST} failed" >> "${BACKUP_LOG}"
    exit 1
fi
SCRIPT
Enter fullscreen mode Exit fullscreen mode

Make the script executable:

sudo chmod +x /usr/local/bin/backup.sh
Enter fullscreen mode Exit fullscreen mode

Script walkthrough:

Section Purpose
set -euo pipefail -e exits on any command failure. -u treats unset variables as errors. -o pipefail propagates failures through pipes. Together, they prevent silent partial failures.
TIMESTAMP date +%Y-%m-%d-%H%M produces a sortable, human-readable timestamp (e.g. 2026-11-10-0735).
tar -czf Creates a gzip-compressed tar archive of /var/app/data. The -C and basename pattern avoids embedding absolute paths in the archive.
aws s3 cp Uploads via the S3 VPC Endpoint using the instance role's temporary credentials. Uses the absolute path /usr/local/bin/aws for cron compatibility.
if/else logging Logs success or failure with a timestamp to /var/log/backup.log, creating a local audit trail.
rm -f on success Deletes the local archive only after a confirmed upload, preventing disk space exhaustion. On failure, the archive is preserved for manual retry.

Step 24: Test the backup script manually

Run the script once by hand to verify it works before scheduling it with cron.

sudo /usr/local/bin/backup.sh
Enter fullscreen mode Exit fullscreen mode
# Check the backup log should show a SUCCESS entry
cat /var/log/backup.log
Enter fullscreen mode Exit fullscreen mode
# Confirm the local archive was deleted after successful upload
ls /tmp/backup-*.tar.gz 2>/dev/null || echo "No local archives — cleanup worked"
Enter fullscreen mode Exit fullscreen mode

Error Hit: Backup script upload rejected by S3

Issue: When I ran backup.sh manually, the upload failed and S3 refused the operation entirely. No archive reached the bucket.

Root cause: The BUCKET variable in the script held a bucket name that didn't match the real bucket. S3 couldn't resolve the target. The ec2-backup-role policy also allows s3:PutObject only on one exact bucket ARN, so a request aimed at any other name falls outside what the role permits and is rejected.

Fix: I corrected BUCKET in /usr/local/bin/backup.sh to match the actual bucket name and re-ran the script:

sudo /usr/local/bin/backup.sh && cat /var/log/backup.log
Enter fullscreen mode Exit fullscreen mode

The log then showed a SUCCESS entry.

Step 25: Verify the backup in the S3 console

Confirm the archive uploaded by the script is present in the S3 bucket.

  1. Navigate to S3 Console → Buckets → <BUCKET_NAME> → backups/.

Step 26: Schedule the backup with cron and verify it runs

Add a cron entry to root's crontab that runs backup.sh every 10 minutes.

sudo crontab -e
Enter fullscreen mode Exit fullscreen mode

If prompted for an editor, choose nano. Add this line at the end, then save and exit (Ctrl+O, Enter, Ctrl+X):

*/10 * * * * /usr/local/bin/backup.sh
Enter fullscreen mode Exit fullscreen mode

The five fields are minute, hour, day of month, month and day of week, so */10 * * * * fires at :00, :10, :20, :30, :40 and :50 of every hour. Confirm it was saved with sudo crontab -l.

Wait 20–30 minutes for two or three runs, then check the log and the bucket:

cat /var/log/backup.log
Enter fullscreen mode Exit fullscreen mode

If nothing appears, check whether cron is running the script:

grep CRON /var/log/syslog | tail -10
Enter fullscreen mode Exit fullscreen mode

Then open S3 Console → Buckets → <BUCKET_NAME> → backups/ and confirm several backups exist.


Phase 5: Serverless Audit Trail & End-to-End Validation

Step 27: Create the DynamoDB table

Create a DynamoDB table to store one audit record per backup, keyed by the S3 object key.

  1. Navigate to DynamoDB Console → Tables → Create table.
  2. Configure:
Field Value
Table name backup-log
Partition key backup-id (String)
Sort key Leave empty
Table settings Customize settings
Read/write capacity settings On-demand

3. Leave all other settings (encryption, tags) at their defaults.
4. Click Create table.
5. Wait for the table status to change to Active.

Step 28: Create the Lambda execution role

Create an IAM role that the Lambda function will assume. It needs exactly two permissions: writing audit records to the backup-log DynamoDB table, and writing logs to CloudWatch Logs.

  1. Navigate to IAM Console → Roles → Create role.
  2. Configure the trust policy:
Field Value
Trusted entity type AWS service
Use case Lambda

3. Click Next.
4. On the Add permissions page, do not attach any managed policies yet. Click Next.
5. Configure:

Field Value
Role name lambda-backup-audit-role
Description Lambda role for backup audit DynamoDB PutItem and CloudWatch Logs only

6. Click Create role.

Now add an inline policy with the exact permissions needed:

7. Navigate to IAM Console → Roles → select lambda-backup-audit-role.
8. Go to the Permissions tab → Add permissions → Create inline policy.
9. Switch to the JSON tab and paste:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "dynamodb:PutItem",
            "Resource": "arn:aws:dynamodb:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/backup-log"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<AWS_REGION>:<AWS_ACCOUNT_ID>:*"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Replace <AWS_REGION> with your region and <AWS_ACCOUNT_ID> with your 12-digit AWS account ID (visible in the top-right corner of the console).

10. Click Next → Name the policy lambda-audit-inline-policy → Create policy.

Step 29: Create the Lambda function

Create a Python Lambda function that processes S3 PUT event notifications and writes an audit record to DynamoDB for each new backup.

  1. Navigate to Lambda Console → Functions → Create function.
  2. Select Author from scratch.
  3. Configure:
Field Value
Function name backup-audit-logger
Runtime Python 3.12 (or latest available Python runtime)
Architecture x86_64
Permissions → Execution role Use an existing role → lambda-backup-audit-role

4. Click Create function.

Step 30: Add the function code

Replace the default function code with the audit logger.

  1. In the Lambda function's Code tab, replace the contents of lambda_function.py with:
import boto3
import urllib.parse

table = boto3.resource("dynamodb").Table("backup-log")


def lambda_handler(event, context):
    for rec in event["Records"]:
        key = urllib.parse.unquote_plus(rec["s3"]["object"]["key"])
        table.put_item(
            Item={
                "backup-id": key,
                "size_bytes": rec["s3"]["object"]["size"],
                "uploaded_at": rec["eventTime"],
                "bucket": rec["s3"]["bucket"]["name"],
            }
        )
    return {"statusCode": 200}
Enter fullscreen mode Exit fullscreen mode

2. Click Deploy to save and deploy the function.

Step 31: Add the S3 trigger

Configure the Lambda function to fire on s3:ObjectCreated:Put events for objects with the backups/ prefix.

  1. In the Lambda function page, click Add trigger.
  2. Configure:
Field Value
Trigger source S3
Bucket <BUCKET_NAME>
Event types PUT (or specifically s3:ObjectCreated:Put)
Prefix backups/
Suffix Leave empty

3. Check the acknowledgment box ("I acknowledge that using the same S3 bucket...").
4. Click Add.

Expected result: The trigger appears in the function's Configuration → Triggers tab, showing the S3 bucket and the backups/ prefix filter.

Step 32: Wait for new backups and verify DynamoDB records

Allow the cron job to produce at least one new backup (up to 10 minutes), then check DynamoDB for the corresponding audit record. If you already have 3+ backups in S3 from Phase4, the Lambda trigger will fire for any new backup uploaded after the trigger was created. It does not retroactively process existing objects.

Wait for at least 2–3 cron cycles (20–30 minutes), then check DynamoDB:

  1. Navigate to DynamoDB Console → Tables → backup-log → Explore table items.

Expected result: One record per backup uploaded after the trigger was created. Each record contains:

Field Example Value
backup-id backups/backup-2026-11-10-0810.tar.gz
size_bytes 456 (varies)
uploaded_at 2026-11-10T08:10:00.000Z
bucket <BUCKET_NAME>

Step 33: Check the Lambda CloudWatch logs

Verify that the Lambda function executed successfully by reviewing its CloudWatch log output.

  1. Navigate to CloudWatch Console → Log Management → /aws/lambda/backup-audit-logger.
  2. Click on the most recent log stream.

No error messages between START and END indicates the function completed successfully.

Troubleshooting: Lambda fails with DynamoDB ValidationException

Issue: After adding the S3 trigger, no records appeared in the backup-log table. The CloudWatch logs for backup-audit-logger showed every invocation failing with:

ValidationException: Missing the key backup-id in the item

Root cause: A typo in the put_item() call. The code wrote the partition key as backup_id (underscore), but the table's partition key is backup-id (hyphen). DynamoDB requires key names to match the table schema exactly, so every write was rejected.

Fix: I changed the key name in the item dict in lambda_function.py and clicked Deploy:

table.put_item(
    Item={
        "backup-id": key,   # I corrected this to match the table's partition key exactly
        "size_bytes": rec["s3"]["object"]["size"],
        "uploaded_at": rec["eventTime"],
        "bucket": rec["s3"]["bucket"]["name"],
    }
)
Enter fullscreen mode Exit fullscreen mode

After the next cron cycle, new items appeared in backup-log and the log stream showed a clean START/END with no errors. Objects uploaded while the function was failing don't get audit records, because Lambda doesn't replay them.

Lesson: DynamoDB attribute names are case-sensitive and must match exactly. Hyphens are legal, but they require ExpressionAttributeNames in query and update expressions, so underscores are safer for future tables.

Phase 6: End-to-End Validation Proofs

This phase collects the final proof points required: internet isolation, backup over VPC Endpoint, and the budget alert.

Step 34: Prove internet access fails but S3 backup succeeds

From the app server, demonstrate that outbound internet requests fail (NAT Gateway was deleted) but the S3 backup upload still works (via the Gateway Endpoint).

SSH to the app server (via the bastion) and run:

# Test internet access — should fail (timeout after 5 seconds)
curl -m 5 https://example.com
Enter fullscreen mode Exit fullscreen mode

Now run a manual backup to prove S3 access works:

# Run the backup script — should succeed via the VPC Endpoint
sudo /usr/local/bin/backup.sh

# Verify the latest entry in the backup log
tail -1 /var/log/backup.log
Enter fullscreen mode Exit fullscreen mode

Step 35: Create a budget alert

Create an AWS Budget to alert you if costs exceed a threshold.

  1. Navigate to AWS Billing & Cost Management → Budgets → Create a budget.
  2. Select Customize (Advanced) → Cost budget → Next.
  3. Configure:
Field Value
Budget name backup-pipeline-budget
Period Monthly
Budget amount Fixed → $5.00 (or any low threshold appropriate for this lab)
Start month Current month

4. Click Next.
5. Add an alert threshold:

Field Value
Threshold 80% of budgeted amount (Actual cost)
Email recipients Your email address

6. Click Next → Next → Create budget.


Phase 7: Resource Cleanup

Delete everything to stop billing. Follow this order, because several resources can't be deleted while others still depend on them.

  1. Lambda function backup-audit-logger (Lambda → Functions → Actions → Delete).
  2. DynamoDB table backup-log (type confirm).
  3. S3 bucket: first Empty it (type permanently delete), then Delete it (type the bucket name). The S3 trigger is stored on the bucket, so it goes with it.
  4. EC2 instances bastion and app-server: terminate both and wait for Terminated.
  5. S3 Gateway Endpoint s3-gateway-endpoint.
  6. Security groups app-server-sg and bastion-sg.
  7. Subnets public-subnet and private-subnet.
  8. Route tables public-rtb and private-rtb.
  9. Internet Gateway backup-pipeline-igw: Detach from VPC first, then delete.
  10. VPC backup-pipeline-vpc.
  11. IAM: delete the roles ec2-backup-role and lambda-backup-audit-role (the Lambda inline policy goes with its role), then the policy ec2-backup-put-only.
  12. CloudWatch log group /aws/lambda/backup-audit-logger.

I kept backup-pipeline-budget as a standing cost guardrail.

Finally, confirm nothing is left:

Navigate to each service console and confirm no project resources remain:

Console What to verify
EC2 → Instances No running instances (terminated instances disappear after ~1 hour)
S3 → Buckets <BUCKET_NAME> no longer listed
VPC → Your VPCs backup-pipeline-vpc no longer listed
DynamoDB → Tables backup-log no longer listed
Lambda → Functions backup-audit-logger no longer listed
IAM → Roles ec2-backup-role and lambda-backup-audit-role no longer listed
VPC → Elastic IPs No allocated Elastic IPs from this project

What I Would Improve in a Real Company

This lab trades some production safeguards for speed and low cost. In a real environment I would add the following.

S3 versioning and Object Lock. Versioning is disabled here. The role cannot delete objects, but PutObject can still overwrite an existing key, so a compromised server could replace a good backup with garbage. Versioning keeps every previous version. Object Lock makes backups immutable for a set retention period.

Lifecycle rules. At one backup every ten minutes, the bucket grows without limit. I would expire backups after a retention window, expire noncurrent versions and abort incomplete multipart uploads. Larger, long-lived archives can move to cheaper Glacier tiers.

Alerts. A failing function like the Lambda failure in Step 33 only shows up if someone reads the logs. I would add:

  • a CloudWatch alarm on the Lambda Errors metric, notifying an SNS topic
  • an alarm that fires when no new backup has landed recently, because cron sends no failure notification
  • an on-failure destination or dead-letter queue on the Lambda so failed audit events aren't lost

Encryption. SSE-S3 is on by default. I would move to SSE-KMS with a customer-managed key for key-level access control and CloudTrail auditing. The EC2 role would then need kms:GenerateDataKey on that key. I would also add a bucket policy that denies non-TLS requests (aws:SecureTransport) and encrypt the DynamoDB table with the same approach.


Conclusion

In this project I built an automated backup pipeline for a private EC2 server. The server has no public IP and no route to the internet. Every ten minutes it archives its data and uploads it to S3 through a Gateway VPC Endpoint, and each upload triggers a Lambda function that writes an audit record to DynamoDB.

Three ideas carried the design:

  • Least privilege. The server's role allows s3:PutObject and nothing else, and the Lambda role allows only a DynamoDB write and CloudWatch Logs. Step 20 proves it: the delete attempt was denied.
  • Private networking. A temporary NAT Gateway installed the AWS CLI, and then I removed it. The Gateway Endpoint gives S3 access at no extra cost, with no path to the wider internet.
  • An independent audit trail. The Lambda function records every backup without relying on the server that produced it.

The two errors I hit along the way, a mistyped bucket name and a mismatched DynamoDB key, taught me as much as the build itself. Small, exact details decide whether a cloud system works.

Good cloud security rarely comes from one big control. It comes from many small, deliberate restrictions that add up.

Top comments (0)