DEV Community

Cover image for Server Data Archival to Amazon S3
Tejas Shinkar
Tejas Shinkar

Posted on

Server Data Archival to Amazon S3

#aws #cloud #devops #sideprojects

Building a production-oriented archival workflow with Python, EC2, IAM and S3

A common infrastructure problem sounds simple:

What should happen to data when an old server is about to be decommissioned?

Deleting the server isn't the difficult part. The difficult part is making sure the right data is preserved, the migration is verified, and the process can be safely repeated without creating duplicates or silently losing files.

That became the problem I decided to solve.

1. The Problem

Imagine a legacy server with a /data directory containing years of application data across logs/, reports/, application/, and backups/.

Before decommissioning it, we need to:

  • Identify which files are still within the retention period
  • Archive eligible files
  • Skip older files
  • Preserve the directory structure
  • Upload them securely to Amazon S3
  • Verify that uploads actually succeeded
  • Record what happened
  • Handle failures
  • Safely rerun the process without duplicating data

Instead of treating this as an S3 upload exercise, I approached it as an infrastructure automation problem.

2. Breaking the Problem Down

I divided the work into three sections.

Section 1 — Script Development

First, I focused on the automation itself. The Python script needed to follow a clear pipeline:

Scan → validate → check retention → check existing archive → upload → verify → log

This gave us a clear separation between the business logic and the AWS infrastructure.

Section 2 — Server Setup

Next, I created a simulated legacy server using an Amazon Linux EC2 instance. Instead of simply running everything as root, I created a /data directory for the source server data, /opt/server-archival for the application, /var/log/server-archival for application logs, a dedicated archiver system user, and a Python virtual environment.

The goal was to make the environment behave more like something we'd actually encounter on a managed server.

Section 3 — Migration & Hardening

Once the script and server were ready, I connected the environment to S3. The EC2 instance received an IAM role scoped to only s3:PutObject and s3:GetObject — no AWS credentials were hardcoded into the Python script.

The final workflow became:

Legacy EC2
    │
    │ /data
    ▼
Python Archival Script
    │
    ├── Retention Check
    ├── Idempotency Check
    ├── Upload
    ├── Verification
    └── Logging
    │
    │ IAM Role
    ▼
Amazon S3
    │
    └── archive/
         ├── logs/
         ├── reports/
         └── ...
Enter fullscreen mode Exit fullscreen mode

3. Retention Logic

The requirement was to archive files within the last 5 years. Initially, I considered simply subtracting 5 × 365 days, but that isn't completely calendar-accurate because of leap years — so the implementation was changed to use calendar-aware date calculation.

This also gave us a chance to test the boundary condition:

  • File just inside the 5-year window → Archive
  • File just outside the window → Skip

That kind of boundary testing is easy to overlook in a basic lab.

4. Preserving the File Structure

S3 doesn't have traditional directories — instead, object keys are used. For example, /data/reports/report.csv became s3://legacy-server-archive-bucket/archive/reports/report.csv. The archive/ prefix gives us a clean separation between the original server path and the S3 archive.

5. Making the Process Idempotent

This was one of the most important parts of the project. A migration script shouldn't blindly upload the same file every time it runs, so before uploading, the script checks whether the corresponding S3 object already exists:

Does the object already exist? Yes → skip. No → upload.

This makes the workflow idempotent — if the script crashes halfway through and we run it again, already archived files don't need to be uploaded again.

6. Verification

Uploading successfully isn't enough. After the upload, the script performs an S3 HeadObject check to verify that the object exists:

Not just: upload → assume success. Instead: upload → verify → log success/failure.

That makes the workflow safer for an actual migration scenario.

7. Logging

The script maintains structured logs in a timestamp | level | message format:

INFO | /data/reports/report.csv → Uploaded and verified
INFO | /data/backups/backup.tar.gz → Skip
INFO | /data/logs/app.log → Already archived, skipping upload
Enter fullscreen mode Exit fullscreen mode

At the end, it generates a summary covering files scanned, eligible, uploaded, already archived, skipped, and failed — making the execution auditable instead of relying only on terminal output.

8. Problems We Actually Faced

This project became much more valuable because things didn't always work on the first attempt.

Amazon Linux vs Ubuntu. The initial bootstrap approach used apt, but the EC2 instance was Amazon Linux, so the setup had to be adapted to use dnf.

Python runtime. Boto3 raised a runtime-support warning with the original Python version. Instead of ignoring it, I upgraded the environment to Python 3.11 and recreated only the virtual environment — the application, data, and logs remained separate.

The S3 403 problem. This was probably the most interesting AWS issue. Our IAM policy intentionally didn't include s3:ListBucket. When checking whether an object existed, S3 returned 403 AccessDenied instead of the expected 404 Not Found. At first this looked like an error in the idempotency logic, but it turned into an important AWS permissions lesson: S3's response to a missing object can depend on what permissions the caller has. We adapted the check accordingly without simply adding broader permissions, preserving the least-privilege design.

9. Testing With Realistic Load

Instead of testing with only 2–3 files, I created a larger test dataset — 57 files spread across different directories, file ages, eligible files, old files, and already-archived files.

Metric First Run Second Run
Files scanned 57 57
Eligible 33 33
Uploaded 30 0
Already archived 3 33
Skipped 24 24
Failed 0 0

This gave us actual evidence that the idempotency mechanism worked.

10. Production-Oriented Practices

Throughout the project, I deliberately tried to move beyond simply making the code "work."

🔐 Least-privilege IAM — only the required S3 object permissions were granted.
🔑 No credentials in code — the EC2 IAM role provided temporary AWS credentials through the instance role.
👤 Dedicated service user — the archival application didn't need to run as root.
🔄 Idempotency — repeated execution doesn't create duplicate uploads.
🛡️ Validation — the server path is validated before processing.
🔎 Verification — every successful upload is checked.
♻️ Retry handling — Boto3 was configured with standard retry behavior.
📝 Structured logging — execution results and failures are recorded.
🧪 Failure & boundary testing — the workflow was tested against different file ages, permissions, and AWS behavior.
🏃 Dry-run capability — the script can identify what would be archived without actually uploading it.

11. What I Took Away

The biggest learning wasn't how to upload a file to S3. It was learning to think about a cloud problem as a system:

What happens if it fails?
What happens if I run it twice?
What permissions does it really need?
How do I verify the result?
How do I know what happened after the process finishes?

Those questions changed the project from a simple AWS exercise into a production-oriented cloud automation workflow.

Final Architecture 

Legacy Server (EC2)
       │
       │ /data
       ▼
Python Archival Automation
       │
       ├── Validate
       ├── Retention Filter
       ├── Idempotency
       ├── Upload
       ├── Verification
       └── Logging
       │
       ▼
IAM Role
(Least Privilege)
       │
       ▼
Amazon S3
       │
       └── archive/
Enter fullscreen mode Exit fullscreen mode

Tech stack: AWS EC2 · Amazon S3 · IAM · Python · Boto3 · Linux · Cloud Automation

The project is available on GitHub as server-data-archival, including the implementation, IAM policy, documentation, and test evidence.

Top comments (1)

Collapse
 
raknaos profile image
Raknaos •

The 403-instead-of-404 detail is the most valuable line in the writeup, because it breaks the mental model most people carry: HeadObject on a key you have no s3:ListBucket for doesn't answer "not there", it answers "I can't tell you". Treating that as "missing" and uploading again is what turns a least-privilege policy into duplicate objects later, so the honest state is a third one — unknown — and it should fail closed rather than widen permissions to make the check pretty. Did you end up logging it as its own outcome, or do those keys fall into "skipped"?

The two-run table (30 uploaded then 0, 3 already archived then 33) is also the part most archival scripts never show. Idempotency claims without a second run against the same dataset are just a code reading, and 57 files with deliberately mixed ages is cheap to build and catches the classifier drift that a 3-file test hides.