DEV Community: rizasaputra

Securely Exporting MongoDB Atlas Snapshots to S3 Over AWS PrivateLink

rizasaputra — Sat, 21 Feb 2026 06:36:05 +0000

Organizations with strict regulatory compliance requirements often need to ensure that sensitive backup data never traverses the public internet. Exporting Atlas snapshots to your own S3 bucket also provides additional control over retention policies, lifecycle management, and disaster recovery strategies beyond Atlas's built-in backup capabilities.

MongoDB Atlas supports exporting snapshots to S3 over AWS PrivateLink—keeping all traffic on private IP addresses within the AWS network. Atlas exposes a dedicated object storage private endpoint for backup exports; you create it via the Atlas API/CLI, and Atlas provisions and manages the underlying AWS PrivateLink infrastructure for you.

This guide provides a step-by-step implementation to meet compliance requirements around data movement and network isolation by exporting Atlas snapshots to S3 over PrivateLink.

Current Limitations

Before diving in, understand the constraints:

AWS only: Your Atlas cluster must be hosted on AWS (not GCP or Azure)
Same-region only: Your Atlas cluster and S3 bucket must be in the same AWS region.
Requires M10+ Atlas clusters
Additional cost: $0.01/hour for PrivateLink connection

Prerequisites

You'll need:

MongoDB Atlas cluster (M10+) hosted on AWS with Cloud Backup enabled
AWS account with permissions to create IAM roles and S3 buckets
Atlas cluster and S3 bucket in the same AWS region

Phase 1: Installing Required Tools

Installing AWS CLI

macOS

brew install awscli

Or using the installer:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Linux

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Configure your credentials:

aws configure

Verify:

aws --version

Installing Atlas CLI

macOS

brew install mongodb-atlas-cli

Linux

# Debian/Ubuntu
wget https://fastdl.mongodb.org/mongocli/mongodb-atlas-cli_latest_linux_x86_64.deb
sudo dpkg -i mongodb-atlas-cli_latest_linux_x86_64.deb

# RHEL/CentOS/Fedora
wget https://fastdl.mongodb.org/mongocli/mongodb-atlas-cli_latest_linux_x86_64.rpm
sudo rpm -i mongodb-atlas-cli_latest_linux_x86_64.rpm

Verify:

atlas --version

Phase 2: Authenticating with Atlas

Creating Atlas API Keys

Log into MongoDB Atlas
Go to your project, then expand the sidebar, choose "Project Identity & Access", and then "Applications", and then "API Keys"
Click "Create API Key"
Name it descriptively (e.g., "PrivateLink S3 Export")
Assign "Project Owner" role (required for PrivateLink and backup exports)
Save both the Public Key and Private Key securely

Authenticate the CLI

atlas auth login

Select "API Keys" and enter your credentials:

? Select authentication type: API Keys (for existing automations)
? Public API Key: <your-public-key>
? Private API Key: <your-private-key>

Verify:

atlas auth whoami

Phase 3: Creating the S3 Bucket

Set your region and create a private bucket for snapshots:

# Set your region (must match your Atlas cluster region)
export AWS_REGION=REPLACE-WITH-YOUR-AWS-REGION-CODE-LIKE-us-east-1
export BUCKET_NAME=REPLACE-WITH-YOUR-ATLAS-SNAPSHOT-BUCKET-NAME

# Create the bucket
aws s3 mb s3://$BUCKET_NAME --region $AWS_REGION

Verify public access is blocked:

aws s3api get-public-access-block --bucket $BUCKET_NAME

All four settings should be true.

Phase 4: Setting Up Unified AWS Access

Atlas uses a unified AWS access model where you authorize an IAM role once, and it can be used across multiple Atlas features (backups, encryption, etc.).

Step 1: Create Atlas Cloud Provider Access Role

Set your project ID:

export PROJECT_ID=YOUR_PROJECT_ID

Create the access role in Atlas:

atlas cloudProviders accessRoles aws create --projectId $PROJECT_ID --output json | tee atlas-access-role.json

This returns something like this:

{
  "providerName": "AWS",
  "atlasAWSAccountArn": "arn:aws:iam::012345678999:root",
  "atlasAssumedRoleExternalId": "xxxxxxxx-1234-5678-9000-xxxxyyyyzzzz",
  "createdDate": "2026-02-21T03:55:10Z",
  "featureUsages": [],
  "roleId": "xxxxyyyyzzzz123412341234"
}

Save these values:

ATLAS_AWS_ACCOUNT=$(jq -r '.atlasAWSAccountArn' atlas-access-role.json)
EXTERNAL_ID=$(jq -r '.atlasAssumedRoleExternalId' atlas-access-role.json)
ROLE_ID=$(jq -r '.roleId' atlas-access-role.json)

echo "Atlas AWS Account: $ATLAS_AWS_ACCOUNT"
echo "External ID: $EXTERNAL_ID"
echo "Role ID: $ROLE_ID"

Step 2: Create IAM Role in AWS

Create the trust policy using the values from Atlas:

cat > atlas-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "$ATLAS_AWS_ACCOUNT"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "$EXTERNAL_ID"
        }
      }
    }
  ]
}
EOF

Create the IAM role:

export ROLE_NAME=REPLACE-WITH-YOUR-AWS-IAM-ROLE-NAME

aws iam create-role \
  --role-name $ROLE_NAME \
  --assume-role-policy-document file://atlas-trust-policy.json \
  --description "Role for MongoDB Atlas unified AWS access"

Step 3: Attach S3 Permissions

Create the S3 policy:

cat > atlas-s3-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSnapshotExport",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::$BUCKET_NAME",
        "arn:aws:s3:::$BUCKET_NAME/*"
      ]
    }
  ]
}
EOF

Create and attach the policy:

# Get your AWS account ID
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Create policy
aws iam create-policy \
  --policy-name MongoDBAtlasSnapshotExportPolicy \
  --policy-document file://atlas-s3-policy.json

# Attach to role
aws iam attach-role-policy \
  --role-name $ROLE_NAME \
  --policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/MongoDBAtlasSnapshotExportPolicy

Step 4: Authorize the IAM Role in Atlas

Get the IAM role ARN:

IAM_ROLE_ARN=$(aws iam get-role --role-name $ROLE_NAME \
  --query 'Role.Arn' --output text)

echo "IAM Role ARN: $IAM_ROLE_ARN"

Authorize the role in Atlas:

atlas cloudProviders accessRoles aws authorize $ROLE_ID \
  --projectId $PROJECT_ID \
  --iamAssumedRoleArn $IAM_ROLE_ARN

Optional: Add S3 Bucket Policy

For defense in depth, restrict bucket access to only the IAM role:

cat > bucket-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOnlyAtlasRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "$IAM_ROLE_ARN"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::$BUCKET_NAME",
        "arn:aws:s3:::$BUCKET_NAME/*"
      ]
    }
  ]
}
EOF

aws s3api put-bucket-policy \
  --bucket $BUCKET_NAME \
  --policy file://bucket-policy.json

Phase 5: Creating Object Storage Private Endpoint

Create the Private Endpoint

Convert AWS region to Atlas format:

# Convert ap-southeast-3 to AP_SOUTHEAST_3 (Atlas format)
ATLAS_REGION=$(echo $AWS_REGION | tr '[:lower:]' '[:upper:]' | tr '-' '_')
echo "Atlas Region: $ATLAS_REGION"

Create the private endpoint:

cat > private-endpoint-payload.json <<EOF
{
  "cloudProvider": "AWS",
  "regionName": "$ATLAS_REGION"
}
EOF

# Create the private endpoint
atlas api cloudBackups createBackupPrivateEndpoint \
  --cloudProvider AWS \
  --groupId $PROJECT_ID \
  --file private-endpoint-payload.json \
  --output json | tee private-endpoint-response.json

# Extract endpoint ID
PRIVATE_ENDPOINT_ID=$(jq -r '.id' private-endpoint-response.json)
echo "Private Endpoint ID: $PRIVATE_ENDPOINT_ID"

Monitor Private Endpoint Status

The private endpoint goes through several states: INITIATING → PENDING_ACCEPTANCE → ACTIVE

atlas api cloudBackups getBackupPrivateEndpoint \
  --cloudProvider AWS \
  --groupId $PROJECT_ID \
  --endpointId $PRIVATE_ENDPOINT_ID

Wait until the status is ACTIVE before proceeding. This typically takes a few minutes as Atlas provisions the PrivateLink infrastructure.

Phase 6: Configuring Export Bucket with Private Networking

Now that the private endpoint is active, you can create an export bucket that uses it.

Create Export Bucket with Private Networking

Create the export bucket configuration:

cat > export-bucket-payload.json <<EOF
{
  "bucketName": "$BUCKET_NAME",
  "cloudProvider": "AWS",
  "iamRoleId": "$ROLE_ID",
  "requirePrivateNetworking": true
}
EOF

# Create the export bucket
atlas api cloudBackups createExportBucket \
  --groupId $PROJECT_ID \
  --file export-bucket-payload.json \
  --output json | tee export-bucket-response.json

# Extract bucket ID
BUCKET_ID=$(jq -r '._id' export-bucket-response.json)
echo "Export Bucket ID: $BUCKET_ID"

When requirePrivateNetworking is set to true, Atlas uses the object storage private endpoint you created earlier. All exports will flow through PrivateLink.

Verify Export Bucket Configuration

atlas api cloudBackups getExportBucket \
  --groupId $PROJECT_ID \
  --exportBucketId $BUCKET_ID

Confirm that requirePrivateNetworking is set to true.

Phase 7: Exporting Snapshots Over PrivateLink

With everything configured, we can now export snapshots. All exports will automatically use PrivateLink.

List Available Snapshots

export CLUSTER_NAME=REPLACE-WITH-YOUR-CLUSTER-NAME
atlas backups snapshots list $CLUSTER_NAME --projectId $PROJECT_ID

This shows all snapshots with IDs and timestamps.

Manual Export

Export a specific snapshot:

# Create export payload
export SNAPSHOT_ID=REPLACE-WITH-YOUR-SELECTED-SNAPSHOT-ID

cat > export-payload.json <<EOF
{
  "snapshotId": "$SNAPSHOT_ID",
  "exportBucketId": "$BUCKET_ID",
  "customData": [
    { "key": "exported_via", "value": "privateLink" }
  ]
}
EOF

# Start the export
atlas api cloudBackups createBackupExport \
  --clusterName $CLUSTER_NAME \
  --groupId $PROJECT_ID \
  --file export-payload.json

Monitor Export Progress

atlas backups exports jobs list $CLUSTER_NAME \
  --projectId $PROJECT_ID

Status progresses: QUEUED → IN_PROGRESS → SUCCESSFUL

Export duration depends on snapshot size and network throughput.

Verify in S3

Once the export job state changes to SUCCESSFUL, you can verify the export in S3.

aws s3 ls s3://$BUCKET_NAME/ --recursive

Snapshots are organized by path:

/exported_snapshots/<orgUUID>/<projectUUID>/<clusterName>/<initiationDateOfSnapshot>/<timestamp>/

Phase 8: Automating Exports with Backup Policies

On top of manual exports, you can configure Atlas to automatically export snapshots on a schedule.

Configure Automatic Export Schedule

# Create schedule update payload
cat > schedule-payload.json <<EOF
{
  "autoExportEnabled": true,
  "export": {
    "exportBucketId": "$BUCKET_ID",
    "frequencyType": "monthly"
  }
}
EOF

# Update the backup schedule
atlas api cloudBackups updateBackupSchedule \
  --clusterName $CLUSTER_NAME \
  --groupId $PROJECT_ID \
  --file schedule-payload.json

Available frequency types:

monthly: Export once per month
yearly: Export once per year

Atlas automatically exports snapshots matching the frequency type.

View Current Schedule

atlas api cloudBackups getBackupSchedule \
  --clusterName $CLUSTER_NAME \
  --groupId $PROJECT_ID

This shows both snapshot and export schedules.

Summary

This guide covered the complete implementation of MongoDB Atlas snapshot exports to S3 over AWS PrivateLink. The key steps:

Create S3 bucket and IAM role with minimal permissions
Create object storage private endpoint and export bucket in Atlas with requirePrivateNetworking: true
Export snapshots and configure automated export scheduling using backup policies

Key implementation considerations:

Atlas cluster must run on AWS in the same region as the destination S3 bucket
The requirePrivateNetworking: true flag enables PrivateLink for all exports to that bucket
Atlas automatically manages PrivateLink infrastructure—no manual VPC endpoint setup required
Use unified AWS access (cloud provider access roles) for IAM role setup
PrivateLink connection costs $0.01/hour (billed separately); data processing charge included in $0.125/GB export price
Native Atlas backup schedule policies eliminate the need for external automation

This architecture is appropriate for organizations with regulatory requirements mandating private-only networking for data movement.

Reference Documentation

Rebuilding an Old Website Using Kiro (and What Went Wrong)

rizasaputra — Wed, 31 Dec 2025 13:02:56 +0000

I rebuilt an old website I hadn’t touched in 7 years using Kiro and technology I never tried. I went from implementing auth in less than 1 hour, to spending 6 days figuring out a simple CRUD with file upload, to a very productive stretch where I finished the remaining CMS and site features from scratch without a framework in 4 days, and finally spending another 3 days painfully cleaning up AI-generated garbage just to make sure everything actually runs in production.

You’ve probably seen plenty of similar posts about AI coding assistants already, but here’s my experience anyway:

Learn the concept first when you know nothing about what you're building

If you're trying something new, take time to actually learn the concept. In my case, I spent days stuck on a simple CRUD because I was totally not familiar with how Cloudflare R2, Workers, and Cloudflare Image Resizing actually work, so I couldn’t direct the AI to do what I want properly. It was only after I spent time reading the docs and understanding the concepts that I could finally understand the mess and garbage I had produced, clean it up, and make it work.

Spec driven development is powerful, but…

I found there are many cases where I’m very clear on what I want to do, and vibe coding without writing any spec is much more productive. For dev teams, I think this means requirement spec for AI can’t replace specs written by PM or QA, at least not for the moment.

Keep specs small

If you’re using spec driven development (Kiro style), I found I can only be productive when there are at most 4 requirements. 2–3 requirements per spec works much better. More than that, I lose the cognitive capability required to read the requirements and design the solution, let alone understand the code being generated. Split your spec. Period.

Steering files are very important

The best hack I found is simply taking time to keep steering files updated. Whenever I finished implementing something, updating the steering files helped the AI understand things that were previously ambiguous, or things where I had already changed my mind.

Before this, I often had to correct the output. With updated steering files, the result was much closer to what I wanted from the start. I didn’t really measure it, but the time saving difference was very noticeable.

AI does silently break things

Even with steering files, and with Kiro searching and reading the codebase before implementing tasks, it still messed things up. It could reinvent implementations instead of reusing existing code which leads to hidden instability, add conflicting CSS rules that go unnoticed until you actually build and deploy, or inject overly strict security configs that make the app unusable. I also found Kiro has a tendency to over-engineer things.

To understand what actually happened, one trick I used was committing to git frequently so I could compare the overall file diff. For me, this was much easier than trying to follow small, scattered diffs shown during task execution.

Checkpointing is underrated

Kiro has a feature I really like: checkpointing. If I was in the middle of implementing something and changed my mind, I could easily restore both the chat and the code to a previous point. It’s basically having an undo button for coding.

Being able to undo and retry with a different approach easily is extremely valuable, and honestly, it reminded me why I could take some random club from Serie C to European champions in Football Manager back when I still had a life.

ChatGPT is the steroid booster for Kiro

Maybe it’s because I’m using Auto model selection so I might be served a weaker model, but I found ChatGPT’s reasoning when engineering the solution is often better.

Kiro has a tendency to just accept and follow whatever you say, and quickly jump into doing stuff. With Kiro, doing and debugging feels more like trial and error. With ChatGPT, since it’s chat-based, you actually get time to think things through properly.

In the end, the best setup for me was me and ChatGPT co-bossing Kiro to actually work and develop lol.

Humans beat AI in short-range sprinting

Did I say vibe coding can be more productive? The only thing that’s sometimes even more productive is implementing or fixing the code myself, especially when I’m already very clear on what needs to be written. I found humans, or at least me, retain context better than AI for a small project, so I don’t need to keep scanning existing code just to get going. I simply can’t type as fast as the AI.

Before closing this out, here’s the before and after.

Old site

Rebuild

More importantly, I now have a solid foundation to assist others and scale AI coding assistant usage into something more complex.

Understanding AWS Elastic Beanstalk Worker Timeout

rizasaputra — Tue, 16 Apr 2019 11:06:03 +0000

In case you never heard it before, AWS has this orchestration service called Beanstalk. Basically it means, instead of setting up EC2 instances, load balancers, SQS, CloudWatch, etc manually, you can use this Beanstalk to orchestrate the setup. All you need to do is zip your code (or package it if you use Java), upload the zip, and do some setup from a centralised dashboard and you are done. If you don't need to have a fine control over your setup, this can really help you to get a running system quickly.

AWS Beanstalk provide two types of environment: Web server environment, and Worker environment. On Web server environment, you will get a server configured with your platform choices (can be Java, Node JS, Docker, etc), and you also can setup a load balancer easily. On worker environment, you can have server and SQS messaging platform to run heavy background jobs or scheduled jobs. I will focus on Worker environment on this post.

A Beanstalk worker instance has a daemon process which continuously listen to an SQS queue. Whenever it detects a message in the queue, the daemon will send a HTTP POST request locally to http://localhost/ on port 80 with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST. You can configure the daemon to post to different URL, connect to existing queue, and configure the timeouts.

There are 3 basic timeout you can configure from the worker environment dashboard:

1. Connection timeout

This is the number of seconds we want to wait for our application when we try to establish a new connection with the app. The value is between 1 to 60 seconds.

2. Inactivity timeout

This is the number of seconds we allow our application to do the processing and return response. You can specify between 1 to 36000 seconds.

3. Visibility timeout

This is the number of seconds to lock the message before returning to queue, and you can specify it to be between 1 to 43200 seconds. I personally feel this is the confusing part, so I'll try my best to explain.

When the daemon read a message from the SQS queue, it will send POST request to your application. If your application can process the request and return good response (e.g HTTP 200) within the Inactivity timeout, great. The whole processing is done, and the message is deleted automatically from the queue.

However, if your application failed to give response within Inactivity timeout period, the message will be sent back to SQS to be retried after Visibility timeout reached, calculated from the time your app start processing the message.

So, for example, let's say you set Inactivity timeout to be 30 seconds, and Visibility timeout to be 60 seconds. You then send a message to the queue at 10:00:00. At 10:00:30, if your application cannot give response after 30 seconds, the daemon will flag the request as failed. At 10:01:00, after 60 seconds reached, the daemon will re-throw the message to the queue and the whole process will be repeated until it reaches the Max Retries setting (default 10 times).

Now, what if your application need 45 seconds to do the heavy background processing in above example? In this case, at 10:00:00 the request will be fired and the processing starts. At 10:00:30, the daemon will flag the processing as failed, but the actual processing will still continue on the background. At 10:00:45 your app finally gives response, but no one is listening to the response. At 10:01:00, the message is back to the queue and the whole heavy processing is repeated even though it was actually success. So you will want to set the Inactivity timeout and Visibility timeout to be a safe value relative on your expected app processing time, and keep both values relatively close. The default settings from AWS put the Inactivity timeout at 299 seconds and Visibility timeout at 300 seconds, only differ by 1 second.

Another thing you need to be careful is to make sure you set up Visibility timeout higher than the Inactivity timeout. Now, consider this example:

You set Inactivity timeout at 60 seconds and Visibility timeout at 30 seconds. Your app needs 45 seconds processing time. In this scenario, when the processing time reached the seconds 30, the message will be made visible again in the queue and then automatically be consumed again by your server. Your server ended up doing double work when it is actually not necessary.

Phew! Now we should be able to config our worker environment properly to avoid timeout. But then, everything changed when Nginx configuration attack.

Nginx

If you are using Node JS for your worker environment platform, AWS will give Nginx out of the box to act as proxy between the actual client and your app. Nginx introduces another layer of timeout, which by default is 60 seconds (I think, never really checked, sorry). Now let's dive to example on how this might cause you problem.

Let's say your application needs 120 seconds to do processing, and you already set up both Visibility timeout and Inactivity timeout to some safe number, e.g 300 seconds. When a request started to be processed at 10:00:00, at 10:01:00 Nginx will spit out timeout. The daemon will pick this signal as literal error, thus will not delete the message from the queue. Meanwhile, your app will continue process the request in the background until it reached 120 seconds and shouts the response to no one since the daemon is no longer listening. As we reached the 300 seconds, the whole process is repeated again.

There is an easy way to distinguish whether your timeout is introduced by Inactivity timeout or Nginx timeout. You can grab Nginx access log from the worker environment at /var/log/nginx/access.log. Then, you can inspect the HTTP response status from the request processed by your app. If the status is 499, it means your app hit Inactivity timeout. However, if the HTTP response is 504, then there's good chance your app hit Nginx timeout limit.

Alright, so, we also need to extend the Nginx timeout. However, extending Nginx timeout is not as easy as the other timeouts since you cannot do it from AWS Console. You can SSH to your worker server instance and change the config directly, but in this case, every time your server rebuild you will need to apply the config changes manually. There is a better method.

You can add a folder in your application source code and name it as .ebextensions. You need to name it as is and are not able to name the folder with other names. Inside the folder, add a file and name it nginx_timeout.config. You can name the file however you want, but it's better to have a descriptive name. One important point about the file naming is, the file must end with .config extension. Inside that file, you can simply straight copy paste this:

files:
  /etc/nginx/conf.d/timeout.conf:
    mode: "000644"
    owner: root
    group: root
    content: |
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
      proxy_read_timeout 300;
      send_timeout 300;

What it does is simply that it will create a file /etc/nginx/conf.d/timeout.conf which will be automatically read by Nginx default standard config. That file states that various Nginx timeout value should be 300 instead of the whatever the default values from standard config, and that's it! Now our worker environment should be good to go digest all heavy processing you feed without hitting timeouts.