DEV Community: Aleksy Bohdziul

Partial Catchment Delineation for Inundation Modeling

Aleksy Bohdziul — Thu, 19 Feb 2026 07:34:53 +0000

A System-Oriented Approach to Scalable Hydrological Feature Engineering

Traditional watershed delineation wasn't designed for machine learning at scale. The standard approach treats each watershed as a complete, self-contained unit, which makes sense when you're studying individual rivers. But it creates real problems when you need to train models across hundreds of locations.

We ran into this on a recent flood prediction project. The study area had almost no hydrological data, no stream gauges, no riverbed surveys, nothing you'd typically use for hydraulic modeling. So we trained an LSTM model to predict discharge instead. It worked, but there was a catch: the model could only predict flow at watershed outlets. One outlet per watershed meant we didn't have enough training points.

The obvious solution was to create more outlets by subdividing watersheds along the river. But traditional catchment boundaries overlap heavily when you do this, which breaks parallel processing and makes selective updates nearly impossible. We needed many independent spatial units for the ML model, but we also needed to preserve the downstream flow aggregation that hydrological modeling depends on.

Our solution was to delineate watersheds at the reach level instead. Each reach gets its own partial catchment, smaller units that remain hydrologically valid but can be computed and updated independently. It's a compromise between what the data pipeline needs and what the hydrology requires.

Getting the geometry right was only part of the problem. We tested multiple DEM sources, modified the river network repeatedly, and needed to recompute catchments constantly during development. Treating delineation as a one-time preprocessing step wasn't viable. We needed to version every intermediate result, recompute selectively when inputs changed, and compare outputs across iterations.

This pushed us toward Dagster's asset model. Instead of treating catchments as temporary pipeline outputs, we manage them as persistent spatial assets with explicit dependencies and lineage tracking.

The following sections cover the hydrological rationale, the technical implementation, and how asset orchestration made this approach practical for production use.

Catchment Delineation in the Context of Inundation Modeling

Inundation modeling relies on an accurate representation of how water accumulates and propagates through a river network. Traditionally, this begins with watershed delineation derived from a digital elevation model, followed by hydraulic simulation over the resulting domain. When applied to large regions, however, this workflow introduces practical limitations. Entire catchments must be processed as single units, even when only a small portion of the river network is relevant for a given prediction or model update.

From a data engineering perspective, this creates an undesirable coupling between upstream and downstream regions. A change in DEM preprocessing, stream burning strategy, or river vector alignment forces recomputation of large spatial extents, even when the change is localized. This coupling becomes a bottleneck when experimenting with multiple configurations or when operating a system that must adapt continuously to new data.

The core insight behind partial catchment delineation is that hydraulic dependency flows downstream, but computational dependency does not need to. By separating catchments into smaller, reach-aligned units, it becomes possible to preserve hydrological correctness while dramatically improving computational flexibility.

The Core Idea: Reach-Level and Progressive Catchments

We segmented each river into short reaches (100 m) and delineated a catchment for each reach. From those building blocks we constructed larger, progressively downstream catchments.

The method introduced here distinguishes between two complementary spatial constructs: reach-level catchments and progressive catchments.

Reach Catchments (non-overlapping)

Reach-level catchments are defined for individual river segments, bounded upstream by the nearest confluence and downstream by the segment's outlet. These units do not overlap and collectively partition the drainage area of the river network. Their non-overlapping nature makes them well suited for parallel processing, independent feature extraction, and localized recomputation.

Visualize the landscape divided into narrow, adjacent drainage areas:

Each reach catchment drains only to its own 100 m river segment
None of them include upstream contributions
Their boundaries tile the basin without overlaps

Progressive Catchments (overlapping by design)

Progressive catchments, by contrast, represent the cumulative upstream area contributing to a given river reach. Each progressive catchment is constructed by aggregating all upstream reach-level catchments along the river network. This structure mirrors traditional hydrological reasoning and provides a direct bridge to downstream hydraulic modeling.

Now start combining those catchments as you move downstream:

Progressive Catchment 1 = Reach 1
Progressive Catchment 2 = Reach 1 + Reach 2
Progressive Catchment 3 = Reach 1 + Reach 2 + Reach 3

Visually:

the first progressive catchment is small and upstream
each subsequent one contains the previous
downstream catchments fully envelop upstream ones

This is why we call them progressive: each one represents the basin area contributing flow up to that point along the river.

By maintaining both representations explicitly, the system can operate at two levels simultaneously. Reach-level catchments support scalable computation and machine learning workflows, while progressive catchments preserve the physical context required for inundation modeling.

Why not delineate progressive catchments directly?

We could have. It would actually be simpler than whatever we're actually doing. But:

We also needed the reach catchments for inundation simulation later
Running delineation logic twice felt like a smell

So we delineate once, and compose later.

Tributaries

Where a tributary joins:

its reach catchments are merged into the progressive catchment only after the confluence
upstream progressive catchments on the main stem remain unaffected

In other words:

reach catchments are spatial building blocks
progressive catchments are cumulative assemblies of those blocks

What the Two Catchment Types Are Used For

Progressive catchments → model training

Our LSTM predicts discharge at outlet points. For each progressive catchment, we derive features such as precipitation, temperature, humidity, pressure, the aridity index, and others.

All inputs are provided as raster datasets. Catchment geometries are used as spatial masks to extract and aggregate pixel values.

This workflow requires repeated spatial joins, raster masking, and temporal aggregation over large geospatial datasets. We implement and orchestrate these pipelines using Dagster, which allows us to manage dependencies, partition computations, and scale processing across large spatial-temporal datasets.

Reach catchments → inundation mapping

Each reach gets its own discharge estimate (derived from differences between progressive catchments), which later feeds the inundation simulation.

Once reach-level and progressive catchments are established, they become the foundation for feature extraction. Terrain attributes, land cover statistics, soil properties, and hydrological indices can be computed independently for each reach-level catchment. These features serve as inputs to machine learning models predicting discharge or inundation extent.

Progressive catchments then provide a natural mechanism for aggregating upstream contributions. Features derived at the reach level can be accumulated downstream in a controlled, traceable manner. This separation simplifies both training and inference: models operate on consistent, non-overlapping units, while hydraulic context is reintroduced through aggregation.

At this stage, the delineation method transitions from a GIS exercise into a data orchestration problem. Each derived feature depends on specific preprocessing choices, spatial units, and upstream dependencies. Managing these relationships manually quickly becomes infeasible.

DEM (Digital Elevation Model) Preprocessing

Implementing partial catchment delineation at high spatial resolution exposes a range of practical challenges. Reliable catchment delineation depends far more on DEM preprocessing than on the delineation algorithm itself.

High-resolution DEMs (1 m × 1 m in our case) amplify artifacts that are negligible at coarser scales, including spurious sinks, artificial barriers, and noise-induced flow paths. Stream burning and sink filling become necessary, but their parameters introduce additional degrees of freedom that affect downstream results.

Below we summarize the preprocessing steps that proved essential for stable and repeatable results.

Depression filling

Raw DEMs frequently contain spurious sinks caused by:

measurement noise
vegetation and built structures
interpolation artifacts

Left untreated, these sinks interrupt downstream connectivity and lead to fragmented or incomplete catchments. We therefore applied depression filling prior to any flow calculations.

Our goal was not to aggressively flatten terrain, but to ensure continuous drainage paths with minimal elevation modification. Priority-flood-style algorithms worked well in practice and preserved overall terrain structure.

Stream burning

Even after sink removal, we observed inconsistencies between modeled flow paths and known river locations. To address this, we burned the vector river network into the DEM by lowering elevations along river centerlines.

Aligning raster-based flow accumulation with vector river networks proved particularly sensitive. Small positional discrepancies between datasets can lead to misaligned pour points, fragmented catchments, or unrealistic drainage patterns. These issues are not purely geometric; they directly influence the stability and reproducibility of downstream features.

This step serves two purposes:

it enforces hydrologically plausible drainage paths
it reduces sensitivity to small elevation errors in flat or low-gradient terrain

Stream burning significantly improved watershed stability, especially near confluences and in wide floodplains where DEM gradients are weak.

Flow accumulation and its limitations

We initially experimented with flow accumulation to:

identify channelized flow paths
snap pour points automatically to areas of high contributing area

However, the high spatial resolution of the DEM (1 m × 1 m) introduced significant noise into flow accumulation outputs. Minor elevation perturbations resulted in fragmented or unrealistic accumulation patterns, making automated snapping unreliable.

As a result, we limited the use of flow accumulation and instead relied more heavily on burned-in river vectors and explicit reach endpoints for pour point placement.

During later experiments we found that using D-infinity for calculating flow paths rather than D8 significantly improved flow accumulation calculation, but due to it being discovered too late, we weren't able to implement it before the end of first phase of the project.

Spatial alignment issues

During development we discovered small but still significant horizontal offsets between the DEM and the river vector dataset, on the order of a few meters. At some point we discovered that our river geometries were offset by a couple of meters relative to the DEM.

These discrepancies led to:

pour points falling outside effective drainage paths
unstable catchment boundaries
inconsistent results across neighboring reaches
weird catchment boundaries
several hours of existential doubt

While stream burning mitigated some of these effects, resolving DEM-vector alignment remains an important area for future improvement. This is still on our list of things to fix properly. For now, burning rivers into DEM somewhat alleviated the issue.

Rather than attempting to eliminate these uncertainties entirely, we treated them as explicit dimensions of experimentation. Different preprocessing strategies were preserved as separate artifacts, allowing their effects to be compared systematically. This approach only becomes feasible when intermediate results are treated as first-class entities rather than overwritten pipeline outputs.

Overall, careful DEM preprocessing proved essential not only for hydrologic correctness, but also for producing geometries stable enough to support downstream machine-learning workflows.

Implementation Outline

Below is a cleaned-up, simplified sketch of the workflow. The real code is longer, louder, and contains more comments written at 2 a.m.

# 1. Load DEM and preprocess
filled_dem = fill_depressions(dem)
burned_dem = burn_streams(filled_dem, river_lines)
flow_dir   = d8_flow_direction(burned_dem)

# 2. Split rivers into fixed-length reaches
reaches = split_lines(river_lines, segment_length=100)

# 3. Create pour points at reach outlets
pour_points = reaches.geometry.apply(get_downstream_endpoint)

# 4. Delineate reach catchments
reach_catchments = delineate_watersheds(
    flow_dir=flow_dir,
    pour_points=pour_points
)

# 5. Build progressive catchments
progressive = []
current = None
for reach in ordered_downstream(reach_catchments):
    current = reach if current is None else union(current, reach)
    progressive.append(current)

The devil, as always, lives in:

tributary joins
reach ordering
and spatial indexing performance

Joining tributaries means:

identifying parent-child relationships between reaches
merging reach catchments in the correct downstream order
avoiding double-counting areas

Asset-Based Orchestration of Spatial Dependencies

To make this workflow operational, we modeled reach-level catchments, progressive catchments, and derived features as explicit assets within Dagster. Each asset represents a durable spatial artifact with well-defined dependencies on upstream inputs. Changes in DEM preprocessing, river network alignment, or feature definitions propagate through the asset graph in a controlled way.

This asset-oriented approach allows recomputation to be both selective and explainable. When a preprocessing parameter changes, only the affected reach-level catchments and their downstream aggregates are recomputed. Historical artifacts remain available for comparison, enabling systematic evaluation of alternative configurations.

Dagster's lineage tracking plays a critical role here. Each feature can be traced back through the chain of spatial transformations that produced it, providing transparency during debugging and model validation. Rather than reasoning about pipeline execution order, the system reasons about data state and dependency.

Operational Implications

Treating partial catchment delineation as an orchestrated asset graph changes the operational profile of inundation modeling workflows. Iteration becomes cheaper because recomputation is localized. Failures become easier to diagnose because dependencies are explicit. Experimentation becomes safer because previous states are preserved rather than overwritten.

Perhaps most importantly, this approach aligns hydrological reasoning with modern data platform design. Physical dependencies are respected, but they no longer dictate computational coupling. The system can evolve incrementally, accommodating new data sources, preprocessing strategies, and modeling approaches without requiring full recomputation of the spatial domain.

Lessons Learned

DEM preprocessing is very important
1 m DEMs are great until you compute derivatives
River vectors and DEMs rarely agree — believe neither blindly

Conclusion

Segmenting rivers into reach-level catchments gave us:

more training points
spatially consistent features
and a clean bridge between ML discharge prediction and inundation modeling

Partial catchment delineation proved valuable not because it produced a single optimal representation of a watershed, but because it enabled a shift in how spatial dependencies are managed at scale. By decomposing watersheds into reach-level units and reconstructing downstream context through progressive aggregation, we gained a representation that supports both hydrological correctness and computational scalability.

The effectiveness of this approach ultimately depended on its orchestration. Without an asset-oriented framework, the complexity introduced by multiple delineation strategies and iterative experimentation would quickly become unmanageable. By modeling spatial artifacts explicitly and preserving their lineage, we were able to integrate hydrology, machine learning, and geospatial preprocessing into a coherent, production-ready system.

If nothing else, this workflow taught us humility, patience, and how many ways water can refuse to flow downhill.

While this article focused on inundation modeling, the underlying pattern extends to any domain where high-resolution geospatial data meets iterative, data-driven workflows. Partial decomposition of space, combined with asset-based orchestration, offers a practical path toward scalable and trustworthy spatial modeling systems.

How to Debug a Node.js App on AWS ECS Fargate Using Port Forwarding (Step-by-Step Guide)

Aleksy Bohdziul — Tue, 18 Nov 2025 23:00:00 +0000

At some point in their lives, every software engineer eventually faces the task of debugging an app live on a remote server. If that app happens to be running on ECS Fargate, getting into that container safely is possible, but not immediately obvious.

Full disclosure: I haven’t done this exact thing before, but I have used the same port-forwarding trick to peek into our RDS instances via an ECS task acting as a jumpbox. So yes, I tested these instructions, and yes, they actually work.

The Setup

Here’s what we have:

A Node.js app deployed to ECS Fargate
Inside a private VPC subnet
Without SSH access (Fargate doesn’t do that)
VS Code (or some other IDE with Node.js inspector support)

Step 1: Enable Node.js inspector

First, make sure your ECS task’s running Node.js process with the inspector enabled. This can be achieved by passing a flag to the node command.

node --inspect=127.0.0.1:9229 server.js

Or by adding the inspect flag to the NODE_OPTIONS ****environmental variable.

NODE_OPTIONS="--inspect=127.0.0.1:9229"

Since we’re using ECS, adding flag to the node command would require modifying Dockerfile or overriding command in the task definition, so I’d suggest the environmental variable approach.

{
  "containerDefinitions": [
    {
      "name": "...",
      "image": "...",
      "essential": true,
      "environment": [
        {
          "name": "NODE_OPTIONS",
          "value": "--inspect=127.0.0.1:9229"
        }
      ] 
    }
  ]
}

Step 2: Enable ECS Exec and SSM Access

Before you can connect, your ECS task needs permission and execution enabled:

Attach this IAM policy to your ECS task execution role:

arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Enable ECS Exec on your service:

aws ecs update-service \
  --cluster my-cluster \
  --service my-service \
  --enable-execute-command

If you’re changing it through the console, it’s hidden under Troubleshooting → Enable Execute Command.

Step 3: Find Your Running Task

List your tasks:

aws ecs list-tasks --cluster my-cluster

You’ll get something like:

{
  "taskArns": [
    "arn:aws:ecs:us-east-1:123456789012:task/my-cluster/abc123def456ghi789"
  ]
}

The last part of the ARN (abc123def456ghi789) is your Task ID.

Step 4: Set Up Port Forwarding via SSM

Here’s where the real black magic starts. AWS won’t just give you a SSM target ID for the Fargate task, you have to construct it yourself using this template:

ecs:<cluster_name>_<task_id>_<container_runtime_id>

Get the container runtime ID:

aws ecs describe-tasks \
 --cluster my-cluster \
 --task <task_arn> \
 --query "tasks[].containers"

You’ll get a massive JSON blob, and somewhere in that mess hides the runtimeId field you actually care about. There might be a few of them actually, one for each container in the task. And yes, the runtimeId contains task id in it.

Then start the port forwarding:

aws ssm start-session \
 --target "ecs:my-cluster_cb39a47ef2ef45f4b947236bf00aeadd_cb39a47ef2ef45f4b947236bf00aeadd-3935363592" \
 --document-name AWS-StartPortForwardingSessionToRemoteHost \
 --parameters '{"host":["127.0.0.1"],"portNumber":["9229"],"localPortNumber":["9229"]}'

This opens a local port (9229) and forwards traffic securely to the Node.js process in your container.

💡 Pro tip: We’re forwarding to 127.0.0.1 here, but you can forward to any IP or hostname accessible from the ECS task.

Step 5: Connect VS Code Debugger

In .vscode/launch.json:

{
  "type": "node",
  "request": "attach",
  "name": "Attach to Fargate",
  "address": "localhost",
  "port": 9229,
  "localRoot": "${workspaceFolder}",
  "remoteRoot": "/usr/src/app"
}

And that’s all.

If you don’t know the remoteRoot , then you can get into the container and look around to check it.

aws ecs execute-command --cluster my-cluster \
 --task <task_arn> \
 --interactive \
 --command "/bin/sh" \
 --container <container name>

Alternative: Chrome debugger (chrome://inspect). Usually works, but tends to have issues with source maps.

TL;DR Cheat Sheet

# 1. Make sure your Node.js app runs with --inspect flag
NODE_OPTIONS="--inspect=127.0.0.1:9229"

# 2. Give ECS task SSM permissions
# 3. Enable ECS Exec
aws ecs update-service —service my-service —cluster my-cluster —enable-execute-command

# 4. Get you task arn
aws ecs list-tasks --cluster my-cluster

# 5. Get your container runtime id
aws ecs describe-tasks \
 --cluster <cluster> \
 --region us-east-2 \
 --task <task_arn> \
 --query "tasks[].containers"

# 6. Start port forwarding
aws ssm start-session \
 --target ecs:<cluster>_<task>_<runtime_id> \
 --document-name AWS-StartPortForwardingSessionToRemoteHost \
 --parameters '{"host":["127.0.0.1"],"portNumber":["9229"],"localPortNumber":["9229"]}'

Then attach your VS Code debugger to localhost:9229 and voila.

How to Securely Connect to Medusa.js Production Database on AWS?

Aleksy Bohdziul — Fri, 29 Aug 2025 11:00:54 +0000

Let’s imagine something for a second.

You're minding your own business, managing AWS infrastructure for a client with a pretty standard e-commerce setup: a Medusa.js backend, a Next.js storefront, and most importantly for this story, a PostgreSQL RDS instance safely stashed away in a private subnet where nothing from the outside world can touch it. Exactly how the AWS gods intended.

Then, one day, your client says:

"Hey, I need to get access to the production database"

Now, there are plenty of legit reasons to want this kind of access: analytics, dashboards, audits, maybe some light database spelunking. In this case, it’s for Metabase, which as far as you can tell, magically turns SQL into colourful charts.

So sure, let's help them out. You're a helpful DevOps engineer. You write Terraform. You breathe YAML. You’ve stared into the void of broken networking configs and lived to tell the tale. This? This is doable.

The only question is: how do we do it securely, without slapping a public IP on the database and calling it a day?

That’s exactly what this post is about: how to securely connect to a Medusa.js production database on AWS, without compromising your infrastructure or your sleep.

The Problem

We’ve got:

A managed RDS PostgreSQL database (though this applies to pretty much any RDS engine)
A Metabase instance living somewhere outside the VPC
A need to connect Metabase to the database

Sounds simple enough, but of course it's not just a one-click "make it public" button.

But wait, some of you might be asking: what's actually stopping us from "just" connecting to the database? Well, our database sits in a private subnet. Which means... (flips through AWS docs), it doesn’t have a route to an internet gateway. More importantly, it doesn’t have a public IP address. It’s only reachable via its private IP from within the VPC. Oh, there is also the security group, which allows access only from the Medusa backend.

Now, in theory, I could completely disregard all the security concerns, toss the database into a public subnet, give it a public IP, and call it a day. But I'd probably also get tossed out of a job, and fairly so.

So, making the database publicly accessible is off the table. That leaves us with one goal: somehow access that private IP from the outside. Luckily (or unluckily), there are a few ways to make that happen.

So, what are our options?

Option 1: Port Forwarding

The first and most common approach is good old port forwarding, which is basically asking a server inside your private network to kindly act as a middleman and pass your packets along to the database, like a helpful bouncer who also does package delivery

To make this work, we need what's called a jump box (or bastion host if you're feeling fancy). This is just a plain old EC2 instance living in your VPC with access to the private subnet and your database (Don’t forget to update your RDS security group to allow traffic from the jump box).

Here’s a minimal Terraform snippet to spin one up:

resource "aws_instance" "jump_box" {
  ami           = "ami-0abcdef1234567890" # Replace with latest Amazon Linux 2 AMI
  instance_type = "t3.nano"
  subnet_id = aws_subnet.private_subnet.id
  vpc_security_group_ids = [aws_security_group.jump_box_sg.id]
  key_name = "your-ssh-key"
  tags = {
    Name = "jump-box"
  }
}

resource "aws_security_group" "jump_box_sg" {
  name = "jump-box-sg"
  description = "Allow SSH"
  vpc_id = aws_vpc.main.id
  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = ["your-ip-address/32"]
  }
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Now that we’ve got our jump box, let’s explore how we can tunnel traffic through it.
For that, we can use the classic way or the AWS way.

Option 1.1: The Classic Way - SSH Port Forwarding

This is the method your senior Linux admin probably used in 2009, and honestly, it still works just fine.

You just need to move your EC2 instance to a public subnet, give it an Elastic IP, SSH key (if you’re using terraform, check the parameter key_name of aws_instance), and make sure the port 22 is open (preferably make it accessible only to specific IP addresses).

Assuming you’ve got your jump box set up and your private key in hand, here’s the command:

ssh -i ~/.ssh/your-key.pem \
  -N -L 5430:your-db.xxxxx.rds.amazonaws.com:5432 \
  ec2-user@your-jumpbox-public-ip

This sets up a tunnel from localhost:5430 → your RDS:5432. Just point your client at localhost:5430 and you’re good. (You can change the local port if you need to)

Most SQL tools support built-in SSH tunneling if you don’t want to set up the tunnel manually.

Pros:

It just works
It’s compatible with most solutions which need to connect to SQL database

Cons:

EC2 instance needs patching and monitoring
You’re exposing an SSH port (even if restricted)
You’ll start getting unsolicited connections attempts the moment you open the port
You have to manage SSH keys (on AWS it’s not that big of a deal)

This is the second option we chose and well, it just worked.

Option 1.2: The AWS Way - SSM Port Forwarding

Ah, good old SSM. In theory, this is the cleaner option. No public access, no open ports, just straight up AWS magic. You enable SSM, and then use “aws ssm start-session” to port-forward to the database.

Prerequisites:

SSM Agent installed on your EC2 instance (Amazon Linux already has it preinstalled)
AWS CLI with SSM plugin installed
AmazonSSMManagedInstanceCore policy attached to your jump box
ssm:StartSession and ssm:DescribeInstanceInformation permissions for your account

Once you have all of that, you can just run the following command:

aws ssm start-session \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"host":["your-db.xxxxx.rds.amazonaws.com"], "portNumber":["5432"],"localPortNumber":["5430"]}'

Like the previous options, this will open a tunnel allowing you to connect to the database on localhost:5430.

Pros:

No public IPs or open ports
No SSH keys to manage
IAM-based access
You can audit access in the AWS CloudTrail
Feels like you’re doing something correct

Cons:

Requires AWS CLI
It probably doesn’t actually solve your problem, since most solutions don’t support this.

Here’s the thing, while port forwarding with SSM is nice for dev and devops access, if you need something like a Metabase to connect to your database then SSM won’t help you.

So yes, we tried this first, and just after that found out that the access is needed for Metabase.

Option 2: VPN

In case you somehow don’t know, a VPN (Virtual Private Network) is basically a magic tunnel that lets machines outside your VPC pretend they’re inside it. Once connected, your laptop can access your internal resources, as if they were part of your precious private subnet all along.

This one’s probably overkill for most use cases like ours, but hey, it exists. You can spin up a VPN (e.g. AWS Client VPN or a WireGuard setup) and let your client connect to your internal network that way. Great if you already have a VPN setup, but if you don’t, then do you really want to?

There are a few flavors here:

AWS Client VPN: The “I want AWS to hold my hand” option.
Roll-your-own VPN (OpenVPN, WireGuard): AKA “I like pain.”
Third-party VPNs: Where you pay to inflict the pain on someone else.

We’ll use AWS Client VPN because we’re not trying to impress anyone, we’re just trying to get this over with before lunch.

resource "aws_ec2_client_vpn_endpoint" "example" {
  description = "Client VPN for private RDS access"
  client_cidr_block = "10.0.10.0/22"
  server_certificate_arn = "arn:aws:acm:your-cert-arn"
  authentication_options {
    type = "certificate-authentication"
    root_certificate_chain_arn = "arn:aws:acm:your-root-ca-arn"
  }
  connection_log_options {
    enabled = false
  }
  split_tunnel = true
  vpc_id = aws_vpc.main.id
  dns_servers = ["8.8.8.8"] # shrug
}

resource "aws_ec2_client_vpn_network_association" "example" {
  client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.example.id
  subnet_id = aws_subnet.private_subnet.id
}

resource "aws_ec2_client_vpn_authorization_rule" "example" {
  client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.example.id
  target_network_cidr = "10.0.0.0/16"
  authorize_all_groups = true
}

Oh right, you’ll need to create certificates. With ACM. Or OpenSSL. Or just write them by hand. Whichever works.

What this actually does:

Spins up a VPN endpoint (as if we needed more resources to take care of).
Associates it with your VPC so users can crawl around your private subnets.
Lets anyone who can connect access your RDS instance as if they were in the same network.

Pros:

Enterprise-y!
Secure and scalable
Useful for broader access needs
Works great if you already have a VPN setup (we don’t)

Cons:

You have to use the VPN
Doesn’t work with Metabase
Certs, IAM, CIDR blocks, DNS resolution issues
Can very easily get very expensive

Bonus Consideration: Read Replicas

If your client is doing heavy analytical workloads, consider offloading queries to an RDS read replica. That way, your production DB doesn’t get overwhelmed by analytical queries, and you get some nice isolation between transactional and analytical use.

Keep in mind:

Read replicas can lag behind the primary DB
You still need to expose the replica through one of the methods above

But it’s a nice option if performance and reliability matter.

Wrapping Up

So, your client wants access to his Medusa.js database. It’s a valid ask, but you still want to do it responsibly.

Here’s the quick TLDR:

Use SSH port forwarding if you need quick, compatible access from outside (e.g. Metabase).
Use SSM for internal-only dev/admin access, just don’t expect it to work with any tool like Metabase.
VPN if you’re going full enterprise or need multi-service access.
Add Read replicas if query load is a concern.

And above all: don’t just throw the DB in a public subnet. That way lies sadness, audit findings, and probably a very uncomfortable meeting.

Speed Up Your Next.js App: Optimizing S3 Images with Cloudflare Images

Aleksy Bohdziul — Wed, 09 Jul 2025 07:00:00 +0000

Let’s talk about something near and dear to every developer’s heart: making stuff faster without tearing your hair out.

So you’ve got your images chilling in an S3 bucket — classic move. But if you’re serving them straight from there, you’re probably pushing chunky, uncompressed images across the internet like it’s 2008. On the flip side, maybe you’re leaning on Next.js’s built-in image optimization… which is cool until your server starts wheezing under the load like it just ran a marathon.

But don’t worry — there’s a better way. In this post, we’ll walk through a dead-simple setup using Cloudflare Images to optimize and cache your images right at the edge, leaving your compute power untouched and your pages blazing fast.

What You’ll Need

Before we dive in, make sure you’ve got the basics:

An AWS account (for your S3 bucket full of glorious JPEGs). If you're using something else for file storage, that’s totally fine too — we don’t judge.
A Cloudflare account — free tier works great.
A domain hooked up to Cloudflare.
And since this post is all about Next.js, we’ll assume you’ve already got that part set up and humming along. If not… this might be a weird place to start.

Setup Guide: The Part You’re Actually Here For

Time to get our hands slightly dirty (but like… not real dirty — this is all pretty painless). Let’s hook everything up so Cloudflare can work its optimization magic on your S3 images.

1. Set Up Your Bucket

If you already have an S3 bucket full of images, awesome — you’re halfway there. If not, go ahead and create one. Give it a nice name, something you won’t be embarrassed about later — you’ll be seeing it a lot.

Now the important part: Cloudflare Images needs to be able to access your images. If you’re already serving images from S3, you’ve probably handled this with either public access or presigned URLs. If not, here’s the quick-and-dirty setup:

Make the bucket public (yes, really)

Go to your bucket permissions
Make sure “Block public access” is turned **off**

Then slap this bucket policy on there to allow public reads (it’s just below “Block public access”:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-bucket-name>/*"
        }
    ]
}

Don’t forget to swap in your actual bucket name, unless you want Cloudflare to look for and come back very confused.

That’s it — your images are now ready to be fetched and optimized like champs.

2. Next stop: Cloudflare land (a.k.a. Where the Magic Happens)

Head to your Cloudflare dashboard, choose your domain, and go to the Images → Transformations tab. Click the little “Enable” toggle next to your domain.

Still on the same page, click your domain to open the settings. Under Sources, select Specified Origins and add your S3 bucket’s public domain (like https://your-bucket.s3.eu-west-1.amazonaws.com). You can select “Any origin” here… but unless you enjoy living dangerously, stick with the specified option.

3. Cloudflare Image Format — Fancy URLs That Do All the Work

Cloudflare’s image transformation magic works through simple (and surprisingly readable) URLs. The format looks like this:

https://<your-domain>/cdn-cgi/image/width=<width>,quality=<quality>,format=<format>/<S3-image-url>

Let’s break that down:

width – how wide you want the image in pixels (e.g., 800)
quality – how much quality to keep (1 to 100 — 75 is a solid sweet spot)
format – choose from auto, avif, webp, or jpeg. Just go with auto unless you’re feeling experimental
S3-image-url – the full URL to the image in your bucket, like:

https://bucket-name.s3.eu-west-1.amazonaws.com/awesome-cat.jpg

Here’s a complete example:

https://your-domain.com/cdn-cgi/image/width=800,quality=75,format=auto/https://bucket-name.s3.eu-west-1.amazonaws.com/awesome-cat.jpg

Cloudflare does all the heavy lifting — resizing, compressing, and even converting to modern formats. Your users get fast, lightweight images, and your server gets to nap.

Want to get fancy? There are more transformation options here, but width, quality, and format cover 90% of what you’ll need.

4. Plugging It Into Next.js

Alright, time to wire this into your Next.js app. I’m going to assume you’re using the <Image> component from next/image — because if you’re not, now’s a great time to start.

Option A: The Quick and Dirty Way

The quick-and-dirty method: just feed the transformed Cloudflare URL directly into the component.

import Image from "next/image"

<Image
  src="https://your-domain.com/cdn-cgi/image/width=800,format=auto/https://bucket-name.s3.eu-west-1.amazonaws.com/cat-wearing-a-crown.jpg"
  width={800}
  height={500}
  alt="A regal cat wearing a golden crown"
/>

Done. It works. Your cat is now optimized.

Option B: The Cleaner, More Reusable Way (Custom Loader)

If you want to keep your code tidy and avoid sprinkling Cloudflare URLs everywhere, you can roll a custom image loader.

First, update your next.config.js:

module.exports = {
  images: {
    loaderFile: "./cloudflare-loader.ts"
  },
}

And if you’re using remote images, you’ll probably need to add a remote pattern too:

remotePatterns: [
  {
    protocol: "https",
    hostname: "bucket-name.s3.eu-west-1.amazonaws.com",
  },
]

Then, create a new file called cloudflare-loader.ts in your project root:

import { ImageLoaderProps } from "next/image"

const BUCKET_DOMAIN = "bucket-name.s3.eu-west-1.amazonaws.com";
const CLOUDFLARE_DOMAIN = "<your domain>";

export default function cloudflareImageLoader({
  src,
  width,
  quality = 75,
}: ImageLoaderProps): string {
  try {
    if (!src.startsWith("/")) {
      const parsedSrc = new URL(src)
      if (parsedSrc.hostname === BUCKET_DOMAIN) {
        const params = [`width=${width}`, `quality=${quality}`, "format=auto"]
        return `${CLOUDFLARE_DOMAIN}/cdn-cgi/image/${params.join(",")}/${src}`
      }
    }
  } catch (error) {
    console.error("Cloudflare image loader error:", error)
  }
  // Fallback to Next.js built-in image optimization
  return `/_next/image?url=${encodeURIComponent(src)}&w=${width}&q=${quality}`
}

This setup:

Applies Cloudflare optimization only to images from your S3 bucket
Falls back to Next.js optimization for everything else
Keeps your JSX clean and your dev brain happy

Result? A Happier, Faster App

Now you’ve got:

On-the-fly image optimization
CDN delivery from Cloudflare’s edge nodes
No need to pre-process or move your images
Better scores in Lighthouse, Core Web Vitals, and all that jazz

Wrapping Up

Optimizing images is one of the lowest-effort, highest-impact improvements you can make to a modern web app. And thanks to Cloudflare Images + your trusty old S3 bucket, you don’t need to rearchitect your whole stack to do it.

So go ahead, optimize those cat pictures. Your users — and their data plans — will thank you.