DEV Community: Kiran Gunturu

AWS Glue Concurrency Is Capped by One Subnet, Not Your Whole VPC

Kiran Gunturu — Sun, 11 Oct 2026 18:24:36 +0000

TL;DR

If you run many AWS Glue jobs concurrently inside a VPC, your concurrency
ceiling is the number of free IPs in a single subnet, not the sum of
free IPs across your VPC. Attaching multiple Glue connections does not
pool IPs across subnets; it only provides failover. I proved this with a
series of controlled tests, and the fix is boring but effective: size a
dedicated subnet per workload to workers_per_job x max_concurrency.

The setup

I work on a regulated data platform where several products share one
VPC. Each product runs a set of Glue ETL jobs. I started hitting failures
under concurrency with errors like:

The specified subnet does not have enough free addresses to satisfy the
request. Please provide a connection with a subnet with ips available.

On paper I had plenty of headroom, ~96 free IPs across six subnets. So
why were jobs failing at 8-10 concurrent runs? The answer is in how Glue
consumes IPs.

The mental model: ENIs = workers = IPs

When a Glue job connects to VPC resources, Glue creates Elastic Network
Interfaces (ENIs) in your subnet. The number of ENIs tracks the number of
workers (DPUs) for the job, and each ENI consumes one private IP in the
subnet.

So a single job with 4 workers ~= 4 IPs. Ten such jobs running at once ~= 40
IPs, all drawn from one subnet. That's the part that surprises people.

The myth: "multiple connections give me more IPs"

A Glue job can have multiple connections attached, and each connection can
point to a different subnet. It's tempting to assume Glue will spread ENIs
across those subnets to pool IP capacity. It won't.

Here's what actually happens:

Glue evaluates connections in the order they're listed and picks the first suitable one. All ENIs for that run are created in that one subnet, in a single AZ.
Additional connections are failover only: Glue rolls over to the next connection if the first has a networking/AZ problem. It is not load-balancing and not high availability.
Multiple connections to the same subnet with different DB URLs just means "reach different databases," not "get more IPs."
Glue does not switch connections dynamically mid-run to chase free IPs.

Net effect: your effective IP budget is the free-IP count of whichever single
subnet Glue lands on, usually your largest one.

The experiments

I ran a controlled matrix to pin down the behavior. Subnet labels are
generic; the numbers are representative.

#	Jobs submitted	Subnet(s)	Available IPs	Workers/job	Glue ver	IPs used	Result
1	3	Subnet-A	26	4	3.0	12	Passed
2	3	Subnet-A	26	4, 3, 2	3.0	9	Passed
3	3	Subnet-A	26	max 4 (autoscale)	3.0	12	Passed
4	1	Subnet-A	26	4	4.0	4	Passed
5	8	Subnet-A + B	52 (26+26)	4	3.0	26	6 succeeded, 2 failed
6	10	Subnet-A + B	36	max 4 (autoscale)	3.0	27	All succeeded

The two multi-subnet tests are the interesting ones:

Test 5: Two subnets attached (26 free IPs each). Eight jobs at 4 workers need 32 IPs. Only the first subnet was used: it holds 26 IPs, so 6 jobs got their 24 IPs and the remaining 2 failed for lack of addresses. Zero IPs were drawn from the second subnet.
Test 6: Ten jobs with auto-scaling enabled. All succeeded, but every ENI still came from a single subnet. Auto-scaling let the jobs fit into fewer IPs; it did not spread them across subnets.

What I observed

A Glue job can attach multiple connections; each points to a subnet.
Glue uses exactly one connection/subnet per job run.
Connection selection is order-based, with rollover only on failure: it's failover, not balancing.
Multiple connections to the same subnet = access to different databases, not more IPs.
Multiple connections exist for fault tolerance/failover, never for HA or IP pooling.
With auto-scaling on, N concurrent jobs all land in the single subnet of the first working connection.
With auto-scaling off, jobs beyond the subnet's IP budget fail outright.
Partial free IPs are not shared across subnets.
Therefore, spreading IPs thinly across many subnets buys you nothing: Glue won't switch subnets to use them.

The capacity math

Plan per single subnet, not per VPC:

IPs needed  ~=  workers_per_job x max_concurrent_jobs

Then remember AWS reserves 5 IPs in every subnet, and leave a little ENI
headroom. Worked examples:

10 concurrent x 4 workers = 40 IPs in one subnet. A /27 (32 total, ~27 usable) can't do it. You need at least a /26 (64 total, ~59 usable).
5 concurrent x 4 workers = 20 IPs. A /27 (~27 usable) is comfortable.

My original layout, six small subnets (/27s and /28s) totaling ~96 free
IPs, looked generous but capped me at ~26 concurrent-worker-IPs because Glue
only ever touched one /27.

What I changed

One dedicated subnet per workload, sized to that workload's peak workers x concurrency. Workload A gets its own /27; the heavier Workload B gets a larger block.
Stopped treating multiple connections as capacity. They're failover. If I genuinely need cross-VPC reachability, that's VPC peering, which is about reachability, still not IP pooling.
Used auto-scaling deliberately. It reduces IP pressure but adds latency for large table ingests, so it's a conscious trade-off, not a free win.
Evaluated Glue job queuing. AWS's built-in job queuing automatically handles concurrency limits and retries jobs instead of failing them immediately, worth enabling if bursty submission is your pattern.

Key takeaways

Glue concurrency is bounded by one subnet's free IPs, full stop.
ENIs = workers = IPs. Budget accordingly.
Multiple connections = failover, not more capacity.
Right-size a dedicated subnet per workload; don't scatter IPs across many small subnets and expect Glue to pool them.

The fastest Glue job is the one you never start: removing per-job overhead from a 191-job pipeline

Kiran Gunturu — Sun, 04 Oct 2026 15:15:50 +0000

In my last post I covered making a production AWS Glue pipeline run faster by tuning individual jobs. This post is about the bigger change that followed. It had nothing to do with how the jobs ran. It was about how many times we started them.

All costs below use Glue's standard rate of $0.44 per DPU-hour, billed per second with a 1-minute minimum per job run (Glue 2.0+).

The pipeline

It's an Oracle-to-vendor feed with five layers, orchestrated by Step Functions:

Layer	Glue jobs	Worker config	What it does
Ingestion	63	G.1X × 4 (4 DPU)	Oracle → S3, one job per source table
Transformation	63	G.1X × 4 (4 DPU)	Business transforms
Publish	63	G.1X × 4 (4 DPU)	Write final output
Outbound	1	G.2X × 6 (12 DPU)	Package into ~5 GB split files, zip, encrypt
SFTP	1	Python Shell, 1 DPU	Ship to the vendor

Step Functions runs the layers in order, and each layer fans its jobs out through a Map state. Running daily, that's 191 Glue job starts every day.

Tuning helped, up to a point

The optimizations from the last post made a real difference:

Stage	Before (min)	After (min)	Change
Ingestion	23.38	6.38	−73%
Transform	10.02	5.43	−46%
Publish	25.23	5.15	−80%
Outbound	8.00	5.55	−31%
SFTP	5.02	5.54	noise (bandwidth-bound)
End to end	1h 15m 13s	29.3 min	−61%

26M records, 1.3 GB zip.

What that did to cost

Wall-clock time and cost aren't the same thing here. A layer's wall clock is roughly its slowest job, but you pay for every job in the layer. A single minute of wall clock on a 63-job G.1X × 4 layer can bill up to 63 × 4 = 252 DPU-minutes.

The estimates below treat every job in a layer as running for the layer's full duration. That makes them an upper bound: jobs that finish early cost less, so actual spend is lower. But the same method applies to every row, so the comparisons hold.

Stage	DPU-hours before	DPU-hours after	Cost / run before	Cost / run after
Ingestion	98.2	26.8	$43.21	$11.79
Transform	42.1	22.8	$18.52	$10.04
Publish	106.0	21.6	$46.63	$9.52
Outbound	1.6	1.1	$0.70	$0.49
SFTP	1.0	0.1	$0.44	$0.04
Total	248.9	72.4	$109.49	$31.87

SFTP ran on G.2X × 6 before being moved to a 1-DPU Python Shell job.

Estimated cost per run fell by about 71%, more than runtime did. That's because nearly all of the cost sits in the three 63-job layers, and those improved the most. Outbound and SFTP are single jobs, so even their slow minutes are cheap.

The run went from over an hour to under 30 minutes. But while going through the run history, I found two days that showed where the rest of the time was going.

The run that processed almost nothing

On two days the source sent almost no data: 5,822 records, a 75 KB file.

Those runs still took 25 and 27 minutes. Ingestion alone took 5 minutes to read a few thousand rows across 63 jobs. Nearly all of that time was spent before any data was touched.

Every Glue job pays the same fixed cost before it does any work:

flowchart LR
    subgraph FIXED["Fixed cost: paid on every job start, regardless of data volume"]
        A["Provision workers"] --> B["Install libraries"]
        B --> C["Start Spark context"]
        C --> D["Open JDBC connection"]
    end
    D --> E["Read and write data<br/>scales with volume"]

That cost doesn't depend on data volume. A job reading 5,000 rows pays the same startup as a job reading 5 million. Glue 2.0+ also bills a 1-minute minimum per job run, so even a very short job costs a full minute of DPU time.

What the empty run cost

Run	Records	Runtime	DPU-hours	Cost
Near-empty day	5,822	25.3 min	73.3	$32.23
Normal day	26M	1h 15m	248.9	$109.49

Both runs are pre-tuning, estimated the same way as above.

The near-empty run cost about 29% of a full run while processing 0.02% of the data. That 29% is roughly what we paid every day just to start 191 jobs.

We were paying that startup cost 191 times a day. Tuning had reduced the time spent on actual work, which only made the fixed overhead a bigger share of what was left. Further tuning wasn't going to change that, because the overhead comes from starting the job, not from the work it does.

Removing the overhead: start fewer jobs

The downstream consumer didn't need the feed every day. So instead of making each job start faster, we reduced how often the jobs start.

We made the cadence configurable: daily, weekly, or monthly. A weekly or monthly run processes the whole period's data in one pass through the same pipeline.

While building this, we took a closer look at the transformation layer. The architecture called for it as a separate stage, but in practice its 63 jobs weren't applying any business transformations. They read the ingested data and passed it through to publish unchanged. That meant 63 extra rounds of provisioning, library installs, and Spark startup every run, just to move data from one S3 prefix to another. In the daily run that layer still took 5.4 minutes. So weekly and monthly runs skip it, and each one starts 128 jobs instead of 191.

flowchart LR
    SF(["Step Functions"]) --> ING["Ingestion<br/>63 Glue jobs<br/>Oracle → S3"]
    ING --> CAD{"Cadence?"}
    CAD -->|"Daily"| TR["Transform<br/>63 Glue jobs<br/>pass-through only"]
    TR --> PUB["Publish<br/>63 Glue jobs"]
    CAD -->|"Weekly / Monthly<br/>skip transform"| PUB
    PUB --> OUT["Outbound<br/>1 Glue job<br/>~5 GB splits, zip, encrypt"]
    OUT --> SFTP["SFTP<br/>1 Glue job"]
    SFTP --> V(["Vendor"])

    classDef removed fill:#eeeeee,stroke:#999999,stroke-dasharray:5 5,color:#666666
    class TR removed

Here's what that does to the number of job starts:

Cadence	Jobs / run	Glue job starts / month	Glue job starts / year
Daily	191	5,921	69,715
Weekly	128	~567	~6,674
Monthly	128	128	1,536

xychart-beta
    title "Glue job starts per month"
    x-axis ["Daily", "Weekly", "Monthly"]
    y-axis "Job starts" 0 --> 6500
    bar [5921, 567, 128]

Weekly means ~90% fewer job starts than daily. Monthly means ~98% fewer. Every one of those removed starts is a round of provisioning, library installs, and Spark startup that no longer happens.

The billing floor of a job start

Because Glue bills at least one minute per job run, every job start has a minimum price, even if the job does no work at all:

Job type	Minimum billed per start
G.1X × 4 (4 DPU)	$0.029
G.2X × 6 (12 DPU)	$0.088
Python Shell (1 DPU)	$0.007

Multiply that by the job count and cadence, and you get the least the pipeline can cost just for starting its jobs:

Cadence	Floor per run	Floor per month	Floor per year
Daily (191 jobs)	$5.64	$174.80	$2,058
Weekly (128 jobs)	$3.79	$16.79	$198
Monthly (128 jobs)	$3.79	$3.79	$45

These are lower bounds. Every job runs longer than a minute, so actual cost is higher, but the floor alone shows the pattern. The transform layer's share of the daily floor was $1.85 per run, about $675 a year, just to start 63 jobs that didn't change the data.

What the pass-through layer cost

	Per daily run	Per month (31 runs)	Per year
Transform layer (63 jobs, no business logic)	$10.04	$311	$3,663

Post-tuning estimate, same method as above.

What it did to runtime

Dropping the pass-through layer is a win on its own, but I wanted to measure the cadence change by itself. So the baseline here is a daily run without transform: 29.3 − 5.43 ≈ 23.9 minutes. Every gain below comes from running less often, not from removing a layer.

Cadence	Avg run time	Avg records / run	Runs / month	Pipeline time / month
Daily (excl. transform)	~23.9 min	21M	31	~12.3 h
Weekly	65.4 min	110.5M	~4.4	~4.8 h
Monthly	2h 43m	540M	1	~2.7 h

Weekly: average of four full 7-day runs (107–114M records, 51–71 min each). Monthly: average of the July and August runs (505M and 575M records). The daily monthly total is an estimate based on the post-tuning run time.

A weekly run processes about 5× the records of a daily run in about 2.7× the time. A monthly run processes about 26× the records in about 6.8× the time. If runtime grew with data volume, those ratios would be about the same. They aren't, because the fixed startup cost is paid once per run no matter how much data the run carries.

xychart-beta
    title "Records processed per minute (millions)"
    x-axis ["Daily (excl. transform)", "Weekly", "Monthly"]
    y-axis "M records / min" 0 --> 4
    bar [0.9, 1.7, 3.3]

The jobs, Spark configs, and code were the same at every cadence. The weekly cadence runs about twice as efficiently as daily, and monthly about four times, only because less of each run is spent starting up.

Over a month, weekly cadence cuts total pipeline time by about 61% compared with daily, and monthly cuts it by about 78%.

What it did to cost

Monthly Glue cost by layer at each cadence (daily includes transform, since that's what production ran):

Layer	Daily / month	Weekly / month	Monthly / month
Ingestion	$365.49	$204.23	$67.34
Transform	$311.09	$0 (skipped)	$0 (skipped)
Publish	$295.03	$114.21	$94.81
Outbound	$15.14	$6.73	$3.83
SFTP	$1.26	$0.27	$0.23
Total	$988.01	$325.44	$166.21
Per year	$11,633	$3,832	$1,995

Upper-bound estimates, same method as above. Daily uses the post-tuning run. Weekly uses the average layer times of the four full weeks. Monthly uses the average of the July and August runs.

Compared with daily, weekly cadence cuts estimated Glue cost by about 67% and monthly by about 83%. Nearly all of the savings come from the 63-job layers: ingestion and publish run far fewer times, and transform drops out completely. The single-job layers barely register either way.

The trade-off

The cost is data freshness. With a weekly feed, the vendor can see data up to seven days old. With monthly, up to a month old. That was acceptable for this consumer, which is why the change was an option at all. If your downstream needs daily data, this approach doesn't apply, and reducing per-job overhead means something different, such as consolidating many small table jobs into fewer larger ones.

The daily-cadence totals above are estimated from a 21M-record run. Low-volume days run shorter, so the real saving is somewhat less than the table shows. It's still the largest single improvement we made to this pipeline.

What I'd check on any Glue pipeline

Find a run with almost no data. Its runtime is your fixed overhead floor. If that floor is a large share of a normal run, job tuning will stop paying off quickly.
Count job starts, not just runtime. 191 jobs × 1 run a day looks fine on a per-run dashboard. At nearly 70,000 job starts a year, the overhead is easier to see.
Ask whether the consumer needs the current cadence. Daily is often the default rather than a requirement. The cheapest overhead to remove is on runs nobody needed.
Look for pass-through layers. A stage can exist in the architecture diagram without changing the data. If a layer's output matches its input, every job in it is pure overhead.
Sum DPU-hours across parallel jobs, not wall clock. A Map state with 63 jobs finishes when the slowest one does, but you pay for all 63.

The takeaway

Tuning took a daily run from 1h15m to 29 minutes, and that was worth doing. But the change that mattered more was noticing that a lot of each run was startup, not work. Changing the cadence and dropping a pass-through layer took monthly pipeline time from over 12 hours to under 3, cut job starts from 5,921 to 128, and cut estimated monthly Glue cost from about $988 to about $166, without touching any job code.

When a pipeline is built from many small jobs, the fastest job is often the one you never start.

Cutting a production AWS Glue pipeline's runtime by about half, plus two bugs that taught me more

Kiran Gunturu — Sat, 26 Sep 2026 14:23:42 +0000

I spent the last few months tuning a production pipeline on AWS Glue and PySpark: Oracle to S3 ingestion, then packaging, encryption, and transfer to a downstream analytics platform. The goal was to cut runtime without touching business logic, and across four workstreams it came to roughly a 56% reduction end to end.

Here are the four changes that moved the needle, then two investigations that never became wins but taught me more. All numbers are from my own runs, checked against logs, with anything environment-specific genericized.

Ingestion: crossing a concurrency wave boundary

The 63-table ingestion ran through a Step Functions Map at concurrency 30. With 63 items that forces three waves, and the wall clock is the last item to finish, not the slowest job. The run sat at 13:55.

Raising Map concurrency to 50 dropped it to two waves and about 8 minutes. Waves are a cliff, so the payoff is in crossing a boundary, not nudging the number. Largest-first packing helps too (start the heavy tables in wave one so none is stuck in the final wave).

Publish: stop rewriting the same bytes

The publish layer wrote output, then rewrote it: a rename pass, a newline pass, and a standardize pass, each re-reading the whole thing. It was redoing on the driver what the executors had already produced. Writing the final bytes once, straight from the executors, and parallelizing the reconciliation instead of running it serially, took that stage from about 25 minutes to 7. No logic changed, just how many times the data got touched.

Outbound packaging and encryption

The outbound job zips files, GPG-encrypts each ZIP, and uploads with SSE-KMS. It ran on Glue Spark with 6 DPUs on G.2X, and it was slow for a dumb reason: everything ran serially on the driver, with Spark only used to build one logging DataFrame at the end. Six DPUs of executors sat idle while the driver zipped one file at a time.

Compression was the CPU-bound phase, so I threaded just that and dropped DEFLATE to level 1:

with ThreadPoolExecutor(max_workers=N) as pool:
    pool.map(lambda part: build_zip(part, compresslevel=1), parts)

Compression dropped from about 9 minutes to 4. Level 1 does make the zips slightly larger, but that was an easy trade here since runtime was the constraint, not archive size. It's worth knowing which side you're optimizing for: if a downstream had tight storage or bandwidth limits, I'd push the level back up and accept the slower compression. For this pipeline the faster, slightly bigger output won. Giving each batch its own temp key handled the rest.

Encryption had two traps. GPG compresses before encrypting by default, but the input was already zipped, so it burned CPU re-compressing incompressible data and the output actually grew (one batch went 5246 MB to 5312 MB). --compress-algo none fixes it. Then parallelizing encryption the way I did compression OOM-killed the driver, because each multi-GB ZIP was read fully into memory alongside an armored copy. I kept encryption serial as the safe landing. The proper fix, staged separately, streams file-to-file with gpg.encrypt_file so no whole ZIP is in RAM, gives each call its own GNUPGHOME, and writes unarmored binary (about 33% smaller). The blocker was never concurrency, it was the memory model.

SFTP transfer: when Spark is the wrong tool

A separate job pushed large files to an external SFTP endpoint on Glue Spark (6 DPU, G.2X) and did zero distributed work. It was bandwidth-bound at about 11 MB/s on roughly 5 GB files, and raising connector concurrency did nothing, because the ceiling is throughput, not session overhead. Moving it to a Glue Python Shell job at 1 DPU kept the same throughput and cut annual cost from about $63 to $11 at $0.44/DPU-hour. A single-stream, IO-bound transfer is a script, not distributed compute.

Two investigations that never became wins

A JDBC partition that made every connection full-scan the table. The ingestion runtime made no sense: 8.9M rows took 8 minutes, but 25M in prod ran 2h45m and climbing. I assumed skew, but the bucket-distribution query came back near-uniform, which killed that theory. The real cause was that the partition column is a computed expression Oracle can't index, so every connection full-scans the whole window just to find its slice, and at prod scale those concurrent scans blow the buffer cache and stall. It stayed a diagnosis: the fix is to partition on a prunable column, which I haven't implemented and measured yet.

A hang that fixed itself on rerun. The same job hung 2h42m one run, then finished in 14 minutes on an identical rerun. The cause was a JDBC socket read with no timeout: one connection died silently and blocked its task forever while the others finished, and Spark can't close a stage until every task returns. Layered timeouts (queryTimeout, oracle.jdbc.ReadTimeout, oracle.net.CONNECT_TIMEOUT) plus speculation break and reschedule a dead read. One trap: oracle.net.CONNECT_TIMEOUT is milliseconds as a connection property but seconds bare in the URL.

The through-line

The wins were all mechanism, not logic: cross a concurrency boundary, stop touching the same bytes twice, parallelize the phase that fits the resource you have headroom in, and right-size the runtime to the work. The investigations were the other half of the lesson, that the bottleneck is almost never where you first guess. Skew looked obvious until one query killed it, and half the real story lived in the source database where the Spark UI can't see it. Mostly it's proving what the bottleneck actually is before changing anything.

Building an AWS + Snowflake with Terraform: Lessons from My CI/CD Review

Kiran Gunturu — Mon, 21 Sep 2026 17:46:40 +0000

Every analytics platform has an unglamorous foundation: the raw/landing tier where source data first lands before anything clever happens to it. It's easy to click that together by hand once. It's much harder to make it reproducible across dev, UAT, and prod and to prove to yourself that what's in prod is what you reviewed.

So I built the landing tier of a banking-style (FCC/AML) data platform entirely as Terraform: a single root module orchestrating five child modules, with remote state, environment promotion, and a GitLab pipeline gating every change. Then I reviewed my own code as if it were a PR and found a handful of design gaps that are worth more than the happy-path walkthrough. Both halves are below.

What the stack provisions

One root config wires together five modules across AWS and Snowflake:

Why modular, not one big file

The temptation with Terraform is to drop everything into main.tf. Splitting into aws_vpc, aws_ec2, aws_s3, aws_glue, and snowflake_database modules buys three things:

Explicit dependencies. The EC2 module consumes the VPC module's subnet_id and security_group_id outputs an implicit dependency Terraform resolves for you. The Glue module carries an explicit depends_on = [module.s3_raw] so the bucket exists before the job references it. S3 and Snowflake share nothing, so they apply in parallel.
Reuse. The same VPC module can back a completely different stack.
Reviewable blast radius. A change to the Glue module can't silently touch networking.

State that won't corrupt itself

The backend is S3 for the state file plus a DynamoDB table for locking, both encrypted:

terraform {
  backend "s3" {
    bucket         = "your-tf-state-bucket"
    key            = "tf_state/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "your-tf-lock-table"
    encrypt        = true
  }
}

The DynamoDB lock is the part people skip and regret. Without it, two applys running at once — a teammate and a CI job, say can race and corrupt state. The lock makes the second one wait.

Promotion by branch, not by copy-paste

Instead of separate root configs per environment, one config selects its variable file from the branch, driven by workflow:rules:

main → main.tfvars
uat → uat.tfvars
everything else (dev + feature branches) → dev.tfvars

First match wins, and merge-request events are dropped up front. The pipeline stages are validate → plan → apply → destroy:

plan writes a saved plan file and apply consumes exactly that artifact, so you apply the plan you reviewed - not a fresh one computed against drifted state.

Here's the pipeline running for real across the dev, uat, and main branches:

Now the part that actually taught me something

Reviewing this as a PR, three design gaps stood out. These are the interesting bit, because they're the mistakes that look fine until they bite:

1. Merge requests produce no pipeline at all. Because MR events are killed with when: never, there's no validate and no plan at review time. Reviewers approve blind - they never see what the change would actually do to infrastructure. The fix is a dedicated MR job that runs validate and plan and posts the plan output on the merge request, so approval is based on evidence.

2. main auto-applies to prod with no gate - but prod can't be destroyed. The pipeline applies to production automatically on merge to main, yet destroy is manual and blocked on main. That asymmetry is backwards: teardown is protected, but the far riskier unattended apply to prod isn't. Production apply is exactly where you want a manual approval gate.

3. Secrets in *.tfvars, and a .gitignore that misses them. The variable files carried credentials, and .gitignore only excluded *.tfvars.json - so *.tfvars was tracked. Credentials belong in CI variables or a secrets manager, the variables should be marked sensitive = true, *.tfvars should be ignored, and any exposed secret must be rotated and scrubbed from history.

A few smaller ones: the Glue module hardcoded its job name and script path instead of using the variables passed in, so it pointed at a bucket the stack never created; the pipeline had no terraform fmt check or security scanner (tfsec/Checkov); and the manual destroy used --auto-approve.

Takeaways

If you're standing up a landing zone of your own, the patterns that held up: split into modules with clear input/output contracts, lock your state, and promote one config across environments by variable file rather than duplicating configs. And the lesson the gaps taught me: the CI/CD design is the real security boundary, not the Terraform. A tidy .tf file with a pipeline that lets prod apply unreviewed and secrets slip into git isn't safe - it just looks safe.

Terraform makes infrastructure reproducible. Your pipeline is what makes it reviewable. Get that second part right.

The full code

If you want to dig into the actual Terraform - all five modules, the GitLab CI pipeline, and the environment variable files - the complete project is here:

🔗 https://gitlab.com/kiran.gntdm-group/terraform

Questions and feedback are welcome - happy to talk through any of the design choices.