I just Dev

Posted on Jul 16

I Asked GPT to Build the Same Infrastructure in AWS CDK, Terraform, Pulumi, and SST. Here's What Happened.

#aws #terraform #devops #ai

Most Infrastructure as Code debates ask the same questions:

Which tool has the best syntax?
Which one scales across teams?
Which one supports multiple clouds?
Which one produces the safest deployments?

But AI coding agents add a new question:

Which IaC framework gives an LLM the best chance of generating infrastructure that is correct, understandable, and easy to repair?

That is not exactly the same as asking which framework is best for humans.

A framework can be concise but difficult for an agent to validate. Another can be verbose but backed by an excellent preview engine. A third can turn an incorrect property into a precise compiler error that the model can fix immediately.

So I compared six popular ways to define AWS infrastructure:

AWS CDK
Terraform
Pulumi
SST
AWS SAM
AWS CloudFormation

The result surprised me less than the reason behind it.

AWS CDK came out on top—but not simply because it uses TypeScript or Python. It won because it creates one of the strongest feedback loops for an AI coding workflow.

Important disclosure: This is a small, directional, opinionated desk test—not a statistically controlled benchmark. The scores are engineering heuristics, not universal pass rates. Model choice, framework versions, supplied documentation, provider schemas, and the quality of the validation loop can all change the result.

The test

I used the same infrastructure request for each framework:

Build a production-minded serverless URL-shortening API on AWS.

It must include an API Gateway HTTP API, a Lambda function, a DynamoDB table using on-demand billing, encryption, point-in-time recovery, least-privilege permissions, environment-specific naming and tags, retained logs, and useful stack outputs.

Do not hardcode credentials. Use current stable syntax, pin important dependencies or providers, explain the file structure, and include commands to format, validate, preview, and deploy the stack.

For AWS CDK, I used TypeScript as the main pass and treated Python as a spot-check. Both are officially supported CDK languages, but TypeScript has a practical advantage for this kind of experiment: it was the first CDK language, many examples are written in it, and its type checker produces detailed repair signals.

I evaluated the first-pass result across five questions:

Did it choose the correct resources and connect them properly?
Did it avoid obviously excessive IAM permissions?
Was the generated code consistent with the framework’s current model?
How useful were the compiler, validator, preview, or synthesis errors?
How much human correction would I expect before a safe deployment?

I did not treat a successful generation as permission to deploy into production. A clean-looking answer from an LLM is not a security review.

The scorecard

Framework	LLM Code Quality	Reliability	Fixes Needed	Best For	Overall
AWS CDK (TypeScript/Python)	9/10	8.5/10	Low	AWS-first teams	Winner
Terraform	8/10	8/10	Medium	Multi-cloud teams	Strong
Pulumi	8.5/10	8/10	Low–Medium	Developers who prefer code	Strong
SST	8.5/10	7.5/10	Low–Medium	Full-stack and serverless apps	Promising
AWS SAM	7/10	7/10	Medium	Focused Lambda stacks	Good
CloudFormation	6/10	6.5/10	High	Existing native AWS templates	Limited as an LLM authoring surface

Again, these numbers are directional. They describe this workflow, not the total capability or maturity of each platform.

1. AWS CDK: the best repair loop

AWS CDK lets you define AWS infrastructure with general-purpose languages and then synthesizes the application into CloudFormation.

That gives an AI agent several layers of feedback:

The language parser catches broken syntax.
TypeScript can catch incorrect properties, imports, and argument types.
cdk synth can catch construct-level and synthesis problems.
cdk diff shows the infrastructure changes before deployment.
CloudFormation remains the deployment engine underneath.

That sequence matters.

LLMs will make mistakes. The real question is whether a tool turns those mistakes into clear, local, repairable errors.

Consider the intent behind a line such as:

table.grantReadWriteData(handler);

The developer—or agent—is expressing a relationship between two resources. The construct can generate the required IAM policy rather than forcing the model to manually assemble a large policy document.

This does not guarantee least privilege, but it narrows the problem and makes the intent easier to inspect.

Why TypeScript performed especially well

TypeScript gave the strongest feedback in this comparison:

Misspelled properties usually failed early.
Incorrect object shapes produced useful diagnostics.
Imports and symbols were discoverable.
Refactoring resource names was less fragile.
The model could use compiler output as structured repair context.

What about Python?

Python is a fully supported AWS CDK language and is an entirely reasonable choice, especially for Python-heavy teams.

Its generated CDK APIs use familiar Python conventions such as keyword arguments and snake_case names. It is often more concise than TypeScript.

For agent-heavy workflows, however, TypeScript has a slight edge because more mistakes are surfaced by static analysis before synthesis. Python can still produce an excellent workflow when paired with type checking, linting, tests, and cdk synth.

My practical rule is simple:

Use the language your team can review. If the team is neutral and expects heavy AI-agent usage, start with CDK in TypeScript.

Where CDK still fails

CDK is not magically safe.

An LLM can still:

Choose an overly high-level construct without understanding the resources it creates.
Mix low-level Cfn* resources with higher-level constructs awkwardly.
Invent a property from an older API version.
Generate broad IAM permissions.
Hide expensive or destructive infrastructure behind a small amount of code.
Produce a clean synthesis that is still architecturally wrong.

cdk synth proves that CDK can produce a template. It does not prove that the architecture is secure, affordable, or appropriate.

2. Terraform: the strongest declarative baseline

Terraform performed well because its configuration language is constrained, widely used, and supported by a strong validation and planning workflow.

A good agent loop looks like this:

Generate HCL
   ↓
terraform fmt
   ↓
terraform validate
   ↓
terraform plan
   ↓
Policy and security checks
   ↓
Human approval

The plan step is a major advantage. It gives both the human and the model a concrete description of intended infrastructure changes.

Terraform was also the clearest choice when the prompt implied multi-cloud requirements. Its provider model is designed to interact with cloud platforms, SaaS products, and other APIs.

Where the LLM struggled

The most common risks were not basic HCL syntax. They were version and provider semantics:

Using an argument that belongs to another provider version.
Confusing similarly named resources or data sources.
Producing technically valid IAM JSON with excessive permissions.
Omitting state and backend decisions.
Creating implicit dependencies that were difficult to understand.
Assuming an AWS behavior that the provider does not model in the expected way.

Terraform providers have their own versions and release cycles. That means “write Terraform for AWS” is not enough context for a reliable agent. The model should know the Terraform version, AWS provider version, module contract, and organization conventions.

With current provider documentation and a strict module interface, Terraform can easily outperform a code-first tool.

3. Pulumi: the strongest general-purpose alternative

Pulumi has many of the qualities that helped CDK:

Infrastructure can be defined in TypeScript, JavaScript, Python, Go, .NET, Java, or YAML.
Normal language tooling can catch errors.
Teams can use familiar testing libraries and package systems.
The same approach can span AWS, Azure, Google Cloud, Kubernetes, and many other providers.

For straightforward resources, the generated Pulumi code was clean and readable. In a TypeScript workflow, the LLM benefited from many of the same compiler signals that helped CDK.

Why it did not take first place

The difficult parts appeared around framework-specific concepts rather than syntax:

Handling values that are only known after deployment.
Correctly composing outputs and resource dependencies.
Choosing the right provider or package.
Managing state and stack configuration.
Avoiding accidental mixing of ordinary runtime logic with deployment-time logic.

These concepts are learnable, but an LLM can produce code that looks like normal TypeScript while misunderstanding Pulumi’s execution model.

For a multi-cloud team that wants general-purpose languages, Pulumi may be the best overall choice. My score does not mean CDK is universally better; it means CDK had an advantage in this AWS-only prompt.

4. SST: the smallest surface area—and the biggest version-context risk

SST produced some of the shortest and most readable infrastructure definitions in the comparison.

Its current model lets developers define an application in code through sst.config.ts, use higher-level components, link resources to application code, and work across a large provider ecosystem. SST currently uses Pulumi behind the scenes for providers and the deployment engine, with Terraform providers bridged through Pulumi.

For full-stack and serverless applications, that is a powerful combination.

An LLM has fewer lines to generate, fewer resource identifiers to manually connect, and a vocabulary that often maps directly to application concepts.

Why the reliability score was lower

SST has evolved significantly, and older examples can describe APIs or project structures that no longer match the current documentation.

That creates a classic LLM failure mode: the model remembers several generations of a framework and combines them into one plausible-looking answer.

The solution is not to avoid SST. The solution is to provide better context:

Pin the SST version.
Attach or retrieve the current component documentation.
Tell the model not to use examples from older major versions.
Validate the generated configuration immediately.

With current documentation in context, SST could be the best choice in this list for a modern serverless or full-stack product team.

5. AWS SAM: excellent when the problem is actually serverless

AWS SAM is an open-source framework focused on building serverless applications. Its shorthand resources are transformed into CloudFormation during deployment, and the SAM CLI supports local development and testing workflows.

That narrow scope helps the LLM.

For familiar combinations such as Lambda, API Gateway, event sources, and DynamoDB, the model usually knows the basic shape of a SAM template.

The weaknesses appeared as the stack expanded:

YAML nesting errors were easy to miss.
IAM policies became verbose.
Complex conditions and intrinsic functions reduced readability.
Resources outside the serverless abstraction required regular CloudFormation definitions.
Large templates became harder to repair without schema-aware tooling.

For a small Lambda application, SAM may be a better choice than its score suggests. A focused tool can be safer than a more flexible tool when the problem fits its boundaries.

6. CloudFormation: essential deployment engine, difficult authoring surface

CloudFormation is the native AWS service for modeling and provisioning collections of resources as stacks. It is also the deployment engine underneath AWS CDK and AWS SAM.

Its coverage and importance are not “limited.”

The Limited label in my scorecard means limited as my first-choice direct LLM authoring surface for this task.

Directly generated CloudFormation tended to be:

Much longer than the equivalent high-level code.
More dependent on exact logical references.
More vulnerable to subtle intrinsic-function mistakes.
Harder to refactor.
Expensive in tokens when the model had to revise a large template.
More likely to contain hand-written IAM documents.

CloudFormation is often an excellent target artifact. It is simply not always the easiest source format for an AI agent to author and repair directly.

Why typed, code-first IaC often works well with LLMs

The easy conclusion is:

“LLMs prefer programming languages over YAML.”

I think that explanation is incomplete.

The real advantage is feedback density.

1. Symbols reduce ambiguity

Typed APIs give the model a constrained set of classes, properties, and methods. A wrong guess can become a compiler error instead of surviving as plausible text.

2. Errors are local

A TypeScript diagnostic often points to one property on one line. That is high-quality repair context for an agent.

3. High-level relationships reduce token count

A call that links a function to a table can replace a larger set of resource references and IAM statements. Fewer generated tokens mean fewer places for inconsistencies to appear.

4. Existing developer tooling becomes agent tooling

Formatters, linters, compilers, test runners, package managers, and IDE diagnostics all become part of the model’s feedback loop.

5. Refactoring is safer

Renaming a symbol through code is generally less fragile than manually changing logical references across a large template.

But code-first IaC also introduces risks. Abstractions can hide resources, ordinary control flow can make infrastructure harder to predict, and a small code change can synthesize into a large deployment change.

Typed code is not automatically safe. It is simply easier to place inside a rich repair loop.

The real winner is not CDK. It is closed-loop validation.

The biggest lesson from this comparison is that LLM performance depends more on the workflow around the framework than on the framework’s syntax.

A trustworthy AI-assisted infrastructure workflow should look like this:

Prompt and architecture constraints
              ↓
Generate the smallest vertical slice
              ↓
Format and lint
              ↓
Compile, validate, or synthesize
              ↓
Preview the infrastructure diff or plan
              ↓
Run policy, security, and cost checks
              ↓
Human review
              ↓
Deploy to an isolated environment
              ↓
Integration tests
              ↓
Explicit production approval

The model should never be rewarded merely for producing a green deployment command.

It should be rewarded for producing a reviewable change with clear assumptions, a safe preview, and evidence that the result meets the architecture constraints.

A practical playbook for using LLMs with IaC

No matter which framework you choose, these practices improve the result:

Pin versions. Give the model the framework, language, CLI, provider, and library versions.
Supply current documentation. Do not rely entirely on the model’s memory for rapidly changing APIs.
Start with one vertical slice. Generate one API route, one function, and one data store before asking for the entire platform.
Require assumptions. Ask the model to list unresolved architecture and security decisions before generating code.
Run deterministic validation. Use cdk synth, cdk diff, terraform validate, terraform plan, pulumi preview, sam validate, or the equivalent workflow.
Feed exact errors back. Compiler and validator output is better context than “it does not work.”
Review IAM manually. Treat wildcard actions and wildcard resources as findings that require justification.
Inspect destructive changes. Deletion, replacement, public access, networking, encryption, retention, and backups need explicit review.
Use an isolated account or environment. Do not give an experimental agent broad production credentials.
Keep a human approval gate. A successful plan is evidence for review, not permission to deploy.

Which framework should you choose for an AI-assisted workflow?

Your situation	My starting point
AWS-first team using TypeScript	AWS CDK
AWS-first team using Python	AWS CDK in Python or Pulumi in Python
Multi-cloud platform team	Terraform
Multi-cloud team that prefers general-purpose code	Pulumi
Modern full-stack or serverless product	SST
Focused Lambda/API application	AWS SAM
Existing native AWS templates or organization standards	CloudFormation

The most important qualifier is this:

Choose the framework your team can validate, review, and operate—not merely the one an LLM can generate in the fewest lines.

Final verdict

AWS CDK won this small comparison because it converted model uncertainty into useful feedback.

TypeScript caught API mistakes. CDK synthesis caught infrastructure problems. CDK diff made the resulting changes reviewable. CloudFormation remained available underneath as the deployment engine.

Terraform was the strongest declarative option. Pulumi was the strongest code-first multi-cloud alternative. SST was the most concise for modern application infrastructure. SAM remained excellent inside its serverless boundary. CloudFormation remained essential, but less attractive as the direct authoring surface for an AI agent.

The larger point is not that everyone should migrate to CDK.

It is this:

In the age of coding agents, we should evaluate Infrastructure as Code by the quality of its repair loop—not only by the elegance of its syntax.

Now I want to see a larger test.

Which combination should be next: GPT vs Claude vs Gemini across CDK, Terraform, and Pulumi? And what should count as a real failure—invalid syntax, unsafe IAM, a broken deployment, or an architecture that technically works but should never reach production?

Share the framework, model, and first error you encountered. That discussion may be more useful than another abstract “Terraform vs CDK” debate.

DEV Community