So what is Data API and how can it help you?
You know how it goes.
You get a support ticket from a developer: “Please create a DB user for my service.”
Or worse: “I need to query the DB, can you create me a personal user?”
It’s not the most exciting task a DevOps engineer can get, but it’s critical for keeping the business running. And when this process does not scale, DevOps quickly becomes a bottleneck.
So we built a self-service mechanism. A developer can simply use a Slack bot to request a database user, without involving DevOps.
Win for the developer, win for DevOps.
Under the hood, the architecture was actually pretty solid.
We stored the desired users list in an S3 bucket.
Each change triggered an event notification to a central SNS topic, which kicked off a Step Function in every AWS account.
From there, Lambda functions handled the heavy lifting: creating the DB user with the right permissions (read-only, read-write, admin), generating the secret, and even registering it in the database proxy.
On paper, this was robust. In reality, distributed systems are never perfect.
Sometimes an event was delayed. Sometimes a Lambda failed. Sometimes permissions drifted. And occasionally, a user was simply not created.
The real problem was not the failure itself.
It was visibility.
From the developer’s perspective, they clicked a button in Slack and… nothing happened. No clear feedback, no way to validate the outcome.
That is when we decided to build a Validator workflow.
The idea sounded simple: create a Lambda function that checks whether the DB user exists and report the result back to the user.
But then reality hit again.
We needed to validate users across multiple environments, multiple AWS accounts, and multiple databases. And as we all know, connecting to a database usually means one thing: network access from inside the VPC.
In a multi-account setup, that left us with two main options.
The first option was VPC peering. Technically possible, but not something we were comfortable with. We did not want to expose production VPCs to other environments just for validation purposes.
The second option was to deploy a Lambda function in every account and trigger it cross-account. That worked, but now we were managing dozens of Lambdas, IAM roles, permissions, and invocation logic. The validator itself was becoming another distributed system.
At that point, we stepped back and asked a simple question:
What if there was a way to query all of our databases from a central account, without VPC peering and without deploying Lambda functions everywhere?
Turns out, there is.
And it is called Data API.
Data API is a feature of Amazon Aurora that allows you to interact with a database using API calls, without requiring direct network connectivity to the database VPC. No security groups, no subnets, no peering. Just IAM-authenticated API calls.
This changes the game.
Instead of running validation logic inside every VPC, we could run a single Lambda in a central account and query each database directly using Data API. Same code, same workflow, no networking complexity.
Under the hood, Data API is also what powers the RDS Query Editor in the AWS console. When you run a query from the UI, you are already using it, whether you realize it or not.
Cross-account access (the part that actually matters)
Because the validator runs in a central account, we still needed a secure way to access databases that live in other environments.
Data API removes the networking requirement, but IAM is still enforced.
In practice, this meant creating a cross-account IAM role in each environment. The central account is allowed to assume this role, and the role itself has permissions to call the Data API on the local Aurora cluster.
We deployed this role to all environments using Terraform, so every account followed the same trust policy and permission boundaries. No manual setup, no snowflakes.
From the validator’s perspective, the flow is simple:
Assume the role in the target account
Call the Data API
Run the validation query
Return the result to the user
Data API solves the networking problem.
Cross-account IAM roles solve the permissions problem.
Together, they let us centralize access without compromising security.
When Data API actually makes sense
Data API is not a silver bullet. It has throughput limits, different latency characteristics, and it only works with Aurora. You would not use it for high-volume application traffic.
But for control-plane operations like validation, auditing, administrative workflows, and platform automation, it is extremely powerful.
In our case, Data API allowed us to reduce infrastructure complexity, standardize access across environments, and give developers fast, reliable feedback without pulling DevOps into every request.
Sometimes the best solution is not adding more infrastructure, but removing it.
Top comments (1)
I wish I could find your page a little sooner, so could escape finding out
Data APIby myself. My reason was to do post-cluster database/user managment, so the DBs are ready for the app developers without any manual intervention.I also did a post on that, which may interest you.
dev.to/santanu_das/series/35002
-S