Boost Developer productivity and DBOps efficiency with AWS Aurora Cloning

#aws #githubactions #rds #lambda

Context

There are numerous situations where we need an independent and isolated DB. For example, developers want to test their code or the Ops team needs to run upgrade tests and etc. The common approach is restoring a snapshot. However, it can become expensive and inefficient over time for repetitive tasks.

AWS Aurora Clone

The "AWS Aurora Clone" becomes handy in this situation, where it provides a more efficient solution in terms of operation and cost.

The Aurora clone uses copy-on-write protocol and works in a way that it shares the same volume between the source and clone cluster(s), but the updates are only visible to the respective cluster.

I have shared, how the Aurora clone has been integrated into our GH actions to enable developers, spin up an independent and isolated Aurora RDS cluster, which contains the same data as the given (source) cluster to test their code on demand.

Solution

In my organization, we have dedicated AWS accounts for GH runners and applications.

Application accounts have two lambda functions.

Aurora Clone - Which is invoked by the GH action and performs the Aurora clone operation.
Aurora Clone Purge - Which is responsible for purging the cloned DB cluster, after a certain period of days where it is no longer needed; Invoked by an Event bridge scheduler.

Workflow

A developer creates a PR with a label clone-db
GH action invokes the lambda function that requires the name of the SOURCE_DB_CLUSTER and PR_NUMBER as inputs.

aws lambda invoke \
    --function-name arn:aws:lambda:$AWS_REGION:$AWS_ACCOUNT_ID:function:aurora_clone \
    --region $AWS_REGION \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "SOURCE_DB_CLUSTER":"$SOURCE_DB_CLUSTER", "PR_NUMBER":"$PR_NUMBER" }' response.json