Access Google Cloud Storage from AWS Lambda using Workload Identity Federation

In this post we will look at how to access Google cloud storage from AWS Lambda functions using Google's Workforce Identity Federation.

Typically, when you access the resources belonging to one cloud from another cloud or from any other environment, you would use a service account credentials file, but this has a couple of downsides to it, one of them being the fact that anyone can use the service account credentials file to access the resources and the other being the fact that the credentials file have long expiry times.

Using the Workforce Identity Federation the caller or the consuming entity which is AWS Lambda in this case can access the destination resource without using a credentials file. This is achieved by creating a Workforce Identity Pool on Google Cloud IAM and providing the necessary permissions for the members of the identity pool to access the Google Cloud Storage bucket and downloading a configuration file, which will help the Google Cloud Storage client library to authenticate and access the required cloud storage bucket.

There is also a helpful YouTube video on the topic, however bear in mind that this video is quite old and this uses service account impersonation as a means to access the destination resource and also at the time of the recording of this video, workforce identity could only be configured using command-line, but lately Google has provided another option which does not require any service account impersonation for the authentication to work and also the entire configuration could be done using Google cloud console UI, the details of which we will see in this post. However, this YouTube video will definitely help you get a basic understanding of this topic.

Create Workload Identity Pool

• Visit Console > IAM > Workload Identity Federation

• Click on Create Pool and enter name and description

• Enter the provider details, which is your AWS account information in this case

• Finally, you can leave the defaults as it is for the attribute mapping

• Click Save

Once the identity pool has been created, the next step is to provide necessary permissions for the members of the identity pool to access the Google cloud storage bucket.

• Visit GCS > Bucket > Permissions tab

• Click Grant Access

• For the principal name use the following:

principalSet://iam.googleapis.com/projects/<GCS_PROJECT_NUMBER>/locations/global/workloadIdentityPools/<WORKLOAD_IDENTITY_POOL_ID>/attribute.aws_role/arn:aws:sts::<AWS_ACCOUNT_NUMBER>:assumed-role/<LAMBDA_FUNCTION_EXECUTION_ROLE>

Let me explain the variables that are used in the above block.

Variable	Description
GCS_PROJECT_NUMBER	Note that this is not Google Cloud Project ID, instead it is the Google Cloud Project number.
WORKLOAD_IDENTITY_POOL_ID	this is the name of the identity pole that we created earlier, this example it will be aws-pool-1
AWS_ACCOUNT_NUMBER	Your AWS account number
LAMBDA_FUNCTION_EXECUTION_ROLE	this is the land of function execution role that you were assigned for your lambda function which you would create and assign in AWS console IAM.

AWS Setup

• Download the workload identity pool configuration file.
• Note: you don't have to create any service account for this set up to work.

• Add this file to any location on your AWS lambda code.
• Bundle this file with the other files that are created for deployment on AWS Lambda.
• It is also safe to commit this file to your repository, because this file does not have actual credentials, instead it only has some configuration which will be used by the client libraries to validate against the workload identity pool.
• Create an environment variable, GOOGLE_APPLICATION_CREDENTIALS pointing to the location of the file in your lambda bundle, so that the client libraries can use this environment variable to locate the configuration file.

In summary, this method does not require you to create any long living service account files which is always a security risk, instead you could simply create a workload identity pool and provide the necessary permissions for the members of the pool to access any of your Google Cloud resources and include the config file as part of your deployment bundle and the client libraries will use this file to authenticate the execution environment to access the Google cloud resources that the identity member has got access to.

References

https://cloud.google.com/iam/docs/workload-identity-federation#mapping
https://cloud.google.com/docs/authentication/application-default-credentials#GAC