I have 2 projects on Gitlab. One with terraform to provision infrastructure, another with Lambda code. Lambda is configured to be deployed from S3 bucket.
In terraform I have a dummy zip file uploaded to s3. I had to give any ZIP otherwise Terraform complained during apply.
I use a separate project to keep Lambda code, which has separate build pipeline and deploy to the same s3 bucket. It should overwrite the dummy ZIP on deploy.
The problem is that whenever Terraform executes the real ZIP is overwritten with the dummy one again, so I need to deploy the real ZIP from another project again.
I don't want to keep function code together with Terraform project.
I also don't want my Terraform build to always trigger another project build (containing function source code), because most often Lambda does not change but other resources are modified in Terraform configuration.
I thought about having another Lambda triggered when ZIP is uploaded to S3 which would verify if it is a real or dummy one.
In case of dummy one it would trigger another project deployment using Gitlab API.
Is there any easy solution to my problem?
Top comments (8)
Can I ask about the purpose of the dummy zip? Is it intended to provision an empty lambda?
I'm not sure I understand the specific problem but I'll tell you that the boundaries between what is infra (generally terraform) and what is business logic (usually serverless/lambda) are far from clear cut for me.
yes, the dummy zip is just to allow provisioning of lambda with Terraform. I may be wrong, but when I configured Lambda to be deployed from s3 I had to give s3 key and it had to exist, otherwise Terraform complained, so I had to put anything there.
What I would do is either move the terraform part to the lambda repo or get rid of it if you already have a way to deploy the lambda.
In my mind there's no such thing as "provisioning" a lambda and then "deploying" the code as one does in a serverful approach, and I believe keeping that mindset is what leads to the awkwardness you're dealing with. You don't gain anything by having an empty lambda placeholder waiting for your code, as the code and the infra are almost one and the same.
Hope that's helpful (and I'm making sense). Feel free to share code if you want more specific advice.
This makes sense. Unfortunately the infrastructure is hybrid and consists of some ECS services inside vpc. Lambdas need to have access to VPC, so it needs subnet IDs, security group IDs and also URLs to the ECS services and if I keep all in one project then I can use references, otherwise I would have to pass the values of provisioned resources as input arguments to the separate Terraform configuration.
In addition, separate team is responsible for infrastructure and Terraform and separate for coding.
So in your opinion we should split the Terraform config so that both teams are responsible for their own parts?
Ok i see how keeping it in an infra repo makes sense in that case. I usually deal with much smaller stuff so take anything I say with a grain of salt.
A more workable approach might be adding the lambda code as a dependency of the infra. Either as a git submodule or grabbing with a shell script from terraform.
But again, far from an expert on that kind of situation so I'll stop here :P
I'll think about your suggestion as well. Thanks :)
I just found this serverless.com/blog/definitive-gui...
This makes a lot of sense to separate app infrastructure from shared infrastructure.
This discussion gave me some more ideas on how to build separate TF configurations dependent on each other. github.com/hashicorp/terraform/iss...