Best way to make infrastructure by Terraform

Hi,
I am new to terraform; I've been involving in a project and trying to make a CICD pipeline in order to work on data through multiple tiers and of course proper way to automate moving data between dev, staging and prod environment. I have some questions:

• What could be the best practice to set up terraform structure?
• Dividing modules into specific submodules is recommended?
• Is it good to use terraform for invoking Lambda or serverless is the better solution?

Well, I need your #help and Looking forward to hearing from you.

Thanks,

Comments

[Rolf Streefkerk]

See here for some good guidelines:
github.com/ozbillwang/terraform-be...

In general you need to think hard on the structure of Terraform, I propose you do these 3 things;

• use modules, keep the general resources code separate from your solution code. Later on you can even put the modules in separate git repo's and use versioning to avoid changes from breaking other solutions.
• use a solutions/services directory, this is where you wire up all the modules together into a solution or a service. Basically your application.
• use an environment directory, this is where you define the exact parameters of your deployment per environment (dev, test, production). There are differences and you should keep them separated at all times.

This could look like this:

- modules
  - ec2
  - vpc
  - rds
  - apigateway
- solution
  - api
  - application_a
- environment
  - dev
  - test
  - prod

You can check out my public repo for an example:

rpstreef / openapi-tf-example

Example of how you can use OpenAPI with AWS API Gateway, Also includes integrations with AWSLambda, AWS Cognito, AWS SNS and CloudWatch logs

OpenAPI with AWS API Gateway, Lambda, Cognito, SNS and CloudWatch logs

This repo only deploys the infrastructure via Terraform. The source code, this repo, will be deployed automatically via AWS CodePipeline. By default, it's configured to automatically deploy at every push to the master branch.

Check the companion articles series 'OpenAPI' on dev.to.

Get started

The essentials

• Download Terraform v0.12.x here
• You will need Node v12.x from here
• Git, to clone this Repo, from here
• Create a free AWS account (requires credit card) here
• Finally, download the AWS CLI tool
• Setup your AWS local profile, see this guide how it's done.
• Manually setup an AWS S3 Bucket for Terraform state storage.

To get the API running

If you meet all the pre-requisites, do the following

• In your AWS development account create the S3 bucket for your Terraform state files
  • Optionally, encrypt the S3 bucket and enable versioning…

View on GitHub

[SalSal]

Thank you for answer the questions, was really great help to make my mind clear!
If you could also know your opinion on this as well?
which one is better to create a bunch of s3 buckets for multiple environment? write a module to create them once or use simple module and apply multiple times?

[Rolf Streefkerk]

Do they have the same functionality across these environments (like development, test etc) then you would have 1 S3 module and in your application directory you create the single bucket. It will be named slightly different per environment (include the name environment name, dev, test,prod).

Then in your environment directory you supply the correct naming for each environment, then you have to deploy each of them separate with their respective tfstate file per environment.

That is probably the easiest way to do it and then make sure you store the Terraform state in each environment where it belongs to as well.

To target each of them, you can create profiles in your ./.aws/config file and reference them in the Terraform tfstate and your remote state configuration.

I hope that helps

[SalSal]

Well, there are five buckets at least which will repeat in dev, stage and prod! but as far as they contain their environment name in their name, should be change per environment.

[Bob Nadler]

I highly recommend getting some structured training to become familiar with the fundamentals first. Hands-on courses like Using Terraform to Manage Applications and Infrastructure are likely to answer many of your questions.

There is a lot of free content on these topics around, but IMO paid services like Linux Academy and Cloud Guru provide a higher quality learning experience even if you're not working towards a certificaiton.

Good luck!

[SalSal]

Thanks, just registered to complete the course.

[Aaron Berry]

Is it good to use terraform for invoking Lambda or serverless is the better solution?

Terraform should deal with declaring and provisioning your required infrastructure only. You can invoke Lambda functions in terraform to fetch data from that lambda function to use in your terraform modules however but it shouldn't be done to perform actions that may change infrastructure or data unknowingly.

If you need to call your lambda functions for another reason it might be good to use something like Ansible to script this action and call this as a separate step in your pipeline.

[andre aliaman]

1. What could be the best practice to set up terraform structure? For me, structure will give others who see your code a better understanding of the way you thinking when trying to solve the problem. So if you shared this project with your team/others, clearer is better.

One of my favorite of the terraform Structure is from Charity. She create the article here: charity.wtf/2016/03/30/terraform-v...

the way how she explains the problem, the process, the way how she looks at that article gives me some idea of how I create Terraform projects.

1. Dividing modules into specific submodules is recommended?
   Depends on how big your projects and how granular you want. If you only work for small services/small projects maybe not too much submodule will be better. Since too much modules/submodules will be more hard maintaining it later

2. Is it good to use terraform for invoking Lambda or serverless is the better solution?
   Depends again. If your terraform is part of big projects for maintaining your whole environments, I think it a bit of risk if you want to use terraform for invoke Lambda. Since one error/problem will risk your state.