Upload Multiple Files to S3 using Terraform

Introduction

Whether to make a static website or just backing up files, we often need sending files to s3 buckets. The simplest way is to run s3 cp or s3 sync commands, but integrating these commands in CI/CD causes a problem, they will always run and even if there is no change, the files will be updated which can take a lot of time depending on file size.
As terraform is stateful, we can just use terraform aws resources to provision files to s3 bucket. I will show you three ways you can achieve this, and also tell which works better for which situation.
Let's get started.

Method1: Provision Files using s3 cp / s3 sync

We can use a terraform null resource to run s3 cp or s3 sync command like this -

resource "null_resource" "update_source_files" {
    provisioner "local-exec" {
        command     = "aws s3 sync app/ s3://my-s3-bucket" 
    }
}

Bu there is a problem, this alone can't detect if a file content is changed. Thus it will not update files after the first apply.

Method2: Provision file using template_files module

Alternatively you can use this module to provision files -

module "template_files" {
    source                  = "hashicorp/dir/template"
    base_dir                = "app"
}

resource "aws_s3_object" "objects" {
    for_each                = "${module.template_files.files}"
    bucket                  = "s3://my-s3-bucket"
    key                     = each.key
    content_type            = each.value.content_type
    source                  = each.value.source_path
    content                 = each.value.content
    etag                    = each.value.digests.md5
}

This works like a charm. But there's a problem. It takes a lot of times to plan and apply. The module traverses each files. For one of our frontends, we had around 1500 files and it took 20 minutes to plan apply in CICD.
If time is not a deal breaker to you then you can use this.

Method3: Provision files using s3_bucket_object

Finally, this is another way -

resource "aws_s3_bucket_object" "provision_source_files" {
    bucket  = "s3://my-s3-bucket"
    for_each = fileset("app/", "**/*.*")

    key    = each.value
    source = "app/${each.value}"
}

fileset("app/", "*/.*") means to take every files in the app directory.

This works just like method2. But it's way faster. from 19 minutes it took only 6 minutes in whole.

Conclusion: Best method?

If you are using your cli, you can use the first method. I often use this as an on the go solution. But with null resource it becomes useless unless somehow file change is tracked. The second method is time consuming. So I would suggest the 3rd method.
But hey, no solution is perfect for everyone. That's why we have multiple solutions for a problem (every problems). So just try and find what suits better for your problem.

Best wishes.