Delete Old Backup/Logfiles/Temp files from EFS using Lambda, PowerShell & Event Bridge

Recently i was working on a requirement where a legacy application was migrated to AWS using Lift and Shift strategy. This application used to generate lot of log data and Temporary files.

The log, temp and backup folder size used to reach around 20GB in 2 days, we were using EFS and there were multiple EC2 instances that were connected to EFS.

After analyzing each file we decided that temporary files were just used for pre-processing and they were only required one time so we decided to delete those along with older log files (except access logs) & Old backup files.

I decided to use Powershell Lambda and integrate EFS with Lambda, this function will delete all un-necessary data from EFS. This function will execute periodically using Event Bridge & delete un-necessary files.

We will create Lambda function first using Visual Studio Code

Step 1 : Download Visual Studio Code
Step 2 : Execute below commands in powershell

$region_code = 'ap-south-1'
$secretKey='your secret key'
$accessKey='your access key'

#Install-Module AWSPowerShell -Scope CurrentUser
#Install-Module AWSLambdaPSCore -Scope CurrentUser

import-Module AWSPowerShell -Force
Import-Module AWSLambdaPSCore -Force

#configure script credentials
Initialize-AWSDefaultConfiguration -AccessKey $accessKey -SecretKey $secretKey -Region ap-south-1
Set-AWSCredential -AccessKey $accessKey -SecretKey $secretKey -StoreAs default

#shows existing function list from AWS
Get-LMFunctionList

Get-AWSPowerShellLambdaTemplate

We'll be using Basic Barebone Script and adding our script commands to it

Create a Starter Lambda function

New-AWSPowerShellLambda -ScriptName DeleteUnusedFiles -Template Basic

You will get this project folder in explorer pane in your Visual Studio Code or else you can import the folder, this project gets created in the same

Step 3 : Add below command to your Powershell script, this is to ensure that our Lambda function is successfully able to get

cd /mnt/efs
Remove-Item * -Include *.tmp
ls /mnt/efs

Step 4 : upload Lambda function to AWS

$publishPSLambdaParams = @{                                                                       
     name = "DeleteUnusedFiles "
     ScriptPath = ".\DeleteUnusedFiles\DeleteUnusedFiles.ps1"
     Region = "ap-south-1"
     IAMRoleArn = "delete_backup_rp_role"
     }

Publish-AWSPowerShellLambda @publishPSLambdaParams

I have created an EFS file system and Access points as below (You can refer below video for more information on setting up EFS and Mount Targets) :

Below are the details for my Accesspoint:

Once EFS is created and Access points are ready, you need to login to your EC2 instance and mount EFS

1) install EFS Mount Helper

sudo yum install -y amazon-efs-utils

2) create folder as efs

mkdir /efs

3) Mount EFS

sudo mount -t efs -o tls,accesspoint=fsap-9999999999999999 fs-8888888888888888:/ efs

Important Step

Remember to configure security group in EFS as below

Once above steps are completed, we will integrate EFS with Lambda as below:

**** Important, you must associate Lambda to a VPC first, without doing this step, you wont be able to attach EFS

1) Create Role with below permissions and associate it with Lambda function

2) Open your Lambda Function (In this example i am using PowerShell lambda function)
3) Navigate to Configuration tab
4) Open File-System Subtab

Select EFS File System, Access point and Local Mount Point as per below screenshot and Hit save/create button.

Once successfully published you can test your function by opening your Lambda function and clicking on Test, after creating Test event.

As you can see in below screenshot we can see the result of ls command which shows files that are there in the EFS share and hence this shows that we can access the EFS from Lambda.

Once this is successfully created you can schedule your Lambda function using Event Bridge to execute periodically.

Troubleshooting:

If you get an error stating "Cannot configure .NET CLI", open below file from Path in Notepad++

C:\Program Files\WindowsPowerShell\Modules\AWSLambdaPSCore\3.0.1.0\Private_DeploymentFunctions.ps1

Find for keyword $LASTEXITCODE
Comment all occurrence and conditions that are using this variable and run below commands

import-Module AWSPowerShell -Force
Import-Module AWSLambdaPSCore -Force

Try to upload Lambda function again using Powershell, it should work now.

If you have any other comment, suggestion for this post then do let me know.

Please do subscribe my YouTube channel https://www.youtube.com/c/Cloudperceptor/videos

Thanks,
Ujjwalkumar Soni