DEV Community: Tejas

Using an Assumed Role with AWS CLI

Tejas — Mon, 04 Sep 2023 03:57:33 +0000

Configuring Default Profile

To communicate with your AWS Resources and different AWS APIs, AWS provides its CLI commands and you can do so much work using the commands. To use these commands you need to get the profiles set. So let’s do this!

To configure a default AWS profile for your CLI, AWS provides a very handy command.

You can set a default profile by running this command:

    aws configure

Terminal prompts for AWS Access Key, Secret Key, Region, and Output Format. Configuring a default profile is useful if you are going to use that profile frequently.

After running this and providing appropriate values, your ~/.aws/config file will look like this:

[default]
region=ap-south-1
output=json

~/.aws/credentials file will look like below:

[default]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Note: In Windows, the AWS directory will be located in C:/Users/<userid>/.aws

Configuring Named Profiles

AWS CLI can have multiple profiles so that you can switch between different profiles to work with different projects. You can add the profile name you want using the —-profile option.

For example, if you want to have a profile named dev_account, you can do that by running:

aws configure --profile dev_account

The same process would follow as configuring the default profile.

Great! You have configured a named profile for yourself🙌.

Now, your config file will look like this with multiple profiles configured:

[default]
region=ap-south-1
output=json
[profile dev_account]
region=us-east-1
output=json

and your credentials file will look like this:

[default]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
[dev_account]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Get the list of profiles by running the following in the terminal:

aws configure list-profiles

It will display profiles with the name you added now. Cheers🚀.

Configuring an IAM Role using CLI

You can also be having different roles having different permissions. So it is useful to configure them into CLI and to access the roles using commands.

For example, if you want a profile for a role named dev_account_readonly, you can do that by running:

aws configure --profile dev_account_readonly

The terminal prompts for the same configurations, but you can skip entering Access Key and Secret Access Key while creating a role profile, these credentials will be taken from the source profile we set ahead.
To set the required properties for the role profile, use the below commands and provide appropriate values:

aws configure set source_profile dev_account --profile dev_account_readonly
aws configure set role_arn ROLE_ARN --profile dev_account_readonly
aws configure set mfa_serial MFA_SERIAL_ARN --profile dev_account_readonly

Here, we set the source profile as the profile that has the permission to assume the role and IAM Role ARN. If the account is using Multi-Factor Authentication, then you also need to set the ARN of that MFA device.

After configuring the role your config file will be looking like this:

[default]
region=ap-south-1
output=json
[profile dev_account]
region=us-east-1
output=json
[profile dev_account_readonly]
region=us-east-1
output=json
source_profile=dev_account
role_arn=ROLE_ARN
mfa_serial=MFA_SERIAL_ARN

To verify if the role has been configured or not, you can do so by running below command:

aws iam get-role --role-name dev_account_readonly --profile dev_account_readonly

Replace the role and profile name with the names you set. All the information regarding the IAM Role will be returned in the JSON format.

That’s all you require for setting the profiles in your CLI and use the Assumed Role🎉!

Workaround with MFA and AWS-SDK to automate manual work

Tejas — Tue, 29 Aug 2023 05:51:22 +0000

Problem Statement

Recently, due to a misconfiguration in the source system, the ETL developed by the team failed and all the extracted data ended up in the retry mechanism because the ETL function was not expecting this kind of information. However, not all the information was supposed to be discarded, as some of it was genuine and had a customer impact. Therefore, it was not possible to avoid or discard all the information at once by deleting all records.

So, it ended up after to and fro discussion, that all the retry records (~2500) will need to be checked manually and delete which are not genuine and update retry count in DynamoDB Table which needs to be retried as failed records will happen in sequence for each type and was given strict instruction that you can’t just “delete files”.

All this information and data was stored in the AWS cloud (DynamoDB, S3). Therefore, to check all 2500 files one by one is very tedious and cumbersome process. So, after using some Sheldon brain, we came up with plan that we will create some code using TypeScript and AWS-SDK as we had access so, it can be automated. So, when we wrote a script, it came as blocker that due to Multi-Factor Authentication (MFA) it was not allowing us to process it as we had specific roles and profiles to be used. Thereby, so we figure it out a way, we can use a MFA and automated this manual work which will be in step-by-step in this blog.

Pre-requisite for the solution to work

Now that you have the problem at hand and you so want to have the solution but hold on! There are bunch of items you need to have configured to make it work for you.

AWS CLI: You need to have AWS CLI installed on your machine. This is required to set up the profile to use for AWS-SDK. Here’s the link to install AWS CLI on your system.
AWS Profile: After installing AWS CLI, AWS Profile needs to be configured in the CLI so that AWS-SDK can use it to make API calls. Here’s a guide on how to configure profiles in AWS CLI.
Note: Please make sure you have both config and credentials file inside your aws directory else AWS SDK would complain for not finding the config file.

The solution to make our lives easier

Okay, now that you know both the problems and pre-requisite to solve it, let’s dive deep into the codebase that’s doing all the magic.

First we need to have clients installed for DynamoDB, which will be used to make API calls to the table and other module to generate credentials. So, let’s do it:

yarn add @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb @aws-sdk/credential-providers

Cool, now you have got all the things required to make your lives easier. So let’s start writing some TypeScript.

First we need to generate the credentials to pass it to the DynamoDB client. So, let’s write the function that returns the credentials.

import { fromIni } from "@aws-sdk/credential-providers";

function getCreds() {
  try {
    return fromIni({ // returns the credentials using your AWS Profile
      profile: "aws-profile",
      mfaCodeProvider: async serial => {
        const mfaCode = await prompt(`Type your AWS MFA code: ${serial}`);
        return mfaCode;
      },
    });
  } catch (err) {
    console.error(err);
    throw err;
  }
}

What’s happening here?

fromIni function is reading your credentials and config files that are stored in your aws directory. It will have your access and secret keys and other necessary configuration. Mixes it with your MFA code and voila, you have the credentials to make the API calls for which you have access to.
Hmm, but how would you make the terminal ask you for the MFA code? For that NodeJS has an internal package called readline . Using this we will prompt for the MFA code. Let’s look into the prompt function:

import * as readline from "readline";

function prompt(query: string): Promise<string> {
    // first create the readline interface
  const rl = readline.createInterface({
      input: process.stdin,
    output: process.stdout,
  });

    // prompt to answer a question
  return new Promise(resolve =>
    rl.question(`${query}\n`, ans => {
      rl.close();
      resolve(ans);
    })
  );
}

Just there. Let’s add the functions to make API calls to DynamoDB table and update the records in bulk, which is not natively supported by DynamoDB.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  ExecuteStatementCommand,
  DynamoDBDocumentClient,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

const updateOrder = async (
    order: SomeSchema,
    docClient: DynamoDBDocumentClient
  ): Promise<void> => {
    const updateCommand = new UpdateCommand({
      TableName: "some-table-name",
      Key: {
        id: order.id,
      },
      // update retryCount and bucketName for each unique record
      UpdateExpression: "set retryCount = :x, bucketName = :y",
      ExpressionAttributeValues: {
        ":x": 0,
        ":y": "someS3BucketName",
      },
    });
    await docClient.send(updateCommand);
  };

async function fetchAndUpdateOrders(): Promise<void> {
  const client = new DynamoDBClient({
    credentials: getCreds(), // retrieve the credentials
  });
  const docClient = DynamoDBDocumentClient.from(client);
  const command = new ExecuteStatementCommand({
    Statement: 'SELECT * FROM "some-table-name"',
  });

  const data = await docClient.send(command); // store all the retrieved data in this variable
  const orders = data.Items as Array<SomeSchema>;
  console.log(`Found a total of ${orders.length} orders`);
  await Promise.all(
        // loop through all the orders and update them
    orders.map(order => {
      console.log(
        `Updating order with id: ${order.id}`
      );
      return updateOrder(order, docClient);
    })
  );
}

That’s it. You have the solution to your problem. Now just call the fetchAndUpdateOrders function and your problem’s gone forever(until something else breaks again!).

fetchAndUpdateOrders()
  .then(() => console.log("Finished updating the records!"))
  .catch(err => {
    console.error("Something bad happened!");
    console.error(err);
  });

At the end, you will have the whole solution looking like this:

import { fromIni } from "@aws-sdk/credential-providers";
import * as readline from "readline";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  ExecuteStatementCommand,
  DynamoDBDocumentClient,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

function getCreds() {
  try {
    return fromIni({ // returns the credentials using your AWS Profile
      profile: "aws-profile",
      mfaCodeProvider: async serial => {
        const mfaCode = await prompt(`Type your AWS MFA code: ${serial}`);
        return mfaCode;
      },
    });
  } catch (err) {
    console.error(err);
    throw err;
  }
}

function prompt(query: string): Promise<string> {
    // first create the readline interface
  const rl = readline.createInterface({
      input: process.stdin,
    output: process.stdout,
  });

    // prompt to answer a question
  return new Promise(resolve =>
    rl.question(`${query}\n`, ans => {
      rl.close();
      resolve(ans);
    })
  );
}

const updateOrder = async (
    order: SomeSchema,
    docClient: DynamoDBDocumentClient
  ): Promise<void> => {
    const updateCommand = new UpdateCommand({
      TableName: "some-table-name",
      Key: {
        id: order.id,
      },
      // update retryCount and bucketName for each unique record
      UpdateExpression: "set retryCount = :x, bucketName = :y",
      ExpressionAttributeValues: {
        ":x": 0,
        ":y": "someS3BucketName",
      },
    });
    await docClient.send(updateCommand);
  };

async function fetchAndUpdateOrders(): Promise<void> {
  const client = new DynamoDBClient({
    credentials: getCreds(), // retrieve the credentials
  });
  const docClient = DynamoDBDocumentClient.from(client);
  const command = new ExecuteStatementCommand({
    Statement: 'SELECT * FROM "some-table-name"',
  });

  const data = await docClient.send(command); // store all the retrieved data in this variable
  const orders = data.Items as Array<SomeSchema>;
  console.log(`Found a total of ${orders.length} orders`);
  await Promise.all(
        // loop through all the orders and update them
    orders.map(order => {
      console.log(
        `Updating order with id: ${order.id}`
      );
      return updateOrder(order, docClient);
    })
  );
}

fetchAndUpdateOrders()
  .then(() => console.log("Finished updating the records!"))
  .catch(err => {
    console.error("Something bad happened!");
    console.error(err);
  });

Say hello to seamless bulk updates & goodbye to manual tweaks!