DEV Community: Toshimitsu Takahashi

Seamless Local Development with MFA-Enabled Roles on AWS

Toshimitsu Takahashi — Mon, 03 Jun 2024 15:10:23 +0000

This article is a translation of https://zenn.dev/tilfin/articles/ab53ae77a8378f.

The Challenge

Many of us have an AWS account for our main login portal and separate AWS accounts for each project. We often need to assume roles to obtain temporary credentials and access resources like S3 and DynamoDB on AWS from our local development environment.

The challenge lies in the short-lived nature of these credentials. In a development environment where the server automatically reloads upon code changes, manually renewing credentials and restarting the server every hour becomes a tedious task. This becomes even more cumbersome with MFA (Multi-Factor Authentication), requiring manual one-time password entry.

This article outlines the steps I took to streamline this process and enable seamless development without interruptions.

Temporary Credential Issuance

While aws sts assume-role is the standard AWS CLI command for this purpose, I've developed a tool called swrole to simplify this process.

https://github.com/tilfin/homebrew-aws/tree/master?tab=readme-ov-file#swrole

swrole 1.1 Release

This latest update introduces multiple ways to handle one-time passwords (OTP). While version 1.0 only supported interactive input, version 1.1 allows you to pass the OTP directly as a command argument using -t 123456. Additionally, you can now define generate_token within the target profile section of your ~/.aws/config file. This enables automatic OTP retrieval from services like 1Password.

~/.aws/credentials

[my-company]
aws_access_key_id = XXXXXXXXXX
aws_secret_access_key = XXXXXXXXXX

~/.aws/config

[profile my-company]
region = ap-northeast-1

[profile project-dev]
role_arn = arn:aws:iam::111111111111:role/developer
mfa_serial = arn:aws:iam::000000000000:mfa/member
source_profile = my-company
region = ap-northeast-1
generate_token = op item get "AWS Account" --otp

In this example, generate_token is configured to fetch the OTP for "AWS Account" using the 1Password CLI.

Leveraging Process Credential Provider in AWS SDKs

During my research, I discovered the credential_process configuration option in ~/.aws/config. This allows AWS SDKs to execute a specified command and utilize the resulting JSON output (from standard output) as credentials.

https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html

Recognizing its potential, I've incorporated this functionality into swrole version 1.1. Using the -j option now outputs the credential JSON in the format expected by the Process credential provider.

However, directly using credential_process with the project-dev profile won't work seamlessly due to the SDK's credential prioritization. We need to define a separate profile specifically for application development.

~/.aws/config

[profile app-dev]
credential_process = swrole -j project-dev

~/.aws/credentials

[app-dev]
region = ap-northeast-1

While other SDKs are untested, using ~/.aws/credentials for credential_process works with AWS SDK JavaScript v3.

Now, you can run your development server with this AWS profile, triggering the seamless credential fetching process:

$ AWS_PROFILE=app-dev pnpm dev

Important: Ensure your AWS SDK utilizes proper credential caching to avoid issues with concurrent requests using the same one-time password.

Additional Notes

I encountered some challenges with AWS SDK JavaScript v3's credential caching behavior related to this approach. You can find a detailed explanation in this article:

Optimizing Credential Configuration in AWS SDK for JavaScript v3: Understanding Cache Mechanisms and Best Practices

Furthermore, this method eliminates the need for manual OTP input even with the AWS CLI itself.

Optimizing Credential Configuration in AWS SDK for JavaScript v3: Understanding Cache Mechanisms and Best Practices

Toshimitsu Takahashi — Mon, 03 Jun 2024 15:04:49 +0000

This article is a translation of https://zenn.dev/tilfin/articles/56f8dc56b83901.

Background

The AWS SDK for JavaScript v3 introduced a significant shift in client configuration compared to v2. While v2 allowed for global SDK settings, v3 adopts a per-client instance runtime configuration approach. Consequently, credential retrieval, which was internally cached in v2 even with randomly generated clients for each AWS resource, now presents potential challenges in v3 with frequent credential requests.

Delving into Credential Retrieval Logic

The simplest approach involves directly setting AWS access and secret keys as credentials in the constructor arguments of AWS resource clients. However, this method is not ideal for real-world application development. Applications deployed on EC2 or ECS typically retrieve credentials from instance or container metadata services, while Lambda functions often obtain them via environment variables. The AWS SDK v3 implements these credential supply logics as Credential Providers.

You can also specify a Credential Provider for the client's credentials. Details are described on the following page:
https://github.com/aws/aws-sdk-js-v3/tree/main/packages/credential-providers

If nothing is specified for credentials, the function set in credentialDefaultProvider is executed first to generate a Credential Provider, which is then used as credentials. The default credentialDefaultProvider is defaultProvider. It attempts various acquisition methods. By using this method, you can automatically refer to ~/.aws/credentials in a local environment and metadata in a deployed environment.

To explicitly specify the implementation of defaultProvider, use fromNodeProviderChain.

Credential Caching Mechanism

Credential retrieval occurs each time a client invokes a command. However, Credential Providers, which implement the various retrieval logics provided by the AWS SDK, do not have a caching mechanism. The only exception is the memoize function of @smithy/property-provider used internally by defaultProvider. Furthermore, the re-acquisition logic for time-limited credentials, including session tokens, is also implemented only here.

https://github.com/aws/aws-sdk-js-v3/blob/main/packages/credential-provider-node/src/defaultProvider.ts#L59

https://github.com/smithy-lang/smithy-typescript/blob/main/packages/property-provider/src/memoize.ts#L46

What are the Best Practices?

Based on the insights so far, in most cases where you are implementing an application that operates within the same AWS account, "using defaultProvider", or "specifying nothing", is the best approach. However, there is a potential pitfall here.

As mentioned at the beginning, in v3, since each client has a different Credential Provider, caching is also done on a per-client basis. For example, in a batch processing loop where the code "creates a client and retrieves an item", there is a possibility that credential acquisition will fail. Below is an issue where an error occurred because the access limit of EC2's metadata endpoint was reached.

https://github.com/aws/aws-sdk-js-v3/issues/4867

As a workaround, the issue's comments suggest caching (reusing) the instances of each AWS client themselves. However, in applications that rely on many AWS services and where command transmissions to them occur intensively, this may not be avoidable. There is also an improvement proposal for this.

https://github.com/aws/aws-sdk-js-v3/issues/4612

Conclusion

Currently, the best practice is to reuse defaultProvider. Since fromNodeProviderChain is provided by the SDK, generate it at the module level as shown below and reuse it.

import { fromNodeProviderChain } from "@aws-sdk/credential-providers";

export const credentialProvider = fromNodeProviderChain();

new S3Client({ credentials: credentialProvider });

new DynamoDBClient({ credentials: credentialProvider });

This will prevent the retrieval process from being concentrated internally.

References

https://zenn.dev/luma/articles/bd3c59b3d7682d

The recommendation of structured logging and the reason I have made Ougai for Ruby

Toshimitsu Takahashi — Sun, 20 May 2018 14:14:07 +0000

Structured logging

The structured logging is to make a log that is easy to process mechanically.

Comparison of usual logs and structured logs

The usual logs are basically just timestamps, level, and message strings. Information to be the context of events (events) to be kept as logs is appropriately embedded in messages. It is easy for human to read on a console.

In the structured log, information which becomes the context embedded in the message is independently given to the field of the log structure. Therefore, it is easier to analyze later. And the output logs are often text based JSON.

Let's compare this. The log of normal Ruby Logger and my own Ougai will be as follows. It is a log that "user created article" (articles shaved).

logger.info "User created article  (user_id=#{user.id} article_id=#{article.id}"

I, [2018-05-13T17:51:08.772542 #6253]  INFO -- : User created article  (user_id=123 article_id=45)

logger.info "User created article", user_id: user.id, article_id: article.id

{"pid":6253,"level":20,"time":"2018-05-13T17:52:25.147+09:00","msg":"User created article","user_id":123,"article_id":45}"

※ It differs from the default format

As you can see, you have to embed the normal logs and convert the incidental information into strings. On the other hand, structured logs are longer by JSON effect, so it is difficult to read in the console. However, it is not a problem if you use log viewer with switchover in the operating environment of the formatter or parse mechanism.

Structured logs are easier to analyze is that, for example, if you want to extract only the logs of certain users, simply grep "user_id=10" will cause to hook entries whose user_id is like 101 in normal logs. On the other hand structured logs (mainly using JSONPath) can be easily and reliably filtered by like a $.user_id = 10.

Log management service and structured log

SaaS for log management and analysis such as AWS CloudWatch Logs, Stackdriver Logging on Google Cloud and Loggly, Logentries all support structured logs, or you can not make use of those services unless you do. However, since the library itself provided by each service does not correspond to the structured output, I think that it is best to relay by Fluentd and send it by present situation best.

The reason for making Ougai

Originally I wrote the front API server in Node.js and participated frequently in the project to create backend management service with Ruby / Rails. Node.js had a famous JSON logger named Bunyan, and I liked it. But Ruby has libraries that handle several structured logs, but each library was as a framework. Node.js itself has functions for console output like console.info, but Ruby has its own Logger, though. In another micro service, each service is often implemented in multiple languages, but if possible, I wanted to unify the format in order to analyze the log cross-sectionally.

With these backgrounds, I built my own logger based on Ruby's original Logger, which is a logger that has the Bunyan compatible log output format described above. That is Ougai. (I named Ougai from a Japanese literary person after Bunyan was named from the English literary.)

GitHub: tilfin/Ougai - A Ruby structured logging is capable of handling a message, custom data or an exception easily and generates JSON or human readable logs.

Since it is an extension of Ruby's original Logger class, even if it is introduced suddenly, it does not crash simply by appearing in the message field. You can gradually move the context to an independent field. The format is standard, for JSON output it is Pino compatible with Bunyan of Node.js, Readable which is colorfully easy to read at the terminal using awesome_print is included. Of course for JSON you can browse with dedicated log viewer commands owned by Bunyan and Pino respectively. There is also a function to make common information easy to put in by hook.

https://github.com/tilfin/ougai/wiki

The wiki also contains examples of settings to use it in combination with Rails, Sidekiq, Fluentd etc.

https://rubygems.org/gems/ougai

Transgate is Agent-based taskflow framework for Node.js

Toshimitsu Takahashi — Thu, 23 Nov 2017 12:56:09 +0000

I made Transgate Agent-based taskflow framework in Node.js .

Why did I make it?

Because the various flows got messed up during I wrote a program to operate my home appliances. They are to obtain temperature and humidity from dyson cool fan regularly, to save it in the database, and to handle messages coming from Google Home / Assistant + IFTTT and operate IRKit. According to the temperature, I also wanted to automatically operate the air conditioner via IRKit. How do you write them?

What thing？

Suddenly I want you to imagine sorting of baggage such as airport. The agent receives items from the gate, processes them, and sends them to another gate. The agent does not know what is going on the other side of the gate. When the agent comes up with an empty item, it finishes the work. The image of the framework architecture looks like this.

The agent can receive items from the gate and send new items to another gate. Item is a simple Object. An agent can concentrate on its own task. So even if the number of previous process or next process increases or decreases, it moves without problems if the schema of the item does not change. And input / output is simple so unit testing is easy. Since the agent does not know the substance of the gate, it can easily be replaced with the input source gate as the stub and the output destination gate as the mock.

Actors in this framework

Gate is an endpoint of Input/Output. For example, file storage, database, queue or API service.
Agent is a worker to process an item between Input/Output gates and does not know anything opposite gates.
Item is an entity as each task target, to be exchanged between gates, and an Object or a JSON. null indicates the terminator.

An example

Let's explain through the home control program that triggered making this framework. By the way this program is running as a daemon on Raspberry PI in my shoebox.

Flow diagram

Main program (main.js)

const {
  Agent,
  HttpClientGate,
  HttpServerGate,
  IntervalGate,
  JointGate,
  StdoutGate,
  duplicator,
  mixer,
} = require('transgate');

const pino = require('pino')();
const config = require('konfig-yaml')();

const MongoGate = require('./lib/mongo_gate');
const IRKitGate = require('./lib/irkit_gate');

// Agent
const AnalysisCommander = require('./lib/analysis_commander');
const DysonCoolLinkRecorder = require('./lib/dyson/cool_link_recorder');
const EnvironmentalAnalyzer = require('./lib/environmental_analyzer');

// Gate
const slackGate = new HttpClientGate({ endpoint: config.slack.webhook_url });
const iftttGate = new HttpServerGate({ port: config.port });
const irkitGate = new IRKitGate(config.irkit.endpoint);
const intervalGate = new IntervalGate(60);
const mongoGate = new MongoGate(config.mongodb.endpoint, config.mongodb.collection);
const drToEaGate = new JointGate();

(async () => {
  try {
    await Agent.all(
      new AnalysisCommander(iftttGate, { irkitGate, slackGate }),
      new DysonCoolLinkRecorder(intervalGate, duplicator(mongoGate, drToEaGate)),
      new EnvironmentalAnalyzer(drToEaGate, { irkitGate, slackGate }),
    );
  } catch(err) {
    pino.error(err);  
    await iftttGate.close();
    await mongoGate.close();
  }

  intervalGate.clear();
})()
.catch(err => {
  pino.error(err);
});

7 Gates

slackGate posts a text message to slack. Even if it is not specially implemented, it will be an instance of HttpClientGate. The item JSON is { "text": "<text message>" } 。
iftttGate uses JSON received from IFTTT's webhook as an item. The item JSON is { "target": "TV", "text": "<speaking words>" } 。
irkitGate instructs infrared transmitter with HTTP interface. The item JSON is { "command": "celling_light_off" } 。
intervalGate creates items at regular intervals. Item is {" time ": <Date instance>}. In this case run agent processing every minute.
mongoGate registers items sent to MongoDB's designated collection.
drToEaGate is a joint flow of items from DysonCoolLinkRecorder (described later) to EnvironmentalAnalyzer.

3 Agents

AnalysisCommander receives JSON from the IFTTT webhook as an item and specifies the infrared signal to be sent to IRKit from the operation target and text. Post it when slack can not interpret wording.
DysonCoolLinkRecorder gets temperature and humidity from the Dyson PureCoolLink fan every 1 minute and sends it to the gate which becomes the joint and write to MongoDB across the duplicator.
When EnvironmentalAnalyzer exceeds the threshold value from the temperature through the joint, it requests IRKit to operate the air conditioner. When you operate automatically, record it in slack.

Agent implementation

Create a subclass of Agent. Write the code of processing the item on received in the main method and sending a new item to the specified gate. We use the before / after hook method to control (start / stop) the initialization process and another process (eg headless chrome) here.

The following is an implementation example of EnvironmentalAnalyzer. When the room temperature becomes 17 degrees Celsius degree or less, turn on the air conditioner.

const { Agent } = require('transgate');

module.exports = 
class EnvironmentalAnalyzer extends Agent {
  async before() {
    this._preTemp = null;
    this._airconAlive = false;
  }

  async main(item, { irkitGate, slackGate }) {
    const curTemp = item.temp;

    if (this._preTemp && this._preTemp > 17 && curTemp <= 17) {
      if (!this._airconAlive) {
        await irkitGate.sendAll({ command: 'aircon_on' });
        this._airconAlive = true;
        await slackGate.send({ text: `Turn on aircon because temp is down to ${curTemp}` });          
      }
    }

    this._preTemp = curTemp;
  }
}

The reason why the constructor and the input gate are concealed is for the implementation of the specification that when the agent receives null, it sends null to the next gate and ends himself.

The features

Suitable for complex daemons and batch programs.
Not suitable for processing large volume because it is not assumed that the same agent is running in parallel.
Gates and agents that appear and the task flow of items can be defined in the main program. Therefore, the whole can be grasped by only it.
Processing of agents can be written in sync with async / await in a pseudo manner. Even if the number of agents increases, they will not be heavy like thread-based.
Since it is easy to replace the gate, it is easy to write unit tests of the agent and to confirm partial execution.

Predicting answers to questions

Will the referenced services be all gates?

It's no. Between gates is limited to one way. The agent does not know the detail specification of the gate. In other words, you can not throw a request and get a response to it. It is possible to loop, not round-trip, The response is determined for the request to send out because it is stateless. The gate becomes the part that the trigger to the agent and the part where the agent sends the result.

How to inform the kicker when the series of the flow is over?

The queue system needs to send a completion notice when the task is completed. In such a case, you can turn the flow by giving the item its context. Make sure the last gate is responsible for sending completion notification.

Should the logger be a gate?

If logs are the outputs themselves, they should be gated. Then you can easily replace the gate with something that will be jointed to the Agent later and then throw it to the log analysis service from there.

How much logic can we include in the gate?

The gate should be as simple as possible. The agent is designed to make it easier to test. But if you put the logic in the gate itself, you can not replace the input / output destination and test it. However, if it is simple and the common logic in the project, it may be implemented in the gate. If it is complicated, you should make an agent for it, connect the agents by a joint gate.

I would be pleased if you are interested in Transgate.

Japanese version