DEV Community: Michael Wu

Level up compliance confidence with AWS Config

Michael Wu — Fri, 15 Oct 2021 11:23:31 +0000

With the ever increasing adoption of moving workloads to public cloud providers like AWS, satisfying regulatory compliance has emerged as one of the most demanding requirements for businesses that operate on the cloud.

As an AWS customer, this means that you need to establish a proper governance model for your business operations to continue, particularly for those that operate under specific regulations. This "model", is conceptualised under AWS Control Tower as a collection of guardrails. A guardrail is simply a rule that describes what is allowed or disallowed in your AWS account. Now, a guardrail can be of two forms, a preventive guardrail or a detective guardrail.

A preventive guardrail basically enforces a policy, such as using SCP to deny IAM users to specific APIs, so that those users in that AWS account cannot perform certain actions.
On the other hand, a detective guardrail reports a violation after evaluations of a resource deem it non-compliant under the rule.

Now, not every AWS customer uses Control Tower, or even AWS Organizations. This blog post aims at walking you through how to leverage AWS Config to set up detective guardrails in your AWS account.

Config rule

In order to detect non-compliant resources, you need to first define what a compliant resource should look like, and this is done via a Config Rule.

A config rule evaluates resources based on desired configuration settings. For example,

SNS topics should have at least one active subscriber.

or,

IAM users should not be inactive for more than 90 days.

You can define as many rules as you like. Since the main purpose of AWS Config is to track resource inventory and changes in an AWS account, it maintains an ever growing list of built-in rules that you can choose to enable to monitor resources of your interest.

You can also create your own rules, backed by a lambda function. Custom rules offer greater flexibility to monitor resource types that are currently not supported by AWS Config. One thing worth noting in relation to security is that you can edit the lambda resource-based policy to restrict the lambda invocation permission only to the config rule that's calling it, instead of granting access to the entire Config service principal. AWS recommends it as a security best practice for developing custom rules.

Okay we have rules, now what?

That's it! That's all you need to do to set up detective guardrails. Now that you are armed with a bit of automation to monitor resources on your behalf, wouldn't it be nicer to take it further by rectifying the non-compliant resources found? This is called a remediation. AWS Config allows you to apply remediation automation as an SSM document to respond to evaluation results from config rules. Like managed Config rules, SSM also maintains a list of automation runbooks that AWS Config can use for resource remediation. You are also free to create your own automation runbooks as SSM documents. How to create an automation runbook is outside the scope of this post, but AWS has an excellent walkthrough of authoring a custom runbook.

Combining rules and remediation?

Yes, AWS Config supports bringing config rules and remediation into one deployment, referred to as a conformance pack. A conformance pack typically comprises a bunch of config rules and remediation actions, declared in a yaml file, similar to a Cloudformation template, but with a few catches. The deployment of a conformance pack is underpinned by Cloudformation, you can get the underlying stack arn by describing the conformance pack.

What if I have multiple accounts?

Both AWS Config rule and conformance pack deployments have APIs that target an AWS Organization. On top of simplified deployment, it has added benefits of bringing newly joined accounts into line automatically, and deleting the organizational guardrails when an account leaves the Organization. Organizational config rules and conformance packs are easily identifiable via a special prefix added to the resources deployed into a member account. Cleaning up organizational resources can sometimes get messy because you can only issue the delete API from the management account or the delegated admin account. Given my experience, oftentimes the org level delete API will report failed because of failures on deletion in one or more member accounts. You need to continuously issue the same API from the management or delegated admin account until all members are cleared.

Parallel task error handling in Step Functions

Michael Wu — Tue, 17 Nov 2020 07:10:13 +0000

The Problem

When you run concurrent tasks in step functions, either as a Parallel state or a Map state, you might want to wait until all tasks complete, regardless of whether they would succeed or fail. By default, Map or Parallel state will wait for all to succeed before ending or proceeding to the next state, however they will be terminated if a single iteration or branch has failed, and all other in-flight executions will be stopped.

While this behaviour might echo the philosophy of fail-fast, there are scenarios in which you might want to hold on terminating such an overarching state should anything fail. For example:

Given 3 branches of workload I want to run in parallel, I want a converged response at the end, containing response from each branch, be it a success or failure.
Given 5 items in an array that I want to run through a set of steps concurrently, I want to know which ones succeed and which ones fail (and reasoning).

The Solution

The solution is to handle the possible exceptions and use a Pass state to propagate result.

`Parallel` state

To demonstrate the problem in a parallel state, let's look at what happens if a branch fails.

In this example, you will see I have two branches of workload, long success represents a long-running, eventually successful workload, while quick fail represents a failed workload that fails before long success completes. The screenshot shows how the parallel state (and the entire state machine) has been cancelled because quick fail branch has failed while long success has been stopped too (greyed out). This is inline with what AWS documentation says

If any branch fails, because of an unhandled error or by transitioning to a Fail state, the entire Parallel state is considered to have failed and all its branches are stopped. If the error is not handled by the Parallel state itself, Step Functions stops the execution with an error.

Now let's see what happens if we handle whatever error is thrown from quick fail and use a Pass state to return the result for this branch.

You can see that quick fail step becomes amber, which means it's in a Caught Error status, and it's got a following step of handle failure which is a Pass state that simply returns an output and ends the branch execution.

The big difference here is that while the branch of quick fail has completed, the overall parallel state is still in-flight, as indicated by the blue colour, due to the ongoing long success branch. Eventually, the long success branch will complete, and together with the output from handle failure, both execution results will be aggregated into the Final State.

The definition of the state machine with error handling would look something like this



{
  "StartAt": "Parallel",
  "States": {
    "Parallel": {
      "Type": "Parallel",
      "Next": "Final State",
      "Branches": [
        {
          "StartAt": "long success",
          "States": {
            "long success": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:ap-southeast-2:123456789098:function:long-success",
              "End": true
            }
          }
        },
        {
          "StartAt": "quick fail",
          "States": {
            "quick fail": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:ap-southeast-2:123456789098:function:quick-fail",
              "Catch": [
                {
                  "ErrorEquals": [
                    "Exception"
                  ],
                  "Next": "handle failure",
                  "ResultPath": "$.error"
                }
              ],
              "End": true
            },
            "handle failure": {
              "Type": "Pass",
              "End": true
            }
          }
        }
      ]
    },
    "Final State": {
      "Type": "Pass",
      "End": true
    }
  }
}

Given the snippet above, you would expect to see an array output from Final State with two elements in it, something like this



[
  <long-success-response>,
  {
    <input-json-blob>,
    "error": {
      "Error": "Exception",
      "Cause": "{\"errorMessage\": \"quick fail function has failed\", \"errorType\": \"Exception\", \"stackTrace\": [\"  File \\\"/var/task/lambda_function.py\\\", line 6, in lambda_handler\\n    raise Exception(\\\"quick fail function has failed\\\")\\n\"]}"
    }
  }
]

From here, you have the ability to process response from each individual branch without failing the entire state machine.

`Map` state

Similarly, you can achieve the same result in the Map state, if your parallelism is dynamic.

A successful definition using this approach would look something like this



{
  "StartAt": "Map",
  "States": {
    "Map": {
      "Type": "Map",
      "ItemsPath": "$.array",
      "ResultPath": "$.array",
      "Next": "Final State",
      "Iterator": {
        "StartAt": "mapper",
        "States": {
          "mapper": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:ap-southeast-2:123456789098:function:sfn-mapper",
            "Catch": [
              {
                "ErrorEquals": [
                  "Exception"
                ],
                "Next": "handle mapper error",
                "ResultPath": "$.error"
              }
            ],
            "ResultPath": "$.data",
            "End": true
          },
          "handle mapper error": {
            "Type": "Pass",
            "End": true
          }
        }
      }
    },
    "Final State": {
      "Type": "Pass",
      "End": true
    }
  }
}

In this example, I use "ResultPath": "$.data" and "ResultPath": "$.error" to log success and fail under the key data and error in the response to make them easy to distinguish. The mapper state runs a lambda function that either returns a success response or throws an error, depending on the execution input. The input would look something like this



{
  "array": [
    {
      "wait": 5,
      "throw": true,
      "name": "one"
    },
    {
      "wait": 10,
      "throw": false,
      "name": "two"
    }
  ]
}

The lambda function sfn-mapper looks like this (using Python as an example)



import json
import time

def lambda_handler(event, context):
    time.sleep(int(event["wait"]))
    if bool(event["throw"]):
        raise Exception(f"{event['name']} has thrown an error")

    return f"{event['name']} has succeeded after {event['wait']} seconds"

When you run such state machine, the end state will look similar to this

Since we are handling exceptions on each iteration, the entire state machine has succeeded. If you check out the response from Final State, you will see something like this based on the above input example.



{
  "array": [
    {
      "wait": 5,
      "throw": true,
      "name": "one",
      "error": {
        "Error": "Exception",
        "Cause": "{\"errorMessage\": \"one has thrown an error\", \"errorType\": \"Exception\", \"stackTrace\": [\"  File \\\"/var/task/lambda_function.py\\\", line 7, in lambda_handler\\n    raise Exception(f\\\"{event['name']} has thrown an error\\\")\\n\"]}"
      }
    },
    {
      "wait": 10,
      "throw": false,
      "name": "two",
      "data": "two has succeeded after 10 seconds"
    }
  ]
}

Notice the error and data key respective to the corresponding iteration.

I hope you've enjoyed this walk-through. Error handling in Parallel or Map state could be really useful in instances where you want greater control over how to process individual response in these states.