This is how you can test your cfn-guard rules

In my previous blog, How do you prove that your infrastructure is compliant.

I explained how you can prove your infrastructure is compliant using CloudFormation Guard.
But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository CloudFormation Guard.

You will notice that the project itself offers a testing framework.
Alright! Let’s build a ruleset and write some tests for it!

Let’s build a ruleset for S3

We will start with a sample for S3, create a file called: rules/s3.guard.

let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]

rule BucketEncryption when %buckets !empty {
  %buckets.Properties {
    BucketEncryption.ServerSideEncryptionConfiguration[*] {
      ServerSideEncryptionByDefault.SSEAlgorithm IN ["AES-256", "aws:kms"] <<Ensure all S3 buckets use encryption-at-rest>>
    }
  }
}

Now we will write the test, called: rules/s3_tests.yaml.

---
- name: "Skip if not present"
  input:
     Resources: {}
  expectations:
    rules:
      BucketEncryption: SKIP
- name: "No encryption used"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
  expectations:
    rules:
      BucketEncryption: FAIL

- name: "Bucket with KMS Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: aws:kms
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with AES-256 Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: AES-256
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with UNKNOWN Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: UNKNOWN
  expectations:
    rules:
      BucketEncryption: FAIL

Now it’s time to run the tests:

cfn-guard test --rules-file rules/s3.guard --test-data rules/s3_tests.yaml

This will give the following output:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

You might be thinking, that looks nice! But the output is quite verbose, and you need to execute a command per rule/test set.

It gets even worse when a test does fail. For this example I altered the rules/s3.guard file to use aws:kmz instead of aws:kms.

Then it looks like this:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  FAILED Rules:
    BucketEncryption: Expected = PASS, Evaluated = FAIL

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

If you look at Test Case #3 it says: BucketEncryption: Expected = PASS, Evaluated = FAIL. That makes it hard to spot what test failed.

cfn-guard-test

For this reason I created a python package called cfn-guard-test. This package

makes it easier to run many rule/test sets. This is especially useful in the CI/CD pipelines.

Please read the repository for the installation instructions.

Let’s run the tests again!

cfn-guard-test --verbose

The output now looks something like:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 5
Failed 0

That is a nice and clean overview. It lists the number of passed and failed tests. For the next example I altered the

rules/s3.guard file to use aws:kmz instead of aws:kms. This will again trigger a failure:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 4
Failed 1

Rule BucketEncryption failed on #3 "Bucket with KMS Encryption" in rules/s3.guard

Well you have to agree. That immediately sums up how many passed and failed. And more it shows that the BucketEncryption

rule fails. And it failed on the test case "Bucket with KMS Encryption".
You don’t want to spend time figuring out what is wrong. This is why we use tooling, to make our life easier.
This tool also has the ability to generate JUnit report.

cfn-guard-test --verbose \
  --junit-path "reports/cfn-guard.xml"

Conclusion

cfn-guard-test can help you run many tests. And get an aggregated report of

the results. This tool will help you detect bugs in your rules easier and quicker.