DEV Community: Jeroen Willemsen

Why WrongSecrets moved to the OWASP Github Organization

Jeroen Willemsen — Tue, 01 Nov 2022 08:08:19 +0000

After 1 year of active development under my personal Github Id commjoen, it is time to migrate our OWASP project WrongSecrets to the OWASP Github organization.

And now you might wonder why, why...now?

Prove Value First

We wanted to wait to move the project to the OWASP organization. We were curious if this project would be liked, maintained, or contributed to after its inception. Instead, we wanted to test it for a year and get feedback. Projects under the OWASP GitHub organization should not be "dead weights" or "pet experiments", which would lower the organization's value. The OWASP organization should contain solid projects with active maintainers aiming to deliver quality. Moving a project too early with some noise will not help the organization. Now, the time is right to move this project to the OWASP organization as it has proven its value to many!

Your Contribution safe with OWASP

We often heard that people were reluctant to commit to (a fork of) the old repo under our personal accounts. After all: what would hold us back from taking it away from the internet, and creating a commercial product for ourselves, while you put your sweat and tears in as a volunteer?
By moving this project to the OWASP GitHub organization, we hope to remove this fear of potential new contributors and make it much more open to all who might be interested.
We would love to pursue the new OWASP "Production" status, possibly in the future. To do so, we must also bring our repositories under the OWASP Github organization.

No Commercial Interest

Please note that, as we said at the beginning, we have no commercial interest in the project. It was born out of our desire to have fun and to educate - we hope you can have a lot of fun with it!

What is Next?

There are tons of things next, have a look at our github issues to see what the future has in store for this awesome project. Feel free to reach out to us through Slack as well!

Special Thanks

We would like to thank Harold Blankenship for helping us to migrate our repositories to the OWASP organization. Harold has shown to be a great help in many needs of this project. It would not have gone so fast without him.

6 Pointers on Application Logging

Jeroen Willemsen — Wed, 05 Jan 2022 08:58:57 +0000

When you have a service running somewhere you need to find out whether it is functioning correctly. Besides the possible tests, liveness checks, metrics, you can use application logging. But what makes an application log “Good”? In this blog we discuss 6 pointers on application logging to help you on your way!

1. Purpose is everything!

Before you start writing any log-output, first step back a little. Now ask yourself: where are you logging for? What is the goal and who is your intended audience? Do you intent to have a log which is targeted at application troubleshooting? Or is this a log used to capture all relevant events as part of an audit-log?
Once you have the purpose and the audience of the log defined, you can give it a go. Below you can find a couple of different purposes for logging. Of course, there are many more you could think of, but let’s just pick up a few:

Purpose case: logging for application maintenance

When you setup logging for your application maintenance, you should think at least of:

Do you want to inform people about certain steps which are executed by the application?
Which exceptions or errors in your application are actually worth logging in terms of faulty code paths? Does informing about them lead to better maintenance and/or business value?
When is a certain state of the application worth waking up the standby for? There are various ways how you can resolve this. One of them is: use the difference between error level logging and warning level logging. error level would mean: get into action now, warning level could mean: when you are ready, pick it up. Many exceptions and errors that your code throws might not be worth picking up, and should not be logged as error or warning.

There are a lot of different things to consider. We often see applications throwing too many exceptions into the log, though the execution path through the exception still leads to the actual state which delivers value to a customer. Therefore, logging to help with your application maintenance goes beyond writing and storing logs: it starts with(re)designing your application control flows.
What can help here, is to ask yourself per user story: “what do you need to log to know that the functionality works correctly?” & “what do you need to log when the functionality fails?”

Purpose case: Audit Logs

Audit logs have very different requirements than application logs. In an audit log you need to capture every event that leads to the relevant state of change in your application or devices. This is rather different than the log for application maintenance: here the reason for an internal server error due to the lack of a database connection is less relevant. However, the steps your users went through in order to execute a transaction is key to the system.
Integrity of the audit log is important as the logs can serve as evidence in court if it comes to legal disputes about actions of a user or the system. This requires integrity protection of the audit log, as well as ensuring that all relevant actions are recorded. This works best if you design your logs for easy correlation: make sure you link the relevant logs from the start! This can be via a session-identifier, trace-IDs, et cetera.
Note that audit log messages require their own aggregation as well. So make sure that you format the messages in the same way.

Purpose case: Security Event Logs

Security event logs should be designed to inform the security team about any event they should be aware of. Ask the security team what they need from the logs: does this include the “happy flows” in your audit-logs? Dos this include all the error-states in your application logs? A security event log should be based on a threat model. What are you afraid of? Which scenario might be problematic to the security of the system and/or users?

2. Secrets in Logging

We often have to load secrets, such as credentials and/or cryptographic keys into our application. These secrets can be easily misused, when they fall into the wrong hands. Think of passwords to powerful administrative backends, or signing keys used to proof that a transaction came from the system to which the key belongs. In both cases, an attacker can easily do harm with them. Therefore, it is better not to put these kind of secrets in logging. After all: anybody with access to these logs might be able to leverage the secrets for a privilege escalation. A nice example of this can be found at challenge 8 of OWASP WrongSecrets.

3. Sensitive Data in Logging

Where passwords are an easy“no-go” for logging, there are quite a few items that could be“too sensitive” to log into your application logs without any additional obfuscation/encryption/masking:

Financial, Health & Business information which should not be shared publicly
Personal Identifiable Information as defined by the GDPR or other applicable laws and regulations.
Application source code
Any other type of information which is deemed too sensitive based on the underlying threatmodel.

Does that mean that you cannot log all of this information at all? Of course it should be, if relevant, part of your audit-log. But it might be a good idea to not share this in your application debug log.

4. Logging and Security

There are many security topics that are relevant if it comes to logging data. Let’s touch upon a few:

Never trust external input. Never just insert raw data into your logs. Instead make sure that you only record data from trusted zones. Next: make sure you do proper encoding & data sanitization. WebGoat contains a nice challenge which shows what happens if you do not clean your data properly.
Harden your setup. The recent events with Log4J show that we should ensure that our logging infrastructure is patched, hardened, and monitored.
Protect integrity of your logs: Integrity of data becomes all the more relevant when the logs need to show what has happened. Audit-logs, transaction-logs, and such often need to be protected by either having signed/HMACed messages or using a WORM(Write Once, Read Many) storage solution.
Protect the data. Logs should be protected at least as the same level as the data the process is dealing with. This means that you need to setup access management for logs, encrypt them at rest, et cetera.

5. Stacktraces and Logging

Stacktraces require special attention. When stacktraces encompass business objects in their descriptions, they can leak information which we just listed not to log. Therefore, always be careful with what you put in the context of an exception. Make sure that either your business objects, or your logging & exception handling configuration ensures that confidential information is cleared out during a stacktrace.

6. Actionable Logs(with context)

Make sure you don’t log just for the sake of logging. Try to understand what you need to record for which purpose, and keep it to that level. We often see log-items such as “Received incoming message”, and “starting processing message” in a row without any context of the message. This makes it hard to understand what is going on. It can help when you are debugging in order to see whether your received message is then being processed, but it does not help fellow developers after that. After all: the more log statements, the higher the risk that important events might be missed. That is why we recommend to prune your log messages(or lower their log-level) when you are done debugging.

Where to Go Next?

There are many more things you need to consider when designing your application logging, such as:

What information is needed for higher level business goals?
Which sources should you include to provide context to your logs? Think about the logs coming from your host-machines, cloud infrastructure, etc..
Where do you store the logs over time?

For now: check your application logs and get in touch with the parties that rely on your logging information! Take the 6 pointers presented here and evaluate together: do the logs deliver the value you need? Is the application is designed in a sensible way when it comes to erroring and exception handling.

WebGoat 8: https://github.com/WebGoat/WebGoat
OWASP cheatsheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
Log4J:https://xebia.com/log4j-a-10-step-mitigation-plan/
Wrongsecrets: https://github.com/commjoen/wrongsecrets

Log4J – A 10 step mitigation plan

Jeroen Willemsen — Sat, 18 Dec 2021 06:31:29 +0000

Co-authored with Edzo Botjes and Jesse Houwing

Update: which vulnerabilities are present?

The initial blog below was written around CVE-2021-45046 and CVE-2021-44228, in which Log4J its JNDI functionalities could be misused in some cases to get to an RCE (Remote Code execution). The recommendation back then was to upgrade to Version 2.12.2 when you are on Java7 and version 2.16.0 in all other cases.
Now a new vulnerability is discovered: CVE-2021-45105. This allows for a DoS (Denial of Service) attack on Log4J when you use the log4j-core JAR in combination with a specific configuration which involves using “Context Lookups like ${ctx:loginId}or $${ctx:loginId}” according to the post by the Apache Log4J team. There are 2 fixes for this:

Update the Log4J core JAR to version 2.17.0;
Update your Log4J configuration as described here by the Apache Log4J team.

We recommend to FIRST make sure you have no systems running on a version of Log4J core below version 2.16.0 (or 2.12.2 in case of Java7). If you do, patch them to the latest available version (2.17.0 for Java 8 and above, 2.12.2 for Java 7).
Last, given that the recent security news attracted a lot of security researchers their attention to various open-source libraries, we expect that more CVEs on various libraries will be reported soon.

There is already a lot of attention on the #Log4J vulnerability. It is all over the news while we write this blog. Many customers have asked us what to do. In this blog we give some advice on how to deal with the Log4j vulnerability and similar vulnerabilities in the future.

What to do first?

1. Check if you are affected by the vulnerability

There are various ways in which you can check whether you are vulnerable:
By means of static analysis:

If you write software that runs on the Java Virtual Machine (JVM) based on Java/Scala/Kottlin/Groovy/Clojure check whether you are using Log4J with a version below 2.16.0.
Check what other software you are running and see if the software is vulnerable according to the list published by the NCSC-NL. CISA.gov has a similar registry.

By means of dynamic analysis:

There are various tools, nicely again enumerated by NCSC-NL at their Github repository with which you can detect whether you are vulnerable. See which one you like best given the situation you are in.

2. Patch! Patch! Patch!

Once you found vulnerable software, patch it! When you write your own software, it means: update Log4J at least to version 2.16.0.
When you work with third party software that contains a vulnerable version of Log4J, then upgrade the given software or follow the patch procedure outlined by your vendor. If there is no upgrade or guidance available, contact the vendor of the software and ask for their support. Does that not help either? Read on to check on other possible solutions.

3. What if you cannot patch?

If you cannot upgrade the library, there might be various solutions to the problem:

One approach that might work here, is removing the class from the application which actually causes the problem. Or better phrased: introduces the vulnerability. In this case, that is the “JndiLookup.class” file. This file needs to be deleted from the JAR file. You can delete the file easily by means of tools like Zip and 7-Zip as the JAR file is basically a “zip file” with java .class files and additional resources. One way to do this is by using 7-zip: “c:\Program Files\7-Zip\7z.exe” d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class" . The Log4J team advises to execute the following command when using a Mac or Linux: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class .
Another approach involves applying configuration changes to the Log4J library to make the vulnerable library safe again. However, these required changes differ per version of the library in use. These configuration changes might actually not always be effective and sometimes come with unwanted side-effects. You can best read up about it here.
Take the vulnerable system offline and try to find a workaround for the functionality provided by the system until there is a patch.
There are runtime patches such as log4j-jndi-be-gone that can be injected into a process. This doesn’t require any changes to application files in case they’re signed, or otherwise not editable, plus these tools will also remove the vulnerability in case the vulnerability is picked up from another directory due to misconfiguration (class path).

But what about the WAFs? We sometimes hear that it is recommended to use a Web-application-firewall (WAF) to prevent attackers to exploit the vulnerability. Though a WAF can help as a defense-in-depth control, it should not be your primary and only defense. A WAF helps to delay an attacker: when configured correctly, it can stop many of the exploits that are currently found in the wild. However, a sophisticated attacker might still be able to bypass the WAF. Therefore a WAF can help as a temporal stopgap while you have vulnerable components, but we recommend to use this as a second layer of defense. The actions described at 2 & 3 should be your primary defense.

A little note: when you write your own software and use JAR-shading

Jar-shading is a technique which is sometimes applied by library/tool creators. This process involves packaging multiple – often supportive- JAR files together, which are then renamed in order to ensure that the authors of the library/tool have better control over the dependencies used. This means that the Log4J JAR that is vulnerable might be present in the library/tool without being easily detected as its name is changed. In that case, solutions like the earlier mentioned log4j-jndi-be-gone might help as well.

What to do next?

There are a few things you need to take care of:

4. Security and Event Monitoring

Make sure you have a proper logging and Security Event Monitoring (SEM) system in place. Make sure you are able to detect when the Log4Shell exploit is applied at your system and ensure you can detect odd behavior at your system in scope. Want to get started with SEM? There are plenty of mature solutions in your cloudprovider and mature open source tools available for your stack.

5. Prepare for Incident Response

Always be ready to do incident response. Make sure you have a run-book and a call–tree ready when security incidents happen. If you think/detect that your application landscape is successfully exploited, contact a professional incident response team.

What about the future?

There are a few things you can do to speed up your response to similar future events:

6. Always know what you are running: the Software Bill of Materials

Make sure you know what you are running on your platform. The Software Bill of Materials (SBoM) describes all the various software components on which your system is based. If you keep an active track of your SBoM with tools like OWASP dependencyTrack, it becomes easier to know whether software you are using is vulnerable. Additionally there are great open-source tools, like the OWASP Dependency Checker, Trivy, Clair, and many others which you can use as part of your CI/CD pipeline to detect whether some of the software you are building has known vulnerabilities.

7. Stay up to date automatically

When you write your own software,use tools like DependaBot, Renovate, Snyk, etc, to automatically create pull/merge-requests and compatibility scores. It’s much easier to apply a patch to a system when it is already using the most current version of a library. Some of these tools also provide a compatibility score and guidance on how to solve breaking changes.

8. Understand the risks of your own application landscape: Document and Threatmodel

When vulnerabilities rise at your system, it is good to understand what risks are involved when these get exploited. For this, it can help to document the overall architecture of your application landscape and do active threatmodelling.

9. Isolate your applications and services

Software will always contain vulnerabilities. This is why you need to isolate your software to make sure that an exploit of one vulnerable component does not automatically mean that the full system is compromised. Isolation can be done in various ways:

Implement network segregation and separation. Make sure that egress and ingress filtering is in place and that network traffic is logged in order to be analyzed by SEM software.
Limit components their runtime capabilities: make sure that they do not have to execute privileged actions, and ensure that proper containerisation takes place.

10. Remain blameless: collaborate and evolve

Building software, maintaining software and operating the complex systems that we use today, are no easy task. This means that mistakes will be made and unforeseen things will happen. These events can often lead to security incidents if not detected early.
In order to respond to these events quickly, it is key to collaborate in a blameless culture in order to ensure steady communication on any of the mistakes made. It matters how you react to unforeseen events, and improve on what you have learned. That is how you improve your resilience as a person, a team and as an organisation.

Ten Pitfalls you should look out for in AWS IAM

Jeroen Willemsen — Sat, 18 Dec 2021 06:18:11 +0000

Coauthored with Ben de Haan

In our previous blog, we shortly touched upon security challenges that we can come upon when working with Terraform on AWS. In this blog we want to dig a little deeper into IAM by explaining 10 pitfalls you should look out for when you configure AWS IAM. Let’s start our journey and tackle them one by one.

1. Implicit denies

For this first pitfall, understanding policy evaluation logic is key.
Say we want to grant a role access to everything but IAM. Essentially there are two ways to configure this: implicitly and explicitly.

Take a look at the following two policies:
Implicit:

{
  "Version": "2012-10-17",
  "Sid": "ImplicitDeny",
  "Statement": [
    {
      "Effect": "Allow",
      "NotAction": "iam:*",
      "Resource": "*"
    }
  ]
}

Explicit:

{
  "Version": "2012-10-17",
  "Sid": "ExplicitDeny",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "iam:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

Both of these policies grant access to everything but IAM. The difference is in how they do it. The first policy implicitly “denies” IAM by not including it in the allowed statements. The second one has an explicit deny on all IAM actions. This difference is crucial.

Evaluated on their own, the outcome of each policy would be the same. But what happens when we attach a different policy (or statement) to the same role, which allows IAM? Say, something like this:

{
  "Version": "2012-10-17",
  "Sid": "AllowIam",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iam:*",
      "Resource": "*"
    }
  ]
}

Unfortunately, we bypassed the first policy: IAM is now also in the list of allowed actions. The second policy still has the desired effect. This is because 1) a deny trumps an allow and 2) IAM evaluates denies first.

This means it’s safer to use an explicit deny rather than an implicit one if you want to prevent unintended access. Like the Zen of Python states, “Explicit is better than implicit”. Apply that to your AWS policies too!

For more information on policy evaluation, check out the AWS docs on this topic.

2. The additive resource policies

This next pitfall is related to understanding policy evaluation logic as well.
Resource policies can take different shapes or forms. The S3 bucket policy is the most notable example of a resource policy. Other examples are KMS key policies or Lambda function invocation permissions.

Resource policies are additive – this means they can grant access in addition to what you define in IAM. We often see resource policies that grant access too broadly.

A role can’t access a resource if you don’t allow it in IAM. The issues arise when you don’t consider the access outside of the AWS account you’re working and testing in.

Encryption for all

For example, consider the following key policy:

{
  "Version": "2012-10-17",
  "Sid": "EnableKeyManagement",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal" : { "AWS" : "*" },
      "Action": "kms:*",
      "Resource": "*"
    }
  ]
}

There are three *‘s in this policy, two of which we don’t really have an immediate problem with. Starting from the bottom, since the resource policy is only tied to one resource, * is acceptable. Even starred out, it only applies to the resource it is attached to.

Next, for the action, kms:* isn’t really an issue here since this policy’s purpose is to facilitate key management. To improve it a little, you could scope it down so the managing party wouldn’t be able to encrypt/decrypt with the key without making a policy change.

The last * is where the problem lies. This policy grants access to anyone and everything with an AWS account. Everyone who knows or predicts your key ID will be able to do whatever with it. This would set you up for high API costs at best, or complete denial of service and possibly data leaks at worst.

With S3 being a favorable storage solution for many, we see a lot of breaches that involve too openly configured S3 buckets. It doesn’t help that there are bucket policies, bucket ACLs, public access blocks, and S3 access points on top of IAM. Anyway, that’s a topic for a different deep dive.

3. Inline policies and named policies

When you create a role, user, or group, you have a choice to attach inline policies or named policies (see also. AWS recommends that you use named policies where possible, and we’re of the same opinion.

Named policies have several advantages. They offer reusability, central change management, versioning, and rollback. For AWS’ opinion on this, check the docs.

That said, it’s not necessarily a problem to use inline policies – especially for the one-to-one relationships Lambda functions have with roles.

The problem here is that you can combine these two to create a complex mess for yourself. Do yourself and your teammates a favor and be consistent. Use named policies unless they don’t fit your use case, and don’t combine the two.

If you are already lost in the woods, one of the tools that can help you to understand what happens is the AWS policy simulator (AWS account required). It doesn’t play nice with Service Control Policies (SCPs), though.

4. Extensive role chaining

IAM Roles are a temporary identities in AWS. They can be assumed by those who need them and are allowed to use them. Users, groups, services and applications can use these roles in various ways. Let’s for now focus on your users. They can assume a role, if: 1) the user can invoke sts:AssumeRole with that role in IAM and 2) the role’s trust relationship allows it.

Let’s start with the assume role policy. This policy lets a user assume a predefined role. Take the following example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::123412341234:role/*"
    }
  ]
}

Note: 123412341234 is your normal AWS AccountID. This policy allows a user to assume any role for the given AWS account with the defined AccountID.

Next up is the trust relationship, which is defined in a trust policy. An IAM role defines who can assume that role, by including that trust policy. Take a look at the following example:

 ...Role definitions for "example-role"...
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "AWS": "arn:aws:iam::123412341234:role/dev-engineer"
              },
              "Action": "sts:AssumeRole",
              "Condition": {}
            }
          ]
        }
  }

Here we say that anyone with the role dev-engineer in the account with the ID 123412341234 is allowed to assume the example-role to which this trust policy is attached.

You can easily extend the example above by allowing another role assumption from this example-role. Maybe there are some other roles the dev-engineer or the example-role can assume (see also: role chaining).

By chaining these AssumeRole calls, it’s possible that the dev-engineer can now suddenly do more than we intended. We have seen various examples where an organization allowed (external) engineers to first use a sort of view-only role to have a look at some of the configurations in AWS, after which they could assume a role that could see classified secrets. This happened because one other engineering role had to look at those as well, using the view-only role as a step in between:

Handling role chaining

Always check the trust policies attached to a role, make sure that you do not add an overly broad trust policy to powerful roles – that is with powerful policies attached. Make use of the Condition clause of the AssumeRole Policy Document as explained in this blog post from AWS.
Always check if you attached a proper Assume Role Policy to the principals, if the AWS IAM configuration is under your control.
Leverage AWS SSO. AWS SSO is able to provision all of the roles your users need across the organization. Users log in to AWS SSO and jump to their target role in just one step. This virtually eliminates the need for role chaining and greatly simplifies the setup. Happy users, happy you.

Note: Role assumption has long been difficult to trace. This has become a lot easier when looking at CloudTrail logs by means of the SourceIdentity. For more information: have a look at this AWS blog post.

5. /root in trust policy documents

Remember the statement "AWS": "arn:aws:iam::123412341234:role/dev-engineer" in the trust policy of section 4? If it would have stated "AWS": "arn:aws:iam::123412341234:root" instead, that’d be a red flag. If you include this, you rely completely on the IAM setup of account 123412341234 – do this only if you fully and completely trust it. That should be almost never.

The principal statement above basically implies that the role to which the trust policy is attached, can be assumed by any authenticated and authorized principal in the 123412341234 account. So if you do this, make sure you’re in full control of that account and ensure the role has limited privileges.

6. What does your service allow you to do?

One often overlooked pitfall is the scope of policies attached to the service roles of various services. When a user cannot access data in bucket A, but can access Sagemaker, then clever usage of Sagemaker can lead to leading the contents of bucket A to the user.
One of the interesting Sagemaker policies AWS references in its docs is this one:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "sagemaker:DescribeEndpointConfig",
                "sagemaker:DescribeModel",
                "sagemaker:InvokeEndpoint",
                "sagemaker:ListTags",
                "sagemaker:DescribeEndpoint",
                "sagemaker:CreateModel",
                "sagemaker:CreateEndpointConfig",
                "sagemaker:CreateEndpoint",
                "sagemaker:DeleteModel",
                "sagemaker:DeleteEndpointConfig",
                "sagemaker:DeleteEndpoint",
                "cloudwatch:PutMetricData",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:DescribeLogStreams",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
            ],
            "Resource": "*"
        }
    ]
}

There are several potential problematic statements in there, especially combined with the Resource: "*" part. Essentially this policy allows passing any role (potentially highly privileged ones), and allows putting in objects and fetching them from any S3 bucket. It also allows publishing logs to everywhere, deleting all Sagemaker models, etc.

You can expect this from a managed policy, as they are built for compatibility rather than security. Take this into account when selecting or copy-pasting readily available policies.

Similarly, QuickSight can serve confidential content to readers. Therefore it is key to configure these services in a way that only shows information according to the entitlements of the users.

Another example is in EC2: When users can execute actions on an EC2 instance, and the instance has access to privileged data on S3 with its role, then users can use this EC2 instance to access that data.

If you want to play around with privilege escalation, check out IAM-Vulnerable. Highly recommended!

7. The “more control by fine-grained control” fallacy

There’s a German proverb that more or less says: Who has the choice, has the pain (“Wer die Wahl hat, hat die Qual”). IAM is fantastic because of the fine-grained access it allows you to configure. This is also where it gets painful, and the complexity here is one of the reasons multiple AWS accounts are a best practice.

Sometimes you have to do some grunt work to get the policy just right, while taking into account all of the other policies that apply to the same entity. Wrapping your head around everything can be hard.
To help you, here are a few pointers:

Structure your policies. For example, if you need to grant permissions related to a specific feature, give it an appropriate name and put everything required for said feature in that policy. Resist the temptation to edit other policies just because you “already have some s3 permissions in there”. Try to apply this logic to the statements within your policy as well.
Sometimes it’s easier to add a new AWS account than to get your policies just right. AWS accounts form a natural boundary and hit the reset button on ‘what can access what’ within your account. In determining when you need a separate AWS account, think of the blast radius. What could possibly go wrong if something’s breached? How will that impact other resources in your account?

8. IAM Service limits

You’ll probably hit IAM service limits sooner or later. You write a policy that’s too big, attach too many policies to a role or group, or attach too many SCPs to an organizational unit. Even if your Terraform plan or CloudFormation change set looks alright, these limits may break your pipeline and cause you to refactor your setup.

It’s useful to check the AWS IAM and STS limits every so often. Especially do so when working with SCPs, as problems here could disrupt your entire organization.

9. Attribute-based access control (ABAC) and tag policies

ABAC is a contentious topic. AWS states it is important to making security scalable, but AWS users and security experts argue that has major flaws.

On the one hand it saves you from constantly updating your least-privilege policies when you add new resources and services. On the other hand not all resources and services support tagging. This renders it useless in some scenarios.

Our view is that you shouldn’t rely on tag-based authorization within an AWS account, but it could work for granting access through role assumptions.

The reason here is that a lot of resources don’t support fine-grained access based on tags. Quite often, if you can create resources, you can set tags for them. If you can create resources, you can probably also create roles and policies. And let’s face it, you probably need to. The problem is that this renders authorization based on these tags moot within the account that you’re able to do so – even by extension (via CI/CD).

Similarly, if you want to create a role for creating EC2 instances with certain tags, and that same application should terminate only EC2 instances with said tags, good luck creating a policy that effectively narrows this down.

On the other hand, setting tags on a group and only allowing role assumptions for roles that have the same tags works decently. If that’s your use case, rock that ABAC.

10. Not looking back

Okay, you noticed some misconfigurations. Or maybe you didn’t, and you have a bigger problem. The thing is, IAM is very much dynamic. Therefore, you need to put in place proper monitoring. Concretely, you’ll want to:

Evaluate high-risk events (do some threat modeling to know what they are).
Set up AWS config to track configurations. Trust us, you’ll want to be able to create a timeline on several resources in case of an incident. And no, that won’t necessarily be in your version-controlled infrastructure as code.
Create an organization-wide AWS Access Analyzer. This way you can keep track of the external exposure on S3 buckets, roles, KMS keys, and lambda layers.

Now what?

Whew, that’s a lot to take in. Apart from finding and selecting the AWS service you need, IAM is probably the most complex part of AWS. So keep that policy evaluation logic close at hand, and think of the structure you want in your IAM setup. Create something structured. It’s natural to pivot to a different solution or architecture when your organization or stack changes, but don’t try to do a bit of everything. That’ll leave you with a tangled mess. Create separate AWS accounts, and by all means re-read this blog on IAM pitfalls every once in a while.

If you find you still have problems, or want a thorough assessment of your AWS setup, don’t hesitate to reach out!

Secure Deployment: Three Security Highlights For Terraform on AWS

Jeroen Willemsen — Wed, 08 Dec 2021 06:01:34 +0000

Coauthored with Ben de Haan

So, you want to build your infrastructure in AWS and use Terraform for your Infrastructure as Code? And you want to do it securely? In this blog we highlight three things you should pay attention to from a security perspective: your IAM roles and trust relationships, your Terraform state, and your detection & monitoring. Of course, each of these topics deserve their own in-depth blog, but we’ll start highlighting three key touchpoints.

If you have read our previous blogs on secure deployment, then you know by now that you can best deploy infrastructure using a secure CI/CD pipeline. If you followed up on that, then you already have a great starting point for your cloud-infrastructure. Hopefully you only deploy your production environment through infrastructure as code with protected branches and protected runners. So what’s next? Let’s talk about three key items: Identity and Access Management (IAM), the Terraform state, and monitoring.

IAM

AWS allows you to segregate your cloud infrastructure over multiple AWS accounts. An account is a natural boundary and you should absolutely use different accounts for e.g. production, staging, and development. The accounts can -and should- be managed using AWS Organizations. AWS Organizations is useful to glue the different AWS accounts together. This enables you to automatically provision them based on organizational structure and attach the appropriate guard rails.

Organization Account Access Role

When you create a new AWS account in the organization, it automatically gets the OrganizationAccountAccessRole. This role provides admin permissions to anyone who can assume it from the organizations root account. So far, so good.

The very next step you want to take is to set up separate roles for your cloud engineers or SRE team, DevOps team, data platform teams, and so on. Additionally, you’ll need roles for the CI runners that they use. Some of these roles might have to do some terraforming themselves.

It would be easy to just use the OrganizationAccountAccessRole and extend that trust relationship to SRE teams or pipelines. The risk here is that anyone who can assume such a role can do pretty much anything with the AWS account. Much like doing things with root, don’t use the OrganizationAccountAccessRole unless you really need to. Ensure you have an emergency process for when you do. Know who or what can assume administrative roles. With the emergency process in place, limit further use to your privileged CI/CD pipeline.

Terraform State

Your Terraform state is used by Terraform to keep track of real world resources. The state serves as a basis to make appropriate changes to your infrastructure.

What Could go Wrong in Terraform State

You (or an attacker) could manipulate the Terraform state by hand. If you do, you could lure Terraform into creating resources which your actual Terraform code never asked for. This can cause headaches in trying to find drift in the cloud account from your definitions in Terraform state.

Terraform state has its challenges when it comes to locking and such, which can provide security challenges of its own. More importantly: we often find secrets in Terraform state, such as “initial passwords” which often remain the same, as well as access-tokens used by custom providers. And how about the values that you want to store in AWS Secrets manager? They often end up in the Terraform state as well.
Essentially, there’s a few things to pay attention to when you think of a potential compromise of your Terraform state:

unintended/unauthorized changes via state manipulation can cause untracked infrastructure changes;
blast radius of states can extend across environments, meaning a compromise can potentially impact all environments at once;
secret leaks via state provide additional entry points for attackers.

How to Secure Terraform State

Try to prevent write access to the Terraform state apart from the privileged runners mentioned earlier as well as a “break glass” user for emergencies (which should be behind MFA).
Split Terraform states by environment and/or other ways that make sense for your organization. Be sure to check out workspaces.
Limit the ability to extract the state and do a local plan. Developers shouldn’t need direct access to production, as that is the task of the CI/CD pipeline. Use the aforementioned “break glass” role in case of emergency.
Try to prevent storing secrets in Terraform state altogether by moving secrets management to a solution like HashiCorp Vault or something similar. Alternatively: make sure it ends up encrypted in Terraform state for the time being.

Detection & Monitoring

Even when you use infrastructure as code, you can easily make mistakes. You can assign overly broad permissions to roles or users, you can easily open up things like your S3 storage buckets too far, etcetera. Therefore:

Always enable AWS Config and AWS Organizations’ service control policies (SCPs) to provide guard rails across the organization. For inspiration on SCPs, check out asecure.cloud’s SCP repository.
Next, make sure you store your CloudTrail logs for a decently long period of time.
Use the Access Analyzer to spot other possible mistakes.
Make sure you enable access logging for your storage solutions, especially for those that keep the Terraform state.
Finally: logging is nice, but don’t forget to make the information actionable. Set up an alerting system that alerts something and/or someone. Have it alert when highly privileged roles are assumed, the AWS root user is used, or when something else than your CI/CD pipeline accesses the Terraform state storage for production. Start small and extend based on risk – ideally you do regular threat modeling to discover detectable attack paths.

Next Steps

That concludes our three security highlights for Terraform on AWS. Now that we touched upon IAM, Terraform state, and Detection & Monitoring, it’s time for the next steps.

Want to catch things earlier? Try out tools like Terraform compliance, Prowler, Scoutsuite, or Inspec to inspect your security posture or even prevent misconfigurations from reaching production. You could also use driftctl to see whether there is any drift between your Terraform code and what’s in your environment. Want to have a deep dive on these topics? Let us know!

Secure Deployment: 10 Pointers on Secrets Management

Jeroen Willemsen — Mon, 06 Dec 2021 20:15:29 +0000

Coauthored with Ben de Haan

In a previous blog we talked about secure deployment. Secrets management is an important part of that. So what does that mean? In this blog we’ll give some pointers on how to do secrets management well in the perspective of a secure deployment. It’s easy to start saying “use tool X to store the secret” or “have all these detection tools in place!”, but that would lead to blind spots. Instead, let’s take a look at some pointers that would help you increase secret security holistically.

1. Inventory your secrets

Many security incidents actually stem from the fact that a “secret” was not treated as a secret, or was not properly protected. In addition to data leaks, there are other risks related to secrets: your application can get compromised when secrets are leaked. Many applications no longer function without their secrets. If you want to protect your secret well, you first need to know what you classify as a secret and where you store it currently. Based on that, you can take actions to further secure the secret.

2. Prepare to migrate them

Your technology stack evolves continuously, and you will switch eventually. You may have done a migration from “Jenkins” to “GitHub actions” or from your “secret shell script” to “ vault”. Similarly you can switch locations for your secret: from your Kubernetes Secrets to your “vault” or to your cloud provider (or vice a versa). All with good reason. The key aspect remains: you need to prepare to migrate a secret, because you want to make sure your secrets remain secret. Therefore you need to be able to tell what the purpose of the secret is and how it is consumed. It helps to store the metadata on a secret close enough to the secret so you can easily identify what/when you need to migrate.

3. Make secrets temporal

No secret is safe infinitely. The chances of a secret leak having major impact on your organization are a lot less if you tune down the validity. This means that you should make sure secrets have a limited lifetime and can be changed when needed. Depending on the type of secret it becomes easier to make them temporal. A cloud-provider access token, for instance, can be very short-lived. This is especially important for high-privileged access tokens. On the other hand: your TLS private keys might have to be stable a little longer in case you have to support a setup without perfect forward secrecy. If you can ensure that your public-private key pair only lives for a year, it is still “temporal”. Similarly, rotate your passwords of accounts you hold dear on a regular basis – a password manager makes this a whole lot easier.

4. Make sure they are not in code/config

Secrets in code/config are easy to read by others, and more importantly, easy to exfiltrate by tools. By now, it’s common knowledge that a secret should not be checked into git. However, places like your Terraform state bucket, environment configuration files, and your Kubernetes config maps, are often overlooked. In the ideal scenario your secrets only live in memory or, preferably, in software/hardware security modules. So when you generate your secret, a fair share of (secure) randomness should be involved. Your configuration code can entail length of a secret, the type, possible used alphabet, or other criteria, but it should not contain the secret itself. In this case, the instructions on how generating and using the secret are in code, but the actual secrets only exist in the software or hardware security module.

5. Bring the secret close to the intended recipient

If you expose a secret to more than just the intended recipient, you increase the risk of disclosing it to unauthorized parties. For instance: If a specific application on your Kubernetes cluster requires a secret, it’s better to let the application retrieve it instead of the worker node. There’s a simple reason for this: if all pods can access the secret through the worker node, any of these pods can leak it. This doesn’t just apply to Kubernetes, all technology stacks face similar versions of this problem.

6. Ensure least privilege access

Least privilege access in secrets management goes two ways:

You want to make sure all entities only have the privileges they need – possibly temporarily (see #3)
The secret scopes only to provide access to what its user requires

In practice, this means you will want to create separate secrets for separate functions or entities, and prevent unconstrained access. Sane defaults and temporary privilege elevation are key here. You don’t want everyone to be root all the time, and unconstrained secret access is by extension the same. If one of your team members gets compromised, you’ll be really happy the attacker has limited access by default.

7. Don’t operate by yourself

When handling secrets, whether through automated generation, or through manual import into your system: make sure you cooperate. Changing a secret can have a lot of impact on the consumers of the secret. Therefore, it’s good to do this together -pair programming style-, to make sure that no errors are made. Pair programming helps spreading the knowledge of the process as well! Last, you can help each other by ensuring copies of an imported secret are destroyed in case of a manual import.

8. Do not reuse secrets

Do not reuse passwords for various services, do not reuse keys for various purposes. Every time you use a secret for a given entity, and that entity gets compromised, so is your secret. Therefore a single purpose per secret helps to reduce the impact in case of an exposed secret.

9. Monitor secret access

There are various ways to monitor access to your secrets. We’ll discuss ‘traditional’ audit logging and monitoring, canaries, and external services detecting leaks concerning your organization.

Audit logging

The oldest and generally most robust way of monitoring whether a secret is accessed or modified, is simply using audit logs. This can be via application access-logs, cloud audit monitoring, or system services such as AuditD. Audit logs are a fundamental control in establishing what has happened. Things can get tricky, though: an attacker could use a compromised (service) account to obtain a secret, in which case you’d just see the (service) account accessing the secret. This means you won’t catch such an attack if you’re not correlating across different log sources.

Canaries

Some data breach stories involve an external entity informing an organization that it might have been breached. This doesn’t have to be the case. Like canaries in coal mines were used to warn of gas, you can put canary tokens, users, or access keys in various places. The moment these are accessed or used, they alert you of possible mischief, giving you an early warning (which might be your only one). One service that you could get started with is canarytokens.org.

HaveIBeenPwned

Another way of seeing whether a part of your secrets have been leaked, is checking haveibeenpwned. This service can tell you, based on your email address, whether your data has showed up in a breach. Please note that, if your email address did not show up, it does not mean it never got breached. Another indicator you can use is the pwned passwords service. This can tell you if the password of your choice already has been leaked before.

Pastebin / dark web monitoring

Not a real solution, but hey: you can try to monitor the internet yourself as well, to see if your secret ends up somewhere ;-). It is often better to ask specialized third parties to do this for you. Be advised: the effectiveness of these parties might vary.
Pro-tip: it is useful to enable tooling to monitor your own repositories! Even various open source tools can already help greatly!

10. Make secrets part of your threat model

Secrets are a key asset, so naturally they are also the object of several possible threats. When you threat model, be sure to include them! Note that it can be hard to quantify “secret is leaked” in terms of likelihood or impact. However, if you identify a threat as “Admin password in our git repo is reused to delete our main account”, then it becomes a lot easier to see what the possible level of risk is.

Where to go from here

These were a lot of pointers to start with! What should you do now?

Make sure your inventory of secrets is up to date: Once you know which secrets you have, you can determine what to do with them.
Make sure you move ahead in risk-based fashion: conduct threatmodeling exercises in order to prioritize what to do with which secret.
If you’re late to the party and have a lot of secrets lying around, don’t worry: chunk up the work in small pieces, and most importantly: start!

If you have any questions, remarks, or additions, don’t hesitate to reach out!

Our experience with resources on secure deployment

Jeroen Willemsen — Mon, 06 Dec 2021 20:02:36 +0000

Coauthored with Ben de Haan

Have you ever searched for resources on “Secure Software Deployment”? Most of the results revolve around the pentesting or putting security tools in your CI/CD pipeline. It would be the same as researching how to improve your cake baking skills, but end up with manuals of kitchen appliances. We want to address this gap: in this blog, we want to give you key pointers for a secure deployment.

So, what should you think of? We’ll start with a few aspects that we believe are important to think of when you work on a secure deployment. After that, we will touch upon the areas that you need to work on to actually achieve it. Finally, we’ll advise where to go from here.

Components of a secure deployment

Deployment of your software normally focuses on the migration of your artifacts to a given target environment. Such a deployment often requires instructions, which are present in code. Often the build process is still part of the deployment, as some artifacts require additional build steps in the target environment.

This leaves us with four main components that are relevant for a deployment:

the instructions for the artifact and its deployment as code;
building the artifact;
deploying the build artifact to the target environment;
the target environment.

Here, we’ll focus on the deployment itself – secure build and secured artifacts are building blocks that the deployment depends on.

The build tools you need, the way you want to code, and how you integrate code changes, are topics worthy of their own articles. Let’s focus on the deployment step instead. So what would you need?

Requirements, requirements, requirements…

Let’s start with an important note: doing secure deployment over a secure CI/CD pipeline is hard. It takes continuous effort, involving improvement of the security of the components we just mentioned. To get you started, we compiled a list of pointers based on our experience and the resources found at the bottom of this blog.

We’ll move on to what you should take care of, covering process, technology, and people:

Process

Formalization & repeatability: first make sure that you understand what you want to achieve. Either document this in a formal process or in code -or both, together-, to be able to automate the building and the deployment process.
Reproducibility: every deployment should be reproducible. This becomes clear when you want to replay an incident in a different environment. You want to find out what went wrong, and why. This means that you’ll have to store some of the produced artifacts for a longer period of time. Alternatively, an artifact deployment process should have guarantees to end up in the same state with the exact same properties as when you deployed said artifacts previously. Possibility to rollback: this is often overlooked, but if you cannot rollback from a compromising deployment… You are in trouble.
Provenance & traceability: in order to make sure that you know who or what started a task, you must be sure that you can trace the origin – whether it was a commit, an approval of a pull/merge-request, or a deployment approval. So, make sure that this applies to your audit logging and build or deployment pipeline.
Access management: a DevOps engineer should not be allowed to just deploy anything anywhere. There should be segregation of duties in place. This holds for the technology in use as well: you should make sure that your means to deploy (e.g. agent/runner) should only be allowed to deploy to the designated target environment. For instance: when your runner deploys a container to data-science test-environment, that same runner should not be authorized to deploy to your production environment at the same time.

Technology

Integrity verification: there are a few places where integrity matters:
- the integrity of the artifact when retrieved/made ready for deployment
- the integrity of the artifact after deployment to the target environment
- In case you have a mutable artifact, the integrity of the configuration instructions as well as the target result are key.
Automated security checks: you can have various automated security checks to detect “low hanging fruit”, often spread over the build, deployment, and post-deployment phase. Think of dependency checks, SAST, DAST, docker container checks, host-audits, and/or cloud provider configuration security checks. Where you put which check depends on the check, the nature of the artifact, and the target environment. Security of the build & deployment pipeline: there are many things to say about the security of your CI/CD pipeline. It is your “production environment” which should be taken care of just as good as your actual production environment. You can find more pointers in this presentation.
How you manage your secrets: of course, you should be careful with what you make public. Whether it is your intellectual property in source code, or the Personally Identifiable Information of your users. The actual secrets that require additional protection in this phase should be your authentication, encryption, and signing means. Think of deployment credentials, other passwords, keys, and such. Verify you deploy them securely, use temporary credentials where possible or otherwise rotate the secrets on a timely basis.

People

Manual verification (or “4 eyes”): tools do not find everything, certainly not when starting out. Make sure you have manual verification in place. Where this happens depends on your process. Often, this is at the pull/merge request level or there’s a deployment approval. Make sure that the author of the code is not the one who does the manual verification. Otherwise they can approve the deployment of their own malware.
Training: train your engineers to understand the goals you want to achieve with a secure deployment and enable them to use the tooling well. In fact, it might be a good idea to train them to build the pipeline themselves for best results. Recognize and report: your engineers should be able to recognize and report on bypassing the security controls. Reward them for the reporting!

Of course, there are many more opinionated aspects of a secure deployment. For instance: should your artifact be immutable at the target environment? What should be your artifact promotion strategy? (E.g. which environments do you have outside of production? and how do you propagate through?). And then we need to talk about the amount of testing of each phase of your build- and deployment needs to take place.

Where to go from here

These were a lot of pointers to start with! What should you do now?

Start risk based. Make a threat model for your SDLC process and take care of the pointers on a scored risk basis. CI/CD security is complex and a continuous team effort. Make sure you embark on this journey together.
Don’t get trapped in analysis paralysis. Start small, but more importantly: start!
If you have any questions, remarks, or additions, don’t hesitate to reach out!

Resources

OWASP SAMM on secure build
OWASP SAMM on secure deployment
UK NCSC Guidance on secure development and deployment
BSIMM on software environment
OWASP DSOMM

Cheating and building secure iOS games

Jeroen Willemsen — Fri, 03 Dec 2021 08:32:27 +0000

You probably have one of the million games where you earn achievements and unlock specials on your iPad or iPhone. If you develop games, you’ve probably wondered about people cheating your games? In this blog we’re going to show you how to try cheating out yourself and how to build secure iOS games.The actual question at hand: how can you break the integrity of an app on iOS?

Cheating is influencing state

A lot of games on the iOS App Store have absolutely zero protection against cheating. Either because developers don’t even consider adding some security or because they might think it’s very hard or expensive to implement. In this post we’re going to be looking at some games that use local state, such as a database or plists (or other formats) on your device. These kinds of games can be cheated by opening up some local files and editing values to your liking.

Influencing the local state

For this example we’re going to use a random game that we found in the appstore. This game does not send any game-related data to the server and thus probably saves everything locally.

We’re using a jailbroken device to be able to access the device file system. This reveals that the game saves a database file in the apps documents folder. When we open this with an SQLite DB Browser we can read and modify the database. In this case the SQLite database contains two interesting keys named registry-stash-stars and registry-stash-coins. We edit these files with SQLite and re-upload the database to the device.

We restart the game and we have 15000 stars and 5000 coins!

Securing your iOS game as a developer

The best way to prevent local state manipulation is, you guessed it: server side validation. Of course, if you don’t want to work with servers, you could also take a look at encrypting your data (hiding it) and then HMAC it (verifying integrity). Don’t forget to store the keys for encryption and the HMAC creation in the keychain. It won’t be watertight but you just got rid of quite some cheaters.

DEV Community: Jeroen Willemsen

Why WrongSecrets moved to the OWASP Github Organization

Prove Value First

Your Contribution safe with OWASP

No Commercial Interest

What is Next?

Special Thanks

6 Pointers on Application Logging

1. Purpose is everything!

Purpose case: logging for application maintenance

Purpose case: Audit Logs

Purpose case: Security Event Logs

2. Secrets in Logging

3. Sensitive Data in Logging

4. Logging and Security

5. Stacktraces and Logging

6. Actionable Logs(with context)

Where to Go Next?

Read More:

Log4J – A 10 step mitigation plan

Update: which vulnerabilities are present?

What to do first?

1. Check if you are affected by the vulnerability

2. Patch! Patch! Patch!

3. What if you cannot patch?

A little note: when you write your own software and use JAR-shading

What to do next?

4. Security and Event Monitoring

5. Prepare for Incident Response

What about the future?

6. Always know what you are running: the Software Bill of Materials

7. Stay up to date automatically

8. Understand the risks of your own application landscape: Document and Threatmodel

9. Isolate your applications and services

10. Remain blameless: collaborate and evolve

Ten Pitfalls you should look out for in AWS IAM

1. Implicit denies

2. The additive resource policies

Encryption for all

3. Inline policies and named policies

4. Extensive role chaining

Handling role chaining

5. /root in trust policy documents

6. What does your service allow you to do?

7. The “more control by fine-grained control” fallacy

8. IAM Service limits

9. Attribute-based access control (ABAC) and tag policies

10. Not looking back

Now what?

Secure Deployment: Three Security Highlights For Terraform on AWS

IAM

Organization Account Access Role

Terraform State

What Could go Wrong in Terraform State

How to Secure Terraform State

Detection & Monitoring

Next Steps

Secure Deployment: 10 Pointers on Secrets Management

1. Inventory your secrets

2. Prepare to migrate them

3. Make secrets temporal

4. Make sure they are not in code/config

5. Bring the secret close to the intended recipient

6. Ensure least privilege access

7. Don’t operate by yourself

8. Do not reuse secrets

9. Monitor secret access

Audit logging

Canaries

HaveIBeenPwned

Pastebin / dark web monitoring

10. Make secrets part of your threat model

Where to go from here

Further reading

Our experience with resources on secure deployment

Components of a secure deployment

Requirements, requirements, requirements…

Process

Technology

People