DEV Community: Vulcanus

Why a High Code Coverage Is Nothing Worth and Leads Only to False Conclusions

Vulcanus — Sun, 09 Jan 2022 00:04:30 +0000

Code coverage shows the degree in percentage to which an application is executed by a testing framework. It is used for quality assurance and "helps" to increase quality of software.

In this article I will demonstrate why code coverage will not help increasing quality, can be faked and what you can do instead to ensure quality.

Most tools show the percentage of code coverage by the executed lines of code. Normally unit, integration and end to end tests help to increase the coverage, which means that by theory you have to write the test by yourself to get the code tested. But is that really the only way? Let's see if that is really the case with an C# example.

namespace ConsoleApp1
{
    public class SimpleClass
    {
        public string SimpleProperty
        {
            get;
            set;
        }

        public string SimpleMethod(bool value)
        {
            return value ? "simple yes" : "simple no";
        }

        public int SimpleMethod2(string value)
        {
            return value.Length;
        }

        public bool SimpleMethod3()
        {
            return true;
        }
    }
}

I have created a small class that has a property and three methods doing basic things. When a test is written for a class like this, it might look like in the code below.

[Test]
public void Test1()
{
   SimpleClass simpleClass = new SimpleClass();
   string value = simpleClass.SimpleMethod(true);
   Assert.IsTrue(value == "simple yes");
}

This simple test executes one method and asserts the output.

Code coverage of Test1

There is a more complex way to "test" a class but it automates the execution of the methods which means that this code will automatically "test" other classes too. The code is very basic and more of an example. It should only help to show my point.

[Test]
public void Test2()
{
    Assembly assembly = Assembly.Load("ClassLibrary1");
    Type[] objectTypes = assembly.GetTypes();

    var parameterInstances = new Dictionary<Type, List<object>>();

    foreach (Type objectType in objectTypes)
    {
        if (objectType.IsClass)
        {
            ConstructorInfo objectConstructor = objectType.GetConstructor(Type.EmptyTypes);
            object objectInstance = objectConstructor.Invoke(new object[] { });

            var methodInfos = objectType.GetMethods();
            foreach (var methodInfo in methodInfos)
            {
                var parameterInfos = methodInfo.GetParameters();
                var methodParameters = new List<List<object>>();
                for (int i = 0; i < parameterInfos.Length; i++)
                {
                    Type parameterType = parameterInfos[i].ParameterType;
                    if (!parameterInstances.ContainsKey(parameterType))
                    {
                        if (parameterType.IsValueType)
                        {
                            parameterInstances.Add(parameterType, CreateValPossibilities(parameterType));
                        }
                        else
                        {
                            parameterInstances.Add(parameterType, CreateRefPossibilities(parameterType));
                        }
                    }
                    methodParameters.Add(parameterInstances[parameterType]);
                }

                List<List<object>> parameterPermutations = GetAllPossibleCombinations(methodParameters);
                foreach (var parameterPermutation in parameterPermutations)
                {
                    methodInfo.Invoke(objectInstance, parameterPermutation.ToArray());
                }
            }
        }
    }
}

To explain what this code does with the power of reflections:

It loads the "ClassLibrary1" assembly and collects all types it contains
It iterates through the type collection and creates an instance of the ones being a class
It iterates through the methods of that class, creates parameters with specific values of certain types and creates a list of permutations to pass them to those methods for execution (working kind of like a Fuzzer)

Guess the code coverage of SimpleClass …

Code coverage of Test2

This code sample will not be able to execute every code line in other scenarios since there are branches you can not cover that simply. Sure you can tailor the code to have some higher degree of line coverage, but this does not enhance quality. By theory we now have achieved "quality" when reading this KPI. Executing code without real context to just execute it does not make any sense. But what should you strive for instead?

Context! We need context to know about the environment, the users, the input and the processes. To explain you, why we need context, let's make an example with SimpleClass and have a look on SimpleMethod2.

public int SimpleMethod2(string value)
{
    return value.Length;
}

This method takes as parameter a string and returns the length of it. When executing a test with a string, we covered the complete method and reached 100% here. Cool? Well, when you look more closely, you will notice it. Do we really ensure "quality"?

The int here is a Int32 struct where the lowest possible value is -2147483648 and the highest 2147483647. Since the Length property of string is a type of Int32 the theoretical length of a string is the highest possible value of Int32. Practically you will not really reach this number even close since the Common Language Runtime (CLR) limits the size of single objects by 2GB (well, you could in fact change that in the config but normally there is no reason for that). By default a string uses UTF-16 which needs two bytes for a single character and in fact this means you probably reach a length of 1073741823.
Ok, the length of the string will not cause any problems but what can? Strings are reference types which means we have the power of null! When the parameter is null, the code will crash.

NullReferenceException

My point? Exactly this. Even though we have a high code coverage in this method, we still have the possibility that the application crashes.
We need context to understand what can happen and model our test cases that way. Instead of caring much about the code coverage KPI, we should cover the different use cases and misuse cases.

Use case and misuse case combined

Shortly explained, while a use case defines the interactions between an actor and a system that contains a set of steps to accomplish a certain goal, does the misuse case describe the process of malicious acts against a system. But how can we ensure quality with context?

We have to take one step back and define what quality is and what quality characteristics are going to be needed to ensure "quality."
Quality is conditional, subjective and is understood differently by different people. While a consumer focuses on specification quality, producers have conformance quality in mind. Knowing the different focus of each group, we can dig deeper more specifically. To do this we can use the quality models defined in ISO 25010 which is part of the 25000 series also known as SQuaRE (Systems and software Quality Requirements and Evaluation).
The first model is quality in use. It shows how the product meets the needs of the users to achieve specific goals.

Quality In Use model

The second one is product quality and consists of eight characteristics which are composed of sub characteristics.

System/Software Product Quality model

Knowing now these quality models helps to see the usage and the product through different perspectives. With those characteristics in mind, we can enhance the requirements and refine them to be more specific about the needs. Now together with the use cases and misuse cases, we can test the code against different scenarios and functional as well as non-functional requirements. Knowing the different scenarios allows you also to build your own KPIs. With a high context coverage, you will also inherently reach a high code coverage, but this does not work the other way around.

Testing your code automatically is important and working with effective KPIs helps you to focus on blind spots. Some KPIs are more effective than others so keep in mind what data is collected and processed to display you the result since a KPI can lead to false conclusions.

Critical Security Areas That Software Engineers Have To Know To Secure Their Solutions

Vulcanus — Sat, 01 Jan 2022 12:44:16 +0000

Societies in industrialized countries depend more and more on software. The rising impact of cyber-physical systems on the real world as well as the amount of personally identifiable information collected and processed shows that systems must be secure. This is not easy to achieve due to rising complexity of those systems.

Organizations have information security departments that support securing business functions and train employees in a variety of security topics to show how to react on certain events and how to handle classified information. While there are trainings for administrators to show them how to secure a system, often little to none effort is put into teaching software engineers and developers on how to develop a secure software solution. This is quite a big issue I'd like to address and raise awareness about.

The Open Web Application Security Project (OWASP) created the "OWASP TOP 10 Proactive Controls project (OPC)" to encourage developers starting with application security. This blog entry summarizes the content of it and adds hints and information to it too. Please keep in mind that this should only raise awareness and is a starting point to help get deeper into this topic.

Define Security Requirements

Security falls under the category non-functional requirements. It should define the needed security functionality the software has to satisfy. To save up time and not re-invent the wheel on each new project, you can select security requirements from a catalog. There is a general one called "Application Security Verification Standard (ASVS)" and one for the mobile named "Mobile Application Verification Standard (MASVS)". They contain a collection of requirements which are best practices for each listed category. Fortunately they have mapped those requirements with CWE (common weakness enumeration which is basically a list of software and hardware weaknesses). Depending on the used tools, those CWEs can be automatically scanned in your code.

Leverage Security Frameworks and Libraries

Normally third-party libraries/frameworks are included into the project to re-use already written code. You should only use those from trusted sources, which are actively maintained and used by many applications. Keep them up to date and encapsulate the library, so you only expose the required need into your software. There are quite some dependency checkers out there to help you select dependencies and keep them up to date.

Secure Database Access

This has some subitems, but we'll go through them quite quickly. Secure your queries. We all know those funny SQL injection jokes, but you can solve this problem quite easily with query parameterization. You can find a cheat sheet from OWASP here and another one from Bobby Tables here.
Database management systems are not always "secure by default" configured. There are guidelines and benchmarks available out there which you should check out like here.
Access to the database should be properly authenticated. This should take place over a secure channel, and your credentials should be properly secured. Besides authenticating with credentials, you should also check out if it's possible to access it instead with your managed identity.
The last point is secure communication. Encrypting your data in transit by having a end-to-end communication security when sensitive data is transmitted over any network. This can be done via TLS. There are guides there helping you choose the minimum allowed TLS version and choosing a cipher suite.

Encode And Escape Data

And here we reach injection attacks again. By encoding characters we ensure that special characters are not processed for malicious intends. This means that the content will be displayed but not executed. For example, instead of sending "<script>" we encode special characters inputted by the user and send "<script>" which will be displayed on the browser like "<script>" but will not be executed. This output encoding should be applied before the content is passed to the target interpreter, to defend against XSS. There are some examples for C#, Java and PHP. There are also other types of encoding and injection defenses like "shell escaping" for os command input. Forms of escaping are not limited to those examples listed here. Look for guidelines and best practices when using user input for certain operations.

Validate All Inputs

You might have already noticed that you can't trust input from clients which means that the data has to be validated before usage.

Exploits of a Mom

Data should be checked that its syntax and semantic is valid.
Syntax validity ensures that data are in a expected form and should not allow any deviations. If three digits are expected, it should be checked that the input consists only of digits and has three digits in length.
Semantic validity accepts input only in an acceptable range specified by the applications functionality and its context. For example a start date has to be before an end date.

Those validations can be performed both on the client and server side, but security related validations always have to be done on the server side, since the validation on client side can be bypassed.

There are in general two approaches for syntax validation: blacklisting and whitelisting. While blacklisting blocks exact texts (e.g. "<script>") to prevent injections, is whitelisting checking for data set matches. In general it is recommended to use whitelisting on a minimal approach, instead of blacklisting since it is prone to bypasses. In other words, whitelisting limits attack surface, while blacklisting detects and stops obvious attacks.
Bear in mind that those validations have limits. Since complexity allows more variations and posibilities, valid data cann still be dangerous.

Another way to check whether data matches a specific pattern is the usage of regular expressions. This should be used with caution since expressions can get quite complex as well as hard to maintain. It also enables a regular expression denial of service attack (ReDOS) which produces a denial of service due to the exploitation of the exponential time worst-case scenario.

There are plenty validation libraries that can be leveraged to validate data. PHP has filter functions, and Java has the Hibernate Validator and C# the FluentValidation. You can also sanitize your data to erase not needed data in your input. Please keep in mind that input validation should not be your primary method to prevent injections and other attacks.

Another problem you might encounter is the validation of serialized data. Avoid deserializing data from untrusted sources. If this is not possible, you might want to implement integrity checks or encryption to prevent tampering. Enforce strict type constraints and possibly run code in a low privilege environment like in a temporary container to deserialize data. Log exceptions and failures such as the not expecting incoming type or failure in deserialization.

Apart from serialized data, there is also the problem with autobinding. Some frameworks support automatic binding of HTTP request parameters to server-side objects consumed by the application. Those bindings enable an attack vector to exploit a vulnerability called "Mass assignment". For example the user can set a parameter like "isAdmin" to true to elevate privileges. There are two ways to handle this, by either avoid autobinding and use Data Transfer Objects (TDOs) which are basically POCOs, or setup whitelist rules to define which fields are allowed to be auto-bound. There is a cheat sheet by OWASP you might to check out here to get more information on how to resolve this issue.

Enforce Access Controls

Access Controls manages the access to systems as well as resources and ensures that only authorized users/systems have access. It should be forced that all requests go through the access control to ensure that every request is checked and authorized to pass. You may already come across of the terms privilege, right and permission. Those are not interchangeable terms! For more information, read Wentz Wus article.

This topic should be thoroughly designed up front and taken early into account in the designing phase.
There are different types of access controls that should be considered (but not limited to):

Discretionary Access Control (DAC): lets people manage content by their own.
Mandatory Access Control (MAC): restricts access based on sensitivity (by a label) of the information and the authorization of the user to access such sensitivity.
Role Based Access Control (RBAC): controls access to resources based on defined rules. Performed actions are identified with roles rather than with individual subject identities.
Attribute Based Access Control (ABAC) manages the request based on policies which combines attributes of the user and object.

There are two principles that you should embed when using access controls. The first is deny by default. This means that if a request is not specifically allowed it must be denied. The second is the principle of least privilege. Ensure that only the least and only necessary access is possible. Often when time passes, it can happen that privilege creep occurs. This means that an identity accumulated access rights and has higher privileges than necessary, so keep that in mind and check regularly if certain permissions and rights are needed.

Linked to a granular access control is the programmed check of permissions. Many applications use access controls that are role based. This limits the developers and adds dangers to it.

if(user.hasRole("SuperUser") || user.hasRole("Admin"))
{
    doAction();
}

Instead your should implement rights and check for them.

if(user.hasRight("RightX"))
{
    doAction();
}

With this, you don't need to re-deploy everything if new roles are added that should have the privilege to perform actions and it is easier to maintain. It also enables for a more granular access control which helps administrators to configure the system more securely.

Furthermore, you should ensure that all access control failures should be logged to ensure non-repudiation.

Data Protection

Data can contain sensitive information which requires more protection, since it may fall under laws and regulations. It is important to classify data in your system to determine sensitivity. Depending on those classifications it may also add security requirements to the system/infrastructure that collects, processes or stores this data.

Data can be in three states. At rest, in transit or in use. Depending on the classification you have to secure the data in each state to avoid information disclosure.

Application secrets should never be stored in code, configs or other files. Keep them in a secret vault like Azure KeyVault or Amazon KMS. Besides security this also gives more flexibility on configuring your solution.

I would like to add here to also think about data retention as well as backup strategies. For data retention you have to keep laws and regulations in mind. Consult with a specialist or lawyer to know what the requirements are.

Backup strategies should not only be planned and executed, but also the results tested. There is a saying that you do not have a backup when you did not test it. In the end, you do not want to be on several tech blogs because of some deletion incident and backup failure like GitLab.

Implement Security Logging and Monitoring

The concept of Security logging is to log security information during runtime. It can be used for forensic analysis and investigations, as well as satisfying regulatory compliance requests. Monitoring is the live review of logs using various forms of automation.

Logging solutions must be designed, build and managed in a secure way. Encode and validate dangerous characters before logging to prevent injections or forging attacks. Ensure log integrity to protect against tampering. When logging, remove any sensitive information to avoid information disclosure. There should also be a common logging format to be consistent. Keep also an eye on time syncing across systems to have consistent timestamps in your logs. Furthermore, you should forward logs to a central, secure logging service to allow centralized monitoring and securing log data.

To identify potentially malicious activties, following activies can be logged as high severity:

Submitted data that is outside of an expected range
Submitted data that contains changes that should not be modifiable
Requests violating access control rules

For a more comprehensive list, check out OWASPs AppSensor Detection_points tab here.
When encountering those activties, the application should respond to possible attacks and shut them down.

There is also a small cheat sheet related to application security logging here.

Handle all Errors and Exceptions

Exceptions can happen in various ways and should be handled accordingly. This handling occurs in all areas of the application including business logic and security features. It is also important for intrusion detection. Certain attacks against the application may trigger errors which can help detect attacks in progress.

Manage exceptions in a centralized manner to avoid duplicated try/catch blocks and ensure all unexpected behavior is correctly handled. When displaying the error message to the user, be sure that you do not leak any critical information but still provide enough information to respond properly.
When logging error messages, provide enough information so that the support, forensics and incident response teams understand the problem.

To help discover possible failures early, you can also use Netflix's Chaos Monkey. It randomly terminates VMs and containers to show how resilient your services are.

Some more Information

Code Review

Besides the mentioned areas, you should also have a look at OWASP's Code Review Guide. It is quite comprehensive but also raises awareness on different topics, containing code examples and figures to illustrate in an easy way to show what has to be considered when reviewing code besides clean code practices and business logic.

Do not write your own encryption

Some developers are tempted to rollout their own encryption. Schneir's Law states

Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break.

Since those persons believe that they can not break their cipher, they use it as evidence that it is unbreakable. This is a good example of the Dunning-Kruger effect. There is a discussion on why it is discouraged to write your own encryption on Stack Exchange.
Therefore use existing solutions that are recognized by the industry and follow their best practices.