Amazon CodeGuru Reviewer: already time for retirement?

Or why you might not have heard about CodeGuru Reviewer and that's okay

Written by @samudurand

Introduction

Amazon CodeGuru Reviewer is a relatively recent AWS service (2020) which promises to improve code quality and find security vulnerabilities. Just like its partner released around the same time, CodeGuru Profiler, CodeGuru Reviewer is based on Machine Learning models built using AWS' many years of coding experience.

The recent release of a new service in the same series, CodeGuru Security, brought the two older services back to my attention. I decided it was time to look into what they offered, and whether it would be worth including in the deployment services we offer to our clients. I was also wondering why it seems to have completely disappeared less than a year after its release, at least from most community blog posts, despite regular feature releases and a new language addition (Javascript).

This article focuses on exploring and demonstrating with real examples CodeGuru Reviewer features, as well as determining its cost and potential usefulness. While reading this article, you should quickly realize that the features are not that great in its current state, and why I think that this service might altogether disappear or be merged into, or replaced by CodeGuru security.

What is CodeGuru Reviewer?

CodeGuru Reviewer is a "static" analysis tool that will connect to Git repositories, and analyze the code of the main branch to figure out if there are critical or hard-to-find bugs or issues, which could have an impact on:

• performance: deep loops, heavy logging, inefficient code...
• maintainability: bad practices, redundant code...
• concurrency: thread-safety, race conditions...
• security: hard coded secrets, un-sanitized inputs...

The analysis is performed using a set of detectors, built based on machine learning, and classified by:

• language: Java, Javascript, Python
• tags: like access-control or resource-leak
• severity level: 5 levels from Critical (example: Hardcoded Credentials) down to Info (example: Stack Trace Exposure))
• Category: either Security or Code Quality

Once it finds such issues, it will highlight and describe them in a human-friendly way in one of the following places:

• in the CodeGuru AWS console (always visible there for all analysis)
• as comments in the Pull Request analyzed (for incremental analysis)
• in the Github Security tab (when integrating via CI/CD)

Demo

I have experimented with CodeGuru Reviewer on a couple of my personal repositories. I cannot give access to my AWS account, but if you want to see what PR reviews look like with this tool, have a look at this PR on my fork of Sl4j, which I created just for this purpose.

Supported languages and repositories

CodeGuru Reviewer supports the following languages:

• Java (8?)
• Python (3.8?)
• JavaScript (ECMAScript 2021?)

Note: the versions are educated guesses, based on the experiences of myself and others with the tool, since the docs don't mention any language version (and doesn't answer community related questions).

As for Git hosting services, it supports AWS CodeCommit, Bitbucket, GitHub and Amazon S3 (not Gitlab, cloud or on-premise).

How to use it?

CodeGuru Reviewer can be used in one of two ways:

• directly as a reviewer on your repository
• as part of your CI/CD pipeline

As a reviewer

Using that approach, the configuration is simple:

1. configure CodeGuru as a reviewer in your repository
2. let CodeGuru automatically run a full repository scan (which can also be triggered manually) on the main branch
3. check the findings in the AWS console

Once CodeGuru has access to your repo, any Pull Request will trigger an incremental review, focused on the code changes you made. The findings will be shown as:

• PR comments and
• browsable items in the AWS console (just like full scans)

As part of the CI/CD

As an alternative, CodeGuru Reviewer can run reviews triggered by Github Actions. The findings will then be available in the Github "Security" tab.

You might want to use this approach if:

• you want to control when PR reviews should happen (you prefer to avoid new comments at every commits) 
• you want to integrate CodeGuru as part of a pipeline

However this approach has a few drawbacks:

• you can only use it on Github, or on other Github actions supporting pipelines (AWS recently made GH actions usable on CodeCommit)
• the code will be downloaded to an S3 bucket that you have to cleanup yourself, which can end up being pricey if the codebase is very large

Pricing

Now that we know what we can do with CodeGuru Reviewer, let's see how much it would cost us to use the pricing described in the docs.

The pricing model is based almost solely on the size of the codebase. The only other criteria is the number of full scan run, which shouldn't be more than twice a month. The number of incremental scan (in PRs) does not affect the price.

Let's take an example, and say that we have a project with:

• 3 repositories containing each 150k lines of code in their largest branch
• 3 full scans performed per month on 1 of the repositories instead of the free maximum of 2 

The result is a monthly cost of $150. A good way to see if this price is expensive or not is to compare it with other static analysis tools offering similar code quality and security checks.

The most popular competitor among those is probably SonarQube, which offers for its cloud hosting option and 500k lines of code, a similar price of $150. It is worth noting that SonarQube offers a much larger set of features and languages supported, so the comparison only by price is not exactly fair.

Where it falls short

Documentation

The documentation is not consistent and often out-of-date:

• some parts mention Javascript while the rest of the docs only mention Java and Python
• no language version is mentioned anywhere
• on the main page of CodeGuru, CodeGuru Reviewer is not listed directly anymore, suggesting changes in progress behind the scenes

Languages

Only 3 languages are supported, and not their latest versions. For Java this becomes quickly a blocker, since the supported version is only 8 (while the most recent LTS is 17), and unsupported code breaks the analysis.

Another point worth mentioning is that CodeGuru can only review one language at a time. If your project has multiple languages, you will not be able to select which you want to scan. It will be determined automatically by the number of files. From the documentation:

If a repository contains Java and Python files, then CodeGuru Reviewer generates recommendations for the language for which there are more files. [...] If the number of Java and Python files is the same, then only Java recommendations are generated.

Findings

While the security findings can be pretty elaborate and helpful, the code quality and performance focused findings are not that impressive and can often be detected by more basic or powerful tools like SonarQube (paying) or PMD (free). To see what I mean you can have a look at the list of Java code quality detectors, which is pretty short and contains a lot of simple findings like:

• general syntax error detection
• constructor deprecation in favour of builders
• simple SQS misconfigurations

There is also a few detectors capable of finding more elusive bugs but they are not many.

Conclusion

CodeGuru Reviewer offers several interesting features with useful recommendations, especially in the security domain. The findings it provides for that area are not easy to identify with standard static analysis. The fact that it can integrate as a normal reviewer or via CI/CD is also convenient and makes it easy to quickly generate results.

However, when it comes to code quality, there are many alternative tools that provide similar insights, for more languages and at a lower cost, or even for free (if you are willing to plug and configure together multiple open source tools). Furthermore, the poor quality of the documentation in some areas (which is not common for most AWS services), and the recent documentation updates removing mentions of CodeGuru Reviewer, give the impression that this tool is not the focus of the AWS team.

The final hint that something will probably happen soon was the announcement of the CodeGuru Security service, and the modification of the main CodeGuru page to point instead towards this new service. CodeGuru Security at first glance seems to be a modified version of the Reviewer, with a focus on security. This is pure speculation at this point, but I suspect that CodeGuru Reviewer will soon be either dropped or maybe merged into the newer and more focused CodeGuru Security.

So as a final advice, I would recommend not investing your time in CodeGuru Reviewer since its current state does not make it a very attractive tool, and even suggest that the time invested might be wasted if the tool is not maintained or discreetly disappears. However, I would keep an eye out for the future evolutions of CodeGuru Security, which seem much more promising.