DEV Community: Secure Code Warrior

Try This Online Java Gotchas Quiz

Alan Richardson — Fri, 19 Feb 2021 11:07:48 +0000

A previous blog post described the Java Gotcha "Bitwise vs Boolean Operator".

Java Gotchas - Bitwise vs Boolean Operator

We added a variant of this, and some other Java Gotchas into a fun little Quiz called "Challenge The Sensei".

scw.typeform.com/to/rgw7q7OE

If you've read the blog post above then you'll be in a good place to ace at least one of the questions.

But your friends might not, so if you find the quiz fun, you can share it with them and see if they score as well as you.

Since we don't want to just quiz you. We want to try and use this to help educate and codify the knowledge. So we have created a Github repo that has runnable code examples for the problem, and the solution.

github.com/SecureCodeWarrior/challenge-the-sensei

This is a Sensei enabled repo.

When you clone the repo and load it into IntelliJ, assuming you have the Secure Code Warrior Sensei IntelliJ Plugin installed, then it will automatically see that you have a .sensei folder, and load the Sensei recipes.

When browsing the code in the IDE you should see IntelliJ prompt you that the error exists in the code, and this should make it easier to see the gotcha in the code:

Hover over the highlighted code then you'll see a prompt telling you about the error
Use the Show Context Action key: alt+enter (Windows) option+enter (macOS) and we may have a QuickFix available that can fix the code.

Sensei recipes have been added for:

We are adding more recipes and more explanatory text to cover the rest of the code in the future... but don't let that stop you from having a look at the code and spotting the error yourself.

And remember to try the quiz and "Challenge the Sensei"

Running IntelliJ Inspections From Continuous Integration

Alan Richardson — Mon, 15 Feb 2021 14:06:05 +0000

IntelliJ IDEA offers functionality to help improve our coding, within the IDE when writing code as Intentions. Intentions can be used in batch to inspect code for patterns throughout the source and even extending to Command-Line analysis or added to Continuous Integration. This post covers the out of the box IntelliJ functionality and expanding with custom Intentions created in Sensei.

IntelliJ Inspections

The Inspections feature of IntelliJ drives the display of many of the errors that are reported dynamically in the IDE when coding e.g.

detecting abstract classes that can be converted to interfaces,
identifying redundant class fields which can be local,
warning about uses of deprecated methods,
etc.

These Inspections highlight code that matches in the IDE as Intention Actions which often have an associated QuickFix.

The real-time IDE highlighting when code matches an Inspection can help us improve our coding dynamically. After identifying the issue in the code, using IntelliJ Intention Actions to QuickFix the code can reinforce better patterns.

Inspections Profile

Inspections can run as a batch from within the IDE, and from the Command Line or in a Continuous Integration process.

The key to working with IntelliJ inspections in a batch is through the use of an Inspections Profile.

IntelliJ has two default Inspection Profiles: one stored in the Project, and one stored in the IDE.

New Inspection Profiles can be created to configure specific plugins or use-cases e.g.

Run Checkstyle real-time scan only
Run a specific set of Sensei rules
Run the HTML checks

The Inspections in a profile can be enabled or disabled from the IntelliJ Preferences. The Preferences dialog is also an easy way to learn the range of Inspections available.

The ‘tool’ icon allows you to duplicate a Profile and create a new Profile to collect a specific set of rules.

Running an Inspection Profile in the IDE

Inspection Profiles can be run from within the IDE using the Analyze \ Inspect Code... menu.

The Analyze functionality allows you to control the scope that the Inspection will run against e.g. the whole project, including or excluding test sources, or against a specific set of files.

You can also manage the Inspection Profiles from here to create or configure a particular profile.

Clicking [OK] on the “Specify Inspection Scope” dialog will trigger IntelliJ into running all the selected Inspections in the profile across the defined scope.

IntelliJ will report the results of running the Inspections in the Inspection Results tab.

The Sensei plugin from Secure Code Warrior allows you to create custom code matching recipes. Sensei tightly integrates with IntelliJ to make these custom recipes as natural to use as the IntelliJ Intention Actions. Meaning they are loaded into IntelliJ as Inspections and can be grouped, enabled, and disabled using Inspection Profiles. Creating a custom Inspection Profile and then using the Analyze Inspect Code functionality is the recommended way of running Sensei recipes in bulk across a project.

Running an Inspection Profile from the Command Line

IntelliJ has the ability to run inspections from the command line as documented by JetBrains:

https://www.jetbrains.com/help/idea/working-with-the-ide-features-from-command-line.html

I primarily use Mac OS, and can run a single instance of IntelliJ from the command line with:

open -na "IntelliJ IDEA CE.app"

To support easier execution I add this to a shell command script.

vi /usr/local/bin/idea

The contents of the script are from the official documentation provided by IntelliJ.

#!/bin/sh

open -na "IntelliJ IDEA CE.app" --args "$@"

I then made this executable to allow me to simplify the command line inspection process.

chmod 755 /usr/local/bin/idea

The official intellij docs describe the general form of the inspection command as:

idea inspect <project> <inspection-profile> <output> [<options>]

In practice, I fully qualify the paths:

idea inspect /Users/user/GitHub/sensei-blog-examples /Users/user/GitHub/sensei-blog-examples/.idea/inspectionProfiles/senseiprofile.xml /Users/user/GitHub/sensei-blog-examples/scan-results

This runs all the Inspections I added to the senseiprofile and reports the results in the scan-results folder.

Viewing Inspection Results

We can report these results from within Continuous Integration, as we’ll see later.

We can also view them within IntelliJ itself using the Analyse \ View Offline Inspection Results… feature.

This will load the results into the Inspection Results tab.

This is officially documented on the JetBrains site:

https://www.jetbrains.com/help/idea/command-line-code-inspector.html#inspection-results

This might be used during a code review process if the command line execution was incorporated into a Continuous Integration process and the reviewers wanted to check the full source context of any of the Inspection result entries.

Inspection Profiles in Continuous Integration

When adding the Command Line inspection into Continuous Integration we ideally want a report to be generated automatically and there are a number of options open to us.

TeamCity offers out of the box support for Inspection Profiles in Continuous Integration.

https://www.jetbrains.com/help/teamcity/inspections.html

The Jenkins Warnings NG plugin supports the Command Line output from IntelliJ Inspections as one of the report formats.

Community projects like idea CLI Inspector exist to support using Inspection Profiles in other CI tooling i.e.

https://github.com/bentolor/idea-cli-inspector

The future of Inspection Profiles in a CI process looks even brighter with the introduction of the JetBrains Qodana project. The Qodana project is a headless version of IntelliJ with official Github Actions and Docker images.

https://github.com/JetBrains/Qodana

Qodana is currently in beta, but the Sensei team is monitoring it so that it becomes an officially supported platform for running Sensei rules as part of Continuous Integration.

Summary

Intention Actions allow us to reinforce coding patterns and quickly fix them in the IDE when we make mistakes during coding.

Inspection Profiles allow us to collect these into profiles which can run in batch as an Analyze and Inspect Code action. This can be useful if we encounter a pattern and want to double-check if we have missed that anywhere else in our code.

Inspection Profiles can be run from the command line and even incorporated into Continuous Integration processes supporting a “trust, but verify” model and catch any accidental slippage.

All of the above is built in IntelliJ functionality and JetBrains are improving their Continuous Integration process with the introduction of Qodana.

Sensei recipes are loaded into IntelliJ to act as native Intention Actions and be collected into Inspection Profiles to support batch checking through Inspect Code and Continuous Integration support provided by the official JetBrains Command Line execution functionality.

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for "sensei secure code". Or install from the Jetbrains Marketplace Online

Learn more about Sensei

Java Gotchas - Bitwise vs Boolean Operators

Alan Richardson — Thu, 11 Feb 2021 15:09:24 +0000

Java Gotchas - Bitwise vs Boolean Operators

"Java Gotcha" - a common mistake pattern that is easy to accidentally implement.

A fairly simple Java Gotcha to accidentally fall into is: using a Bitwise operator instead of a Boolean Comparison operator.

e.g. a simple mistype can result in writing "&" when you really meant to write "&&".

A common heuristic we learn when reviewing code is:

"&" or "|" when used in a conditional statement is probably not intended.

In this section we will explore this heuristic a little and identify ways we can identify and fix this.

What's the Problem? Bitwise operations work fine with Booleans

Using Bitwise operators with Booleans is perfectly valid, so Java will not report a syntax error.

If I construct a JUnit Test to explore a truth table for both Bitwise OR (|) and Bitwise AND (&) then we will see that the outputs from the Bitwise operator match the truth table. Given this, we might think that the use of Bitwise operators is not an issue.

AND Truth Table

    @Test
    void bitwiseOperatorsAndTruthTable(){
        Assertions.assertEquals(true, true & true);
        Assertions.assertEquals(false, true & false);
        Assertions.assertEquals(false, false & true);
        Assertions.assertEquals(false, false & false);
    }

OR Truth Table

    @Test
    void bitwiseOperatorsOrTruthTable(){
        Assertions.assertEquals(true, true | true);
        Assertions.assertEquals(true, true | false);
        Assertions.assertEquals(true, false | true);
        Assertions.assertEquals(false, false | false);
    }

Truth table images were created using the web.standfor.edu truth table tool.

Issue: Short Circuit Operation

The real issue is the difference in behaviour between Bitwise (&, |) and Boolean (&&, ||) operators.

A Boolean operator is a short circuit operator and only evaluates as much as it needs to.

e.g.

if (args != null & args.length() > 23) {
    System.out.println(args);
}

In the above code, both boolean conditions will evaluate, because the Bitwise operator has been used:

args != null
args.length() > 23

This leaves my code open to a NullPointerException if args is null because we will always perform the check for args.length, even when args is null because both boolean conditions have to be evaluated.

Boolean Operators Short Circuit Evaluation

When an && is used e.g.

if (args != null && args.length() > 23) {
    System.out.println(args);
}

As soon as we know that args != null evaluates to false the condition expression evaluation stops.

We don't need to evaluate the right-hand side.

Whatever the outcome of the right-hand side condition, the final value of the Boolean expression will be false.

But this would never happen in production code

This is a pretty easy mistake to make, and is not always picked up by Static Analysis tools.

I used the following Google Dork to see if I could find any public examples of this pattern:

filetype:java if "!=null & "

This search brought back some code from Android in the RootWindowContainer

isDocument = intent != null & intent.isDocument()

This is the type of code that might pass a code review because we often do use Bitwise operators in assignment statements to mask values. But in this instance, the outcome is the same as the if statement example above. If intent is ever null, then a NullPointerException will be thrown.

Very often we get away with this construct because we often code defensively and write redundant code. The check for != null may well be redundant in most use cases.

This is an error made by programmers in production code.

I don't know how current the results for the search are, but when I ran the search there were results back with code from: Google, Amazon, Apache... and me.

A recent pull request on one of my open source projects was to address exactly this error.

if(type!=null & type.trim().length()>0){
    acceptMediaTypeDefinitionsList.add(type.trim());
}

How to Find This

When I checked my sample code in a few static analysers, none of them picked up this hidden self-destruct code.

As a team at Secure Code Warrior, we created and reviewed a fairly simple Sensei recipe that could pick this up.

Because Bitwise operators are perfectly valid, and often used in assignments we focussed on the use-case of if statements, and the use of Bitwise &, to find the problematic code.

search:
  expression:
    anyOf:
    - in:
        condition: {}
    value:
      caseSensitive: false
      matches: ".* & .*"

This uses a regular expression to match " & " when it is used as a condition expression. e.g. in an if statement.

To fix it, we again relied on regular expressions. This time using the sed function in the QuickFix to globally replace the & in the expression with &&.

availableFixes:
  - name: "Replace bitwise AND operator to logical AND operator"
    actions:
      - rewrite:
          to: "{{#sed}}s/&/&&/g,{{{ . }}}{{/sed}}"

End Notes

This covers the most common misuse of Bitwise operator, i.e. when a Boolean operator was actually intended.

There are other situations where this could crop up e.g. the assignment example, but when writing recipes we have to attempt to avoid false-positive identification, otherwise recipes will be ignored or turned off. We build recipes to match the most common occurrences. As Sensei evolves, we may well add additional specificity into the search functionality to cover more matching conditions.

In its current form, this recipe would identify many of the live use-cases, and most importantly, the one that was reported in my project.

NOTE: A fair few secure code warriors contributed to this example and recipe review - Charlie Eriksen, Matthieu Calie, Robin Claerhaut, Brysen Ackx, Nathan Desmet, Downey Robersscheuten. Thanks for your help.

The source code and recipes for this post can be found in the sensei-blog-examples repository in the Secure Code Warrior GitHub account, in the pojoexamples module.

https://github.com/securecodewarrior/sensei-blog-examples

Learn more about Sensei

What is Static Analysis

Alan Richardson — Mon, 01 Feb 2021 11:34:49 +0000

TLDR; An overview of Static Analysis and some IntelliJ IDE plugins which can help you improve your coding process.

What is Static Analysis?

Static Analysis is the automated analysis of source code without executing the application.

When the analysis is performed during program execution then it is known as Dynamic Analysis.

Static Analysis is often used to detect:

Security vulnerabilities.
Performance issues.
Non-compliance with standards.
Use of out of date programming constructs.

How does a Static Analysis tool work?

The basic concept common to all Static Analysis tools is searching source code to identify specific coding patterns that have some sort of warning or information associated with them.

This could be as simple as "JUnit 5 test classes do not need to be 'public'". Or something complex to identify like "Untrusted String input being used in an SQL execution statement".

Static Analysis tools vary in how they implement this functionality.

source code parsing technology to create an Abstract Syntax Tree (AST),
text Regular Expression matching,
a combination of the above.

Regular Expression matching on text is very flexible, easy to write rules to match, but can often lead to a lot of false positives and the matching rules are ignorant of the surrounding code context.

AST matching treats the source code as program code, and not just files filled with text, this allows for more specific, contextual matching and can reduce the number of false positives reported against the code.

Static Analysis in Continuous Integration

Static Analysis is often performed during the Continous Integration (CI) process to generate a report of compliance issues which can be reviewed to receive an objective view of the code-base over time.

Some people use Static Analysis as an objective measure of their code quality by configuring the static analysis tool to only measure specific parts of the code, and only report on a subset of rules.

The objectivity is provided by the rules used since these do not vary in their evaluation of code over time. Clearly, the combination of rules used and their configuration is a subjective decision and different teams choose to use different rules at different times.

Having the Static Analysis performed in CI is useful but might delay the feedback to the programmer. Programmers don't receive feedback when coding, they receive feedback later when the code is run through the Static Analysis tool. Another side-effect of running the Static Analysis in CI is that the results are easier to ignore.

To help make teams pay more attention to the results from Static Analysis it is usually possible to configure a threshold metric in the build process to fail the build if the metric is exceeded e.g. a number of rules triggered.

Static Analysis in the IDE

To receive feedback faster, there are many IDE plugins that run the Static Analysis rules in the IDE on demand, or periodically as the code changes.

The rule violations can then be seen in the IDE as the programmer is writing code, and to make the rules harder to ignore, the violations can often be configured to render as underlined code in the editor.

I personally find this a useful way to improve my coding, particularly when working with a new library that is covered by the Static Analysis tool. Although it can be 'noisy' with false positives, or rules you are not interested in. But this is solved by taking the extra step to configure the Static Analysis tool to ignore certain rules.

Fixing Code Based on Static Analysis Rules

With most Static Analysis tools, the fixing of the rule is left to the programmer, so they have to understand the cause of the rule violation and how to fix it.

Very few static analysis tools also include the ability to fix the violations because the fix is so often contextual to the team and the technology used and their agreed coding styles.

Default Rules

False confidence in the quality of the rules may arise when the Static Analysis tools come with default rules, it is tempting to believe that they cover all the issues that a programmer might encounter, and all the circumstances that rule should apply. Sometimes the circumstances in which a rule should apply can be subtle and may not be easy to detect.

The hope is that by using a Static Analysis tool, and researching the rules and violations in more detail, that programmers will develop the skill to detect and avoid the issue in the context of their specific domain.

When the domain requires contextual rules, the Static Analysis tools may not have any rules that match your domain or library, and additionally, the tools can often be difficult to configure and expand.

Annoyances

None of these 'annoyances' are insurmountable:

false positives
lack of fixes
configuration to ignore rules
adding context-specific rules

But they are often used as excuses to avoid using the Static Analysis tools in the first place, which is a pity because the use of Static Analysis can be enormously useful, as a way to:

highlight better approaches to junior developers
gain fast feedback on clear coding violations
identify obscure issues that the programmer has not encountered before
reinforce that the programmer has adopted a good coding approach (when no violations are reported)

IDE Based Static Analysis Tools

As an individual contributor to a project, I like to use Static Analysis tools that run from within the IDE so that I receive fast feedback on my code.

This supplements any pull request review process, and CI integration that a project may have.

I try to identify tools that will give me an edge, and improve my individual workflow.

When tools run in the IDE, because they tend to share the same basic GUI and configuration approach, it can be tempting to view them interchangeably.

The tools may have overlapping functionality or rule sets but to gain maximum advantage I install multiple tools to take advantage of their strengths.

The Static Analysis IDE tools I actively use when coding are:

the inbuilt IntelliJ Inspections - common coding patterns
SpotBugs - common errors
SonarLint - common usage patterns
CheckStyle - common style patterns
Sensei from Secure Code Warrior - custom rule creation

I use them all because they work well together to augment and supplement each other.

IntelliJ Inspections

If you use IntelliJ then you are already using their Inspections.

These are Static Analysis rules which are flagged in the IDE. Some of them also have QuickFix options to rewrite the code to address the issue.

The rules are configurable on and off, and to choose the error level used to highlight it in the IDE.

There are a lot of good IntelliJ Inspections. I know that because I read through them while writing this. I use the IntelliJ Inspections as the defaults and haven't configured them, but to gain full value from the Inspections you should read through them, identify those relevant to your coding style, and configure the warning level so that you notice them in the code.

The great thing about the IntelliJ Inspections is that they come free with the IDE and they help build the muscle memory of:

noticing warnings and errors in the source as you write code
hovering the mouse over flagged code to learn the rule violations
using alt+enter to apply a QuickFix for the issue

SpotBugs

The SpotBugs IntelliJ plugin uses Static Analysis to try and alert you to bugs in your code.

SpotBugs can be configured from the IntelliJ Preferences to scan your code, the actual rules used can be found in the Detector tab.

I tend to use SpotBugs after I've written and reviewed my code, then I'll run the 'Analyze Project Files Including Test Sources'.

This does help me identify bugs, dead code, and obvious optimizations. It also forces me to research some of the flagged violations to help me decide what to do.

SpotBugs will find issues but does not offer any QuickFix actions to attempt to resolve the issues.

SpotBugs is easy to configure and I find it to be a useful objective second opinion to consult in my IDE.

SonarLint

The SonarLint plugin.

SonarLint can be configured from the IntelliJ Preferences to select which rules the code is validated against.

By default, SonarLint runs in realtime and shows issues for the current code that you are editing.

SonarLint does not offer quick fixes but the documentation associated with the violation reports is usually clear and well documented.

I've found SonarLint to be useful in the past for alerting me to new Java features that I was aware of in the newer versions of Java.

CheckStyle

The CheckStyle plugin offers a mix of formatting and code-quality rules.

The CheckStyle plugin comes bundled with 'Sun Checks' and 'Google Checks'.

The definitions of these can be easily found online.

CheckStyle adds the most value when a project has spent the time creating its own ruleset. Then the IDE plugin can be configured to use that ruleset and programmers can perform a scan, prior to committing the code to CI.

CheckStyle is very often used as a build failing plugin for CI processes when the number of CheckStyle violations exceeds a threshold.

Sensei

Sensei uses Static Analysis based on an Abstract Syntax Tree (AST) for matching code and for creating QuickFixes, this allows for very specific identification of code with issues.

The AST allows QuickFixes associated with a recipe to understand the surrounding code e.g. when adding a new class into the code, any import for that class will only be added to the source file once, and not for each replacement.

Sensei was created to make it easy to build custom matching rules which may not exist, or which would be hard to configure, in other tools.

Rather than amend a configuration file, all the configuration can be performed in the GUI. When creating new recipes the GUI makes it easy to see which code the recipe matches. And when defining the QuickFixes the before and after state of the code can be compared immediately. This makes it easier to create very contextual recipes i.e. unique to teams, or technology, and even individual programmers.

I use Sensei in combination with other Static Analysis tools e.g. most Static Analysis tools will find issues, but not fix them. A common use case for Sensei is to replicate the other tool's matching search in Sensei, and expand it with a Quick Fix. This has the benefit that the custom fix applied already meets the coding standards for your project.

I periodically find myself creating Sensei recipes that already exist in the IntelliJ Intensions set because the Intension report doesn't quite match the context I've created or because the QuickFix provided by IntelliJ doesn't match the code pattern I want to use.

I augment the existing tools, rather than attempt to fully replace them.

Sensei can also be very useful when you identify a contextual variant of a common rule e.g. if you are using an SQL library not supported by the Static Analysis tool, but the common SQL rules in the Static Analysis engine still apply, then you can create library-specific variants of those rules using Sensei.

Sensei does not come out of the box with a lot of generic recipes like the Static Analysis tools mentioned, its strength is in making it easy to create new recipes, complete with QuickFixes configured to match your specific coding style and use-cases.

NOTE: we are working on a public repository of recipes to cover generic use-cases, and you can find it here.

Summary

I tend to pick tools that work together, are configurable, and easy to expand to meet my specific context. I've used Static Analysis tools in the IDE for years to help me identify issues, and learn more about the programming languages and libraries I use.

I use all of the tools mentioned in combination:

IntelliJ Intentions helps flag common code issues out the box, often with associated QuickFixes.
SpotBugs finds simple bugs I might have missed and alerts me to performance issues.
SonarLint identifies Java features I was unaware of and prompts me to additional ways of modelling my code.
CheckStyle helps me conform to an agreed coding style that is also enforced during CI.
Sensei helps me create QuickFixes to augment common scenarios found by Static Analysis tools and create specific project or technology recipes that can be hard to configure in another tool.

Learn more about Sensei

How to catch and fix a Guice dependency injection issue using Sensei

Alan Richardson — Mon, 25 Jan 2021 10:18:42 +0000

How to catch and fix a dependency injection issue using Sensei

The Sensei project itself has its own set of recipes that have built up over time, this is an example of one of the scenarios the Sensei team built a recipe to cover. A misconfiguration of Guice, which led to a NullPointerException being reported at runtime during our testing.

This could be generalised to many Dependency Injection scenarios where code is syntactically correct, but because the wiring configuration was incorrect, an error slips through.

This often happens when we are learning the technology, and we keep making the simple mistake of
forgetting to wire things up. But this also happens to experienced professionals because, well... we all make mistakes, and we may not have Unit Tests to cover everything.

RunTime Exceptions from Incorrect Dependency Injection Wiring

The code below fails at runtime with a NullPointerException.

Injector injector = Guice.createInjector(
                 new SystemOutModule());
CountReporter reporter = injector.getInstance(
                  CountReporter.class);

String [] lines5 = {"1: line", "2: line",
              "3: line", "4: line", "5: line"};

reporter.reportThisMany(Arrays.asList(lines5));
Assertions.assertEquals(5, reporter.getCount());

The code is syntactically correct but fails because we missed out a requestStaticInjection in our SystemOutModule configuration.

public class SystemOutModule extends AbstractModule {

    @Override
    protected void configure() {
        binder().bind(ILineReporter.class).to(
             SystemOutReporter.class);
    }
}

When we try to use the reporter, created using the Injector, it is not fully instantiated and we receive a NullPointerException when we call reportThisMany.

We may well have missed that in our code review, or we didn't have unit tests that triggered the dependency injection, and it slipped out into our build.

Warning Signs

In this case, there is a warning sign, the CountReporter has a static field annotated with @Inject but... the CountReporter class itself is package private. In a complicated code base this could be a warning sign that it isn't used because the Module class configuring the bindings needs to be in the same package for this to work.

class CountReporter {

    @Inject
    private static ILineReporter reporter;

Another error that we made, which we might have picked up in a code review, is that we forgot to actually bind the fields in the SystemOutModule configure method.

binder().requestStaticInjection(
                   CountReporter.class);

Had we written the requestStaticInjection code then the Syntax Error generated when trying to use the CountReporter would have alerted us to the simple error.

> 'reporters.CountReporter' is not public in 'reporters'.
  Cannot be accessed from outside package

Sadly. We forgot, and there were no syntactic warning signs in the code.

How could Sensei help?

We probably wouldn't use Sensei to pick up the missing requestStaticInjection since all
our Guice configuration wiring would need to use that method, and we can't guarantee that
all wiring is going to be as simple as this use-case.

We could write a Sensei rule to look for some warning signs that our code is not up to scratch.

In this case that would mean:

Find any classes with @Inject annotated fields
Where the classes are not public.

The above was the warning sign that they were unlikely to have been wired up.

By creating a recipe, then we will have a warning sign early, during the coding, and reduce the reliance on our pull requests or resolving our tech debt to allow us to add Unit Tests.

How to create a recipe?

The task I want to complete is:

Create a recipe that matches fields annotated with @Inject which are in protected private classes

That should hopefully give us enough warning to identify any Modules using it and add the missing wiring code.

In my CountReporter class, I will use Alt+Enter to Create a new Recipe and I will start from scratch

I will name this and add a description:

Name: Guice: Injected Field Not Public
Description: If the Injected field is not public then the code might not be wired up
Level: Warning

The search I write looks for a field with the Inject annotation which has not been scoped as public.

search:
  field:
    with:
      annotation:
        type: "com.google.inject.Inject"
    in:
      class:
        without:
          modifier: "public"

Fix

The QuickFix in the recipe will amend the injected class, but that is not the only code I have to change.

availableFixes:
- name: "Change class to public. Remember to also request injection on this class"
  actions:
  - changeModifiers:
      visibility: "public"
      target: "parentClass"

When the recipe is triggered I still have a manual step to perform in my code, adding the line containing requestStaticInjection to fully instantiate the object.

public class SystemOutModule extends AbstractModule {

    @Override
    protected void configure() {
        binder().bind(ILineReporter.class).to(SystemOutReporter.class);
        // instantiate via dependency injection
        binder().requestStaticInjection(
                      CountReporter.class);
    }
}

I could potentially write another recipe to pick this up. I probably wouldn't do that unless forgetting to add the static injection became a semi-regular error that I made when coding.

Summary

If we ever find ourselves making a mistake with a common root pattern, then Sensei can help codify the knowledge around detecting and fixing the issue, and then hopefully, it won't slip through code reviews and into production.

Sometimes the recipes we write identify heuristic patterns i.e. matching them doesn't guarantee that there is a problem, but is likely that there is a problem.

Also, the recipes and QuickFixes that we write, don't have to be fully comprehensive, they need to be good enough that they help us identify and fix problems without being overcomplicated. Because when they become overcomplicated they become harder to understand and harder to maintain.

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code”.

The source code and recipes for this post can be found in the sensei-blog-examples repository in the Secure Code Warrior GitHub account, in the guiceexamples module.

https://github.com/securecodewarrior/sensei-blog-examples

Learn more about Sensei

Automatically Amending Method and Class Visibility for JUnit 5

Alan Richardson — Thu, 14 Jan 2021 14:29:43 +0000

One of the joys of programming is the constant learning required to keep up to date. One of the issues is that we build up familiarity and patterns of usage that can impact the adoption of new approaches. Sensei can help migration by identifying deprecated patterns and prompting us with the fix to use going forward.

As an example, when I migrated from JUnit 4 to JUnit 5, I was used to writing all my test classes and methods as public. But with JUnit 5 they can be package private.

e.g. instead of:

public class Junit5VisibilityTest {

    @Test
    public void thisDoesNotNeedToBePublic(){
        Assertions.assertTrue(true);
    }
}

I really want to write:

class Junit5VisibilityTest {

    @Test
    void thisDoesNotNeedToBePublic(){
        Assertions.assertTrue(true);
    }
}

It took me a while to build the muscle memory to code to this, and I still slip up once in a while.

Using Sensei

With Sensei I can create recipes that find the public methods and classes, and amend the declarations to be package private automatically.

To achieve this I created a recipe:

Name - JUnit: JUnit 5 test methods do not need to be public
Description - JUnit 5 test methods do not need public visibility
Level - Error

I classed it as Error because I want to stamp out this coding practice and I want higher visibility of the issue when I'm writing code in the IDE.

Amending the Class Declaration

To find the classes, I search for any class which has a child annotation of @Test from Junit 5 i.e. org.junit.jupiter.api.Test

And where the class has modifier public:

search:
  class:
    with:
      child:
        annotation:
          type: "org.junit.jupiter.api.Test"
    modifier: "public"

Then the quick fix changes the modifier to remove the visibility so that it is the default, and the default is package private which is what I'm looking for.

availableFixes:
- name: "remove public visibility from JUnit 5  Test class"
  actions:
  - changeModifiers:
      visibility: ""

Amending the Method Declarations

The method declaration amendment recipe is much the same as the class recipe.

First I search for public methods annotated with @Test from JUnit 5.

search:
  method:
    annotation:
      type: "org.junit.jupiter.api.Test"
    modifier: "public"

And then I change the modifier to be default visibility.

availableFixes:
- name: "Remove @Test method public visibility"
  actions:
  - changeModifiers:
      visibility: ""

Hint: Amending Multiple Methods

Sensei has the ability to apply the QuickFix to all the violations in the current file.

When I use alt+enter to apply the QuickFix.

If I expand the QuickFix name menu, I can see an option to:

"Fix All: 'JUnit: JUnit 5 test methods do not need to be public' problems in the file"

When I select that option then Sensei will amend all the occurrences of the problem, not just the one I select.

Amending the class

In the same way that a method does not need to be public, neither does the class.

I can create a recipe and a QuickFix to amend the class.

Name - JUnit: Junit 5 Test classes do not need to be public
Description - Junit 5 Test classes do not need to be public
Level - Error

When I find a class that is public and has a method with a @Test annotation. Then I want to change the visibility.

search:
  class:
    modifier: "public"
    anyOf:
    - child:
        method:
          annotation:
            type: "Test"

I can make the change to the class definition with the changeModifiers action again.

availableFixes:
- name: "Remove @Test class public visibility"
  actions:
  - changeModifiers:
      visibility: ""

Summary

A static analysis tool initially alerted me to this recommended approach in JUnit. But the static analysis tool didn't help me build the muscle memory to change my code as I program.

Use the 'Level' to alert you. When it is a problem I am trying to stamp out in my coding I initially make it 'Error' and then reduce this as I wean myself off the coding approach.

Remember you can use Sensei to fix all the issues in the current file at the same time, by using the drop-down menu option when applying the QuickFix.

By creating a Sensei recipe, I can see my old coding approach in real-time. And QuickFix it, to reinforce the approach if I occasionally slip up in my coding.

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code”.

The source code and recipes for this can be found in the sensei-blog-examples repository in the Secure Code Warrior GitHub account, in the junitexamples module.

github.com/securecodewarrior/sensei-blog-examples

Automatically Adding a Private Constructor with Sensei

Alan Richardson — Wed, 13 Jan 2021 16:20:41 +0000

TLDR; Sensei can identify a Utility class coding pattern, and automatically generate a private constructor to make it impossible to instantiate the class.

In a utility class, when the fields and methods are static, there is no obvious reason why I would ever instantiate it.

e.g. UtilityClass utility = new UtilityClass();

The code below is a simple implementation of a Utility class.

public class UtilityClass {

    public static final Boolean ULTIMATE_TRUTH = true;

    public static boolean getTrue(){
        return ULTIMATE_TRUTH;
    }
}

This is the type of coding pattern that Static Analysis tools can pick up, but they often don't supply the ability to fix the issue.

I can use Sensei to identify the coding pattern, and automatically generate a private constructor to make it impossible for me to instantiate the class.

Now that I know I can fix the problem. I’ll refine the search conditions to show the recipe when it is most appropriate.

Searching for the Class

I create a new recipe on the Utility class called Static Classes: create private constructor.

Name: Static Classes: create private constructor
Description: create a private constructor for static classes
Level: Error

And initially, I'll search for a class.

search:
  class: {}

This will match any class, which is good enough to let me get started writing a Quick Fix, and once I have a Quick Fix that works, I'll refine the search to make it highlight when there is more likely to be a class that requires a private constructor.

Quick Fix

For the Quick Fix, I will want to generate a private constructor.

In the example class this would look like:

~~~~
private UtilityClass(){}
~~~~

To add the above code to my class, my Quick Fix will add a Method, and the name of the method will be a Mustache template that uses the name of the class.

availableFixes:
- name: "add private constructor"
  actions:
  - addMethod:
      method: "private {{{ name }}}(){}"

In the GUI Editor, I use the Show Variables to create the Mustache template and then edit the field to add the private modifier, brackets, and braces to make it syntactically correct.

And this would now allow me to add a private constructor to any class.

The QuickFix preview helps me because I can see the generated code as I write the Mustache template.

Search for Missing Constructors

Ideally, I don't want to create a recipe that flags an error against every class. So I'll add some additional conditions in the search so that we only match on classes that do not have a constructor.

search:
  class:
    without:
      child:
        method:
          constructor: true

The YAML is slightly different from the GUI.

In the GUI I configure it to look for a class without a child method which is a constructor 'yes'. We use 'yes' in the GUI instead of 'true' to make the GUI a little more human friendly.

This recipe will now only reveal itself for any class without a constructor.

Narrow Search for Likely Culprits

So I might want to go further and look for the presence of static methods or fields.

I look for any class without a constructor and which has either all public static fields or all public static methods.

~~~~
search:
class:
with:
anyOf:
- child:
method:
allOf:
- modifier: "public"
- modifier: "static"
- child:
field:
allOf:
- modifier: "static"
- modifier: "public"
without:
child:
method:
constructor: true
~~~~

Since Sensei is used to help me, as a programmer, in the IDE, rather than to statically analyze the code and report all errors, this filter is good enough to rule out most classes in my code base where I might have a good reason to have a default public constructor.

In some projects, this might be a step too far because the utility classes could have private methods, so I might choose to look for the presence of 'any' public static methods, rather than 'all'.

~~~~
- child:
field:
anyOf:
- modifier: "static"
- modifier: "public"
~~~~

Hints

Sensei is not designed to replace a Static Analysis tool. Sensei can augment a Static Analysis tool for common issues associated with your coding process, or technology. By replicating enough of the matching to highlight an issue, and supporting the development process by generating the QuickFix code.

What I'm trying to do is create a simple enough recipe that includes all the situations I need it, but filter it so that it doesn't get suggested in every class.

When working on recipes I try to de-risk them, in this case, I wasn't sure if I could create the private constructor so I created the QuickFix first. Then refactored the search conditions to make them more specific.

Sometimes when working on recipes I'm not sure how to perform the search, so I work on that first.

I find recipes easier to create when I build them incrementally, switching between refactoring of the QuickFix and the search.

Code Links

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code”.

The source code and recipes for this can be found in the sensei-blog-examples repository in the Secure Code Warror github account, in the pojoexamples module.

https://github.com/securecodewarrior/sensei-blog-examples

Improving A Personal Programming Process Using Sensei

Alan Richardson — Thu, 10 Dec 2020 15:36:05 +0000

Adding Annotations and Method Renaming

For this post, I've recreated a 'bad' coding approach that I used when I was learning JUnit, and will demonstrate how to convert the ‘bad’ pattern to an agreed, and "better", coding pattern using Sensei.

When I was learning JUnit, I could only keep so much in my head at any one time. I constantly forgot how to skip tests when they were not working.

If we are working in a Team then we can use code reviews on pull requests to help enforce coding styles. And we can shorten the feedback cycle when pair programming with a more experienced programmer.

We can also augment our process with tooling and have the tools prompt us to do the right thing. Thoughtworks describe this as "tools over rules," in their Technology Radar listing for Sensei:

"make it easy to do the right thing over applying checklist-like governance rules and procedures"

Disabling a JUnit Test

Ideally, I would, as we all know, use the @Disabled annotation and write:

~~~~
@Disabled
@test
void canWeAddTwoNumbers(){
Assertions.fail("this test was skipped and should not run");
}
~~~~

But, when learning, I had to train myself to use @Disabled.

When I forgot how to disable a Test method I would remove the @Test annotation and rename the test:

~~~~
class SkipThisTest {

void SKIPTHIScanWeAddTwoNumbers(){
    Assertions.fail("this test was skipped and should not run");
}

}
~~~~

It wasn't good, but it got the job done. I didn't have something like Sensei to help me remember and so I fell into using poor coding patterns.

The tasks I've taken on board for this readme are to:

Create a rule which finds methods that have been 'skipped' or 'disabled' by renaming the method.
Create a QuickFix to rename the method and add both an @test and @Disabled annotation.

Recipe Settings

The first step I take with Sensei is to "add new recipe" and search for the coding pattern I want the recipe to act on.

Name: JUnit: Make @Disabled @Test from SKIPTHIS
Short Description: Stop naming methods SKIPTHIS, use @Disabled @Test instead

And my search is very simple. I use a basic regex to match the method name.

~~~~
search:
method:
name:
matches: "SKIPTHIS.*"
~~~~

QuickFix Settings

The QuickFix is a little more complicated because it will rewrite the code, and I'll use a few steps to achieve my final code.

I want to:

add an @Test annotation to the method
add an @Disabled annotation to the method
amend the method name

Adding the annotations is simple enough using the addAnnotation fix. If I use a fully qualified name for the annotation then Sensei will automatically add the imports for me.

~~~~
availableFixes:

name: "Add @Disabled and @test Annotation" actions:
- addAnnotation: annotation: "@org.junit.jupiter.api.Test"
- addAnnotation: annotation: "@org.junit.jupiter.api.Disabled" ~~~~

The actual renaming seems a little more complicated but I'm just using a regex replacement, and the generic way to do this with Sensei is to use sed in a rewrite action.

Because the rewrite actions a Mustache templates, Sensei has so functional extensions in the template mechanism. A function is represented with {{#...}} so for sed the function is {{#sed}}. The function takes two arguments which are comma-separated.

The first argument is the sed statement:

s/(.*) SKIPTHIS(.*)/$1 $2/

The second argument is the String to apply the sed statement to, which in this case is the method itself, and this is represented in the Mustache variables as:

{{{.}}}

  - rewrite:
       to: "{{#sed}}s/(.*) SKIPTHIS(.*)/$1 $2/,{{{.}}}{{/sed}}"

The sed implementation requires that when the arguments themselves contain commas, that they are wrapped with {{#encodeString}} and {{/encodeString}}
- e.g. {{#encodeString}}{{{.}}}{{/encodeString}}

Reverse Recipe

Since this is an example, and we might want to use this in demos, I wanted to explore how to reverse out the above change using a Sensei recipe.

Thinking it through I want to find a method annotated with @Disabled
but only in the class SkipThisTest where I do the demo:

Name: JUnit: demo in SkipThisTest remove @Disabled and revert to SKIPTHIS
Short Description: remove @Disabled and revert to SKIPTHIS for demo purposes in the project
Level: warning

The Recipe Settings Search is very simple, matching the annotation in a specific class.

~~~~
search:
method:
annotation:
type: "Disabled"
in:
class:
name: "SkipThisTest"
~~~~

To avoid making the code look like it is an error I defined the general setting on the recipe to be a Warning. Warnings are shown with highlights in the code and it doesn't make the code look like it has a major problem.

For the Quick fix, since we have matched the method, I use the rewrite action and populate the template using the variables.

availableFixes:
- name: "Remove Disabled and rename to SKIPTHIS..."
  actions:
  - rewrite:
      to: "{{{ returnTypeElement }}} SKIPTHIS{{{ nameIdentifier }}}{{{ parameterList\
        \ }}}{{{ body }}}"

I basically add every variable except the modifier (since I want to get rid of the annotations) and add the SKIPTHIS text into the template.

This fix has the weakness that by removing the modifiers, I remove any other annotations as well.

Add another Action

I can add another named fix, to give me a choice when the alt+enter is used to display the QuickFix.

availableFixes:
- name: "Remove Disabled and rename to SKIPTHIS..."
  actions:
  - rewrite:
      to: "{{{ returnTypeElement }}} SKIPTHIS{{{ nameIdentifier }}}{{{ parameterList\
        \ }}}{{{ body }}}"
      target: "self"
- name: "Remove Disabled, keep other annotations, and rename to SKIPTHIS..."
  actions:
  - rewrite:
      to: "{{#sed}}s/(@Disabled\n.*@Test)//,{{{ modifierList }}}{{/sed}}\n\
        {{{ returnTypeElement }}} SKIPTHIS{{{ nameIdentifier }}}{{{ parameterList\
        \ }}}{{{ body }}}"
      target: "self"

Here, I added an additional line in the new Quick Fix.

{{#sed}}s/(@Disabled\n.*@Test)//,{{{ modifierList }}}{{/sed}}

This takes the modifier list, encodes it as a string, then uses sed to remove the line with @Disabled from the string, but leaves all other lines in the modifier, i.e. it leaves all other annotations alone.

NOTE: Remember to add the "," in the sed, otherwise you will see a comment added to your preview because this is how Sensei alerts you to syntax errors in the sed command.

Nested `sed` calls

I was lucky that I could match both the @Disabled and @Test in a single search and replace.

In the event that the code is more complicated and I wanted to have a sequence of sed commands then I can do that by nesting them:

{{#sed}}s/@Test//,{{#sed}}s/@Disabled\n//,{{{ modifierList }}}{{/sed}}{{/sed}}

In the above example, I apply the @Test replacement to the results of applying the @Disabled replacement on the {{{ modifierList }}}.

Summary

sed is a very flexible way to achieve code rewriting and it is possible to nest the sed function calls for complicated rewrite conditions.

Recipes like this often end up being temporary because we are using them to improve our programming process, and once we have built up the muscle memory and no longer use the poor programming pattern we can remove or disable them in the Cookbook.

NOTE: Rather than add two available fixes, I could clone the recipe, but if I do, then I have to remember to uncheck the "Add disable entry for cloned recipe" checkbox, otherwise only the cloned recipe will be available for use.

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code”.

All the code for this blog post can be found on GitHub in the junitexamples module of our blog examples repository
https://github.com/SecureCodeWarrior/sensei-blog-examples

Migrating from System.out.println to a Logger in Java

Alan Richardson — Mon, 30 Nov 2020 13:32:53 +0000

When writing the code below I made a bunch of mistakes:

~~~~
private String getCountdownString() {

    String output = "";
    String prefix="";

    for(int countdown = 10; countdown > 0; countdown-- ){

        output = output + prefix + countdown;

        System.out.println(output);

        prefix=", ";
    }

    System.out.println(output);

    return output;
}

~~~~

Initially, I had countdown++ and the loop didn't finish.

And I used countdown > 1 so didn't get the output I wanted.

In the end, I littered my code with System.out.println to help me debug.

And I realised, I really need to learn how to use a logger.

Research

Fortunately, I read through the Sensei documentation and decided to use the "Getting Started" guide to help me create a recipe to convert from System.out.println and encourage me to use a logger:

java.util.logging.Logger

Video - Using Sensei to convert `System.out.println` to a Logger

Creating a Recipe

The first thing I do is click on the println then alt+enter to create a new recipe.

I create it with the following details:

Name: Logger: use logger instead of println
Description: use logger - stop using System.out.println
Level: Error

And I'll start by matching methodcall with the name println

~~~~
search:
methodcall:
name: "println"
~~~~

And the preview shows me all the matches in my code.

I can see that all the matches in my code are for System.out.println but I don't trust that long term this will be the only match. I want to match a more qualified statement I want to change.

I expand the matcher to search for a methodcall on a field named out in the class System.

~~~~
search:
methodcall:
name: "println"
"on":
field:
in:
class:
name: "System"
name: "out"
~~~~

I could, if I wanted, fully qualify the System name to java.lang.System

Amending the Code to log

Next, I want to create the QuickFix.

First I want to amend the line of code that logs the output:

availableFixes:
- name: "use Logger"
  actions:
  - rewrite:
      to: "logger.log(Level.INFO, {{{ arguments.0 }}})"

I don't have to remember the mustache format. I used the Show Variables in the GUI to show me the argument and double-clicked on it. Then the GUI filled in the appropriate matching mustache template.

When I try it out, I can see that I still have to alt+enter to import the Level enum. But if I amend my QuickFix to have a fully qualified item then Sensei will add the import for me e.g.

It will replace System.out.println(output); with

~~~~
logger.log(Level.INFO, output);
~~~~

And add an import for the enum:

~~~~
import java.util.logging.Level;
~~~~

If I rewrite to:

logger.log(java.util.logging.Level.INFO, {{{ arguments.0 }}})

And this will work, but I will still have to remember the syntax to instantiate the logger in the first place.

Amending the code to add the logger field

I can amend my QuickFix to create the logger field for me as well.

I will code the logger first, and then add that to my recipe so I never have to code it again.

~~~~
Logger logger2 = Logger.getLogger(SysOutTest.class.getName());
~~~~

I tend to write the code first because then I can use IntelliJ code completion and syntax checking to make sure I get it right. As a side-effect, it will then be in the code preview when I edit the recipe to add the QuickFix lines that will create that code.

And when I write the code, I want to use a different field name (here I'm using logger2) because Sensei is clever enough not to add a duplicate field, so I have to fool it by using a different name.

So I'll amend the recipe to create this code by adding a field called logger.

availableFixes:
- name: "use Logger"
  actions:
  - rewrite:
      to: "logger.log(java.util.logging.Level.INFO, {{{ arguments.0 }}})"
  - addField:
      field: "java.util.logging.Logger logger = Logger.getLogger({{{ containingClass.name\
        \ }}}.class.getName())"
      target: "parentClass"

Note that I changed SysOutTest to be a mustache variable so that it picks up the name of any class I use this recipe in. And again, I didn't remember the mustache syntax, I used the GUI show variables to find the replacement I needed.

And by fully qualifying the Logger to java.util.logging.Logger, Sensei will add the import and write the line of code that I want i.e.

~~~~
Logger logger = Logger.getLogger(SysOutTest.class.getName());
~~~~

One useful thing about this recipe is that because it will only add the logger field once, I can use this on any existing code where I have used System.out.println and use Sensei to change all the occurrences in my code.

Next Steps

Once I get used to this, I'll eventually train myself away from using System.out.println.

And I can use Sensei to help me proactively write code by creating a second recipe which helps me create a logger.

e.g. I can match on a class, where there is no field called logger, and add one.

I created a recipe of level Information

Name: Logger: add logger
Description: Add logger to class
Level: Information

To match on a class without a logger field:

~~~~
search:
class:
without:
child:
field:
name: "logger"
~~~~

And then I will reuse part of the QuickFix we saw earlier

availableFixes:
  - name: "Add Logger"
    actions:
      - addField:
          field: "java.util.logging.Logger logger = Logger.getLogger({{{ containingClass.name\
        \ }}}.class.getName())"
          target: "self"

Note the difference in target here compared to the first QuickFix. This uses self because our Search matched the class. The first QuickFix uses parentClass because we matched code within the class itself.

Summary

This represents one of the key flows associated with using Sensei to help improve your personal programming skillset:

create a recipe to help with your immediate 'best practice'
once you know how to use that best practice... create a recipe to make your workflow faster

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code”. More Install Instructions

The source code and recipes for this can be found in the sensei-blog-examples repository in the Secure Code Warrior GitHub account, in the pojoexamples module test folder.

github.com/securecodewarrior/sensei-blog-examples

Team Support for Sharing Programming Knowledge with Sensei

Alan Richardson — Fri, 27 Nov 2020 13:53:28 +0000

TLDR; When one person creates a recipe to improve their code quality or productivity, everyone on the team can benefit when the cookbooks are shared. Sensei supports multiple mechanisms for sharing cookbooks e.g. shared folders, Github, and via HTTP downloads.

The Secure Code Warrior Sensei plugin allows anyone to create a recipe to match poor quality coding patterns and fix them to be better quality. One thing we knew we had to solve, was how to share those recipes across a team so that the knowledge encoded in the recipe was transferred.

Sensei provides a number of mechanisms for sharing cookbooks:

Store Cookbooks in the Project Under Version Control
Storing Cookbooks in a shared folder
Store Cookbooks in Github
Zipped files over HTTP(s)

By sharing the cookbooks, Sensei helps teams collaborate on knowledge sharing. The collaboration helps improve communication and embed the agreed code quality approaches.

For example, sharing a cookbook allows:

team members to share useful recipes with each other.
team leads to codify agreed coding practices for junior staff.
- To identify common violations with a quick fix for the agreed version.
increased Inter-team co-operation.
- An AppSec team might create recipes to highlight a problem in the code, and the development team could write the quick fix.

The next few sections explain how to implement each of the sharing mechanisms.

Store Cookbooks in the Project Under Version Control

The project .sensei folder is the default option when creating a cookbook file.

`project://.sensei’

All cookbooks and recipes would be stored in a .sensei folder in your project.

The easiest approach to sharing is to add the project .sensei folder to version control.

Then the .sensei folder can be managed like any other shared code artifact associated with the project. The cookbooks are stored as YAML configuration, making them easy to merge during any commit and review process.

This is the approach taken for the public sensei-blog-examples project.

https://github.com/SecureCodeWarrior/sensei-blog-examples

The .sensei folder contains the cookbook with all the recipes, and they are available to anyone that clones the repository.

Store Cookbooks in Any Folder

Teams can also use cookbooks stored in central locations.

Saving the cookbook to any folder with shared write access permissions will allow the whole team to update the recipes, and import them into any project that they happen to be working on.

The location would be set to the directory path.

Store Recipes in Github

Sensei can also access recipes that are stored in a Github repo. Both private and public repositories are supported.

Github over SSH

SSH Repository access is configured using the following syntax for the Location

git@github.com:SecureCodeWarrior/acookbook.git

For this to work, the repository would contain the contents of a cookbook folder.

e.g. our Basic Protection Cookbook takes this approach and could be added as a "Locally configured Cookbook" using the location git@github.com:SecureCodeWarrior/cookbook-basic-protection-set.git

It is also possible to configure the branch and the subfolder for the cookbook e.g. in the master branch in the cookbook subfolder

e.g.

git@github.com:SecureCodeWarrior/sensei-blog-examples.git|master|.sensei

An SSH key needs to be configured for private repositories.

https://github.com/settings/keys

And the key should not have a passphrase.

Github over HTTPS

It is also possible to access public repositories over HTTPS, and the same repo.git|branch|folder syntax is used e.g.

https://github.com/SecureCodeWarrior/sensei-blog-examples.git|master|.sensei

or for the Basic Protection Cookbook

https://github.com/SecureCodeWarrior/cookbook-basic-protection-set.git

Zipped over HTTP(s)

Sensei can also access cookbooks which are zipped, over HTTP or HTTPS.

e.g.

http://localhost:8000/rules.sensei.zip

The zip cookbook file should contain the contents of a cookbook folder e.g the rules.sensei file.

Sharing Summary

Sensei supports using multiple cookbooks so that an individual programmer can have recipes that support their own learning and productivity.

More importantly, we know that teams work most effectively when knowledge is shared. Having shared team repositories, e.g. for a specific project, or a specific library, or for a shared set of migration patterns, can help boost team productivity and codify the team’s experience.

When a cookbook is shared, multiple teams can use the same cookbook which can also improve inter-team collaboration from different disciplines e.g. AppSec to development.

With four core sharing mechanisms available, Sensei hopefully has at least one approach you can use to increase collaboration on knowledge sharing.

You can install Sensei from within IntelliJ using "Preferences \ Plugins" (Mac) or "Settings \ Plugins" (Windows) then just search for “sensei secure code” (or find it on the JetBrains marketplace here)

All the Sensei blog posts code and recipes are on Github in:

github.com/SecureCodeWarrior/sensei-blog-examples

Using Documentation Links with Sensei

Alan Richardson — Wed, 11 Nov 2020 13:57:30 +0000

One of the difficulties with learning a new library, or sharing agreed practices across our team is documenting and creating examples.

Very often we create small example projects, but we don't have them open when working with actual code.

I've often thought it would be great to have the ability to link to our examples, or online examples and be able to goto a URL for more explanation when needed.

With Java, we have JavaDoc comments, which can have a see annotation:

/**
 * @see <a href="https://junit.org/junit5/docs/current/user-guide/#writing-tests-annotations">Junit 5 Annotation docs</a>
 */

JavaDoc like this in 3rd party libraries is a great help because we can use the Quick Documentation functionality in IntelliJ to have access to more detailed examples.

But, we all know that comments don't get updated as often as code, and web presence maintenance is often disconnected from library maintenance and sometimes performed by a different team entirely.

How Sensei Helps

Sensei provides the ability to match on library annotations and methods to provide links to long-form documentation on a wiki or third party tutorial site.

As an example, I'm using the @Test annotation from JUnit.

The JavaDoc is very detailed, and the Quick Documentation view explains how to use the annotation.

But the official documentation on the web site is often easier to read and has more examples.

When a team starts learning a library, having a set of recommended tutorials, can be very useful.

Sensei has a goto action that can open a URL, allowing us to link to external sites and examples for documentation that we, as a team, find useful.

Implementing the Goto URL

To implement this I would create a search that matches the @Test annotation from Junit.

search:
  annotation:
    owner:
      method: {}
    type: "org.junit.jupiter.api.Test"

And then I would add goto actions for each of the URLs I find useful.

e.g.

The example below would create a single Action JUnit Annotations (learn) which would open both URLs in a browser at the same time.

availableFixes:
- name: "Learn about JUnit Annotations"
  actions:
  - goto:
      type: "URL"
      value: "https://junit.org/junit5/docs/current/user-guide/#writing-tests-annotations"
  - goto:
      type: "URL"
      value: "https://junit.org/junit5/docs/current/user-guide/#writing-tests-classes-and-methods"

And when I activate it in IntelliJ with Alt+Enter I see the context menu which I can select to jump to the documentation.

Multiple Actions

I might choose to have multiple Actions so that each URL or tutorial has its own option in the alt+enter Quick Fix pop up menu.

For example, for the @Parameterized annotation, I might want to link to the official documentation and a set of online example code.

I would simply create a recipe that searches for the annotation:

search:
  annotation:
    owner:
      method: {}
    type: "org.junit.jupiter.params.ParameterizedTest"

And links off to the sites I identified as being useful:

availableFixes:
- name: "JUnit Annotations (learn)"
  actions:
  - goto:
      type: "URL"
      value: "https://junit.org/junit5/docs/current/user-guide/#writing-tests-annotations"
- name: "What is a JUnit Test? (learn)"
  actions:
  - goto:
      type: "URL"
      value: "https://junit.org/junit5/docs/current/user-guide/#writing-tests-classes-and-methods"

Both links would then appear in the pop-up dialog.

Who would benefit?

I would have found this useful when using and learning libraries, especially when leading teams and helping them adopt a new library.

This could also benefit teams creating libraries, by creating a standard set of documentation recipes to help guide people through the adoption of the library or new features in the library.

This would be especially useful if the code maintenance and documentation maintenance are performed by different teams.

You can install Sensei from within IntelliJ using preferences > plugins (just search for “sensei secure code”).

All the code for this blog post is on Github in the junitexamples module in github.com/SecureCodeWarrior/sensei-blog-examples

What is Sensei?

Alan Richardson — Wed, 11 Nov 2020 13:46:47 +0000

What is Sensei?

The Sensei plugin provides an easy way to find specific code patterns in your source code, and then apply rewrite rules to amend the matching code. All within the Intellij IDE, and in real-time.

For example, you could create a rule that matches on JUnit @Disabled annotations which do not have a reason, Sensei would then tell you about the issue by highlighting the code in the IDE.

Additionally, when you alt+enter, you can have the option to Add a todo comment parameter.

When selected, this would amend the code to add a boilerplate reason, which you can then amend, and if you don’t, it will show up in your TODO panel.

e.g. @Disabled would become @Disabled("TODO: add a description here")

Sensei combines the functionality of a Static Analysis code scanner with a code rewriting engine.

Demo Video

IntelliJ Intention Actions

Based on the above description, the obvious alternative (if you were not using Sensei) is to use the IntelliJ Intention Actions functionality.

Sensei differs from IntelliJ Intention Actions because the aim with Sensei is to provide a way to create matchers and rewriters which are project specific, or even local to an individual developer.

We have tried to put together a GUI that makes both the matching and rewrite rules easy to write and experiment with.

How can it help me personally improve?

When I'm learning a new library, it takes me time to build up muscle memory around the methods and formatting. So I might choose to create personal recipes which

link off to the official documentation or tutorial pages
have boilerplate templates which are most effective
fix poor coding practices
add boilerplate code to help use a library

I can use Sensei to build temporary recipes which prompt me for the current practices I've chosen to use and help me build up effective habits. And the recipes are temporary because I can remove them when I outgrow them.

How can it help my team improve?

In the same way we can help teams build up muscle memory around agreed coding standards.

Creating cookbooks of recipes that we apply when we find the same comments in pull request reviews. Since the cookbooks are stored in version control with the project, they are available to everyone on the project. And we can switch them off when we no longer need prompting.

Sensei helps provide feedback early

What we've tried to build with Sensei is a way of pulling the feedback that helps us improve, and reminders of corrective action, as early into the coding process as we can.

Rather than wait for

the results of a static analyser
the comments from a code review

We can instead see the feedback, for custom standards we want to enforce, as we code.

And we have either reminders, or actual rewrite rules, to help us write code that complies with the standards.

Sensei is flexible

In that way, Sensei is a bit of a mix, since it’s:

part Static Analyser
part coding tutor
part rewrite engine

Sensei is flexible enough to make the job of saying "What is Sensei?" that little bit harder.

Sensei fills a gap in the programming workflow

We've tried to make Sensei the missing piece of the programmer workflow that helps you improve specific elements in your coding style, or library use, that you and your team are currently working with.

This flexibility means that it takes a little more time to get to grips with Sensei than a static analysis tool or the built-in IntelliJ Intensions. Still, by spending the time to experiment, you will gain a new way to speed up your learning in your personal development process.

How to experiment?

Once you have downloaded and installed Sensei from the Intellij Marketplace, we recommend installing via the plugins manager in IntelliJ preferences (just search for 'Sensei secure code' and you'll find it).

We have also created a getting started guide on the Intellij Marketplace - get started

The easiest way to make Sensei work for you is to look at your coding process and consider:

What documentation do you keep looking up?
- You could add some Sensei recipes that link back to that documentation.
What simple mistakes do you keep making?
- You could temporarily codify that poor coding pattern as a matcher, and write a Quick Fix rewrite that amends the code to be what you really want to write.
What boilerplate code do you write to use a library?
- You could create a Quick Fix rule to write the code for you.

Since Sensei is designed to work alongside whatever static analysis tool you're using, if you find that the same violations are being reported from static analysis, then you could replicate the condition in a Sensei recipe. You can then add a Quick Fix to help train you, not just to identify the mistake but also to move quickly to writing the correct code.

To learn more about Sensei - check out our Examples Project on Github

It contains code examples, recipes and explanations.

github.com/SecureCodeWarrior/sensei-blog-examples

DEV Community: Secure Code Warrior

Try This Online Java Gotchas Quiz

Running IntelliJ Inspections From Continuous Integration

IntelliJ Inspections

Inspections Profile

Running an Inspection Profile in the IDE

Running an Inspection Profile from the Command Line

Viewing Inspection Results

Inspection Profiles in Continuous Integration

Summary

Java Gotchas - Bitwise vs Boolean Operators

Java Gotchas - Bitwise vs Boolean Operators

What's the Problem? Bitwise operations work fine with Booleans

AND Truth Table

OR Truth Table

Issue: Short Circuit Operation

Boolean Operators Short Circuit Evaluation

But this would never happen in production code

How to Find This

End Notes

What is Static Analysis

What is Static Analysis?

How does a Static Analysis tool work?

Static Analysis in Continuous Integration

Static Analysis in the IDE

Fixing Code Based on Static Analysis Rules

Default Rules

Annoyances

IDE Based Static Analysis Tools

IntelliJ Inspections

SpotBugs

SonarLint

CheckStyle

Sensei

Summary

How to catch and fix a Guice dependency injection issue using Sensei

How to catch and fix a dependency injection issue using Sensei

RunTime Exceptions from Incorrect Dependency Injection Wiring

Warning Signs

How could Sensei help?

How to create a recipe?

Fix

Summary

Automatically Amending Method and Class Visibility for JUnit 5

Using Sensei

Amending the Class Declaration

Amending the Method Declarations

Hint: Amending Multiple Methods

Amending the class

Summary

Automatically Adding a Private Constructor with Sensei

Searching for the Class

Quick Fix

Search for Missing Constructors

Narrow Search for Likely Culprits

Hints

Code Links

Improving A Personal Programming Process Using Sensei

Adding Annotations and Method Renaming

Disabling a JUnit Test

Recipe Settings

QuickFix Settings

Reverse Recipe

Add another Action

Nested sed calls

Summary

Migrating from System.out.println to a Logger in Java

Research

Video - Using Sensei to convert System.out.println to a Logger

Creating a Recipe

Amending the Code to log

Amending the code to add the logger field

Next Steps

Summary

Team Support for Sharing Programming Knowledge with Sensei

Store Cookbooks in the Project Under Version Control

Store Cookbooks in Any Folder

Store Recipes in Github

Github over SSH

Github over HTTPS

Nested `sed` calls

Video - Using Sensei to convert `System.out.println` to a Logger