Three things your linter shouldn’t tell you

My coworker Grayson originally wrote this blog post. It was first posted on the r2c engineering blog. At r2c we are linter lovers at heart and value tools that help us write better code, faster. However, sometimes we’re left wanting for a bit more from linters…

Oh, ESLint rules aren’t enabled by default? Which is better, Airbnb’s or Google’s rule list? Bandit is giving me thousands of errors? Oh, those are just asserts. Why does Bandit consider assert a security violation? 🤔 Shouldn’t this be easier?…

Wouldn’t it be great to have a code check tool where you could just run check and it showed you the results you cared about?

We’re making a new developer tool called Bento. Developer time is valuable, and we frequently hear from developers that their linters demand a lot from them. It’s our belief that your linter shouldn’t bother you with things you don’t care about, so in Bento we turn off checks based on these three things:

Linters Shouldn’t Tell You about Style Violations

Frankly, we don’t care about one or two lines between function definitions or whether there should be spaces around binary operators. With deterministic formatters like Black for Python and Prettier for JavaScript, there is no need to spend valuable development time manually addressing style violations when a formatter can do this automatically.

Therefore, Bento’s default configuration explicitly disables all style and preference checks. Bento only reports violations that might** actually be a problem**.

For ESLint (with the Airbnb config), this amounts to disabling all checks labeled “style” by both ESLint and Airbnb, checks with “prefer” in the name, and a handful of checks that our manual examination determined did not indicate the presence of an actual bug.

For Flake 8, this amounts to most pycodestyle checks.

Linters Shouldn’t Tell You about Unpopular Checks

We hypothesized we could get a good estimate for checks that developers like and dislike by mining the configuration files for code checking tools in open source code. Using our at-scale code analysis platform, app.r2c.dev, we mined the ESLint configurations of the top 1,000 npm packages (as of April 2019), and 128 Bandit configurations on GitHub.

Disabled ESLint checks:

Perhaps unsurprisingly, about 38% of repositories globally disabled no-console. Apparently, people like console.log, which makes sense! We disable this check in Bento.

Disabled Bandit checks:

Of the Bandit configs, nearly 80% disable assert_used - B101! We disable this check in Bento, too. About 45% also disable random - B311, so we disable it as well.

As you might imagine, this is the first step in tailoring checks to your code. In the future, Bento will show you the checks you want to see by identifying which checks get fixed… and which get ignored. This will be incredibly powerful—and necessary—as we add support for more code checks.

Linters Shouldn’t Tell You about “Bad Checks”

Granted, the definition of “bad” is subjective! 🤗 Generally speaking, our definition of a “bad check” is a check with a high false-positive rate and a low chance of being a bug.

As an example, B105 checks for hardcoded_password_strings. In theory, this check is great!

• it prevents checking in hard-coded secrets, and
• it prevents easily guessable defaults — for example, admin:password.

However, in practice, this check is very noisy. The check itself looks for assignments or comparison operators that “look like a password.” This check commonly has a false positive on the variable name token, frequently used in string parsing operations.

We use our static analysis platform to test checks on real-world code. We can test on thousands of open source repositories at once to see where a check yields a true or false positive. Usually, it is easy to spot false positives—such as token in string parsers.

#https://github.com/cobbler/cobbler/blob/91e3ff1d2a05eeb24107ef72e09c1a3d800ae304/cobbler/web/templatetags/site.py#L245
while not self.at_end():
    token = self.get_token()
    if token == 'not':    # B105 fires here
        if self.at_end():
            raise self.error_class('No variable provided after "not".')
        token = self.get_token()
        negate = True

To be more scientific, we can measure the accuracy of the check. After rigorously reviewing 20 repositories, including Django and Ansible, B105 hits nothing of importance 93% of the time. While the theory is good, this check is rather noisy. Therefore, Bento disables this check. (FYI, neither Django nor Ansible use hardcoded passwords!…except in their test files. 😉)

Checks we ship in Bento are reviewed this way. If a check is “bad,” it’s disabled. If “badness” cannot be determined, we leave it enabled. We figure our users will tell us if it’s bad! 😅

Run Bento on your code

We care a lot about code quality at r2c, and we love our linters! We’ve made it even easier to write better code faster by filtering out the checks that we don’t want linters to tell us about. You can run Bento on your code to take advantage of these checks by following this link: https://bento.dev/.

If you have feedback or comments for us, email us or make an issue on GitHub.