DEV Community: Pablo Estrada

Introducing Semgrep and r2c

Pablo Estrada — Thu, 29 Oct 2020 17:19:32 +0000

This post is by Isaac Evans, CEO and co-founder of r2c.

Free, fast, open-source, offline, customizable. These are not often words that describe code scanning tools, and that's a shame.

We founded r2c to bring world-class security tools to developers based on our conviction that software will run the most exciting parts of the future: everything from medical equipment to robots to autonomous cars. The security process should not be the foe but rather the enabler of rapid software development. If developers lack tooling that is easy to set up and understand—or if a developer has to convince their manager to spend a few million dollars on advanced security tools each time they change jobs, the future is bleak.

Before founding r2c, we worked on security and developer tools for large companies and governments. It was eye-opening to see that despite massive budgets, their security programs were generally a generation or more behind the tech giants. When it came to security tools for developers, most teams were jaded about scanning code for vulnerabilities; they hated the tools they had to use and usually ignored them beyond doing the minimum necessary to satisfy a compliance checkbox.

What about code scanning at places like Facebook, Apple, Amazon, Netflix, and Google? They don't generally use traditional commercial security tools which ask "how can we find every bug?" Instead, they focus on custom tooling that can build guardrails for developers. This doesn't require million-dollar tools, PhDs in program analysis, or days of compute time. It looks much more like unit tests for security.

We believe there is a gap between traditional compliance tools and simple linters that's ripe for a new approach, and we were fortunate to find partners from Redpoint Ventures and Sequoia Capital who agreed. With them, we raised a \$13M Series A round of funding to build a security tool that developers might actually love. We've been working on it quietly for a while now, and we're finally ready to announce it to the world!

Semgrep

Semgrep, our open-source product, is specifically designed for eradicating bug classes.
Developers and security engineers can say "this is the safe pattern we always use for (e.g. parsing XML)", write a rule in a few minutes, and enforce that on every editor save, commit, and pull request.

Semgrep is ideal for building security guardrails: start by using frameworks designed with security in mind, then automatically flag code that strays from the secure-by-default path. This is an approach used by Google, Facebook, Amazon, Dropbox, Stripe, Netflix, and others—a topic Clint Gibler and I presented on at Global AppSec 2020. This approach increases developer productivity, reduces attack surface, minimizes the areas for human inspection and audit, and allows the security team to scalably protect code written by thousands of developers.

The idea behind Semgrep is simple: it feels like a regular search (grep) but is syntax-aware. You can learn Semgrep in a few minutes! And Semgrep can be used for more than just security issues: performance, internationalization, or just annoyances committed by accident.

$ semgrep -e foo(1) matches all equivalent variations. See a live example of matching exec calls

What's Next?

Semgrep started as an open-source project at Facebook and we're lucky to have its original author, Yoann Padioleau, on our team at r2c. Since we released the first post-Facebook version (0.4) earlier this year, we've released 25 new versions, added support for 8 new languages, reworked the parsers so we could collaborate with Github on tree-sitter, been joined by thousands of enthusiastic GitHub followers, and seen over 100K pulls of the Semgrep Docker image.

Our roadmap contains more program analysis features to support the sorts of secure-by-default enforcement that large technology companies are already leveraging so heavily (constant propagation, taint tracking, and more), as well as support for many more languages.

Batteries Included

Along with this release of Semgrep, we're announcing the availability of Semgrep Community, a free, hosted service for managing Semgrep CI as well as Semgrep Teams, a paid service which adds additional features for managing Semgrep that are useful for enterprises. Both these offerrings provide SaaS infrastructure for operating a modern AppSec program. They enable central definition of code standards for your projects and show results where you already work: GitHub, GitLab, Slack, Jira, VS Code, and more.

We're also excited that Semgrep Registry already has 900+ rules written by r2c and the community—you can start running on your project right now! Or if you like to DIY, try writing your own.

Preventing SQL injection: a Django author's perspective

Pablo Estrada — Tue, 12 May 2020 21:42:44 +0000

This is a guest post co-authored by Jacob Kaplan-Moss, co-creator of Django, and Grayson Hardaway.

What’s SQL Injection?

SQL Injection (SQLi) is one of the most dangerous classes of web vulnerabilities. Thankfully, it’s becoming increasingly rare — thanks mostly to increasing use of database abstraction layers like Django’s ORM — but where it occurs it can be devastating.

SQLi happens when code incorrectly constructs SQL queries that contain user input. For example, imagine writing a search function without knowing about SQLi:


def search(request):
    query = request.GET['q']
    sql = f"SELECT * FROM some_table WHERE title LIKE '%{query}%';"

    cursor = db.cursor()
    cursor.execute(sql)
    ...

Can you spot the problem? Notice that the query comes from the browser: request.GET['q']. Think about what might happen if that query contains a single quote. What happens when the SQL string is constructed?

Consider if an attacker searches for ' OR 'a'='a. In this case the constructed SQL would become:

SELECT * FROM some_table WHERE title LIKE '%%' OR 'a'='a';

So that’s bad; now we’re returning the entire contents of the table. This could be a data breach, or it could overwhelm your database server.

But it gets worse; imagine now that the attacker searches for '; DELETE FROM some_table. Now, the constructed SQL becomes:

SELECT * FROM some_table WHERE title LIKE '%%'; DELETE FROM some_table;

Uh oh.

General concepts for preventing SQLi

We’ll get to Django specifics shortly, but first it’s important to really understand the fundamental rules of preventing SQL injection:

Never trust any data submitted by the user
Always use "parameterized statements" when directly constructing SQL queries

Anything that comes from the user could be maliciously constructed. Even things that seem safe, like browser headers (e.g., things like the user agent, request.META['HTTP_USER_AGENT'] in Django) are trivial to tamper with either directly in the browser or with tools like Burp or Charles.

Practically, in Django this means nearly anything that hangs off the HttpRequest object, i.e., the request parameter that’s passed as the first argument to view functions. Though there are some exceptions, it’s probably best to consider anything on request as fundamentally untrustworthy.

However, just because some piece of data isn’t attached to request right now doesn't mean that you can trust it. For example, consider something like an image caption. You might access it through an API that doesn’t mention a request:

image = Image.objects.get(...)
sql = f"""SELECT * FROM images WHERE similarity(caption, '{image.caption}') > 0.5;
...

But if that image caption was previously entered by a user…it’s still dangerous. So this brings us around to the second rule: always use parameterized statements.

Parameterized statements are a mechanism to pass any dynamic parameters separate from the SQL query. They’re either interpreted directly by the database or safely escaped before being added to the query. Almost every database client on the planet supports parameterized statements — and if yours doesn’t, find a different one.

Here’s what the search function from above would look like with parameterized statements:

def search(request):
    cursor = db.cursor()
    cursor.execute(
        "SELECT * FROM some_table WHERE title LIKE '%?%'",
        [request.GET['q']]
    )

Notice the ? in the SQL string, and the second parameter to execute. This second argument is the parameter list; items in this list are safely injected into the query to replace the question marks.

PEP-249, the Python database API standard requires parameterized statements, though different libraries may use different syntax for the placeholders (%-style parameters, :named parameters, numeric parameters, etc.).

You can use code analysis tools to check for SQL injections. Bento is one such tool that has several checks for common SQL injection problems. This can catch many common errors; but it’s still a best practice to use parameterized statements and one of the techniques below to completely prevent this attack.

Preventing SQLi in Django

Django’s ORM uses parameterized statements everywhere, so it is highly resistant to SQLi. Thus, if you’re using the ORM to make database queries you can be fairly confident that your app is safe.

However, there are still a few cases where you need to be aware of injection attacks; a very small minority of APIs are not 100% safe. These are where you should focus your auditing, and where your automated code analysis should focus its checks.

Raw Queries

Occasionally, the ORM isn’t expressive enough, and you need raw SQL. Before you do, consider whether there are ways to avoid it -- for example, building a Django model on top of a database view, or calling a stored procedure can help prevent the need to embed raw SQL in your Python.

But, sometimes raw SQL is unavoidable. There are several APIs for doing this, but all are somewhat dangerous. In order of desirability, these are the APIs that Django provides:

Raw queries, for example:

    sql = "... some complex SQL query here ..."
    qs = MyModel.objects.raw(sql, [param1, param2])
    # ^ note the parameterized statements in the line above

The RawSQL annotation, for example:

    from django.db.models.expressions import RawSQL

    sql = "... some complex subquery here ..."
    qs = MyModel.objects.annotate(val=RawSQL(sql, [param1]))
    # ^ note the parameterized statement in the line above

Use database cursors directly, for example:

    from django.db import connection
    sql = "... some complex query here ..."
    with connection.cursor() as cursor:
     cursor.execute(sql, [param1])
     # ^ again, note the parameterized statement in the line above

AVOID: Queryset.extra() (no example: this is unsafe, so it's just included for completeness).

To use these APIs safely:

Read the first part of this article and make sure you understand parameterized statements before proceeding.
Don’t use extra(). It’s difficult (if not impossible) to use in a way that’s 100% safe, and should be considered deprecated.
Always pass parameterized statements — even if your parameter list is empty. That is, you should write something like:

    sql = 'SELECT * FROM something;'
    qs = MyModel.objects.raw(sql, [])

This is to remind you to later add parameters to this list, and to make it easier for automated tools like Bento to find potentially incorrect API usage.

The query itself should always be a static string, rather than one formed from concatenation or any other string processing. Again, this is to make it easier for automated tools to find incorrect API usage.

Automatic Prevention

It is good practice to use code analysis tools to catch preventable mistakes — to err is human, as the saying goes. Bento will automatically check Django code for SQL injection patterns. The following will check your codebase all at once for SQL injections caused by something hanging off of the request object.

pip3 install bento-cli && \
  bento init && \
  BENTO_REGISTRY=r/r2c.python.django.security.injection.sql bento check -t semgrep --all .

Better than checking your current code, however, is checking your future code! Bento is designed to be run as a pre-commit hook or in continuous integration (CI) environments. Bento is diff-aware and will only check commits, ensuring a speedy workflow while keeping your code secure. When you init Bento on your project, it will automatically set itself up to check commits.

This commit-based workflow is especially powerful for ensuring certain patterns never enter your codebase. To practically eliminate SQL injection from your codebase, you should automatically detect that your code:

Always uses parameterized queries.
Never uses .extra().

Bento can detect these patterns by using a different registry:

BENTO_REGISTRY=r/r2c.python.django.security.audit bento check -t semgrep --all .

This set of rules will highlight many more findings even when there is not a vulnerability. It is much stricter and can be overwhelming if you check your code all at once. However, you can also archive your findings with Bento, which will suppress findings until you’re ready to deal with them. This lets you continuously check your code for these patterns without being overwhelmed by findings.

Under the hood, Bento is powered by Semgrep. Semgrep is a tool for easily detecting and preventing bugs and anti-patterns in your codebase. It combines the convenience of grep with the correctness of syntactical and semantic search. This has advantages over normal grep — the most obvious one being that Semgrep is not thwarted by line boundaries.

Let’s say you wanted to detect the following SQL injection:

def search(request):
    search_term = request.GET['search_term']
    cursor = db.cursor()
    cursor.execute("SELECT \* FROM table WHERE field=" + search_term)

This can be expressed in Semgrep like so:

$VAR = request.GET[...]
...
$CUR.execute("..." + \$VAR)

Detecting a pattern like this in a commit-based workflow is invaluable because it effectively eliminates this pattern of SQL injection from your codebase! You can check this out in action at https://sgrep.live/0X5.

Other ORMs

Finally, if you are continually finding that Django’s ORM isn't expressive enough, you may want to experiment with replacing Django’s ORM with SQLAlchemy, which is a more powerful and expressive ORM. You’ll lose out on many of Django’s conveniences like the admin, model forms, and model-based generic views, but will gain a more powerful and expressive API that’s still safe.

Custom ORM additions

Finally, there are a few potentially-dangerous areas that may be unsafe even though you’re not using raw SQL directly. Django allows for the creation of custom aggregates and custom expressions -- e.g. a third-party library could write APIs such that something like Document.objects.filter(title__similar_to=other_title) would work.

Django’s core ORM -- the core expressions, annotations, and aggregations -- are all mature and battle-hardened. The odds of a SQLi in the core parts of the ORM is very, very low. But ORM additions -- especially ones that you write yourself -- can still be a source of risk.

To mitigate the risk of injection from these advanced features, I suggest the following:

First, be cautious about including custom expressions/aggregates from third-party apps. You should audit those third-party apps carefully. Is the app mature, stable, and maintained? Are you confidant that any security issues would be promptly fixed and responsibly disclosed? And, of course, be sure to pin your dependencies to prevent newer and potentially less secure versions from being installed without your explicit direction.

Similarly, be cautious about writing your own custom aggregates. Carefully read the beginning of this article, and Django's documentation about avoiding SQL injection in custom expressions. As the documentation shows, if possible you should avoid doing any string interpolation in custom expressions. If you can’t, you'll need to escape any expression parameters yourself. This is tricky to do right, and will depend on the specifics of your database engine and Python wrapper API. Consult an expert before diving in here!

The django.security.audit registry in Bento will detect if a custom ORM addition is defined in your codebase; you could also quickly audit third-party apps with this. The exploitation conditions are very nuanced, so if you find this in your project, be sure to consult that expert!

Wrapping up

Django was designed to be resilient against SQL injection (and other common web vulnerabilities). Most common uses of Django will be automatically protected, so SQLi vulnerabilities in real-world Django apps are thankfully rare.

However, when they occur, SQLi vulnerabilities are devastating. It’s well worth your time to audit your codebase to ensure you’re safe. Bento can help by flagging several common vulnerabilities. Now that you understand the concepts, and why certain errors are flagged, you should be better equipped to write safe code.

Our quest to make world-class security and bugfinding available to all developers, for free

Pablo Estrada — Wed, 05 Feb 2020 17:40:37 +0000

by Isaac Evans, CEO and co-founder @ r2c

This post was originally published on the Bento blog in late December 2019.

Why we’re building

One thing we’ve learned at r2c is that most Python or JavaScript developers have never heard of—let alone tried—the tools some devs use to find deep flaws in code: like Codenomicon, which found Heartbleed, or Zoncolan at Facebook, which finds more top-severity security issues than any human effort. Not only do these tools find severe issues, they save time by pointing out hundreds of thousands of issues before humans do.

We believe every developer deserves access to powerful tools, but most don’t know about or can’t afford them. r2c’s mission is to make those tools available to those who want to find bugs, discover security problems, and save time but don’t work for a giant company that prioritizes these problems with nearly unlimited resources.

That’s why we’re excited to release Bento! It’s a free and opinionated toolkit for easily adopting linters and program analysis in a codebase. It includes analysis we’ve written and packages fantastic community-created tools, all running offline (no code is ever shipped off your machine). Over the next few months we’ll release more novel checks and include existing tools; subscribe for updates.

Some members of our team wrote early versions of these tools at places like Facebook. r2c started by building infrastructure to make it easy to run static analysis tools at massive scale (see our paper co-published at USENIX) but our goal has always been to take the learnings from scaling analysis to benefit individual developers directly: folks helping small teams writing voter registration systems for their city, non-profits who serve communities targeted by powerful hostile actors, startups who handle sensitive data about fellow humans, or developers who just want to automate away code review.

How can I get Bento now?

Bento is in alpha, but you can try it right away:

pip3 install bento-cli

Here’s a short demo:
youtube: https://www.youtube.com/embed/rGwd1aEF8Yk

A lot of love from our small team has gone into Bento. Please try it on your Python or JavaScript projects and send us feedback!

But is this just a glorified linter?

Well yes, but actually, no; Bento is currently a union of curated AST-based lints, including new ones written by us, tuned to find bugs that matter. Our roadmap takes us far beyond AST-based linting though: finding sql injection through taint analysis, detecting dangerous dependency upgrades, etc.

Linters have done a good job reaching developers and improving code consistency, especially style. But we want to surface issues and checks that are deep and avoid arguing about spaces vs tabs in code review. Bento ships with configurations that are tuned on real-world data and focuses the finding on correctness and security. They are based on using our platform to analyze swathes of open-source repositories and see what checks developers turn on and off (Three Things Your Linter Shouldn’t Tell You. Our opinion is that you should forget about style and use a deterministic, zero-config formatter (Black for Python or Prettier for JavaScript).

As opposed to other tools that try to measure code-quality or concatenate linter output, we have skin in the analysis game; we’re already making some contributions back to the tools we include. We’re collaborating with a few linter authors already and we would love to offer free compute resources on our platform for measuring check quality to anyone else who might be interested (hello@r2c.dev).

Here’s what’s coming next

Our immediate focus is writing custom analysis tools to find security and other issues for users of the Flask web framework. If you or someone you know uses Flask and has ideas on what we might detect, send us a note or make an issue!

Bento core values

Our first releases are about making it easy to install, adopt, and get started before we ship everything on our roadmap.

Find bugs that matter
Bento automatically enables and configures relevant analysis based on your dependencies and frameworks, and it will never report style-related issues. You won’t painstakingly configure your tooling, we did that already!

Go fast
No one should have to dig through thousands of linter results and fix them before they can start using a tool. Bento ships with a built-in archiving feature that lets you establish a baseline without fixing all the issues at once and just look at any new problems entering the codebase.

This philosophy also applies to setup: Bento auto-configures in about 30 seconds, it’s easy to install in a Docker container, and it can even install itself as a pre-commit hook automatically.

Get better over time
Bento automatically tailors itself to your project by enabling checks that correspond to your language, framework, and dependencies. As time goes on and based on community feedback, we’ll be writing and shipping new checks that you can adopt automatically. And we want your feedback!

A workflow for writing with linters

Pablo Estrada — Mon, 16 Dec 2019 16:21:52 +0000

I recently wrote about using linters to check my writing, primarily using a tool called textlint. This post describes how I tweaked my workflow to run text snippets through textlint using Visual Studio Code.

Although textlint is accessible through the command line, I’ve found its Visual Studio Code integration to be pretty smooth, so how I consume textlint’s output. To process my text through Visual Studio Code, I configured an extension and configured an Alfred snippet to help me to do this easily.

Advanced New File extension

The extension Advanced New File is described as, “an easier way of creating a new file inside a project.” You can specify the name of the file on creation and if you define a path that doesn't exist yet the folders will be created.

To trigger this extension I added this keyboard shortcut to keybindings.json:

  {
    "key": "cmd+n",
    "command": "newFile.createNewFile"
  }

I often use small text snippet files and it’s handy to sync them via Dropbox. Here’s how I configured the extension to create new files in Dropbox by default:

"newFile.defaultBaseFileName": "newFile",
"newFile.relativeTo": "root", // "root" or "project"
"newFile.defaultFileExtension": ".txt",
"newFile.rootDirectory": "~/Dropbox/snippets",
"newFile.showPathRelativeTo": "root", // "project" or "none"
"newFile.expandBraces": false,

Automating filenames using Alfred

Instead of having to manually type in a new filename, I use Alfred’s Snippets. Snippets auto-expand short keystroke combinations into longer text. The classic example is address expansion: instead of manually typing your postal address, enter a keyword like !address and the text is automatically replaced by your complete postal address.

Snippets can also use dynamic placeholders, like {date} or {clipboard}. When creating a new file, I use the {date} placeholder by typing in [date]and Alfred automatically replaces this with the current date in the format I define in Alfred’s preferences. From inside VS Code, the workflow looks like this:

Using these extensions and shortcuts, it’s super fast to create a new text file and name it accordingly. When I write in these files, textlint and other extensions for writing check my work on the fly.

How I use linters to check my writing

Pablo Estrada — Thu, 05 Dec 2019 16:31:55 +0000

I installed textlint, a pluggable linter for text. This post describes how I configured it, along with initial results and observations.

At work, my team is developing Bento, a free program analysis toolkit for finding bugs that matter in your code. Our program analysis team writes code checks that find problems in projects using specific frameworks such as Flask and Requests. We’ve also added curated checks from some tools such as Flake 8 and ESLint, and characterized the performance of checks that Bento enables or disables.

As we fine-tuned Bento’s configuration, it got less noisy, had a higher signal-to-noise ratio, and became more useful. Since I spend a lot of time writing, I wanted to have similar software analyze and improve my writing, like Bento analyzes and improves source code.

Inspiration

I found helpful articles like Chris Ward’s Customizing Visual Studio Code for Writing. That and others kept pointing to textlint, so I decided to try it.

textlint is described as, “the pluggable natural language linter for text and markdown.” It’s written in JavaScript and inspired by ESLint. Like ESLint, textlint’s pluggable design brings flexibility, extensibility, some initial configuration requirements, and collaboration through contribution.

Installation

textlint suggests per-project installation, but I installed it globally since I wasn’t sure how it would perform, and I didn’t want to check in textlint configs or plugin code to any repository while I tinkered with it. I’m using textlint to check my private files more so than files on which I collaborate with others, so global installation has worked well so far. I saved my configuration file to /usr/local/lib/node_modules/textlint/.textlintrc.

I’m using textlint through Visual Studio Code and the vscode-textlint extension, and added this to my settings.json:

"textlint.configPath": "/usr/local/lib/node_modules/textlint/.textlintrc",
"textlint.nodePath": "/usr/local/lib/node_modules/textlint",

Once install, textlint will do nothing straight out of the box since it has no default rules. The user must select and install rules, and this is where the configuration tasks begin. Note that if you install textlint globally then you must also install each reference rule globally.

The textlint project page lists several dozen rules, about half for English and half for Japanese. It’s not too hard to go through each rule and decide if it’s worth trying. Each rule must be installed (in my case globally) and then configured in .textlintrc. Some rules have more options, and hence need more configuration, than others.

Configuration

I decided to install and configure a bunch of rules, and then observe their behavior. Then I’ll individually turning off rules that don’t seem helpful. I chose these rules:

textlint-rule-no-todo
textlint-rule-alex
textlint-rule-abbr-within-parentheses
textlint-rule-stop-words
textlint-rule-write-good
textlint-rule-en-max-word-count
textlint-rule-en-capitalization
textlint-rule-rousseau
textlint-plugin-html
check-ends-with-period
textlint-rule-terminology
textlint-rule-max-comma
textlint-rule-no-start-duplicated-conjunction
textlint-rule-period-in-list-item
textlint-rule-en-max-word-count
textlint-rule-diacritics
textlint-rule-spellchecker

You can find links to these on the textlint collection of rules page. It took a while to install and configure all the above rules, and I’m not sure I’ve configured them optimally. Here’s a GitHub Gist with my configuration

Consuming output

Results appear inline through squiggly underlines and in Visual Studio Code’s Problems panel, like this:

textlint’s output is also available through the command line and looks like this:

Results and observations

With these checks enabled and configured, textlint can product a lot of output. I’m still working through which textlint rules to fine-tune and which to mute. I’ve configured some redundant checks, and occasionally a useful comparison of rules appears:

In other cases the rules are entirely redundant, such as this case in which I’ve enabled Write Good’s “weasel” and Rousseau’s “weasel” rules:

I also installed Michael Vernier’s SpellChecker extension for Visual Studio Code and configured it in Visual Studio Code’s settings.json:

"spellchecker.ignoreWordsList": [
        "list",
        "of",
        "ignored",
        "words",
    ],
    "spellchecker.ignoreRegExp": [
        "/\\\\(.*\\\\.(jpg|jpeg|png|md|gif|svg|pdf|PDF|SVG|JPG|JPEG|PNG|MD|GIF)\\\\)/g", //remove links to image and markdown files
        "/((http|https|ftp|git)\\\\S*)/g", //remove hyperlinks
        "/^(\\\\s*)(\\\\w+)?(\\\\s*[\\\\w\\\\W]+?\\\\n*)(\\\\s*)\\\\n*$/gm", //remove code blocks
        "/^>[\\\\s\\\\S]*$/gm", //remove quoted text
    ],
    "spellchecker.ignoreFileExtensions": [".textlintrc", ".yml", ".json"],
    "spellchecker.checkInterval": 1500,

and Street Side Software’s Spelling Checker for Visual Studio Code, though I haven’t spent enough time with them to form a solid opinion.

Future wishes

I wish each rule included test files, so I could run each rule and understand its behavior at installation time. This way I could better tune my config from the start. As a workaround I’ve tested rules on the command line using this structure: textlint --rule [RULE NAME] [FILENAME]. It’s rather nice to see textlint’s output in the command line instead of at the bottom of my IDE.

Some of the rules have “auto-fix” functionality, but I haven’t tested this enough yet.

Three things your linter shouldn’t tell you

Pablo Estrada — Thu, 21 Nov 2019 19:37:00 +0000

My coworker Grayson originally wrote this blog post. It was first posted on the r2c engineering blog. At r2c we are linter lovers at heart and value tools that help us write better code, faster. However, sometimes we’re left wanting for a bit more from linters…

Oh, ESLint rules aren’t enabled by default? Which is better, Airbnb’s or Google’s rule list? Bandit is giving me thousands of errors? Oh, those are just asserts. Why does Bandit consider assert a security violation? 🤔 Shouldn’t this be easier?…

Wouldn’t it be great to have a code check tool where you could just run check and it showed you the results you cared about?

We’re making a new developer tool called Bento. Developer time is valuable, and we frequently hear from developers that their linters demand a lot from them. It’s our belief that your linter shouldn’t bother you with things you don’t care about, so in Bento we turn off checks based on these three things:

Linters Shouldn’t Tell You about Style Violations

Frankly, we don’t care about one or two lines between function definitions or whether there should be spaces around binary operators. With deterministic formatters like Black for Python and Prettier for JavaScript, there is no need to spend valuable development time manually addressing style violations when a formatter can do this automatically.

Therefore, Bento’s default configuration explicitly disables all style and preference checks. Bento only reports violations that might** actually be a problem**.

For ESLint (with the Airbnb config), this amounts to disabling all checks labeled “style” by both ESLint and Airbnb, checks with “prefer” in the name, and a handful of checks that our manual examination determined did not indicate the presence of an actual bug.

For Flake 8, this amounts to most pycodestyle checks.

Linters Shouldn’t Tell You about Unpopular Checks

We hypothesized we could get a good estimate for checks that developers like and dislike by mining the configuration files for code checking tools in open source code. Using our at-scale code analysis platform, app.r2c.dev, we mined the ESLint configurations of the top 1,000 npm packages (as of April 2019), and 128 Bandit configurations on GitHub.

Disabled ESLint checks:

Perhaps unsurprisingly, about 38% of repositories globally disabled no-console. Apparently, people like console.log, which makes sense! We disable this check in Bento.

Disabled Bandit checks:

Of the Bandit configs, nearly 80% disable assert_used - B101! We disable this check in Bento, too. About 45% also disable random - B311, so we disable it as well.

As you might imagine, this is the first step in tailoring checks to your code. In the future, Bento will show you the checks you want to see by identifying which checks get fixed… and which get ignored. This will be incredibly powerful—and necessary—as we add support for more code checks.

Linters Shouldn’t Tell You about “Bad Checks”

Granted, the definition of “bad” is subjective! 🤗 Generally speaking, our definition of a “bad check” is a check with a high false-positive rate and a low chance of being a bug.

As an example, B105 checks for hardcoded_password_strings. In theory, this check is great!

it prevents checking in hard-coded secrets, and
it prevents easily guessable defaults — for example, admin:password.

However, in practice, this check is very noisy. The check itself looks for assignments or comparison operators that “look like a password.” This check commonly has a false positive on the variable name token, frequently used in string parsing operations.

We use our static analysis platform to test checks on real-world code. We can test on thousands of open source repositories at once to see where a check yields a true or false positive. Usually, it is easy to spot false positives—such as token in string parsers.

#https://github.com/cobbler/cobbler/blob/91e3ff1d2a05eeb24107ef72e09c1a3d800ae304/cobbler/web/templatetags/site.py#L245
while not self.at_end():
    token = self.get_token()
    if token == 'not':    # B105 fires here
        if self.at_end():
            raise self.error_class('No variable provided after "not".')
        token = self.get_token()
        negate = True

To be more scientific, we can measure the accuracy of the check. After rigorously reviewing 20 repositories, including Django and Ansible, B105 hits nothing of importance 93% of the time. While the theory is good, this check is rather noisy. Therefore, Bento disables this check. (FYI, neither Django nor Ansible use hardcoded passwords!…except in their test files. 😉)

Checks we ship in Bento are reviewed this way. If a check is “bad,” it’s disabled. If “badness” cannot be determined, we leave it enabled. We figure our users will tell us if it’s bad! 😅

Run Bento on your code

We care a lot about code quality at r2c, and we love our linters! We’ve made it even easier to write better code faster by filtering out the checks that we don’t want linters to tell us about. You can run Bento on your code to take advantage of these checks by following this link: https://bento.dev/.

If you have feedback or comments for us, email us or make an issue on GitHub.

Looking to speak with Flask developers

Pablo Estrada — Fri, 15 Nov 2019 19:13:59 +0000

Hi Flask Fans,

I’m Pablo and I work for a small startup in San Francisco. We’re creating a tool specifically to help Flask developers. We’d love to interview some of you and get some insight into Flask developers’ needs and wants.

The interview would last about 30 minutes and would help us better understand how you use Flask and what development tools you’d find useful. As a thanks for taking the time to speak with us, we’ll give you a $50 Amazon gift card.

The tool is called Bento in case you’d like to learn more about it before deciding.

If you’re interested, please submit this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSc2IQrkM3NX7Uka5JGiU-AkZ4Pt1SgOWtRBCt729ULgcXQ23Q/viewform?usp=sf_link

You can also send me a private message if you prefer!

Thanks,
Pablo