DEV Community

Cover image for I Compared an Offline Security Audit With SonarQube on 4 Real Projects with Observer
getobserver
getobserver

Posted on Edited on

I Compared an Offline Security Audit With SonarQube on 4 Real Projects with Observer

#security #opensource #php #devtools

I wasn't planning to compare Observer with SonarQube.

I was just curious.

I've been building Observer because I wanted a simpler first pass when looking at an unfamiliar codebase.

The problem I kept running into was not finding enough information.

It was having too much information and still not knowing where to start.

So I built a local audit workflow around a simple question:

"What should I look at first?"

Recently I decided to test that idea against SonarQube.

I used the exact same 4 projects:

  • Monica: Laravel
  • WordPress: PHP
  • OWASP Juice Shop: Node.js
  • FinDocAnalyzer: Laravel

The results from my tests

Project Observer SonarQube
Monica 169 2,530+
WordPress 244 2,470+
Juice Shop 226 312+
FinDocAnalyzer 32 63+

Measured on v0.6.0.

At first those numbers look strange.

But I don't think they're really a "who found more?" comparison.

The tools are doing different jobs.

SonarQube has a broad code-quality and static-analysis focus.

Observer is more focused on security triage and actionable findings.

That difference was actually the most interesting part of the experiment.

When I audit a project, I don't always want to start with a giant list.

I want to quickly understand:

  • Where are the risky areas?
  • What should I investigate first?
  • Where is the actual code?
  • What can I do about it?

That's what Observer is trying to make easier.

The basic workflow 

observer analyze . --out report.html
Enter fullscreen mode Exit fullscreen mode

It runs locally as a single binary and generates a self-contained HTML report. No account, no telemetry, and --assert-offline enforces that your code never leaves your machine.

What's new in v0.7.0

Running these comparisons exposed a few rough edges, and v0.7.0 fixes them:

  • Scans that don't hang. Deep scans on large codebases used to be killed by a fixed 5-minute budget. Timeouts now scale with file count (a WordPress-sized repo of ~4,600 files gets roughly 7–9 minutes), and you can override them with OBSERVER_SCAN_TIMEOUT=30m.
  • A time estimate up front. After file discovery, Observer prints an estimate so you know what you're in for.
  • Fairer counts. Semgrep now scans the same file set as Observer's built-in engine, with vendor, node_modules, dist, build and *.min.js excluded, so finding counts are apples-to-apples.
  • A more honest comparison page. The README and landing page now include a "Code leaves your machine?" row and label which engine (built-in, Semgrep, PHPStan) produced each finding.

GitHub: https://github.com/sanks205/getobserver

Release notes: https://github.com/sanks205/getobserver/releases/tag/v0.7.0

I'm curious how you handle this

When a code scan finishes, do you normally start with security findings, code quality, dependencies, runtime errors, or something else?

Top comments (0)