JSPS: A VS Code extension for more powerful codebase searches

JS Powered Search is a VS Code extension that lets you search your codebase using the full power of JavaScript. It's powerful, flexible, stateful, and fast. Today I'm going to briefly walk through what it does and how it works. But first:

Why another search tool?

In my mind there are three levels of complexity when it comes to codebase searches.

Level 1

"I need to find every instance of the word ScrollingModalComponent."

Okay, no problem. Click the Search icon and do a text search. VS Code lets you toggle case sensitivity and whole-word searching, which is a pretty great starting point.

Level 2

"I need to find every instance of ScrollingModal, RollingModal, ScrollingModalComponent, and RollingModalComponent, case insensitive. I don't want to find ChildScrollingModal."

Sounds like you need a Regular Expression. Something like \b(sc)?rollingmodal(component)?\b should do the trick. Click the RegEx toggle in VS Code's search pane, pop in the RegEx, and you're all set. RegExes are pretty useful and will usually do what you need, but their logical abilities are very limited. And if you don't write them very often, it can be time-consuming to figure out the right syntax. (Pro tip: use RegExr.)

Level 3

"I need to find every TypeScript file that's at least 100 lines long, exports a ScrollingModalComponent, uses a .filter method two or more times, and does not contain the term RollingModal."

Granted, perhaps you could figure out a RegEx for this. And if you're an avid RegEx golfer, maybe you even want to. But for most of us this sounds like a headache. Especially if Level 2 was already a headache.

Be honest: you're going to end up doing a plain-text search for "ScrollingModalComponent" and checking each file by hand. If the search only comes back with ten or so results, no big deal. But if there are tens or hundreds of results? That's going to take up the rest of your day for sure.

You may wonder, "why can't I just write some custom Array.filter logic and run all my files through it?"

Now you can.

What does JSPS do?

JSPS has two main operations: Scaffold and Search.

Scaffold produces a template file that you can alter to define the search functionality you need. It looks like this:

export interface SearchOptions {
  includeFilePatterns?: string[]; // globs to include, e.g. ['**/*.ts']. Searches all files by default.
  excludeFilePatterns?: string[]; // globs to exclude.
  includeNodeModules?: boolean; // (default: false) true if node_modules should be searched. Strongly discouraged.
  maxFileSizeInKB?: number; // (default: 1000) any files larger than this will be skipped.
  onlyTestLinesInMatchingFiles?: boolean; // (default: false) true if searchByLine should only be used on files that pass searchByFile
}

export function getSettings(): SearchOptions {
  return {
    // includeFilePatterns: [],
    // excludeFilePatterns: [],
    // includeNodeModules: false,
    // maxFileSizeInKB: 1000,
    // onlyTestLinesInMatchingFiles: false
  };
}

export interface LineSearchMetadata {
  fileName: string;
  filePath: string;
}

export interface LineSearchOptions {
  // A function that accepts a line of text and determines whether it matches your search.
  // If you only want to search by file, set this method to undefined.
  doesLineMatchSearch?: (line: string, metadata: LineSearchMetadata) => boolean;
}

export function searchByLine(): LineSearchOptions {
  return {
    doesLineMatchSearch: (line, metadata) => {
      return line.includes("exactly what I'm looking for");
    },
  };
}

export interface FileSearchMetadata {
  fileName: string;
  filePath: string;
  lines: string[]; // The file text as an array of lines
}

export interface FileSearchOptions {
  // A function that accepts a file (as a text string) and determines whether the file matches your search.
  // If you only want to search by line, set this method to undefined.
  doesFileMatchSearch?: (
    fileContents: string,
    metadata: FileSearchMetadata
  ) => boolean;
}

export function searchByFile(): FileSearchOptions {
  return {
    doesFileMatchSearch: (file, metadata) => {
      return file.includes("another thing I'm looking for");
    },
  };
}

Three functions are exported. One returns your desired search settings, one returns a function for matching by line, and one returns a function for matching by file.

So if you want to find every instance of "ScrollingModalComponent" in your codebase, that's a three-step process:

1. Delete the doesFileMatchSearch function. We won't need it here.
2. Update doesLineMatchSearch to the following:

doesLineMatchSearch: (line, metadata) => {
  return line.includes("ScrollingModalComponent");
},

3 . Run the JSPS Search command.

All the search settings have sensible defaults, so you most likely won't need to change them for simple searches. And in seconds, you'll have a full list of search results. You can click any of them to jump to the file and line in question.

Let's revisit that level 3 search.

I need to find every TypeScript file that's at least 100 lines long, exports a ScrollingModalComponent, uses a .filter method two or more times, and does not contain the term RollingModal.

JSPS can do that without breaking a sweat. Here's what the search definition would look like:

export interface SearchOptions {
  includeFilePatterns?: string[]; // globs to include, e.g. ['**/*.ts']. Searches all files by default.
  excludeFilePatterns?: string[]; // globs to exclude.
  includeNodeModules?: boolean; // (default: false) true if node_modules should be searched. Strongly discouraged.
  maxFileSizeInKB?: number; // (default: 1000) any files larger than this will be skipped.
  onlyTestLinesInMatchingFiles?: boolean; // (default: false) true if searchByLine should only be used on files that pass searchByFile
}

export function getSettings(): SearchOptions {
  return {
    includeFilePatterns: ["**/*.ts"],
    // excludeFilePatterns: [],
    // includeNodeModules: false,
    // maxFileSizeInKB: 1000,
    onlyTestLinesInMatchingFiles: true,
  };
}

export interface LineSearchMetadata {
  fileName: string;
  filePath: string;
}

export interface LineSearchOptions {
  // A function that accepts a line of text and determines whether it matches your search.
  // If you only want to search by file, set this method to undefined.
  doesLineMatchSearch?: (line: string, metadata: LineSearchMetadata) => boolean;
}

export function searchByLine(): LineSearchOptions {
  return {
    doesLineMatchSearch: (line, metadata) => {
      return line.startsWith("export") && line.includes('ScrollingModalComponent');
    },
  };
}

export interface FileSearchMetadata {
  fileName: string;
  filePath: string;
  lines: string[]; // The file text as an array of lines
}

export interface FileSearchOptions {
  // A function that accepts a file (as a text string) and determines whether the file matches your search.
  // If you only want to search by line, set this method to undefined.
  doesFileMatchSearch?: (
    fileContents: string,
    metadata: FileSearchMetadata
  ) => boolean;
}

export function searchByFile(): FileSearchOptions {
  return {
    doesFileMatchSearch: (file, metadata) => {
      return (
        metadata.lines.length >= 100 &&
        metadata.lines.filter((l) => l.includes(".filter")).length >= 2 &&
        !file.includes("RollingModal")
      );
    },
  };
}

Here's what happens when you run this as a JSPS Search:

Once again, you have the results you need and all you had to do was write a bit of string logic; no RegEx required (although if case sensitivity was an issue, you might need some RegEx for word boundaries or something).

The final low-key superpower of JSPS is that search definition files are totally saveable and reusable. You can commit a few to your repo as a "table of contents" for new developers; you can collaborate on one with VS Code Live Share; you can send a JSPS file to your team to show them the impact of a large refactor; you can keep advanced searches on your hard drive in case you need them later. It's totally up to you.

How it works

VS Code has a very developer-friendly extension API with above-average documentation (I have some complaints about its Googleability, but that's a topic for another day). Creating a file based on a template is pretty ho-hum stuff with the vscode.window and TextDocument APIs, so the scaffolding part didn't take long to set up.

The Search command was more difficult. JSPS has to grab the full text of the current editor window, transpile it to JavaScript, then compile it into a dynamic JavaScript module. On-the-fly transpilation is a basic feature of the typescript package, but compiling a string to a JavaScript module isn't a basic feature of anything. For that I'm using undocumented features of the Module package (which is built into Node) via the require-from-string NPM package. It's not a best practice, but it works--with the caveat that I'm pretty sure import won't work in search definition files. That's on the roadmap to look at later.

From that point, all JSPS has to do is validate your search definition, find files in the workspace that match the provided include and exclude globs (which is easy, thanks again to the VS Code API), and test each one with the functions you wrote.

Of course, a search utility like this has a lot of UI and error reporting needs. Luckily, the VS Code API has a ton of UI features to choose from.

What's next?

There are some known issues with JSPS, but from my testing over the last few days it works pretty well. I'm going to be refining it and fixing edge cases for a while (you're welcome to join in at the official GitHub repo) before I take it out of preview and publish version 1.0.

In the meantime, I'd love to have your feedback. Do you see an opportunity to improve performance or UX? Does it break on a particular OS or with particular types of files? What features would you like to see added?

Long-term, I think a find-and-replace feature that capitalizes on the power of JSPS Search would make a lot of sense, so that's something I want to build soon.

I hope JSPS is useful when your codebase searches need a power boost. Enjoy!