Integrating the ruff language server

I want to add linting capabilities to Satyrn (a Jupyter notebook client application I'm working on), so I looked into integrating ruff, a python linting and code formatting tool. This tool follows the Language Server Protocol, so as long as I can figure out how to follow that protocol, I should be able to use it in my app.

This post covers some background on the Language Server Protocol and then goes into a demonstration of how to integrate the ruff language server into a simple code editor built using CodeMirror. The code for the demonstration is availiable on github.

The Langauge Server Protocol

All the cool features in your IDE like jump-to-definition, code formatting, linting, and autocomplete are made possible by the Language Server Protocol (LSP). Without this standard it would be quite challenging for developers of these tools to support all the popular code editors out there, and likewise for the developers of IDEs to integrate language support for so many different languages.

Fortunately the LSP defines a standard interface between a code editor and a Language Server (LS), exposing information about the code editor to the LS, and suggested actions from the LS to the code editor. This standard has lead to an explosion of language-specific plugins availiable in most IDEs.

From the LSP website:

The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort.

In order to empower the LS with the context it needs to do its job properly, the LSP requires the code editor to send information about the open document so the LS can keep a copy of the active document in memory. As the user edits the open document these updates have to be written to the LS to keep it in sync.

When the user manually triggers a feature of the LS (like formatting a selection of code) a request is sent to the LS and actions are returned (eg edit action to update the selected code). The surprising thing about this process to me is that these requests are typically sent over stdout and stdin, instead of over the network. I’m use to building cloud software, where most processes are packaged up in their own docker containers and only communicate over the network, so this was new for me.

The LSP also requires the code editor listen out for notifications from the LS. This enables features like linting, where the LS can warn the user about new errors detected in their code as they type. The specifications of the LSP and all the current capabilities are well documented. I’m particularly interested to see what kind of new intelligent actions will be added to the LSP over the coming years.

Integrating the ruff language server

To learn more about how to integrate a Language Server (LS) into a code editor, I decided to create a simple python code editor with linting and code formatting capabilities (using the ruff language server).

I built the editor using ruff, CodeMirror, VS Code’s LSP client library, and Electron.

The rest of the post goes into some detail on the implementation. This post assumes some familiarity with React and Electron, but I will try to abstract those details so it’s not super important for you to follow along.

Creating a simple code editor

You can create a simple code editor in javascript in a matter of minutes by working with one of the many amazing open source projects like CodeMirror or Monaco Editor. Below is a minimal example of a code editor using CodeMirror. There is a button to format your code, but it does not do anything yet.

import ReactCodeMirror from '@uiw/react-codemirror'
import { python } from '@codemirror/lang-python'
import { useState } from 'react'

function App(): JSX.Element {
  const [code, setCode] = useState('')

  function handleFormatCode() { // TODO: format code }

  function handleCodeChange(newCode) => {
    setCode(newCode)
  }

  return (
    <>
      <ReactCodeMirror 
        value={code} 
        extensions={[python()]} 
        onChange={handleCodeChange} 
        height="200px"
      />
      <button onClick={handleFormatCode}>Format Code</button>
    </>
  )
}

The ruff CLI and language server

Ruff is a python code formatter and linter. You can use it as a drop-in replacement for black for code formatting. You can pip install it and run it from the command line via ruff format example.py.

If you are integrating it into a code editor like we are about to you can also use the ruff server --preview command to start the language server (you need --preview because its in beta).

Instead of launching this from the command line we can launch it via javascript from within our app (the node backend of our electron app):

import { ChildProcess, spawn } from 'child_process'

const ruffServer: ChildProcess = spawn('ruff', ['server', '--preview'])

As long as we run the above command somewhere when our app starts up this will start the ruff language server in the background, ready to help us with code formatting and linting.

Sending code to ruff

So that ruff can do its job correctly it needs to be initialized and have an up-to-date copy of the code from our editor.

We’ll essentially need our editor to emit information about the document that is being edited and edits made to the document. And also listen out for notifications from the language server to do useful stuff like display linting errors.

The language server protocol is well documented and there are great open source libraries we can use to abstract some of the machinery.

Initializing the server

We’ll use VS Code’s language server client library to interact with the ruff server

import { Connection, createConnection, StreamMessageReader, 
        StreamMessageWriter,  InitializeParams, InitializeResult
} from 'vscode-languageserver/node'

// Create the connection and start listening
const connection = createConnection(
  new StreamMessageReader(ruffServer.stdout),
  new StreamMessageWriter(ruffServer.stdin)
)
connection.listen()

// Send initialization request and notification
await connection.sendRequest<InitializeResult>('initialize', {
  processId: process.pid,
  rootUri: null,
  capabilities: {}
})
await connection.sendNotification('initialized')

This is quite an interesting piece of code, because it shows you how the LSP client library communicates to the ruff child process via stdout and stdin. Our first message is sending the initialization request.

Send ruff the code

After initializing the language server we tell it we are working on a document. You will usually use the path to your document as the document ID, but since we’re not persisting our document yet we’ll create our own path. We’ll also need to keep track of the document version so that we can tell ruff what version number our edits are as we update it.

import { TextDocumentItem } from 'vscode-languageserver/node'

const documentId = `file:///document_${Date.now()}.py`
let documentVersion: number = 1
connection.sendNotification('textDocument/didOpen', {
  textDocument: TextDocumentItem.create(
    documentId, 
    'python', // set the language
    documentVersion, 
    '' // open a blank document
  )
})

Editing our file

The simplest approach to sending file edits to the ruff server is to overwrite the entire document whenever the user changes the code. From our react app we can forward all code changes to our electron backend:

function handleCodeChange(newCode) => {
  setCode(newCode)
  window.api.send('update-code', newCode)
}

Then we can forward all changes from the electron backend to the ruff server:

function forwardCodeChange(connection, documentId, documentVersion, newCode){
  connection.sendNotification('textDocument/didChange', {
    textDocument: { uri: documentId, version: documentVersion },
    { text: newCode }
  })
}

forwardCodeChange(connection, documentId, documentVersion, newCode)
documentVersion += 1

Now the ruff server is kept in sync with the code in our code editor. A more advanced technique will send incremental edits from the text editor to the ruff server (checkout github for the implementation).

Formatting our code

Finally we can hookup the button to format our code!

First let's fill in the function we left empty earlier to pass the code to the electron backend when the user presses the button:

function handleFormatCode(): void {
  window.api.invoke<string>('format-code', code).then((formattedCode) => {
    setCode(formattedCode)
  })
}

And on the backend we can make the request to the ruff server and get the formatted code back. Notice the ruff server sends us TextEdit objects, which contain a range and the new text to replace the old text. I've skipped the details of applyChanges but you can see my approach on github if you are interested.

import { TextEdit } from 'vscode-languageserver/node'

function formatCode(connection, documentId, code) {
  const formattingResult: TextEdit[] = connection.sendRequest<TextEdit[]>(
    'textDocument/formatting', {
    textDocument: { uri: documentId },
    options: { tabSize: 4, insertSpaces: true }
  })
  const formattedCode = applyChanges(code, formattingResult)
  return formattedCode
}

And that’s it, now our format button is hooked up and works!

Linting

To listen out for linting errors from the language server we need to add these lines to our electron backend:

import { EventEmitter } from 'events'

const diagnosticsEmitter = new EventEmitter()

connection.onNotification('textDocument/publishDiagnostics', (params) => {
  diagnosticsEmitter.emit('diagnostics', params.diagnostics)
})

When the ruff server detects an error and wants to notify our editor it will emit diagnostics over this channel. We can add a callback function to our language server client to listen out for these diagnostics and forward them on to our code editor in the frontend:

languageServer.onDiagnostics((diagnostics) => {
  mainWindow.webContents.send('diagnostics', diagnostics)
})

CodeMirror has a nice utility for displaying linting errors, but we’ll need to convert the linting errors we get from ruff to fit their datamodel

import { Diagnostic as CMDiagnostic } from '@codemirror/lint'
import { Diagnostic as LSPDiagnostic } from 'vscode-languageserver-protocol'

function convertLSPDiagnostics(lspDiagnostic: LSPDiagnostic[]): CMDiagnostic[] {
  return lspDiagnostic.map((d) => ({
    from: d.range.start.character,
    to: d.range.end.character,
    severity: convertSeverity(d.severity),
    message: d.message,
    source: d.source
  }))
}

function convertSeverity(severity: number | undefined): 'info' | 'warning' | 'error' {
  switch (severity) {
    case 1:
      return 'error'
    case 2:
      return 'warning'
    default:
      return 'info'
  }
}

And finally we can display those diagnostics in our CodeMirror editor using the lintGutter and linter extensions:

// ... existing imports ...
import { linter, lintGutter } from '@codemirror/lint'

function App(): JSX.Element {
  // ... existing code ...

  const [diagnostics, setDiagnostics] = useState<CMDiagnostic[]>([])

  useEffect(() => {
    window.api.receive('diagnostics', (receivedDiagnostics: LSPDiagnostic[]) => {
      const cmDiagnostics = convertLSPDiagnostics(receivedDiagnostics)
      setDiagnostics(cmDiagnostics)
    })
    return (): void => {
      window.api.removeListeners('diagnostics')
    }
  }, [])

  const linterExtension = linter(() => diagnostics)

  return (
    <>
      <ReactCodeMirror
        value={code}
        height="200px"
        extensions={[python(), linterExtension, lintGutter()]}
        onUpdate={handleCodeUpdate}
      />
      <button onClick={handleFormatCode}>Format Code</button>
    </>
  )
}

And voilà! Now we have linting errors!

Finale

I hope you enjoyed this post looking at integrating the ruff language server into a code editor to unlock linting and code formatting capabilities. I am looking forward to taking these learnings and supercharging Satyrn. If you want to learn more about the implementation check out the code on github.