DEV Community

Rodrigo Mello
Rodrigo Mello

Posted on

Spawning a PTY in Rust: How broll Captures Your Terminal Without You Noticing

#rust #cli #terminal #opensource

When you run broll start, something subtle happens. A new shell opens, and it looks exactly like your normal terminal. Your prompt, aliases, keybindings, colors: everything works. But behind the scenes, every byte of output flows through broll's recording pipeline before it reaches your screen.

The trick that makes this possible is a pseudo-terminal (PTY). This post explains what PTYs are, how broll spawns one in Rust, how it injects shell hooks using OSC escape sequences to distinguish commands from output, and how it uses the Drop trait to guarantee safe cleanup.

What Is a PTY and Why Do You Need One?

A PTY is a pair of virtual devices: a "master" side and a "slave" side. The slave side looks like a real terminal to any program connected to it. The master side gives you programmatic access to everything the program writes, and lets you send input as if a human were typing.

Without a PTY, you could try capturing output by piping stdout. But that breaks interactive programs. Tools like vim, less, and htop need a real terminal to query its size, move the cursor, and read keypresses. They detect when they are connected to a pipe and either refuse to run or fall back to a degraded mode.

A PTY solves this because the child process sees a genuine terminal device. It can call ioctl to get the terminal size, switch to raw mode, and use escape sequences. Meanwhile, the parent process reads and writes through the master side, acting as an invisible intermediary.

Spawning the PTY with portable-pty

broll uses the portable-pty crate (version 0.8) to abstract over platform differences. Here is the setup from src/recorder/mod.rs:

let (cols, rows) = crossterm::terminal::size().unwrap_or((80, 24));

let pty_system = NativePtySystem::default();
let pair = pty_system
    .openpty(PtySize {
        rows,
        cols,
        pixel_width: 0,
        pixel_height: 0,
    })
    .context("Failed to open PTY")?;
Enter fullscreen mode Exit fullscreen mode

openpty returns a PtyPair containing the master and slave ends. The size is initialized from the real terminal's dimensions so the child shell renders correctly from the start.

Next, broll builds the command to spawn in the PTY:

let mut cmd = CommandBuilder::new(&shell);
cmd.env(SESSION_ENV_VAR, &session_id);
cmd.env("BROLL_SESSION", session_label);
cmd.cwd(&work_dir);

let mut child = pair.slave.spawn_command(cmd)?;
drop(pair.slave);
Enter fullscreen mode Exit fullscreen mode

Two details matter here. First, SESSION_ENV_VAR (BROLL_SESSION_ID) is injected so broll can detect nested sessions and prevent recording inside a recording. Second, pair.slave is dropped immediately after spawning. The slave side is only needed to create the child process. Keeping it open would prevent the master from receiving EOF when the child exits.

After spawning, broll obtains a reader and writer for the master side:

let mut reader = pair.master.try_clone_reader()?;
let writer = pair.master.take_writer()?;
Enter fullscreen mode Exit fullscreen mode

The reader is used by the main thread to consume PTY output. The writer is moved into a separate thread that forwards stdin. This separation is the foundation of broll's three-thread architecture, which the next post in this series covers in detail.

Shell Hook Injection with OSC Sequences

Recording raw terminal output is useful, but broll wants to do more: it needs to know which bytes are part of a command you typed and which bytes are that command's output. To do this, it injects shell hooks that emit invisible markers.

What Are OSC Sequences?

OSC stands for Operating System Command. These are escape sequences in the format \x1b]<code>;<data>\x07 that terminals use for metadata like setting the window title. Terminals silently ignore OSC codes they do not recognize, which makes them perfect for embedding custom signals in the byte stream.

broll uses private OSC code 777 (a convention for application-specific extensions):

const PREEXEC_MARKER_PREFIX: &str = "\x1b]777;broll-exec;";
const PREEXEC_MARKER_END: char = '\x07';
const PRECMD_MARKER: &str = "\x1b]777;broll-cmd\x07";
Enter fullscreen mode Exit fullscreen mode

The preexec marker fires just before a command runs and includes the command text. The precmd marker fires when the command finishes and the prompt is about to appear.

Injecting the Hooks

broll cannot modify the user's shell configuration permanently. Instead, it creates a temporary rc file that first sources the user's existing config, then appends the hook functions:

fn create_hook_rc(shell_name: &str) -> Option<tempfile::TempDir> {
    let broll_data = dirs::data_dir()?.join("broll");
    std::fs::create_dir_all(&broll_data).ok()?;
    let tmp_dir = tempfile::Builder::new()
        .prefix("broll-")
        .tempdir_in(&broll_data)
        .ok()?;

    let hook_code = match shell_name {
        "zsh" => {
            let user_zshrc = dirs::home_dir()
                .map(|h| h.join(".zshrc"))
                .filter(|p| p.exists());
            let source_line = user_zshrc
                .map(|p| format!("[[ -f \"{}\" ]] && source \"{0}\"\n", p.display()))
                .unwrap_or_default();

            let rc_path = tmp_dir.path().join(".zshrc");
            let content = format!(
                concat!(
                    "{}",
                    "_broll_preexec() {{\n",
                    "  local cmd=\"${{1//$'\\a'/}}\"\n",
                    "  printf '\\e]777;broll-exec;%s\\a' \"$cmd\"\n",
                    "}}\n",
                    "_broll_precmd() {{ printf '\\e]777;broll-cmd\\a'; }}\n",
                    "autoload -Uz add-zsh-hook\n",
                    "add-zsh-hook preexec _broll_preexec\n",
                    "add-zsh-hook precmd _broll_precmd\n",
                ),
                source_line,
            );
            std::fs::write(&rc_path, content).ok()?;
            Some(())
        }
        // bash variant uses PROMPT_COMMAND similarly
        _ => None,
    };
    hook_code.map(|_| tmp_dir)
}
Enter fullscreen mode Exit fullscreen mode

For zsh, the trick is setting the ZDOTDIR environment variable to point at the temp directory. zsh loads $ZDOTDIR/.zshrc on startup instead of ~/.zshrc:

match shell_name.as_str() {
    "zsh" => {
        cmd.env("ZDOTDIR", tmp.path().to_str().unwrap_or("/tmp"));
    }
    "bash" => {
        let rc_path = tmp.path().join(".bashrc");
        cmd.args(["--rcfile", rc_path.to_str().unwrap_or("/tmp/.bashrc")]);
    }
    _ => {}
}
Enter fullscreen mode Exit fullscreen mode

For bash, the --rcfile argument serves the same purpose.

The State Machine

On the storage thread, a simple two-state machine processes the markers:

#[derive(PartialEq)]
enum CaptureState {
    /// Between precmd and preexec. Prompt + user typing. Skip this.
    Idle,
    /// Between preexec and precmd. Real command output. Capture this.
    Capturing,
}
Enter fullscreen mode Exit fullscreen mode

When the storage thread sees a preexec marker, it extracts the command text, stores it as an input chunk, and transitions to Capturing. When it sees a precmd marker, it renders the accumulated output through a VT100 virtual terminal and stores the result. Then it transitions back to Idle.

The tempfile::TempDir return value is important. TempDir implements Drop, so when the variable holding it goes out of scope (when the session ends), the temporary directory and its contents are automatically deleted. The caller stores it in _tmp_dir, where the underscore prefix tells the reader (and the compiler) that the binding exists solely for its destructor.

The RAII RawModeGuard

When the PTY is active, broll puts the real terminal into raw mode so that every keystroke reaches the child process immediately (instead of being line-buffered). But if the program panics or returns early, the terminal must be restored. Leaving a terminal in raw mode is a miserable user experience: no echo, no line editing, no Ctrl+C.

broll uses the RAII (Resource Acquisition Is Initialization) pattern to guarantee cleanup:

struct RawModeGuard;

impl RawModeGuard {
    fn enable() -> Result<Self> {
        crossterm::terminal::enable_raw_mode()?;
        Ok(Self)
    }
}

impl Drop for RawModeGuard {
    fn drop(&mut self) {
        let _ = crossterm::terminal::disable_raw_mode();
    }
}
Enter fullscreen mode Exit fullscreen mode

The Drop trait is Rust's destructor. When a RawModeGuard goes out of scope, whether through normal return, early ? propagation, or a panic, disable_raw_mode() runs automatically. The let _ = discards any error from the cleanup call, because there is nothing useful to do if disabling raw mode fails during stack unwinding.

In start_session, the guard is created right before the I/O loop:

let _raw_guard = RawModeGuard::enable()?;
Enter fullscreen mode Exit fullscreen mode

And explicitly dropped before printing exit messages:

drop(_raw_guard);
eprintln!("broll: session {} ended", &session_id[..8]);
Enter fullscreen mode Exit fullscreen mode

This explicit drop call ensures raw mode is disabled before the final message, so \n works correctly in the output.

A Minimal PTY Example

Here is a standalone example that spawns a PTY, runs ls, and captures the output. Add portable-pty = "0.8" to your dependencies:

use portable_pty::{CommandBuilder, NativePtySystem, PtySize, PtySystem};
use std::io::Read;

fn main() -> anyhow::Result<()> {
    let pty_system = NativePtySystem::default();
    let pair = pty_system.openpty(PtySize {
        rows: 24,
        cols: 80,
        pixel_width: 0,
        pixel_height: 0,
    })?;

    let mut cmd = CommandBuilder::new("ls");
    cmd.arg("-la");

    let mut child = pair.slave.spawn_command(cmd)?;
    drop(pair.slave); // Must drop so master gets EOF

    let mut reader = pair.master.try_clone_reader()?;
    let mut output = String::new();
    reader.read_to_string(&mut output)?;

    child.wait()?;
    println!("Captured output:\n{}", output);

    Ok(())
}
Enter fullscreen mode Exit fullscreen mode

The key insight: after drop(pair.slave), the master reader will receive EOF when the child exits. Without that drop, read_to_string would block forever.

Putting It Together

The flow when a user runs broll start looks like this:

  1. Create a PTY pair sized to match the real terminal.
  2. Build a temporary rc file with OSC-emitting hooks.
  3. Spawn the user's shell in the PTY slave with ZDOTDIR (or --rcfile) pointing to the temp rc.
  4. Drop the slave side.
  5. Enable raw mode with a RawModeGuard.
  6. Start I/O threads (covered in the next post).
  7. The shell loads the hooks. Every command triggers preexec/precmd markers.
  8. The storage thread parses markers, renders output through VT100, and stores chunks in SQLite.
  9. When the shell exits, the master reader gets EOF, the channel closes, and cleanup runs in reverse order.

The user sees a normal shell. The hooks are invisible. The temp files clean themselves up. And every command, along with its output, ends up in a searchable database.

Try it out:

Top comments (0)