What Is the Earshot Open-Source Toolkit?
Earshot is a Python toolkit described as "ears for coding agents: let AI hear, review and make sound." It is MIT-licensed, CPU-only, and available as a CLI, a Python API, and an MCP server. It was released publicly in mid-October 2026 by developer Leon Lin (@LexnLin), who introduced it in a thread on X on October 15. The official source is the Earshot GitHub repository.
In practice, Earshot can:
Describe and classify audio with sound labels and measurements.
Match a file against a text description.
Find specific sound events inside longer recordings.
Analyze audio alongside video, including sync to picture cuts.
Evaluate sound design against a profile of professional launch films.
Generate sounds in code (forge) and convert recordings into editable code (fit).
Plug into coding agents such as Claude Code via MCP.
Why Do AI Coding Agents Need Audio Analysis?
Writing audio code is not the same as knowing how it sounds. Without listening, an agent has no feedback loop. Earshot also gives agents something richer than transcripts or file metadata to reason about. Practical cases:
Checking whether a generated keyboard sound feels realistic.
Finding sound effects that match a scene.
Catching timing problems between sound and picture.
Comparing generated sounds with reference recordings.
Improving sound effects through iterative testing.
Key Features of Earshot
Feature
What It Does
Practical Use Case
earshot describe
Describes a file with calibrated tags, AudioSet labels, and measurements
Confirm a generated sound is what you intended
earshot match
Scores how well a text description fits the audio (strong, plausible, weak, no match)
Check that a sound fits "a deep thocky keyboard press"
earshot rank
Sorts many candidate files by a description
Pick the best of several downloaded effects
earshot find
Finds the best matching moments as start and end trims
Extract one clean key press from a long recording
earshot compare / ab
Finds nearest references; compares two sounds in words
See how a generated sound differs from a reference
earshot realness
Estimates real or pro-produced versus code-synthesized
Check whether a sound reads as synthetic
earshot timeline
Listening transcript with events, loudness, beat grid, and sync to cuts
Review a film's audio second by second
earshot sheet
Annotated PNG of events, spectrogram, waveform, and loudness
Give image-reading agents a visual of the audio
earshot critique
Scores a video against a profile of seven professional launch films
Review a product demo's sound design
earshot measure
Model-free measurements: LUFS, true peak, LRA, spectrum, tempo and key
Check loudness before export
earshot forge
Generates sounds from synthesis recipes and improves them through a scoring loop
Build a keypress, whoosh, or UI click in code
earshot fit
Listens to a recording and writes an editable Python recipe that recreates it
Turn a reference sound into tweakable code
earshot gallery
Builds a static HTML gallery of generated sound variations
Audition many forge outputs in a browser
earshot mcp
Exposes the tools as an MCP server
Connect Earshot to Claude Code
earshot doctor
Checks the install, models, and CPU settings
Troubleshoot setup
Every command prints a plain-language summary first. The --json flag returns the full result and --out f.json saves it.
What's New in Version 0.2.0
The 0.2.0 release (October 16) added earshot fit and Forge v2. According to the release notes, the library of generated effects grew from 5 types (15 sounds) to 34 types (183 sounds) covering motion-design staples: whooshes, risers, impacts, sub drops, logo stings, UI clicks, toggles, pops, dings, shimmers, typing, and glitches. It also added a held-out benchmark, a sharper critic, a faster search, and earshot gallery. The release notes report that on 12 real recordings, fit renders reached a median CLAP similarity of 0.945. That is the developer's own measurement, so treat it as indicative, not as independent proof.
How Earshot Works
You supply an audio file or video.
Earshot analyzes it with machine-learning models plus signal measurements.
It returns evidence: descriptions, scores, timestamps, or a scorecard.
The coding agent acts on that evidence, for example by adjusting a recipe or moving an effect.
The loop repeats until the result improves.
Three components do the analysis:
CLAP audio-text embeddings (laion/larger_clap_general) place audio and text in a shared space, which makes "does this sound match this sentence?" answerable.
The AudioSet classifier (MIT/ast-finetuned-audioset-10-10-0.4593) assigns labels from the AudioSet ontology.
Model-free measurements such as loudness, true peak, envelope, and spectrum do not depend on any model and act as a cross-check.
That cross-check matters because forge optimizes against Earshot's own models. The README warns that a sound can please the models without truly improving (a Goodhart effect), and recommends judging results with model-free measurements, spectrograms next to references, and human listening.
How to Install Earshot
Prerequisites: Python 3.10 or newer, Git, FFmpeg and FFprobe on PATH, and about 1.4 GB of disk space for model downloads.
Step 1: Clone the repository and create a virtual environment.
git clone https://github.com/Leonxlnx/earshot && cd earshot
python -m venv .venv
Step 2: Install CPU-only PyTorch, then Earshot. README calls the virtual environment's Python directly. On macOS and Linux that is .venv/bin/python; on Windows use .venv/Scripts/python.
.venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/python -m pip install -e ".[mcp,llm,dev]"
If you prefer to activate the environment, use source .venv/bin/activate on macOS and Linux, or .venv\Scripts\activate on Windows, then run python -m pip. Check the README if a command differs on your system.
Step 3: Run the diagnostic.
.venv/bin/python -m earshot doctor --load
The doctor command checks the install, models, data files, and CPU settings. The first model downloads two open models (about 1.4 GB) into .cache/hf inside the repo, or into EARSHOT_HOME. After that, Earshot works offline. Optional environment variables: EARSHOT_THREADS, EARSHOT_HOME, EARSHOT_PRECISION, and EARSHOT_OFFLINE=1.
Speed note: each CLI call is a new process and loads the models first (the README says roughly 20 to 40 seconds). For more than a few calls, use the MCP server or a single Python script.
How to Use Earshot: Practical Examples
These examples assume the earshot command is available in your environment. Otherwise use python -m earshot with your virtual environment's Python.
Example A: Describe a file
earshot describe sound.wav
Returns what the file sounds like: envelope shape, closest tags, and measurements. Use case: checking that a generated "keyboard click" reads as one.
Example B: Match a description
earshot match sound.wav "a deep mechanical keyboard press"
Returns a verdict from strong to no match, with evidence. Add --alts to compare against competing descriptions. Use case: validating a sound before it goes into a mix.
Example C: Find a sound in a recording
earshot find recording.wav "a single keyboard key press" --top 4
Returns time ranges you can use as trims. Use case: pulling one clean key press from a minute of typing.
Example D: Critique a video
earshot critique film.mp4
Returns a verdict and timestamped fixes ordered by severity. Use case: reviewing a product demo. The scorecard reflects the style of its seven reference films, not every good film.
Example E: Turn a recording into editable code
earshot fit sound.wav -o recipe.py
Writes a readable Python recipe with knobs for new variations. Use case: making ten variants of a click from one reference. Licensing note: a recipe fitted to a recording stays tied to that recording's licence, so do not assume fitted code is free to use commercially.
Example F: Browse generated sounds
earshot gallery output_directory -o gallery.html
Builds a static HTML gallery of sound variations. Use case: auditioning forge outputs side by side. Confirm the exact flags with the earshot gallery --help.
Bonus: generate a sound with forge
earshot forge
earshot forge whoosh --target "a fast crisp air whoosh" --refs path/to/real/whooshes/ --dir made/whoosh
The forge command synthesizes candidates from physical-modeling recipes, scores them with a criterion built from Earshot's own analysis, and keeps the best through an evolutionary search. Output is a set of WAV files plus a report.json with every candidate's parameters and scores.
How to Connect Earshot to Claude Code Using MCP
The Model Context Protocol (MCP) is an open standard that lets AI clients call external tools. Running earshot mcp exposes Earshot's functions to a compatible client such as Claude Code.
claude mcp add earshot --scope user -e EARSHOT_THREADS=3 -- /absolute/path/to/earshot/.venv/bin/python -m earshot mcp
On Windows, point to the virtual environment's Windows interpreter, for example ...\earshot.venv\Scripts\python.exe, rather than copying the Unix path. You can also configure it in a .mcp.json file:
{"mcpServers": {"earshot": {"command": "/path/to/earshot/.venv/bin/python",
"args": ["-m", "earshot", "mcp"], "env": {"EARSHOT_THREADS": "3"}}}}
Why the absolute path matters: the client launches the server itself, without your virtual environment activated, so the full path guarantees the right interpreter and dependencies.
Verifying the setup: run Claude mcp list, or check /mcp inside Claude Code. Models load on the first call and then stay in memory, which avoids the CLI's per-call startup cost. On slow machines, start Claude Code with MCP_TIMEOUT=60000.
What the agent can do: describe a generated sound, match it to a target, rank candidates, find trims, and re-run after each change. The repository also ships two agent skills (skills/earshot and skills/real-sound) that you can copy into ~/.claude/skills/. The optional llm_critique tool reads GEMINI_API_KEY from the environment; never write a literal key into a file.
Limitations and Alternatives
Limitations:
Brand-new project. Released October 15, 2026, with version 0.2.0 a day later. Expect fast change and rough edges.
Small evidence base. The author's blind test (8 of 15 correct, essentially chance) is one listener and 15 trials, and the README says so. Independent coverage has also noted that the evidence is small.
Models are imperfect. Loudness, timing, sync, and "is this the kind of sound I asked for" on exposed sounds are reliable. Tags inside busy mixes, music genre, and mood are hints only.
Self-scoring bias. Generated sounds are judged by Earshot's own critic, so human review is still needed.
No judgment of taste, emotion, speech content, or licences.
Startup and memory cost. Local models use RAM, and CLI calls have cold starts.
Licensing of fitted code (see Example E).
See the repository's LIMITS.md and VALIDATION.md for details.
Alternatives: FFmpeg covers loudness and format work, librosa covers spectral analysis, and CLAP or AudioSet models can be called directly. Earshot's value is packaging these into an agent-friendly workflow with plain-language output.
Conclusion
The Earshot open-source toolkit gives coding agents a way to hear what they build. Through its CLI, Python API, and Claude Code MCP integration, it can describe sounds, match them to text, locate events, critique video sound design, generate synthetic audio, and convert recordings into editable code, all on a local CPU. It is young, fast-moving, and open about its limits, but for developers building games, films, or demos with AI agents it is a practical place to start. Install it from the Earshot GitHub repository, run doctor --load, and try an earshot described on a file you care about.
Top comments (0)