This is a simplified guide to an AI model called Beat_this maintained by Xavriley. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
beat_this is a beat and downbeat tracking model from the ISMIR 2024 paper "Beat This! Accurate Beat Tracking Without DBN Postprocessing" by xavriley and collaborators at CPJKU. The model detects precise beat positions and downbeat boundaries in audio files without relying on Dynamic Bayesian Network postprocessing, achieving state-of-the-art F1 scores while maintaining generality across diverse music genres. The architecture alternates convolutions with transformers operating either over frequency or time dimensions, and is trained on multiple datasets including solo instruments, pieces with time signature changes, and classical music with high tempo variations. The main model (final0, final1, final2) weighs approximately 78 MB each, with a smaller variant available at 8.1 MB. The most critical detail before using it: the model achieves good results specifically because it avoids meter and tempo constraints that traditional systems impose, but this means it can still fail on difficult and underrepresented genres and performs worse on continuity metrics compared to methods using postprocessing.
Best use cases
Music information retrieval and analysis workflows. If you build music analysis software that needs to segment tracks into beat-aligned sections for tempo detection, structural analysis, or synchronization with other modalities, beat_this provides clean beat and downbeat annotations without requiring external postprocessing pipelines. The model outputs precise timestamps suitable for downstream music information retrieval tasks like onset detection or harmonic analysis.
Rhythm-aware music production tools. For digital audio workstations, beat detection plugins, or metronome applications, this model provides frame-level accuracy suitable for real-time audio alignment and grid snapping. The two input parameters—constant tempo assumption and DBN postprocessing toggle—let you trade accuracy for continuity depending on whether your source material has stable rhythm or requires smoothing.
Music transcription and notation systems. Beat and downbeat detection forms a foundational layer for automatic music transcription pipelines. Feeding the model's output into tempo curve estimation and time signature inference produces richer musical analysis than beat detection alone, particularly useful for music notation software or academic music analysis tools.
Dataset annotation and validation. If you maintain collections of annotated music data, this model can automatically generate beat annotations for new tracks, which human annotators can then correct. The availability of multiple trained seeds (final0, final1, final2) allows you to ensemble predictions for higher confidence annotation.
Classical and complex music analysis. The model explicitly handles tempo variations, time signature changes, and solo instruments—scenarios where simpler beat trackers fail. If your music corpus includes Bach, contemporary classical, or experimental music, this model outperforms systems trained only on pop or dance music datasets.
Limitations
The model trains on multiple public datasets but explicitly excludes GTZAN for evaluation fairness; if you run inference on GTZAN files, results will be misleadingly high. The model struggles with difficult and underrepresented genres—the paper acknowledges this limitation directly. Performance on continuity metrics (how smoothly beat timing progresses) is worse than DBN-postprocessed systems, making it less suitable for applications requiring strictly monotonic beat sequences without jitter. The Replicate API defaults to using DBN postprocessing (use_dbn: true) and constant tempo assumption (constant_tempo: true), which partially reverses the paper's main contribution of avoiding DBN; disabling use_dbn requires setting it to false explicitly. The model uses PyTorch 2.0+, ffmpeg for non-WAV audio, and optional CUDA support—CPU inference is slower and lacks float16 optimization. The input schema accepts only audio as a URI, meaning you must upload files to a publicly accessible URL rather than passing raw bytes. Output format from Replicate's API is undocumented (schema shows title: Output with no further specification), so the actual return structure requires inspection during your first API call. The model file size (78 MB for main variants) requires downloading checkpoints on first use, which takes time on bandwidth-limited systems.
How it compares
demixing by jimothyjohn separates instruments and vocals from audio—a different task than beat detection. Choose beat_this if you need tempo and rhythm information; choose demixing if you need to isolate individual instruments before processing.
all-in-one-music-structure-analysis by cwalo performs comprehensive analysis including BPM, downbeats, and structure in one call. Use this alternative if you need multiple outputs (structure, demuxing, BPM) simultaneously; choose beat_this if you want the most accurate beat and downbeat detection specifically, as it focuses on that single task and achieves state-of-the-art F1 scores.
music by elevenlabs and music-2.6 by minimax are generative models that compose music from prompts—the opposite of beat_this, which analyzes existing audio. These are irrelevant if your goal is beat tracking, but useful if you need to create rhythmically structured content.
musicgen by charlesmccarthy generates music from text or composition plans, again a generative task rather than analysis. Not comparable to beat_this unless you plan to analyze generated music afterward.
Technical specifications
The model uses a hybrid convolutional-transformer architecture that alternates between frequency-domain and time-domain transformer blocks. Training uses a custom loss function tolerant to small time shifts in beat annotations (typically ±70 ms), addressing a core weakness in prior beat tracking systems where perfect frame alignment was unrealistic. The model trains on spectrograms (22 kHz sample rate, monophonic) preprocessed from raw audio using pedalboard. Multiple model variants exist: the main final0/1/2 are trained on all datasets except GTZAN with three random seeds; small0/1/2 offer a 10x smaller model (8.1 MB) with slight accuracy drops; single_final0/1/2 use a single train/validation split; and fold0-7 provide 8-fold cross-validation variants for fair evaluation on datasets used in training. The model supports CPU and GPU inference (CUDA recommended), with optional float16 precision for recent GPUs. Inference can be distributed across multiple GPUs using the --touch-first and --skip-existing command-line flags. The Replicate deployment defaults to using DBN postprocessing from madmom, which contradicts the paper's main claim about avoiding DBN—this requires explicit configuration to disable. Input audio supports any format readable by torchaudio with ffmpeg backend; output format is a .beats TSV file compatible with Sonic Visualizer.
- Architecture: Alternating convolutional and partial transformer blocks (frequency and time transformers)
- Model sizes: 78 MB (main/final), 8.1 MB (small), 78 MB (single/fold variants)
- Training data: Multiple datasets including solo instruments, time signature changes, classical music; excludes GTZAN
- Audio preprocessing: 22 kHz monophonic spectrograms via pedalboard
- Loss function: Shift-tolerant loss (~70 ms tolerance for beat annotation timing)
- Inference compute: GPU (CUDA) preferred; CPU fallback available; float16 option on recent GPUs
-
Output format: TSV
.beatsfile with beat and downbeat timestamps - Dependencies: PyTorch 2.0+, tqdm, einops, soxr, rotary-embedding-torch; ffmpeg for non-WAV audio; optional madmom for DBN
- License: Check the repository LICENSE file (linked in metadata)
Model inputs and outputs
Inputs
- audio (string, URI, required): URL pointing to an audio file in WAV, MP3, FLAC, or other formats supported by torchaudio with ffmpeg
-
constant_tempo (boolean, default:
true): Assume the source material has constant tempo; usetruefor pop/electronic music,falsefor classical or live recordings with tempo drift -
use_dbn (boolean, default:
true): Apply Dynamic Bayesian Network postprocessing for temporal smoothing; set tofalseto use the raw model output without DBN constraints
Outputs
-
Output (object): The schema indicates only
title: Outputwith no documented structure; returns beat and downbeat annotations (likely as a.beatsfile or JSON array of timestamps based on the command-line tool behavior)
Getting started
import replicate
# Initialize client (assumes REPLICATE_API_TOKEN environment variable)
client = replicate.Replicate()
# Run beat detection on an audio file
output = client.run(
"xavriley/beat_this:26142842c6dc94673820f0a9762214fa7109015d966031d8e6137bef4fd14323",
input={
"audio": "https://example.com/path/to/your/audio.mp3",
"constant_tempo": True,
"use_dbn": True
}
)
print(output)
For local Python use without Replicate, install the package and use directly:
from beat_this.inference import File2Beats
# Load model (downloads automatically on first run)
file2beats = File2Beats(checkpoint_path="final0", device="cuda", dbn=False)
# Get beat and downbeat positions
beats, downbeats = file2beats("path/to/audio.mp3")
# Save to Sonic Visualizer format
from beat_this.utils import save_beat_tsv
save_beat_tsv(beats, downbeats, "output.beats")
Frequently asked questions
Q: Should I disable use_dbn when running on Replicate?
A: It depends on your use case. The paper's main contribution is achieving state-of-the-art results without DBN, so set use_dbn: false if you want the raw model predictions. However, DBN provides smoother, more musically consistent beat sequences at the cost of introducing meter constraints; use use_dbn: true if you need temporal continuity or notice jittery output from the raw model.
Q: What audio formats does beat_this accept?
A: The model accepts WAV, MP3, FLAC, and any format that ffmpeg can decode, provided ffmpeg is installed and torchaudio is configured to use it as a backend. For Replicate, pass audio as a URI string rather than raw bytes.
Q: How accurate is this model compared to manual annotation?
A: The paper reports state-of-the-art F1 scores on the GTZAN test set when using the final0/1/2 models. However, accuracy degrades on difficult and underrepresented genres, and it performs worse on continuity metrics (how smoothly beat timing changes) compared to DBN-postprocessed systems. For fair evaluation on datasets used in training, use the cross-validation variants (fold0-7 or single_final variants).
Q: Can I use this model commercially?
A: Check the LICENSE file in the GitHub repository for the specific open-source license terms. The code is publicly available and the paper is published, but license restrictions may apply depending on your use case.
Q: Which model variant should I use: final, small, or fold?
A: Use final0 (the default) for general-purpose beat tracking on unseen music. Use small0/1/2 if model size or inference speed matters more than accuracy—they sacrifice some F1 score for 10x smaller file size. Use fold* or single_final* only for fair research evaluation on datasets that contributed to training data; otherwise results will be biased upward.
Q: How long does inference take on CPU vs GPU?
A: The README does not specify exact inference times, but indicates GPU is strongly recommended. The command-line tool defaults to GPU with CPU fallback; float16 mode on recent GPUs improves speed. For batch processing, distribute across multiple GPUs with separate process instances.
Q: What happens if the audio has tempo changes or unusual time signatures?
A: The model is specifically designed to handle tempo variations and time signature changes, as demonstrated on classical and contemporary music in the training data. However, disable constant_tempo: true if you suspect tempo drift, and be aware that very extreme or rapid tempo changes may still cause failures.
Q: Does the model work for non-Western music or underrepresented genres?
A: The paper explicitly acknowledges that the model "can still fail, especially for difficult and underrepresented genres." The training datasets focus on Western music traditions. If you work with non-Western, experimental, or niche genres, test on a representative sample before production use.
Top comments (0)