DEV Community: PartFit 3D

Running Spotify's basic-pitch in the browser for audio-to-MIDI

PartFit 3D — Wed, 30 Sep 2026 06:57:40 +0000

I wanted a quick MIDI sketch of a melody I had sung or played, without uploading the recording to anyone's server. Spotify's basic-pitch model ships a TensorFlow.js build, so the whole thing can run in a browser tab. Here is what it took to make that feel responsive, and where it still falls short.

Keep the model out of the first page load

TensorFlow.js plus the model is too heavy for the initial bundle, and most visitors read the page before they drop a file. So the library is only loaded through a dynamic import():

let modulePromise: Promise<typeof import('@spotify/basic-pitch')> | null = null;

function loadModule() {
  if (!modulePromise) {
    modulePromise = import('@spotify/basic-pitch');
  }
  return modulePromise;
}

The model weights (about 1 MB) are served as static files and cached after the first run. The download starts as soon as someone picks a file, so it overlaps with them choosing which part of the audio to convert.

One trap: basic-pitch pins its own TensorFlow.js version. Adding @tensorflow/tfjs as a separate dependency loads a second copy that re-registers every kernel, so don't.

Split inference from note extraction

The model produces three per-frame matrices: note activations (frames), onsets, and pitch contours. Turning those into notes is a separate, cheap step controlled by two thresholds and a minimum note length.

That split matters for the interface. Running the model over a 30 second clip takes a few seconds; turning its output into notes takes milliseconds. So the raw output is kept, and moving a sensitivity slider only re-runs the second step:

// Runs once per segment
const output = await analyze(audioBuffer);

// Runs on every slider change
const notes = await notesFromOutput(output, {
  onsetThreshold,
  frameThreshold,
  minNoteLengthMs,
  minPitchMidi,
  maxPitchMidi,
});

One detail: outputToNotesPoly mutates the frame and onset arrays when you pass a pitch range, so you need to copy them before each call if you want to reuse the originals.

The model expects mono audio at 22050 Hz. OfflineAudioContext handles the mixdown and resampling without any extra library.

Let people hear the mistakes

A piano roll doesn't tell you much about whether the transcription is right. What helped more was an A/B player: the original audio and a simple WebAudio synth of the detected notes play on one shared playhead, and you switch between them. A wrong octave or a missing note is obvious by ear in a second.

Clean up with plain functions

Model output needs tidying before it is useful in a DAW. Each cleanup step is a pure function over a note array, which keeps them easy to test and combine:

remove notes shorter than a threshold
remove notes much quieter than the rest
merge repeated notes separated by tiny gaps
trim overlaps so a monophonic line stays monophonic
quantize to a grid based on an estimated tempo

Export uses @tonejs/midi to write a standard .mid.

Where it doesn't work

basic-pitch does well on a single line: a sung melody, a guitar riff, a piano part. On a dense full mix it gives you something rough at best. Limiting a run to 15 to 60 seconds also keeps memory in check on phones, where long clips can run out of resources.

The result is MidiDraft, if you want to try it on your own recordings. The examples page includes one clip it deliberately gets wrong, so you can see what failure looks like before trusting it with your own audio.

How Browser-Local Mesh Processing Keeps STL and 3MF Files Private

PartFit 3D — Tue, 04 Aug 2026 08:11:09 +0000

When a maker opens a model-preparation tool, the file may contain more than triangles. A prototype can reveal dimensions, mechanical interfaces, product direction, or customer work. That makes the architecture of the tool—not just its privacy policy—important.

A browser-local workflow keeps the model on the user's device. The page itself is downloaded from a server, but parsing, geometry operations, fit checks, and export happen inside the browser. This is a useful pattern for STL and 3MF utilities because modern browsers already provide the APIs and performance needed for substantial client-side work.

This article walks through the main pieces of that architecture and the trade-offs to expect.

1. Treat the file picker as a local boundary

The browser's File API gives an application access only to files the user explicitly chooses. Reading a selected file with arrayBuffer() does not upload it:

const [file] = fileInput.files;
const bytes = await file.arrayBuffer();

At this point the data is in the tab's memory. It crosses the network only if application code sends it with fetch, a form submission, a WebSocket, or another network API.

That distinction should be visible in both the implementation and the product copy. “Runs in your browser” is not automatically the same as “never uploads your model.” A trustworthy design avoids model-upload endpoints, does not place raw geometry in analytics events, and makes network behavior easy to inspect.

2. Parse STL and 3MF without assuming they are equivalent

STL is comparatively simple: it represents a triangle surface, usually in binary form, with no standard unit metadata. A parser typically produces a flat list of vertex positions and, optionally, normals.

3MF is a ZIP-based package containing XML and related assets. It can preserve units, object structure, transforms, and other information that STL loses. A browser-local 3MF pipeline therefore needs to:

read the package in memory;
locate the model relationships and XML parts;
apply declared units and object transforms;
convert referenced meshes into a consistent internal representation.

The safest internal representation is explicit about units and coordinate systems. Converting everything to millimeters early prevents subtle fit-check errors later.

For large files, parsing and mesh construction should run in a Web Worker. That keeps the main thread responsive while the worker turns bytes into typed arrays. Transferable ArrayBuffer objects can move data between the worker and UI without expensive copies.

3. Fit checking is a geometry problem, not a slicer

A build-volume check starts with the model's axis-aligned bounding box. For every vertex, track the minimum and maximum coordinate on each axis:

sizeX = maxX - minX;
sizeY = maxY - minY;
sizeZ = maxZ - minZ;

Those dimensions can be compared with a printer's usable width, depth, and height. A configurable safety margin is worth including because nominal build volume is not always the practical printable volume.

Orientation matters. Rotating the model changes its axis-aligned bounds, sometimes enough to make it fit without a cut. The UI should therefore recompute dimensions after each transform and show the result clearly—ideally as a per-part Fit or Too Large status.

This remains model preparation, not slicing. The browser tool can orient geometry, split it, inspect the mesh, and export parts. A slicer such as Bambu Studio, Cura, or PrusaSlicer still turns those parts into toolpaths and G-code.

4. A plane cut needs more than deleting triangles

Splitting a triangle mesh by a plane sounds simple until a triangle crosses the plane. A robust cut pipeline generally does the following:

classify each triangle vertex by signed distance to the plane;
keep triangles entirely on either side;
split crossing triangles at their plane intersections;
collect the resulting boundary segments;
join segments into closed loops;
triangulate those loops to cap the two open cut surfaces.

Floating-point tolerance is critical. Vertices that are mathematically on the plane may land slightly above or below it in binary arithmetic. Using an epsilon consistently avoids cracks, duplicated slivers, and unstable classifications.

Repeated cuts add another requirement: every resulting mesh needs its own identity, transform, dimensions, and validation state. An auditable parts list is much easier to reason about than a destructive “latest result only” workflow.

5. Validate topology before export

A file can look fine in a preview while still causing trouble downstream. Useful browser-side checks include:

degenerate triangles with near-zero area;
boundary edges that appear only once;
non-manifold edges shared by more than two faces;
disconnected components;
inconsistent winding or normals;
dimensions that still exceed the selected build volume.

These checks do not guarantee that every slicer will accept every model, but they expose common failure modes before the user downloads several parts.

Export should preserve the same units and geometry the fit checker evaluated. STL export is widely compatible but loses metadata. 3MF can retain richer structure and is often a better handoff format when the downstream slicer supports it.

6. Make privacy testable

Privacy claims become more credible when users can verify them. A browser-local model tool can support that in several ways:

keep core processing usable without an account;
avoid model-related network requests;
document which telemetry, if any, is collected;
keep filenames and model-derived measurements out of analytics;
clear in-memory state when the user resets the workspace;
use a restrictive Content Security Policy where practical.

The browser's developer tools should show the page assets loading, but no request containing the selected model. That is a concrete, testable promise.

7. Know the limits of the approach

Browser-local processing is not free of constraints. Memory is finite, mobile devices vary widely, and a complex cut can create many new triangles. Large files benefit from workers, typed arrays, progress reporting, cancellation, and early estimates of memory use.

The browser is also not a full CAD kernel. Plane cuts and mesh checks are realistic; exact solid modeling, feature-history editing, and repair of every pathological mesh are different problems.

The best architecture is honest about that boundary: do focused model preparation locally, then hand the results to a dedicated slicer.

Closing thought

Local-first design is especially well matched to 3D-printing files. It reduces upload latency, keeps prototypes under the user's control, and makes privacy a property of the system rather than only a policy statement.

We used these principles while building PartFit 3D, a browser-based STL and 3MF splitter with build-volume checks, repeated plane cuts, per-part fit status, mesh checks, and slicer-ready export. The broader pattern applies to many file-processing tools: if the browser can do the work safely and responsively, the user's data may not need to leave the device at all.