Six-Stem Separation Benchmark: BS-RoFormer vs HT-Demucs

We ran the 50 songs of the MUSDB18-HQ test set through the same separation code StemX runs on every upload, and scored the stems against the original multitracks. BS-RoFormer-SW, the engine every split starts on, has a median vocals SDR of 11.98 dB (95% CI 10.41–13.50). Below are all four comparable stems for it and for HT-Demucs 6s, the engine that takes over when the first one fails or runs out of time. After that: the speed of each on the NVIDIA L4 the service runs on, and what this test can't tell you.

This replaces our earlier benchmark of the UVR models we used before, which scored short excerpts with SI-SDR only.

Results: SDR

SDR (signal-to-distortion ratio) is the standard score in the source separation literature, in decibels. Higher is better. Each track gets the median over its one-second frames, and the table gives the median across tracks, with a bootstrap 95% confidence interval for that median.

Results: SI-SDR

SI-SDR ignores overall level, so a stem that is right but quieter than the original isn't penalised. We compute it once over each whole track, both channels together.

Primary against fallback, track by track

Both engines separated the same tracks, so we also compared them track by track. For SDR:

• Vocals: BS-RoFormer-SW scored higher on 50 of 50 tracks. The median per-track difference is +3.24 dB (95% CI +3.01 dB to +3.54 dB).
• Drums: BS-RoFormer-SW scored higher on 48 of 50 tracks. The median per-track difference is +3.41 dB (95% CI +3.01 dB to +3.84 dB).
• Bass: BS-RoFormer-SW scored higher on 48 of 50 tracks. The median per-track difference is +3.02 dB (95% CI +2.32 dB to +3.54 dB).
• Other (guitar + piano + other): BS-RoFormer-SW scored higher on 46 of 50 tracks. The median per-track difference is +3.46 dB (95% CI +2.97 dB to +4.08 dB).

An interval that includes 0 dB means this test can't tell the two engines apart on that stem.

Speed on an NVIDIA L4

Seconds of processing per minute of audio, over all 207.8 minutes of the set. The first column is the model alone. The second adds what the service does after it: a low-pass when the source has no high frequencies, a bass check that also runs HT-Demucs 6s on the mix when BS-RoFormer-SW did the split, and a correction so the six stems add back up to the mix.

Fallbacks during this run: 0 of 50 tracks for BS-RoFormer-SW. A fallback would have put HT-Demucs 6s's output in the BS-RoFormer-SW column, so the count matters.

How we measured

• Dataset: the MUSDB18-HQ test set, 50 full-length songs as uncompressed 44.1 kHz stereo WAV with the original vocals, drums, bass and other stems (Rafii et al., Zenodo, DOI 10.5281/zenodo.3338373). The archive was downloaded inside Google Cloud and its MD5 checked against the one Zenodo publishes. It is for educational and research use, so we publish scores only, never the audio.
• The path under test: each mixture went through the service's own decode, separation and post-processing code with production settings, in a worker image built from the same code and pinned model weights as the live service. BS-RoFormer-SW ran behind its usual time budget, with HT-Demucs 6s as the fallback. HT-Demucs 6s also ran on its own, as it does when it takes over.
• Six stems onto four: StemX returns six stems and MUSDB has four. Vocals, drums and bass map one to one. Guitar, piano and other are summed into MUSDB's "other".
• SDR: BSSEval v4 using museval 0.4.1's own implementation, one-second windows and hops, all four targets scored together, frames where any source is silent skipped. That is the method used in SiSEC 2018 and in most papers since.
• Confidence intervals: 10,000 bootstrap resamples of the tracks, percentile method, fixed seed.
• Hardware: one NVIDIA L4 on Google Cloud Run, run on 2026-10-12.

What this test can't tell you

• Guitar and piano are not scored. MUSDB has no separate guitar or piano stems, so they are judged only as part of "other". A guitar that ends up in the piano stem costs nothing here. We have not found a dataset with guitar and piano stems whose terms let us publish results, so for now we can't give you a figure for either.
• Training overlap is unknown. Neither set of weights comes with a track list we could check against this test set. If some of these songs were in training, the scores are higher than you would get on music the models have never heard.
• 50 songs is a small, narrow set. The songs are professional multitracks and lean towards rock and pop. The confidence intervals show the sampling error for this set; they don't cover genres it lacks.
• SDR is not listening. Stems with the same score can sound different, and a few decibels of SDR can be hard to hear on one track and obvious on another.
• Your files may not be WAV masters. Many uploads are MP3s. Lossy encoding removes detail the models never get back, which this test, run on uncompressed audio, does not measure.

Reproduce it

The raw results, one row per track, engine and stem, are in this CSV, with SDR, SI-SDR, SIR, SAR, ISR and the timings. The run used worker image v0.6.1-bench2 and three commands:

1. scripts/gcp-bench.sh stage downloads MUSDB18-HQ in Cloud Build, checks it and keeps the test set.
2. scripts/gcp-bench.sh musdb v0.6.1-bench2 full separates and scores every track on an L4.
3. python -m stemx.bench summarize turns the CSV into the medians and intervals on this page.

For tempo and key accuracy, see all StemX benchmarks. To hear the engines on your own music, the free preview splits 60 seconds of any song into all six stems.