{"id":604,"date":"2026-11-06T13:29:14","date_gmt":"2026-11-06T21:29:14","guid":{"rendered":"https:\/\/jasonsla.de\/?p=604"},"modified":"2026-11-06T13:29:15","modified_gmt":"2026-11-06T21:29:15","slug":"voice-swap-pipeline-seed-vc","status":"publish","type":"post","link":"https:\/\/jasonsla.de\/2026\/11\/06\/voice-swap-pipeline-seed-vc\/","title":{"rendered":"Then I Taught It to Sing as Someone Else: Voice-Swap on the Home GPU Server"},"content":{"rendered":"\n

In the previous post I wrote about teaching compute01<\/code> \u2014 my home GPU server \u2014 to write full songs from text prompts with ACE-Step. Text goes in, a finished song with vocals comes out. It is a genuinely useful little studio, but it has one limitation that starts to itch after the first dozen songs: the singer never changes. ACE-Step’s vocalist is whatever the checkpoint is, every time, in every key, on every track.<\/p>\n\n\n\n

The fix I wanted was not another generation run with a different prompt. I wanted an audio pipeline: take any song the server has already rendered, pull the vocal out, re-voice it with a reference singer, and remix it over the original instrumental. No re-generation, no re-rolling the dice on a take. And I wanted it to stay entirely on the box \u2014 no cloud API, no uploading anyone’s voice anywhere.<\/p>\n\n\n\n

That pipeline is now built, and it works. The server can take a song it wrote, separate the vocal, shift it into the reference voice’s range, convert the timbre to a reference voice with Seed-VC, loudness-match the result, and mix it back over the instrumental \u2014 all on two P40 GPUs, all local.<\/p>\n\n\n\n

The Pipeline<\/h2>\n\n\n\n
\"Flowchart
The voice-swap pipeline: split, pitch, convert, loudness-match, remix.<\/figcaption><\/figure>\n\n\n\n

The whole thing is one command:<\/p>\n\n\n\n

~\/seedvc\/venv\/bin\/python ~\/seedvc\/voiceswap.py <song> --voice <name> \\\n        [--pitch-shift N] [--vocal-gain dB]<\/code><\/pre>\n\n\n\n
The target voice lives as a single reference clip in ~\/voices\/\/ref.wav<\/code> \u2014 a short clip of clean audio of whoever you want to hear sing (Seed-VC takes 1 to 30 seconds, and the pipeline uses up to 25). --pitch-shift<\/code> defaults to auto, and --vocal-gain<\/code> is there for when the converted vocal needs a nudge in the mix.<\/p>\n\n\n\n
There are five stages, and two of them are the actual engineering:<\/p>\n\n\n\n
1. Stem split.<\/strong> htdemucs<\/code> separates the song into vocals and instrumental. This is the only stage I did not have to think hard about \u2014 htdemucs is boring and reliable, and on a GPU it does a full song in a few seconds.<\/p>\n\n\n\n
2. Auto-pitch correction.<\/strong> This is the first piece of real engineering. A tenor melody re-voiced by a baritone sounds strained, so the vocal is first moved into the reference voice’s range. The pipeline estimates the median fundamental frequency (f0) of the source vocal with librosa’s PYIN algorithm, does the same for the reference, and shifts the source by the nearest whole number of semitones between the two, capped at \u00b112. A --pitch-shift 0<\/code> flag skips this if you want the original key preserved.<\/p>\n\n\n\n
3. Seed-VC conversion.<\/strong> Seed-VC’s f0-conditioned 44.1 kHz singing model re-voices the stem with the reference’s timbre while keeping the source’s melody and lyrics \u2014 it preserves the pitch contour, so the song stays the song. The conversion runs at 40 steps (30\u201350 is the recommended window) with a cfg of 0.7.<\/p>\n\n\n\n
4. EBU R128 loudness match.<\/strong> The second piece of real engineering, and the one people notice least until it is missing. A converted vocal that is a few LUFS louder than the rest of the track does not sit in the mix \u2014 it floats on top of it, like a karaoke track recorded by a different engineer. The pipeline measures integrated loudness of the source vocal (EBU R128) and aligns the converted vocal to it before anything is mixed.<\/p>\n\n\n\n
5. Remix.<\/strong> ffmpeg amix<\/code> blends the converted vocal over the original instrumental. No time stretching, no re-alignment \u2014 the conversion preserves length to within a few tens of milliseconds, so the remix is just a mix, and the result is exported as a 320 kbit\/s CBR MP3.<\/p>\n\n\n\n
Before and After<\/h2>\n\n\n\n
The same two songs from the previous post, run through the pipeline with my voice as the reference. Play the original first, then the swap: same lyrics, same melody, same instrumental, different singer. Both use a fixed \u221212 semitone shift (exactly one octave), so the vocal stays in the song’s key.<\/p>\n\n\n\n
“Compute Zero One” (synth-pop)<\/h3>\n\n\n\n