Neurodyne: Neural Pitch Manipulation with Representation Learning and Cycle-Consistency GAN
Abstract
Pitch manipulation is the process of producers adjusting the pitch of an audio segment to a specific key and intonation, which is essential in music production. Neural-network-based pitch-manipulation systems have been popular in recent years due to their superior synthesis quality compared to classical DSP methods. However, their performance is still limited due to their inaccurate feature disentanglement using source-filter models and the lack of paired in- and out-of-tune training data. This work proposes Neurodyne to address these issues. Specifically, Neurodyne uses adversarial representation learning to learn a pitch-independent latent representation to avoid inaccurate disentanglement and cycle-consistency training to create paired training data implicitly. Experimental results on global-key and template-based pitch manipulation demonstrate the effectiveness of the proposed system, marking improved synthesis quality while maintaining the original singer identity.
What is Neurodyne?
Neurodyne is a neural pitch-manipulation system that has high audio quality and pitch accuracy using adversarial representation learning and cycle-consistency training, as illustrate in the figure above. It avoids artifacts from inaccurate disentanglement by learning a pitch-independent latent representation and creates paired in- and out-of-tune training data implicitly in the training proecss for better generalization. To illustrate the effectiveness of Neurodyne, we conduct evaluation in both global-key and template-based pitch manipulation scenarios.
Global-key Pitch Manipulation
We conduct global-key pitch manipulation to illustrate the robustness of our proposed system. In global-key pitch manipulation, utterances will be globally manipulated by a specific amount of semitones, including: -12, -6, -3, 0, 3, 6, 12.
We select different baselines in global-key pitch manipulation, including BigVGAN, World, TD-PSOLA, DiffPitcher, SiFi-GAN, and PC-NSF.
| Style | Manipulated Keys | GT |
BigVGAN (ICLR 2023) |
WORLD (IEICE TIS 2016) |
TD-PSOLA (Eurospeech) |
DiffPitcher (WASPAA 2023) |
SiFi-GAN (ICASSP 2023) |
PC-NSF (TASLP 2023) |
Neurodyne |
|---|---|---|---|---|---|---|---|---|---|
| Pop | -12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Pop | -6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Pop | -3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Pop | 0 | ||||||||
| Pop | +3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Pop | +6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Pop | +12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | -12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | -6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | -3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | 0 | ||||||||
| Opera | +3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | +6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Opera | +12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | -12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | -6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | -3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | 0 | ||||||||
| Children | +3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | +6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Children | +12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | -12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | -6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | -3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | 0 | ||||||||
| Folk | +3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | +6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Folk | +12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | -12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | -6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | -3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | 0 | ||||||||
| Jazz | +3 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | +6 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / | ||||||||
| Jazz | +12 | / | / | ||||||
| / | / | ||||||||
| / | / | ||||||||
| / | / |
Template-based Pitch Manipulation
We conduct template-based pitch manipulation to evaluate the effectiveness of our proposed model in real-world scenarios. In template-based pitch manipulation, an out-of-tune audio segment will be adjusted based on an in-tune reference.
We select different baselines in template-based pitch manipulation, including World, TD-PSOLA, DiffPitcher, SiFi-GAN, and PC-NSF.
| Out-Of-Tune Source | In-Tune Reference |
WORLD (IEICE TIS 2016) |
TD-PSOLA (Eurospeech) |
DiffPitcher (WASPAA 2023) |
SiFi-GAN (ICASSP 2023) |
PC-NSF (TASLP 2023) |
Neurodyne |
|---|---|---|---|---|---|---|---|
Comparison with Plugins
We compare Neurodyne with SOTA commercial autotune softwares under template-based pitch manipulation to further illustrate its effectivenss. All the out-of-tune samples are mannually manipulated by a music producer.
| Out-Of-Tune Source | In-Tune Reference |
Newtone (FL Studio) |
Melodyne (Celemony) |
Neurodyne |
|---|---|---|---|---|