Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis

Abstract

Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTS, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in this demo page.

Target Text

Prompt Example

VALL-E

Fine-tune

Ours

It is true that the horses are here, but the Hurons are gone; let us, then, hunt for the path by which they parted.

If I must say it, mother, I want to go away, and get out of this dead level.

The first mill plant was placed in the woollen factory of james Harrison at Newburgh, New York, about september fifteenth eighteen eighty one.

There their sad condition evoked for a time general commiseration.

Now in this of Burne Jones, the landscape is clearly full of light everywhere, color or glass light: that is, the outline is prepared for modification of color only.

When they entered the stage box on the left the first act was well under way, the scene being the interior of a cabin in the south of Ireland.

Target Text

Prompt Example

VALL-E (20s)

Fine-tune

Ours

It is true that the horses are here, but the Hurons are gone; let us, then, hunt for the path by which they parted.

If I must say it, mother, I want to go away, and get out of this dead level.

The first mill plant was placed in the woollen factory of james Harrison at Newburgh, New York, about september fifteenth eighteen eighty one.

There their sad condition evoked for a time general commiseration.

Now in this of Burne Jones, the landscape is clearly full of light everywhere, color or glass light: that is, the outline is prepared for modification of color only.

When they entered the stage box on the left the first act was well under way, the scene being the interior of a cabin in the south of Ireland.

Target Text

Prompt Example

Fine-tune

Ours

It is true that the horses are here, but the Hurons are gone; let us, then, hunt for the path by which they parted.

If I must say it, mother, I want to go away, and get out of this dead level.

The first mill plant was placed in the woollen factory of james Harrison at Newburgh, New York, about september fifteenth eighteen eighty one.

There their sad condition evoked for a time general commiseration.

Now in this of Burne Jones, the landscape is clearly full of light everywhere, color or glass light: that is, the outline is prepared for modification of color only.

When they entered the stage box on the left the first act was well under way, the scene being the interior of a cabin in the south of Ireland.

Target Text

Prompt Example

VALL-E

Fine-tune

Ours

Dig anywhere in Pompeii and you'll uncover an ancient treasure, a snapshot of a lost Roman world. Most of those who lived in the city probably escaped the eruption of Mount Vesuvius, but what they left behind is still being unearthed 2,000 years later. The new discoveries from this excavation include a huge oven, capable of producing 100 loaves a day, a shrine to make offerings to the gods that's decorated with snakes, and a bed reduced to ashes in a fire. There are skeletons too – three individuals crushed under falling masonry. The archaeologists plan to erect a walkway, so tourists can see what's going on. It will prove popular. This is the dig that recently revealed a fresco, depicting a bread meal that looked like it might be a forerunner of pizza.

(Failed)

Target Text	Name	Prompt	Ours
Let's go drink until we can't feel feelings anymore.	Sponge Bob
Uh, it's not like the internet to go crazy about something small and stupid.	Peter Griffin
Then I would never talk to that person about boa constrictors, or primeval forests, or stars. I would bring myself down to his level.	Rick
In what a disgraceful light might it not strike so vain a man!	Morty

Target Text

Name

Prompt

Ours

Let's go drink until we can't feel feelings anymore.

Sponge Bob