[ad_1]
Beforehand, we offered the 1,000 languages initiative and the Common Speech Mannequin with the purpose of constructing speech and language applied sciences obtainable to billions of customers around the globe. A part of this dedication entails growing high-quality speech synthesis applied sciences, which construct upon initiatives resembling VDTTS and AudioLM, for customers that talk many alternative languages.
 |
After growing a brand new mannequin, one should consider whether or not the speech it generates is correct and pure: the content material have to be related to the duty, the pronunciation right, the tone applicable, and there needs to be no acoustic artifacts resembling cracks or signal-correlated noise. Such analysis is a significant bottleneck within the growth of multilingual speech programs.
The preferred methodology to guage the standard of speech synthesis fashions is human analysis: a text-to-speech (TTS) engineer produces a couple of thousand utterances from the newest mannequin, sends them for human analysis, and receives outcomes a couple of days later. This analysis part usually entails listening exams, throughout which dozens of annotators take heed to the utterances one after the opposite to find out how pure they sound. Whereas people are nonetheless unbeaten at detecting whether or not a chunk of textual content sounds pure, this course of could be impractical — particularly within the early levels of analysis initiatives, when engineers want speedy suggestions to check and restrategize their method. Human analysis is dear, time consuming, and could also be restricted by the provision of raters for the languages of curiosity.
One other barrier to progress is that totally different initiatives and establishments usually use numerous rankings, platforms and protocols, which makes apples-to-apples comparisons unattainable. On this regard, speech synthesis applied sciences lag behind textual content technology, the place researchers have lengthy complemented human analysis with computerized metrics resembling BLEU or, extra not too long ago, BLEURT.
In “SQuId: Measuring Speech Naturalness in Many Languages“, to be offered at ICASSP 2023, we introduce SQuId (Speech High quality Identification), a 600M parameter regression mannequin that describes to what extent a chunk of speech sounds pure. SQuId relies on mSLAM (a pre-trained speech-text mannequin developed by Google), fine-tuned on over one million high quality rankings throughout 42 languages and examined in 65. We exhibit how SQuId can be utilized to enrich human rankings for analysis of many languages. That is the most important printed effort of this kind so far.
Evaluating TTS with SQuId
The primary speculation behind SQuId is that coaching a regression mannequin on beforehand collected rankings can present us with a low-cost methodology for assessing the standard of a TTS mannequin. The mannequin can due to this fact be a worthwhile addition to a TTS researcher’s analysis toolbox, offering a near-instant, albeit much less correct various to human analysis.
SQuId takes an utterance as enter and an optionally available locale tag (i.e., a localized variant of a language, resembling “Brazilian Portuguese” or “British English”). It returns a rating between 1 and 5 that signifies how pure the waveform sounds, with a better worth indicating a extra pure waveform.
Internally, the mannequin contains three parts: (1) an encoder, (2) a pooling / regression layer, and (3) a completely linked layer. First, the encoder takes a spectrogram as enter and embeds it right into a smaller 2D matrix that accommodates 3,200 vectors of dimension 1,024, the place every vector encodes a time step. The pooling / regression layer aggregates the vectors, appends the locale tag, and feeds the outcome into a completely linked layer that returns a rating. Lastly, we apply application-specific post-processing that rescales or normalizes the rating so it’s inside the [1, 5] vary, which is frequent for naturalness human rankings. We practice the entire mannequin end-to-end with a regression loss.
The encoder is by far the most important and most essential piece of the mannequin. We used mSLAM, a pre-existing 600M-parameter Conformer pre-trained on each speech (51 languages) and textual content (101 languages).
 |
| The SQuId mannequin. |
To coach and consider the mannequin, we created the SQuId corpus: a group of 1.9 million rated utterances throughout 66 languages, collected for over 2,000 analysis and product TTS initiatives. The SQuId corpus covers a various array of programs, together with concatenative and neural fashions, for a broad vary of use instances, resembling driving instructions and digital assistants. Handbook inspection reveals that SQuId is uncovered to an unlimited vary of of TTS errors, resembling acoustic artifacts (e.g., cracks and pops), incorrect prosody (e.g., questions with out rising intonations in English), textual content normalization errors (e.g., verbalizing “7/7” as “seven divided by seven” moderately than “July seventh”), or pronunciation errors (e.g., verbalizing “robust” as “toe”).
A standard difficulty that arises when coaching multilingual programs is that the coaching information might not be uniformly obtainable for all of the languages of curiosity. SQuId was no exception. The next determine illustrates the dimensions of the corpus for every locale. We see that the distribution is essentially dominated by US English.
 |
| Locale distribution within the SQuId dataset. |
How can we offer good efficiency for all languages when there are such variations? Impressed by earlier work on machine translation, in addition to previous work from the speech literature, we determined to coach one mannequin for all languages, moderately than utilizing separate fashions for every language. The speculation is that if the mannequin is giant sufficient, then cross-locale switch can happen: the mannequin’s accuracy on every locale improves because of collectively coaching on the others. As our experiments present, cross-locale proves to be a strong driver of efficiency.
Experimental outcomes
To know SQuId’s general efficiency, we evaluate it to a customized Massive-SSL-MOS mannequin (described within the paper), a aggressive baseline impressed by MOS-SSL, a state-of-the-art TTS analysis system. Massive-SSL-MOS relies on w2v-BERT and was educated on the VoiceMOS’22 Problem dataset, the most well-liked dataset on the time of analysis. We experimented with a number of variants of the mannequin, and located that SQuId is as much as 50.0% extra correct.
 |
| SQuId versus state-of-the-art baselines. We measure settlement with human rankings utilizing the Kendall Tau, the place a better worth represents higher accuracy. |
To know the impression of cross-locale switch, we run a collection of ablation research. We differ the quantity of locales launched within the coaching set and measure the impact on SQuId’s accuracy. In English, which is already over-represented within the dataset, the impact of including locales is negligible.
 |
| SQuId’s efficiency on US English, utilizing 1, 8, and 42 locales throughout fine-tuning. |
Nevertheless, cross-locale switch is way more efficient for many different locales:
 |
| SQuId’s efficiency on 4 chosen locales (Korean, French, Thai, and Tamil), utilizing 1, 8, and 42 locales throughout fine-tuning. For every locale, we additionally present the coaching set dimension. |
To push switch to its restrict, we held 24 locales out throughout coaching and used them for testing solely. Thus, we measure to what extent SQuId can take care of languages that it has by no means seen earlier than. The plot beneath reveals that though the impact is just not uniform, cross-locale switch works.
 |
| SQuId’s efficiency on 4 “zero-shot” locales; utilizing 1, 8, and 42 locales throughout fine-tuning. |
When does cross-locale function, and the way? We current many extra ablations within the paper, and present that whereas language similarity performs a task (e.g., coaching on Brazilian Portuguese helps European Portuguese) it’s surprisingly removed from being the one issue that issues.
Conclusion and future work
We introduce SQuId, a 600M parameter regression mannequin that leverages the SQuId dataset and cross-locale studying to guage speech high quality and describe how pure it sounds. We exhibit that SQuId can complement human raters within the analysis of many languages. Future work contains accuracy enhancements, increasing the vary of languages lined, and tackling new error sorts.
Acknowledgements
The writer of this put up is now a part of Google DeepMind. Many because of all authors of the paper: Ankur Bapna, Joshua Camp, Diana Mackinnon, Ankur P. Parikh, and Jason Riesa.
[ad_2]