Project name

Text To Speech

Project members

Dong Wang, Zhiyong Zhang

Introduction

We are interested in a flexible syntehsis based on neural model . The basic idea is that since the neural model can be traind with multiple conditions, we can treat speaker and emotion as the conditional factors. We use the speaker vector and emotion vector as addiiontal input to the model, and then train a single model that can produce sound of different speakers and different emotions.

In the following experiments, we use a simple DNN architecture to implement the training. The vocoder is WORD.

Experiments

Mono-speaker

The first step is mono-speaker systems. We trained three systems: a female, a male and a child, each with a single network. The performance is like the ofllowing.

Synthesis text:好雨知时节，当春乃发声，随风潜入夜，润物细无声

Female[1]

Male[2]

Child[3]

Multi-speaker

Now we combine all the data from male, female and child to train a single model.

Without Speaker-vector

The first experiment is that the data are blindly combined, without any indicator of speakers.

Female & Male[4]

Female & Child[5]

Male & Child[6]

With Speaker-vector

Now we use speaker vector as an indicator of the speaker trait.

Specific person

Firstly, use the speaker fector to specifiy a particular person:

Female[7]

Male[8]

Interpolate of different person

Now let's produce interpolated voice by interpolating two speakers: female and amle.

Female & Male with different ratio

(1) 0.0:1.0[9]

(2) 0.1:0.9[10]

(3) 0.2:0.8[11]

(4) 0.3:0.7[12]

(5) 0.4:0.6[13]

(6) 0.5:0.5[14]

(7) 0.6:0.4[15]

(8) 0.7:0.3[16]

(9) 0.8:0.2[17]

(10) 0.9:0.1[18]

(11) 1.0:0.0[19]

Mono-speaker Multi-Emotion

Using emotion vectors can specify which emotio to use, and the emotion can be also interpolated.

Specific emotion

Neutral emotion [20]
Happy emotion [21]
Sorrow emotion [22]
Angry emotion [23]

Interpolation emotion

Angry & neutral with different ratio

(1) 0.0:1.0 [24]
(2) 0.1:0.9 [25]
(3) 0.2:0.8 [26]
(4) 0.3:0.7 [27]
(5) 0.4:0.6 [28]
(6) 0.5:0.5 [29]
(7) 0.6:0.4 [30]
(8) 0.7:0.3 [31]
(9) 0.8:0.2 [32]
(10) 0.9:0.1 [33]
(11) 1.0:0.0 [34]

Multi-speaker Multi-emotion

Finally, all the data (different speakers and different emotions) are combined together. Note that only the child voice has different emotions of training data. We hope that this emotion can be learned so that we can generate voice of other speakers with emotion, although they do not have any training data with emtoions.

Female

angry [35]
happy [36]
neutral [37]
sorrow [38]

Male

angry [39]
happy [40]
neutral [41]
sorrow [42]

MLPG Comparation

We compare the different implementation of mlpg AS merlin does(mlpg.py and fast_mlpg.py). There are three implementations:

mlpg: As mlpg.py while compute all the dimension of delta features(including lf0/bap/mgc, the dim is 1/5/60 respectively)
mlpg-lossy: Wrong implementation of mlpg.py by only considering the first dimension of global co-variance.
fast-mlpg: As fast_mlpg.py in merlin.

Computation Time(Estimation)

   alg.    |    lf0(dim=1)    |    bap(dim=5)   |   mgc(dim=60) 
mlpg-lossy |      100000      |     130000      |   160000    
   mlpg    |      130000      |     500000      |   6200000    
fast-mlpg  |      60000       |     300000      |   3580000
 avg-rate  |      1:1.3:0.6   |     1:4:2+      |   1:40:20+

Synthesis waves

text

5='好雨知时节，当春乃发声，随风潜入夜，润物细无声。'
13='大熊猫最大的愿望就是拍一张自己的照片。'

no-mlpg

5 [43]
13 :*5 [44]

mlpg-lossy

5 [45]
13 :*5 [46]

mlpg

5 [47]
13 :*5 [48]

fast-mlpg

5 [49]
13 :*5 [50]

TTS-project-synthesis

目录