Deep Speech Factorization-2

Introduction

Speech signals involve complex factors, each contributing in an unknown and secrete way. Recent developed deep learning methods have built up some interesting tools for discovering these latent factors. These tools include various unsupervised models such as VAE, GAN, supervised learning methods such as multi-task learning, knowledge distillation, etc. These tools allow us to decipher secretes of speech signal, based on big data, rather than hypothesis.

These will lead to an unprecedented breakthrough in speech information processing. Some of the signals for this breakthrough includes:

• In speaker recognition, speaker factors can be learned within a very small speech segment.
• In speech synthesis, speaking styles can be learned as latent variables and discovered in an unsupervised way, and speaker factors can be used to change the speaker trait.
• In speech recognition, learning multiple tasks in a collaborative way has shown to be successful.

In previous studies (Phase 1), we have found that using cascade learning, speech signals can be factorized into content, speaker and emotion at the frame level. In this Phase 2, we will try to answer the following questions:

• Can we factorize speech signals in an unsupervised way?
• How supervised and unsupervised factorizations are integrated?
• How to deal with language discrepancy in factorization?
• How to discover optimal factorization architectures?

People

Dong Wang, Yunqi Cai, Haoran Sun, Zhiyuan Tang, Lantian Li

Research direction

Basic research

• Collaborative learning with AutoML
• VAE/dVAE factorization
• Supervised VAE for factorization
• ASR + TTS cycle training

Applied reseach

• Pretraining for ASR, SID, EMD (BERT in speech)
• Low-resource ASR, TTS
• Signal compression, cleaning up, etc.

Related publications

1. Yang Zhang and Lantian Li and Dong Wang, "VAE-based regularization for deep speaker embedding", Interspeech 2019
2. Lantian Li, Yixiang Chen, Ying Shi, Zhiyuan Tang, and Dong Wang, “Deep speaker feature learning for text-independent speaker verification,”, Interspeech 2017.
3. Lantian Li, Dong Wang, Yixiang Chen, Ying Shing, Zhiyuan Tang, http://wangd.cslt.org/public/pdf/spkfact.pdf
4. Lantian Li, Zhiyuan Tang, Dong Wang, FULL-INFO TRAINING FOR DEEP SPEAKER FEATURE LEARNING, http://wangd.cslt.org/public/pdf/mlspk.pdf
5. Zhiyuan Thang, Lantian Li, Dong Wang, Ravi Vipperla "Collaborative Joint Training with Multi-task Recurrent Model for Speech and Speaker Recognition", IEEE Trans. on Audio, Speech and Language Processing, vol. 25, no.3, March 2017.
6. Dong Wang,Lantian Li,Ying Shi,Yixiang Chen,Zhiyuan Tang., "Deep Factorization for Speech Signal", https://arxiv.org/abs/1706.01777

Talks

• Back to Matrix (Dong Wang, 2019/11/02)
• Nolinear_Flow (Yunqi Cai, 2019/11/02)
• Flow_introduction (Yunqi Cai, 2019/11/04)
• Previous_experiments (Haoran Sun, 2019/11/02)

Further reading

Old

1. Barlow, Unsupervised learning, Neural computation, 1989. [1]

1. Geoffrey E. Hinton， Terrence Joseph Sejnowski, "Unsupervised Learning: Foundations of Neural Computation", MIT Press, 1999[2]Associated review from DL Wang [3]

1. Anthony J. Bell and Terrence I. Sejnowski, "An Information-Maximization Approach to Blind Separation and Blind Deconvolution", Neural computation 1995.[4]

1. Gustavo Deco and Wilfried Brauer , "Higher Order Statistical Decorrelation without Information Loss", NIPS 1994. [5]

1. Aapo Hyvarinen and Petteri Pajunen, "Nonlinear independent component analysis: Existence and uniqueness results", Neural networks, 1999. [6]

Linguistics

1. Hiroya Fujisaki, “Prosody, models, and spontaneous speech,” in Computing prosody, pp. 27–42. Springer,1997.

ML

1. Laurent Dinh David Krueger Yoshua Bengio,"NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION", ICLR 2015 [7]
2. Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, "DENSITY ESTIMATION USING REAL NVP", ICLR 2017 [8]

1. Kingma, et al., "Auto-encoding variational Bayes". 2014.
2. Danilo Jimenez Rezende et al., "Variational Inference with Normalizing Flows", 2016.
3. Kingma et al., "Improving Variational Inference with Inverse Autoregressive Flow", 2016.
4. Oord, "Neural Discrete Representation Learning", 2017.
5. Kingma, "Glow: Generative Flow with Invertible 1×1 Convolutions", 2018.
6. Goodfellow et al., "Generative adversarial nets", 2014.
7. Zhu et al., "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks", 2017.
8. Chen et al., "Infogan: Interpretable representation learning by information maximizing generative adversarial nets", 2016.
9. Hu et al., "On unifying deep generative models", 2017.
10. Makhzani, "Adversarial Autoencoders", 2015.
11. Locatello et al., "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations", 2019.

ASR

1. Chan et al., "Listen, attend and spell: A neural network for large vocabulary conversational speech recognition", 2016.
2. Prabhavalkar et al., "A Comparison of Sequence-to-Sequence Models for Speech Recognition", 2017.
3. Chiu et al., "State-of-the-art Speech Recognition With Sequence-to-Sequence Models", 2018.
4. Pratap, "wav2letter++: The Fastest Open-source Speech Recognition System", 2018
5. Ren et al., "Almost Unsupervised Text to Speech and Automatic Speech Recognition", 2019
6. Tsai et al., "Learning Factorized Multimodal Representations", 2019

SID

1. E. Variani, X. Lei, E. McDermott, I. Lopez Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,”2014.
2. G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end textdependent speaker verification,” 2016.

TTS

1. Wang, et al., "Tacotron: A fully end-to-end text-to-speech synthesis model." CoRR, abs/1703.10135, 2017.
2. van den Oord, et al., "Parallel WaveNet: Fast high-fidelity speech synthesis.", CoRR, abs/1711.10433, 2017.
3. van den Oord, et al., "WaveNet: A generative model for raw audio". CoRR, abs/1609.03499, 2016a
4. Nal Kalchbrenner et al., "Efficient Neural Audio Synthesis", 2018 (WaveRNN)
5. Hsu et al., "Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization", NIPS 2018.
6. Wei-Ning Hsu, DISENTANGLING CORRELATED SPEAKER AND NOISE FOR SPEECH SYNTHESIS VIA DATA AUGMENTATION AND ADVERSARIAL FACTORIZATION, ICASSP 2019.

Flow Model

• Overview

• Papamakarios, "Normalizing Flows for Probabilistic Modeling and Inference" [9]

• History

• Whitening:

• Richard M. Johnson. The minimal transformation to orthonormality. Psychometrika, 31: 61–66, 03 1966. [10]
• Friedman, Exploratory projection pursuit, 1987, [11]

• Gaussianization

• Chen et al., Gaussianization, NIPS 2001 [12]
• Tabak et al., Density estimation by dual ascent of the log-likelihood [13]

• Normalization flow

• Tabak et al., A family of non-parametric density estimation, [14] (First composition)
• Rippel et al., High-dimensional probability estimation with deep density model, [15] (first DNN parameterization)
• Laurent Dinh David Krueger Yoshua Bengio,"NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION", ICLR 2015 [16] (First VP)
• Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, "DENSITY ESTIMATION USING REAL NVP", ICLR 2017 [17] (First NVP)
• Danilo Jimenez Rezende et al., Variational Inference with Normalizing Flows, [18] (First apply to variational inference)

VAE Model

• Auto-Encoding Variational Bayes, [19] (First composition)
• Auto-Encoding Variational Bayes,[20]
• Variational Lossy Autoencoder,[21]
• VAE with a VampPrior,[22]
• The Variational Fair Autoencoder,[23]
• Grammar Variational Autoencoder,[24]
• beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,[25]
• Deep Variational Information Bottleneck,[26]
• Hyperspherical Variational Auto-Encoders,[27]
• Understanding disentangling in β-VAE,[28]
• Isolating Sources of Disentanglement in Variational Autoencoders,[29]
• Structured Disentangled Representations,[30]
• Variational Autoencoders with Jointly Optimized Latent Dependency Structure,[31]
• Variational Autoencoders Pursue PCA Directions,[32]
• Correlated Variational Auto-Encoders,[33]
• Mixed-curvature Variational Autoencoders,[34]
• From Variational to Deterministic Autoencoders,[35]
• Variational Autoencoders for Highly Multivariate Spatial Point Processes Intensities,[36]
• Variational Laplace Autoencoders,[37]
• Disentangling Disentanglement in Variational Autoencoders,[38]
• Modeling assumptions and evaluation schemes: On the assessment of deep latent variable models,[39]
• Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders,[40]
• A Joint Generative Model for Zero-Shot Learning,[41]
• A Contrastive Divergence for Combining Variational Inference and MCMC,[42]
• Resisting Adversarial Attacks Using Gaussian Mixture Variational Autoencoders,[43]
• Hierarchical Decompositional Mixtures of Variational Autoencoders,[44]
• Entropic GANs meet VAEs: A Statistical Approach to Compute Sample Likelihoods in GANs,[45]
• InfoVAE: Balancing Learning and Inference in Variational Autoencoders,[46]
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing,[47]
• Variational Autoencoders for Collaborative Filtering,[48]
• Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data,[49]
• Deepcoder: Semi-parametric variational autoencoders for automatic facial action coding,[50]
• Mitigating Posterior Collapse in Strongly Conditioned Variational Autoencoders,[51]
• When Do Variational Autoencoders Know What They Don't Know?,[52]
• Learning Latent Subspaces in Variational Autoencoders,[53]
• Hierarchical Variational Autoencoders for Music,[54]
• Learning Structured Output Representation using Deep Conditional Generative Models,[55]
• A Classifying Variational Autoencoder with Application to Polyphonic Music Generation,[56]
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse,[57]
• Learning Hierarchical Priors in VAEs,[58]

Tools

• VAE: https://jmetzen.github.io/2015-11-27/vae.html
• WaveRNN + VQVAE: https://github.com/mkotha/WaveRNN