“2014-11-03”版本间的差异

来自cslt Wiki
跳转至： 导航、 搜索
（→‎QA）
Lyy（讨论 | 贡献）
第2行： 第2行：
 
=== AM development ===
 
=== AM development ===
  
−
==== Contour ====
+
==== Environment ====
−
* NAN problem
+
* buy two 760-GPUs.
−
:* nan recurrence
+
* sale the old GPU.
−
  ------------------------------------------------------------
+
−
    grid/atr.  |  Reproducible  |    add.
+
−
  ------------------------------------------------------------
+
−
    grid-10    |    yes        | 
+
−
  ------------------------------------------------------------
+
−
    grid-12    |    no          | "nan" in different position
+
−
  ------------------------------------------------------------
+
−
    grid-14    |    yes        | 
+
−
  ------------------------------------------------------------
+
−
:* buy 760-GPU
+
  
 
==== Sparse DNN ====
 
==== Sparse DNN ====
 
* Performance improvement found when pruned slightly
 
* Performance improvement found when pruned slightly
 
* Experiments show that  
 
* Experiments show that  
−
* Suggest to use TIMIT / AURORA 4 for training
+
* Waiting for result of AURORA 4  
 
* HOLD
 
* HOLD
  
 
==== RNN AM====
 
==== RNN AM====
 
* Initial nnet seems no very well, need to be pre-trained or test lower learn-rate.
 
* Initial nnet seems no very well, need to be pre-trained or test lower learn-rate.
−
* For AURORA4 1h/epoch, 100 epochs done.
+
* For AURORA 4 1h/epoch, more than 200 epochs have done.
 
* Using AURORA 4 short-sentence with a smaller number of targets.
 
* Using AURORA 4 short-sentence with a smaller number of targets.
 +
* Adjusting the learning rate.
 +
* Trying toolkit of Microsoft.
  
 
====Noise training====
 
====Noise training====
−
* First draft of the noisy training journal paper.
+
* Paper has been submitted.
−
* Second version released.
+
−
* Paper Correction (Yinshi, Liuchao, Lin Yiye), be going.
+
  
 
====Drop out & Rectification & convolutive network====
 
====Drop out & Rectification & convolutive network====
第80行： 第70行：
 
               dp-1.0            |  9.94          |  11.33          |  12.05          |  8.32
 
               dp-1.0            |  9.94          |  11.33          |  12.05          |  8.32
 
     ---------------------------------------------------------------------------------------------------------
 
     ---------------------------------------------------------------------------------------------------------
−
:** Losing important features, enlarge the hidden-layer dim to 2048.
+
      baseline_dp0.4_lr0.008    |  9.52          |  12.01          |  11.75          |  9.44
−
:** Follow the standard dnn training learn-rate to avoid the different learn-rate changing time of various DNN training.
+
  ---------------------------------------------------------------------------------------------------------
−
:** Test out of known noise test-data.
+
      baseline_dp0.4_lr0.0001    |  9.92          |  14.22          |  13.59          |  10.24
−
:** Continue the droptout on normal trained XEnt NNET , eg wsj(learn-rate:1e-4/1e-5). (++)
+
  ---------------------------------------------------------------------------------------------------------
 +
      baseline_dp0.4_lr0.00001  |  9.06          |  13.27          |  13.14          |  9.33
 +
  ---------------------------------------------------------------------------------------------------------
 +
      baseline_dp0.8_lr0.008    |  9.16          |  11.23          |  11.42          |  8.49
 +
  ---------------------------------------------------------------------------------------------------------
 +
      baseline_dp0.8_lr0.0001    |  9.22          |  11.52          |  11.77          |  8.82
 +
  ---------------------------------------------------------------------------------------------------------
 +
      baseline_dp0.8_lr0.00001  |  9.12          |  11.27          |  11.65          |  8.68
 +
  ---------------------------------------------------------------------------------------------------------
 +
        dp-0.4_follow-std-lr    |  11.33          |  14.60          |  13.50          |  10.95
 +
  ---------------------------------------------------------------------------------------------------------
 +
        dp-0.8_follow-std-lr    |  9.77          |  12.01          |  11.79          |  8.93
 +
  ---------------------------------------------------------------------------------------------------------
 +
          dp-0.4_4-2048          |  11.69          |  16.13          |  14.24          |  11.98
 +
  ---------------------------------------------------------------------------------------------------------
 +
          dp-0.8_4-2048          |  9.46          |  11.60          |  11.98          |  8.78
 +
  ---------------------------------------------------------------------------------------------------------
 +
 
 +
:** Test baseline of 2048 dimension.
 +
:** Test with AURORA4 of 7000 (clean + noisy).
 +
:** Follow the standard DNN training learn-rate to avoid the different learn-rate changing time of various DNN training. Similar performance is obtained.
 +
:** Find and test unknown noise test-data.(+)
 +
:** Have done the droptout on normal trained XEnt NNET , eg wsj(learn-rate:1e-4/1e-5).  
 
:** Draft the dropout-DNN weight distribution. (++)
 
:** Draft the dropout-DNN weight distribution. (++)
  
 
* Rectification
 
* Rectification
 
:* Still NAN error, need to debug.  
 
:* Still NAN error, need to debug.  
−
   1) AURORA4 -15h
+
   1) AURORA 4 -15h
 
   (1) Train: train_clean
 
   (1) Train: train_clean
 
       learn-rate/testcase(WER)  | test_clean_wv1  | test_airport_wv1 | test_babble_wv1 | test_car_wv1  
 
       learn-rate/testcase(WER)  | test_clean_wv1  | test_airport_wv1 | test_babble_wv1 | test_car_wv1  
 
     ---------------------------------------------------------------------------------------------------------
 
     ---------------------------------------------------------------------------------------------------------
 
           std-baseline        |  6.04          |  29.91          |  27.76          |  16.37
 
           std-baseline        |  6.04          |  29.91          |  27.76          |  16.37
 +
    ---------------------------------------------------------------------------------------------------------
 +
          lr0.00001            |  8.30          |  43.85          |  46.42          |  29.80
 +
    ---------------------------------------------------------------------------------------------------------
 +
          lr0.0001            |  6.57          |  31.11          |  30.65          |  19.65
 +
    ---------------------------------------------------------------------------------------------------------
 +
          lr0.0006            |  6.19          |  29.23          |  28.45          |  17.31
 +
    ---------------------------------------------------------------------------------------------------------
 +
          lr0.0008            |  6.17          |  28.10          |  27.46          |  14.97
 
     ---------------------------------------------------------------------------------------------------------
 
     ---------------------------------------------------------------------------------------------------------
 
           lr0.001              |  6.28          |  30.01          |  30.26          |  20.81
 
           lr0.001              |  6.28          |  30.01          |  30.26          |  20.81
第110行： 第130行：
 
         lr-0.001_l1-0.000001    |  6.30          |  31.91          |  29.23          |  21.52
 
         lr-0.001_l1-0.000001    |  6.30          |  31.91          |  29.23          |  21.52
 
     ---------------------------------------------------------------------------------------------------------
 
     ---------------------------------------------------------------------------------------------------------
 +
 +
:* Train rectifier on baseline.
 +
:* Combine drop out and rectifier.
 
:* Change the learn-rate in the middle of the training, Modify the train_nnet.sh script(Liu Chao).
 
:* Change the learn-rate in the middle of the training, Modify the train_nnet.sh script(Liu Chao).
−
:* Using maximum learning-rate.
 
  
−
* MaxOut (++)
+
* MaxOut (+)
 +
:* 6min/epoch, can't use high lr.
 +
 
 +
* P-norm
  
 
* Convolutive network (+)
 
* Convolutive network (+)
−
:* Test more configurations
+
  ORG_DNN.4-1200 WER: 4.50
 +
 
 +
                    |  WER | hid-layers | hid-dim | delta-order | splice | lda-dim | learn-rate | cnn_init_opts
 +
------------------------------------------------------------------------------------------------------------------
 +
  cnn_std_baseline  | 4.86 |    5      | 1200   |      0      |    5 |  198  |  0.008 | "--patch-dim1 7"   
 +
 
 +
:* AURORA 4
 +
:* READ paper
  
 
====Denoising & Farfield ASR====
 
====Denoising & Farfield ASR====
第126行： 第158行：
  
 
* Spike detection and removal.
 
* Spike detection and removal.
−
* Add more silence tag "#" in pure-silence utterance text(train).
 
−
:* xEntropy model be training
 
−
:* need to test baseline.
 
 
* Sum all sil-pdf as the silence posterior probability.
 
* Sum all sil-pdf as the silence posterior probability.
 
:* Program done, to tune the threshold
 
:* Program done, to tune the threshold
第136行： 第165行：
 
* Seems ROS model is superior to the normal one with faster speech
 
* Seems ROS model is superior to the normal one with faster speech
 
* Suggest to extract speech data of different ROS, construct a new test set(+)
 
* Suggest to extract speech data of different ROS, construct a new test set(+)
−
* Tencent training data done
+
* Tencent training data with 100h
  
 
==== low resource language AM training ====
 
==== low resource language AM training ====
第150行： 第179行：
 
   |      1        | 16.87        |      |        |
 
   |      1        | 16.87        |      |        |
 
   |      0        | 16.88        |      |        |   
 
   |      0        | 16.88        |      |        |   
 +
 +
mpe:
 +
| Nnet-structure |  WER  |
 +
|  baseline    | 16.12 |
 +
|    4-0-1      |      |
 +
|    4-1-1      | 16.10 |
 +
|    4-2-1      | 15.66 |
 +
|    3-1-1      | 16.10 |
 +
|    3-2-1      | 15.64 |
 +
|    2-2-1      | 15.73 |
 +
|    1-3-1      | 15.91 |
 +
|    0-4-1      |      |
 +
 +
 +
remark: 4-0-1 means 4 hidden-layers from 6000h_CN, 0 hidden-layer from random generation, 1 output-layer.
 
:**  feature_transform = uyghur_transform + 6000_N*hidden-layers
 
:**  feature_transform = uyghur_transform + 6000_N*hidden-layers
 
   nnet.init = random (4-N)*hidden-layers + output-layer
 
   nnet.init = random (4-N)*hidden-layers + output-layer
第173行： 第217行：
  
 
* sub word unit language model is ready. on testing.
 
* sub word unit language model is ready. on testing.
 +
 
====Scoring====
 
====Scoring====
−
* Harmonics program done, experiment to be done.
+
* Timber Comparison on testing
−
* Initial experiment shows more timber data are required
+
  
 
====Confidence====
 
====Confidence====
第186行： 第230行：
 
* EER ~ 11.2% (GMM-based system)
 
* EER ~ 11.2% (GMM-based system)
 
* test different number of components; fast i-vector computing
 
* test different number of components; fast i-vector computing
 +
 +
===Language ID===
 +
* GMM-based language is ready.
 +
* Delivered to Jietong
  
 
===Emotion detection===
 
===Emotion detection===
  
 
* Sinovoice is implementing the server
 
* Sinovoice is implementing the server
−
 
−
 
  
 
==Text Processing==
 
==Text Processing==

2014年11月3日 (一) 09:10的版本

Speech Processing

AM development

Environment

  • buy two 760-GPUs.
  • sale the old GPU.

Sparse DNN

  • Performance improvement found when pruned slightly
  • Experiments show that
  • Waiting for result of AURORA 4
  • HOLD

RNN AM

  • Initial nnet seems no very well, need to be pre-trained or test lower learn-rate.
  • For AURORA 4 1h/epoch, more than 200 epochs have done.
  • Using AURORA 4 short-sentence with a smaller number of targets.
  • Adjusting the learning rate.
  • Trying toolkit of Microsoft.

Noise training

  • Paper has been submitted.

Drop out & Rectification & convolutive network

  • Drop out
  • dataset:wsj, testset:eval92 
       std |  dropout0.4 | dropout0.5 | dropout0.6 | dropout0.7 | dropout0.8
    ------------------------------------------------------------------------- 
       4.5 |     5.39    |    4.80    |   4.75     |  4.36      |    4.55
    • Frame-accuarcy seems not consistent with WER.
    • Using the train-data as cv, verify the learning ability of the model.
  • AURORA4 dataset 
  (1) Train: train_clean      
   drop-retention/testcase(WER) | test_clean_wv1  | test_airport_wv1 | test_babble_wv1 | test_car_wv1 
   ---------------------------------------------------------------------------------------------------------
          std-baseline          |  6.04           |  29.91           |  27.76          |  16.37
   ---------------------------------------------------------------------------------------------------------
             dp-0.4             |  6.61           |  29.59           |  30.12          |  19.40
   ---------------------------------------------------------------------------------------------------------
             dp-0.5             |  6.40           |  28.07           |  27.88          |  19.88
   ---------------------------------------------------------------------------------------------------------
             dp-0.6             |  6.36           |  26.68           |  24.85          |  18.32
   ---------------------------------------------------------------------------------------------------------
             dp-0.7             |  6.13           |  25.53           |  23.90          |  15.69
   ---------------------------------------------------------------------------------------------------------
             dp-0.8             |  5.94           |  24.94           |  23.67          |  15.77
   ---------------------------------------------------------------------------------------------------------
             dp-0.9             |  5.96           |  27.30           |  25.63          |  15.46
   ---------------------------------------------------------------------------------------------------------
 
  (2) Train: train_nosiy
   drop-retention/testcase(WER) | test_clean_wv1  | test_airport_wv1 | test_babble_wv1 | test_car_wv1 
   ---------------------------------------------------------------------------------------------------------
          std-baseline          |  9.60           |  11.41           |  11.63          |  8.64
   ---------------------------------------------------------------------------------------------------------
             dp-0.3             |  12.91          |  16.55           |  15.37          |  12.60
   ---------------------------------------------------------------------------------------------------------
             dp-0.4             |  11.48          |  14.43           |  13.23          |  11.04
   ---------------------------------------------------------------------------------------------------------
             dp-0.5             |  10.53          |  13.00           |  12.89          |  10.24
   ---------------------------------------------------------------------------------------------------------
             dp-0.6             |  10.02          |  12.32           |  11.81          |  9.29
   ---------------------------------------------------------------------------------------------------------
             dp-0.7             |  9.65           |  12.01           |  12.09          |  8.89
   ---------------------------------------------------------------------------------------------------------
             dp-0.8             |  9.79           |  12.01           |  11.77          |  8.91
   ---------------------------------------------------------------------------------------------------------
             dp-1.0             |  9.94           |  11.33           |  12.05          |  8.32
   ---------------------------------------------------------------------------------------------------------
     baseline_dp0.4_lr0.008     |  9.52           |  12.01           |  11.75          |  9.44
  ---------------------------------------------------------------------------------------------------------
     baseline_dp0.4_lr0.0001    |  9.92           |  14.22           |  13.59          |  10.24
  ---------------------------------------------------------------------------------------------------------
     baseline_dp0.4_lr0.00001   |  9.06           |  13.27           |  13.14          |  9.33
  ---------------------------------------------------------------------------------------------------------
     baseline_dp0.8_lr0.008     |  9.16           |  11.23           |  11.42          |  8.49
  ---------------------------------------------------------------------------------------------------------
     baseline_dp0.8_lr0.0001    |  9.22           |  11.52           |  11.77          |  8.82
  ---------------------------------------------------------------------------------------------------------
     baseline_dp0.8_lr0.00001   |  9.12           |  11.27           |  11.65          |  8.68
  ---------------------------------------------------------------------------------------------------------
       dp-0.4_follow-std-lr     |  11.33          |  14.60           |  13.50          |  10.95
  ---------------------------------------------------------------------------------------------------------
       dp-0.8_follow-std-lr     |  9.77           |  12.01           |  11.79          |  8.93
  ---------------------------------------------------------------------------------------------------------
         dp-0.4_4-2048          |  11.69          |  16.13           |  14.24          |  11.98
  ---------------------------------------------------------------------------------------------------------
         dp-0.8_4-2048          |  9.46           |  11.60           |  11.98          |  8.78
  ---------------------------------------------------------------------------------------------------------
    • Test baseline of 2048 dimension.
    • Test with AURORA4 of 7000 (clean + noisy).
    • Follow the standard DNN training learn-rate to avoid the different learn-rate changing time of various DNN training. Similar performance is obtained.
    • Find and test unknown noise test-data.(+)
    • Have done the droptout on normal trained XEnt NNET , eg wsj(learn-rate:1e-4/1e-5).
    • Draft the dropout-DNN weight distribution. (++)
  • Rectification
  • Still NAN error, need to debug. 
 1) AURORA 4 -15h
 (1) Train: train_clean
     learn-rate/testcase(WER)  | test_clean_wv1  | test_airport_wv1 | test_babble_wv1 | test_car_wv1 
   ---------------------------------------------------------------------------------------------------------
          std-baseline         |  6.04           |  29.91           |  27.76          |  16.37
   ---------------------------------------------------------------------------------------------------------
          lr0.00001            |  8.30           |  43.85           |  46.42          |  29.80
   ---------------------------------------------------------------------------------------------------------
          lr0.0001             |  6.57           |  31.11           |  30.65          |  19.65
   ---------------------------------------------------------------------------------------------------------
          lr0.0006             |  6.19           |  29.23           |  28.45          |  17.31
   ---------------------------------------------------------------------------------------------------------
          lr0.0008             |  6.17           |  28.10           |  27.46          |  14.97
   ---------------------------------------------------------------------------------------------------------
          lr0.001              |  6.28           |  30.01           |  30.26          |  20.81
   ---------------------------------------------------------------------------------------------------------
          lr0.003              |  6.44           |  32.01           |  32.24          |  17.82
   ---------------------------------------------------------------------------------------------------------
          lr0.005              |  6.47           |  33.49           |  34.75          |  18.15
   ---------------------------------------------------------------------------------------------------------
          lr0.007              |  6.72           |  35.85           |  39.72          |  18.03
   ---------------------------------------------------------------------------------------------------------
        lr-0.001_l1-0.001      |  83.19          |  98.57           |  98.84          |  97.77
   ---------------------------------------------------------------------------------------------------------
        lr-0.001_l1-0.0001     |  7.58           |  32.94           |  34.29          |  23.42
   ---------------------------------------------------------------------------------------------------------
       lr-0.001_l1-0.00001     |  6.21           |  29.15           |  28.24          |  19.50
   ---------------------------------------------------------------------------------------------------------
       lr-0.001_l1-0.000001    |  6.30           |  31.91           |  29.23          |  21.52
   ---------------------------------------------------------------------------------------------------------
  • Train rectifier on baseline.
  • Combine drop out and rectifier.
  • Change the learn-rate in the middle of the training, Modify the train_nnet.sh script(Liu Chao).
  • MaxOut (+)
  • 6min/epoch, can't use high lr.
  • P-norm
  • Convolutive network (+) 
 ORG_DNN.4-1200 WER: 4.50

                   |  WER | hid-layers | hid-dim | delta-order | splice | lda-dim | learn-rate	| cnn_init_opts
------------------------------------------------------------------------------------------------------------------
 cnn_std_baseline  | 4.86 |     5      | 1200	  |      0      |    5 	 |   198   |   0.008	| "--patch-dim1 7"
  • AURORA 4
  • READ paper

Denoising & Farfield ASR

  • ICASSP paper submitted.
  • HOLD

VAD

  • Spike detection and removal.
  • Sum all sil-pdf as the silence posterior probability.
  • Program done, to tune the threshold
  • rearrange the ending point of the detected speech

Speech rate training

  • Seems ROS model is superior to the normal one with faster speech
  • Suggest to extract speech data of different ROS, construct a new test set(+)
  • Tencent training data with 100h

low resource language AM training

  • Use Chinese NN as initial NN, change the last layer
  • Various the used Chinese trained DNN layer numbers.
    • feature_transform = 6000h_transform + 6000_N*hidden-layers 
 nnet.init = random (4-N)*hidden-layers + output-layer
 | N / learn_rate | 0.008         | 0.001 | 0.0001 |
 |   baseline     | 17.00(14*2h)  |       |        |
 |       4        | 17.75(9*0.6h) | 18.64 |        |
 |       3        | 16.85         |       |        |
 |       2        | 16.69         |       |        |
 |       1        | 16.87         |       |        |
 |       0        | 16.88         |       |        |

mpe: | Nnet-structure | WER | | baseline | 16.12 | | 4-0-1 | | | 4-1-1 | 16.10 | | 4-2-1 | 15.66 | | 3-1-1 | 16.10 | | 3-2-1 | 15.64 | | 2-2-1 | 15.73 | | 1-3-1 | 15.91 | | 0-4-1 | |


remark: 4-0-1 means 4 hidden-layers from 6000h_CN, 0 hidden-layer from random generation, 1 output-layer.

    • feature_transform = uyghur_transform + 6000_N*hidden-layers 
 nnet.init = random (4-N)*hidden-layers + output-layer
 Note: This is reproduced Yinshi's experiment
 | N / learn_rate | 0.008 | 0.001 | 0.0001 |
 |   baseline     | 17.00 |       |        |
 |       4        | 28.23 | 30.72 | 37.32  |
 |       3        | 22.40 |       |        |
 |       2        | 19.76 |       |        |
 |       1        | 17.41 |       |        |
 |       0        |       |       |        |
    • feature_transform = 6000_transform + 6000_N*hidden-layers 
 nnet.init = uyghur (4-N)*hidden-layers + output-layer
 | N / learn_rate | 0.008 | 0.001 | 0.0001 |
 |   baseline     | 17.00 |       |        |
 |       4        | 17.80 | 18.55 | 21.06  |
 |       3        | 16.89 | 17.64 |        |
 |       2        |       |       |        |
 |       1        |       |       |        |
 |       0        |       |       |        |


  • sub word unit language model is ready. on testing.

Scoring

  • Timber Comparison on testing

Confidence

  • Reproduce the experiments on fisher dataset.
  • Use the fisher DNN model to decode all-wsj dataset
  • preparing scoring for puqiang data

Speaker ID

  • Preparing GMM-based server.
  • EER ~ 11.2% (GMM-based system)
  • test different number of components; fast i-vector computing

Language ID

  • GMM-based language is ready.
  • Delivered to Jietong

Emotion detection

  • Sinovoice is implementing the server

Text Processing

LM development

Domain specific LM

  • domain lm
  • weibo lm with pruning 0 10 10 20 20 testing done. weibo lm with pruning 0 10 8 8 8 under testing. weibo lm without pruning 4/8 done.
  • merger weibo、baiduhi and baiduzhidao lm and test (this week)
  • confirm the size of alpa with xiaomin for business application.(like e-13)
  • get the general test data from miaomin .this test set may get from online.
  • new dict.
  • train lm on baiduhi, baiduzhida with new 150k dict and test (this week)
  • new toolkit:find method to update the new dict. can get new wordlist from sougou and get word information from baidu.(two week)

tag LM

  • set new test
  • fix the bug
  • record test set and test the unknown address (this week)

RNN LM

  • rnn
  • RNNLM=>ALPA make a report
  • test RNNLM on Chinese data from jietong-data
  • check the rnnlm code.
  • lstm+rnn
  • check the lstm-rnnlm code

Word2Vector

W2V based doc classification

  • Initial results variable Bayesian GMM obtained. Performance is not as good as the conventional GMM.
  • Non-linear inter-language transform: English-Spanish-Czch: wv model training done, transform model on investigation
  • SSA-based local linear mapping still on running.
  • k-means classes change to 2.
  • Knowledge vector started
  • format the data
  • yuanbin will continue this work with help of xingchao.
  • Character to word conversion
  • prepare the task: word similarity
  • prepare the dict.
  • Google word vector train
  • some ideal will discuss on weekly report.

Translation

  • v4.0 demo released
  • cut the dict and use new segment-tool

QA

  • lucene Optimization
  • rewrite the method to select the 50 standard question not same template.(this week)
  • test the boost keyword weight and extract the synonyms word.(this week)
  • check the word segment for template.(this week)
  • min-segment method improve the accuracy.(0.61->0.66)
  • check the query method for getting lucene information and to rewrite the score method like the idf value.
  • test
  • test the different idf vale from baidu sougou in fuzzymatch.(this week)
  • need to check the other 10% error.(this week)
  • spell check
  • simple demo done.
  • new inter will install SEMPRE
取自“http://index.cslt.org/mediawiki/index.php?title=2014-11-03&oldid=12263”