2013-06-21

Data sharing

• LM count files still undelivered!

DNN progress

Experiments

• sparse DNN: sticky training (retrain the nnet while keeping the sparsness)

zero small values(test set: 1900), with extremely sparseness:

threshold	0	0.2	0.3	0.4	0.5
shrinkage%	0.0	66.4	81.6	0.90	0.94
without sticky: WER	7.55	9.46	53.23	98.99	-
with sticky: WER	7.55	7.56	7.87	8.81	9.87

Conclusion: The extremely sparse network can largely pertain the performance of DNN. The structure seems more important than the parameter tuning based on the structure.

• fixed-point DNN forwarding

1. Migrating the Atlas lib to ARM. Done.
2. Atlas/mkl independent implementation. Done. The non Atlas/MKL solution is much slower, but with sparse DNN, the difference is decreased.
3. Working on comparison of Atlas and non-Atlas on ARM.

Tencent exps

1. ：本月DNN方面，一开始大规模数据（6000小时）训练工作，现得到迭代7次结果如下：

Old Baseline New Baeline DNN-1000小时 DNN-6000小时

1900 8.4 6.8 4.3 3.9

2044 22.4 15.7 12.7 10.7

online1 35.6 32.7 25.8 24.6

online2 29.6 27.3 22.1 21.1

map 24.5 15.8 13.4 8.7

general 36 25.1 19.3 15.9

1. ：DNN+序列化dt，未使用Kaldi代码，重写中，接近完成，鉴于其中包含的各种trick，得到有效结果的时间不好预估。
2. ：调研二阶优化方式，LBFGS，Hessian Free算法，以及异步SGD算法，实现多机并行化实现。
3. ：调研RNN相关的Long Short-Term Memery算法在ASR实现。

GPU & CPU merge

1. Hold

RNN LM progress

• some errors found in the training process
• checking debugbing

Embedded progress

• Status:

ARM debug system is ready. Native compiling is possible.

PS system: SGE-based large scale training is ready.

Kaldi system: ARM migration is done.

Semi continuous model based on the Tencent 400h data is done.

Sys	WER	RT(on server)	RT (on arm)
SP model	8.81	0.07	0.91
Tencent(cnt.)	5.04	0.31	-
Tencent(semi.)	6.26	0.07	0.83

• To be done

1. sparse DNN based Kaldi engine
2. sparse DNN based PS engine