LM-release-v0.2

RELEASE TITLE: LM RELEASE
RELEASE VERSION: v0.2
RELEASE TYPE: STEP RELEASE
RELEASE LOCATION: /work4/singular/public/release/lm/v0.2
RELATED BUGDB: 11

1. BACKGROUND:


2. TECHNOLOGY SUMMARY:

利用40G金融语料和64G通用语料，以3G的大小进行分割，用cleaning v0.1进行清洗，用MultilingualSegmenter.jar进行分词，用得到36个LM，按照1:1进行插值合并，按照不同剪枝率进行剪枝，得到2组LM（其中通用领域LM由于体积过大，需先将22个LM合并成2个，再按5gram_1e-9进行剪枝，再将2个LM合并，再剪枝）。再将2组LM按照1:1进行合并，得到一组混合LM。

3. RELEASE COMPONENT:

LM:  LM RELEASE v0.2





















4. TEST RESULT:

Condition:
LM = 1e-5,
BIGLM=1e-9,
Beam=9
max_active=5000

====================================================================
|   testset   | test_1000ju | test_2000ju | test_myhexin_20161019 |
--------------------------------------------------------------------
|     WER     |    28.94    |    38.71    |          7.65         |
====================================================================


5. RELEASE TEAM:

Author: 赵梦原
Contributor: 张之勇，白子薇
Monitor: 刘荣