“Gigabye LM”版本间的差异

2012年9月13日 (四) 05:57的版本

The training is with Gigabytes except the cna data, and ppl testing is based on a sub set from the cna data (big52gb applied)

2gram:

25M 2gram.4000.gz: 0 zeroprobs, logprob= -9.39983e+06 ppl= 161.965 ppl1= 177.141

3gram:

47M 3gram.500.gz:0 zeroprobs, logprob= -6.34868e+06 ppl= 85.1361 ppl1= 94.2525

117M 3gram.1000.gz :0 zeroprobs, logprob= -7.43809e+06 ppl= 80.6408 ppl1= 87.7439

195M 3gram.2000.gz:0 zeroprobs, logprob= -7.95872e+06 ppl= 79.9875 ppl1= 86.5196

221M 3gram.3000.gz:0 zeroprobs, logprob= -8.04799e+06 ppl= 80.2418 ppl1= 86.7277

229M 3gram.4000.gz:0 zeroprobs, logprob= -8.15697e+06 ppl= 82.6585 ppl1= 89.3392

4gram:

205M 4gram.500.gz:0 zeroprobs, logprob= -6.25395e+06 ppl= 79.6739 ppl1= 88.0716

472M 4gram.1000.gz:0 zeroprobs, logprob= -7.21607e+06 ppl= 70.737 ppl1= 76.774

Model	2gram	3gram	size
1	1e-7	1e-7	30M logprob= -8.55982e+06 ppl= 102.796 ppl1= 111.532
2	1e-6	1e-6	5M logprob= -9.26982e+06 ppl= 150.96 ppl1= 164.9
3	1e-7	1e-6	11M logprob= -9.09681e+06 ppl= 137.467 ppl1= 149.913

tri-gram size:

final fst size:

@@ 第29行： / 第29行： @@
 == 2. pruning the 4k 3gram LM. ==
- Model 1gram 2gram 3gram              size        ppl
+{|
-           1e-7  1e-7      1e-7               30M     logprob= -8.55982e+06 ppl= 102.796 ppl1= 111.532
+! Model ||2gram ||3gram  ||            size  ||      ppl
-           1e-6  1e-6      1e-6               5M       logprob= -9.26982e+06 ppl= 150.96   ppl1= 164.9
+|-
-          1e-7  1e-6.5    1e-6.5           11M      logprob= -9.09681e+06 ppl= 137.467 ppl1= 149.913
+| 1          ||           1e-7    ||  1e-7          ||     30M     logprob= -8.55982e+06 ppl= 102.796 ppl1= 111.532
+|-
+|2           ||           1e-6    ||  1e-6          ||     5M       logprob= -9.26982e+06 ppl= 150.96   ppl1= 164.9
+|-
+| 3          ||          1e-7    ||  1e-6          ||     11M      logprob= -9.09681e+06 ppl= 137.467 ppl1= 149.913
+|-
+|}
 == 3. word-based 3-gram ==