• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• preparing EMNLP

• Pick up new task in news generation and do literature review

• Literature review

• Literature review

• Literature review

• Adjust literature review focus

• Literature review

• Literature review

• Try to reproduce sc-lstm work

• Transfer to new task in machine translation and do literature review

• Literature review

• Literature review

• Literature review and code review

• Code review

• Code review and experiment started, but version discrepancy encountered

• Code review and version discrepancy solved

• Code review and experiment
• details about experiment:

 small data, 
 1st and 2nd translator uses the same training data, 
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 42.56

• Entry procedures
• Machine Translation paper reading

• experiment setting:

 small data, 
 1st and 2nd translator uses the different training data, counting 22000 and 22017 seperately
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 36.67 (36.67 is the model at 4750 updates, but we use model at 3000 updates to
                    prevent the case of overfitting, to generate the 2nd translator's training data, for 
                    which the BLEU is 34.96)
 best result of our model: 29.81
 This may suggest that that using either the same training data with 1st translator or different
                   one won't influence 2nd translator's performance, instead, using the same one may
                    be better, at least from results. But I have to give a consideration of a smaller size 
                    of training data compared to yesterday's model.

• code 2nd translator with constant embedding

• Configure environment
• Run tf_translate code
• Read Machine Translation paper

• experiment setting:

 small data, 
 1st and 2nd translator uses the same training data, 
 2nd translator uses constant untrainable embedding imported from 1st translator's decoder

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 43.48
 Experiments show that this kind of series or cascade model will definitely impair the final perfor-
                     mance due to information loss as the information flows through the network from 
                     end to end. Decoder's smaller vocabulary size compared to encoder's demonstrate
                     this (9000+ -> 6000+).
 The intention of this experiment is looking for a map to solve meaning shift using 2nd translator,
                     but result of whether the map is learned or not is obscured by the smaller vocab size 
                     phenomenon.

• literature review on hierarchical machine translation

• Code double decoding model and read multilingual MT paper

• read machine translation paper
• learne lstm model and seq2seq model

• Code double decoding model and experiment
• details about experiment:

 small data, 
 2nd translator uses as training data the concat(Chinese, machine translated English), 
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 43.53

• NEXT: 2nd translator uses trained constant embedding

• understand the difference between lstm model and gru model
• read the implement code of seq2seq model

• read neural machine translation paper
• read tf_translate code

• code and debug double-decoder model
• alter 2017/05/14 model's size and will try after nips

• read neural machine translation paper
• read tf_translate code

• train double-decoder model on small data set but encounter decode bugs

• debug double-decoder model
• the model performs well on develop set, but performs badly on test data. I want to figure out the reason.

• details about experiment:

 hidden_size = 700 (500 in prior)
 emb_size = 510 (310 in prior)
 small data, 
 2nd translator uses as training data the concat(Chinese, machine translated English), 
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 45.21
 But only one checkpoint outperforms the baseline, the other results are commonly under 43.1

• debug double-decoder model

• double-decoder without joint loss generalizes very bad
• i'm trying double-decoder model with joint loss

• details about experiment 1:

 hidden_size = 700
 emb_size = 510
 learning_rate = 0.0005 (0.001 in prior)
 small data, 
 2nd translator uses as training data the concat(Chinese, machine translated English), 
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 42.19
 Overfitting? In overall, the 2nd translator performs worse than baseline

• details about experiment 2:

 hidden_size = 500
 emb_size = 310
 learning_rate = 0.001
 small data, 
 double-decoder model with joint loss which means the final loss  = 1st decoder's loss + 2nd 
 decoder's loss

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 39.04
 The 1st decoder's output is generally better than 2nd decoder's output. The reason may be that 
 the second decoder only learns from the first decoder's hidden states because their states are 
 almost the same.

• DISCOVERY:

 The reason why double-decoder without joint loss generalizes very bad is that the gap between
 force teaching mechanism (training process) and beam search mechanism (decoding process)
 propagates and expands the error to the output end, which destroys the model when decoding.

• next:

 Try to train double-decoder model without joint loss but with beam search on 1st decoder.

• code double-attention one-decoder model
• code double-decoder model

• read neural machine translation paper
• read tf_translate code

• write document of tf_translate project
• read neural machine translation paper
• read tf_translate code

• code and debug double attention model

• read tf_translate code
• write document of tf_translate project

• details about experiment:

 hidden_size = 500
 emb_size = 310
 learning_rate = 0.001
 small data, 
 2nd translator uses as training data both Chinese and machine translated English
 Chinese and English use different encoders and different attention
 final_attn = attn_1 + attn_2
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 when decoding:
   final_attn = attn_1 + attn_2 best result of our model: 43.50
   final_attn = 2/3attn_1 + 4/3attn_2 best result of our model: 41.22
   final_attn = 4/3attn_1 + 2/3attn_2 best result of our model: 43.58

• details about experiment 1:

 hidden_size = 500
 emb_size = 310
 learning_rate = 0.001
 small data, 
 2nd translator uses as training data both Chinese and machine translated English
 Chinese and English use different encoders and different attention
 final_attn = 2/3attn_1 + 4/3attn_2
 2nd translator uses random initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 42.36

• details about experiment 2:

 final_attn = 2/3attn_1 + 4/3attn_2
 2nd translator uses constant initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 45.32

• details about experiment 3:

 final_attn = attn_1 + attn_2
 2nd translator uses constant initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 45.41 and it seems more stable

• run and test tf_translate code
• write document of tf_translate project

• details about experiment 1:

 final_attn = 4/3attn_1 + 2/3attn_2
 2nd translator uses constant initialized embedding

• results (BLEU):

 BASELINE: 43.87
 best result of our model: 45.79

• That only make English word embedding at encoder constant and train all the other embedding and parameters achieves an even higher bleu score 45.98 and the results are stable.
• The quality of English embedding at encoder plays an pivotal role in this model.
• Preparation of big data.

• Only make the English encoder's embedding constant -- 45.98
• Only initialize the English encoder's embedding and then finetune it -- 46.06
• Share the attention mechanism and then directly add them -- 46.20
• Run double-attention model on large data

• Baseline bleu on large data is 30.83 with 30000 output vocab
• Our best result is 31.53 with 20000 output vocab

• Train the model with 40 batch size and with concat(attn_1, attn_2)
• the best result of model with 40 batch size and with add(attn_1, attn_2) is 30.52

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper

• Prepare for APSIPA paper
• Read paper about MT involving grammar

• Prepare for APSIPA paper
• Read paper about MT involving grammar

• Completed APSIPA paper
• Took new task in style translation

• Tried synonyms substitution

• Tried post edit like synonyms substitution but this didn't work

• Trained a GRU language model to determine similar word

• read neural machine translation paper
• read and run tf_translate code

• Trained a GRU language model to determine similar word
• This didn't work because semantics is not captured

• read paper：LSTM Neural Networks for Language Modeling
• read and run ViVi_NMT code

• Tried to figure out new ways to change the text style

• read the API of tensorflow
• debugged ViVi_NMT and tried to upgrade code version to tensorflow1.0

• Trained seq2seq model to solve this problem
• Semantics are stored in fixed-length vectors by a encoder and a decoder generate sequences on this vector

• debugged ViVi_NMT and tried to upgrade code version to tensorflow1.0 (on server)
• installed tensorflow0.1 and tensorflow1.0 on my pc and debugged ViVi_NMT

• Cross-domain seq2seq w/o attention and w/ attention models didn't work because of overfitting

• read the API of tensorflow
• debugged ViVi_NMT and tried to upgrade code version to tensorflow1.0 (on server)

• Read style transfer papers

• debugged ViVi_NMT and tried to upgrade code version to tensorflow1.0 (on server)
• accomplished this task
• found the new version saves more time，has lower complexity and better bleu than before

• Read style transfer papers

• run two versions of the code on small data sets (Chinese-English)
• tested these checkpoint

• recorded experimental results
• found version 1.0 of the code save more training time, has less complexity and these two version of the code has a similar Bleu value
• found that the Bleu is still good when the model is over fitting
• reason: the test set and training set are similar in content and style on small data set

• run two versions of the code on big data sets (Chinese-English)
• read NMT papers

• out of memory（OOM） error occurred when version 0.1 of code was trained using large data set，but version 1.0 worked
• reason: improper distribution of resources by the tensorflow0.1 version leads to exhaustion of memory resources
• I've tried many times, and version 0.1 worked

• tested these checkpoints and recorded experimental results
• the version 1.0 code saved 0.06 second per step than the version 0.1 code

• downloaded the wmt2014 data set
• used the English-French data set to run the code and found the translation is not good
• reason:no data preprocessing is done

• trained translation models using tf1.0 baseline and tf0.1 baseline perspectively
• dataset：zh-en small

• tested these checkpoints
• found the new version takes less time
• found these two versions have similar complexity and bleu values
• found that the bleu is still good when the model is over fitting .
• (reason: the test set and the train set of small data set are similar in content and style)