cslt Wiki - 用户贡献 [zh-cn]

Bi-monthly-2016-08-language Language group

2016-08-31T05:50:57Z

Liuaiting：

AitingLiu [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/c/c0/Bimonthly_report_Aiting.pdf pdf]]

Lishiyao [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/9/90/Lishiyao.pdf ppt]]

Andy Zhang [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/b/b6/Andy%27s_bimonthly_report.pdf slides]]

Jiyuan Zhang [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/9/98/Bimonthly_report_JiyuanZhang.pptx slides]]

Aodong Li [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/6/6e/Bi-monthly_report_aodong.pdf slides]]

Group [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/5/52/NLP-bimonthly-report-20160831.pdf slides]]

文件:Bimonthly report Aiting.pdf

2016-08-31T05:50:29Z

Liuaiting：

Schedule

2016-08-31T02:51:39Z

Liuaiting：/* Daily Report */

=NLP Schedule=

==Members==

===Current Members===

* Yang Feng (冯洋)
* Jiyuan Zhang （张记袁）
* Aodong Li (李傲冬)
* Andi Zhang (张安迪)
* Ziwei Bai （白子薇）
* Aiting Liu (刘艾婷)
* Shiyao Li （李诗瑶）

===Former Members===
* '''Chao Xing (邢超)''' : FreeNeb
* '''Rong Liu (刘荣)''' : 优酷
* '''Xiaoxi Wang (王晓曦)''' : 图灵机器人
* '''Xi Ma (马习)''' : 清华大学研究生
* '''Tianyi Luo (骆天一)''' ： phd candidate in University of California Santa Cruz
* '''Qixin Wang (王琪鑫)''' : MA candidate in University of California
* '''DongXu Zhang (张东旭)''': --
* '''Yiqiao Pan (潘一桥)''' ： MA candidate in University of Sydney

==Work Progress==
===Daily Report===

{|class="wikitable"
! Date !! Person !! start!! leave !! hours ||status
|-
| rowspan="4"|2016/08/16
|Ziwei Bai|| 9:30 || 17:30 || 7 ||learn kernel construction,write the first construction method.
|-
|Andy Zhang||9:30 || 17:30 || 7 || prepare for the ML book, watch Wang's video of neural network
|-
|Aiting Liu|| 9:30 || 17:30 || 7 || learn Wang's instructional video of Neural Network
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/17
|Ziwei Bai|| 9:00 || 18:00 || 8 || rewrite 'the property of Gram matrix' and write the second kernel construction method
|-
|Andy Zhang|| 9:00 || 18:00 || 8 || read materials about Radio basis function networks as well as self-organizing map
|-
|Aiting Liu|| 9:00 || 18:00 || 8 || view the information related to the Hopfield network and Boltzmann machine
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/18
|Ziwei Bai|| 9:30 ||18:30 || 8 ||learn the 'propert of kernel function'&learn chaos' chatting model predict program
|-
|Andy Zhang||9:30 || 18:30 || 8 || continue preparing for the MLbook
|-
|Aiting Liu||9:30 || 18:30 || 8 || prepare for the presentation, learn RBF
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/19
|Ziwei Bai|| 9:00 || 18:00 || 8 || finish the first edition of my work of ML book
|-
|Andy Zhang|| 9:00 || 18:00 || 8 ||learn Hopfield network, prepare for the writing
|-
|Aiting Liu|| 9:00 || 18:00 || 8 || learn Andrew Ng's online course
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/20
|Ziwei Bai|| 9:40 || 18:50 || 8 || perfect my work of the book
|-
|Andy Zhang|| || || ||
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/22
|Ziwei Bai|| 9:15 || 18:15 ||8 || test Chatting model ,retrain word vector
|-
|Andy Zhang|| 9:15 || 18:15 || 8 || start writing chapter 3, version 0.0.
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || || || ||
|-
| rowspan="4"|2016/08/23
|Ziwei Bai|| 9:00 || 18:15||8 || 1、retrain NRM model & word vec 2.review paper‘NRM for STC’
|-
|Andy Zhang||9:00 || 18:15 || 8 || continue writing the ML book. mainly worked on MLP today
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || 9:00 || 18:15 || 8 || read the book Deep Learning and prepare the presentation of Probability Theory
|-
| rowspan="4"|2016/08/24
|Ziwei Bai|| 9:05 || 18:05 || 8 || test Chatting model 0-8,modify ASR Python Server Functional Specification
|-
|Andy Zhang|| 9:05|| 18:05 ||8 || mainly wrote the section of MLP & RBF
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || 9:05 || 18:05 || 8 || finish the preparation of the presentation about Linear Algebra and Probability Theory
|-
| rowspan="4"|2016/08/25
|Ziwei Bai|| 9:20 || 21:00 || 7.5 || 1、test Chatting Model 9-14，and upload to cvss。2、learn Bengio‘s dialogue system by hierarchical NN（except experiment and result）。
|-
|Andy Zhang|| 9:20 || 21:00 || 7.5 || continue writing chapter3
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || 9:20 || 21:00 || 7.5 || revise the PPT and start to learn tensorflow
|-
| rowspan="4"|2016/08/26
|Ziwei Bai|| 9:30 || 18:30 || 8 || prepare for seminar,select 100 posts Suitable for dialogue from weibo data
|-
|Andy Zhang|| 9:30 || 18:30 || 8 || continue writing chapter3
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li || 9:30 || 18:30 || 8 || self-study tensorflow
|-
| rowspan="4"|2016/08/27
|Ziwei Bai|| 9:30 || 20:30 || 10 || 1、filter the movie scipts data based on weibo data word vector (30W+pairs) and try to train Chatting Model with the dataset on the basis of model 31, but load model failed(maybe because the version of tensor flow not consistent).2、find the response Corresponding to the post selected yesterday and test the model (0-34) with it
|-
|Andy Zhang|| || || ||
|-
|Aiting Liu|| || || ||
|-
|Shiyao Li|| || || ||
|-
| rowspan="5"|2016/08/29
|Ziwei Bai|| 9:20 || 18:00 || 7+ || test the Chatting model,count the number of startswith <END> sentences,and natural sentences.
|-
|Andy Zhang|| 9:20 || 17:40 || 7+ || continued working on chapter 3, did some revise work & tried to find ways to draw figures
|-
|Aiting Liu|| 9:20 || 18:20 || 8 || prepare diagrams in the book
|-
|Shiyao Li|| 9:20 || 18:20 || 8 || learn the CNN to solve MNIST using tensorflow
|-
|Aodong Li|| 10:30 || 16:30 || 5 || Read shared paper at the week before the last week
|-
| rowspan="4"|2016/08/30
|Ziwei Bai|| 9:15 ||18:15 || 8 || 1、prepare for bimonthly report.2、test Chatting Model
|-
|Andy Zhang|| 9:15 || 18:15 || 8 || finished version 1 of chapter 3 with the help of Aiting
|-
|Aiting Liu|| 9:15 || 18:15 || 8 || assist Andy to complete chapter3, prepare bimonthly report
|-
|Shiyao Li|| 9:15 || 18:15 || 8 || continue to learn tensorflow and train several models about mnist task
|-
| rowspan="5"|2016/08/31
|Ziwei Bai||9:10 || || ||
|-
|Andy Zhang|| || || ||
|-
|Aiting Liu|| 9:10 || 18:10 || 8 || bimonthly report
|-
|Shiyao Li|| 9:10 || || ||
|-
|Aodong Li|| 08:20 || || ||
|-
|}

===Month Summary===

{| class="wikitable"
!People!! Summary
|-
|Yang Feng ||
*surveyed the line works of neural parsing, neural Turing machine and variants of RBMs and fixed the research topic to jointly parsing and understanding with Turing machine
*learned tensorflow
*produced two versions of results of query paraphrase
|-
|Jiyuan Zhang ||
|-
|Aodong Li ||
*Code:
2. Rare word embedding on word2vec and the related in C
3. Help Lantian for ICASSP
*Paper share: one for bilingual word embedding
*ML book: first version of chapter 2 with Aiting
|-
|Ziwei Bai ||
*finish part of chapter 5 kernel method
*retrain & test Chatting Model
*share 'the use of tensorflow'
|-
|Andy Zhang ||
*shared the principle of word2vec
*read several papers about comparison of sentence similarity
*wrote chapter 3 neural network of the ML book
|-
|Aiting Liu ||
*share two papers:
1.Intrinsic Subspace Evaluation of Word Embedding Representations [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/6/68/Intrinsic_Subspace_Evaluation_of_Word_Embedding_Representations.pdf pdf]]
[[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/1/17/Intrinsic_Subspace_Evaluation_of_Word_Embedding_Representations.pptx slides]]

2.A Sentence Interaction Network for Modeling Dependence between Sentences
[[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/0/04/A_Sentence_Interaction_Network_for_Modeling_Dependence_between_Sentences.pdf pdf]]
[[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/6/6c/A_Sentence_Interaction_Network_for_Modeling_Dependence_between_Sentences.pptx slides]]
*write chapter2 linear model
*help Andy complete chapter3 neural network
|-
|Shiyao Li ||
*prepare the data for question-pair task.
*use SRILM to train the language model and Moses to train the task
*share a paper about LSTM and do a presentation.
|}

===Time Off Table===

{| class="wikitable"
! Date !! Yang Feng !! Jiyuan Zhang !! Aodong Li !! Ziwei Bai !! Andy Zhang !! Aiitng Liu !! Shiyao Li
|-
|2016/8/15 || || || || || || || 8 hours
|-
|2016/8/16 || || || || || || || 8 hours
|-
|2016/8/17 || || || || || || || 8 hours
|-
|2016/8/18 || 2 hours || || || || || || 4 hours
|-
|2016/8/19 || || 6 hours || || || || || 8 hours
|-
|2016/8/22 || || || || || || 8 hours || 8 hours
|-
|2016/8/23 || || 2 hours || || || || 8 hours ||
|-
|2016/8/24 || || || || || || 8 hours ||
|-
|2016/8/25 || || || || || || 8 hours ||
|-
|2016/8/26 || || || 5 hours || || || 8 hours ||
|-
|2016/8/29 || || || 0.5 hours || || || ||
|}

==Past progress==
[[nlp-progress 2016/08]]

[[nlp-progress 2016/05/01 -- 08/16 | nlp-progress 2016/05-07]]

[[nlp-progress 2016/04]]

Bi-monthly-2016-08-language Language group

2016-08-31T00:51:19Z

Liuaiting：

AitingLiu [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/2/26/Bimonthly_report_AitingLiu.pptx ppt]]

Bi-monthly-2016-08-language Language group

2016-08-31T00:50:52Z

Liuaiting：以“AitingLiu [http://cslt.riit.tsinghua.edu.cn/mediawiki/images/2/26/Bimonthly_report_AitingLiu.pptx ppt]”为内容创建页面

AitingLiu [http://cslt.riit.tsinghua.edu.cn/mediawiki/images/2/26/Bimonthly_report_AitingLiu.pptx ppt]

2016-08-29T08:53:48Z

Liuaiting：

{| class="wikitable"
!Date !! People !! Last Week !! This Week
|-
| rowspan="6"|2016/08/22
|Yang Feng ||
*surveyed the line of neural parsing work
*arranged the nlp wiki pages
*arranged the report of tensorflow
||
*start the work of neural grammar
|-
|Jiyuan Zhang ||
*added punctuation to input
*added input vector to the attention layer
*the result of the poem7_49k [[http://192.168.0.51:5555/cgi-bin/cvss/cvss_request.pl?account=zhangjy&step=view_request&cvssid=559 here]]
||
*writing books
*an overview of RNN,RBM,LSTM
|-
|Aodong Li ||
*Mainly focus on writing chapter 2, Linear Model.
*Now we have completed Introduction, Polynomial Regression, Linear Regression, Linear Classification, Probabilistic PCA, part of Probabilistic LDA.
||
*Complete the remaining of chapter 2--PLDA and Bayesian Approach.
|-
|Andy Zhang ||
*read books, papers, etc. about chapter3 neural networks;
*wrote the outine of this chapter
||
*start writing my chapter
|-
|Aiting Liu || ||
|-
|Shiyao Li || ||
|-
| rowspan="6"|2016/08/29
|Yang Feng ||
*continued surveying the line of neural parsing work
*read papers of variants of RBM and neural Turing machine
*learned tensorflow
||
*read more papers of Turing machine and get the full picture of my idea
*start the baseline work of neural Turing machine
|-
|Jiyuan Zhang ||
*shared an overview of LSTM,RNN,RBM
*prepared relevant knowledge for writing book
*ran models of 58k-hybird and 14k-hybird [[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/3/3b/Result_of_58k_and_14k.pdf result]]
||
*the main focus on writing books
|-
|Aodong Li ||
*Finish the first version of chapter 2, Linear Model
*Help Lantian for ICASSP
||
*Revise the Linear Model chapter (Waiting for teacher Wang's reply)
|-
|Andy Zhang ||
*almost finish the first version of chapter3
||
*hope to finish the first version before '''''Sep 2nd''''', main challenge is to draw the pictures myself
*revise on the first version
|-
|Aiting Liu || || *prepare diagrams in chapter3
|-
|Shiyao Li ||
*Prepare the presentation of Linear Algebra and Probability Theory and Information Theory
*start to learn tensorflow
||
*learn some models by tensorflow
|-
|}

Schedule

2016-08-29T08:51:16Z

Liuaiting：/* Work Progress */

Schedule

2016-08-19T09:12:40Z

Liuaiting：/* Daily Report */

Schedule

2016-08-18T09:34:53Z

Liuaiting：/* Daily Report */

Reading table

2016-08-18T05:33:42Z

Liuaiting：/* Aiting Liu */

=Text Processing Team Schedule=

==Members==
===Former Members===
* Rong Liu (刘荣) : 优酷
* Xiaoxi Wang (王晓曦) : 图灵机器人
* Xi Ma (马习) : 清华大学研究生
* DongXu Zhang (张东旭) : --
* Yiqiao Pan (潘一桥)：继续读研

===Current Members===
* Tianyi Luo (骆天一)
* Chao Xing (邢超)
* Qixin Wang (王琪鑫)
* Aodong Li (李傲冬)
* Aiting Liu (刘艾婷)
* Ziwei Bai (白子薇)

==Work Process==
===Paper Share===
====2016-06-23====
Learning Better Embeddings for Rare Words Using Distributional Representations [http://aclweb.org/anthology/D15-1033 pdf]

Hierarchical Attention Networks for Document Classification [https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf pdf]

Hierarchical Recurrent Neural Network for Document Modeling [http://www.aclweb.org/anthology/D/D15/D15-1106.pdf pdf]

Learning Distributed Representations of Sentences from Unlabelled Data [http://arxiv.org/pdf/1602.03483.pdf pdf]

Speech Synthesis Based on HiddenMarkov Models [http://www.research.ed.ac.uk/portal/files/15269212/Speech_Synthesis_Based_on_Hidden_Markov_Models.pdf pdf]

===Research Task===
====Binary Word Embedding(Aiting)====
[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/9/97/Binary.pdf binary]

2016-06-05: find out that tensorflow does not provide logical derivation method.

2016-06-01: complete the first version of binary word embedding model

2016-05-28: complete the word2vec model in tensorflow

2016-05-25: write my own version of word2vec model

2016-05-23:

1.get tensorflow's word2vec model from(https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/embedding)
2.learn word2vec_basic model
3.run word2vec.py and word2vec_optimized.py

2016-05-22：

1.find the tf.logical_xor(x,y) method in tensorflow to compute Hamming distance.
2.learn tensorflow's word2vec model

2016-05-21：

1.read Lantian's paper 'Binary Speaker Embedding'
2.try to find a formula in tensorflow to compute Hamming distance.

====Ordered Word Embedding(Aodong)====

: 2016-07-12 : Complete the mixed initialization version of Rare Word Embedding and start training
: 2016-07-11 :
Improve the predict process of chatting model
Changing some hyperparameters of the chatting model to speed up the training process
: 2016-07-09, 10 : Try to carry out paper's low-freq word experiment, and do some readings both from PRML and Hang Li's paper.
: 2016-07-08 : Do model selections and the model finally set off on the server.
: 2016-07-07 : Complete the chatting model and run on Huilian's server, in order to overcome the GPU memory problem.
: 2016-07-06 : Finally complete translate model in tensorflow!!!! Now I am dealing with the out of memory problem.
: 2016-07-05 :
Code predict process
Although I've got a low cost value, the predict result does not compatible as expected, even the input of predict process from training set
When I tried the Weibo data, program collapsed with an out of memory error.
: 2016-07-04 : Complete Coding training process
: 2016-07-01, 02: The cost function is very bumpy, debug it, while it's quite difficult!
: 2016-06-27, 28, 29 : Coding
: 2016-06-26 : Code tf's GRU and attention model
: 2016-06-25 : Read tf's source code rnn_cell.py and seq2seq.py
: 2016-06-24 :
Code spearman correlation coefficient and experiment
Read Li's paper "Neural Responding Machine for Short-Text Conversation"
: 2016-06-23 :
Share paper "Learning Better Embeddings for Rare Words Using Distributional Representations"
experiment and receive new task
: 2016-06-22 :
Experiment on low-frequency words
Roughly read "Online Learning of Interpretable Word Embeddings"
Roughly read "Learning Better Embeddings for Rare Words Using Distributional Representations"
: 2016-06-21 : Experiment and calculate cosine distance between words
: 2016-06-20 : Something went wrong with my program and fix it, so I have to start it all over again
: 2016-06-04 : Experiment the semantic&syntactic analysis of retrained word vector
: 2016-06-03 : Complete coding retrain process of low-freq word and experiment the semantic&syntactic analysis
: 2016-06-02 : Complete coding predict process of low-freq word and experiment the semantic&syntactic analysis
: 2016-06-01 : Read "Distributed Representations of Words and Phrases and their Compositionality"
: 2016-05-31 :
Read Mikolov's ppt about his word embedding papers
test the randomness of word2vec and there is nothing different in single thread while rerunning the program
Download dataset "microsoft syntactic test set", "wordsim353", and "simlex-999"
: 2016-05-30 : Read "Hierarchical Probabilistic Neural Network Language Model" and "word2vec Explained: Deriving Mikolov's Negative-Sampling Word-Embedding Method"
: 2016-05-27 : Reread word2vec paper and read C-version word2vec.
: 2016-05-24 : Understand word2vec in TensorFlow, and because of some uncompleted functions, I determine to adapt the source of C-versioned word2vec.
: 2016-05-23 :
Basic setup of TensorFlow
Read code of word2vec in TensorFlow
: 2016-05-22 :
Learn about algorithms in word2vec
Read low-freq word papar and learn about 6 strategies

[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/3/39/How_to_deal_with_low_frequency_words.pdf low_freq]

[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/2/2c/Lowv.pdf order_rep]

====Matrix Factorization(Ziwei)====
[http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization.pdf matrix-factorization]

2016-06-23：
prepare for report
2016-05-28：
learn the code 'matrix-factorization.py','count_word_frequence.py',and 'reduce_rawtext_matrix_factorization.py'
problem:I have no idea how to run the program and where the data.

2016-05-23:
read the code 'map_rawtext_matrix_factorization.py'
2016-05-22：
learn the rest of paper ‘Neural word Embedding as implicit matrix factorization’
2016-05-21：
learn the ‘abstract’ and ‘introduction’ of paper ‘Neural word Embedding as implicit matrix factorization’

===Question answering system===

====Chao Xing====
2016-05-30 ~ 2016-06-04 :
Deliver CDSSM model to huilan.
2016-05-29 :
Package chatting model in practice.
2016-05-28 :
Modify bugs...
2016-05-27 :
Train large scale model, find some problem.
2016-05-26 :
Modify test program for large scale testing process.
2016-05-24 :
Build CDSSM model in huilan's machine.
2016-05-23 :
Find three things to do.
1. Cost function change to maximize QA+ - QA-.
2. Different parameters space in Q space and A space.
3. HRNN separate to two tricky things : use output layer or use hidden layer as decoder's softmax layer's input.
2016-05-22 :
1. Investigate different loss functions in chatting model.
2016-05-21 :
1. Hand out different research task to intern students.
2016-05-20 :
1. Testing denosing rnn generation model.
2016-05-19 :
1. Discover for denosing rnn.
2016-05-18 :
1. Modify model for crawler data.
2016-05-17 :
1. Code & Test HRNN model.
2016-05-16 :
1. Work done for CDSSM model.
2016-05-15 :
1. Test CDSSM model package version.
2016-05-13 :
1. Coding done CDSSM model package version. Wait to test.
2016-05-12 :
1. Begin to package CDSSM model for huilan.
2016-05-11 :
1. Prepare for paper sharing.
2. Finish CDSSM model in chatting process.
3. Start setup model & experiment in dialogue system.
2016-05-10 :
1. Finish test CDSSM model in chatting, find original data has some problem.
2. Read paper:
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Neural Responding Machine for Short-Text Conversation
2016-05-09 :
1. Test CDSSM model in chatting model.
2. Read paper :
Learning from Real Users Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
SimpleDS A Simple Deep Reinforcement Learning Dialogue System
3. Code RNN by myself in tensorflow.
2016-05-08 :
Fix some problem in dialogue system team, and continue read some papers in dialogue system.
2016-05-07 :
Read some papers in dialogue system.
2016-05-06 :
Try to fix RNN-DSSM model in tensorflow. Failure..
2016-05-05 :
Coding for RNN-DSSM in tensorflow. Face an error when running rnn-dssm model in cpu : memory keep increasing.
Tensorflow's version in huilan is 0.7.0 and install by pip, this cause using error in creating gpu graph,
one possible solution is build tensorflow from source code.

====Aiting Liu====

2016-07-19 ~ 2016-07-21: preprocess the 200,000 lyrics

2016-07-18: get 200,000 songs from http://www.cnlyric.com/ ( singer list from a-z)

2016-07-08: preprocess the lyrics from baidu music

2016-07-07: get 27963 songs from http://www.cnlyric.com/ (singerlist from A/B/C)

2016-07-06: try to get lyrics from http://www.kuwo.cn/ http://www.kugou.com/ http://y.qq.com/#type=index http://music.163.com/

2016-07-05: write lyrics spider, and get 56306 songs from http://music.baidu.com/

2016-07-04: learn tensorflow

2016-07-01: submit APSIPA2016 paper

2016-06-30: perfection paper

2016-06-29: complete the ordered word embedding's paper

2016-06-26: modify the ordered word embedding's paper

2016-06-25: complete ordered word embedding experiment,get 54 figures

2016-06-23: read Bengio's paper https://arxiv.org/pdf/1605.06069v3.pdf

2016-06-22: read Bengio's paper http://arxiv.org/pdf/1507.04808v3.pdf

2016-06-13:

[[文件:Classification.jpg]]

2016-06-12:

[[文件:Similarity.jpg]]

2016-06-05: complete the binary word embedding, find out that tensorflow does not provide logical derivation method.

2016-06-04: write the binary word embedding model

2016-06-01:

1.Record demo video of our Personalized Chatterbot
2.program the binary word embedding model

2016-05-31: debugging our Personalized Chatterbot

2016-05-30: complete our Personalized Chatterbot

2016-05-29:

1.scan Chao's code and modify it
2.run the modified program to get the eight hundred thousand sentences's whole matrix

2016-05-28:

1.complete the word2vec model in tensorflow
2.complete the first version of binary word embedding model

2016-05-25: .write my own version of word2vec model

2016-05-23:

1.get tensorflow's word2vec model from(https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/embedding)
2.learn word2vec_basic model
3.run word2vec.py and word2vec_optimized.py,we need a Chinese evaluation dataset if we want to use it directly

2016-05-22：

1.find the tf.logical_xor(x,y) method in tensorflow to compute Hamming distance.
2.learn tensorflow's word2vec model

2016-05-21：

1.read Lantian's paper 'Binary Speaker Embedding'
2.try to find a formula in tensorflow to compute Hamming distance.

2016-05-18：

Fetch American TV subtitles and process them into a specific format(12.6M)
(1.Sex and the City 2.Gossip Girl 3.Desperate Housewives 4.The IT Crowd 5.Empire 6.2 Broke Girls)

2016-05-16：Process the data collected from the interview site,interview books and American TV subtitles(38.2M+23.2M)

2016-05-11：

Fetch American TV subtitles
(1.Friends 2.Big Bang Theory 3.The descendant of the Sun 4.Modern Family 5.House M.D. 6.Grey's Anatomy)

2016-05-08：Fetch data from 'http://news.ifeng.com/' and 'http://www.xinhuanet.com/'(13.4M)

2016-05-07：Fetch data from 'http://fangtan.china.com.cn/' and interview books (10M)

2016-05-04：Establish the overall framework of our chat robot,and continue to build database

====Ziwei Bai====
2016-07-18 ~ 2016-07-20:
1、run bottleneck model with different parameters
2、 prepare Bi-weekly report
3、draw a map to compare different model,
2016-07-14 ~ 2016-07-15：
build bottleneck model（non linear layer : sigmoid relu tanh）
2016-07-12 ~ 2016-07-13:
modify the TTS program
1、separate classify and transfer
2、separate lf0 and mgc
2016-07-11：
finish the patent
2016-07-07 ~ 2016-07-08:
1、program LSTM with tensorflow （still has some bug）
2、learn paper 'Fast,Compact,and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers fot Mobile Devices'
2016-07-06:
finish the second edition of patent
2016-07-05：
finish the fisrt edition of patent
2016-07-04：
1、debug and run the chatting model with softmax
2、determine model for patent of ‘LSTM-based modern text to the poetry conversion technology’
2016-07-01：
the model updated yesterday can't converge，try to learn tf.sampled_softmax_loss()
2016-06-30:
convert our chatting model from Negative sample to softmax and convert the cost from cosine to cross-entropy
tf.softmax()
2016-06-29：
learn paper 'Neural Responding Machine for Short-Text Conversation'
2016-06-23：
learn paper ‘Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models’
http://arxiv.org/pdf/1507.04808v3.pdf
2016-06-22：
1、construct vector for word cut by jieba
2、retrain the cdssm model with new word vector(still run)
2016-06-04：
1、modify the interface for QA system
2、pull together the interface and QA system
2016-06-01：
1、add data source and Performance Test results in work report
2、learn pyQt

2016-05-30：
complete the work report
2016-05-29：
write code for inputting a question ,return a answer sets whose question is most similar to the input question
2016-05-25:
1、learn DSSM
2、 complete the first edition of work report
3、construct basic Q&A（name，age，job and so on）
2016-05-23：
write code for searching question in 'zhihu.sogou.com' and searching answer in zhihu
2016-05-21：
learn the second half of paper 'A Neural Conversational Model'
2016-05-18:
1、crawl QA pairs from http://www.chinalife.com.cn/publish/zhuzhan/index.html and http://www.pingan.com/
2、find paper 'A Neural Conversational Model' from google scholar and learn the first half of it.
2016-05-16:
1、find datasets in paper 'Neural Responding Machine for Short-Text Conversation'
2、reconstruct 15 scripts into our expected formula
2016-05-15:
1、find 130 scripts
2、 reconstruct 11 scripts into our expected formula
problem：many files cann't distinguish between dialogue and scenario describes by program.

2016-05-11:
1、read paper“Movie-DiC: a Movie Dialogue Corpus for Research and Development”
2、reconstruct a new film scripts into our expected formula

2016-05-08: convert the pdf we found yesterday into txt，and reconstruct the data into our expected formula

2016-05-07: Finding 9 Drama scripts and 20 film scripts

2016-05-04：Finding and dealing with the data for QA system

===Generation Model (Aodong li)===

: 2016-05-21 : Complete my biweekly report and take over new tasks -- low-frequency words
: 2016-05-20 :
Optimize my code to speed up
Train the models with GPU
However, it does not converge :(
: 2016-05-19 : Code a simple version of keywords-to-sequence model and train the model
: 2016-05-18 : Debug keywords-to-sequence model and train the model
: 2016-05-17 : make technical details clear and code keywords-to-sequence model
: 2016-05-16 : Denoise and segment more lyrics and prepare for keywords to sequence model
: 2016-05-15 : Train some different models and analyze performance: song to song, paragraph to paragraph, etc.
: 2016-05-12 : complete sequence to sequence model's prediction process and the whole standard sequence to sequence lstm-based model v0.0
: 2016-05-11 : complete sequence to sequence model's training process in Theano
: 2016-05-10 : complete sequence to sequence lstm-based model in Theano
: 2016-05-09 : try to code sequence to sequence model
: 2016-05-08 :
denoise and train word vectors of Lijun Deng's lyrics (110+ pieces)
decide on using raw sequence to sequence model
: 2016-05-07 :
study attention-based model
learn some details about the poem generation model
change my focus onto lyrics generation model
: 2016-05-06 : read the paper about poem generation and learn about LSTM
: 2016-05-05 : check in and have an overview of generation model

===jiyuan zhang===
: 2016-05-01~06 :modify input format and run lstmrbm model (16-beat,32-beat,bar)
: 2016-05-09~13:
Modify model parameters and run model ，the result is not ideal yet
According to teacher Wang's opinion, in the generation stage,replace random generation with the maximum probability generation

: 2016-05-24~27 :check the blog's codes and understand the model and input format details on the blog

==Past progress==

[[nlp-progress-2016-05]]

[[nlp-progress-2016-04]]

Schedule

2016-07-21T05:45:48Z

Liuaiting：/* Aiting Liu */

=Text Processing Team Schedule=

==Members==
===Former Members===
* Rong Liu (刘荣) : 优酷
* Xiaoxi Wang (王晓曦) : 图灵机器人
* Xi Ma (马习) : 清华大学研究生
* DongXu Zhang (张东旭) : --
* Yiqiao Pan (潘一桥)：继续读研

===Current Members===
* Tianyi Luo (骆天一)
* Chao Xing (邢超)
* Qixin Wang (王琪鑫)
* Aodong Li (李傲冬)
* Aiting Liu (刘艾婷)
* Ziwei Bai (白子薇)

==Work Process==
===Paper Share===
====2016-06-23====
Learning Better Embeddings for Rare Words Using Distributional Representations [http://aclweb.org/anthology/D15-1033 pdf]

Hierarchical Attention Networks for Document Classification [https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf pdf]

Hierarchical Recurrent Neural Network for Document Modeling [http://www.aclweb.org/anthology/D/D15/D15-1106.pdf pdf]

Learning Distributed Representations of Sentences from Unlabelled Data [http://arxiv.org/pdf/1602.03483.pdf pdf]

Speech Synthesis Based on HiddenMarkov Models [http://www.research.ed.ac.uk/portal/files/15269212/Speech_Synthesis_Based_on_Hidden_Markov_Models.pdf pdf]

===Research Task===
====Binary Word Embedding(Aiting)====
[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/9/97/Binary.pdf binary]

2016-06-05: find out that tensorflow does not provide logical derivation method.

2016-06-01: complete the first version of binary word embedding model

2016-05-28: complete the word2vec model in tensorflow

2016-05-25: write my own version of word2vec model

2016-05-23:

1.get tensorflow's word2vec model from(https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/embedding)
2.learn word2vec_basic model
3.run word2vec.py and word2vec_optimized.py

2016-05-22：

1.find the tf.logical_xor(x,y) method in tensorflow to compute Hamming distance.
2.learn tensorflow's word2vec model

2016-05-21：

1.read Lantian's paper 'Binary Speaker Embedding'
2.try to find a formula in tensorflow to compute Hamming distance.

====Ordered Word Embedding(Aodong)====

: 2016-07-12 : Complete the mixed initialization version of Rare Word Embedding and start training
: 2016-07-11 :
Improve the predict process of chatting model
Changing some hyperparameters of the chatting model to speed up the training process
: 2016-07-09, 10 : Try to carry out paper's low-freq word experiment, and do some readings both from PRML and Hang Li's paper.
: 2016-07-08 : Do model selections and the model finally set off on the server.
: 2016-07-07 : Complete the chatting model and run on Huilian's server, in order to overcome the GPU memory problem.
: 2016-07-06 : Finally complete translate model in tensorflow!!!! Now I am dealing with the out of memory problem.
: 2016-07-05 :
Code predict process
Although I've got a low cost value, the predict result does not compatible as expected, even the input of predict process from training set
When I tried the Weibo data, program collapsed with an out of memory error.
: 2016-07-04 : Complete Coding training process
: 2016-07-01, 02: The cost function is very bumpy, debug it, while it's quite difficult!
: 2016-06-27, 28, 29 : Coding
: 2016-06-26 : Code tf's GRU and attention model
: 2016-06-25 : Read tf's source code rnn_cell.py and seq2seq.py
: 2016-06-24 :
Code spearman correlation coefficient and experiment
Read Li's paper "Neural Responding Machine for Short-Text Conversation"
: 2016-06-23 :
Share paper "Learning Better Embeddings for Rare Words Using Distributional Representations"
experiment and receive new task
: 2016-06-22 :
Experiment on low-frequency words
Roughly read "Online Learning of Interpretable Word Embeddings"
Roughly read "Learning Better Embeddings for Rare Words Using Distributional Representations"
: 2016-06-21 : Experiment and calculate cosine distance between words
: 2016-06-20 : Something went wrong with my program and fix it, so I have to start it all over again
: 2016-06-04 : Experiment the semantic&syntactic analysis of retrained word vector
: 2016-06-03 : Complete coding retrain process of low-freq word and experiment the semantic&syntactic analysis
: 2016-06-02 : Complete coding predict process of low-freq word and experiment the semantic&syntactic analysis
: 2016-06-01 : Read "Distributed Representations of Words and Phrases and their Compositionality"
: 2016-05-31 :
Read Mikolov's ppt about his word embedding papers
test the randomness of word2vec and there is nothing different in single thread while rerunning the program
Download dataset "microsoft syntactic test set", "wordsim353", and "simlex-999"
: 2016-05-30 : Read "Hierarchical Probabilistic Neural Network Language Model" and "word2vec Explained: Deriving Mikolov's Negative-Sampling Word-Embedding Method"
: 2016-05-27 : Reread word2vec paper and read C-version word2vec.
: 2016-05-24 : Understand word2vec in TensorFlow, and because of some uncompleted functions, I determine to adapt the source of C-versioned word2vec.
: 2016-05-23 :
Basic setup of TensorFlow
Read code of word2vec in TensorFlow
: 2016-05-22 :
Learn about algorithms in word2vec
Read low-freq word papar and learn about 6 strategies

[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/3/39/How_to_deal_with_low_frequency_words.pdf low_freq]

[http://cslt.riit.tsinghua.edu.cn/mediawiki/images/2/2c/Lowv.pdf order_rep]

====Matrix Factorization(Ziwei)====
[http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization.pdf matrix-factorization]

2016-06-23：
prepare for report
2016-05-28：
learn the code 'matrix-factorization.py','count_word_frequence.py',and 'reduce_rawtext_matrix_factorization.py'
problem:I have no idea how to run the program and where the data.

2016-05-23:
read the code 'map_rawtext_matrix_factorization.py'
2016-05-22：
learn the rest of paper ‘Neural word Embedding as implicit matrix factorization’
2016-05-21：
learn the ‘abstract’ and ‘introduction’ of paper ‘Neural word Embedding as implicit matrix factorization’

===Question answering system===

====Chao Xing====
2016-05-30 ~ 2016-06-04 :
Deliver CDSSM model to huilan.
2016-05-29 :
Package chatting model in practice.
2016-05-28 :
Modify bugs...
2016-05-27 :
Train large scale model, find some problem.
2016-05-26 :
Modify test program for large scale testing process.
2016-05-24 :
Build CDSSM model in huilan's machine.
2016-05-23 :
Find three things to do.
1. Cost function change to maximize QA+ - QA-.
2. Different parameters space in Q space and A space.
3. HRNN separate to two tricky things : use output layer or use hidden layer as decoder's softmax layer's input.
2016-05-22 :
1. Investigate different loss functions in chatting model.
2016-05-21 :
1. Hand out different research task to intern students.
2016-05-20 :
1. Testing denosing rnn generation model.
2016-05-19 :
1. Discover for denosing rnn.
2016-05-18 :
1. Modify model for crawler data.
2016-05-17 :
1. Code & Test HRNN model.
2016-05-16 :
1. Work done for CDSSM model.
2016-05-15 :
1. Test CDSSM model package version.
2016-05-13 :
1. Coding done CDSSM model package version. Wait to test.
2016-05-12 :
1. Begin to package CDSSM model for huilan.
2016-05-11 :
1. Prepare for paper sharing.
2. Finish CDSSM model in chatting process.
3. Start setup model & experiment in dialogue system.
2016-05-10 :
1. Finish test CDSSM model in chatting, find original data has some problem.
2. Read paper:
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Neural Responding Machine for Short-Text Conversation
2016-05-09 :
1. Test CDSSM model in chatting model.
2. Read paper :
Learning from Real Users Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
SimpleDS A Simple Deep Reinforcement Learning Dialogue System
3. Code RNN by myself in tensorflow.
2016-05-08 :
Fix some problem in dialogue system team, and continue read some papers in dialogue system.
2016-05-07 :
Read some papers in dialogue system.
2016-05-06 :
Try to fix RNN-DSSM model in tensorflow. Failure..
2016-05-05 :
Coding for RNN-DSSM in tensorflow. Face an error when running rnn-dssm model in cpu : memory keep increasing.
Tensorflow's version in huilan is 0.7.0 and install by pip, this cause using error in creating gpu graph,
one possible solution is build tensorflow from source code.

====Aiting Liu====

2016-07-19 ~ 2016-07-21: preprocess the 20,000 lyrics

2016-07-18: get twenty thousand songs from http://www.cnlyric.com/ ( singer list from a-z)

2016-07-08: preprocess the lyrics from baidu music

2016-07-07: get 27963 songs from http://www.cnlyric.com/ (singerlist from A/B/C)

2016-07-06: try to get lyrics from http://www.kuwo.cn/ http://www.kugou.com/ http://y.qq.com/#type=index http://music.163.com/

2016-07-05: write lyrics spider, and get 56306 songs from http://music.baidu.com/

2016-07-04: learn tensorflow

2016-07-01: submit APSIPA2016 paper

2016-06-30: perfection paper

2016-06-29: complete the ordered word embedding's paper

2016-06-26: modify the ordered word embedding's paper

2016-06-25: complete ordered word embedding experiment,get 54 figures

2016-06-23: read Bengio's paper https://arxiv.org/pdf/1605.06069v3.pdf

2016-06-22: read Bengio's paper http://arxiv.org/pdf/1507.04808v3.pdf

2016-06-13:

[[文件:Classification.jpg]]

2016-06-12:

[[文件:Similarity.jpg]]

2016-06-05: complete the binary word embedding, find out that tensorflow does not provide logical derivation method.

2016-06-04: write the binary word embedding model

2016-06-01:

1.Record demo video of our Personalized Chatterbot
2.program the binary word embedding model

2016-05-31: debugging our Personalized Chatterbot

2016-05-30: complete our Personalized Chatterbot

2016-05-29:

1.scan Chao's code and modify it
2.run the modified program to get the eight hundred thousand sentences's whole matrix

2016-05-28:

1.complete the word2vec model in tensorflow
2.complete the first version of binary word embedding model

2016-05-25: .write my own version of word2vec model

2016-05-23:

1.get tensorflow's word2vec model from(https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/embedding)
2.learn word2vec_basic model
3.run word2vec.py and word2vec_optimized.py,we need a Chinese evaluation dataset if we want to use it directly

2016-05-22：

1.find the tf.logical_xor(x,y) method in tensorflow to compute Hamming distance.
2.learn tensorflow's word2vec model

2016-05-21：

1.read Lantian's paper 'Binary Speaker Embedding'
2.try to find a formula in tensorflow to compute Hamming distance.

2016-05-18：

Fetch American TV subtitles and process them into a specific format(12.6M)
(1.Sex and the City 2.Gossip Girl 3.Desperate Housewives 4.The IT Crowd 5.Empire 6.2 Broke Girls)

2016-05-16：Process the data collected from the interview site,interview books and American TV subtitles(38.2M+23.2M)

2016-05-11：

Fetch American TV subtitles
(1.Friends 2.Big Bang Theory 3.The descendant of the Sun 4.Modern Family 5.House M.D. 6.Grey's Anatomy)

2016-05-08：Fetch data from 'http://news.ifeng.com/' and 'http://www.xinhuanet.com/'(13.4M)

2016-05-07：Fetch data from 'http://fangtan.china.com.cn/' and interview books (10M)

2016-05-04：Establish the overall framework of our chat robot,and continue to build database

====Ziwei Bai====
2016-07-18 ~ 2016-07-20:
1、run bottleneck model with different parameters
2、 prepare Bi-weekly report
3、draw a map to compare different model,
2016-07-14 ~ 2016-07-15：
build bottleneck model（non linear layer : sigmoid relu tanh）
2016-07-12 ~ 2016-07-13:
modify the TTS program
1、separate classify and transfer
2、separate lf0 and mgc
2016-07-11：
finish the patent
2016-07-07 ~ 2016-07-08:
1、program LSTM with tensorflow （still has some bug）
2、learn paper 'Fast,Compact,and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers fot Mobile Devices'
2016-07-06:
finish the second edition of patent
2016-07-05：
finish the fisrt edition of patent
2016-07-04：
1、debug and run the chatting model with softmax
2、determine model for patent of ‘LSTM-based modern text to the poetry conversion technology’
2016-07-01：
the model updated yesterday can't converge，try to learn tf.sampled_softmax_loss()
2016-06-30:
convert our chatting model from Negative sample to softmax and convert the cost from cosine to cross-entropy
tf.softmax()
2016-06-29：
learn paper 'Neural Responding Machine for Short-Text Conversation'
2016-06-23：
learn paper ‘Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models’
http://arxiv.org/pdf/1507.04808v3.pdf
2016-06-22：
1、construct vector for word cut by jieba
2、retrain the cdssm model with new word vector(still run)
2016-06-04：
1、modify the interface for QA system
2、pull together the interface and QA system
2016-06-01：
1、add data source and Performance Test results in work report
2、learn pyQt

2016-05-30：
complete the work report
2016-05-29：
write code for inputting a question ,return a answer sets whose question is most similar to the input question
2016-05-25:
1、learn DSSM
2、 complete the first edition of work report
3、construct basic Q&A（name，age，job and so on）
2016-05-23：
write code for searching question in 'zhihu.sogou.com' and searching answer in zhihu
2016-05-21：
learn the second half of paper 'A Neural Conversational Model'
2016-05-18:
1、crawl QA pairs from http://www.chinalife.com.cn/publish/zhuzhan/index.html and http://www.pingan.com/
2、find paper 'A Neural Conversational Model' from google scholar and learn the first half of it.
2016-05-16:
1、find datasets in paper 'Neural Responding Machine for Short-Text Conversation'
2、reconstruct 15 scripts into our expected formula
2016-05-15:
1、find 130 scripts
2、 reconstruct 11 scripts into our expected formula
problem：many files cann't distinguish between dialogue and scenario describes by program.

2016-05-11:
1、read paper“Movie-DiC: a Movie Dialogue Corpus for Research and Development”
2、reconstruct a new film scripts into our expected formula

2016-05-08: convert the pdf we found yesterday into txt，and reconstruct the data into our expected formula

2016-05-07: Finding 9 Drama scripts and 20 film scripts

2016-05-04：Finding and dealing with the data for QA system

===Generation Model (Aodong li)===

: 2016-05-21 : Complete my biweekly report and take over new tasks -- low-frequency words
: 2016-05-20 :
Optimize my code to speed up
Train the models with GPU
However, it does not converge :(
: 2016-05-19 : Code a simple version of keywords-to-sequence model and train the model
: 2016-05-18 : Debug keywords-to-sequence model and train the model
: 2016-05-17 : make technical details clear and code keywords-to-sequence model
: 2016-05-16 : Denoise and segment more lyrics and prepare for keywords to sequence model
: 2016-05-15 : Train some different models and analyze performance: song to song, paragraph to paragraph, etc.
: 2016-05-12 : complete sequence to sequence model's prediction process and the whole standard sequence to sequence lstm-based model v0.0
: 2016-05-11 : complete sequence to sequence model's training process in Theano
: 2016-05-10 : complete sequence to sequence lstm-based model in Theano
: 2016-05-09 : try to code sequence to sequence model
: 2016-05-08 :
denoise and train word vectors of Lijun Deng's lyrics (110+ pieces)
decide on using raw sequence to sequence model
: 2016-05-07 :
study attention-based model
learn some details about the poem generation model
change my focus onto lyrics generation model
: 2016-05-06 : read the paper about poem generation and learn about LSTM
: 2016-05-05 : check in and have an overview of generation model

===jiyuan zhang===
: 2016-05-01~06 :modify input format and run lstmrbm model (16-beat,32-beat,bar)
: 2016-05-09~13:
Modify model parameters and run model ，the result is not ideal yet
According to teacher Wang's opinion, in the generation stage,replace random generation with the maximum probability generation

: 2016-05-24~27 :check the blog's codes and understand the model and input format details on the blog

==Past progress==

[[nlp-progress-2016-05]]

[[nlp-progress-2016-04]]