Document classification test

Problem And Solve

• How to import the sparse data of vsm to weka

Document classification of Sougou data

• DATA

• Data from SougouLab [1],using SogouC.reduced(30M)
• 9-Classes:财经,IT,健康,体育,旅游,教育,招聘,文化,军事
• train and test: train(),test(),dev()

• Text preprocessing

• Segment word using wordlist of 9W.(tencent)
• Remove stop word.stop_wordlist is

• Some Tools

• weka
• scw
• google word2ve
• LDA

• class map

C000007 汽车 C000008 财经 C000010 IT C000013 健康 C000014 体育 C000016 旅游 C000020 教育 C000022 招聘 C000023 文化 C000024 军事

VSM Test

• Data

• dimension:9402

• Method

• document reprenstion: use the tf-idf weight for word weight
• classifier: Native Bayes

• Result

classification result

	财经	IT	健康	体育	旅游	教育	招聘	文化	军事	sum
ACC-test	0.72139	0.72139	0.75124	0.82089	0.79602	0.61194	0.70647	0.79104	0.72913
ACC-train	0.678	0.718	0.708	0.708	0.73