Segmentation-Free Word Embedding for Unsegmented Languages

Takamasa Oshikiri


Abstract
In this paper, we propose a new pipeline of word embedding for unsegmented languages, called segmentation-free word embedding, which does not require word segmentation as a preprocessing step. Unlike space-delimited languages, unsegmented languages, such as Chinese and Japanese, require word segmentation as a preprocessing step. However, word segmentation, that often requires manually annotated resources, is difficult and expensive, and unavoidable errors in word segmentation affect downstream tasks. To avoid these problems in learning word vectors of unsegmented languages, we consider word co-occurrence statistics over all possible candidates of segmentations based on frequent character n-grams instead of segmented sentences provided by conventional word segmenters. Our experiments of noun category prediction tasks on raw Twitter, Weibo, and Wikipedia corpora show that the proposed method outperforms the conventional approaches that require word segmenters.
Anthology ID:
D17-1080
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
767–772
Language:
URL:
https://aclanthology.org/D17-1080
DOI:
10.18653/v1/D17-1080
Bibkey:
Cite (ACL):
Takamasa Oshikiri. 2017. Segmentation-Free Word Embedding for Unsegmented Languages. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 767–772, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Segmentation-Free Word Embedding for Unsegmented Languages (Oshikiri, EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1080.pdf