[go: up one dir, main page]

Information and Media Technologies
Online ISSN : 1881-0896
ISSN-L : 1881-0896
Media (processing) and Interaction
An Efficient Algorithm for Unsupervised Word Segmentation with Branching Entropy and MDL
Valentin ZhikovHiroya TakamuraManabu Okumura
Author information
JOURNAL FREE ACCESS

2013 Volume 8 Issue 2 Pages 514-527

Details
Abstract

This paper proposes a fast and simple unsupervised word segmentation algorithm that utilizes the local pre-dictability of adjacent character sequences, while searching for a least-effort representation of the data. The model uses branching entropy as a means of constraining the hypothesis space, in order to efficiently obtain a solution that minimizes the length of a two-part MDL code. An evaluation with corpora in Japanese, Thai, English, and the “CHILDES” corpus for research in language development reveals that the algorithm achieves a F-score, comparable to that of the state-of-the-art methods in unsupervised word segmentation, in a significantly reduced computational time. In view of its capability to induce the vocabulary of large-scale corpora of domain-specific text, the method has potential to improve the coverage of morphological analyzers for languages without explicit word boundary markers. A semi-supervised word segmentation approach is also proposed, in which the word boundaries obtained through the unsupervised model are used as features for a state-of-the-art word segmentation method.

Content from these authors
© 2013 Japanese Society for Artificial Intelligence
Previous article Next article
feedback
Top