Patent Number: 6,311,152

Title: System for chinese tokenization and named entity recognition

Abstract: A system (100, 200) for tokenization and named entity recognition of ideographic language is disclosed. In the system, a word lattice is generated for a string of ideographic characters using finite state grammars (150) and a system lexicon (240). Segmented text is generated by determining word boundaries in the string of ideographic characters using the word lattice dependent upon a contextual language model (152A) and one or more entity language models (152B). One or more named entities is recognized in the string of ideographic characters using the word lattice dependent upon the contextual language model (152A) and the one or more entity language models (152B). The contextual language model (152A) and the one or more entity language models (152B) are each class-based language models. The lexicon (240) includes single ideographic characters, words, and predetermined features of the characters and words.

Inventors: Bai; Shuanhu (Singapore, SG), Wu; Horng Jyh Paul (Singapore, SG), Li; Haizhou (Singapore, SG), Loudon; Gareth (Singapore, SG)

Assignee: Kent Ridge Digital Labs

International Classification: G06F 17/28 (20060101); G06F 017/27 (); G06F 017/20 (); G10L 015/18 ()

Expiration Date: 10/30/2018