Method and system for language identification
A method and system for language identification are provided. The system includes a feature set of a plurality of character strings of varying length with associated information. The associated information includes one or more significance scores for a character string for one or more of a plurality of languages. Means are provided for detecting character strings from the feature set within a token from an input text. The system uses a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string. The associated information can also include significance scores based on linguistic rules.
International Business Machines Corporation
G G G
G06F G06F G06F
inventional inventional inventional
Alexandre Daniel Alexei Brian Richard Mayo Pavel
Troussov McCloskey Nevidomski O'Donovan Carlgren Takeuchi Volkov