eu.monnetproject.translation.chunker

Class FairlyGoodTokenizer

java.lang.Object

eu.monnetproject.translation.chunker.FairlyGoodTokenizer

public class FairlyGoodTokenizer
extends Object

Fairly good tokenizer all-purpose, all-weather model (TM) See http://maccaronic.wordpress.com/2012/11/20/a-simple-regular-expression-for-tokenization-of-most-natural-language/

Author:

John McCrae

Constructor Summary

Constructors

Constructor and Description
FairlyGoodTokenizer()

Method Summary

Modifier and Type	Method and Description
static String[]	split(String string)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FairlyGoodTokenizer

public FairlyGoodTokenizer()

Method Detail

split

public static String[] split(String string)