org.cleartk.token.tokenizer
Class PennTreebankTokenizer

java.lang.Object
  org.cleartk.token.tokenizer.Tokenizer_ImplBase
      org.cleartk.token.tokenizer.PennTreebankTokenizer

All Implemented Interfaces:

Tokenizer

public class PennTreebankTokenizer
extends Tokenizer_ImplBase

Copyright (c) 2007-2008, Regents of the University of Colorado
All rights reserved.

Author:

Philip Ogren This code was derived directly (i.e. translated) from a python script that Steven Bethard gave me called tokenizer.py. This copyright of this script is owned by the Regents of the University of Colorado and therefore there is no restriction on our use and distribution of this derivitive work here. The python script was derived from a sed script available here: http://www.cis.upenn.edu/~treebank/tokenization.html Steve's script fixes several misc. errors., Philip Ogren

Field Summary

static Pattern	ampersandPattern
static String	ampersandRegex
static Pattern	beginOrEndPattern
static String	beginOrEndRegex
static Pattern	bracesPattern
static String	bracesRegex
static String	closedBracesRegex
static Pattern	colonPattern
static String	colonRegex
static Pattern	commaPattern
static String	commaRegex
static Pattern	dashPattern
static String	dashRegex
static Pattern	dollarSignPattern
static String	dollarSignRegex
static Pattern	doubleQuotePattern
static String	doubleQuoteRegex
static Pattern	ellipsisPattern
static String	ellipsisRegex
static Pattern	extraSpacePattern
static String	extraSpaceRegex
static Pattern	multipleWhitespacePattern
static String	multipleWhitespaceRegex
static Pattern	nonFinalPunctPattern
static String	nonFinalPunctRegex
static Pattern	nonPeriodPunctPattern
static String	nonPeriodPunctRegex
static Pattern	oneWordAbbreviationPattern
static String	oneWordAbbreviationRegex
static String	openBracesRegex
protected Pattern[]	patterns
static Pattern	periodPattern
static String	periodRegex
static Pattern	quotePattern
static String	quoteRegex
static Pattern	singleQuotePattern
static String	singleQuoteRegex
static Pattern	tAbbreviationPattern
static String	tAbbreviationRegex
static Pattern[]	threeWordAbbreviationPatterns
static String[]	threeWordAbbreviationRegexes
static Pattern	tripleQuotePattern
static String	tripleQuoteRegex
static Pattern[]	twoWordAbbreviationPatterns
static String[]	twoWordAbbreviationRegexes

Constructor Summary

PennTreebankTokenizer()

Method Summary

String[]

getTokenTexts(String text) Tokenizes the input text and returns a string array corresponding to the tokens.

Methods inherited from class org.cleartk.token.tokenizer.Tokenizer_ImplBase

getTokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

openBracesRegex

public static String openBracesRegex

closedBracesRegex

public static String closedBracesRegex

bracesRegex

public static String bracesRegex

bracesPattern

public static Pattern bracesPattern

ellipsisRegex

public static String ellipsisRegex

ellipsisPattern

public static Pattern ellipsisPattern

commaRegex

public static String commaRegex

commaPattern

public static Pattern commaPattern

dollarSignRegex

public static String dollarSignRegex

dollarSignPattern

public static Pattern dollarSignPattern

ampersandRegex

public static String ampersandRegex

ampersandPattern

public static Pattern ampersandPattern

dashRegex

public static String dashRegex

dashPattern

public static Pattern dashPattern

colonRegex

public static String colonRegex

colonPattern

public static Pattern colonPattern

nonFinalPunctRegex

public static String nonFinalPunctRegex

nonFinalPunctPattern

public static Pattern nonFinalPunctPattern

periodRegex

public static String periodRegex

periodPattern

public static Pattern periodPattern

nonPeriodPunctRegex

public static String nonPeriodPunctRegex

nonPeriodPunctPattern

public static Pattern nonPeriodPunctPattern

singleQuoteRegex

public static String singleQuoteRegex

singleQuotePattern

public static Pattern singleQuotePattern

tripleQuoteRegex

public static String tripleQuoteRegex

tripleQuotePattern

public static Pattern tripleQuotePattern

doubleQuoteRegex

public static String doubleQuoteRegex

doubleQuotePattern

public static Pattern doubleQuotePattern

quoteRegex

public static String quoteRegex

quotePattern

public static Pattern quotePattern

oneWordAbbreviationRegex

public static String oneWordAbbreviationRegex

oneWordAbbreviationPattern

public static Pattern oneWordAbbreviationPattern

twoWordAbbreviationRegexes

public static String[] twoWordAbbreviationRegexes

twoWordAbbreviationPatterns

public static Pattern[] twoWordAbbreviationPatterns

threeWordAbbreviationRegexes

public static String[] threeWordAbbreviationRegexes

threeWordAbbreviationPatterns

public static Pattern[] threeWordAbbreviationPatterns

tAbbreviationRegex

public static String tAbbreviationRegex

tAbbreviationPattern

public static Pattern tAbbreviationPattern

beginOrEndRegex

public static String beginOrEndRegex

beginOrEndPattern

public static Pattern beginOrEndPattern

extraSpaceRegex

public static String extraSpaceRegex

extraSpacePattern

public static Pattern extraSpacePattern

multipleWhitespaceRegex

public static String multipleWhitespaceRegex

multipleWhitespacePattern

public static Pattern multipleWhitespacePattern

patterns

protected Pattern[] patterns

Constructor Detail

PennTreebankTokenizer

public PennTreebankTokenizer()

Method Detail

getTokenTexts

public String[] getTokenTexts(String text)

Tokenizes the input text and returns a string array corresponding to the tokens.

Specified by:

getTokenTexts in class Tokenizer_ImplBase