org.cleartk.token.tokenizer
Class Subtokenizer

java.lang.Object
  org.cleartk.token.tokenizer.Tokenizer_ImplBase
      org.cleartk.token.tokenizer.Subtokenizer

All Implemented Interfaces:

Tokenizer

public class Subtokenizer
extends Tokenizer_ImplBase

Copyright (c) 2007-2008, Regents of the University of Colorado
All rights reserved.

This tokenizer is named 'Subtokenizer' because it is designed to over-generate tokens that can then be used as input for another tokenization approach. Specifically, this tokenizer can be used as input for BIO-styled tokenization using a classifier. Each token generated by this tokenizer would be assigned a B-TOKEN (or something similar) or I-TOKEN for a given set of gold-standard tokens or based on the results of a classifier.

Please see the corresponding unit tests for examples of how this tokenizer produces tokens.

Author:

Philip

Field Summary

static Pattern	multipleWhitespacePattern
static String	multipleWhitespaceRegex
static Pattern	subtokensPattern
static String	subtokensRegex

Constructor Summary

Subtokenizer()

Method Summary

String[]

getTokenTexts(String text)

Methods inherited from class org.cleartk.token.tokenizer.Tokenizer_ImplBase

getTokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

subtokensRegex

public static String subtokensRegex

subtokensPattern

public static Pattern subtokensPattern

multipleWhitespaceRegex

public static String multipleWhitespaceRegex

multipleWhitespacePattern

public static Pattern multipleWhitespacePattern

Constructor Detail

Subtokenizer

public Subtokenizer()

Method Detail

getTokenTexts

public String[] getTokenTexts(String text)

Specified by:

getTokenTexts in class Tokenizer_ImplBase