org.cleartk.token.tokenizer
Class Tokenizer_ImplBase

java.lang.Object
  org.cleartk.token.tokenizer.Tokenizer_ImplBase

All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

PennTreebankTokenizer, Subtokenizer

public abstract class Tokenizer_ImplBase
extends Object
implements Tokenizer

Copyright (c) 2007-2008, Regents of the University of Colorado
All rights reserved.

Constructor Summary

Tokenizer_ImplBase()

Method Summary

List<Token>	getTokens(String text) performs tokenization on the input text and returns the character offsets corresponding to the begining and end of each token w.r.t.
abstract String[]	getTokenTexts(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer_ImplBase

public Tokenizer_ImplBase()

Method Detail

getTokenTexts

public abstract String[] getTokenTexts(String text)

getTokens

public List<Token> getTokens(String text)

performs tokenization on the input text and returns the character offsets corresponding to the begining and end of each token w.r.t. the input text.

Specified by:

getTokens in interface Tokenizer

Parameters:

text - the text that you want tokenized.

Returns:

offsets corresponding to the begining and end of each token. The dimensions of the returned array will be number of tokens returned by getTokens by 2 (one for token begin and one for token end).

 
          int[][] offsets = getTokenOffsets("example text"); int
         beginFirstToken = offsets[0][0]; int endFirstToken =
         offsets[0][1]; int begin4thToken = offsets[3][0]; int end4thToken
         = offsets[3][1];