|
||||||||||
| PREV NEXT | FRAMES NO FRAMES | |||||||||
| Packages that use Token | |
|---|---|
| org.cleartk.token.tokenizer | |
| Uses of Token in org.cleartk.token.tokenizer |
|---|
| Methods in org.cleartk.token.tokenizer that return types with arguments of type Token | |
|---|---|
List<Token> |
Tokenizer.getTokens(String text)
|
List<Token> |
Tokenizer_ImplBase.getTokens(String text)
performs tokenization on the input text and returns the character offsets corresponding to the begining and end of each token w.r.t. |
|
||||||||||
| PREV NEXT | FRAMES NO FRAMES | |||||||||