org.canova.nlp.vectorizer

Class TextVectorizer<VECTOR_TYPE>

java.lang.Object

org.canova.nlp.vectorizer.TextVectorizer<VECTOR_TYPE>

All Implemented Interfaces:

Vectorizer<VECTOR_TYPE>

Direct Known Subclasses:

TfidfVectorizer

public abstract class TextVectorizer<VECTOR_TYPE>
extends Object
implements Vectorizer<VECTOR_TYPE>

Baseline text vectorizer that includes some common elements to text analysis such as the tokenizer factory

Author:

Adam Gibson

Nested Class Summary

Nested classes/interfaces inherited from interface org.canova.api.vector.Vectorizer

Vectorizer.RecordCallBack

Field Summary

Fields

Modifier and Type	Field and Description
protected VocabCache	cache
static String	MIN_WORD_FREQUENCY
protected int	minWordFrequency
static String	STOP_WORDS
protected Collection<String>	stopWords
static String	TOKENIZER
protected TokenizerFactory	tokenizerFactory

Constructor Summary

Constructors

Constructor and Description
TextVectorizer()

Method Summary

Modifier and Type	Method and Description
abstract TokenizerFactory	createTokenizerFactory(Configuration conf) Create tokenizer factory based on the configuration
abstract void	doWithTokens(Tokenizer tokenizer) Increment counts, add to collection,...
void	fit(RecordReader reader)
void	fit(RecordReader reader, Vectorizer.RecordCallBack callBack)
void	initialize(Configuration conf)
protected String	toString(Collection<Writable> record)
protected Counter<String>	wordFrequenciesForRecord(Collection<Writable> record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.canova.api.vector.Vectorizer

createVector, fitTransform, fitTransform, transform

Field Detail

tokenizerFactory

protected TokenizerFactory tokenizerFactory

minWordFrequency

protected int minWordFrequency

MIN_WORD_FREQUENCY

public static final String MIN_WORD_FREQUENCY

See Also:

Constant Field Values

STOP_WORDS

public static final String STOP_WORDS

See Also:

Constant Field Values

TOKENIZER

public static final String TOKENIZER

See Also:

Constant Field Values

stopWords

protected Collection<String> stopWords

cache

protected VocabCache cache

Constructor Detail

TextVectorizer

public TextVectorizer()

Method Detail

initialize

public void initialize(Configuration conf)

Specified by:

initialize in interface Vectorizer<VECTOR_TYPE>

fit

public void fit(RecordReader reader)

Specified by:

fit in interface Vectorizer<VECTOR_TYPE>

fit

public void fit(RecordReader reader,
                Vectorizer.RecordCallBack callBack)

Specified by:

fit in interface Vectorizer<VECTOR_TYPE>

wordFrequenciesForRecord

protected Counter<String> wordFrequenciesForRecord(Collection<Writable> record)

toString

protected String toString(Collection<Writable> record)

doWithTokens

public abstract void doWithTokens(Tokenizer tokenizer)

Increment counts, add to collection,...

Parameters:

tokenizer -

createTokenizerFactory

public abstract TokenizerFactory createTokenizerFactory(Configuration conf)

Create tokenizer factory based on the configuration

Parameters:

conf - the configuration to use

Returns:

the tokenizer factory based on the configuration