org.jwat.warc
Class WarcReader

java.lang.Object
  org.jwat.warc.WarcReader

Direct Known Subclasses:

WarcReaderCompressed, WarcReaderUncompressed

public abstract class WarcReader
extends Object

Base class for WARC reader implementations.

Author:

nicl

Field Summary

protected boolean	bBlockDigest Block Digest enabled/disabled.
protected boolean	bIsCompliant Compliance status for records parsed up to now.
protected String	blockDigestAlgorithm Default block digest algorithm to use if none is present in the record.
protected String	blockDigestEncoding Default encoding scheme used to encode block digest into a string, if none is detected from the record.
protected boolean	bPayloadDigest Payload Digest enabled/disabled.
protected long	consumed Number of bytes consumed by this reader.
Diagnostics<Diagnosis>	diagnostics Reader level errors and warnings or when no record is available.
protected int	errors Aggregated number of errors encountered while parsing.
protected WarcFieldParsers	fieldParser WARC field parser used.
protected HeaderLineReader	headerLineReader Header line reader used to read the WARC headers.
protected Exception	iteratorExceptionThrown Exception thrown while using the iterator.
protected HeaderLineReader	lineReader Line reader used to read version lines.
protected String	payloadDigestAlgorithm Default payload digest algorithm to use if none is present in the record.
protected String	payloadDigestEncoding Default encoding scheme used to encode payload digest into a string, if none is detected from the record.
protected int	payloadHeaderMaxSize Max size allowed for a payload header.
protected int	warcHeaderMaxSize Max size allowed for a WARC header.
protected WarcRecord	warcRecord Current WARC record object.
protected int	warnings Aggregate number of warnings encountered while parsing.

Constructor Summary

WarcReader()

Method Summary

abstract void	close() Close current record resource(s) and input stream(s).
String	getBlockDigestAlgorithm() Get the default block digest algorithm.
boolean	getBlockDigestEnabled() Get the readers block digest on/off status.
String	getBlockDigestEncoding() Get the default block digest encoding scheme.
abstract long	getConsumed() Get number of bytes consumed by this reader.
Exception	getIteratorExceptionThrown() Gets an exception thrown in the iterator if any or null.
abstract WarcRecord	getNextRecord() Parses and gets the next record.
abstract WarcRecord	getNextRecordFrom(InputStream in, long offset) Parses and gets the next record from an Inputstream.
abstract WarcRecord	getNextRecordFrom(InputStream in, long offset, int buffer_size) Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream.
abstract long	getOffset() Get the current offset in the WARC InputStream.
String	getPayloadDigestAlgorithm() Get the default payload digest algorithm.
boolean	getPayloadDigestEnabled() Get the readers payload digest on/off status.
String	getPayloadDigestEncoding() Get the default payload digest encoding scheme.
int	getPayloadHeaderMaxSize() Get the max size allowed for a payload header.
abstract long	getStartOffset() Get the offset of the current or next WARC record.
int	getWarcHeaderMaxSize() Get the max size allowed for a WARC header.
protected void	init() Method used to initialize a readers internal state.
boolean	isCompliant() Returns a boolean indicating if all records parsed so far are compliant.
abstract boolean	isCompressed() Is this reader assuming GZip compressed input.
Iterator<WarcRecord>	iterator() Returns an Iterator over the records as they are being parsed.
boolean	setBlockDigestAlgorithm(String digestAlgorithm) Set the default block digest algorithm.
void	setBlockDigestEnabled(boolean enabled) Set the readers block digest on/off status.
void	setBlockDigestEncoding(String encodingScheme) Set the default block digest encoding scheme.
boolean	setPayloadDigestAlgorithm(String digestAlgorithm) Set the default payload digest algorithm.
void	setPayloadDigestEnabled(boolean enabled) Set the readers payload digest on/off status.
void	setPayloadDigestEncoding(String encodingScheme) Set the default payload digest encoding scheme.
void	setPayloadHeaderMaxSize(int size) Set the max size allowed for a payload header.
void	setWarcHeaderMaxSize(int size) Set the max size allowed for a WARC header.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

bIsCompliant

protected boolean bIsCompliant

Compliance status for records parsed up to now.

consumed

protected long consumed

Number of bytes consumed by this reader.

errors

protected int errors

Aggregated number of errors encountered while parsing.

warnings

protected int warnings

Aggregate number of warnings encountered while parsing.

bBlockDigest

protected boolean bBlockDigest

Block Digest enabled/disabled.

blockDigestAlgorithm

protected String blockDigestAlgorithm

Default block digest algorithm to use if none is present in the record.

blockDigestEncoding

protected String blockDigestEncoding

Default encoding scheme used to encode block digest into a string, if none is detected from the record.

bPayloadDigest

protected boolean bPayloadDigest

Payload Digest enabled/disabled.

payloadDigestAlgorithm

protected String payloadDigestAlgorithm

Default payload digest algorithm to use if none is present in the record.

payloadDigestEncoding

protected String payloadDigestEncoding

Default encoding scheme used to encode payload digest into a string, if none is detected from the record.

warcRecord

protected WarcRecord warcRecord

Current WARC record object.

iteratorExceptionThrown

protected Exception iteratorExceptionThrown

Exception thrown while using the iterator.

warcHeaderMaxSize

protected int warcHeaderMaxSize

Max size allowed for a WARC header.

payloadHeaderMaxSize

protected int payloadHeaderMaxSize

Max size allowed for a payload header.

lineReader

protected HeaderLineReader lineReader

Line reader used to read version lines.

headerLineReader

protected HeaderLineReader headerLineReader

Header line reader used to read the WARC headers.

fieldParser

protected WarcFieldParsers fieldParser

WARC field parser used.

diagnostics

public final Diagnostics<Diagnosis> diagnostics

Reader level errors and warnings or when no record is available.

Constructor Detail

WarcReader

public WarcReader()

Method Detail

init

protected void init()

Method used to initialize a readers internal state. Must be called by all constructors.

isCompliant

public boolean isCompliant()

Returns a boolean indicating if all records parsed so far are compliant.

Returns:

a boolean indicating if all records parsed so far are compliant

isCompressed

public abstract boolean isCompressed()

Is this reader assuming GZip compressed input.

Returns:

boolean indicating the assumption of GZip compressed input

getBlockDigestEnabled

public boolean getBlockDigestEnabled()

Get the readers block digest on/off status.

Returns:

boolean indicating block digest on/off

setBlockDigestEnabled

public void setBlockDigestEnabled(boolean enabled)

Set the readers block digest on/off status. Digest, however, will only be computed if either a Warc-Block-Digest header is present or an optional algorithm has been chosen. The Warc-Block-Digest always takes precedence.

Parameters:

enabled - boolean indicating block digest on/off

getPayloadDigestEnabled

public boolean getPayloadDigestEnabled()

Get the readers payload digest on/off status.

Returns:

boolean indicating payload digest on/off

setPayloadDigestEnabled

public void setPayloadDigestEnabled(boolean enabled)

Set the readers payload digest on/off status. Digest, however, will only be computed if either a Warc-Payload-Digest header is present or an optional algorithm has been chosen. The Warc-Payload-Digest always takes precedence.

Parameters:

enabled - boolean indicating payload digest on/off

getBlockDigestAlgorithm

public String getBlockDigestAlgorithm()

Get the default block digest algorithm.

Returns:

default block digest algorithm

setBlockDigestAlgorithm

public boolean setBlockDigestAlgorithm(String digestAlgorithm)

Set the default block digest algorithm. This algorithm is only used in case no WARC payload digest header is present in the record.

Parameters:

digestAlgorithm - block digest algorithm (null means default block digest is disabled)

getPayloadDigestAlgorithm

public String getPayloadDigestAlgorithm()

Get the default payload digest algorithm.

Returns:

default payload digest algorithm

setPayloadDigestAlgorithm

public boolean setPayloadDigestAlgorithm(String digestAlgorithm)

Set the default payload digest algorithm. This algorithm is only used in case no WARC payload digest header is present in the record.

Parameters:

digestAlgorithm - payload digest algorithm (null means default payload digest is disabled)

getBlockDigestEncoding

public String getBlockDigestEncoding()

Get the default block digest encoding scheme.

Returns:

default block digest encoding scheme

setBlockDigestEncoding

public void setBlockDigestEncoding(String encodingScheme)

Set the default block digest encoding scheme. This scheme is only used if none can be inferred from an existing block digest header.

Parameters:

encodingScheme - encoding scheme (null means default block digest is not encoded)

getPayloadDigestEncoding

public String getPayloadDigestEncoding()

Get the default payload digest encoding scheme.

Returns:

default payload digest encoding scheme

setPayloadDigestEncoding

public void setPayloadDigestEncoding(String encodingScheme)

Set the default payload digest encoding scheme. This scheme is only used if none can be inferred from an existing payload digest header.

Parameters:

encodingScheme - encoding scheme (null means default payload digest is not encoded)

getWarcHeaderMaxSize

public int getWarcHeaderMaxSize()

Get the max size allowed for a WARC header.

Returns:

max size allowed for a WARC header

setWarcHeaderMaxSize

public void setWarcHeaderMaxSize(int size)

Set the max size allowed for a WARC header.

Parameters:

size - max size allowed

getPayloadHeaderMaxSize

public int getPayloadHeaderMaxSize()

Get the max size allowed for a payload header.

Returns:

max size allowed for a payload header

setPayloadHeaderMaxSize

public void setPayloadHeaderMaxSize(int size)

Set the max size allowed for a payload header.

Parameters:

size - max size allowed

close

public abstract void close()

Close current record resource(s) and input stream(s).

getStartOffset

public abstract long getStartOffset()

Get the offset of the current or next WARC record.

Returns:

offset of the current of next WARC record

getOffset

public abstract long getOffset()

Get the current offset in the WARC InputStream.

Returns:

offset in WARC InputStream

See Also:

WarcRecordBase#getStartOffset()

getConsumed

public abstract long getConsumed()

Get number of bytes consumed by this reader.

Returns:

number of bytes consumed by this reader

getNextRecord

public abstract WarcRecord getNextRecord()
                                  throws IOException

Parses and gets the next record. This method is for linear access to records.

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public abstract WarcRecord getNextRecordFrom(InputStream in,
                                             long offset)
                                      throws IOException

Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Parameters:

in - InputStream used to read next record

offset - offset provided by caller

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public abstract WarcRecord getNextRecordFrom(InputStream in,
                                             long offset,
                                             int buffer_size)
                                      throws IOException

Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Parameters:

in - InputStream used to read next record

offset - offset provided by caller

buffer_size - buffer size to use

Returns:

the next record

Throws:

IOException - io exception in parsing process

getIteratorExceptionThrown

public Exception getIteratorExceptionThrown()

Gets an exception thrown in the iterator if any or null.

Returns:

exception thrown in the iterator if any or null

iterator

public Iterator<WarcRecord> iterator()

Returns an Iterator over the records as they are being parsed. Any exception thrown during parsing is accessible through the getIteratorExceptionThrown method.

Returns:

Iterator over the records