org.jwat.warc
Class WarcReaderUncompressed

java.lang.Object
  org.jwat.warc.WarcReader
      org.jwat.warc.WarcReaderUncompressed

public class WarcReaderUncompressed
extends WarcReader

WARC Reader implementation for reading uncompressed files.

Author:

nicl

Field Summary

protected ByteCountingPushBackInputStream	in WARC file InputStream.
static int	PUSHBACK_BUFFER_SIZE Buffer size used by PushbackInputStream.
protected long	startOffset Start offset of current or next valid record.

Fields inherited from class org.jwat.warc.WarcReader

bBlockDigest, bIsCompliant, blockDigestAlgorithm, blockDigestEncoding, bPayloadDigest, consumed, diagnostics, errors, fieldParser, headerLineReader, iteratorExceptionThrown, lineReader, payloadDigestAlgorithm, payloadDigestEncoding, payloadHeaderMaxSize, warcHeaderMaxSize, warcRecord, warnings

Method Summary

void	close() Close current record resource(s) and input stream(s).
long	getConsumed() Get number of bytes consumed by this reader.
WarcRecord	getNextRecord() Parses and gets the next record.
WarcRecord	getNextRecordFrom(InputStream rin, long offset) Parses and gets the next record from an Inputstream.
WarcRecord	getNextRecordFrom(InputStream rin, long offset, int buffer_size) Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream.
long	getOffset() Get the current offset in the WARC InputStream.
long	getStartOffset() Get the offset of the current or next WARC record.
boolean	isCompressed() Is this reader assuming GZip compressed input.

Methods inherited from class org.jwat.warc.WarcReader

getBlockDigestAlgorithm, getBlockDigestEnabled, getBlockDigestEncoding, getIteratorExceptionThrown, getPayloadDigestAlgorithm, getPayloadDigestEnabled, getPayloadDigestEncoding, getPayloadHeaderMaxSize, getWarcHeaderMaxSize, init, isCompliant, iterator, setBlockDigestAlgorithm, setBlockDigestEnabled, setBlockDigestEncoding, setPayloadDigestAlgorithm, setPayloadDigestEnabled, setPayloadDigestEncoding, setPayloadHeaderMaxSize, setWarcHeaderMaxSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUSHBACK_BUFFER_SIZE

public static final int PUSHBACK_BUFFER_SIZE

Buffer size used by PushbackInputStream.

See Also:

Constant Field Values

in

protected ByteCountingPushBackInputStream in

WARC file InputStream.

startOffset

protected long startOffset

Start offset of current or next valid record.

Method Detail

isCompressed

public boolean isCompressed()

Description copied from class: WarcReader

Is this reader assuming GZip compressed input.

Specified by:

isCompressed in class WarcReader

Returns:

boolean indicating the assumption of GZip compressed input

close

public void close()

Description copied from class: WarcReader

Close current record resource(s) and input stream(s).

Specified by:

close in class WarcReader

getStartOffset

public long getStartOffset()

Description copied from class: WarcReader

Get the offset of the current or next WARC record.

Specified by:

getStartOffset in class WarcReader

Returns:

offset of the current of next WARC record

getOffset

public long getOffset()

Description copied from class: WarcReader

Get the current offset in the WARC InputStream.

Specified by:

getOffset in class WarcReader

Returns:

offset in WARC InputStream

See Also:

WarcRecordBase#getStartOffset()

getConsumed

public long getConsumed()

Description copied from class: WarcReader

Get number of bytes consumed by this reader.

Specified by:

getConsumed in class WarcReader

Returns:

number of bytes consumed by this reader

getNextRecord

public WarcRecord getNextRecord()
                         throws IOException

Description copied from class: WarcReader

Parses and gets the next record. This method is for linear access to records.

Specified by:

getNextRecord in class WarcReader

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset,
                                    int buffer_size)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

buffer_size - buffer size to use

Returns:

the next record

Throws:

IOException - io exception in parsing process