org.jwat.warc
Class WarcReaderCompressed

java.lang.Object
  org.jwat.warc.WarcReader
      org.jwat.warc.WarcReaderCompressed

public class WarcReaderCompressed
extends WarcReader

WARC Reader implementation for reading GZip compressed files.

Author:

nicl

Field Summary

protected int	bufferSize Buffer size, if any, to use on GZip entry InputStream.
static int	PUSHBACK_BUFFER_SIZE Buffer size used by PushbackInputStream.
protected GzipReader	reader WARC file InputStream.
protected long	startOffset Cached start offset used after the reader is closed.

Fields inherited from class org.jwat.warc.WarcReader

bBlockDigest, bIsCompliant, blockDigestAlgorithm, blockDigestEncoding, bPayloadDigest, consumed, diagnostics, errors, fieldParser, headerLineReader, iteratorExceptionThrown, lineReader, payloadDigestAlgorithm, payloadDigestEncoding, payloadHeaderMaxSize, warcHeaderMaxSize, warcRecord, warnings

Method Summary

void	close() Close current record resource(s) and input stream(s).
long	getConsumed() Get number of bytes consumed by this reader.
WarcRecord	getNextRecord() Parses and gets the next record.
WarcRecord	getNextRecordFrom(InputStream rin, long offset) Parses and gets the next record from an Inputstream.
WarcRecord	getNextRecordFrom(InputStream rin, long offset, int buffer_size) Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream.
long	getOffset() Get the current offset in the WARC GzipReader.
long	getStartOffset() Get the current offset in the WARC GzipReader.
boolean	isCompressed() Is this reader assuming GZip compressed input.

Methods inherited from class org.jwat.warc.WarcReader

getBlockDigestAlgorithm, getBlockDigestEnabled, getBlockDigestEncoding, getIteratorExceptionThrown, getPayloadDigestAlgorithm, getPayloadDigestEnabled, getPayloadDigestEncoding, getPayloadHeaderMaxSize, getWarcHeaderMaxSize, init, isCompliant, iterator, setBlockDigestAlgorithm, setBlockDigestEnabled, setBlockDigestEncoding, setPayloadDigestAlgorithm, setPayloadDigestEnabled, setPayloadDigestEncoding, setPayloadHeaderMaxSize, setWarcHeaderMaxSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUSHBACK_BUFFER_SIZE

public static final int PUSHBACK_BUFFER_SIZE

Buffer size used by PushbackInputStream.

See Also:

Constant Field Values

reader

protected GzipReader reader

WARC file InputStream.

bufferSize

protected int bufferSize

Buffer size, if any, to use on GZip entry InputStream.

startOffset

protected long startOffset

Cached start offset used after the reader is closed.

Method Detail

isCompressed

public boolean isCompressed()

Description copied from class: WarcReader

Is this reader assuming GZip compressed input.

Specified by:

isCompressed in class WarcReader

Returns:

boolean indicating the assumption of GZip compressed input

close

public void close()

Description copied from class: WarcReader

Close current record resource(s) and input stream(s).

Specified by:

close in class WarcReader

getStartOffset

public long getStartOffset()

Get the current offset in the WARC GzipReader.

Specified by:

getStartOffset in class WarcReader

Returns:

offset in WARC InputStream

getOffset

public long getOffset()

Get the current offset in the WARC GzipReader.

Specified by:

getOffset in class WarcReader

Returns:

offset in WARC InputStream

See Also:

WarcRecordBase#getStartOffset()

getConsumed

public long getConsumed()

Description copied from class: WarcReader

Get number of bytes consumed by this reader.

Specified by:

getConsumed in class WarcReader

Returns:

number of bytes consumed by this reader

getNextRecord

public WarcRecord getNextRecord()
                         throws IOException

Description copied from class: WarcReader

Parses and gets the next record. This method is for linear access to records.

Specified by:

getNextRecord in class WarcReader

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

Returns:

the next record

Throws:

IOException - io exception in parsing process

getNextRecordFrom

public WarcRecord getNextRecordFrom(InputStream rin,
                                    long offset,
                                    int buffer_size)
                             throws IOException

Description copied from class: WarcReader

Parses and gets the next record from an Inputstream wrapped by a BufferedInputStream. This method is mainly for random access use since there are serious side-effects involved in using multiple PushBackInputStream instances.

Specified by:

getNextRecordFrom in class WarcReader

Parameters:

rin - InputStream used to read next record

offset - offset provided by caller

buffer_size - buffer size to use

Returns:

the next record

Throws:

IOException - io exception in parsing process