org.jwat.warc
Class WarcRecord

java.lang.Object
  org.jwat.warc.WarcRecord

All Implemented Interfaces:

PayloadOnClosedHandler

public class WarcRecord
extends Object
implements PayloadOnClosedHandler

This class represents a parsed WARC record header block including possible validation and format warnings/errors encountered in the process. The payload of the WARC record is accessible through a wrapped payload object.

Author:

nicl

Field Summary

protected boolean	bClosed Has record been closed before.
protected boolean	bIsCompliant Is this record compliant ie. error free.
protected boolean	bMisplacedCr Did the reader detect a misplaced CR while parsing newlines.
protected boolean	bMisplacedLf Did the reader detect a misplaced LF while parsing newlines.
protected boolean	bMissingCr Did the reader detect a missing CR while parsing newlines.
protected boolean	bMissingLf Did the reader detect a missing LF while parsing newlines.
protected boolean	bPayloadClosed Has payload been closed before.
WarcDigest	computedBlockDigest Computed block digest.
WarcDigest	computedPayloadDigest Computed payload digest.
Diagnostics<Diagnosis>	diagnostics Validation errors and warnings.
WarcHeader	header WARC header.
protected HttpHeader	httpHeader HTTP header content parsed from payload.
protected ByteCountingPushBackInputStream	in Input stream used to read this record.
Boolean	isValidBlockDigest Is Warc-Block-Digest valid.
Boolean	isValidPayloadDigest Is Warc-Payload-Digest valid.
protected Payload	payload Payload object if any exists.
protected WarcReader	reader Reader instance used, required for file compliance.

Constructor Summary

protected

WarcRecord() Non public constructor to allow unit testing.

Method Summary

protected void	addErrorDiagnosis(DiagnosisType type, String entity, String... information) Add an error diagnosis of the given type on a specific entity with optional extra information.
protected void	addWarningDiagnosis(DiagnosisType type, String entity, String... information) Add a warning diagnosis of the given type on a specific entity with optional extra information.
void	close() Close resources associated with the WARC record.
static WarcRecord	createRecord(WarcWriter writer)
HeaderLine	getHeader(String field) Get a non-standard WARC header or null, if nothing is stored for this header name.
List<HeaderLine>	getHeaderList() Get a List of all the non-standard WARC headers found during parsing.
Payload	getPayload() Return Payload object.
InputStream	getPayloadContent() Payload content InputStream getter.
long	getStartOffset() Get the record offset relative to the start of the WARC file InputStream.
boolean	hasPayload() Specifies whether this record has a payload or not.
boolean	isClosed() Check to see if the record has been closed.
boolean	isCompliant() Returns a boolean indicating the ISO compliance status of this record.
protected int	parseNewLines(ByteCountingPushBackInputStream in) Looks forward in the inputstream and counts the number of newlines found.
static WarcRecord	parseRecord(ByteCountingPushBackInputStream in, WarcReader reader) Given an InputStream it tries to read and validate a WARC header block.
void	payloadClosed() Called when the payload object is closed and final steps in the validation process can be performed.
protected void	processComputedDigest(WarcDigest computedDigest, String digestAlgorithm, String digestEncoding, String digestName) Adjust algorithm and encoding information about computed block digest.
protected Boolean	processWarcDigest(WarcDigest warcDigest, WarcDigest computedDigest, String digestName) Auto-detect encoding used in WARC digest header and compare it to the internal one, if it has been computed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

reader

protected WarcReader reader

Reader instance used, required for file compliance.

in

protected ByteCountingPushBackInputStream in

Input stream used to read this record.

bIsCompliant

protected boolean bIsCompliant

Is this record compliant ie. error free.

diagnostics

public final Diagnostics<Diagnosis> diagnostics

Validation errors and warnings.

isValidBlockDigest

public Boolean isValidBlockDigest

Is Warc-Block-Digest valid. (Null is equal to not tested)

isValidPayloadDigest

public Boolean isValidPayloadDigest

Is Warc-Payload-Digest valid. (Null is equal to not tested)

header

public WarcHeader header

WARC header.

bMissingCr

protected boolean bMissingCr

Did the reader detect a missing CR while parsing newlines.

bMissingLf

protected boolean bMissingLf

Did the reader detect a missing LF while parsing newlines.

bMisplacedCr

protected boolean bMisplacedCr

Did the reader detect a misplaced CR while parsing newlines.

bMisplacedLf

protected boolean bMisplacedLf

Did the reader detect a misplaced LF while parsing newlines.

bPayloadClosed

protected boolean bPayloadClosed

Has payload been closed before.

bClosed

protected boolean bClosed

Has record been closed before.

payload

protected Payload payload

Payload object if any exists.

httpHeader

protected HttpHeader httpHeader

HTTP header content parsed from payload.

computedBlockDigest

public WarcDigest computedBlockDigest

Computed block digest.

computedPayloadDigest

public WarcDigest computedPayloadDigest

Computed payload digest.

Constructor Detail

WarcRecord

protected WarcRecord()

Non public constructor to allow unit testing.

Method Detail

createRecord

public static WarcRecord createRecord(WarcWriter writer)

parseRecord

public static WarcRecord parseRecord(ByteCountingPushBackInputStream in,
                                     WarcReader reader)
                              throws IOException

Given an InputStream it tries to read and validate a WARC header block.

Parameters:

in - InputStream containing WARC record data

reader - WarcReader used, with access to user defined options

Returns:

WarcRecord or null

Throws:

IOException - io exception in the process of reading record

payloadClosed

public void payloadClosed()
                   throws IOException

Called when the payload object is closed and final steps in the validation process can be performed.

Specified by:

payloadClosed in interface PayloadOnClosedHandler

Throws:

IOException - io exception in final validation processing

isClosed

public boolean isClosed()

Check to see if the record has been closed.

Returns:

boolean indicating whether this record is closed or not

processWarcDigest

protected Boolean processWarcDigest(WarcDigest warcDigest,
                                    WarcDigest computedDigest,
                                    String digestName)

Auto-detect encoding used in WARC digest header and compare it to the internal one, if it has been computed.

Parameters:

warcDigest - digest from WARC header

computedDigest - internally compute digest

Returns:

WARC digest validity indication

processComputedDigest

protected void processComputedDigest(WarcDigest computedDigest,
                                     String digestAlgorithm,
                                     String digestEncoding,
                                     String digestName)

Adjust algorithm and encoding information about computed block digest.

Parameters:

computedDigest - internally compute digest

digestAlgorithm - default algorithm

digestEncoding - default encoding

close

public void close()
           throws IOException

Close resources associated with the WARC record. Mainly payload stream if any.

Throws:

IOException - if unable to close resources

getStartOffset

public long getStartOffset()

Get the record offset relative to the start of the WARC file InputStream.

Returns:

the record offset relative to the start of the WARC file

getHeaderList

public List<HeaderLine> getHeaderList()

Get a List of all the non-standard WARC headers found during parsing.

Returns:

List of HeaderLine

getHeader

public HeaderLine getHeader(String field)

Get a non-standard WARC header or null, if nothing is stored for this header name.

Parameters:

field - header name

Returns:

WARC header line structure or null

hasPayload

public boolean hasPayload()

Specifies whether this record has a payload or not.

Returns:

true/false whether the ARC record has a payload

getPayload

public Payload getPayload()

Return Payload object.

Returns:

payload or null

getPayloadContent

public InputStream getPayloadContent()

Payload content InputStream getter.

Returns:

Payload content InputStream

isCompliant

public boolean isCompliant()

Returns a boolean indicating the ISO compliance status of this record.

Returns:

a boolean indicating the ISO compliance status of this record

addErrorDiagnosis

protected void addErrorDiagnosis(DiagnosisType type,
                                 String entity,
                                 String... information)

Add an error diagnosis of the given type on a specific entity with optional extra information. The information varies according to the diagnosis type.

Parameters:

type - diagnosis type

entity - entity examined

information - optional extra information

addWarningDiagnosis

protected void addWarningDiagnosis(DiagnosisType type,
                                   String entity,
                                   String... information)

Add a warning diagnosis of the given type on a specific entity with optional extra information. The information varies according to the diagnosis type.

Parameters:

type - diagnosis type

entity - entity examined

information - optional extra information

parseNewLines

protected int parseNewLines(ByteCountingPushBackInputStream in)
                     throws IOException

Looks forward in the inputstream and counts the number of newlines found. Non newlines characters are pushed back onto the inputstream.

Parameters:

in - data inputstream

Returns:

newlines found in inputstream

Throws:

IOException - if an error occurs while reading data