org.jwat.warc
Class WarcWriter

java.lang.Object
  org.jwat.warc.WarcWriter

Direct Known Subclasses:

WarcWriterCompressed, WarcWriterUncompressed

public abstract class WarcWriter
extends Object

Base class for WARC writer implementations.

Author:

nicl

Field Summary

protected boolean	bExceptionOnContentLengthMismatch Configuration for throwing exception on content-length mismatch.
Diagnostics<Diagnosis>	diagnostics Writer level errors and warnings or when writing byte headers.
protected WarcFieldParsers	fieldParser WARC field parser used.
protected WarcHeader	header Current WARC header written.
protected Long	headerContentLength Content-Length from the WARC header.
protected OutputStream	out Outputstream used to write WARC records.
protected long	payloadWrittenTotal Total bytes written for current record payload.
protected static int	S_HEADER_WRITTEN State after header has been written.
protected static int	S_INIT State after writer has been constructed and before records have been written.
protected static int	S_PAYLOAD_WRITTEN State after payload has been written.
protected static int	S_RECORD_CLOSED State after record has been closed.
protected int	state Current state of writer.
protected byte[]	stream_copy_buffer Buffer used by streamPayload() to copy from one stream to another.
protected DateFormat	warcDateFormat WARC DateFormat as specified by the WARC ISO standard.

Constructor Summary

WarcWriter()

Method Summary

abstract void	close() Close WARC writer and free its resources.
protected void	closeRecord_impl() Close the WARC record by writing two newlines.
abstract void	closeRecord() Close the WARC record.
boolean	exceptionOnContentLengthMismatch() Does this writer throw an exception if the content-length does not match the payload amount written.
protected void	init() Method used to initialize a readers internal state.
abstract boolean	isCompressed() Is this writer producing compressed output.
void	setExceptionOnContentLengthMismatch(boolean enabled) Tell the writer what to do in case of mismatch between content-length and amount payload written.
long	streamPayload(InputStream in)
protected byte[]	writeHeader_impl(WarcRecord record) Write a WARC header to the WARC output stream.
void	writeHeader(byte[] header_bytes, Long contentLength) Write a raw WARC header to the WARC output stream.
abstract byte[]	writeHeader(WarcRecord record) Write a WARC header to the WARC output stream.
long	writePayload(byte[] b)
long	writePayload(byte[] b, int offset, int len)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S_INIT

protected static final int S_INIT

State after writer has been constructed and before records have been written.

See Also:

Constant Field Values

S_HEADER_WRITTEN

protected static final int S_HEADER_WRITTEN

State after header has been written.

See Also:

Constant Field Values

S_PAYLOAD_WRITTEN

protected static final int S_PAYLOAD_WRITTEN

State after payload has been written.

See Also:

Constant Field Values

S_RECORD_CLOSED

protected static final int S_RECORD_CLOSED

State after record has been closed.

See Also:

Constant Field Values

warcDateFormat

protected DateFormat warcDateFormat

WARC DateFormat as specified by the WARC ISO standard.

fieldParser

protected WarcFieldParsers fieldParser

WARC field parser used.

stream_copy_buffer

protected byte[] stream_copy_buffer

Buffer used by streamPayload() to copy from one stream to another.

bExceptionOnContentLengthMismatch

protected boolean bExceptionOnContentLengthMismatch

Configuration for throwing exception on content-length mismatch. (Default is true)

diagnostics

public final Diagnostics<Diagnosis> diagnostics

Writer level errors and warnings or when writing byte headers.

state

protected int state

Current state of writer.

out

protected OutputStream out

Outputstream used to write WARC records.

header

protected WarcHeader header

Current WARC header written.

headerContentLength

protected Long headerContentLength

Content-Length from the WARC header.

payloadWrittenTotal

protected long payloadWrittenTotal

Total bytes written for current record payload.

Constructor Detail

WarcWriter

public WarcWriter()

Method Detail

init

protected void init()

Method used to initialize a readers internal state. Must be called by all constructors.

isCompressed

public abstract boolean isCompressed()

Is this writer producing compressed output.

Returns:

boolean indicating whether compressed output is produced

exceptionOnContentLengthMismatch

public boolean exceptionOnContentLengthMismatch()

Does this writer throw an exception if the content-length does not match the payload amount written.

Returns:

boolean indicating if an exception is thrown or not

setExceptionOnContentLengthMismatch

public void setExceptionOnContentLengthMismatch(boolean enabled)

Tell the writer what to do in case of mismatch between content-length and amount payload written.

Parameters:

enabled - boolean indicating exception throwing on/off

close

public abstract void close()

Close WARC writer and free its resources.

closeRecord

public abstract void closeRecord()
                          throws IOException

Close the WARC record.

Throws:

IOException - if an exception occurs while closing the record

closeRecord_impl

protected void closeRecord_impl()
                         throws IOException

Close the WARC record by writing two newlines.

Throws:

IOException - if an exception occurs while closing the record

writeHeader

public void writeHeader(byte[] header_bytes,
                        Long contentLength)
                 throws IOException

Write a raw WARC header to the WARC output stream.

Parameters:

header_bytes - raw WARC record to output

Throws:

IOException - if an exception occurs while writing header data

writeHeader

public abstract byte[] writeHeader(WarcRecord record)
                            throws IOException

Write a WARC header to the WARC output stream.

Parameters:

record - WARC record to output

Throws:

IOException - if an exception occurs while writing header data

writeHeader_impl

protected byte[] writeHeader_impl(WarcRecord record)
                           throws IOException

Write a WARC header to the WARC output stream.

Parameters:

record - WARC record to output

Throws:

IOException - if an exception occurs while writing header data

streamPayload

public long streamPayload(InputStream in)
                   throws IOException

Parameters:

in - input stream containing payload data

Returns:

written length of payload data

Throws:

IOException - if an exception occurs while writing payload data

writePayload

public long writePayload(byte[] b)
                  throws IOException

Throws:

IOException

writePayload

public long writePayload(byte[] b,
                         int offset,
                         int len)
                  throws IOException

Throws:

IOException