org.cloudgraph.hbase.mapreduce

Class GraphInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable>

org.cloudgraph.hbase.mapreduce.GraphInputFormat

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class GraphInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable>
implements org.apache.hadoop.conf.Configurable

A graph based input-specification for MapReduce jobs which splits an underlying root table by region and by the scans resulting from a given query, then provides graph record readers for each split which assemble and serve data graphs to client mapper extensions.

Data graphs are assembled within a record reader based on the detailed selection criteria within a given query, and may be passed to a recognizer and potentially screened from client mappers potentially illuminating business logic dedicated to identifying specific records.

A graph recognizer is used only when query expressions are present which reference properties not found in the row key model for a target graph.

Since:

0.5.8

Author:

Scott Cinnamond

See Also:

TableSplit, GraphRecordReader, GraphRecordRecognizer

Field Summary

Fields

Modifier and Type	Field and Description
static String	QUERY Serialized query which encapsulates meta data which drives the creation is one or more scans against any number of tables, starting with the root.
static String	RECOGNIZER Boolean property indicating whether a graph recognizer is necessary for the current query
protected static String	ROOT_TABLE_NAME Internal Job parameter that specifies root the input table as derived from a deserialized query
protected static String	ROOT_TABLE_NAMESPACE Internal Job parameter that specifies root the input table namespace as derived from a deserialized query
static String	SCAN_CACHEBLOCKS Set to false to disable server-side caching of blocks for this scan.
static String	SCAN_CACHEDROWS The number of rows for caching that will be passed to scanners.
protected static String	SCAN_MAXVERSIONS The maximum number of version to return.
protected static String	SCAN_TIMERANGE_END The ending timestamp used to filter columns with a specific range of versions.
protected static String	SCAN_TIMERANGE_START The starting timestamp used to filter columns with a specific range of versions.
protected static String	SCAN_TIMESTAMP The timestamp used to filter columns with a specific timestamp.
protected static String	SCANS Internal Job parameter that specifies the scan list.
static String	TABLE_MAPPINGS Serialized store mappings or configuration which links physical tables with any number of data graphs/mappings.

Constructor Summary

Constructors

Constructor and Description
GraphInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
org.apache.hadoop.conf.Configuration	getConf()
protected List<org.apache.hadoop.hbase.client.Scan>	getScans() Allows subclasses to get the list of Scan objects.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
protected org.apache.hadoop.hbase.util.Pair<byte[][],byte[][]>	getStartEndKeys()
protected boolean	includeRegionInSplit(byte[] startKey, byte[] endKey) Test if the given region is to be included in the InputSplit while splitting the regions of a table.
String	reverseDNS(InetAddress ipAddr) Uses InetAddress in case of DNS lookup failure.
void	setConf(org.apache.hadoop.conf.Configuration configuration)
protected void	setScans(List<org.apache.hadoop.hbase.client.Scan> scans) Allows subclasses to set the list of Scan objects.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

QUERY

public static final String QUERY

Serialized query which encapsulates meta data which drives the creation is one or more scans against any number of tables, starting with the root.

See Also:

Constant Field Values

TABLE_MAPPINGS

public static final String TABLE_MAPPINGS

Serialized store mappings or configuration which links physical tables with any number of data graphs/mappings. This data is typically packaged into a jar in an e.g. cloudgraph-config.xml file, but where these mappings are dynamic and loaded at runtime, the job (spark,mapreduce) child VM's must have access to and load this mapping dynamically. The mapping must be serialized as XML and be marshalled and be unmarshallable using the root CloudGraphStoreMapping which may have any number of table mapping elements.

See Also:

Constant Field Values

RECOGNIZER

public static final String RECOGNIZER

Boolean property indicating whether a graph recognizer is necessary for the current query

See Also:

Constant Field Values

SCANS

protected static final String SCANS

Internal Job parameter that specifies the scan list.

See Also:

Constant Field Values

ROOT_TABLE_NAME

protected static final String ROOT_TABLE_NAME

Internal Job parameter that specifies root the input table as derived from a deserialized query

See Also:

Constant Field Values

ROOT_TABLE_NAMESPACE

protected static final String ROOT_TABLE_NAMESPACE

Internal Job parameter that specifies root the input table namespace as derived from a deserialized query

See Also:

Constant Field Values

SCAN_TIMESTAMP

protected static final String SCAN_TIMESTAMP

The timestamp used to filter columns with a specific timestamp.

See Also:

Constant Field Values

SCAN_TIMERANGE_START

protected static final String SCAN_TIMERANGE_START

The starting timestamp used to filter columns with a specific range of versions.

See Also:

Constant Field Values

SCAN_TIMERANGE_END

protected static final String SCAN_TIMERANGE_END

The ending timestamp used to filter columns with a specific range of versions.

See Also:

Constant Field Values

SCAN_MAXVERSIONS

protected static final String SCAN_MAXVERSIONS

The maximum number of version to return.

See Also:

Constant Field Values

SCAN_CACHEBLOCKS

public static final String SCAN_CACHEBLOCKS

Set to false to disable server-side caching of blocks for this scan.

See Also:

Constant Field Values

SCAN_CACHEDROWS

public static final String SCAN_CACHEDROWS

The number of rows for caching that will be passed to scanners.

See Also:

Constant Field Values

Constructor Detail

GraphInputFormat

public GraphInputFormat()

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration configuration)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                       throws IOException,
                                                              InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable>

Throws:

IOException

InterruptedException

reverseDNS

public String reverseDNS(InetAddress ipAddr)
                  throws NamingException,
                         UnknownHostException

Uses InetAddress in case of DNS lookup failure.

Parameters:

ipAddr - the address

Returns:

the reverse address

Throws:

NamingException

UnknownHostException

getStartEndKeys

protected org.apache.hadoop.hbase.util.Pair<byte[][],byte[][]> getStartEndKeys()
                                                                        throws IOException

Throws:

IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                                                    org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                                             throws IOException,
                                                                                                                                    InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,GraphWritable>

Throws:

IOException

InterruptedException

includeRegionInSplit

protected boolean includeRegionInSplit(byte[] startKey,
                                       byte[] endKey)

Test if the given region is to be included in the InputSplit while splitting the regions of a table.

This optimization is effective when there is a specific reasoning to exclude an entire region from the M-R job, (and hence, not contributing to the InputSplit), given the start and end keys of the same.
Useful when we need to remember the last-processed top record and revisit the [last, current) interval for M-R processing, continuously. In addition to reducing InputSplits, reduces the load on the region server as well, due to the ordering of the keys.

Note: It is possible that endKey.length() == 0 , for the last (recent) region.
Override this method, if you want to bulk exclude regions altogether from M-R. By default, no region is excluded( i.e. all regions are included).

Parameters:

startKey - Start key of the region

endKey - End key of the region

Returns:

true, if this region needs to be included as part of the input (default).

getScans

protected List<org.apache.hadoop.hbase.client.Scan> getScans()

Allows subclasses to get the list of Scan objects.

setScans

protected void setScans(List<org.apache.hadoop.hbase.client.Scan> scans)

Allows subclasses to set the list of Scan objects.

Parameters:

scans - The list of Scan used to define the input