org.cloudgraph.hbase.graph

Class ParallelGraphAssembler

java.lang.Object

org.cloudgraph.hbase.graph.DefaultAssembler

org.cloudgraph.hbase.graph.DistributedAssembler

org.cloudgraph.hbase.graph.ParallelGraphAssembler

All Implemented Interfaces:

HBaseGraphAssembler, org.plasma.sdo.access.DataGraphAssembler

public class ParallelGraphAssembler
extends DistributedAssembler

Constructs a data graph in parallel starting with a given root using the given "selection graph", where processing proceeds as a breadth-first traversal and tasks are dynamically added based on thread availability within a shared thread pool.

While the result graph may be of any arbitrary size or depth, because the traversal is breadth-first, many tasks are typically spawned at the "base" of the graph, exhausting the available pool threads. Each subgraph task can spawn further sub tasks based on thread availability, but typically this means each task will traverse and process a healthy segment of the total graph. Since the actual size or depth of the result graph is not known until discovered on traversal, a fixed number of parallel tasks cannot be initially created, but must be dynamically spawned during graph discovery.

The assembly is triggered by calling the ParallelGraphAssembler#assemble(Result resultRow) method which recursively reads HBase keys and values re-constituting the data graph. The assembly traversal is driven by HBase column values representing the original edges or containment structure of the graph.

Since every column key in HBase must be unique, and a data graph may contain any number of nodes, a column key factory is used both to persist as well as re-constitute a graph. A minimal amount of "state" information is therefore stored with each graph which maps user readable sequence numbers (which are used in column keys) to UUID values. The nodes of the resulting data graph are re-created with the original UUID values.

Since:

0.6.2

Author:

Scott Cinnamond

See Also:

StatefullColumnKeyFactory, ParallelSubgraphTask

Field Summary

Fields inherited from class org.cloudgraph.hbase.graph.DistributedAssembler

distributedReader

Fields inherited from class org.cloudgraph.hbase.graph.DefaultAssembler

charset, graph, keyFactories, root, rootTableReader, rootType, selection, snapshotDate

Constructor Summary

Constructors

Constructor and Description
ParallelGraphAssembler(org.plasma.sdo.PlasmaType rootType, org.plasma.query.collector.Selection selection, DistributedReader distributedReader, Timestamp snapshotDate, ThreadPoolMappingProps config) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	assemble(org.plasma.sdo.PlasmaDataObject target, long targetSequence, EdgeReader sourceCollection, org.plasma.sdo.PlasmaDataObject source, org.plasma.sdo.PlasmaProperty sourceProperty, RowReader rowReader, int level) Creates a single task and begins the traversal from root in the current thread.
ThreadPoolMappingProps	getConfig()

Methods inherited from class org.cloudgraph.hbase.graph.DistributedAssembler

assemble, assembleEdge, assembleExternalEdge, clear

Methods inherited from class org.cloudgraph.hbase.graph.DefaultAssembler

assembleData, assembleUUID, createChild, createChild, createChild, createRoot, fetchGraph, fetchRootType, fetchRootUUID, fetchType, fetchUUID, findMetaDataColumnValue, findMetaDataColumnValue, findRootType, findRootUUID, getDataColumnValue, getDataColumnValue, getDataGraph, getKeyFactory, getMetaDataColumnValue, getMetaDataColumnValue, getProperties, getProperties, link

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.plasma.sdo.access.DataGraphAssembler

getDataGraph

Constructor Detail

ParallelGraphAssembler

public ParallelGraphAssembler(org.plasma.sdo.PlasmaType rootType,
                              org.plasma.query.collector.Selection selection,
                              DistributedReader distributedReader,
                              Timestamp snapshotDate,
                              ThreadPoolMappingProps config)

Constructor.

Parameters:

rootType - the SDO root type for the result data graph

selection - selected SDO properties. Properties are mapped by selected types required in the result graph.

snapshotDate - the query snapshot date which is populated into every data object in the result data graph.

distributedReader - the distributed reader

minPoolSize - the minimum or core size of the underlying thread pool used for all tasks executed under this assembler

maxPoolSize - the maximum size of the underlying thread pool used for all tasks executed under this assembler

Method Detail

getConfig

public ThreadPoolMappingProps getConfig()

assemble

protected void assemble(org.plasma.sdo.PlasmaDataObject target,
                        long targetSequence,
                        EdgeReader sourceCollection,
                        org.plasma.sdo.PlasmaDataObject source,
                        org.plasma.sdo.PlasmaProperty sourceProperty,
                        RowReader rowReader,
                        int level)
                 throws IOException

Creates a single task and begins the traversal from root in the current thread.

Specified by:

assemble in class DistributedAssembler

targetSequence - the current data object sequence

source - the source or parent data object

sourceProperty - the source (reference) property

rowReader - the current row reader

level - the current graph level

Throws:

IOException - if a remote or network exception occurs.