org.gridgain.grid.dataload

Interface GridDataLoader<K,V>

All Superinterfaces:

AutoCloseable

All Known Implementing Classes:

GridDataLoaderImpl

@GridOnlyAvailableIn(value=DATA_GRID)
public interface GridDataLoader<K,V>
extends AutoCloseable

Data loader is responsible for loading external data into cache. It achieves it by properly buffering updates and properly mapping keys to nodes responsible for the data to make sure that there is the least amount of data movement possible and optimal network and memory utilization.

Note that loader will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the loader.

Also note that GridDataLoader is not the only way to load data into cache. Alternatively you can use GridCache.loadCache(GridBiPredicate, long, Object...) method to load data from underlying data store. You can also use standard cache put(...) and putAll(...) operations as well, but they most likely will not perform as well as this class for loading data. And finally, data can be loaded from underlying data store on demand, whenever it is accessed - for this no explicit data loading step is needed.

GridDataLoader supports the following configuration properties:

• perNodeBufferSize(int) - when entries are added to data loader via addData(Object, Object) method, they are not sent to in-memory data grid right away and are buffered internally for better performance and network utilization. This setting controls the size of internal per-node buffer before buffered data is sent to remote node. Default is defined by DFLT_PER_NODE_BUFFER_SIZE value.
• perNodeParallelLoadOperations(int) - sometimes data may be added to the data loader via addData(Object, Object) method faster than it can be put in cache. In this case, new buffered load messages are sent to remote nodes before responses from previous ones are received. This could cause unlimited heap memory utilization growth on local and remote nodes. To control memory utilization, this setting limits maximum allowed number of parallel buffered load messages that are being processed on remote nodes. If this number is exceeded, then addData(Object, Object) method will block to control memory utilization. Default is defined by DFLT_MAX_PARALLEL_OPS value.
• autoFlushFrequency(long) - automatic flush frequency in milliseconds. Essentially, this is the time after which the loader will make an attempt to submit all data added so far to remote nodes. Note that there is no guarantee that data will be delivered after this concrete attempt (e.g., it can fail when topology is changing), but it won't be lost anyway. Disabled by default (default value is 0).
• isolated(boolean) - defines if data loader will assume that there are no other concurrent updates and allow data loader choose most optimal concurrent implementation.
• updater(GridDataLoadCacheUpdater) - defines how cache will be updated with loaded entries. It allows to provide user-defined custom logic to update the cache in the most effective and flexible way.
• deployClass(Class) - optional deploy class for peer deployment. All classes loaded by a data loader must be class-loadable from the same class-loader. GridGain will make the best effort to detect the most suitable class-loader for data loading. However, in complex cases, where compound or deeply nested class-loaders are used, it is best to specify a deploy class which can be any class loaded by the class-loader for given data.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DFLT_MAX_PARALLEL_OPS Default max concurrent put operations count.
static int	DFLT_PER_NODE_BUFFER_SIZE Default per node buffer size.

Method Summary

Methods

Modifier and Type	Method and Description
GridFuture<?>	addData(Collection<? extends Map.Entry<K,V>> entries) Adds data for loading on remote node.
GridFuture<?>	addData(K key, V val) Adds data for loading on remote node.
GridFuture<?>	addData(Map.Entry<K,V> entry) Adds data for loading on remote node.
GridFuture<?>	addData(Map<K,V> entries) Adds data for loading on remote node.
long	autoFlushFrequency() Gets automatic flush frequency.
void	autoFlushFrequency(long autoFlushFreq) Sets automatic flush frequency.
String	cacheName() Name of cache to load data to.
void	close() Closes data loader.
void	close(boolean cancel) Loads any remaining data and closes this loader.
void	deployClass(Class<?> depCls) Optional deploy class for peer deployment.
void	flush() Loads any remaining data, but doesn't close the loader.
GridFuture<?>	future() Gets future for this loading process.
boolean	isolated() Gets flag value indicating that this data loader assumes that there are no other concurrent updates to the cache.
void	isolated(boolean isolated) Sets flag indicating that this data loader should assume that there are no other concurrent updates to the cache.
int	perNodeBufferSize() Gets size of per node key-value pairs buffer.
void	perNodeBufferSize(int bufSize) Sets size of per node key-value pairs buffer.
int	perNodeParallelLoadOperations() Gets maximum number of parallel load operations for a single node.
void	perNodeParallelLoadOperations(int parallelOps) Sets maximum number of parallel load operations for a single node.
GridFuture<?>	removeData(K key) Adds key for removal on remote node.
void	tryFlush() Makes an attempt to load remaining data.
void	updater(GridDataLoadCacheUpdater<K,V> updater) Sets custom cache updater to this data loader.

Field Detail

DFLT_MAX_PARALLEL_OPS

static final int DFLT_MAX_PARALLEL_OPS

Default max concurrent put operations count.

See Also:

Constant Field Values

DFLT_PER_NODE_BUFFER_SIZE

static final int DFLT_PER_NODE_BUFFER_SIZE

Default per node buffer size.

See Also:

Constant Field Values

Method Detail

cacheName

@Nullable
String cacheName()

Name of cache to load data to.

Returns:

Cache name or null for default cache.

isolated

boolean isolated()

Gets flag value indicating that this data loader assumes that there are no other concurrent updates to the cache. Default is false.

Returns:

Flag value.

isolated

void isolated(boolean isolated)
              throws GridException

Sets flag indicating that this data loader should assume that there are no other concurrent updates to the cache. Should not be used when custom cache updater set using updater(GridDataLoadCacheUpdater) method. Default is false.

Parameters:

isolated - Flag value.

Throws:

GridException - If failed.

perNodeBufferSize

int perNodeBufferSize()

Gets size of per node key-value pairs buffer.

Returns:

Per node buffer size.

perNodeBufferSize

void perNodeBufferSize(int bufSize)

Sets size of per node key-value pairs buffer.

This method should be called prior to addData(Object, Object) call.

If not provided, default value is DFLT_PER_NODE_BUFFER_SIZE.

Parameters:

bufSize - Per node buffer size.

perNodeParallelLoadOperations

int perNodeParallelLoadOperations()

Gets maximum number of parallel load operations for a single node.

Returns:

Maximum number of parallel load operations for a single node.

perNodeParallelLoadOperations

void perNodeParallelLoadOperations(int parallelOps)

Sets maximum number of parallel load operations for a single node.

This method should be called prior to addData(Object, Object) call.

If not provided, default value is DFLT_MAX_PARALLEL_OPS.

Parameters:

parallelOps - Maximum number of parallel load operations for a single node.

autoFlushFrequency

long autoFlushFrequency()

Gets automatic flush frequency. Essentially, this is the time after which the loader will make an attempt to submit all data added so far to remote nodes. Note that there is no guarantee that data will be delivered after this concrete attempt (e.g., it can fail when topology is changing), but it won't be lost anyway.

If set to 0, automatic flush is disabled.

Automatic flush is disabled by default (default value is 0).

Returns:

Flush frequency or 0 if automatic flush is disabled.

See Also:

flush()

autoFlushFrequency

void autoFlushFrequency(long autoFlushFreq)

Sets automatic flush frequency. Essentially, this is the time after which the loader will make an attempt to submit all data added so far to remote nodes. Note that there is no guarantee that data will be delivered after this concrete attempt (e.g., it can fail when topology is changing), but it won't be lost anyway.

If set to 0, automatic flush is disabled.

Automatic flush is disabled by default (default value is 0).

Parameters:

autoFlushFreq - Flush frequency or 0 to disable automatic flush.

See Also:

flush()

future

GridFuture<?> future()

Gets future for this loading process. This future completes whenever method close(boolean) completes. By attaching listeners to this future it is possible to get asynchronous notifications for completion of this loading process.

Returns:

Future for this loading process.

deployClass

void deployClass(Class<?> depCls)

Optional deploy class for peer deployment. All classes loaded by a data loader must be class-loadable from the same class-loader. GridGain will make the best effort to detect the most suitable class-loader for data loading. However, in complex cases, where compound or deeply nested class-loaders are used, it is best to specify a deploy class which can be any class loaded by the class-loader for given data.

Parameters:

depCls - Any class loaded by the class-loader for given data.

updater

void updater(GridDataLoadCacheUpdater<K,V> updater)

Sets custom cache updater to this data loader.

Parameters:

updater - Cache updater.

removeData

GridFuture<?> removeData(K key)
                         throws GridException,
                                GridInterruptedException,
                                IllegalStateException

Adds key for removal on remote node. Equivalent to addData(key, null).

Parameters:

key - Key.

Returns:

Future fo this operation.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

addData

GridFuture<?> addData(K key,
                    @Nullable
                    V val)
                      throws GridException,
                             GridInterruptedException,
                             IllegalStateException

Adds data for loading on remote node. This method can be called from multiple threads in parallel to speed up loading if needed.

Note that loader will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the loader.

Parameters:

key - Key.

val - Value or null if respective entry must be removed from cache.

Returns:

Future fo this operation.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

addData

GridFuture<?> addData(Map.Entry<K,V> entry)
                      throws GridException,
                             GridInterruptedException,
                             IllegalStateException

Adds data for loading on remote node. This method can be called from multiple threads in parallel to speed up loading if needed.

Note that loader will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the loader.

Parameters:

entry - Entry.

Returns:

Future fo this operation.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

addData

GridFuture<?> addData(Collection<? extends Map.Entry<K,V>> entries)
                      throws IllegalStateException

Adds data for loading on remote node. This method can be called from multiple threads in parallel to speed up loading if needed.

Note that loader will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the loader.

Parameters:

entries - Collection of entries to be loaded.

Returns:

Future for this load operation.

Throws:

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

addData

GridFuture<?> addData(Map<K,V> entries)
                      throws IllegalStateException

Adds data for loading on remote node. This method can be called from multiple threads in parallel to speed up loading if needed.

Note that loader will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the loader.

Parameters:

entries - Map to be loaded.

Returns:

Future for this load operation.

Throws:

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

flush

void flush()
           throws GridException,
                  GridInterruptedException,
                  IllegalStateException

Loads any remaining data, but doesn't close the loader. Data can be still added after flush is finished. This method blocks and doesn't allow to add any data until all data is loaded.

If another thread is already performing flush, this method will block, wait for another thread to complete flush and exit. If you don't want to wait in this case, use tryFlush() method.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

See Also:

tryFlush()

tryFlush

void tryFlush()
              throws GridException,
                     GridInterruptedException,
                     IllegalStateException

Makes an attempt to load remaining data. This method is mostly similar to flush(), with the difference that it won't wait and will exit immediately.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

IllegalStateException - If grid has been concurrently stopped or close(boolean) has already been called on loader.

See Also:

flush()

close

void close(boolean cancel)
           throws GridException,
                  GridInterruptedException

Loads any remaining data and closes this loader.

Parameters:

cancel - True to cancel ongoing loading operations.

Throws:

GridException - If failed to map key to node.

GridInterruptedException - If thread has been interrupted.

close

void close()
           throws GridException,
                  GridInterruptedException

Closes data loader. This method is identical to calling close(false) method.

The method is invoked automatically on objects managed by the try-with-resources statement.

Specified by:

close in interface AutoCloseable

Throws:

GridException - If failed to close data loader.

GridInterruptedException - If thread has been interrupted.