org.gridgain.grid.ggfs.mapreduce

Class GridGgfsTask<T,R>

java.lang.Object

org.gridgain.grid.compute.GridComputeTaskAdapter<GridGgfsTaskArgs<T>,R>

org.gridgain.grid.ggfs.mapreduce.GridGgfsTask<T,R>

All Implemented Interfaces:

Serializable, GridComputeTask<GridGgfsTaskArgs<T>,R>

Direct Known Subclasses:

GridGgfsTaskNoReduceAdapter

public abstract class GridGgfsTask<T,R>
extends GridComputeTaskAdapter<GridGgfsTaskArgs<T>,R>

GGFS task which can be executed on the grid using one of GridGgfs.execute() methods. Essentially GGFS task is regular GridComputeTask with different map logic. Instead of implementing GridComputeTask.map(List, Object) method to split task into jobs, you must implement createJob(GridGgfsPath, GridGgfsFileRange, GridGgfsTaskArgs) method.

Each file participating in GGFS task is split into GridGgfsFileRanges first. Normally range is a number of consequent bytes located on a single node (see GridGgfsGroupDataBlocksKeyMapper). In case maximum range size is provided (either through GridGgfsConfiguration.getMaximumTaskRangeLength() or GridGgfs.execute() argument), then ranges could be further divided into smaller chunks.

Once file is split into ranges, each range is passed to GridGgfsTask.createJob() method in order to create a GridGgfsJob.

Finally all generated jobs are sent to Grid nodes for execution.

As with regular GridComputeTask you can define your own logic for results handling and reduce step.

Here is an example of such a task:

 public class WordCountTask extends GridGgfsTask<String, Integer> {
     @Override
     public GridGgfsJob createJob(GridGgfsPath path, GridGgfsFileRange range, GridGgfsTaskArgs<T> args) throws GridException {
         // New job will be created for each range within each file.
         // We pass user-provided argument (which is essentially a word to look for) to that job.
         return new WordCountJob(args.userArgument());
     }

     // Aggregate results into one compound result.
     public Integer reduce(List<GridComputeJobResult> results) throws GridException {
         Integer total = 0;

         for (GridComputeJobResult res : results) {
             Integer cnt = res.getData();

             // Null can be returned for non-existent file in case we decide to ignore such situations.
             if (cnt != null)
                 total += cnt;
         }

         return total;
     }
 }
 

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
GridGgfsTask()

Method Summary

Methods

Modifier and Type	Method and Description
abstract GridGgfsJob	createJob(GridGgfsPath path, GridGgfsFileRange range, GridGgfsTaskArgs<T> args) Callback invoked during task map procedure to create job that will process specified split for GGFS file.
Map<? extends GridComputeJob,GridNode>	map(List<GridNode> subgrid, GridGgfsTaskArgs<T> args) This method is called to map or split grid task into multiple grid jobs.

Methods inherited from class org.gridgain.grid.compute.GridComputeTaskAdapter

result

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.gridgain.grid.compute.GridComputeTask

reduce

Constructor Detail

GridGgfsTask

public GridGgfsTask()

Method Detail

map

@Nullable
public final Map<? extends GridComputeJob,GridNode> map(List<GridNode> subgrid,
                                                  @Nullable
                                                  GridGgfsTaskArgs<T> args)
                                                 throws GridException

This method is called to map or split grid task into multiple grid jobs. This is the first method that gets called when task execution starts.

Parameters:

subgrid - Nodes available for this task execution. Note that order of nodes is guaranteed to be randomized by container. This ensures that every time you simply iterate through grid nodes, the order of nodes will be random which over time should result into all nodes being used equally.

args - Task execution argument. Can be null. This is the same argument as the one passed into Grid#execute(...) methods.

Returns:

Map of grid jobs assigned to subgrid node. Unless GridComputeTaskContinuousMapper is injected into task, if null or empty map is returned, exception will be thrown.

Throws:

GridException - If mapping could not complete successfully. This exception will be thrown out of GridComputeTaskFuture.get() method.

createJob

@Nullable
public abstract GridGgfsJob createJob(GridGgfsPath path,
                             GridGgfsFileRange range,
                             GridGgfsTaskArgs<T> args)
                               throws GridException

Callback invoked during task map procedure to create job that will process specified split for GGFS file.

Parameters:

path - Path.

range - File range based on consecutive blocks. This range will be further realigned to record boundaries on destination node.

args - Task argument.

Returns:

GGFS job. If null is returned, the passed in file range will be skipped.

Throws:

GridException - If job creation failed.