org.gridgain.grid.ggfs

Interface GridGgfs

public interface GridGgfs

GridGain File System API. It provides a typical file system "view" on a particular cache:

• list directories or get information for a single path
• create/move/delete files or directories
• write/read data streams into/from files

The data of each file is split on separate data blocks and stored in the cache. You can access file's data with a standard Java streaming API. Moreover, for each part of the file you can calculate an affinity and process file's content on corresponding nodes to escape unnecessary networking.

This API is fully thread-safe and you can use it from several threads.

GGFS Configuration

The simplest way to run a GridGain node with configured file system is to pass special configuration file included in GridGain distribution to ggstart.sh or ggstart.bat scripts, like this: ggstart.sh config/hadoop/default-config.xml

GGFS can be started as a data node or as a client node. Data node is responsible for caching data, while client node is responsible for basic file system operations and accessing data nodes remotely. When used as Hadoop file system, clients nodes usually started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

Integration With Hadoop

In addition to direct file system API, GGFS can be integrated with Hadoop by plugging in as Hadoop FileSystem. Refer to org.gridgain.grid.ggfs.hadoop.v1.GridGgfsHadoopFileSystem or org.gridgain.grid.ggfs.hadoop.v2.GridGgfsHadoopFileSystem for more information.

NOTE: integration with Hadoop is available only in In-Memory Accelerator For Hadoop edition.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PROP_GROUP_NAME Property: group name.
static String	PROP_PERMISSION Property: permission.
static String	PROP_USER_NAME Property: user name.

Method Summary

Methods

Modifier and Type	Method and Description
Collection<GridGgfsBlockLocation>	affinity(GridGgfsPath path, long start, long len) Gets affinity block locations for data blocks of the file, i.e.
Collection<GridGgfsBlockLocation>	affinity(GridGgfsPath path, long start, long len, long maxLen) Get affinity block locations for data blocks of the file.
GridGgfsOutputStream	append(GridGgfsPath path, boolean create) Opens an output stream to an existing file for appending data.
GridGgfsOutputStream	append(GridGgfsPath path, int bufSize, boolean create, Map<String,String> props) Opens an output stream to an existing file for appending data.
GridGgfsConfiguration	configuration() Gets GGFS configuration.
GridGgfsOutputStream	create(GridGgfsPath path, boolean overwrite) Creates a file and opens it for writing.
GridGgfsOutputStream	create(GridGgfsPath path, int bufSize, boolean overwrite, GridUuid affKey, int replication, long blockSize, Map<String,String> props) Creates a file and opens it for writing.
boolean	delete(GridGgfsPath path, boolean recursive) Deletes file.
<T,R> GridFuture<R>	execute(Class<? extends GridGgfsTask<T,R>> taskCls, GridGgfsRecordResolver rslvr, Collection<GridGgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes GGFS task asynchronously with overridden maximum range length (see GridGgfsConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> GridFuture<R>	execute(Class<? extends GridGgfsTask<T,R>> taskCls, GridGgfsRecordResolver rslvr, Collection<GridGgfsPath> paths, T arg) Executes GGFS task asynchronously.
<T,R> GridFuture<R>	execute(GridGgfsTask<T,R> task, GridGgfsRecordResolver rslvr, Collection<GridGgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes GGFS task asynchronously with overridden maximum range length (see GridGgfsConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> GridFuture<R>	execute(GridGgfsTask<T,R> task, GridGgfsRecordResolver rslvr, Collection<GridGgfsPath> paths, T arg) Executes GGFS task asynchronously.
boolean	exists(GridGgfsPath path) Checks if the specified path exists in the file system.
GridFuture<?>	format() Formats the file system removing all existing entries from it.
GridGgfsFile	info(GridGgfsPath path) Gets file information for the specified path.
Collection<GridGgfsFile>	listFiles(GridGgfsPath path) Lists files under the specified path.
Collection<GridGgfsPath>	listPaths(GridGgfsPath path) Lists file paths under the specified path.
GridGgfsMetrics	metrics() Gets metrics snapshot for this file system.
void	mkdirs(GridGgfsPath path) Creates directories under specified path.
void	mkdirs(GridGgfsPath path, Map<String,String> props) Creates directories under specified path with the specified properties.
String	name() Gets GGFS name.
GridGgfsInputStream	open(GridGgfsPath path) Opens a file for reading.
GridGgfsInputStream	open(GridGgfsPath path, int bufSize) Opens a file for reading.
GridGgfsInputStream	open(GridGgfsPath path, int bufSize, int seqReadsBeforePrefetch) Opens a file for reading.
void	rename(GridGgfsPath src, GridGgfsPath dest) Renames/moves a file.
void	resetMetrics() Resets metrics for this file system.
void	setTimes(GridGgfsPath path, long accessTime, long modificationTime) Sets last access time and last modification time for a given path.
long	size(GridGgfsPath path) Determines size of the file denoted by provided path.
GridGgfsPathSummary	summary(GridGgfsPath path) Gets summary (total number of files, total number of directories and total length) for a given path.
GridGgfsFile	update(GridGgfsPath path, Map<String,String> props) Updates file information for the specified path.

Field Detail

PROP_USER_NAME

static final String PROP_USER_NAME

Property: user name.

See Also:

Constant Field Values

PROP_GROUP_NAME

static final String PROP_GROUP_NAME

Property: group name.

See Also:

Constant Field Values

PROP_PERMISSION

static final String PROP_PERMISSION

Property: permission.

See Also:

Constant Field Values

Method Detail

name

@Nullable
String name()

Gets GGFS name.

Returns:

GGFS name.

configuration

GridGgfsConfiguration configuration()

Gets GGFS configuration.

Returns:

GGFS configuration.

exists

boolean exists(GridGgfsPath path)
               throws GridException

Checks if the specified path exists in the file system.

Parameters:

path - Path to check for existence in the file system.

Returns:

True if such file exists, otherwise - false.

Throws:

GridException - In case of error.

info

@Nullable
GridGgfsFile info(GridGgfsPath path)
                  throws GridException

Gets file information for the specified path.

Parameters:

path - Path to get information for.

Returns:

File information for specified path or null if such path does not exist.

Throws:

GridException - In case of error.

summary

GridGgfsPathSummary summary(GridGgfsPath path)
                            throws GridException

Gets summary (total number of files, total number of directories and total length) for a given path.

Parameters:

path - Path to get information for.

Returns:

Summary object.

Throws:

GridGgfsFileNotFoundException - If path is not found.

GridException - If failed.

update

@Nullable
GridGgfsFile update(GridGgfsPath path,
                           Map<String,String> props)
                    throws GridException

Updates file information for the specified path. Existent properties, not listed in the passed collection, will not be affected. Other properties will be added or overwritten. Passed properties with null values will be removed from the stored properties or ignored if they don't exist in the file info.

When working in DUAL_SYNC or DUAL_ASYNC modes only the following properties will be propagated to the secondary file system:

• usrName - file owner name;
• grpName - file owner group;
• permission - Unix-style string representing file permissions.

Parameters:

path - File path to set properties for.

props - Properties to update.

Returns:

File information for specified path or null if such path does not exist.

Throws:

GridException - In case of error.

rename

void rename(GridGgfsPath src,
          GridGgfsPath dest)
            throws GridException

Renames/moves a file.

You are free to rename/move data files as you wish, but directories can be only renamed. You cannot move the directory between different parent directories.

Examples:

• "/work/file.txt" => "/home/project/Presentation Scenario.txt"
• "/work" => "/work-2012.bkp"
• "/work" => "/backups/work" - such operation is restricted for directories.

Parameters:

src - Source file path to rename.

dest - Destination file path. If destination path is a directory, then source file will be placed into destination directory with original name.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If source file doesn't exist.

delete

boolean delete(GridGgfsPath path,
             boolean recursive)
               throws GridException

Deletes file.

Parameters:

path - File path to delete.

recursive - Delete non-empty directories recursively.

Returns:

True in case of success, false otherwise.

Throws:

GridException - In case of error.

mkdirs

void mkdirs(GridGgfsPath path)
            throws GridException

Creates directories under specified path.

Parameters:

path - Path of directories chain to create.

Throws:

GridException - In case of error.

mkdirs

void mkdirs(GridGgfsPath path,
          @Nullable
          Map<String,String> props)
            throws GridException

Creates directories under specified path with the specified properties.

Parameters:

path - Path of directories chain to create.

props - Metadata properties to set on created directories.

Throws:

GridException - In case of error.

listPaths

Collection<GridGgfsPath> listPaths(GridGgfsPath path)
                                   throws GridException

Lists file paths under the specified path.

Parameters:

path - Path to list files under.

Returns:

List of files under the specified path.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

listFiles

Collection<GridGgfsFile> listFiles(GridGgfsPath path)
                                   throws GridException

Lists files under the specified path.

Parameters:

path - Path to list files under.

Returns:

List of files under the specified path.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

open

GridGgfsInputStream open(GridGgfsPath path)
                         throws GridException

Opens a file for reading.

Parameters:

path - File path to read.

Returns:

File input stream to read data from.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

open

GridGgfsInputStream open(GridGgfsPath path,
                       int bufSize)
                         throws GridException

Opens a file for reading.

Parameters:

path - File path to read.

bufSize - Read buffer size (bytes) or zero to use default value.

Returns:

File input stream to read data from.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

open

GridGgfsInputStream open(GridGgfsPath path,
                       int bufSize,
                       int seqReadsBeforePrefetch)
                         throws GridException

Opens a file for reading.

Parameters:

path - File path to read.

bufSize - Read buffer size (bytes) or zero to use default value.

seqReadsBeforePrefetch - Amount of sequential reads before prefetch is started.

Returns:

File input stream to read data from.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

create

GridGgfsOutputStream create(GridGgfsPath path,
                          boolean overwrite)
                            throws GridException

Creates a file and opens it for writing.

Parameters:

path - File path to create.

overwrite - Overwrite file if it already exists. Note: you cannot overwrite an existent directory.

Returns:

File output stream to write data to.

Throws:

GridException - In case of error.

create

GridGgfsOutputStream create(GridGgfsPath path,
                          int bufSize,
                          boolean overwrite,
                          @Nullable
                          GridUuid affKey,
                          int replication,
                          long blockSize,
                          @Nullable
                          Map<String,String> props)
                            throws GridException

Creates a file and opens it for writing.

Parameters:

path - File path to create.

bufSize - Write buffer size (bytes) or zero to use default value.

overwrite - Overwrite file if it already exists. Note: you cannot overwrite an existent directory.

affKey - Affinity key used to store file blocks. If not null, the whole file will be stored on node where affKey resides.

replication - Replication factor.

blockSize - Block size.

props - File properties to set.

Returns:

File output stream to write data to.

Throws:

GridException - In case of error.

append

GridGgfsOutputStream append(GridGgfsPath path,
                          boolean create)
                            throws GridException

Opens an output stream to an existing file for appending data.

Parameters:

path - File path to append.

create - Create file if it doesn't exist yet.

Returns:

File output stream to append data to.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist and create flag is false.

append

GridGgfsOutputStream append(GridGgfsPath path,
                          int bufSize,
                          boolean create,
                          @Nullable
                          Map<String,String> props)
                            throws GridException

Opens an output stream to an existing file for appending data.

Parameters:

path - File path to append.

bufSize - Write buffer size (bytes) or zero to use default value.

create - Create file if it doesn't exist yet.

props - File properties to set only in case it file was just created.

Returns:

File output stream to append data to.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist and create flag is false.

setTimes

void setTimes(GridGgfsPath path,
            long accessTime,
            long modificationTime)
              throws GridException

Sets last access time and last modification time for a given path. If argument is null, corresponding time will not be changed.

Parameters:

path - Path to update.

accessTime - Optional last access time to set. Value -1 does not update access time.

modificationTime - Optional last modification time to set. Value -1 does not update modification time.

Throws:

GridGgfsFileNotFoundException - If target was not found.

GridException - If error occurred.

affinity

Collection<GridGgfsBlockLocation> affinity(GridGgfsPath path,
                                         long start,
                                         long len)
                                           throws GridException

Gets affinity block locations for data blocks of the file, i.e. the nodes, on which the blocks are stored.

Parameters:

path - File path to get affinity for.

start - Position in the file to start affinity resolution from.

len - Size of data in the file to resolve affinity for.

Returns:

Affinity block locations.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

affinity

Collection<GridGgfsBlockLocation> affinity(GridGgfsPath path,
                                         long start,
                                         long len,
                                         long maxLen)
                                           throws GridException

Get affinity block locations for data blocks of the file. In case maxLen parameter is set and particular block location length is greater than this value, block locations will be split into smaller chunks.

Parameters:

path - File path to get affinity for.

start - Position in the file to start affinity resolution from.

len - Size of data in the file to resolve affinity for.

maxLen - Maximum length of a single returned block location length.

Returns:

Affinity block locations.

Throws:

GridException - In case of error.

GridGgfsFileNotFoundException - If path doesn't exist.

metrics

GridGgfsMetrics metrics()
                        throws GridException

Gets metrics snapshot for this file system.

Returns:

Metrics.

Throws:

GridException - In case of error.

resetMetrics

void resetMetrics()
                  throws GridException

Resets metrics for this file system.

Throws:

GridException - In case of error.

size

long size(GridGgfsPath path)
          throws GridException

Determines size of the file denoted by provided path. In case if path is a directory, then total size of all containing entries will be calculated recursively.

Parameters:

path - File system path.

Returns:

Total size.

Throws:

GridException - In case of error.

format

GridFuture<?> format()
                     throws GridException

Formats the file system removing all existing entries from it.

Returns:

Future completing when format is finished.

Throws:

GridException - In case format has failed.

execute

<T,R> GridFuture<R> execute(GridGgfsTask<T,R> task,
                          @Nullable
                          GridGgfsRecordResolver rslvr,
                          Collection<GridGgfsPath> paths,
                          @Nullable
                          T arg)
                      throws GridException

Executes GGFS task asynchronously.

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

Execution future.

Throws:

GridException - If execution failed.

execute

<T,R> GridFuture<R> execute(GridGgfsTask<T,R> task,
                          @Nullable
                          GridGgfsRecordResolver rslvr,
                          Collection<GridGgfsPath> paths,
                          boolean skipNonExistentFiles,
                          long maxRangeLen,
                          @Nullable
                          T arg)
                      throws GridException

Executes GGFS task asynchronously with overridden maximum range length (see GridGgfsConfiguration.getMaximumTaskRangeLength() for more information).

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Optional maximum range length. If 0, then by default all consecutive GGFS blocks will be included.

arg - Optional task argument.

Returns:

Execution future.

Throws:

GridException - If execution failed.

execute

<T,R> GridFuture<R> execute(Class<? extends GridGgfsTask<T,R>> taskCls,
                          @Nullable
                          GridGgfsRecordResolver rslvr,
                          Collection<GridGgfsPath> paths,
                          @Nullable
                          T arg)
                      throws GridException

Executes GGFS task asynchronously.

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

Execution future.

Throws:

GridException - If execution failed.

execute

<T,R> GridFuture<R> execute(Class<? extends GridGgfsTask<T,R>> taskCls,
                          @Nullable
                          GridGgfsRecordResolver rslvr,
                          Collection<GridGgfsPath> paths,
                          boolean skipNonExistentFiles,
                          long maxRangeLen,
                          @Nullable
                          T arg)
                      throws GridException

Executes GGFS task asynchronously with overridden maximum range length (see GridGgfsConfiguration.getMaximumTaskRangeLength() for more information).

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Maximum range length.

arg - Optional task argument.

Returns:

Execution future.

Throws:

GridException - If execution failed.