alluxio.underfs.s3a

Class S3AUnderFileSystem

java.lang.Object

alluxio.underfs.BaseUnderFileSystem

alluxio.underfs.ObjectUnderFileSystem

alluxio.underfs.s3a.S3AUnderFileSystem

All Implemented Interfaces:

UnderFileSystem, java.io.Closeable, java.lang.AutoCloseable

@ThreadSafe
public class S3AUnderFileSystem
extends ObjectUnderFileSystem

S3 UnderFileSystem implementation based on the aws-java-sdk-s3 library.

Nested Class Summary

Nested classes/interfaces inherited from class alluxio.underfs.ObjectUnderFileSystem

ObjectUnderFileSystem.DeleteBuffer, ObjectUnderFileSystem.ObjectListingChunk, ObjectUnderFileSystem.ObjectPermissions, ObjectUnderFileSystem.ObjectStatus, ObjectUnderFileSystem.OperationBuffer<T>, ObjectUnderFileSystem.RenameBuffer

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields inherited from class alluxio.underfs.ObjectUnderFileSystem

mExecutorService, mRootKeySupplier, PATH_SEPARATOR, PATH_SEPARATOR_CHAR

Fields inherited from class alluxio.underfs.BaseUnderFileSystem

mUfsConf, mUri

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	S3AUnderFileSystem(AlluxioURI uri, com.amazonaws.services.s3.AmazonS3 amazonS3Client, java.lang.String bucketName, java.util.concurrent.ExecutorService executor, com.amazonaws.services.s3.transfer.TransferManager transferManager, UnderFileSystemConfiguration conf, boolean streamingUploadEnabled) Constructor for S3AUnderFileSystem.

Method Summary

Modifier and Type	Method and Description
void	cleanup()
protected boolean	copyObject(java.lang.String src, java.lang.String dst)
static com.amazonaws.services.s3.AmazonS3	createAmazonS3(com.amazonaws.auth.AWSCredentialsProvider credentialsProvider, com.amazonaws.ClientConfiguration clientConf, com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration endpointConfiguration, UnderFileSystemConfiguration conf) Create an AmazonS3 client.
static com.amazonaws.auth.AWSCredentialsProvider	createAwsCredentialsProvider(UnderFileSystemConfiguration conf)
boolean	createEmptyObject(java.lang.String key)
static S3AUnderFileSystem	createInstance(AlluxioURI uri, UnderFileSystemConfiguration conf) Constructs a new instance of S3AUnderFileSystem.
protected java.io.OutputStream	createObject(java.lang.String key)
protected boolean	deleteObject(java.lang.String key)
protected java.util.List<java.lang.String>	deleteObjects(java.util.List<java.lang.String> keys)
protected java.lang.String	getFolderSuffix()
protected ObjectUnderFileSystem.ObjectListingChunk	getObjectListingChunk(java.lang.String key, boolean recursive)
protected ObjectUnderFileSystem.ObjectStatus	getObjectStatus(java.lang.String key)
protected ObjectUnderFileSystem.ObjectPermissions	getPermissions()
protected java.lang.String	getRootKey()
java.lang.String	getUnderFSType()
protected java.io.InputStream	openObject(java.lang.String key, OpenOptions options, RetryPolicy retryPolicy)
void	setMode(java.lang.String path, short mode)
void	setOwner(java.lang.String path, java.lang.String user, java.lang.String group)

Methods inherited from class alluxio.underfs.ObjectUnderFileSystem

close, connectFromMaster, connectFromWorker, convertToFolderName, create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, deleteFile, getBlockSizeByte, getChildName, getDirectoryStatus, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileLocations, getFileLocations, getFileStatus, getListingChunkLength, getListingChunkLengthMax, getObjectListingChunkForPath, getParentPath, getSpace, getStatus, isDirectory, isExistingDirectory, isFile, isObjectStorage, isRoot, listInternal, listStatus, listStatus, mkdirs, mkdirsInternal, open, openExistingFile, openExistingFile, parentExists, renameDirectory, renameFile, renameRenamableDirectory, renameRenamableFile, stripPrefixIfPresent, supportsFlush

Methods inherited from class alluxio.underfs.BaseUnderFileSystem

create, deleteDirectory, exists, getAclPair, getActiveSyncInfo, getConfiguration, getFingerprint, getOperationMode, getPhysicalStores, isSeekable, mkdirs, open, resolveUri, setAclEntries, startActiveSyncPolling, startSync, stopActiveSyncPolling, stopSync, supportsActiveSync, validatePath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3AUnderFileSystem

protected S3AUnderFileSystem(AlluxioURI uri,
                             com.amazonaws.services.s3.AmazonS3 amazonS3Client,
                             java.lang.String bucketName,
                             java.util.concurrent.ExecutorService executor,
                             com.amazonaws.services.s3.transfer.TransferManager transferManager,
                             UnderFileSystemConfiguration conf,
                             boolean streamingUploadEnabled)

Constructor for S3AUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

amazonS3Client - AWS-SDK S3 client

bucketName - bucket name of user's configured Alluxio bucket

executor - the executor for executing upload tasks

transferManager - Transfer Manager for efficient I/O to S3

conf - configuration for this S3A ufs

streamingUploadEnabled - whether streaming upload is enabled

Method Detail

createAwsCredentialsProvider

public static com.amazonaws.auth.AWSCredentialsProvider createAwsCredentialsProvider(UnderFileSystemConfiguration conf)

Parameters:

conf - the configuration for this UFS

Returns:

the created AWSCredentialsProvider instance

createInstance

public static S3AUnderFileSystem createInstance(AlluxioURI uri,
                                                UnderFileSystemConfiguration conf)

Constructs a new instance of S3AUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

conf - the configuration for this UFS

Returns:

the created S3AUnderFileSystem instance

createAmazonS3

public static com.amazonaws.services.s3.AmazonS3 createAmazonS3(com.amazonaws.auth.AWSCredentialsProvider credentialsProvider,
                                                                com.amazonaws.ClientConfiguration clientConf,
                                                                com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration endpointConfiguration,
                                                                UnderFileSystemConfiguration conf)

Create an AmazonS3 client.

Parameters:

credentialsProvider - the credential provider

clientConf - the client config

endpointConfiguration - the endpoint config

conf - the Ufs config

Returns:

the AmazonS3 client

getUnderFSType

public java.lang.String getUnderFSType()

setOwner

public void setOwner(java.lang.String path,
                     java.lang.String user,
                     java.lang.String group)

setMode

public void setMode(java.lang.String path,
                    short mode)
             throws java.io.IOException

Throws:

java.io.IOException

cleanup

public void cleanup()

Specified by:

cleanup in interface UnderFileSystem

Overrides:

cleanup in class ObjectUnderFileSystem

copyObject

protected boolean copyObject(java.lang.String src,
                             java.lang.String dst)

Specified by:

copyObject in class ObjectUnderFileSystem

createEmptyObject

public boolean createEmptyObject(java.lang.String key)

Specified by:

createEmptyObject in class ObjectUnderFileSystem

createObject

protected java.io.OutputStream createObject(java.lang.String key)
                                     throws java.io.IOException

Specified by:

createObject in class ObjectUnderFileSystem

Throws:

java.io.IOException

deleteObject

protected boolean deleteObject(java.lang.String key)
                        throws java.io.IOException

Specified by:

deleteObject in class ObjectUnderFileSystem

Throws:

java.io.IOException

deleteObjects

protected java.util.List<java.lang.String> deleteObjects(java.util.List<java.lang.String> keys)
                                                  throws java.io.IOException

Overrides:

deleteObjects in class ObjectUnderFileSystem

Throws:

java.io.IOException

getFolderSuffix

protected java.lang.String getFolderSuffix()

Specified by:

getFolderSuffix in class ObjectUnderFileSystem

getObjectListingChunk

@Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(java.lang.String key,
                                                                                   boolean recursive)
                                                                            throws java.io.IOException

Specified by:

getObjectListingChunk in class ObjectUnderFileSystem

Throws:

java.io.IOException

getObjectStatus

@Nullable
protected ObjectUnderFileSystem.ObjectStatus getObjectStatus(java.lang.String key)
                                                                throws java.io.IOException

Specified by:

getObjectStatus in class ObjectUnderFileSystem

Throws:

java.io.IOException

getPermissions

protected ObjectUnderFileSystem.ObjectPermissions getPermissions()

Specified by:

getPermissions in class ObjectUnderFileSystem

getRootKey

protected java.lang.String getRootKey()

Specified by:

getRootKey in class ObjectUnderFileSystem

openObject

protected java.io.InputStream openObject(java.lang.String key,
                                         OpenOptions options,
                                         RetryPolicy retryPolicy)

Specified by:

openObject in class ObjectUnderFileSystem