alluxio.underfs.s3a

Class S3ALowLevelOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.s3a.S3ALowLevelOutputStream

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable

@NotThreadSafe
public class S3ALowLevelOutputStream
extends java.io.OutputStream

[Experimental] A stream for writing a file into S3 using streaming upload. The data transfer is done using S3 low-level multipart upload. The multipart upload is initialized in the first write() and an upload id is given by AWS S3 to distinguish different multipart uploads. We upload data in partitions. When write(), the data will be persisted to a temporary file mFile on the local disk. When the data mPartitionOffset in this temporary file reaches the mPartitionSize, the file will be submitted to the upload executor mExecutor and we do not wait for uploads to finish. A new temp file will be created for the future write and the mPartitionOffset will be reset to zero. The process goes until all the data has been written to temp files. In flush(), we upload the buffered data if they are bigger than 5MB and wait for all uploads to finish. The temp files will be deleted after uploading successfully. In close(), we upload the last part of data (if exists), wait for all uploads to finish, and complete the multipart upload. close() will not be retried, but all the multipart upload related operations(init, upload, complete, and abort) will be retried. If an error occurs and we have no way to recover, we abort the multipart uploads. Some multipart uploads may not be completed/aborted in normal ways and need periodical cleanup by enabling the PropertyKey.UNDERFS_CLEANUP_ENABLED. When a leader master starts or a cleanup interval is reached, all the multipart uploads older than PropertyKey.UNDERFS_S3_INTERMEDIATE_UPLOAD_CLEAN_AGE will be cleaned.

Field Summary

Fields

Modifier and Type	Field and Description
protected com.amazonaws.services.s3.AmazonS3	mClient The Amazon S3 client to interact with S3.
protected java.lang.String	mKey Key of the file when it is uploaded to S3.

Constructor Summary

Constructors

Constructor and Description
S3ALowLevelOutputStream(java.lang.String bucketName, java.lang.String key, com.amazonaws.services.s3.AmazonS3 s3Client, com.google.common.util.concurrent.ListeningExecutorService executor, long streamingUploadPartitionSize, java.util.List<java.lang.String> tmpDirs, boolean sseEnabled) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
protected void	abortMultiPartUpload()
protected void	abortMultiPartUpload(com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String uploadId) Aborts multipart upload.
void	close()
protected void	completeMultiPartUpload()
protected void	completeMultiPartUpload(com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String uploadId) Completes multipart upload.
protected void	execUpload(com.amazonaws.services.s3.AmazonS3 s3Client, com.amazonaws.services.s3.model.UploadPartRequest request) Executes the upload part request.
protected void	execUpload(com.amazonaws.services.s3.model.UploadPartRequest request)
void	flush()
protected com.amazonaws.services.s3.AmazonS3	getClient()
protected void	initMultiPartUpload()
protected void	uploadPart() Uploads part async.
protected void	waitForAllPartsUpload() Waits for the submitted upload tasks to finish.
void	write(byte[] b)
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mClient

protected com.amazonaws.services.s3.AmazonS3 mClient

The Amazon S3 client to interact with S3.

mKey

protected final java.lang.String mKey

Key of the file when it is uploaded to S3.

Constructor Detail

S3ALowLevelOutputStream

public S3ALowLevelOutputStream(java.lang.String bucketName,
                               java.lang.String key,
                               com.amazonaws.services.s3.AmazonS3 s3Client,
                               com.google.common.util.concurrent.ListeningExecutorService executor,
                               long streamingUploadPartitionSize,
                               java.util.List<java.lang.String> tmpDirs,
                               boolean sseEnabled)

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

s3Client - the Amazon S3 client to upload the file with

executor - a thread pool executor

streamingUploadPartitionSize - the size in bytes for partitions of streaming uploads

tmpDirs - a list of temporary directories

sseEnabled - whether or not server side encryption is enabled

Method Detail

write

public void write(int b)
           throws java.io.IOException

Specified by:

write in class java.io.OutputStream

Throws:

java.io.IOException

write

public void write(byte[] b)
           throws java.io.IOException

Overrides:

write in class java.io.OutputStream

Throws:

java.io.IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws java.io.IOException

Overrides:

write in class java.io.OutputStream

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Specified by:

flush in interface java.io.Flushable

Overrides:

flush in class java.io.OutputStream

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class java.io.OutputStream

Throws:

java.io.IOException

initMultiPartUpload

protected void initMultiPartUpload()
                            throws java.io.IOException

Throws:

java.io.IOException

uploadPart

protected void uploadPart()
                   throws java.io.IOException

Uploads part async.

Throws:

java.io.IOException

execUpload

protected void execUpload(com.amazonaws.services.s3.model.UploadPartRequest request)
                   throws java.io.IOException

Throws:

java.io.IOException

execUpload

protected void execUpload(com.amazonaws.services.s3.AmazonS3 s3Client,
                          com.amazonaws.services.s3.model.UploadPartRequest request)

Executes the upload part request.

Parameters:

request - the upload part request

waitForAllPartsUpload

protected void waitForAllPartsUpload()
                              throws java.io.IOException

Waits for the submitted upload tasks to finish.

Throws:

java.io.IOException

completeMultiPartUpload

protected void completeMultiPartUpload()
                                throws java.io.IOException

Throws:

java.io.IOException

completeMultiPartUpload

protected void completeMultiPartUpload(com.amazonaws.services.s3.AmazonS3 s3Client,
                                       java.lang.String uploadId)
                                throws java.io.IOException

Completes multipart upload.

Throws:

java.io.IOException

abortMultiPartUpload

protected void abortMultiPartUpload()

abortMultiPartUpload

protected void abortMultiPartUpload(com.amazonaws.services.s3.AmazonS3 s3Client,
                                    java.lang.String uploadId)

Aborts multipart upload.

getClient

protected com.amazonaws.services.s3.AmazonS3 getClient()

Returns:

the client