Package org.lwjgl.util.zstd

Class Zstd

  • public class Zstd
extends java.lang.Object
    Native bindings to Zstandard (zstd), a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios.

    Introduction

    zstd, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. The zstd compression library provides in-memory compression and decompression functions. The library supports compression levels from 1 up to maxCLevel which is currently 22. Levels ≥ 20, labeled --ultra, should be used with caution, as they require more memory.

    Compression can be done in:

    • a single step (described as Simple API)
    • a single step, reusing a context (described as Explicit context)
    • unbounded multiple steps (described as Streaming compression)

    The compression ratio achievable on small data can be highly improved using a dictionary in:

    • a single step (described as Simple dictionary API)
    • a single step, reusing a dictionary (described as Bulk-processing dictionary API)

    Advanced experimental functions can be accessed using #define ZSTD_STATIC_LINKING_ONLY before including zstd.h. Advanced experimental APIs shall never be used with a dynamic library. They are not "stable", their definition may change in the future. Only static linking is allowed.

    Streaming compression - HowTo

    A ZSTD_CStream object is required to track streaming operation.

    Use createCStream and freeCStream to create/release resources. ZSTD_CStream objects can be reused multiple times on consecutive compression operations. It is recommended to re-use ZSTD_CStream in situations where many streaming operations will be achieved consecutively, since it will play nicer with system's memory, by re-using already allocated memory. Use one separate ZSTD_CStream per thread for parallel execution.

    Start a new compression by initializing ZSTD_CStream context. Use initCStream to start a new compression operation.

    Use compressStream as many times as necessary to consume input stream. The function will automatically update both pos fields within input and output. Note that the function may not consume the entire input, for example, because the output buffer is already full, in which case input.pos < input.size. The caller must check if input has been entirely consumed. If not, the caller must make some room to receive more compressed data, typically by emptying output buffer, or allocating a new output buffer, and then present again remaining input data.

    At any moment, it's possible to flush whatever data might remain stuck within internal buffer, using flushStream. output->pos will be updated. Note that, if output->size is too small, a single invocation of ZSTD_flushStream() might not be enough (return code > 0). In which case, make some room to receive more compressed data, and call again ZSTD_flushStream().

    endStream instructs to finish a frame. It will perform a flush and write frame epilogue. The epilogue is required for decoders to consider a frame completed. flush() operation is the same, and follows same rules as ZSTD_flushStream().

    Streaming decompression - HowTo

    A ZSTD_DStream object is required to track streaming operations.

    Use createDStream and freeDStream to create/release resources. ZSTD_DStream objects can be re-used multiple times.

    Use initDStream to start a new decompression operation, or ZSTD_initDStream_usingDict() if decompression requires a dictionary.

    Use decompressStream repetitively to consume your input. The function will update both pos fields. If input.pos < input.size, some input has not been consumed. It's up to the caller to present again remaining data. If output.pos < output.size, decoder has flushed everything it could.

    • Method Detail

      • ZSTD_versionNumber

        public static int ZSTD_versionNumber()
        Returns the version number.

      • nZSTD_versionString

        public static long nZSTD_versionString()
        Unsafe version of: versionString

      • ZSTD_versionString

        public static java.lang.String ZSTD_versionString()
        Returns the version string.

      • nZSTD_compress

        public static long nZSTD_compress​(long dst,
                                  long dstCapacity,
                                  long src,
                                  long srcSize,
                                  int compressionLevel)
        Unsafe version of: compress

      • ZSTD_compress

        public static long ZSTD_compress​(java.nio.ByteBuffer dst,
                                 java.nio.ByteBuffer src,
                                 int compressionLevel)
        Compresses src content as a single zstd compressed frame into already allocated dst.

        Hint: compression runs faster if dstCapacity ≥ compressBound(srcSize)

        Returns:
        compressed size written into dst (≤ dstCapacity), or an error code if it fails (which can be tested using isError).

      • nZSTD_decompress

        public static long nZSTD_decompress​(long dst,
                                    long dstCapacity,
                                    long src,
                                    long compressedSize)
        Unsafe version of: decompress
        Parameters:
        dstCapacity - is an upper bound of originalSize to regenerate. If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
        compressedSize - must be the exact size of some number of compressed and/or skippable frames

      • ZSTD_decompress

        public static long ZSTD_decompress​(java.nio.ByteBuffer dst,
                                   java.nio.ByteBuffer src)
        Returns:
        the number of bytes decompressed into dst (≤ dstCapacity), or an errorCode if it fails (which can be tested using isError).

      • nZSTD_getFrameContentSize

        public static long nZSTD_getFrameContentSize​(long src,
                                             long srcSize)
        Unsafe version of: getFrameContentSize
        Parameters:
        srcSize - must be at least as large as the frame header. Hint: any size ≥ FRAMEHEADERSIZE_MAX is large enough.

      • ZSTD_getFrameContentSize

        public static long ZSTD_getFrameContentSize​(java.nio.ByteBuffer src)
        Notes:
        1. a 0 return value means the frame is valid but "empty"
        2. decompressed size is an optional field, it may not be present, typically in streaming mode. When return==ZSTD_CONTENTSIZE_UNKNOWN, data to decompress could be any size. In which case, it's necessary to use streaming mode to decompress data. Optionally, application can rely on some implicit limit, as decompress only needs an upper bound of decompressed size. (For example, data could be necessarily cut into blocks ≤ 16 KB).
        3. decompressed size is always present when compression is completed using single-pass functions, such as compress, compressCCtx, compress_usingDict or compress_usingCDict.
        4. decompressed size can be very large (64-bits value), potentially larger than what local system can handle as a single memory segment. In which case, it's necessary to use streaming mode to decompress data.
        5. If source is untrusted, decompressed size could be wrong or intentionally modified. Always ensure return value fits within application's authorized limits. Each application can set its own limits.
        Parameters:
        src - should point to the start of a ZSTD encoded frame
        Returns:
        decompressed size of src frame content, if known

      • ZSTD_compressBound

        public static long ZSTD_compressBound​(long srcSize)
        Returns the maximum compressed size in worst case single-pass scenario.

      • nZSTD_isError

        public static int nZSTD_isError​(long code)
        Unsafe version of: isError

      • ZSTD_isError

        public static boolean ZSTD_isError​(long code)
        Tells if a size_t function result is an error code.

      • nZSTD_getErrorName

        public static long nZSTD_getErrorName​(long code)
        Unsafe version of: getErrorName

      • ZSTD_getErrorName

        public static java.lang.String ZSTD_getErrorName​(long code)
        Provides readable string from an error code.

      • ZSTD_maxCLevel

        public static int ZSTD_maxCLevel()
        Returns the maximum compression level available.

      • ZSTD_createCCtx

        public static long ZSTD_createCCtx()
        Creates a compression context.

        When compressing many times, it is recommended to allocate a context just once, and re-use it for each successive compression operation. This will make workload friendlier for system's memory. Use one context per thread for parallel execution in multi-threaded environments.

      • nZSTD_freeCCtx

        public static long nZSTD_freeCCtx​(long cctx)
        Unsafe version of: freeCCtx

      • ZSTD_freeCCtx

        public static long ZSTD_freeCCtx​(long cctx)
        Frees memory allocated by createCCtx.

      • nZSTD_compressCCtx

        public static long nZSTD_compressCCtx​(long ctx,
                                      long dst,
                                      long dstCapacity,
                                      long src,
                                      long srcSize,
                                      int compressionLevel)
        Unsafe version of: compressCCtx

      • ZSTD_compressCCtx

        public static long ZSTD_compressCCtx​(long ctx,
                                     java.nio.ByteBuffer dst,
                                     java.nio.ByteBuffer src,
                                     int compressionLevel)
        Same as compress, requires an allocated ZSTD_CCtx (see createCCtx).

      • ZSTD_createDCtx

        public static long ZSTD_createDCtx()
        Creates a decompression context.

        When decompressing many times, it is recommended to allocate a context only once, and re-use it for each successive compression operation. This will make workload friendlier for system's memory. Use one context per thread for parallel execution.

      • nZSTD_freeDCtx

        public static long nZSTD_freeDCtx​(long dctx)
        Unsafe version of: freeDCtx

      • ZSTD_freeDCtx

        public static long ZSTD_freeDCtx​(long dctx)
        Frees memory allocated by createDCtx.

      • nZSTD_decompressDCtx

        public static long nZSTD_decompressDCtx​(long ctx,
                                        long dst,
                                        long dstCapacity,
                                        long src,
                                        long srcSize)
        Unsafe version of: decompressDCtx

      • ZSTD_decompressDCtx

        public static long ZSTD_decompressDCtx​(long ctx,
                                       java.nio.ByteBuffer dst,
                                       java.nio.ByteBuffer src)
        Same as decompress, requires an allocated ZSTD_DCtx (see createDCtx).

      • nZSTD_compress_usingDict

        public static long nZSTD_compress_usingDict​(long ctx,
                                            long dst,
                                            long dstCapacity,
                                            long src,
                                            long srcSize,
                                            long dict,
                                            long dictSize,
                                            int compressionLevel)
        Unsafe version of: compress_usingDict

      • ZSTD_compress_usingDict

        public static long ZSTD_compress_usingDict​(long ctx,
                                           java.nio.ByteBuffer dst,
                                           java.nio.ByteBuffer src,
                                           @Nullable
                                           java.nio.ByteBuffer dict,
                                           int compressionLevel)
        Compression using a predefined Dictionary (see dictBuilder/zdict.h).

        This function loads the dictionary, resulting in significant startup delay.

        When dict == NULL || dictSize < 8 no dictionary is used.

      • nZSTD_decompress_usingDict

        public static long nZSTD_decompress_usingDict​(long dctx,
                                              long dst,
                                              long dstCapacity,
                                              long src,
                                              long srcSize,
                                              long dict,
                                              long dictSize)
        Unsafe version of: decompress_usingDict

      • ZSTD_decompress_usingDict

        public static long ZSTD_decompress_usingDict​(long dctx,
                                             java.nio.ByteBuffer dst,
                                             java.nio.ByteBuffer src,
                                             @Nullable
                                             java.nio.ByteBuffer dict)
        Decompression using a predefined Dictionary (see dictBuilder/zdict.h). Dictionary must be identical to the one used during compression.

        This function loads the dictionary, resulting in significant startup delay.

        When dict == NULL || dictSize < 8 no dictionary is used.

      • nZSTD_createCDict

        public static long nZSTD_createCDict​(long dictBuffer,
                                     long dictSize,
                                     int compressionLevel)
        Unsafe version of: createCDict

      • ZSTD_createCDict

        public static long ZSTD_createCDict​(java.nio.ByteBuffer dictBuffer,
                                    int compressionLevel)
        When compressing multiple messages / blocks with the same dictionary, it's recommended to load it just once.

        ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup delay. ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.

        dictBuffer can be released after ZSTD_CDict creation, since its content is copied within CDict.

      • nZSTD_freeCDict

        public static long nZSTD_freeCDict​(long CDict)
        Unsafe version of: freeCDict

      • ZSTD_freeCDict

        public static long ZSTD_freeCDict​(long CDict)
        Frees memory allocated by createCDict.

      • nZSTD_compress_usingCDict

        public static long nZSTD_compress_usingCDict​(long cctx,
                                             long dst,
                                             long dstCapacity,
                                             long src,
                                             long srcSize,
                                             long cdict)
        Unsafe version of: compress_usingCDict

      • ZSTD_compress_usingCDict

        public static long ZSTD_compress_usingCDict​(long cctx,
                                            java.nio.ByteBuffer dst,
                                            java.nio.ByteBuffer src,
                                            long cdict)
        Compression using a digested Dictionary.

        Faster startup than compress_usingDict, recommended when same dictionary is used multiple times. Note that compression level is decided during dictionary creation. Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)

      • nZSTD_createDDict

        public static long nZSTD_createDDict​(long dictBuffer,
                                     long dictSize)
        Unsafe version of: createDDict

      • ZSTD_createDDict

        public static long ZSTD_createDDict​(java.nio.ByteBuffer dictBuffer)
        Creates a digested dictionary, ready to start decompression operation without startup delay.

        dictBuffer can be released after DDict creation, as its content is copied inside DDict.

      • nZSTD_freeDDict

        public static long nZSTD_freeDDict​(long ddict)
        Unsafe version of: freeDDict

      • ZSTD_freeDDict

        public static long ZSTD_freeDDict​(long ddict)
        Frees memory allocated with createDDict.

      • nZSTD_decompress_usingDDict

        public static long nZSTD_decompress_usingDDict​(long dctx,
                                               long dst,
                                               long dstCapacity,
                                               long src,
                                               long srcSize,
                                               long ddict)
        Unsafe version of: decompress_usingDDict

      • ZSTD_decompress_usingDDict

        public static long ZSTD_decompress_usingDDict​(long dctx,
                                              java.nio.ByteBuffer dst,
                                              java.nio.ByteBuffer src,
                                              long ddict)
        Decompression using a digested Dictionary.

        Faster startup than decompress_usingDict, recommended when same dictionary is used multiple times.

      • ZSTD_createCStream

        public static long ZSTD_createCStream()

      • nZSTD_freeCStream

        public static long nZSTD_freeCStream​(long zcs)
        Unsafe version of: freeCStream

      • ZSTD_freeCStream

        public static long ZSTD_freeCStream​(long zcs)
        Frees memory allocated by createCStream.

      • nZSTD_initCStream

        public static long nZSTD_initCStream​(long zcs,
                                     int compressionLevel)

      • ZSTD_initCStream

        public static long ZSTD_initCStream​(long zcs,
                                    int compressionLevel)

      • nZSTD_compressStream

        public static long nZSTD_compressStream​(long zcs,
                                        long output,
                                        long input)
        Unsafe version of: compressStream

      • ZSTD_compressStream

        public static long ZSTD_compressStream​(long zcs,
                                       ZSTDOutBuffer output,
                                       ZSTDInBuffer input)
        Returns:
        a size hint, preferred nb of bytes to use as input for next function call or an error code, which can be tested using isError.

        Notes:

        1. it's just a hint, to help latency a little, any other value will work fine
        2. size hint is guaranteed to be ≤ CStreamInSize

      • nZSTD_flushStream

        public static long nZSTD_flushStream​(long zcs,
                                     long output)
        Unsafe version of: flushStream

      • ZSTD_flushStream

        public static long ZSTD_flushStream​(long zcs,
                                    ZSTDOutBuffer output)
        Returns:
        nb of bytes still present within internal buffer (0 if it's empty) or an error code, which can be tested using isError

      • nZSTD_endStream

        public static long nZSTD_endStream​(long zcs,
                                   long output)
        Unsafe version of: endStream

      • ZSTD_endStream

        public static long ZSTD_endStream​(long zcs,
                                  ZSTDOutBuffer output)
        Returns:
        0 if frame fully completed and fully flushed, or > 0 if some data is still present within internal buffer (value is minimum size estimation for remaining data to flush, but it could be more) or an error code, which can be tested using isError

      • ZSTD_CStreamInSize

        public static long ZSTD_CStreamInSize()
        Returns the recommended size for input buffer.

      • ZSTD_CStreamOutSize

        public static long ZSTD_CStreamOutSize()
        Returns the recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block in all circumstances.

      • ZSTD_createDStream

        public static long ZSTD_createDStream()

      • nZSTD_freeDStream

        public static long nZSTD_freeDStream​(long zds)
        Unsafe version of: freeDStream

      • ZSTD_freeDStream

        public static long ZSTD_freeDStream​(long zds)
        Frees memory allocated by createDStream.

      • nZSTD_initDStream

        public static long nZSTD_initDStream​(long zds)
        Unsafe version of: initDStream

      • ZSTD_initDStream

        public static long ZSTD_initDStream​(long zds)
        Returns:
        recommended first input size

      • nZSTD_decompressStream

        public static long nZSTD_decompressStream​(long zds,
                                          long output,
                                          long input)
        Unsafe version of: decompressStream

      • ZSTD_decompressStream

        public static long ZSTD_decompressStream​(long zds,
                                         ZSTDOutBuffer output,
                                         ZSTDInBuffer input)
        Returns:
        0 when a frame is completely decoded and fully flushed, an error code, which can be tested using isError, any other value > 0, which means there is still some decoding to do to complete current frame. The return value is a suggested next input size (a hint to improve latency) that will never load more than the current frame.

      • ZSTD_DStreamInSize

        public static long ZSTD_DStreamInSize()
        Returns the recommended size for input buffer.

      • ZSTD_DStreamOutSize

        public static long ZSTD_DStreamOutSize()
        Returns the recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block in all circumstances.

      • ZSTD_COMPRESSBOUND

        public static long ZSTD_COMPRESSBOUND​(long srcSize)
        Pure Java version of ZSTD_compressBound(long).