org.bytedeco.tensorrt.nvinfer

Class IExecutionContext

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.tensorrt.nvinfer.INoCopy

org.bytedeco.tensorrt.nvinfer.IExecutionContext

All Implemented Interfaces:

AutoCloseable

@Namespace(value="nvinfer1")
 @NoOffset
 @Properties(inherit=nvinfer.class)
public class IExecutionContext
extends INoCopy

\class IExecutionContext \brief Context for executing inference using an engine, with functionally unsafe features. Multiple execution contexts may exist for one ICudaEngine instance, allowing the same engine to be used for the execution of multiple batches simultaneously. If the engine supports dynamic shapes, each execution context in concurrent use must use a separate optimization profile. \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
IExecutionContext() Default native constructor.
IExecutionContext(long size) Native array allocator.
IExecutionContext(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
boolean	allInputDimensionsSpecified() \brief Whether all dynamic dimensions of input tensors have been specified
boolean	allInputShapesSpecified() \brief Whether all input shape bindings have been specified
void	destroy() Deprecated. Use delete instead. Deprecated in TRT 8.0. \warning Calling destroy on a managed pointer will result in a double-free error.
boolean	enqueue(int batchSize, Pointer bindings, org.bytedeco.cuda.cudart.CUstream_st stream, org.bytedeco.cuda.cudart.CUevent_st inputConsumed) Deprecated.
boolean	enqueue(int batchSize, PointerPointer bindings, org.bytedeco.cuda.cudart.CUstream_st stream, org.bytedeco.cuda.cudart.CUevent_st inputConsumed) Deprecated. Deprecated in TensorRT 8.4. Superseded by enqueueV2() if the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag.
boolean	enqueueV2(Pointer bindings, org.bytedeco.cuda.cudart.CUstream_st stream, org.bytedeco.cuda.cudart.CUevent_st inputConsumed)
boolean	enqueueV2(PointerPointer bindings, org.bytedeco.cuda.cudart.CUstream_st stream, org.bytedeco.cuda.cudart.CUevent_st inputConsumed) \brief Asynchronously execute inference.
boolean	execute(int batchSize, Pointer bindings) Deprecated.
boolean	execute(int batchSize, PointerPointer bindings) Deprecated. Deprecated in TensorRT 8.4. Superseded by executeV2() if the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. \warning This function will trigger layer resource updates if hasImplicitBatchDimension() returns true and batchSize changes between subsequent calls, possibly resulting in performance bottlenecks.
boolean	executeV2(Pointer bindings)
boolean	executeV2(PointerPointer bindings) \brief Synchronously execute inference a network.
Dims32	getBindingDimensions(int bindingIndex) \brief Get the dynamic dimensions of a binding If the engine was built with an implicit batch dimension, same as ICudaEngine::getBindingDimensions.
boolean	getDebugSync() \brief Get the debug sync flag.
ICudaEngine	getEngine() \brief Get the associated engine.
boolean	getEnqueueEmitsProfile() \brief Get the enqueueEmitsProfile state.
IErrorRecorder	getErrorRecorder() \brief Get the ErrorRecorder assigned to this interface.
String	getName() \brief Return the name of the execution context.
int	getOptimizationProfile() \brief Get the index of the currently selected optimization profile.
IExecutionContext	getPointer(long i)
IProfiler	getProfiler() \brief Get the profiler.
boolean	getShapeBinding(int bindingIndex, int[] data)
boolean	getShapeBinding(int bindingIndex, IntBuffer data)
boolean	getShapeBinding(int bindingIndex, IntPointer data) \brief Get values of an input tensor required for shape calculations or an output tensor produced by shape calculations.
Dims32	getStrides(int bindingIndex) \brief Return the strides of the buffer for the given binding.
IExecutionContext	position(long position)
boolean	reportToProfiler() \brief Calculate layer timing info for the current optimization profile in IExecutionContext and update the profiler after one iteration of inference launch.
boolean	setBindingDimensions(int bindingIndex, Dims32 dimensions) \brief Set the dynamic dimensions of a binding
void	setDebugSync(boolean sync) \brief Set the debug sync flag.
void	setDeviceMemory(Pointer memory) \brief Set the device memory for use by this execution context.
void	setEnqueueEmitsProfile(boolean enqueueEmitsProfile) \brief Set whether enqueue emits layer timing to the profiler If set to true (default), enqueue is synchronous and does layer timing profiling implicitly if there is a profiler attached.
void	setErrorRecorder(IErrorRecorder recorder)
boolean	setInputShapeBinding(int bindingIndex, int[] data)
boolean	setInputShapeBinding(int bindingIndex, IntBuffer data)
boolean	setInputShapeBinding(int bindingIndex, IntPointer data) \brief Set values of input tensor required by shape calculations.
void	setName(BytePointer name)
void	setName(String name) \brief Set the name of the execution context.
boolean	setOptimizationProfile(int profileIndex) Deprecated. Superseded by setOptimizationProfileAsync. Deprecated prior to TensorRT 8.0 and will be removed in 9.0.
boolean	setOptimizationProfileAsync(int profileIndex, org.bytedeco.cuda.cudart.CUstream_st stream) \brief Select an optimization profile for the current context with async semantics.
void	setProfiler(IProfiler profiler) \brief Set the profiler.

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

IExecutionContext

public IExecutionContext()

Default native constructor.

IExecutionContext

public IExecutionContext(long size)

Native array allocator. Access with Pointer.position(long).

IExecutionContext

public IExecutionContext(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

position

public IExecutionContext position(long position)

Overrides:

position in class Pointer

getPointer

public IExecutionContext getPointer(long i)

Overrides:

getPointer in class Pointer

execute

@Cast(value="bool")
 @Deprecated
 @NoException(value=true)
public boolean execute(int batchSize,
                                                                                     @Cast(value="void*const*")
                                                                                     PointerPointer bindings)

Deprecated. Deprecated in TensorRT 8.4. Superseded by executeV2() if the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. \warning This function will trigger layer resource updates if hasImplicitBatchDimension() returns true and batchSize changes between subsequent calls, possibly resulting in performance bottlenecks.

\brief Synchronously execute inference on a batch. This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried using ICudaEngine::getBindingIndex()

Parameters:

batchSize - The batch size. This is at most the max batch size value supplied to the builder when the engine was built. If the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag, please use executeV2() instead, and this batchSize argument has no effect.

bindings - An array of pointers to input and output buffers for the network.

Returns:

True if execution succeeded.

See Also:

ICudaEngine::getMaxBatchSize()

execute

@Cast(value="bool")
 @Deprecated
 @NoException(value=true)
public boolean execute(int batchSize,
                                                                                     @Cast(value="void*const*") @ByPtrPtr
                                                                                     Pointer bindings)

Deprecated.

enqueue

@Cast(value="bool")
 @Deprecated
 @NoException(value=true)
public boolean enqueue(int batchSize,
                                                                                     @Cast(value="void*const*")
                                                                                     PointerPointer bindings,
                                                                                     org.bytedeco.cuda.cudart.CUstream_st stream,
                                                                                     @ByPtrPtr
                                                                                     org.bytedeco.cuda.cudart.CUevent_st inputConsumed)

Deprecated. Deprecated in TensorRT 8.4. Superseded by enqueueV2() if the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag.

\brief Asynchronously execute inference on a batch. This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried using ICudaEngine::getBindingIndex()

Parameters:

batchSize - The batch size. This is at most the max batch size value supplied to the builder when the engine was built. If the network is created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag, please use enqueueV2() instead, and this batchSize argument has no effect.

bindings - An array of pointers to input and output buffers for the network.

stream - A cuda stream on which the inference kernels will be enqueued.

inputConsumed - An optional event which will be signaled when the input buffers can be refilled with new data.

Returns:

True if the kernels were enqueued successfully.

See Also:

ICudaEngine::getMaxBatchSize() \warning Calling enqueue() in from the same IExecutionContext object with different CUDA streams concurrently results in undefined behavior. To perform inference concurrently in multiple streams, use one execution context per stream. \warning This function will trigger layer resource updates if hasImplicitBatchDimension() returns true and batchSize changes between subsequent calls, possibly resulting in performance bottlenecks.

enqueue

@Cast(value="bool")
 @Deprecated
 @NoException(value=true)
public boolean enqueue(int batchSize,
                                                                                     @Cast(value="void*const*") @ByPtrPtr
                                                                                     Pointer bindings,
                                                                                     org.bytedeco.cuda.cudart.CUstream_st stream,
                                                                                     @ByPtrPtr
                                                                                     org.bytedeco.cuda.cudart.CUevent_st inputConsumed)

Deprecated.

setDebugSync

@NoException(value=true)
public void setDebugSync(@Cast(value="bool")
                                                   boolean sync)

\brief Set the debug sync flag. If this flag is set to true, the engine will log the successful execution for each kernel during executeV2(). It has no effect when using enqueueV2().

See Also:

getDebugSync()

getDebugSync

@Cast(value="bool")
 @NoException(value=true)
public boolean getDebugSync()

\brief Get the debug sync flag.

See Also:

setDebugSync()

setProfiler

@NoException(value=true)
public void setProfiler(IProfiler profiler)

\brief Set the profiler.

See Also:

getProfiler()

getProfiler

@NoException(value=true)
public IProfiler getProfiler()

\brief Get the profiler.

See Also:

setProfiler()

getEngine

@Const
 @ByRef
 @NoException(value=true)
public ICudaEngine getEngine()

\brief Get the associated engine.

See Also:

ICudaEngine

destroy

@Deprecated
 @NoException(value=true)
public void destroy()

Deprecated. Use delete instead. Deprecated in TRT 8.0. \warning Calling destroy on a managed pointer will result in a double-free error.

\brief Destroy this object.

setName

@NoException(value=true)
public void setName(String name)

\brief Set the name of the execution context. This method copies the name string.

See Also:

getName()

setName

@NoException(value=true)
public void setName(@Cast(value="const char*")
                                              BytePointer name)

getName

@NoException(value=true)
public String getName()

\brief Return the name of the execution context.

See Also:

setName()

setDeviceMemory

@NoException(value=true)
public void setDeviceMemory(Pointer memory)

\brief Set the device memory for use by this execution context. The memory must be aligned with cuda memory alignment property (using cudaGetDeviceProperties()), and its size must be at least that returned by getDeviceMemorySize(). Setting memory to nullptr is acceptable if getDeviceMemorySize() returns 0. If using enqueueV2() to run the network, the memory is in use from the invocation of enqueueV2() until network execution is complete. If using executeV2(), it is in use until executeV2() returns. Releasing or otherwise using the memory for other purposes during this time will result in undefined behavior.

See Also:

ICudaEngine::createExecutionContextWithoutDeviceMemory()

getStrides

@ByVal
 @Cast(value="nvinfer1::Dims*")
 @NoException(value=true)
public Dims32 getStrides(int bindingIndex)

\brief Return the strides of the buffer for the given binding. The strides are in units of elements, not components or bytes. For example, for TensorFormat::kHWC8, a stride of one spans 8 scalars. Note that strides can be different for different execution contexts with dynamic shapes. If the bindingIndex is invalid or there are dynamic dimensions that have not been set yet, returns Dims with Dims::nbDims = -1.

Parameters:

bindingIndex - The binding index.

See Also:

getBindingComponentsPerElement

setOptimizationProfile

@Cast(value="bool")
 @Deprecated
 @NoException(value=true)
public boolean setOptimizationProfile(int profileIndex)

Deprecated. Superseded by setOptimizationProfileAsync. Deprecated prior to TensorRT 8.0 and will be removed in 9.0.

\brief Select an optimization profile for the current context.

Parameters:

profileIndex - Index of the profile. It must lie between 0 and getEngine().getNbOptimizationProfiles() - 1 The selected profile will be used in subsequent calls to executeV2() or enqueueV2(). When an optimization profile is switched via this API, TensorRT may enqueue GPU memory copy operations required to set up the new profile during the subsequent enqueueV2() operations. To avoid these calls during enqueueV2(), use setOptimizationProfileAsync() instead. If the associated CUDA engine has dynamic inputs, this method must be called at least once with a unique profileIndex before calling execute or enqueue (i.e. the profile index may not be in use by another execution context that has not been destroyed yet). For the first execution context that is created for an engine, setOptimizationProfile(0) is called implicitly. If the associated CUDA engine does not have inputs with dynamic shapes, this method need not be called, in which case the default profile index of 0 will be used (this is particularly the case for all safe engines). setOptimizationProfile() must be called before calling setBindingDimensions() and setInputShapeBinding() for all dynamic input tensors or input shape tensors, which in turn must be called before either executeV2() or enqueueV2(). \warning This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly resulting in performance bottlenecks.

Returns:

true if the call succeeded, else false (e.g. input out of range)

See Also:

IExecutionContext::setOptimizationProfileAsync()

getOptimizationProfile

@NoException(value=true)
public int getOptimizationProfile()

\brief Get the index of the currently selected optimization profile. If the profile index has not been set yet (implicitly to 0 for the first execution context to be created, or explicitly for all subsequent contexts), an invalid value of -1 will be returned and all calls to enqueueV2() or executeV2() will fail until a valid profile index has been set.

setBindingDimensions

@Cast(value="bool")
 @NoException(value=true)
public boolean setBindingDimensions(int bindingIndex,
                                                                                    @ByVal @Cast(value="nvinfer1::Dims*")
                                                                                    Dims32 dimensions)

\brief Set the dynamic dimensions of a binding

Parameters:

bindingIndex - index of an input tensor whose dimensions must be compatible with the network definition (i.e. only the wildcard dimension -1 can be replaced with a new dimension >= 0).

dimensions - specifies the dimensions of the input tensor. It must be in the valid range for the currently selected optimization profile, and the corresponding engine must not be safety-certified. This method requires the engine to be built without an implicit batch dimension. This method will fail unless a valid optimization profile is defined for the current execution context (getOptimizationProfile() must not be -1). For all dynamic non-output bindings (which have at least one wildcard dimension of -1), this method needs to be called before either enqueueV2() or executeV2() may be called. This can be checked using the method allInputDimensionsSpecified(). \warning This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly resulting in performance bottlenecks, if the dimensions are different than the previous set dimensions.

Returns:

false if an error occurs (e.g. bindingIndex is out of range for the currently selected optimization profile or binding dimension is inconsistent with min-max range of the optimization profile), else true. Note that the network can still be invalid for certain combinations of input shapes that lead to invalid output shapes. To confirm the correctness of the network input shapes, check whether the output binding has valid dimensions using getBindingDimensions() on the output bindingIndex.

See Also:

ICudaEngine::getBindingIndex

getBindingDimensions

@ByVal
 @Cast(value="nvinfer1::Dims*")
 @NoException(value=true)
public Dims32 getBindingDimensions(int bindingIndex)

\brief Get the dynamic dimensions of a binding If the engine was built with an implicit batch dimension, same as ICudaEngine::getBindingDimensions. If setBindingDimensions() has been called on this binding (or if there are no dynamic dimensions), all dimensions will be positive. Otherwise, it is necessary to call setBindingDimensions() before enqueueV2() or executeV2() may be called. If the bindingIndex is out of range, an invalid Dims with nbDims == -1 is returned. The same invalid Dims will be returned if the engine was not built with an implicit batch dimension and if the execution context is not currently associated with a valid optimization profile (i.e. if getOptimizationProfile() returns -1). If ICudaEngine::bindingIsInput(bindingIndex) is false, then both allInputDimensionsSpecified() and allInputShapesSpecified() must be true before calling this method.

Returns:

Currently selected binding dimensions For backwards compatibility with earlier versions of TensorRT, a bindingIndex that does not belong to the current profile is corrected as described for ICudaEngine::getProfileDimensions.

See Also:

ICudaEngine::getProfileDimensions

setInputShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean setInputShapeBinding(int bindingIndex,
                                                                                    @Const
                                                                                    IntPointer data)

\brief Set values of input tensor required by shape calculations.

Parameters:

bindingIndex - index of an input tensor for which ICudaEngine::isShapeBinding(bindingIndex) and ICudaEngine::bindingIsInput(bindingIndex) are both true.

data - pointer to values of the input tensor. The number of values should be the product of the dimensions returned by getBindingDimensions(bindingIndex). If ICudaEngine::isShapeBinding(bindingIndex) and ICudaEngine::bindingIsInput(bindingIndex) are both true, this method must be called before enqueueV2() or executeV2() may be called. This method will fail unless a valid optimization profile is defined for the current execution context (getOptimizationProfile() must not be -1). \warning This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly resulting in performance bottlenecks, if the shapes are different than the previous set shapes.

Returns:

false if an error occurs (e.g. bindingIndex is out of range for the currently selected optimization profile or shape data is inconsistent with min-max range of the optimization profile), else true. Note that the network can still be invalid for certain combinations of input shapes that lead to invalid output shapes. To confirm the correctness of the network input shapes, check whether the output binding has valid dimensions using getBindingDimensions() on the output bindingIndex.

setInputShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean setInputShapeBinding(int bindingIndex,
                                                                                    @Const
                                                                                    IntBuffer data)

setInputShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean setInputShapeBinding(int bindingIndex,
                                                                                    @Const
                                                                                    int[] data)

getShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean getShapeBinding(int bindingIndex,
                                                                               IntPointer data)

\brief Get values of an input tensor required for shape calculations or an output tensor produced by shape calculations.

Parameters:

bindingIndex - index of an input or output tensor for which ICudaEngine::isShapeBinding(bindingIndex) is true.

data - pointer to where values will be written. The number of values written is the product of the dimensions returned by getBindingDimensions(bindingIndex). If ICudaEngine::bindingIsInput(bindingIndex) is false, then both allInputDimensionsSpecified() and allInputShapesSpecified() must be true before calling this method. The method will also fail if no valid optimization profile has been set for the current execution context, i.e. if getOptimizationProfile() returns -1.

See Also:

isShapeBinding(bindingIndex)

getShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean getShapeBinding(int bindingIndex,
                                                                               IntBuffer data)

getShapeBinding

@Cast(value="bool")
 @NoException(value=true)
public boolean getShapeBinding(int bindingIndex,
                                                                               int[] data)

allInputDimensionsSpecified

@Cast(value="bool")
 @NoException(value=true)
public boolean allInputDimensionsSpecified()

\brief Whether all dynamic dimensions of input tensors have been specified

Returns:

True if all dynamic dimensions of input tensors have been specified by calling setBindingDimensions(). Trivially true if network has no dynamically shaped input tensors.

See Also:

setBindingDimensions(bindingIndex,dimensions)

allInputShapesSpecified

@Cast(value="bool")
 @NoException(value=true)
public boolean allInputShapesSpecified()

\brief Whether all input shape bindings have been specified

Returns:

True if all input shape bindings have been specified by setInputShapeBinding(). Trivially true if network has no input shape bindings.

See Also:

isShapeBinding(bindingIndex)

setErrorRecorder

@NoException(value=true)
public void setErrorRecorder(IErrorRecorder recorder)

See Also:

/**

getErrorRecorder

@NoException(value=true)
public IErrorRecorder getErrorRecorder()

\brief Get the ErrorRecorder assigned to this interface. Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not been set.

Returns:

A pointer to the IErrorRecorder object that has been registered.

See Also:

setErrorRecorder()

executeV2

@Cast(value="bool")
 @NoException(value=true)
public boolean executeV2(@Cast(value="void*const*")
                                                                         PointerPointer bindings)

\brief Synchronously execute inference a network. This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried using ICudaEngine::getBindingIndex(). This method only works for execution contexts built with full dimension networks.

Parameters:

bindings - An array of pointers to input and output buffers for the network.

Returns:

True if execution succeeded.

See Also:

ICudaEngine::getMaxBatchSize()

executeV2

@Cast(value="bool")
 @NoException(value=true)
public boolean executeV2(@Cast(value="void*const*") @ByPtrPtr
                                                                         Pointer bindings)

enqueueV2

@Cast(value="bool")
 @NoException(value=true)
public boolean enqueueV2(@Cast(value="void*const*")
                                                                         PointerPointer bindings,
                                                                         org.bytedeco.cuda.cudart.CUstream_st stream,
                                                                         @ByPtrPtr
                                                                         org.bytedeco.cuda.cudart.CUevent_st inputConsumed)

\brief Asynchronously execute inference. This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried using ICudaEngine::getBindingIndex(). This method only works for execution contexts built with full dimension networks.

Parameters:

bindings - An array of pointers to input and output buffers for the network.

stream - A cuda stream on which the inference kernels will be enqueued

inputConsumed - An optional event which will be signaled when the input buffers can be refilled with new data

Returns:

True if the kernels were enqueued successfully.

See Also:

ICudaEngine::getMaxBatchSize() \note Calling enqueueV2() with a stream in CUDA graph capture mode has a known issue. If dynamic shapes are used, the first enqueueV2() call after a setInputShapeBinding() call will cause failure in stream capture due to resource allocation. Please call enqueueV2() once before capturing the graph. \warning Calling enqueueV2() in from the same IExecutionContext object with different CUDA streams concurrently results in undefined behavior. To perform inference concurrently in multiple streams, use one execution context per stream.

enqueueV2

@Cast(value="bool")
 @NoException(value=true)
public boolean enqueueV2(@Cast(value="void*const*") @ByPtrPtr
                                                                         Pointer bindings,
                                                                         org.bytedeco.cuda.cudart.CUstream_st stream,
                                                                         @ByPtrPtr
                                                                         org.bytedeco.cuda.cudart.CUevent_st inputConsumed)

setOptimizationProfileAsync

@Cast(value="bool")
 @NoException(value=true)
public boolean setOptimizationProfileAsync(int profileIndex,
                                                                                           org.bytedeco.cuda.cudart.CUstream_st stream)

\brief Select an optimization profile for the current context with async semantics.

Parameters:

profileIndex - Index of the profile. The value must lie between 0 and getEngine().getNbOptimizationProfiles() - 1

stream - A cuda stream on which the cudaMemcpyAsyncs may be enqueued When an optimization profile is switched via this API, TensorRT may require that data is copied via cudaMemcpyAsync. It is the application’s responsibility to guarantee that synchronization between the profile sync stream and the enqueue stream occurs. The selected profile will be used in subsequent calls to executeV2() or enqueueV2(). If the associated CUDA engine has inputs with dynamic shapes, the optimization profile must be set with a unique profileIndex before calling execute or enqueue. For the first execution context that is created for an engine, setOptimizationProfile(0) is called implicitly. If the associated CUDA engine does not have inputs with dynamic shapes, this method need not be called, in which case the default profile index of 0 will be used. setOptimizationProfileAsync() must be called before calling setBindingDimensions() and setInputShapeBinding() for all dynamic input tensors or input shape tensors, which in turn must be called before either executeV2() or enqueueV2(). \warning This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly resulting in performance bottlenecks. \warning Not synchronizing the stream used at enqueue with the stream used to set optimization profile asynchronously using this API will result in undefined behavior.

Returns:

true if the call succeeded, else false (e.g. input out of range)

See Also:

ICudaEngine::getNbOptimizationProfiles(), IExecutionContext::setOptimizationProfile()

setEnqueueEmitsProfile

@NoException(value=true)
public void setEnqueueEmitsProfile(@Cast(value="bool")
                                                             boolean enqueueEmitsProfile)

\brief Set whether enqueue emits layer timing to the profiler If set to true (default), enqueue is synchronous and does layer timing profiling implicitly if there is a profiler attached. If set to false, enqueue will be asynchronous if there is a profiler attached. An extra method reportToProfiler() needs to be called to obtain the profiling data and report to the profiler attached.

See Also:

IExecutionContext::getEnqueueEmitsProfile(), IExecutionContext::reportToProfiler()

getEnqueueEmitsProfile

@Cast(value="bool")
 @NoException(value=true)
public boolean getEnqueueEmitsProfile()

\brief Get the enqueueEmitsProfile state.

Returns:

The enqueueEmitsProfile state.

See Also:

IExecutionContext::setEnqueueEmitsProfile()

reportToProfiler

@Cast(value="bool")
 @NoException(value=true)
public boolean reportToProfiler()

\brief Calculate layer timing info for the current optimization profile in IExecutionContext and update the profiler after one iteration of inference launch. If IExecutionContext::getEnqueueEmitsProfile() returns true, the enqueue function will calculate layer timing implicitly if a profiler is provided. This function returns true and does nothing. If IExecutionContext::getEnqueueEmitsProfile() returns false, the enqueue function will record the CUDA event timers if a profiler is provided. But it will not perform the layer timing calculation. IExecutionContext::reportToProfiler() needs to be called explicitly to calculate layer timing for the previous inference launch. In the CUDA graph launch scenario, it will record the same set of CUDA events as in regular enqueue functions if the graph is captured from an IExecutionContext with profiler enabled. This function needs to be called after graph launch to report the layer timing info to the profiler. \warning profiling CUDA graphs is only available from CUDA 11.1 onwards. \warning reportToProfiler uses the stream of the previous enqueue call, so the stream must be live otherwise behavior is undefined.

Returns:

true if the call succeeded, else false (e.g. profiler not provided, in CUDA graph capture mode, etc.)

See Also:

IExecutionContext::setEnqueueEmitsProfile(), IExecutionContext::getEnqueueEmitsProfile()