Skip to main content

TensorRT

Conversion​

From TensorRT Doc

There are three main options for converting a model with TensorRT:

  • using TF-TRT
  • automatic ONNX conversion from .onnx files
  • manually constructing a network using the TensorRT API (either in C++ or Python)

ONNX Conversion​

A more performant option (than TF-TRT) for automatic model conversion and deployment is to convert using ONNX. ONNX is a framework agnostic option that works with models in TensorFlow, PyTorch, and more.

  • TensorRT supports automatic conversion from ONNX files using either
    • the TensorRT API, or
    • trtexec (see the example in TensorRT Doc).
  • ONNX conversion is all-or-nothing, meaning all operations in your model must be supported by TensorRT (or you must provide custom plug-ins for unsupported operations).
  • The result of ONNX conversion is a singular TensorRT engine that allows less overhead than using TF-TRT.

Manually Constructing a Network using the TensorRT API​

For the most performance and customizability possible, you can also construct TensorRT engines manually using the TensorRT network definition API. See TensorRT Doc for further information.

  • This essentially involves building an identical network to your target model in TensorRT operation by operation, using only TensorRT operations.
  • After a TensorRT network is created, you will then export just the weights of your model from the framework and load them into your TensorRT network

Troubleshooting​

Errors​

Error 1​

ERROR: [TRT]: ModelImporter.cpp:726: ERROR: ModelImporter.cpp:527 In function importModel:        
[4] Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() && "This version of the ONNX parser only supports TensorRT INetworkDefinitions with an explicit batch dimension. P
lease ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."                                                                                                 
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:315 Failed to parse onnx file               
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:966 failed to build network since parsing model errors.
  • problem: Explicit batch is required when you are dealing with Dynamic shapes, otherwise network will be created using implicit batch dimension.
  • fix: Use force-implicit-batch-dim=0.
    • force-implicit-batch-dim=1 means: When a network supports both implicit batch dimension and full dimension, force the implicit batch dimension mode. doc

Note: DeepStream 6.0 uses explicit batch dimension for caffemodels. Some caffemodels use TensorRT plugins/layers which have not been updated for explicit batch dimensions. Add force-implicit-batch-dim=1 in the nvinfer config file for such models to build the models using implicit batch dimension networks. source

Warnings​

WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

and

WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: Output type must be INT32 for shape outputs

both indicate that your network model file was build using INT64 weights. TensorRT doesn't support INT64. Instead it converts INT64 to INT32 if possible. The warning shows up every time it does the clipping (see issue).