Skip to main content

Deepstream

Note: If you want to use Deepstream via ssh, see section "Usage via ssh".

Why do we need it?​

  • Nvidia's Deepstream SDK is the Nvidia recommended way of implementing a detection pipeline on Jetson AGX
    • since Nvidia is also the company which designs the AGX, the Deepstream approach is very likely the most efficient and robust way of implementing the detection our Team can get
  • from this comment:
    • "a significant portion of time has been spent on image preprocessing, postprocessing and display"
    • "if you use C++ to parallelize/pipeline preprocessing, inferencing and postprocessing, you could speed up the whole video pipeline quite a bit [...]"
      • "One good way to achieve that is to use NVIDIA DeepStream SDK"

What is Deepstream?​

Deepstream "automates" setting up the model pipeline properly and efficiently using so called config files. It "automates" in the sense that it aims to minimize the coding effort to set up a Deep Learning model pipeline.

The Deepstream documentation often leaves open some questions. However, in general, it's a pretty good starting point.

There is a huge amount of configuration options in Deepstream 6 and at the time of writing the framework is still growing. Please, read the Deepstream documentation for usage instructions.

You can find some of the common errors and their fixes in the respective sections of the Team Galaxis traffic sign documentation.

Basics​

commanddescription
deepstream-app -hShow help options
deepstream-app --help-allShow all help options
deepstream-app --help-gstShow GStreamer Options
deepstream-app -vPrint DeepStreamSDK version
deepstream-app --version-allPrint DeepStreamSDK and dependencies version (e.g. CUDA version, TensorRT version, etc.)

Increase Inference Throughput​

  • see tips in Deepstream doc
  • Set the interval=0 parameter under [property] in the inference config file to a higher value. E.g. interval=1 means that Deepstream will infer every other frame (or interval=2 for every third frame) and use a tracker to predict the location of the object. This can effectively double the throughput!

Example Apps​

YOLO (sources/objectDetector_Yolo)​

Custom Library nvdsinfer_custom_impl_Yolo​

filedescription
nvdsinfer_custom_impl_Yolo/nvdsinfer_yolo_engine.cppImplementation of 'NvDsInferCreateModelParser'/IModelParser for nvdsinfer to parse custom models. Alternatively, also contains implementation of 'NvDsInferYoloCudaEngineGet' for nvdsinfer to directly create cuda engine. To use the 'NvDsInferYoloCudaEngineGet' interface, enable the macro USE_CUDA_ENGINE_GET_API in nvdsinfer_yolo_engine.cpp
nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cppOutput layer parsing function for detected objects for the Yolo model.
nvdsinfer_custom_impl_Yolo/yoloPlugins.hDeclaration of YoloLayerV3 and YoloLayerV3PluginCreator.
nvdsinfer_custom_impl_Yolo/yoloPlugins.cppImplementation of YoloLayerV3 and YoloLayerV3PluginCreator.
nvdsinfer_custom_impl_Yolo/kernels.cuImplementation of cuda kernels for
nvdsinfer_custom_impl_Yolo/trt_utils.hUtilities to setup tensorRT networks
nvdsinfer_custom_impl_Yolo/trt_utils.cppImplementation of Utilities to setup tensorRT networks
nvdsinfer_custom_impl_Yolo/yolo.hInterface to create Yolo Cuda-Engine
nvdsinfer_custom_impl_Yolo/yolo.cppImplementation to create Yolo Cuda-Engine

Compile nvdsinfer_custom_impl_Yolo​

  • Important: Do not copy nvdsinfer_custom_impl_Yolo/ to some other directory! Otherwise make will fail.
  • on AGX:
    • sudo chmod a+rwx /opt/nvidia/deepstream/deepstream-6.0/sources/objectDetector_Yolo/
    • sudo chmod a+rwx /opt/nvidia/deepstream/deepstream-6.0/sources/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/
    • export CUDA_VER=10.2
    • make -C nvdsinfer_custom_impl_Yolo/ (first set NUM_CLASSES_YOLO in nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp)
      • Important: Do not copy nvdsinfer_custom_impl_Yolo/ to some other directory! Otherwise make will fail.

yolov2​

  • to run on DLA use network-mode=2 (i.e. FP16 mode) in configuration group [property]
    • Nvidia does not provide an INT8 calibration table file for YOLOv2 in Deepstream 6.0.1, so in order to use INT8 mode on DLA a calibration table must be generated first (see TensorRT doc)

yolov4 and yolov7​

Apps from deepstream_python_apps (sources/apps/sample_apps)​

Troubleshooting​

Error 1:

Makefile:25: *** "CUDA_VER is not set".  Stop.
  • instead of sudo make, run make

INT8 Mode​

  • "Calibration must be performed using images representative of those which will be used at runtime." (from TensorRT doc)
    • This means that the provided INT8 calibration table in Deepstream 6.0.1 does not work for our trained traffic sign detection networks! We have to generate our own INT8 calibration tables using images of our real test track! (Better: Don't use INT8 mode.)

Config Files​

DLA​

# check, if a DLA is active:
cat /sys/devices/platform/host1x/15880000.nvdla0/power/runtime_status   #DLA0
cat /sys/devices/platform/host1x/158c0000.nvdla1/power/runtime_status   #DLA1

Usage of deepstream-app via ssh​

Deepstream does not support x11 forwarding via ssh, so you must save all outputs to files! This can be done in the following way:

  1. auf host: xhost +
  2. auf AGX: export DISPLAY=:0
    • oder export DISPLAY=:1 (je nach dem, was klappt)
    • liegt daran, dass x11 forwarding nicht von Deepstream supported wird (May 2022) [komischerweise obwohl Gstreamer x11 forwarding supported]
      • d.h. man muss die DISPLAY Variable der AGX bash shell auf den lokalen AGX Monitor setzen (i.e. export DISPLAY=:0)
  3. Then in the Deepstream config file (e.g. deepstream_app_config_yoloV3.txt) in the [sink0] configuration group set the output to a file (see configuration groups). Example:
[sink0]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
bitrate=2000000
output-file=traffic_sign_detections.mp4
source-id=0

Recording mp4 files​

It may be useful to record .mp4 files for later use, while you have enough light. You can record an .mp4 file from command line by running

gst-launch-1.0 v4l2src device="/dev/video2" num-buffers=3000 ! 'video/x-raw, framerate=60/1' ! videoconvert ! x264enc ! mp4mux ! filesink location=traffic_signs_3000frames.mp4

which records 3000 frames at 60 fps, i.e. the output .mp4 duration will be 50 seconds.

Important: The above command will output an .mp4 file in yuv422p pixel format. However, Deepstream config files will only take yuv420p formatted input files. Run ffmpeg -i file.mp4 -vf format=yuv420p file_yuv420p.mp4 to convert files from yuv422p to yuv420p.

  • actually, you can record yuv420p (aka I420 in the Gstreamer framework) directly, but I have not yet figured out the right gst-launch-1.0 pipeline for this ...
  • if you need to change framerates, you can do this by running ffmpeg -i <input> -filter:v fps=30 <output>

Note: If you stream .mp4 files via ssh, the output will lag significantly. So, always download .mp4 files using scp and play them locally on your computer!

Note: You can get some metadata about an .mp4 file (i.e. duration, fps, encoding, ... etc.) by running e.g. ffmpeg -i traffic_signs_1000frames.mp4.

Docker​

  • xhost + vor dem docker Start!
  • im docker start script: mounte die richtige Kamera (z.B. via docker run --device=/dev/video4)

Troubleshooting​

Check Camera Device​

  • l /dev/video*
  • v4l2-ctl --list-devices
  • always use the right camera device number in your apps!

Display the webcam feed​

You can display the webcam feed of device=/dev/video2 from command line by running

# using mpv
mpv /dev/video2

# using vlc
vlc v4l2:///dev/video2

# using Gstreamer
gst-launch-1.0 v4l2src device=/dev/video2 name=cam_src ! videoconvert ! videoscale ! video/x-raw,format=RGB ! queue ! videoconvert ! ximagesink name=img_origin

Clear Cache​

  • Always do this first!
  • rm .cache/gstreamer-1.0/registry.aarch64.bin
    • Note: .cache/ is in folder /root/

Errors​

Error 1​

# no output at all
  • If gst-inspect-1.0 and gst-launch-1.0 both do not give any output, at all, reboot Ubuntu.

Error 2​

[NvMediaParserParse:357] Video parser parse failed: 0Stream format not found, dropping the frame
  • Problem: The input video source's format is not supported by Deepstream.
  • Fix: Convert the video file to one of Deepstream's supported formats first.
    • e.g. yuv422 (the Realsense default format) is not supported in Deepstream pipelines

Error 3​

** INFO: <bus_callback:180>: Pipeline running

ERROR from src_elem: Internal data stream error.
Debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstV4l2Src:src_elem:
streaming stopped, reason not-negotiated (-4)
** INFO: <bus_callback:160>: incorrect camera parameters provided, please provide supported resolution and frame rate

** INFO: <bus_callback:217>: Received EOS. Exiting ...

Quitting
App run failed
  • problem: here the line ** INFO: <bus_callback:160>: incorrect camera parameters provided, please provide supported resolution and frame rate indicates that the camera resolution settings specified in deepstream_app_config_yoloV4.txt under [source0] are probably not supported by the camera.
  • solution:
    • run v4l2-ctl -d /dev/video2 --list-formats-ext to check supported formats and resolutions
    • choose one of the supported formats and in deepstream_app_config_yoloV4 change 
      [source0]
      camera-width=424
      camera-height=240
      camera-fps-n=60

Error 4​

ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/yolov3_b1_gpu0_int8.engine open error
  • Possible Problem 1: a wrong .engine file name was specified
    • solution:
      • In config_infer_primary_yoloV3.txt change model-engine-file=yolov3_b1_gpu0_int8.engine to model-engine-file=model_b1_gpu0_int8.engine.
      • The right name of the .engine file can be determined by generating a new .engine file by running deepstream-app -c deepstream_app_config_yoloV3.txt and then checking which .engine file was created.
      • Deepstream usually generates model_b1_gpu0_int8.engine for YoloV3. Thus, yolov3_b1_gpu0_int8.engine won't work.
  • Possible Problem 2: the specified .engine file does not exist
    • solution: create an .engine file (Deepstream does this automatically)
  • Note: This ERROR is usually not a problem. It indicates that the .engine file is missing. Deepstream automatically generates a new .engine file after displaying this ERROR.

Error 5​

No EGL Display
nvbufsurftransform: could not get EGL display
  • solution: wenn login über ssh: setze export DISPLAY=:0 oder export DISPLAY=:1 auf dem AGX (s. section "Using Deepstream via ssh")

Error 6​

WARNING: Num classes mismatch. Configured:80, detected by network: 4
  • solution:
    1. in nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp check, if NUM_CLASSES_YOLO = 4 is set correctly
    2. In deepstream_app_config_yoloV3.txt and config_infer_primary_yoloV3.txt check labelfile-path=. Setting it in deepstream_app_config_yoloV3.txt will override the value in config_infer_primary_yoloV3.txt.
    • Therefore, make sure to set it only in one of the two files. Deepstream-app will crash, if the values in both files conflict!

Error 7​

ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:31 Could not open lib: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_yolov4, error string: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_yolov4: cannot open shared object file: No such file or directory
0:00:15.658106044  9401      0x3a0bd30 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary-inference1> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1248> [UID = 1]: Could not open custom lib: (null)
0:00:15.658129358  9401      0x3a0bd30 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary-inference1> error: Failed to create NvDsInferContext instance
0:00:15.658135900  9401      0x3a0bd30 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary-inference1> error: Config file path: /root/docker_volume/realsense_gstream_ds_ws_scp/src/ros2_deepstream/config_files/dstest2_pgie_config.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
[NvMultiObjectTracker] De-initialized
Error: gst-resource-error-quark: Failed to create NvDsInferContext instance (1): gstnvinfer.cpp(841): gst_nvinfer_start (): /GstPipeline:pipeline0/GstNvInfer:primary-inference1:
Config file path: /root/docker_volume/realsense_gstream_ds_ws_scp/src/ros2_deepstream/config_files/dstest2_pgie_config.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
  • problem: the folder deepstream_yolov4/ is not in /opt/nvidia/deepstream/deepstream-6.0/sources/
  • solution:

Error 8​

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
 dlopen error: /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initilaize low level lib.
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
 dlopen error: /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initilaize low level lib.
  • output: the camera window opens, but it is completely black and shows nothing
  • problem: the multi-object tracker libnvds_mot_klt.so could not be found
  • solution:
    • in Deepstream 6.0.1 libnvds_mot_klt.so was renamed to libnvds_nvmultiobjecttracker.so
    • for Deepstream 6.0.1: in path/to/config_files/dstest2_tracker_config.txt set 34 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so instead of 34 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so

Error 9​

Error: gst-stream-error-quark: memory type configured and i/p buffer mismatch ip_surf 0 muxer 3 (1): gstnvstreammux.c(619): gst_nvstreammux_chain (): /GstPipeline:pipeline0/GstNvStreamMux:Stream-muxer
  • problem: the specified streammux resolution is wrong
  • fix: in single_stream_pkg/single_stream_pkg/single_stream_class.py set
324 #        streammux.set_property('width', 1920)
325 #        streammux.set_property('height', 1080)
326         streammux.set_property('width', 640)
327         streammux.set_property('height', 540)
  • important: resolution must be set to exactly one of the following values ! Other resolutions will give the above error.
    • on phth's machine:
      • width: 640 x height: 540
      • 424 x 240
    • on AGX:
      • width: 640 x height: 480

Error 10​

NvDsInferCudaEngineGetFromTltModel: Failed to open TLT encoded model file /root/docker_volume/ds_ws/src/ros2_deepstream/config_files/final_model.etlt
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:724 Failed to create network using custom network creation function
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:789 Failed to get cuda engine from custom library API
0:00:15.477449959 11681      0x23cd730 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<secondary1-nvinference-engine> NvDsInferContext[UID 22]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 22]: build engine file failed
terminate called after throwing an instance of 'nvinfer1::InternalError'
  what():  Assertion val > 0 failed.
  • problem: there is no model file final_model.etlt in the directory src/ros2_deepstream/config_files/
  • fix: download the model file

Error 11​

ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1484 Deserialize engine failed because file path: /root/docker_volume/ds_ws/src/ros2_deepstream/config_files/final_model.etlt_b32_gpu0_fp16.engine open error
  • problem: there is no engine file final_model.etlt_b32_gpu0_fp16.engine in the directory src/ros2_deepstream/config_files/
  • fix: download the engine file

Error 12​

ERROR: [TRT]: ModelImporter.cpp:726: ERROR: ModelImporter.cpp:527 In function importModel:        
[4] Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() && "This version of the ONNX parser only supports TensorRT INetworkDefinitions with an explicit batch dimension. P
lease ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."                                                                                                 
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:315 Failed to parse onnx file               
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:966 failed to build network since parsing model errors.
  • problem: Explicit batch is required when you are dealing with Dynamic shapes, otherwise network will be created using implicit batch dimension.
  • fix: Use force-implicit-batch-dim=0.
    • force-implicit-batch-dim=1 means: When a network supports both implicit batch dimension and full dimension, force the implicit batch dimension mode. doc

Note: DeepStream 6.0 uses explicit batch dimension for caffemodels. Some caffemodels use TensorRT plugins/layers which have not been updated for explicit batch dimensions. Add force-implicit-batch-dim=1 in the nvinfer config file for such models to build the models using implicit batch dimension networks. source