Skip to main content

DLA

NVDLA​

Our team is using the YOLO Darknet framework for model training, testing and developing. After we train the model in the Darknet framework, we convert the darknet .weights model file to TensorRT and we deploy it on the Xavier platform using NVDLA. By Using FP16 half-precision (TODO) together with NVDLA we can get more than 40x speedup!

DLA Supported Layers​

Lists of the layers supported by DLA along with the constraints associated with each layer:

From TensorRT doc:

  • DLA is designed to do full hardware acceleration of convolutional neural networks.
  • DLA supports various layers such as convolution, deconvolution, fully connected, activation, pooling, batch normalization, and so on.
  • DLA does not support Explicit Quantization.
  • DLA is useful for offloading CNN processing from the iGPU, and is significantly more power-efficient for these workloads.

From developer.nvidia.com:

  • For example, DLA software performs fusions to reduce the number of passes to and from system memory.

Conversion​

First, read Wiki: TensorRT.

From TensorRT doc:

  • When building a model for DLA, the TensorRT builder parses the network and calls the DLA compiler to compile the network into a DLA loadable.

From developer.nvidia.com:

  • The TensorRT builder provides the compile time and build time interface that invokes the DLA compiler.
  • Once the plan file is generated, the TRT runtime calls into the DLA runtime stack to execute the workload on the DLA cores.
  • TensorRT also makes it easy to port from GPU to DLA by specifying only a few additional flags.

ONNX Conversion​

Manually Constructing a Network using the TensorRT API​

YOLO v4 on DLA​

If you train a standard YOLO v4 using the "darknet" framework (see github: AlexeyAB) or any other framework many network layers will fall back to GPU because they are not supported on the DLAs (see this comment). In this case the data transfer between processors causes some performance drop.

  • list of layers on GPU vs layers on DLA: [yolov4-dla-gpu-fallback-list](/trafficsigns/YOLOv4 DLA GPU Fallback List)
  • performance comparison GPU only vs GPU + DLA fallback: Nvidia support comment

Solution 1: This means you will have to re-design the YOLO v4 model so that all layers of the model can run on NVDLA (see this comment).

Solution 2: (TODO) Nvidia's TAO Toolkit provides some workarounds for this issue, e.g. the force_relu configuration parameter replaces all activations by ReLU activations:

YOLO v4 vs YOLO v3​

  • in the graph below you can see that
    • yolov4 can achieve an average precision between 38 and 44, and frames per second between 60 and 120, whereas
    • yolov3 achieves an average precision between 31 and 33 and frames per second between 71 and 120.

yoloV4vsV3.jpg

Troubleshooting​

PyTorch DLA Support​