NVIDIA’s DeepStream SDK is a multi-sensor AI-based streaming analytics toolkit for video, image, and audio understanding. DeepStream is an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions that transform real-time pixels and sensor data into actionable insights. DeepStream SDK features hardware-accelerated building blocks, called plugins, which bring deep neural networks and other complex processing tasks into a processing pipeline. The DeepStream SDK allows you to focus on building optimized Vision AI applications without having to design complete solutions from scratch.
The DeepStream SDK uses AI to perceive pixels and generate metadata while offering integration from the edge-to-the-cloud. The DeepStream SDK can be used to build applications across various use cases including retail analytics, patient monitoring in healthcare facilities, parking management, optical inspection (AOI), supply chain management (logistics), operations and manufacturing.
NVIDIA NGC offers collections that easily group relevant information together. We have created the NVIDIA DeepStream SDK collection, so it is easy to find all information in a single place.
Please note that we have consolidated the DeepStream containers as follows:
Type |
Architecture |
Container Name |
Target |
|---|---|---|---|
Triton |
Multi-Arch x86 + Jetson |
deepstream:6.4-triton-multiarch |
Deployment |
DeepStream Samples |
Multi-Arch x86 + Jetson |
deepstream:6.4-samples-multiarch |
Deployment |
Development and Graph Composer |
x86 |
deepstream:6.4-gc-triton-devel |
x86 development. Includes Graph Composer GUI |
NOTE: All DeepStream dockerfiles are available on GitHub for easy customization.
For a full list of new features and changes, and known limitations please refer to the DeepStream 6.4 Release Notes.
Key New Features and Enhancements available on DeepStream 6.4:
Category |
Details |
New Features |
• New Single-View 3D tracking capability available on NvTracker • NvDCF tracker now supported with PVA backend for Jetson devices • Support for AV1 encoder • Two new plugins for image decoding and encoding (CUDA accelerated) • Video template plugin now supports audio-in / video-out • MQTT now support TLS security and keep-alive functionality • Support for GStreamer 1.20.3 • Support for Ubuntu 22.04 and JetPack 6.0 Developer Preview |
Enhancements |
• Further improvements to REST-APIs that control DeepStream pipeline on the fly • Improved Re-ID accuracy on NvTracker • Improved Video4Linux Encoder Plugin • Multiple Performance optimizations |
Key New Features and Enhancements available on GXF and Graph Composer 3.0 and 3.1:
Category |
Details |
New Features |
• New Python APIs in addition to existing C/C++ APIs • Event triggered data-out support • New GXF distributed scheduling support • Support for Ubuntu 22.04 and JetPack 6.0 Developer Preview |
Enhancements |
• Registry and Graph Composer are now independent entities • GXF scheduler optimizations • GXF now supports Bayer, RAW16 and 3D RGBD formats • GXF error reporting and logging improvements • Graph Composer sub-graph support improvements • Graph Composer applications add support for multiple clock sources |
Item |
Documentation |
|---|---|
Documentation |
DeepStream SDK Documentation |
Getting Started |
Quick Start Guide |
Developing with C/C++ |
DeepStream Reference Application (deepstream-app) |
Developing with Python |
Python Application GitHub Repository |
Developing with Graph Composer |
Graph Composer Reference Apps |
DeepStream and TAO Toolkit Integration |
TAO Supported Models |
Deep Dives with DeepStream Ninjas |
DeepStream Multi-Object Trackers |
Additional Examples |
|
Learn More |
New To DeepStream? Start here |
Jetson Series |
Modules |
JetPack |
Latest DeepStream Supported |
|---|---|---|---|
Orin |
Jetson AGX Orin, Jetson Orin NX, Jetson Orin Nano |
(Developer Preview) |
6.4 |
Xavier |
Jetson AGX Xavier, Jetson Xavier NX |
6.3 (legacy) |
|
TX2 |
Jetson TX2 NX Jetson TX2 |
6.0 (legacy) |
|
Nano |
Jetson Nano |
6.0 (legacy) |
Enterprise GPU Architecture |
GPUs |
|---|---|
Ada Lovelace |
L4, L40 |
Ampere |
A2, A10, A16, A30, A40, A100, RTX A6000 |
Hopper |
H100 |
Turing |
T4 |
Volta |
V100 |
Software Dependencies |
x86 |
Jetson (via JetPack) |
|---|---|---|
Operating System |
Ubuntu LTS 22.04 |
Ubuntu LTS 22.04 |
GStreamer |
1.20.3 |
1.20.3 |
Rivermax |
1.40 |
1.40 |
DLFW (Triton) |
23.08 |
23.11 |
TensorRT |
8.6.1 |
8.6.2 |
CUDA |
12.2 |
12.2 |
cuDNN |
8.9.4 |
8.9 |
GPU Driver (RM) |
535 (535 TRD4) |
N/A |
Note:
If you are looking for older versions of DeepStream, please refer to the x86 or Jetson archive. Archived documentation is available here.
DeepStream Support is available via:
Method |
Available to |
|---|---|
Forums |
|
Direct Support |
NVIDIA AI Enterprise License holders |
Known Limitations
· AMQP support is not included inside the container. Please refer AMQP Protocol Adapter Section within the DeepStream 6.4 Plugin Guide Section for instructions on how to install necessary dependencies for enabling AMQP if required.
· There are known bugs and limitations in the SDK. To learn more about those, refer to the release notes.
The following Security Vulnerabilities (CVEs) impact DeepStream as it uses OpenSSL headers for compilation (only). The CVEs are part of the OpenSSL source code:
CVE |
Description |
|---|---|
OpenSSL contains a stack buffer overflow vulnerability in its handling of punycode decoding with X.509 certificate verification. A remote attacker could leverage this to crash the running process by overwriting arbitrary length of stack memory with ASCII `.` characters. The vulnerability requires a certificate authority to have signed a malicious certificate, or the application continues with verification without a trusted certificate issuer. The issue affects both TLS clients and servers. This vulnerability is still under analysis for technical details. |
|
OpenSSL contains a stack buffer overflow vulnerability in its handling of punycode decoding with X.509 certificate verification. A remote attacker could leverage this to crash the running process or potentially divert the execution flow by overwriting 4 bytes of stack memory with arbitrary data. The vulnerability requires a certificate authority to have signed a malicious certificate, or the application continues with verification without a trusted certificate issuer. The issue affects both TLS clients and servers |
The following licenses apply to the DeepStream SDK assets:
Asset |
Applicable EULA |
Notes |
|---|---|---|
SDK |
A copy of the license is available on the following folder of the SDK: |
|
Containers |
License grants redistribution rights allowing developers to build applications on top of the DeepStream containers |
|
Development Containers |
A development-only license. Does not allow redistribution of the container |
|
TAO Models |
All TAO pre-trained models included in the DeepStream SDK are covered by the NVIDIA AI Product License.
|
NOTE: By pulling, downloading, or using the DeepStream SDK, you accept the terms and conditions of the EULA licenses listed above:
Ethical AI
NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.