Blueprint for the Video Search and Summarization Agent
Introduction
Advances in AI video understanding and interaction have the potential to revolutionize how we access, analyze, and interact with video content in various domains. These AI models are capable of:
- Video captioning-Generating text descriptions or summary of videos.
- Question answering-Answering questions about a video's content.
- Video retrieval-Finding specific videos (highlights) based on text queries.
- Action recognition-Identifying actions happening in the video.
The current release of Video Search and Summarization Agent (VSS) demonstrates Video Summarization, Q&A and alerts with accelerated performance on NVIDIA hardware.
Features
VSS supports video upload, live stream support, summarizing on video files, image files and live streams with various configuration options. Features:
- Faster/Quick Long video processing
- Image / Multi-Image support
- Live Stream (RTSP) support
- Supported file formats: mp4, mkv, jpg, png
- Supported codecs: h264/h265 video and Opus/Vorbis audio
- Summarization for videos, images, and live streams
- Q&A for files, images and live-streams
- Event & Alerts
- vLLM acceleration for Cosmos-Reason1 and Cosmos-Reason2
- Multi Node multiple GPU support
- Context aware RAG support for enhanced accuracy & Q&A
- Graph RAG
- Vector RAG
- Support for GPT-4o as the VLM and LLM
- Use OpenAI Compatible hosted VLM models
- Drop-in support for custom VLMs
- Guardrails support
- OpenAI Compatible REST API
- Multi-stream support
- Use of Riva ASR based audio transcription in summarization, QnA, and alerts
- CV pipeline to generate CV metadata and Set of Marks (SOM) Prompting for videos and live streams
Architecture
User Guide
User Guide is available at: https://docs.nvidia.com/vss/index.html
NOTE: This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Deployment Note
The Video Search and Summarization Blueprint is shared as reference and is provided "as is". The security in the production environment is the responsibility of the end users deploying it. When deploying in a production environment, please have security experts review any potential risks and threats; define the trust boundaries, implement logging and monitoring capabilities, secure the communication channels, integrate AuthN & AuthZ with appropriate access controls, keep the deployment up to date, ensure the containers/source code are secure and free of known vulnerabilities. The end users are also responsible for ensuring integrity and authenticity of the models and containers.
Known CVEs
VSS Engine 2.4.1 Container has the following known CVEs:
| CVE | Description |
|---|---|
| GHSA-58pv-8j8x-9vj2 | This impacts jaraco.context < 6.1.0 python package. This does not affect VSS since it does not install user provided python packages. |
| CVE-2025-69223 | This impacts aiohttp < 3.13.3 python package. This does not affect VSS since it gets included as a private package inside ray and ray is not used by VSS. |
| GHSA-f83h-ghpp-7wcc | This impacts pdfminer.six < 20251230 python package. This does not affect VSS since it does not implement PDF parsing. |
| CVE-2025-68973 | This impacts gnupg < 2.4.8. This does not affect VSS since it does not implement GPG encryption. |
| GHSA-mcmc-2m55-j8jj GHSA-mrw7-hf4f-83pf CVE-2025-62372 | This impacts vLLM < 0.11.1 python package. This does not affect VSS since it does not support user provided embeddings. |
| CVE-2026-21441 | This affects urllib3 < 2.6.3 python package. This does not affect VSS since it does not access user provided URLs at runtime. |
| CVE-2025-3887 | This impacts GStreamer H.265 codec parser, Malicious malformed streams can cause stack overflow in H.265 codec parser causing the application to crash. Users must take care that malicious H.265 streams are not added to VSS. This can be remedied by building and installing the GStreamer1.24.2 codec parser library after applying the patch mentioned in https://gstreamer.freedesktop.org/security/sa-2025-0001.html. |
| GHSA-rcfx-77hg-w2wv | This impacts fastmcp < 2.14.0 python package. This does not affect VSS since it already used an updated version of MCP SDK. |
VSS Engine 2.4.0 Container (previous version) has the following known CVEs:
| CVE | Description |
|---|---|
| CVE-2024-8966 | This impacts gradio <= 5.22.0 python package, This impacts the file upload functionality of Gradio UI where an attacker can cause Denial-of-Service (DoS) attack by appending a large number of characters to the end of a multipart boundary. This does not affect VSS since the underlying root cause is already fixed by having a newer version 0.0.18 of python-multipart which does not have this vulnerability. |
| CVE-2025-4565 | This impacts protobuf < 4.25.8 python package, This impacts parsing of untrusted Protocol Buffers data containing an arbitrary number of recursive groups, recursive messages or a series of SGROUP tags leading to unbounded recursions and potential Denial-of-Service when protobuf pure-Python backend is used. This does not affect VSS since python backend of protobuf is not used. |
| CVE-2025-3887 | This impacts GStreamer H.265 codec parser, Malicious malformed streams can cause stack overflow in H.265 codec parser causing the application to crash. Users must take care that malicious H.265 streams are not added to VSS. This can be remedied by building and installing the GStreamer1.24.2 codec parser library after applying the patch mentioned in https://gstreamer.freedesktop.org/security/sa-2025-0001.html. |
GOVERNING TERMS
This blueprint is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products and enables use of separate open source and proprietary software and models governed by their respective licenses: NVIDIA Cosmos-Reason2-8B, NVIDIA Cosmos Reason 1-7B, ReIdentificationNet, Grounding Dino, Facebook Research SAM2, Llama 3.1 70B Instruct NIM, NVIDIA Retrieval QA Llama 3.2 1B Reranking v2 NIM, and NVIDIA Retrieval QA Llama 3.2 1B Embedding v2 NIM. Use of the sample data is governed by the NVIDIA Sample Data License. ADDITIONAL INFORMATION: Apache 2.0 for SAM2. Llama 3.1 Community License Agreement and Llama 3.2 Community License Agreement. Built with Llama.