Quick Start Guide

Microservice offers the following GPU-accelerated AI-based video effects:

• Gaze-Redirection
  • This feature redirects gaze to look at the camera within a certain range of gaze angles and head poses.
• AI Green Screen (AIGS)
  • The Virtual background filter segments a video or still image into foreground and background regions. The Background Blur filter uses the segmentation mask and an input image to produce a blur effect in the background region of the input image.
• Background replacement
  • The Background replacement effect combines AIGS's virtual background filter & background composition. The Virtual background filter is used internally to generate a segmentation mask. Using the mask and an input image we composite & replace the background region of the input video.

In a typical Video Effects Microservice deployment, a service provider configures and launches the Video Effects Microservice on a GPU-based server. A client application (remote or local) connects to the microservice, negotiates the desired video effects and connection parameters via gRPC, and starts streaming video to the microservice (via RTP/UDP, gRPC/TCP or another protocol) and receives the processed video back.

The Video Effects Microservice includes Docker containers which can be used to demonstrate the typical deployment scenario described above. Specifically, this quick start guide provides step-by-step instructions for the following:

1. Configuring the Video Effects Microservice
2. Launching the Video Effects Microservice
3. Testing the microservice using the Test Client, which connects with the launched microservice, streams video to the microservice and receives the processed video back.

Prerequisites

Ensure the following prerequisites are available on your system for running the container:

• Docker >= 19.03
• nvidia-docker: latest nvidia-container-toolkit as described in the installation steps.
• Nvidia driver: 535+ (for example, 535.161.07). Driver dependency is aligned to CUDA version requirement (>= 11.8)
• Machine with GPU of any of below architectures:
  • sm_70 (e.g. V100)
  • sm_75 (e.g. T4)
  • sm_80 (e.g. A100)
  • sm_86 (e.g. A10, A40)
  • sm_89 (e.g. L4, L40)

Getting Started

Download the Video Effects Quick Start scripts in the following way:

1. Select CLI Command to copy the download command.
2. Ensure you have installed the NGC CLI tool.
3. After you install the tool, to start the download, paste the copied command in a Command Prompt window.
4. Login to NGC docker registry.

     docker login nvcr.io
   

This will prompt for username and password. Use $oauthtoken for username and your NGC_API_KEY for password

Configuring Video Effects Microservice

The file config.sh contains the parameters used for configuring various aspects of the Video Effects Microservice. Before launching the microservice, ensure that these parameters are set properly. An example file config.sh is included in the package.

• (Required) service_api_port: The port to be used for gRPC end-point.
• (Required for UDP) service_udp_port_range: Host port number(or range) to be mapped for udp streaming case, e.g. "9001", "9001-9005".
• (Required) gpu_for_service: GPU to be used in service for Video effects inference
• (Required) gpu_for_client: GPU to be used in client.
• (Required for UDP) my_pod_ip: Pod or Host ip where microservice is deployed. Microservice includes this value as udp-host ip in response back to client.
• (Optional) grpc_worker_threads: number of worker threads used by gRPC async server, default=1
• (Optional) max_streams: Maximum number of sessions micro-service can handle, default=1
• (Optional) max_udp_ports_per_stream: Maximum number of udp ports that need to be allocated per stream, default=1
• (Optional) enable_traces: Enable/Disable OpenTelemetry traces.
• (Optional) use_ostream_exporter: If enabled, OpenTelemetry traces and metrics will be printed on standard output. If disabled, OpenTelemetry will use otlp exporter and data is sent to lightstep.

Below lightstep related params are needed to supply if enable_traces=true and use_ostream_exporter=false

• (Optional) lightstep_token_filepath: Lightstep backend needs token for access. This parameter provides a local file path for token.

• (Optional) lightstep_cert_filepath: To access lightstep grpc endpoint, it needs SSL certificate. Use this parameter to provide file path.

• (Optional) lightstep_endpoint: URL to access lightstep backend.

• (Optional) Logging: service prints logs to stdout or stderr. "docker logs " can be used to check service logs.

  • (Optional) GST_DEBUG can be used to control gStreamer logs. service uses gStreamer pipeline for audio processing. For more info, Please refer gStreamer - Printing Debug Information. optional. default:0.

  • (Optional) GLOG_logtostderr can be used to enable glogs. Please find more details related to glogs here. optional. default:0.

  • (Optional) GLOG_v controls the log verbose level. High level means more detailed logs. optional. default:0.

Launching Video Effects Microservice

Set appropriate permissions:

  chmod -R 775 video_effects_v1.2.0

Pull the Video Effects Microservice docker image:

  bash ./video_effects_init.sh

Start the Video Effects service

  bash ./video_effects_start.sh

How to use custom pipline tuning configs:

1. update contents of quality tuning config for respective profile in quality_tuning_configs/
2. for changes to take effect, restart container as follows:

$ bash ./video_effects_stop.sh

$ bash ./video_effects_start.sh

The service is now launched in background mode, follow next steps to launch client docker.

Testing the Microservice using Video Effects Client Application

Command line client is a containerized executable that enables validating various features offered by Video Effects Microservice. This client is provided in the docker in binary form (executable). The client application, once configured, connects to the microservice, negotiates a session, streams video to and receives processed video from the microservice.

These example config files are provided in this package:

• config_grpc.json: uses gRPC streaming
• config_srtp_udp.json: uses secured RTP UDP streaming
• config_rtp_udp.json: uses RTP UDP streaming, contains recommended settings for RTP UDP
• config_grpc_lookaway.json: uses gRPC streaming, contains recommended settings for Gaze redirection
• config_rtp_udp_lookaway.json: uses RTP UDP streaming, contains recommended settings for RTP UDP & Gaze redirection
• config_rtp_udp_composite.json: uses RTP UDP streaming, contains recommended settings for RTP UDP & Background Replacement effect

A detailed config file description is provided in section Config file options.

For the protobuf definition, please refer to protos/video_effects/<api-version>/video_effects.proto.

To test the microservice using this pre-built client,

1. Launch the client docker with an interactive session,

     bash ./video_effects_client_start.sh
   

2. To facilitate the data transfer between host and docker, the currect directory is mounted in docker at /host.
3. You will find a sample client app inside docker at /opt/nvidia/maxine-microservices/bin/video_effects_client. Invoke the application as follows (If distributed span tracing is required, please add --enable_traces at the end):

     GLOG_v=4 video_effects_client --grpc_uri=127.0.0.1:50061 \
       --video_config=/host/configs/config_grpc.json \
       --video_input=/host/sample_inputs/Video.mp4 \
       --video_output=/host/outputs/video_effects_out.mkv
   

Upon launching the client, The client invokes gRPC bi-directional endpoint of microservice deployed at the URI specified by grpc_uri argument and establishes a gRPC channel.

How to use custom config file

1. Edit the sample config files in configs folder in the current directory on host.
2. Launch the client docker. The config files will be mounted at /host/configs inside the docker.
3. Provide the config file path through the client option --video_config.

How to access output file:

1. The directory on host: outputs/ and on client docker: /host/outputs/ are synced
2. the argument --video_output in client launch command allows us to control location of output file
3. use arg --video_ouput as shown in sample client launch command to generate and access output on host machine at: outputs/

How to use custom inputs:

1. The directory on host: sample_inputs/ and the directory in client docker: /host/sample_inputs/ are synced
2. on host machine, copy the custom input files to sample_inputs/.
3. update the args --video_input client launch command to use custom inputs.
4. provide additional args --background_input for background replacement use case

Setting custom Bitrate settings on input and output:

• Following arguments can be used to set custom bitrate values on respective inputs and output

• --input_video_bitrate: The bitrate of driving video data sent to MS, Defaults: GRPC=24Mbps, UDP=1Mbps, See Recommended values for bitrate
• --output_transcode_bitrate: The bitrate to transcode the output with, Defaults: GRPC=24Mbps, UDP=1Mbps, See Recommended values for bitrate

Config file options

The explanation of options in config file is as follows:

{
    "video_enhancement_config": {
        "enable_background_effect": false, # toggle background effect
        "background_effect_config": {
            "background_effect": "BACKGROUND_EFFECT_BLUR", # background replacement vs blur, "BACKGROUND_EFFECT_BLUR" & "BACKGROUND_EFFECT_COMPOSITE" supported
            "background_blur_config": {
                "mode": "VIDEO_EFFECT_MODE_QUALITY", # quality vs perf, value "VIDEO_EFFECT_MODE_PERF" recommended for live streaming
                "blur_strength": 1.0 # blur strength in background blur effect.
            }
        },
        "enable_gaze_redirection": true, # toggle gaze redirect feature
        "gaze_redirection_config": {
            "eye_size_sensitivity": "EYE_SIZE_SENSITIVITY_3", # keep default, recommended value
            "temporal_optimize": true, # keep default, recommended true
            "enable_gaze_look_away": false, # is set to true, enables occasional lookaways in output, to avoid continous stare
            "gaze_look_away_config": {
                "max_look_away_offset": { # controls look away angle, see 'protos/video_effects/v1/video_effects.proto' for more details
                    "value": 5,
                    "unit": "UNIT_DEGREE_ANGLE"
                },
                "min_look_away_interval": { # controls frequency of lookaways, see 'protos/video_effects/v1/video_effects.proto' for more details
                    "value": 100,
                    "unit": "UNIT_FRAME"
                },
                "look_away_interval_range": { # controls frequency of lookaways, see 'protos/video_effects/v1/video_effects.proto' for more details
                    "value": 250,
                    "unit": "UNIT_FRAME"
                }
            }
        }
    },
    "endpoint_config": {
        "input_media_config": {
            "video_input_config": {
                "stream_config": {
                    "stream_type": "GRPC" # input stream type, use "UDP" for RTP/UDP
                },
                "payloader_config": {
                    "type": "PAYLOADER_RTP" # other values: "PAYLOADER_UNSPECIFIED"
                },
                "encoding": "H264" # input encoding
            }
        },
        "output_media_config": {
            "video_output_config": {
                "stream_config": {
                    "stream_type": "GRPC" # output stream type, use "UDP" for RTP/UDP
                },
                "payloader_config": {
                    "type": "PAYLOADER_RTP" # other values: "PAYLOADER_UNSPECIFIED"
                },
                "encoding": "H264" # output encoding
            }
        }
    },
    "quality_profile": "VIDEO_EFFECT_QUALITY_PROFILE_HIGH_QUALITY", # used to tune service componets to deliever expected experience. refer to "video_effects/protos/video_effects/<api-version>/video_effects.proto" for more details
}

Stopping the Video Microservice

  bash ./video_effects_stop.sh

Possible errors

• Values of docker container name, ports can conflict with existing containers.
  • Solution: change the conflicting values in config.sh respectively
• Please use compatible media player like VLC or ffplay player if the output media is not playable using Windows Media Player.

Microservice(s) endpoint proto definitions:

• Video microservice proto protos/video_effects/<api-version>/video_effects.proto.

Diagnostics

The microservice prints logs to stdout or stderr. To access service logs, run the docker logs command:

 docker logs -f [container-name]

• -f to Follow log output

Suggested Reading

Please refer to Video Effect Microservice User Guide(Under resources) on NGC.

License

By pulling and using Maxine software, you accept the terms and conditions of the corresponding license.