Kaldi ASR Integration With Triton Inference Server

NVIDIA

Resource

NVIDIA

Kaldi ASR Integration With Triton Inference Server

Kaldi ASR custom backend for the NVIDIA Triton Inference Server.

Metrics

Throughput is measured using the RTFX metric. It is defined such as : RTFX = (number of seconds of audio inferred) / (compute time in seconds). It is the inverse of the RTF (Real Time Factor) metric, such as RTFX = 1/RTF.

Latency is defined as the delay between the availability of the last chunk of audio and the reception of the inferred text. More precisely, it is defined such as :

Client: Last audio chunk available
t0 <- Current time
Client: Send last audio chunk
Server: Compute inference of last chunk
Server: Generate the raw lattice for the full utterance
Server: Determinize the raw lattice
Client: Receive lattice output
Client: Call callback with output
t1 <- Current time

The latency is defined such as latency = t1 - t0.

Results

Our results were obtained by:

Building and starting the server as described in Quick Start Guide.
Running scripts/run_inference_all_a100.sh, scripts/run_inference_all_v100.sh and scripts/run_inference_all_t4.sh

GPU	Realtime I/O	Number of parallel audio channels	Latency (s)
			90%	95%	99%	Avg
A100	Yes	2000	0.11	0.12	0.14	0.09
V100	Yes	2000	0.42	0.50	0.61	0.23
V100	Yes	1000	0.09	0.09	0.11	0.07
T4	Yes	600	0.17	0.18	0.22	0.14
T4	Yes	400	0.12	0.13	0.15	0.10