Parameters

The configuration is done through the config.pbtxt file available in the model-repo/kaldi_online/ directory. It allows you to specify the following:

Model path

The following parameters can be modified if you want to use your own Kaldi model.

• mfcc_filename
• ivector_filename
• nnet3_rxfilename
• fst_rxfilename
• word_syms_rxfilename

Model configuration

The model configuration parameters are passed to the model and have an impact on both accuracy and performance. The model parameters are usually Kaldi ASR parameters, meaning, if they are, you can reuse the values that are currently being used in the CPU Kaldi ASR pipeline.

• beam
• lattice_beam
• max_active
• frame_subsampling_factor
• acoustic_scale

Inference engine configuration

The inference engine configuration parameters configure the inference engine. They impact performance, but not accuracy.

• max_batch_size: The size of one execution batch on the GPU. This parameter should be set as large as necessary to saturate the GPU, but not bigger. Larger batches will lead to a higher throughput, smaller batches to lower latency.
• num_worker_threads: The number of CPU threads for the postprocessing CPU tasks, such as lattice determinization and text generation from the lattice.
• input.WAV_DATA.dims: The maximum number of samples per chunk. The value must be a multiple of frame_subsampling_factor * chunks_per_frame.

Inference process

Inference is done through simulating concurrent users. Each user is attributed to one utterance from the LibriSpeech dataset. It streams that utterance by cutting it into chunks and gets the final TEXT output once the final chunk has been sent. The -c parameter sets the number of active users being simulated in parallel.

Client command-line parameters

The client can be configured through a set of parameters that define its behavior. To see the full list of available options and their descriptions, use the -h command-line option. The parameters are:

    -v
    -i <Number of iterations on the dataset>
    -c <Number of parallel audio channels>
    -a <Path to the scp dataset file>
    -l <Maximum number of samples per chunk. Must correspond to the server config>
    -u <URL for inference service and its gRPC port>
    -o : Only feed each channel at realtime speed. Simulates online clients.
    -p : Print text outputs
    -b : Print partial (best path) text outputs

Input/Output

The API is currently experimental.

Input

The server execpts chunks of audio each containing up to input.WAV_DATA.dims samples. Per default, this corresponds to 510ms of audio per chunk. The last chunk can send a partial chunk smaller than this maximum value.

The chunk is made of a float array set in the input WAV_DATA, with the input WAV_DATA_DIM containing the number of samples contained in that chunk. Flags can be set to declare a chunk as a first chunk or last chunk for a sequence. Finally, each chunk from a given sequence is associated with a CorrelationID. Every chunk belonging to the same sequence must be given the same CorrelationID.

Output

Once the server receives the final chunk for a sequence (with the END flag set), it will generate the output associated with that sequence, and send it back to the client. The end of the sequencing procedure is:

1. Process the last chunk.
2. Flush and process the Neural Net context.
3. Generate the full lattice for the sequence.
4. Determinize the lattice.
5. Find the best path in the lattice.
6. Generate the text output for that best path.
7. Send the text back to the client.

Even if only the best path is used, we are still generating a full lattice for benchmarking purposes. Partial results (generated after each timestep) are currently not available but will be added in a future release.

Using a custom Kaldi ASR model

Support for Kaldi ASR models that are different from the provided LibriSpeech model is experimental. However, it is possible to modify the Model Path section of the config file model-repo/kaldi_online/config.pbtxt to set up your own model.