NVIDIA Jarvis is an application framework for multimodal conversational AI services that delivers real-time performance on GPUs. The Jarvis framework includes pre-trained conversational AI models, tools in the NVIDIA AI Toolkit, and optimized end-to-end services for speech and natural language understanding (NLU) tasks.
To get started with Jarvis in a few easy steps, go to Jarvis Quick Start.
Jarvis Speech Server: Jarvis Speech Skills is a Docker image containing a toolkit for production-grade conversational AI inference. The Jarvis Speech API server exposes a simple API for performing speech recognition, speech synthesis, and a variety of natural language processing inferences. Pull and run the container by using the commands below. No GPU is required to run the sample clients.
Jarvis Speech Client: Jarvis Speech Clients is a Docker image containing sample command-line drivers for the Jarvis services. Pull and run the container by using the commands below. The client expects that a Jarvis server is running with models deployed, and all command-line drivers accept an optional argument to specify the location of the server. No GPU is required to run the sample clients.
Jarvis Quick Start Scripts: NVIDIA Jarvis includes Quick Start scripts to help you get started with Jarvis AI Services. These scripts are meant for deploying the services locally for testing and running our example applications.
Jarvis Speech Skills Helm chart: Can be used to deploy ASR, NLP, and TTS services automatically. Specifically, it is designed to automate the steps for push-button deployment to a Kubernetes cluster.
In order to use Jarvis AI Services, you are required to be logged in to NVIDIA GPU Cloud (NGC).
You have access to a Volta, Turing, or an NVIDIA Ampere architecture-based GPU.
Download Jarvis quick start scripts via the command-line with the NGC CLI tool by running the following CLI command or selecting the File Browser tab to download the scripts.
ngc registry resource download-version "nvidia/jarvis/jarvis_quickstart:1.2.1-beta"
Initialize and start Jarvis. The initialization step downloads and prepares Docker images and models. The start script launches the server.
Note: This process may take quite some time depending on the speed of your Internet connection and the number of models deployed. Each model is individually optimized for the target GPU after download.
3. Start a container with sample clients for each service.
bash jarvis_start_client.sh ```
jupyter notebook --ip=0.0.0.0 --allow-root --notebook-dir=/work/notebooks
quickstart directory, you can modify the
config.sh file with your preferred configuration. Options include which models to retrieve from NGC, where to store them, and which GPU to use if more than one is installed in your system (see Local (Docker) for more details).
The Jarvis AI Services are a set of gRPC endpoints that implement specific functional building blocks for conversational AI. Jarvis includes:
Jarvis ASR: Takes as input an audio stream or audio buffer and returns the English language text transcript, along with additional optional metadata. This works in both streaming and non-streaming (batch) modes.
Jarvis NLP: Takes as input text and performs a number of analysis algorithms. This includes a named entity recognition model, and a model to transform from text without punctuation to text with punctuation by adding periods, commas, question marks, and capitalization. Jarvis NLP also has an Analyze-Intent method which sequences calls to multiple models: a domain model for selecting which intent model to use and domain-specific intent models that return both intent labels and performs slot filling, returning regions of the query that may be required for fulfillment purposes. In all cases, the models provided may be retrained with NVIDIA TLT to tailor performance/labels to your domain.
Jarvis Core NLP: This is intended to be used to support custom models trained on userâs data via TLT/NeMo. The Jarvis Core NLP service supports three basic operations:
per-sentence classification (examples include sentiment analysis, intent recognition, abusive language detection)
per-word classification (examples include slot filling, named entity recognition, personally-identifying information detection)
text transformation (examples include machine translation, text punctuation, and spelling correction)
These three basic operations support a wide variety of practical use cases. The Jarvis NLP services are built as special cases of Jarvis Core NLP services using pre-trained models.
Jarvis TTS: Takes as input an English-language text string, one or a set of speakers, and returns an audio waveform of the given voice speaking this string. This only supports a batch mode of operation where the system will return the entire audio file once it is complete. In future versions, Jarvis will support a streaming response mode that sends the audio waveform incrementally as it is being computed.
The Jarvis AI Services can be configured at launch time by adding new models via a simple setting file. Helm chart for easy deployment and scaling with Kubernetes is provided, as well as guidance for launching the services locally for testing. This tool consists of a deployment service that uses one or more charts to describe a set of related tasks.
In its standard configuration, Jarvis runs as two separate processes: the API Server which the client should directly connect to, and a Triton Inference Server instance that performs the inferencing requests.
TLT Conversational AI - Transfer Learning Toolkit (TLT) is a python based AI toolkit for taking purpose-built pre-trained Conv AI models and customizing them with your own data.The fine tuned models can be easily exported to Jarvis for inference.
Supported models: Intent Slot
Supported models: Named Entity Recognition (NER)
Supported models: Punctuation
Supported models: Question Answering (BERT) and Question Answering (Megatron)
Supported models: Text Classification (Domain Classification-Weather)
Supported models: Tacotron 2 and WaveGlow
By pulling and using Jarvis software, you accept the terms and conditions of this license.