NVIDIA

BERT QA on Azure ML with Triton Demo

Resource

NVIDIA

BERT QA on Azure ML with Triton Demo

Demo notebook to deploy BERT QA model on Azure ML with Triton Inference Server

Back to File Browser

bert-aks-v100.ipynb

In [1]:

# Copyright 2020 NVIDIA Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

Deploy BERT to Triton Inference Server on AKS

Overview

Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. BERT can be fine-tuned to perform Question-Answering tasks, as shown here. You can also optimize the BERT QA engine to improve the inference performance, as shared here. In this notebook, we will deploy BERT QA model with Triton Inference Server on Azure Kubernetes Service.

Before starting to follow the steps below, you should have created the workspace on Azure Machine Learning and installed Azure Machine Learning SDK for Python

Additionally, you can find more information of the topics discussed below from Azure Documentation, Deploy models with Azure Machine Learning, and more about NVIDIA Triton Inference Server from here

Objectives

This notebook demonstrates:

step-by-step guide with code and explanation to deploy NVIDIA-TensorRT-optimized BERT QA model with Triton Inference Server on Azure Machine Learning.

Requirements
Connect To Your Workspace
Create AKS Compute Target
Register BERT Model
Define Inference Configuration And Environment
Deploy BERT Model
Test The Webservice
Check The Response
Clean Up

Requirements

Ensure you have a config.json specified in the root of this directory. To create one, refer to the How-to-configure-environment.
TensorRT Optimized BERT QA model. Follow the blog to create one

Connect To Your Workspace

We need config.json file downloaded from Azure ML Workspace webpage and place it at the root dir of this notebook. Executing the below cell instantiates the workspace object.

In [1]:

from azureml.core import Workspace

In [ ]:

ws = Workspace.from_config(path="config.json")
ws

Create AKS Compute Target

Following cells create a GPU-accelerated Azure Kubernetes Service(AKS) cluster. You should adjust the variables such as compute_name, compute_loc, vm_size to meet your needs.

In [3]:

from azureml.core.compute import ComputeTarget, AksCompute
from azureml.core.compute_target import ComputeTargetException

In [4]:

compute_name = 'aks-gpu'
compute_loc = 'southcentralus'
vm_size = 'Standard_NC6s_v3'

try:
    gpu_cluster = ComputeTarget(workspace=ws, name=compute_name)
    print(f"Found existing gpu-cluster, {compute_name}")
except ComputeTargetException:
    print(f"Creating new gpu-cluster, {compute_name}")
    
    # Specify the configuration for the new cluster
    compute_config = AksCompute.provisioning_configuration(
        cluster_purpose=AksCompute.ClusterPurpose.DEV_TEST,
        agent_count=1,
        vm_size=vm_size,
        location=compute_loc
    )

    # Create the cluster with the specified name and configuration
    gpu_cluster = ComputeTarget.create(ws, compute_name,
                                       compute_config)

    # Wait for the cluster to complete, show the output log
    gpu_cluster.wait_for_completion(show_output=True)

Creating new gpu-cluster, aks-gpu
Creating...................................................................................
SucceededProvisioning operation finished, operation "Succeeded"

Register BERT Model

BERT model file, following the triton model repository structure, has to be accessible locally. Registering the model in Azure uploads the model artifacts to the cloud. Shown below is the example directory structure when a user uses TensorRT engine for BERT model. Note that Triton Inference Server expects specific folder structure for Triton to be able to access serialized model files. For more details of the model repository setup of Triton, refer to this page

<path/to/notebook/workspace>  
    |--model  
        |--triton  
            |--bert  
                |--1  
                    |--model.plan

You can also read more about the registering model on Azure ML

In [5]:

from azureml.core.model import Model
import os

In [6]:

model_path = os.path.join(os.getcwd(), 'model')

model = Model.register(workspace=ws,
                       model_path=model_path,
                       model_name='bert')

WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /azureml/LocalUpload/201022T063700-f769845d/model/triton/bert/1/model.plan?sv=2019-02-02&sr=b&sig=qOg6AexyXNBm%2BoCEFFg9BUL55PE4Dz5xSGbf2QWp4tU%3D&st=2020-10-22T13%3A27%3A01Z&se=2020-10-23T13%3A37%3A01Z&sp=rcw&comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQTJNVEl6Tmpnek9EUSUzRA%3D%3D&timeout=30
WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', timeout('The write operation timed out',))': /azureml/LocalUpload/201022T063700-f769845d/model/triton/bert/1/model.plan?sv=2019-02-02&sr=b&sig=qOg6AexyXNBm%2BoCEFFg9BUL55PE4Dz5xSGbf2QWp4tU%3D&st=2020-10-22T13%3A27%3A01Z&se=2020-10-23T13%3A37%3A01Z&sp=rcw&comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQTJNVEl6Tmpnek9EUSUzRA%3D%3D&timeout=30
WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', timeout('The write operation timed out',))': /azureml/LocalUpload/201022T063700-f769845d/model/triton/bert/1/model.plan?sv=2019-02-02&sr=b&sig=qOg6AexyXNBm%2BoCEFFg9BUL55PE4Dz5xSGbf2QWp4tU%3D&st=2020-10-22T13%3A27%3A01Z&se=2020-10-23T13%3A37%3A01Z&sp=rcw&comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQTJNVEl6Tmpnek9EUSUzRA%3D%3D&timeout=30

Registering model bert

Define Inference Configuration And Environment

An inference configuration describes how to set up the web-service containing the BERT model you registered in the previous step. Inference configuration includes a entry script that is to be called when web-service inference request is created.

An Environment, or Azure Machine Learning environment, allows developers to control SW dependency management. Azure ML Environment class accoutns for local development solutions such as PIP and Conda. A developer can ceate a custom environment based on Azure ML's curated environment. We clone AzureML-Triton environment from Azure's curated env list and install TensorFlow.

In [7]:

from azureml.core.environment import Environment
from azureml.core.model import InferenceConfig

In [20]:

env_name = 'env-bert-triton'

env = Environment.get(ws, "AzureML-Triton").clone(env_name)

for pip_package in ["tensorflow==1.15"]:
    env.python.conda_dependencies.add_pip_package(pip_package)
#     env.python.conda_dependencies.add_conda_package(pip_package)

inference_config = InferenceConfig(entry_script='score_bert.py',
                                   source_directory='scripts',
                                   environment=env)

In [21]:

env

Out[21]:

{
    "databricks": {
        "eggLibraries": [],
        "jarLibraries": [],
        "mavenLibraries": [],
        "pypiLibraries": [],
        "rcranLibraries": []
    },
    "docker": {
        "arguments": [],
        "baseDockerfile": null,
        "baseImage": "mcr.microsoft.com/azureml/aml-triton:20200918.v1",
        "baseImageRegistry": {
            "address": null,
            "password": null,
            "registryIdentity": null,
            "username": null
        },
        "enabled": false,
        "platform": {
            "architecture": "amd64",
            "os": "Linux"
        },
        "sharedVolumes": true,
        "shmSize": null
    },
    "environmentVariables": {
        "EXAMPLE_ENV_VAR": "EXAMPLE_VALUE"
    },
    "inferencingStackVersion": null,
    "name": "env-bert-triton",
    "python": {
        "baseCondaEnvironment": null,
        "condaDependencies": {
            "channels": [
                "conda-forge"
            ],
            "dependencies": [
                "python=3.6.2",
                {
                    "pip": [
                        "azureml-core==1.16.0",
                        "azureml-defaults==1.16.0",
                        "azureml-contrib-services==1.16.0",
                        "numpy",
                        "inference-schema[numpy-support]",
                        "https://aka.ms/triton/packages/tritonclientutils-2.1.0-py3-none-any.whl",
                        "https://aka.ms/triton/packages/tritonhttpclient-2.1.0-py3-none-any.whl",
                        "tensorflow==1.15"
                    ]
                }
            ],
            "name": "azureml_98872e7f8a24d09d2207491887a2e936"
        },
        "condaDependenciesFile": null,
        "interpreterPath": "python",
        "userManagedDependencies": false
    },
    "r": null,
    "spark": {
        "packages": [],
        "precachePackages": true,
        "repositories": []
    },
    "version": null
}

Entry Script Details

Entry Script should have init() and run() defined inside the script. The shown below is the entry script for BERT we are about to deploy. Notice that the original script contains other helper functions necessary to pre and post process data for inference, but they are removed for brevity. More details of entry script can be found here.

(Note) Original script can be found from scripts/score_bert.py

def init():
    """Initializes the triton client and  point at the specified URL

    Parameter
    ----------
    url : str
        The URL on which to address the Triton server, defaults to
        localhost:8000
    """
    global triton_client
    
    triton_client = tritonhttpclient.InferenceServerClient(url="localhost:8000")

    print(get_model_info())

@rawhttp
def run(request):
    """This function is called every time your webservice receives a request.

    Notice you need to know the names and data types of the model inputs and
    outputs. You can get these values by reading the model configuration file
    or by querying the model metadata endpoint.
    """

    if request.method == 'POST':
        model_name = 'bert'
        input_meta, input_config, output_meta, output_config = \
            parse_model_http(model_name=model_name)
        input_name = input_meta[0]['name']
        input_dtype = input_meta[0]['datatype']
        output_name = output_meta[0]['name']

        req_body = request.get_data(False)

        loaded_body = json.loads(req_body)

        init_bert_config()

        inputs = preprocess(loaded_body)

        res = triton_infer(
            model_name=model_name, input_mapping=inputs,
            binary_data=False, class_count=0)

        result = postprocess(
            results=res, output_name=output_name, batch_size=1, batching=False)

        return AMLResponse(result, 200)
    else:
        return AMLResponse("bad request", 500)

Deploy BERT Model

We deploy the registered BERT model to AKS Compute Target, both created in the previous steps. Read more about the deployment process here.

Deployment process needs deployment configuration where you can modify the machine configuration of your requirement.

In [22]:

from azureml.core.webservice import AksWebservice

service_name = "aks-bert-triton-1"

deployment_config = AksWebservice.deploy_configuration(
    compute_target_name=compute_name,
    gpu_cores=1,
    cpu_cores=1,
    memory_gb=4,
    auth_enabled=True,
)

service = Model.deploy(
    workspace=ws,
    name=service_name,
    models=[model],
    deployment_config=deployment_config,
    inference_config=inference_config,
    overwrite=True,
)

service.wait_for_deployment(show_output=True)

Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running.................
Succeeded
AKS service creation operation finished, operation "Succeeded"

Test The Webservice

Once deployed, you can create a inference request with context and question with which BERT QA model would return an answer.

Feel free to test your own context and question in the cell below. Context should contain answer to the given question.

In [30]:

import json

title = "TensorRT"
context = "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps."
question = "What is TensorRT?"

body = {
    "data": [
        {"title": title, 
        "paragraphs": [
            {"context": context, 
            "qas": [
                { "question": question, 
                "id": "Q1"}
            ]}
        ]}
    ]}

req_data = json.dumps(body, indent=2)

print(req_data)

{
  "data": [
    {
      "title": "TensorRT",
      "paragraphs": [
        {
          "context": "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps.",
          "qas": [
            {
              "question": "What is TensorRT?",
              "id": "Q1"
            }
          ]
        }
      ]
    }
  ]
}

In [24]:

import requests

aks_service = AksWebservice(ws, service_name)

key, _ = aks_service.get_keys()

headers = {'Content-Type': 'application/json',
           'Authorization': 'Bearer ' + key}

resp = requests.post(aks_service.scoring_uri, data=req_data, headers=headers)

Check The Response

In [27]:

print('Title: ')
print(f'\t{title}')
print()
print('Context: ')
print(f'\t{context}')
print()
print('Question: ')
print(f'\t{question}')
print()
print('Response from BERT QA model: ')
print(f'\t{resp.text}')
print()

Title: 
	TensorRT

Context: 
	TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps.

Question: 
	What is TensorRT?

Response from BERT QA model: 
	a high performance deep learning inference platform

What's Next

Head to NVIDIA Deep Learning Example github repo or NGC to download the BERT resources and create your own Question-Answering webservice with BERT on Azure Machine Learning. For more details of how to create TensorRT-optimized BERT QA model, refer to the developer blog.

Clean Up

In [31]:

service.delete()

No service with name aks-bert-triton-1 found to delete.