Quick Start

Package Contents

Unzip the package locally using

unzip app_operator.zip -d app_operator

The operator development package should contain the following.

└─ app_operator
   ├─ Dockerfile
   ├─ main.py
   ├─ build.sh
   ├─ run-local.sh
   ├─ run-triton.sh
   ├─ requirements.txt
   ├─ env.list
   ├─ def
   |  └─ one-operator-pipeline.yml
   ├─ lib
   |  └─ clara-0.1-py3-none-any.whl
   └─ local
      ├─ input
      |  └─ liver_14.nii.gz
      ├─ output
      └─ models
         └─ segmentation_liver_v1
            ├─ 1
            │  └─ model.graphdef
            └─ config.pbtxt

• Dockerfile builds the operator container. This build script can be used to build the operator container, or to package an existing custom inference container image as a Clara operator (please see Packaging Existing Container below).
• main.py is the Python script that is executed when the operator is instantiated. The script reads NIFTI files from an input folder, applies simple transformations to each file to prepare them for inference, and uses the Triton client library to send the inference to the liver segmentation model deployed in Triton, outputting the result of each Triton inference in the output folder.
• build.sh is a Bash script that initiates the docker build.
• run-local is a Bash script that initiates a local (stand-alone) containerized operator. For run-local.sh to run successfully the user must first build the operator container image using build.sh and run run-triton.sh to make the model available to the operator for inference.
• run-triton.sh is a Bash script that initiates a local containerized Triton server.
• requirements.txt lists all the libraries needed for main.py to run.
• env.list lists all the environment variable names required to run the Clara operator locally (baremetal or containerized) for development and debugging purposes.
• def contains the operator and pipeline (one-operator-pipeline.yaml) definitions necessary to deploy the operator in Clara. Note that Clara will accept only the deployment of pipelines, not operators alone, therefore ensure that the operator is available to a container repository reachable by Clara at runtime.
• local is a directory containing the model artifacts necessary to run main.py locally in the state it is distributed.
  • run-triton.sh will load the local/models folder into the Triton container to make it available for inference.
  • run-local will load the local/input folder into the operator container to allow the operator container to read liver_14.nii.gz and send it for inference.
• lib contains the Clara operator Python client library. This library must always be installed for a Clara operator to function correctly.

Run the Example Liver Segmentation Model Locally

Before running the inference model locally make sure to install Docker CE, and NVIDIA Docker 2 to be able to serve the model on a GPU. You may follow the steps here.

To run the packaged liver inference model:

1. Build the Clara example inference operator using

./build.sh

2. Start the Triton inference server using

./run-triton.sh

3. Run the Clara operator locally using

./run-local.sh

4. Once the inference completes check the output of the inference in the local/output directory.

Deploy Single-Operator Pipeline to Existing Clara Cluster

This section assumes that you have installed a Clara cluster which has access to the container image built using build.sh (by default the image is tagged clara/simple-operator:0.1.0). To install or upgrade Clara Deploy SDK, follow the steps outlined on the Installation page of the Clara Deploy User Guide.

Deploy the Model

Before deploying the operator, one must ensure that the inference model is deployed in Clara (if an inference model is used). In the case of the example liver segmentation model, the user must copy the contents under local/models to the Clara model repository directory, by default /clara/common/models, resulting in a directory structure such as

clara
  └─ common
        └─ models
           └─ segmentation_liver_v1
              ├─ 1
              │  └─ model.graphdef
              └─ config.pbtxt

Deploy the Operator

To deploy the singleton-operator example pipeline provided

cd def
clara create pipelines -p one-operator-pipeline.yml

clara create will output a pipeline id, PIPELINE_ID, if deployed successfully.

To run the pipeline with the given data we must first create a job

cd ..
clara create job -p PIPELINE_ID -n liver-job -f local/input

the command will output a job id JOB_ID.

To run the created job use

clara start job -j JOB_ID

Deploy an Existing Inference Container as a Clara Operator

To package a custom inference container as a Clara operator you may update the Dockerfile to do so. For instance, the Dockerfile to package an existing container may look something like

FROM my_inference_container:0.1.0

...

# install Clara-compatible Triton client
ARG TRITON_CLIENTS_URL=https://github.com/NVIDIA/tensorrt-inference-server/releases/download/v1.5.0/v1.5.0_ubuntu1804.clients.tar.gz
RUN mkdir -p /opt/nvidia/trtis-clients \
    && curl -L ${TRITON_CLIENTS_URL} | tar xvz -C /opt/nvidia/triton-clients
RUN pip install --no-cache-dir future==0.18.2 grpcio==1.27.2 protobuf==3.11.3 /opt/nvidia/triton-clients/python/*.whl

# install operator requirements
RUN pip install --upgrade setuptools &&\
    pip install -r requirements.txt

ENTRYPOINT ["bash", "-c", "python -u -m clara.app my_script.py"]

Note that:

1. You must include the Triton client installation steps in the script if you want to use Triton for inference. This step can be omitted if the inference model is packaged in your container.
2. You must always install the whl file in the lib/ directory as it contains the wrapper code necesary for the Clara operator to function correctly.
3. The ENTRYPOINT to your container must use the clara.app Python module for the operator to function correctly. Specifically, if your main script is my_script.py then your container entry-point should be

python -u -m clara.app my_script.py