Advanced

Customizing the Operator Development Kit Example Code

The Operator Development Kit (ODK) can be used as a starting point for the development of custom Clara pipeline operators or the migration of existing container images into Clara pipeline operators. This section explains the focal components of the ODK when building a custom Clara pipeline to perform inference.

Clara Operator-specific Variables in Python

main.py is the functional part of the code the developer may customize for their own purposes. As provided, main.py is an example of decorating the code with the necessary components. Clara Deploy will expect certain components in the pipeline operator definition.

• Enviornment variables
• Inputs
• Pre-transformations
• Infererence
• Post-transformations
• Outputs

The following describes the components used in the ODK sample application.

1. The input component reads data from the input payload. The input directory path is defined by the OPERATOR_INPUT_PATH environment variable. When the environment variable is unavailable, the default path of /input is used.

 input_path = os.getenv('OPERATOR_INPUT_PATH', '/input')

2. The pre-transformation component applies pre-inference transformations to the input data. This component is optional and model dependent.

 pre_transforms = Compose([
 LoadImage(reader="NibabelReader", image_only=True, dtype=np.float32),
 AddChannel(),
 ScaleIntensityRange(a_min=-1000, a_max=500, b_min=0.0, b_max=1.0, clip=True),
 CropForeground(margin=5),
 Resize([192, 192, 64], mode="area"),
 ])

3. The inference uses the Triton Python client to perform inference via the model deployed to Triton Inference Server (see Section "Deploy the Model" below),

 inference_ctx = initialize_inference_server()
 inference_response = inference_ctx.run({ model_input_label : (inference_image,) },
 { model_output_label: InferContext.ResultFormat.RAW },
 batch_size = 1)

4. Post-transformation component applies post-inference transformations to the inference result,

 post_transforms = Compose([
 ToTensor(),
 Activations(sigmoid=True),
 AsDiscrete(threshold_values=True, logit_thresh=0.5),
 ToNumpy(),
 ])

5. Output component finally writes the output data to the path defined by OPERATOR_OUTPUT_PATH. When the environment variable is unavailable, the default value of /output is used.

 output_path = os.getenv('OPERATOR_OUTPUT_PATH', '/output')

The full code set can be found in main.py

Deploying a Single-Operator Pipeline to Existing Clara Deploy Cluster

This section assumes that you have installed a Clara Deploy cluster which has access to the container image built using build.sh (by default the image is tagged clara/simple-operator:0.1.0). To install or upgrade Clara Deploy SDK, follow the steps outlined on the Installation page of the Clara Deploy User Guide.

Deploy the Model

Before deploying the operator, one must ensure that the inference model is deployed in Clara (if an inference model is used). In the case of the example COVID-19 classification model, the user must copy the contents under local/models to the Clara model repository directory, by default /clara/common/models, resulting in a directory structure such as

clara
 └─ common
 └─ models
 └─ covid
 ├─ 1
 │ └─ model.pt
 └─ config.pbtxt

The ODK contains a PyTorch model, however, the user may deploy ONNX, Tensorflow Graphdef, Tensoflow SavedMode, TensorRT, and other formats supported by the Triton Inference Server, with an accompanying config.pbtxt for the model.

Deploy the Operator

To deploy the singleton-operator example pipeline provided start in the top level ODK folder.

cd pipelines
clara create pipelines -p one-operator-pipeline.yml

clara create will output a pipeline id, ``, if deployed successfully.

Before trying to run a job from the newly deployed pipeline, the user must ensure that the container image built using build.sh is reachable from the Kubernetes cluster on which Clara Deploy is running. For instance, if the container image is my_inference_container:0.1 either ensure

• that this image is present in the container repository local to Clara (i.e. in the same machine), or
• available in a public container repository reachable by the Kubernetes cluster where Clara Deploy is deployed (e.g. DockerHub).

To run the pipeline with the given data we must first create a job

cd ..
clara create job -p -n covid-class -f local/input

The output will be a job id ``.

It is important to provide the input option with -f here as this takes the local input data and moves (uploads) it to the expected Clara operator location for input data.

JOB_ID: 
PAYLOAD_ID: 
Payload uploaded successfully.

To run the created job use

clara start job -j 

To check for a successful job use

clara list jobs

Look for the and corresponding for the job.

Inspect the outputs using the and correspondingin the corresponding `` folder which by default Clara always stores in the defined payloads folder.

cd /clara/payloads/

The output of the classification results are stored as text correxponding with each of the inputs as shown below.

├── input
│ ├── volume-covid19-A-0000.nii.gz
│ ├── volume-covid19-A-0001.nii.gz
│ └── volume-covid19-A-0010.nii.gz
├── NVIDIA
│ └── Clara
└── operators
 └── simple-operator
 └── classification-output
 ├── volume-covid19-A-0000.nii_result.txt
 ├── volume-covid19-A-0001.nii_result.txt
 └── volume-covid19-A-0010.nii_result.txt

Package Existing Container

To package a custom inference container as a Clara operator pipeline you may update the Dockerfile and build.sh script. Packaging your own code to run in a Clara operator requires you to define and install environment requirements, an ENTRYPOINT, and any optional dependencies for your inference application.

For example, the Dockerfile to package an existing container may look something like

FROM my_inference_container:0.1.0

...

# install Clara-compatible Triton client
ARG TRITON_CLIENTS_URL=https://github.com/triton-inference-server/server/releases/download/v2.11.0/v2.11.0_ubuntu2004.clients.tar.gz
RUN mkdir -p /opt/nvidia/triton-clients \
 && curl -L ${TRITON_CLIENTS_URL} | tar xvz -C /opt/nvidia/triton-clients
RUN pip install --no-cache-dir future==0.18.2 grpcio==1.27.2 protobuf==3.11.3 /opt/nvidia/triton-clients/python/*.whl

# install operator requirements
RUN pip install --upgrade setuptools &&\
 pip install -r requirements.txt

ENTRYPOINT ["bash", "-c", "python -u my_script.py"]

Note that, you must include the Triton client installation steps in the script if you want to use Triton for inference. This step can be omitted if the inference model is packaged in your container.

Similary, use the build.sh script to package the container. Simply pass the fully qualified name and tag of the desired container to the script.

note: By defulat the script will create an operator named clara/simple-operator:1.0.0if you do not specify the name in the command line.

Once the container is successfuly created, then go back to Deploy The Operator and follow the same steps using your custom container.