NVIDIA

NVIDIA Time Series Prediction Platform

Resource

NVIDIA

NVIDIA Time Series Prediction Platform

NVIDIA Time Series Prediction Platform is a tool designed to compare easily and experiment with arbitrary combinations of forecasting models, time-series datasets, and other configurations.

Back to File Browser

2_TrtInference.ipynb

In [1]:

# Copyright 2023 NVIDIA Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

Please note that due to limitations of this instance we will showcase a different version of the NVIDIA Triton Model Navigator than is visible in the Deep Learning Examples TimeSeriesPredictionPlatform repo. On DLE, we can perform conversion and deployment, while here we are limited to just conversion. This docker image will not work for the deployment on the DLE version of the TSPP.

In [2]:

%load_ext autoreload
%autoreload 2

NOTE: These notebooks are designed to highlight different features of NVIDIA-TSPP. For this reason, all the examples are created to run quickly with only few iterations. The parameters should be tuned to get optimal results.

1 | Introduction

Setup
Export and Conversion
Inference
Benchmark

In [3]:

import os
import numpy as np
import torch
import yaml
from omegaconf import OmegaConf

import model_navigator as nav
import conf.conf_utils
from triton.model import get_model
from triton.dataloader import get_dataloader_fn
from pathlib import Path

from collections import Iterator

curr_workdir = globals()['_dh'][0]

/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Using backend: pytorch
/tmp/ipykernel_1/884079274.py:13: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  from collections import Iterator

Setup

Setup the Checkpoint Directory

In [4]:

# Since this notebook uses the NVIDIA-TSPP container. TSPP code is available at /workspace directory
tspp_ws = "/workspace"

In [5]:

checkpoint_dir = os.path.join(curr_workdir, F'outputs/2_TrtInference_tft_electricity')
os.makedirs(checkpoint_dir, exist_ok = True)

Train a quick TFT model on a single epoch to use for deployment. Take a look at the 1_TsppOverview notebook for data download and preprocessing instructions in addition to an in depth description on training.

In [6]:

dataset = "electricity"

In [7]:

!python {tspp_ws}/data/script_download_data.py --dataset {dataset} --output_dir {tspp_ws}/datasets/

Using backend: pytorch
#### Running download script ###
Getting electricity data...
/workspace/datasets/electricity
Pulling data from https://archive.ics.uci.edu/ml/machine-learning-databases/00321/LD2011_2014.txt.zip to /workspace/datasets/electricity/LD2011_2014.txt.zip
100% [..................................................] 261335609 / 261335609done
Unzipping file: /workspace/datasets/electricity/LD2011_2014.txt.zip
Done.
Aggregating to hourly data
Done.
Download completed.

In [8]:

!python {tspp_ws}/launch_preproc.py dataset={dataset}

{'_target_': 'data.data_utils.Preprocessor', 'config': {'graph': False, 'source_path': '/workspace/datasets/electricity/electricity.csv', 'dest_path': '/workspace/datasets/electricity/', 'time_ids': 'days_from_start', 'train_range': [0, 1315], 'valid_range': [1308, 1339], 'test_range': [1332, 10000], 'dataset_stride': 1, 'scale_per_id': True, 'encoder_length': 168, 'example_length': 192, 'MultiID': False, 'features': [{'name': 'categorical_id', 'feature_type': 'ID', 'feature_embed_type': 'CATEGORICAL', 'cardinality': 371}, {'name': 'hours_from_start', 'feature_type': 'TIME', 'feature_embed_type': 'CONTINUOUS'}, {'name': 'power_usage_weight', 'feature_type': 'WEIGHT', 'feature_embed_type': 'CONTINUOUS'}, {'name': 'power_usage', 'feature_type': 'TARGET', 'feature_embed_type': 'CONTINUOUS', 'scaler': {'_target_': 'sklearn.preprocessing.StandardScaler'}}, {'name': 'hour', 'feature_type': 'KNOWN', 'feature_embed_type': 'CATEGORICAL', 'cardinality': 25}, {'name': 'day_of_week', 'feature_type': 'KNOWN', 'feature_embed_type': 'CATEGORICAL', 'cardinality': 8}, {'name': 'hours_from_start', 'feature_type': 'KNOWN', 'feature_embed_type': 'CONTINUOUS', 'scaler': {'_target_': 'sklearn.preprocessing.StandardScaler'}}, {'name': 'categorical_id', 'feature_type': 'STATIC', 'feature_embed_type': 'CATEGORICAL', 'cardinality': 371}], 'train_samples': 450000, 'valid_samples': 50000, 'binarized': True, 'time_series_count': 369}}
Reading in data from CSV File: /workspace/datasets/electricity/electricity.csv
Sorting on time feature
Mapping nodes
Mapping categoricals to bounded range
Splitting datasets
Calculating scalers
Applying scalers
Applying scalers
Applying scalers
Fixing any nans in continuous features
Fixing any nans in continuous features
Fixing any nans in continuous features
Saving preprocessor state at /workspace/datasets/electricity/tspp_preprocess.bin
Saving processed data at /workspace/datasets/electricity/

In [9]:

!python {tspp_ws}/launch_training.py model=tft dataset={dataset} trainer/criterion=quantile \
    trainer.config.num_epochs=1 +trainer.config.force_rerun=true hydra.run.dir={checkpoint_dir}

Using backend: pytorch
 Training with 1 epochs 
 Epoch 0 
Epoch 0 | step 0 |avg loss 1.424 |walltime 0.879 |
Epoch 0 | step 25 |avg loss 0.604 |walltime 3.716 |
Epoch 0 | step 50 |avg loss 0.414 |walltime 6.558 |
Epoch 0 | step 75 |avg loss 0.324 |walltime 9.397 |
Epoch 0 | step 100 |avg loss 0.276 |walltime 12.215 |
Epoch 0 | step 125 |avg loss 0.251 |walltime 15.043 |
Epoch 0 | step 150 |avg loss 0.239 |walltime 17.873 |
Epoch 0 | step 175 |avg loss 0.231 |walltime 20.697 |
Epoch 0 | step 200 |avg loss 0.228 |walltime 23.519 |
Epoch 0 | step 225 |avg loss 0.219 |walltime 26.349 |
Epoch 0 | step 250 |avg loss 0.216 |walltime 29.176 |
Epoch 0 | step 275 |avg loss 0.215 |walltime 32.007 |
Epoch 0 | step 300 |avg loss 0.208 |walltime 34.844 |
Epoch 0 | step 325 |avg loss 0.208 |walltime 37.678 |
Epoch 0 | step 350 |avg loss 0.205 |walltime 40.497 |
Epoch 0 | step 375 |avg loss 0.206 |walltime 43.323 |
Epoch 0 | step 400 |avg loss 0.206 |walltime 46.152 |
Epoch 0 | step 425 |avg loss 0.203 |walltime 48.994 |
 Calculating Validation Metrics 
 Epoch 0 Validation Metrics: {'val_loss': 0.2297} 
Epoch 0 | step event |avg loss 0.202 |walltime 52.573 |
 Saving checkpoint to best_checkpoint.zip 
 Saving checkpoint to last_checkpoint.zip 
 Training Stopped 
 MAE : 47.242537606396986  RMSE : 409.92711211571003  SMAPE : 9.563726724549715  ND : 0.06218640786648368

Export and Conversion

The batch_size will determine the batch size that the final inference model is optimized for. This is mainly relevant with the NVIDIA TensorRT (TRT) format. Additionally, we set the output path for the final formats of the model and metadata that is used by the NVIDIA Triton Model Navigator.

In [10]:

batch_size = 512
output_path = f'{tspp_ws}/conversion/'

Build the model using the checkpoint_dir from the training above. data_names contains the names of the inputs for the model, which are used for the onnx conversion.

In [11]:

model, data_names = get_model(model_dir=checkpoint_dir)

Build the dataloader using the checkpoint_dir of the training above and the specified batch_size.
Wrap the dataloader in an Iterator class, in this case we use a list. In order to use the export feature below the length of the dataloader must be known. In the case below we set a limit of 5.
Ideally the final models here would be deployed to Triton, where empty Tensors and dictionaries are not accepted. Our typical dataloader outputs a dictionary where some values can be None depending on the dataset.
- To handle the dictionary issue, we pass all the values to the Triton model and rebuild the dictionary on the server side.
- To handle the empty tensors, we replace fill them with np.nans to act as a special value. Again on the server side, we convert tensors with np.nans back to empty tensors.
- These modifications to are present as wrappers on the dataloader and model from the get_dataloader_fn and get_model functions respectively.

In [12]:

total_batches = 5
dataloader = get_dataloader_fn(checkpoint_dir, batch_size)
data_list = []
for i, batch in enumerate(dataloader):
    if i >= total_batches:
        break
    data_list.append(batch[1])

Export will attempt to perform all possible conversion paths for PyTorch. More information on it can be found here. Note that there will be errors that are shown, this is due to model exports failing which is expected. There are also warnings that will be printed relating to the NaNs present in the data, this is expected and an explanation is above. A summary will be shown in the following cell of what worked and what did not work.

In [13]:

raw_data = data_list[0]
package_descriptor = nav.torch.export(
    model=model,
    dataloader=data_list,
    override_workdir=True,
    workdir=Path(output_path),
    input_names=tuple(data_names['inputs']),
    rtol=1000.0, # A very large buffer to ensure that we can get profiling scripts.
    atol=1000.0, # A very large buffer to ensure that we can get profiling scripts.
    target_device='cuda',
    max_workspace_size=10000000000,
)

Unable to get git info: Command '['git', 'config', '--get', 'remote.origin.url']' returned non-zero exit status 1.
2023-01-27 20:09:17 INFO     Navigator API: ============================== Pipeline 'Preprocessing' started ====================================
2023-01-27 20:09:17 INFO     Navigator API: ============================== Infer input metadata. ===============================================
2023-01-27 20:09:17 INFO     Navigator API: Command Infer input metadata. paramters: {
    "framework": "pytorch",
    "model": "<class 'triton.model.ModelWrapper'>",
    "dataloader": "{samples_count: 5, tensors_count: 10, tensors_shapes: [torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 192, 2]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 24, 1])]}",
    "batch_dim": "0",
    "_input_names": "('s_cat__0', 's_cont__1', 'k_cat__2', 'k_cont__3', 'o_cat__4', 'o_cont__5', 'target__6', 'sample_weight__7', 'id__8', 'weight__9')",
    "trt_dynamic_axes": "None",
    "dynamic_axes": "None"
}
2023-01-27 20:09:17 WARNING  Navigator API: No TRT (min, opt, max) values for axes provided. Using values derived from the dataloader: {s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],
 s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],
 k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],
 k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],
 o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],
 o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],
 target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],
 sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],
 id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],
 weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}.
2023-01-27 20:09:17 WARNING  Navigator API: No dynamic axes provided. Using values derived from the dataloader: {'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}
2023-01-27 20:09:17 INFO     Navigator API: ============================== Fetch input model data ==============================================
2023-01-27 20:09:17 INFO     Navigator API: Command Fetch input model data paramters: {
    "framework": "pytorch",
    "dataloader": "{samples_count: 5, tensors_count: 10, tensors_shapes: [torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 192, 2]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 24, 1])]}",
    "sample_count": "100",
    "batch_dim": "0",
    "seed": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:17 WARNING  Navigator API: Requested sample_count (100) is larger than the number of available samples (5). Using 5 samples.
2023-01-27 20:09:17 INFO     Navigator API: ============================== Infer output metadata. ==============================================
2023-01-27 20:09:17 INFO     Navigator API: Command Infer output metadata. paramters: {
    "framework": "pytorch",
    "model": "<class 'triton.model.ModelWrapper'>",
    "dataloader": "{samples_count: 5, tensors_count: 10, tensors_shapes: [torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 192, 2]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 192, 1]), torch.Size([512, 1]), torch.Size([512, 1]), torch.Size([512, 24, 1])]}",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "_output_names": "None",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "forward_kw_names": "None",
    "target_device": "cuda",
    "dynamic_axes": "None",
    "model_params": "<class 'NoneType'>",
    "profiling_sample": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}",
    "conversion_samples": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}"
}
2023-01-27 20:09:17 WARNING  Navigator API: No dynamic axes provided. Using values derived from the dataloader: {'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}
2023-01-27 20:09:17 INFO     Navigator API: ============================== Dump input model data ===============================================
2023-01-27 20:09:17 INFO     Navigator API: Command Dump input model data paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "profiling_sample": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}",
    "correctness_samples": "{samples_count: 5, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}",
    "conversion_samples": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}"
}
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 WARNING  Navigator API: Tensor data contains `NaN` value. Please verify the dataloader and model.
2023-01-27 20:09:17 INFO     Navigator API: ============================== Dump output model data ==============================================
2023-01-27 20:09:17 INFO     Navigator API: Command Dump output model data paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "model": "<class 'triton.model.ModelWrapper'>",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "forward_kw_names": "None",
    "target_device": "cuda",
    "model_params": "<class 'NoneType'>",
    "profiling_sample": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}",
    "correctness_samples": "{samples_count: 5, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}",
    "conversion_samples": "{samples_count: 1, tensors_count: 10, tensors_shapes: {'s_cat__0': (1, 192, 1), 's_cont__1': (1, 1), 'k_cat__2': (1, 192, 2), 'k_cont__3': (1, 192, 1), 'o_cat__4': (1, 1), 'o_cont__5': (1, 1), 'target__6': (1, 192, 1), 'sample_weight__7': (1, 1), 'id__8': (1, 1), 'weight__9': (1, 24, 1)}}"
}
2023-01-27 20:09:17 INFO     Navigator API: ============================== Pipeline 'PyTorch Export' started ===================================
2023-01-27 20:09:17 INFO     Navigator API: ============================== Export PyTorch to TorchScript script ================================
2023-01-27 20:09:17 INFO     Navigator API: Command Export PyTorch to TorchScript paramters: {
    "model_name": "navigator_model",
    "model": "<class 'triton.model.ModelWrapper'>",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:09:17 INFO     Navigator API: TorchScrip export started
2023-01-27 20:09:17 INFO     Navigator API: Command: /usr/bin/python torchscript-script/reproduce_export.py --exported_model_path torchscript-script/model.pt --target_jit_type script --batch_dim 0 --target_device cuda
2023-01-27 20:09:18 WARNING  Navigator API: External errors are usually caused by incompatibilites between the model and the target formats and/or runtimes.
2023-01-27 20:09:18 WARNING  Navigator API: Encountered an error when executing command: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 91, in execute_local_runtime_script
    fire.Fire(func, args)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/export/exporters/pytorch2torchscript.py", line 44, in export
    script_module = torch.jit.script(model)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_script.py", line 1286, in script
    return torch.jit._recursive.create_script_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 477, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 539, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 517, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 539, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 517, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 543, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py", line 394, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
Arguments for call are not valid.
The following variants are available:
  
  aten::get.str(Dict(str, t) self, str key) -> t(*)?:
  Expected at most 1 arguments but found 2 positional arguments.
  
  aten::get.default_str(Dict(str, t) self, str key, t default_value) -> t(*):
  Expected a value of type 't' for argument 'default_value' but instead found type 'NoneType'.
  
  aten::get.int(Dict(int, t) self, int key) -> t(*)?:
  Expected a value of type 'Dict[int, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.default_int(Dict(int, t) self, int key, t default_value) -> t(*):
  Expected a value of type 'Dict[int, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.bool(Dict(bool, t) self, bool key) -> t(*)?:
  Expected a value of type 'Dict[bool, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.default_bool(Dict(bool, t) self, bool key, t default_value) -> t(*):
  Expected a value of type 'Dict[bool, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.float(Dict(float, t) self, float key) -> t(*)?:
  Expected a value of type 'Dict[float, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.default_float(Dict(float, t) self, float key, t default_value) -> t(*):
  Expected a value of type 'Dict[float, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.complex(Dict(complex, t) self, complex key) -> t(*)?:
  Expected a value of type 'Dict[complex, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.default_complex(Dict(complex, t) self, complex key, t default_value) -> t(*):
  Expected a value of type 'Dict[complex, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.Tensor(Dict(Tensor, t) self, Tensor key) -> t(*)?:
  Expected a value of type 'Dict[Tensor, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.
  
  aten::get.default_Tensor(Dict(Tensor, t) self, Tensor key, t default_value) -> t(*):
  Expected a value of type 'Dict[Tensor, t]' for argument 'self' but instead found type 'Dict[str, Tensor]'.

The original call is:
  File "/workspace/models/tft_pyt/modeling.py", line 157
    def forward(self, x: Dict[str, Tensor]):
        # temporal/static categorical/continuous known/observed input 
        s_cat_inp = x.get('s_cat', None)
                    ~~~~~ <--- HERE
        s_cont_inp = x.get('s_cont', None)
        t_cat_k_inp = x.get('k_cat', None)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/core.py", line 139, in transform
    self.output = self.__call__(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/export/pyt.py", line 83, in __call__
    context.execute_local_runtime_script(
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 93, in execute_local_runtime_script
    raise UserError(f"Command to reproduce error:\n{' '.join(cmd)}") from e
model_navigator.framework_api.exceptions.UserError: Command to reproduce error:
/bin/bash torchscript-script/reproduce_export.sh

2023-01-27 20:09:18 INFO     Navigator API: You can disable error suppression for debugging with flag NAV_DEBUG=1
2023-01-27 20:09:18 INFO     Navigator API: Command logs saved in /workspace/conversion/torchscript-script/format.log
2023-01-27 20:09:18 INFO     Navigator API: ============================== Export PyTorch to TorchScript trace =================================
2023-01-27 20:09:18 INFO     Navigator API: Command Export PyTorch to TorchScript paramters: {
    "model_name": "navigator_model",
    "model": "<class 'triton.model.ModelWrapper'>",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:09:18 INFO     Navigator API: TorchScrip export started
2023-01-27 20:09:18 INFO     Navigator API: Command: /usr/bin/python torchscript-trace/reproduce_export.py --exported_model_path torchscript-trace/model.pt --target_jit_type trace --batch_dim 0 --target_device cuda
/workspace/triton/model.py:33: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not torch.isnan(t).any():
/workspace/triton/model.py:47: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  wrapped_input['id'] = id if id.numel() else None
2023-01-27 20:09:19 INFO     Navigator API: Command logs saved in /workspace/conversion/torchscript-trace/format.log
2023-01-27 20:09:19 INFO     Navigator API: ============================== Export PyTorch to ONNX ==============================================
2023-01-27 20:09:19 INFO     Navigator API: Command Export PyTorch to ONNX paramters: {
    "model_name": "navigator_model",
    "model": "<class 'triton.model.ModelWrapper'>",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "forward_kw_names": "None",
    "target_device": "cuda",
    "opset": "14",
    "verbose": "False"
}
2023-01-27 20:09:19 INFO     Navigator API: PyTorch to ONNX export started
2023-01-27 20:09:19 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_export.py --exported_model_path onnx/model.onnx --opset 14 --input_names ["s_cat__0", "s_cont__1", "k_cat__2", "k_cont__3", "o_cat__4", "o_cont__5", "target__6", "sample_weight__7", "id__8", "weight__9"] --output_names ["output__0"] --dynamic_axes {"s_cat__0": [0], "s_cont__1": [0], "k_cat__2": [0], "k_cont__3": [0], "o_cat__4": [0], "o_cont__5": [0], "target__6": [0], "sample_weight__7": [0], "id__8": [0], "weight__9": [0], "output__0": [0]} --batch_dim 0 --forward_kw_names None --target_device cuda
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input s_cat__0
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input s_cont__1
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input k_cat__2
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input k_cont__3
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input o_cat__4
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input o_cont__5
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input target__6
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input sample_weight__7
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input id__8
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input weight__9
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:2028: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input output__0
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/symbolic_opset9.py:4371: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model. 
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/onnx/_internal/jit_utils.py:258: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at /opt/pytorch/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1891.)
  _C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:687: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at /opt/pytorch/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1891.)
  _C._jit_pass_onnx_graph_shape_type_inference(
/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:1178: UserWarning: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (Triggered internally at /opt/pytorch/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp:1891.)
  _C._jit_pass_onnx_graph_shape_type_inference(
2023-01-27 20:09:20 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:09:20 INFO     Navigator API: ============================== Pipeline 'PyTorch Conversion' started ===============================
2023-01-27 20:09:20 INFO     Navigator API: ============================== Convert TorschScript to TorchTensorRT script TensorRTPrecision.FP32 =
2023-01-27 20:09:20 INFO     Navigator API: Command Convert TorschScript to TorchTensorRT paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "max_workspace_size": "10000000000",
    "precision_mode": "TensorRTPrecisionMode.HIERARCHY",
    "target_device": "cuda",
    "verbose": "False",
    "debug": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:20 INFO     Navigator API: Conversion TorchScript to TorchTensorRT started
2023-01-27 20:09:20 WARNING  Navigator API: Exported TorchScript model not found at /workspace/conversion/torchscript-script/model.pt. Skipping conversion.
2023-01-27 20:09:20 INFO     Navigator API: Command logs saved in /workspace/conversion/torch-trt-script-fp32/format.log
2023-01-27 20:09:20 INFO     Navigator API: ============================== Convert TorschScript to TorchTensorRT script TensorRTPrecision.FP16 =
2023-01-27 20:09:20 INFO     Navigator API: Command Convert TorschScript to TorchTensorRT paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "max_workspace_size": "10000000000",
    "precision_mode": "TensorRTPrecisionMode.HIERARCHY",
    "target_device": "cuda",
    "verbose": "False",
    "debug": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:20 INFO     Navigator API: Conversion TorchScript to TorchTensorRT started
2023-01-27 20:09:20 WARNING  Navigator API: Exported TorchScript model not found at /workspace/conversion/torchscript-script/model.pt. Skipping conversion.
2023-01-27 20:09:20 INFO     Navigator API: Command logs saved in /workspace/conversion/torch-trt-script-fp16/format.log
2023-01-27 20:09:20 INFO     Navigator API: ============================== Convert TorschScript to TorchTensorRT trace TensorRTPrecision.FP32 ==
2023-01-27 20:09:20 INFO     Navigator API: Command Convert TorschScript to TorchTensorRT paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "max_workspace_size": "10000000000",
    "precision_mode": "TensorRTPrecisionMode.HIERARCHY",
    "target_device": "cuda",
    "verbose": "False",
    "debug": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:20 INFO     Navigator API: Conversion TorchScript to TorchTensorRT started
2023-01-27 20:09:20 INFO     Navigator API: Command: /usr/bin/python torch-trt-trace-fp32/reproduce_conversion.py --exported_model_path torchscript-trace/model.pt --converted_model_path torch-trt-trace-fp32/model.pt --shapes '{"s_cat__0": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "s_cont__1": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "k_cat__2": {"min": (1, 192, 2), "opt": (512, 192, 2), "max": (512, 192, 2)}, "k_cont__3": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "o_cat__4": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "o_cont__5": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "target__6": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "sample_weight__7": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "id__8": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "weight__9": {"min": (1, 24, 1), "opt": (512, 24, 1), "max": (512, 24, 1)}}' --input_dtypes '["int32", "float32", "int32", "float32", "float32", "float32", "float32", "float32", "int32", "float32"]' --max_workspace_size 10000000000 --precision fp32 --precision_mode hierarchy --target_device cuda --debug False
2023-01-27 20:09:28 WARNING  Navigator API: External errors are usually caused by incompatibilites between the model and the target formats and/or runtimes.
2023-01-27 20:09:28 WARNING  Navigator API: Encountered an error when executing command: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/core.py", line 139, in transform
    self.output = self.__call__(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/convert/pyt.py", line 102, in __call__
    context.execute_external_runtime_script(ts2torchtrt.__file__, args)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 101, in execute_external_runtime_script
    self.execute_cmd(cmd)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 133, in execute_cmd
    raise UserError(
model_navigator.framework_api.exceptions.UserError: Processes exited with error code:1. Command to reproduce error:
/bin/bash torch-trt-trace-fp32/reproduce_conversion.sh

2023-01-27 20:09:28 INFO     Navigator API: You can disable error suppression for debugging with flag NAV_DEBUG=1
2023-01-27 20:09:28 INFO     Navigator API: Command logs saved in /workspace/conversion/torch-trt-trace-fp32/format.log
2023-01-27 20:09:28 INFO     Navigator API: ============================== Convert TorschScript to TorchTensorRT trace TensorRTPrecision.FP16 ==
2023-01-27 20:09:28 INFO     Navigator API: Command Convert TorschScript to TorchTensorRT paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "max_workspace_size": "10000000000",
    "precision_mode": "TensorRTPrecisionMode.HIERARCHY",
    "target_device": "cuda",
    "verbose": "False",
    "debug": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:28 INFO     Navigator API: Conversion TorchScript to TorchTensorRT started
2023-01-27 20:09:28 INFO     Navigator API: Command: /usr/bin/python torch-trt-trace-fp16/reproduce_conversion.py --exported_model_path torchscript-trace/model.pt --converted_model_path torch-trt-trace-fp16/model.pt --shapes '{"s_cat__0": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "s_cont__1": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "k_cat__2": {"min": (1, 192, 2), "opt": (512, 192, 2), "max": (512, 192, 2)}, "k_cont__3": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "o_cat__4": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "o_cont__5": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "target__6": {"min": (1, 192, 1), "opt": (512, 192, 1), "max": (512, 192, 1)}, "sample_weight__7": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "id__8": {"min": (1, 1), "opt": (512, 1), "max": (512, 1)}, "weight__9": {"min": (1, 24, 1), "opt": (512, 24, 1), "max": (512, 24, 1)}}' --input_dtypes '["int32", "float32", "int32", "float32", "float32", "float32", "float32", "float32", "int32", "float32"]' --max_workspace_size 10000000000 --precision fp16 --precision_mode hierarchy --target_device cuda --debug False
2023-01-27 20:09:36 WARNING  Navigator API: External errors are usually caused by incompatibilites between the model and the target formats and/or runtimes.
2023-01-27 20:09:36 WARNING  Navigator API: Encountered an error when executing command: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/core.py", line 139, in transform
    self.output = self.__call__(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/convert/pyt.py", line 102, in __call__
    context.execute_external_runtime_script(ts2torchtrt.__file__, args)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 101, in execute_external_runtime_script
    self.execute_cmd(cmd)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 133, in execute_cmd
    raise UserError(
model_navigator.framework_api.exceptions.UserError: Processes exited with error code:1. Command to reproduce error:
/bin/bash torch-trt-trace-fp16/reproduce_conversion.sh

2023-01-27 20:09:36 INFO     Navigator API: You can disable error suppression for debugging with flag NAV_DEBUG=1
2023-01-27 20:09:36 INFO     Navigator API: Command logs saved in /workspace/conversion/torch-trt-trace-fp16/format.log
2023-01-27 20:09:36 INFO     Navigator API: ============================== Convert ONNX to TensorRT TensorRTPrecision.FP32 =====================
2023-01-27 20:09:36 INFO     Navigator API: Command Convert ONNX to TensorRT paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "max_workspace_size": "10000000000",
    "target_device": "cuda",
    "verbose": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:09:36 INFO     Navigator API: ONNX to TRT conversion started
2023-01-27 20:09:36.935111647 [W:onnxruntime:, session_state.cc:1030 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-01-27 20:09:36.935151153 [W:onnxruntime:, session_state.cc:1032 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2023-01-27 20:09:36 INFO     Navigator API: Command: polygraphy convert onnx/model.onnx --convert-to trt -o trt-fp32/model.plan --tf32 --trt-min-shapes s_cat__0:[1,192,1] k_cat__2:[1,192,2] k_cont__3:[1,192,1] target__6:[1,192,1] --trt-opt-shapes s_cat__0:[512,192,1] k_cat__2:[512,192,2] k_cont__3:[512,192,1] target__6:[512,192,1] --trt-max-shapes s_cat__0:[512,192,1] k_cat__2:[512,192,2] k_cont__3:[512,192,1] target__6:[512,192,1] --pool-limit workspace:10000000000

[38;5;14m[I] Creating ONNX-Runtime Inference Session with providers: ['CUDAExecutionProvider'][0m

2023-01-27 20:11:35 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:11:35 INFO     Navigator API: ============================== Convert ONNX to TensorRT TensorRTPrecision.FP16 =====================
2023-01-27 20:11:35 INFO     Navigator API: Command Convert ONNX to TensorRT paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "max_workspace_size": "10000000000",
    "target_device": "cuda",
    "verbose": "False",
    "trt_profile": "{s_cat__0 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n s_cont__1 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n k_cat__2 [min=(1, 192, 2), opt=(512, 192, 2), max=(512, 192, 2)],\n k_cont__3 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n o_cat__4 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n o_cont__5 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n target__6 [min=(1, 192, 1), opt=(512, 192, 1), max=(512, 192, 1)],\n sample_weight__7 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n id__8 [min=(1, 1), opt=(512, 1), max=(512, 1)],\n weight__9 [min=(1, 24, 1), opt=(512, 24, 1), max=(512, 24, 1)]}"
}
2023-01-27 20:11:35 INFO     Navigator API: ONNX to TRT conversion started
2023-01-27 20:11:35.508307508 [W:onnxruntime:, session_state.cc:1030 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-01-27 20:11:35.508339806 [W:onnxruntime:, session_state.cc:1032 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2023-01-27 20:11:35 INFO     Navigator API: Command: polygraphy convert onnx/model.onnx --convert-to trt -o trt-fp16/model.plan --tf32 --fp16 --trt-min-shapes s_cat__0:[1,192,1] k_cat__2:[1,192,2] k_cont__3:[1,192,1] target__6:[1,192,1] --trt-opt-shapes s_cat__0:[512,192,1] k_cat__2:[512,192,2] k_cont__3:[512,192,1] target__6:[512,192,1] --trt-max-shapes s_cat__0:[512,192,1] k_cat__2:[512,192,2] k_cont__3:[512,192,1] target__6:[512,192,1] --pool-limit workspace:10000000000

[38;5;14m[I] Creating ONNX-Runtime Inference Session with providers: ['CUDAExecutionProvider'][0m

2023-01-27 20:17:01 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:17:01 INFO     Navigator API: ============================== Pipeline 'Correctness' started ======================================
2023-01-27 20:17:01 INFO     Navigator API: ============================== Correctness torchscript trace PyTorchExecutionProvider ==============
2023-01-27 20:17:01 INFO     Navigator API: Command Correctness torchscript paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:01 INFO     Navigator API: Correctness test for: Format.TORCHSCRIPT PyTorchExecutionProvider started.
2023-01-27 20:17:01 INFO     Navigator API: Command: /usr/bin/python torchscript-trace/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpqj2ed1xw --format torchscript --precision None --jit_type trace --runtime PyTorchExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:17:09 INFO     Navigator API: Command logs saved in /workspace/conversion/torchscript-trace/format.log
2023-01-27 20:17:09 INFO     Navigator API: ============================== Correctness onnx CUDAExecutionProvider ==============================
2023-01-27 20:17:09 INFO     Navigator API: Command Correctness onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:09 INFO     Navigator API: Correctness test for: Format.ONNX CUDAExecutionProvider started.
2023-01-27 20:17:09 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpwmjlneso --format onnx --precision None --jit_type None --runtime CUDAExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:17:11 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:17:11 INFO     Navigator API: ============================== Correctness onnx CPUExecutionProvider ===============================
2023-01-27 20:17:11 INFO     Navigator API: Command Correctness onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:11 INFO     Navigator API: Correctness test for: Format.ONNX CPUExecutionProvider started.
2023-01-27 20:17:11 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmp_5c3qdgp --format onnx --precision None --jit_type None --runtime CPUExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:17:14 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:17:14 INFO     Navigator API: ============================== Correctness onnx TensorrtExecutionProvider ==========================
2023-01-27 20:17:14 INFO     Navigator API: Command Correctness onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:14 INFO     Navigator API: Correctness test for: Format.ONNX TensorrtExecutionProvider started.
2023-01-27 20:17:14 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpitfgcouu --format onnx --precision None --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:17:36 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:17:36 INFO     Navigator API: ============================== Correctness trt TensorRTPrecision.FP32 TensorrtExecutionProvider ====
2023-01-27 20:17:36 INFO     Navigator API: Command Correctness trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:36 INFO     Navigator API: Correctness test for: Format.TENSORRT TensorrtExecutionProvider started.
2023-01-27 20:17:36 INFO     Navigator API: Command: /usr/bin/python trt-fp32/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpeoj4eokq --format trt --precision fp32 --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:17:42 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:17:42 INFO     Navigator API: ============================== Correctness trt TensorRTPrecision.FP32 TrtexecExecutionProvider =====
2023-01-27 20:17:42 INFO     Navigator API: Command Correctness trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:17:42 INFO     Navigator API: Correctness test for: Format.TENSORRT TrtexecExecutionProvider started.
2023-01-27 20:17:42 INFO     Navigator API: Command: /usr/bin/python trt-fp32/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpwd89fi2f --format trt --precision fp32 --jit_type None --runtime TrtexecExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:18:23 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:18:23 INFO     Navigator API: ============================== Correctness trt TensorRTPrecision.FP16 TensorrtExecutionProvider ====
2023-01-27 20:18:23 INFO     Navigator API: Command Correctness trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:18:23 INFO     Navigator API: Correctness test for: Format.TENSORRT TensorrtExecutionProvider started.
2023-01-27 20:18:23 INFO     Navigator API: Command: /usr/bin/python trt-fp16/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpe_vv09tc --format trt --precision fp16 --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:18:30 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:18:30 INFO     Navigator API: ============================== Correctness trt TensorRTPrecision.FP16 TrtexecExecutionProvider =====
2023-01-27 20:18:30 INFO     Navigator API: Command Correctness trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "target_device": "cuda",
    "atol": "1000.0",
    "rtol": "1000.0",
    "verbose": "False"
}
2023-01-27 20:18:30 INFO     Navigator API: Correctness test for: Format.TENSORRT TrtexecExecutionProvider started.
2023-01-27 20:18:30 INFO     Navigator API: Command: /usr/bin/python trt-fp16/reproduce_correctness.py --model_name navigator_model --output_names '["output__0"]' --batch_dim 0 --results_path /tmp/tmpp_cnsl71 --format trt --precision fp16 --jit_type None --runtime TrtexecExecutionProvider --enable_xla None --jit_compile None --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:19:12 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:19:12 INFO     Navigator API: ============================== Pipeline 'Generate Configurations' started ==========================
2023-01-27 20:19:12 INFO     Navigator API: ============================== Generate configurations for torchscript trace =======================
2023-01-27 20:19:12 INFO     Navigator API: Command Generate configurations for torchscript paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "opset": "14"
}
2023-01-27 20:19:12 INFO     Navigator API: Command logs saved in /workspace/conversion/torchscript-trace/format.log
2023-01-27 20:19:12 INFO     Navigator API: ============================== Generate configurations for onnx ====================================
2023-01-27 20:19:12 INFO     Navigator API: Command Generate configurations for onnx paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "opset": "14"
}
2023-01-27 20:19:12 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:19:12 INFO     Navigator API: ============================== Generate configurations for trt TensorRTPrecision.FP32 ==============
2023-01-27 20:19:12 INFO     Navigator API: Command Generate configurations for trt paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "opset": "14"
}
2023-01-27 20:19:12 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:19:12 INFO     Navigator API: ============================== Generate configurations for trt TensorRTPrecision.FP16 ==============
2023-01-27 20:19:12 INFO     Navigator API: Command Generate configurations for trt paramters: {
    "framework": "pytorch",
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "opset": "14"
}
2023-01-27 20:19:12 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:19:12 INFO     Navigator API: ============================== Pipeline 'Profiling' started ========================================
2023-01-27 20:19:12 INFO     Navigator API: ============================== Load metadata. ======================================================
2023-01-27 20:19:12 INFO     Navigator API: Command Load metadata. paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion"
}
2023-01-27 20:19:12 INFO     Navigator API: ============================== Load samples. =======================================================
2023-01-27 20:19:12 INFO     Navigator API: Command Load samples. paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0"
}
2023-01-27 20:19:12 INFO     Navigator API: ============================== Performance torchscript trace PyTorchExecutionProvider ==============
2023-01-27 20:19:12 INFO     Navigator API: Command Performance torchscript paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:19:12 INFO     Navigator API: Performance test for: Format.TORCHSCRIPT PyTorchExecutionProvider started.
2023-01-27 20:19:12 INFO     Navigator API: Command: /usr/bin/python torchscript-trace/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmp32xwq4vl --format torchscript --precision None --jit_type trace --runtime PyTorchExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:19:30 INFO     Navigator API: Command logs saved in /workspace/conversion/torchscript-trace/format.log
2023-01-27 20:19:30 INFO     Navigator API: ============================== Performance onnx CUDAExecutionProvider ==============================
2023-01-27 20:19:30 INFO     Navigator API: Command Performance onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:19:30 INFO     Navigator API: Performance test for: Format.ONNX CUDAExecutionProvider started.
2023-01-27 20:19:30 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmp5dl5fb8z --format onnx --precision None --jit_type None --runtime CUDAExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:19:36 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:19:36 INFO     Navigator API: ============================== Performance onnx CPUExecutionProvider ===============================
2023-01-27 20:19:36 INFO     Navigator API: Command Performance onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:19:36 INFO     Navigator API: Performance test for: Format.ONNX CPUExecutionProvider started.
2023-01-27 20:19:36 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmpp3isbtcy --format onnx --precision None --jit_type None --runtime CPUExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:21:10 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:21:10 INFO     Navigator API: ============================== Performance onnx TensorrtExecutionProvider ==========================
2023-01-27 20:21:10 INFO     Navigator API: Command Performance onnx paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:21:10 INFO     Navigator API: Performance test for: Format.ONNX TensorrtExecutionProvider started.
2023-01-27 20:21:10 INFO     Navigator API: Command: /usr/bin/python onnx/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmpvd2psbvh --format onnx --precision None --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:21:56 WARNING  Navigator API: External errors are usually caused by incompatibilites between the model and the target formats and/or runtimes.
2023-01-27 20:21:56 WARNING  Navigator API: Encountered an error when executing command: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/core.py", line 139, in transform
    self.output = self.__call__(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/commands/performance/performance.py", line 354, in __call__
    context.execute_external_runtime_script(performance_script.__file__, args)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 101, in execute_external_runtime_script
    self.execute_cmd(cmd)
  File "/usr/local/lib/python3.8/dist-packages/model_navigator/framework_api/execution_context.py", line 133, in execute_cmd
    raise UserError(
model_navigator.framework_api.exceptions.UserError: Processes exited with error code:139. Command to reproduce error:
/bin/bash onnx/reproduce_profiling.sh

2023-01-27 20:21:56 INFO     Navigator API: You can disable error suppression for debugging with flag NAV_DEBUG=1
2023-01-27 20:21:56 INFO     Navigator API: Command logs saved in /workspace/conversion/onnx/format.log
2023-01-27 20:21:56 INFO     Navigator API: ============================== Performance trt TensorRTPrecision.FP32 TensorrtExecutionProvider ====
2023-01-27 20:21:56 INFO     Navigator API: Command Performance trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:21:56 INFO     Navigator API: Performance test for: Format.TENSORRT TensorrtExecutionProvider started.
2023-01-27 20:21:56 INFO     Navigator API: Command: /usr/bin/python trt-fp32/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmp5_frxplo --format trt --precision fp32 --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:22:06 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:22:06 INFO     Navigator API: ============================== Performance trt TensorRTPrecision.FP32 TrtexecExecutionProvider =====
2023-01-27 20:22:06 INFO     Navigator API: Command Performance trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:22:06 INFO     Navigator API: Performance test for: Format.TENSORRT TrtexecExecutionProvider started.
2023-01-27 20:22:06 INFO     Navigator API: Command: /usr/bin/python trt-fp32/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmp4vdjx2w0 --format trt --precision fp32 --jit_type None --runtime TrtexecExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:22:24 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp32/format.log
2023-01-27 20:22:24 INFO     Navigator API: ============================== Performance trt TensorRTPrecision.FP16 TensorrtExecutionProvider ====
2023-01-27 20:22:24 INFO     Navigator API: Command Performance trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:22:24 INFO     Navigator API: Performance test for: Format.TENSORRT TensorrtExecutionProvider started.
2023-01-27 20:22:24 INFO     Navigator API: Command: /usr/bin/python trt-fp16/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmpvmkp2e0y --format trt --precision fp16 --jit_type None --runtime TensorrtExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:22:34 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:22:34 INFO     Navigator API: ============================== Performance trt TensorRTPrecision.FP16 TrtexecExecutionProvider =====
2023-01-27 20:22:34 INFO     Navigator API: Command Performance trt paramters: {
    "model_name": "navigator_model",
    "workdir": "/workspace/conversion",
    "batch_dim": "0",
    "max_batch_size": "512",
    "input_metadata": "{'s_cat__0': TensorSpec(name='s_cat__0', shape=(-1, 192, 1), dtype=dtype('int64'), optional=False), 's_cont__1': TensorSpec(name='s_cont__1', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'k_cat__2': TensorSpec(name='k_cat__2', shape=(-1, 192, 2), dtype=dtype('int64'), optional=False), 'k_cont__3': TensorSpec(name='k_cont__3', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'o_cat__4': TensorSpec(name='o_cat__4', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'o_cont__5': TensorSpec(name='o_cont__5', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'target__6': TensorSpec(name='target__6', shape=(-1, 192, 1), dtype=dtype('float32'), optional=False), 'sample_weight__7': TensorSpec(name='sample_weight__7', shape=(-1, 1), dtype=dtype('float64'), optional=False), 'id__8': TensorSpec(name='id__8', shape=(-1, 1), dtype=dtype('int64'), optional=False), 'weight__9': TensorSpec(name='weight__9', shape=(-1, 24, 1), dtype=dtype('float32'), optional=False)}",
    "output_metadata": "{'output__0': TensorSpec(name='output__0', shape=(-1, 24, 3), dtype=dtype('float32'), optional=False)}",
    "profiler_config": "ProfilerConfig(batch_sizes=None, measurement_interval=5000, measurement_mode=<MeasurementMode.COUNT_WINDOWS: 'count_windows'>, measurement_request_count=50, stability_percentage=10.0, max_trials=10)",
    "target_device": "cuda",
    "verbose": "False"
}
2023-01-27 20:22:34 INFO     Navigator API: Performance test for: Format.TENSORRT TrtexecExecutionProvider started.
2023-01-27 20:22:34 INFO     Navigator API: Command: /usr/bin/python trt-fp16/reproduce_profiling.py --model_name navigator_model --batch_dim 0 --results_path /tmp/tmphtpi7ysx --format trt --precision fp16 --jit_type None --runtime TrtexecExecutionProvider --enable_xla None --jit_compile None --profiler_config '{"batch_sizes": None, "measurement_interval": 5000, "measurement_mode": "count_windows", "measurement_request_count": 50, "stability_percentage": 10.0, "max_trials": 10}' --max_batch_size 512 --runner_manager_dict '{"input_metadata": [{"name": "s_cat__0", "shape": (-1, 192, 1), "dtype": "int64"}, {"name": "s_cont__1", "shape": (-1, 1), "dtype": "float64"}, {"name": "k_cat__2", "shape": (-1, 192, 2), "dtype": "int64"}, {"name": "k_cont__3", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "o_cat__4", "shape": (-1, 1), "dtype": "float64"}, {"name": "o_cont__5", "shape": (-1, 1), "dtype": "float64"}, {"name": "target__6", "shape": (-1, 192, 1), "dtype": "float32"}, {"name": "sample_weight__7", "shape": (-1, 1), "dtype": "float64"}, {"name": "id__8", "shape": (-1, 1), "dtype": "int64"}, {"name": "weight__9", "shape": (-1, 24, 1), "dtype": "float32"}], "output_metadata": [{"name": "output__0", "shape": (-1, 24, 3), "dtype": "float32"}], "target_device": "cuda"}'
2023-01-27 20:22:53 INFO     Navigator API: Command logs saved in /workspace/conversion/trt-fp16/format.log
2023-01-27 20:22:53 INFO     Navigator API: ============================== Model Navigator summary =============================================
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'Preprocessing' summary ====================================
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Infer input metadata.
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Fetch input model data
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Infer output metadata.
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Dump input model data
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Dump output model data
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'PyTorch Export' summary ===================================
2023-01-27 20:22:53 WARNING  Navigator API: [FAIL] Export PyTorch to TorchScript script
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Export PyTorch to TorchScript trace
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Export PyTorch to ONNX
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'PyTorch Conversion' summary ===============================
2023-01-27 20:22:53 WARNING  Navigator API: [SKIPPED] Convert TorschScript to TorchTensorRT script TensorRTPrecision.FP32
2023-01-27 20:22:53 WARNING  Navigator API: [SKIPPED] Convert TorschScript to TorchTensorRT script TensorRTPrecision.FP16
2023-01-27 20:22:53 WARNING  Navigator API: [FAIL] Convert TorschScript to TorchTensorRT trace TensorRTPrecision.FP32
2023-01-27 20:22:53 WARNING  Navigator API: [FAIL] Convert TorschScript to TorchTensorRT trace TensorRTPrecision.FP16
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Convert ONNX to TensorRT TensorRTPrecision.FP32
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Convert ONNX to TensorRT TensorRTPrecision.FP16
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'Correctness' summary ======================================
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness torchscript trace PyTorchExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness onnx CUDAExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness onnx CPUExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness onnx TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness trt TensorRTPrecision.FP32 TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness trt TensorRTPrecision.FP32 TrtexecExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness trt TensorRTPrecision.FP16 TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Correctness trt TensorRTPrecision.FP16 TrtexecExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'Generate Configurations' summary ==========================
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Generate configurations for torchscript trace
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Generate configurations for onnx
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Generate configurations for trt TensorRTPrecision.FP32
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Generate configurations for trt TensorRTPrecision.FP16
2023-01-27 20:22:53 INFO     Navigator API: ============================== Pipeline 'Profiling' summary ========================================
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Load metadata.
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Load samples.
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance torchscript trace PyTorchExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance onnx CUDAExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance onnx CPUExecutionProvider
2023-01-27 20:22:53 WARNING  Navigator API: [FAIL] Performance onnx TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance trt TensorRTPrecision.FP32 TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance trt TensorRTPrecision.FP32 TrtexecExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance trt TensorRTPrecision.FP16 TensorrtExecutionProvider
2023-01-27 20:22:53 INFO     Navigator API: [ OK ] Performance trt TensorRTPrecision.FP16 TrtexecExecutionProvider
2023-01-27 20:22:53 WARNING  Navigator API: Initially models are not verified. Validate exported models and use PackageDescriptor.set_verified(format, runtime, jit_type, precision) method to set models as verified.

In [14]:

package_descriptor.get_formats_status()

Out[14]:

{'torchscript-script': {'PyTorchExecutionProvider': <Status.FAIL: 'FAIL'>},
 'torchscript-trace': {'PyTorchExecutionProvider': <Status.OK: 'OK'>},
 'onnx': {'CUDAExecutionProvider': <Status.OK: 'OK'>,
  'CPUExecutionProvider': <Status.OK: 'OK'>,
  'TensorrtExecutionProvider': <Status.FAIL: 'FAIL'>},
 'torch-trt-script-fp32': {'PyTorchExecutionProvider': <Status.SKIPPED: 'SKIPPED'>},
 'torch-trt-script-fp16': {'PyTorchExecutionProvider': <Status.SKIPPED: 'SKIPPED'>},
 'torch-trt-trace-fp32': {'PyTorchExecutionProvider': <Status.FAIL: 'FAIL'>},
 'torch-trt-trace-fp16': {'PyTorchExecutionProvider': <Status.FAIL: 'FAIL'>},
 'trt-fp32': {'TensorrtExecutionProvider': <Status.OK: 'OK'>,
  'TrtexecExecutionProvider': <Status.OK: 'OK'>},
 'trt-fp16': {'TensorrtExecutionProvider': <Status.OK: 'OK'>,
  'TrtexecExecutionProvider': <Status.OK: 'OK'>}}

Inference

Post export we can see which formats worked and which formats did not. We have verifed that the NVIDIA TensorRT format works with the TFT model, so below we can boot up a local instance of the NVIDIA TensorRT fp16 model and run a forward pass. We can then calculate the elementwise error of the output. Note that we do expect some error between the PyTorch fp32 model compared to the NVIDIA TensorRT fp16 model. fp16 computation has inherently less information leading to slight errors accumulating as the input passes through the layers.

In [15]:

import numpy as np
from model_navigator.converter.config import TensorRTPrecision
feed_dict = {k: v.detach().cpu().numpy() for k, v in zip(data_names['inputs'], raw_data)}


# Compare in framework model and exported model outputs
output_a = [model(*raw_data).detach().cpu().numpy()]
trt_runner = package_descriptor.get_runner(format=nav.Format.TENSORRT, precision=TensorRTPrecision.FP16)
with trt_runner:
    output_b = trt_runner.infer(feed_dict)

[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan

In [16]:

error = np.absolute(output_a - output_b['output__0'])

In [17]:

error.mean(), error.std()

Out[17]:

(0.0009011049, 0.0008147036)

Benchmark

Now we can benchmark the throughput and latency of the successful conversion formats using the NVIDIA Triton Model Navigator provided scripts. To learn more about profiling inside NVIDIA Triton Model Navigator visit the link here. To learn more about reproducing results from the package visit the link here.

In [18]:

!cd {output_path} && bash torchscript-trace/reproduce_profiling.sh

[32m2023-01-27 20:23:00[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling for pytorch-runner-N0-01/27/23-20:23:00 and batch size: 1 started
[32m2023-01-27 20:23:06[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling result for pytorch-runner-N0-01/27/23-20:23:00 and batch size: 1:
Batch: 1
Request count: 50
Throughput: 229.7775 [infer/sec]
Avg Latency: 4.3521 [ms]
Std Latency: 0.0643 [ms]
p50 Latency: 4.3351 [ms]
p90 Latency: 4.3899 [ms]
p95 Latency: 4.4178 [ms]
p99 Latency: 4.5939 [ms]
[32m2023-01-27 20:23:06[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling for pytorch-runner-N0-01/27/23-20:23:00 and batch size: 512 started
[32m2023-01-27 20:23:15[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling result for pytorch-runner-N0-01/27/23-20:23:00 and batch size: 512:
Batch: 512
Request count: 50
Throughput: 33427.4155 [infer/sec]
Avg Latency: 15.3168 [ms]
Std Latency: 0.0824 [ms]
p50 Latency: 15.2936 [ms]
p90 Latency: 15.4192 [ms]
p95 Latency: 15.4895 [ms]
p99 Latency: 15.5898 [ms]

In [19]:

!cd {output_path} && bash trt-fp16/reproduce_profiling.sh

Warning: Accessing the `severity` property of G_LOGGER is deprecated and will be removed in v0.45.0. Use `module_severity` instead
[32m2023-01-27 20:23:18[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling for trtexec-runner-N0-01/27/23-20:23:18 and batch size: 1 started
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] The trtexec command being run: ['trtexec', '--loadEngine=/workspace/conversion/trt-fp16/model.plan', '--exportOutput=/tmp/tmp9pd4lew3', '--useCudaGraph', '--shapes=s_cat__0:1x192x1,k_cat__2:1x192x2,k_cont__3:1x192x1,target__6:1x192x1', '--loadInputs=s_cat__0:/tmp/tmphopxh3q8,k_cat__2:/tmp/tmp3ubyhvt7,k_cont__3:/tmp/tmponjfhfyo,target__6:/tmp/tmph8x7_kbg']
[32m2023-01-27 20:23:26[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling result for trtexec-runner-N0-01/27/23-20:23:18 and batch size: 1:
Batch: 1
Request count: 2869
Throughput: 939.4170 [infer/sec]
Avg Latency: 1.0645 [ms]
Std Latency: 0.0000 [ms]
p50 Latency: 1.0644 [ms]
p90 Latency: 0.0000 [ms]
p95 Latency: 0.0000 [ms]
p99 Latency: 1.0684 [ms]
[32m2023-01-27 20:23:26[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling for trtexec-runner-N0-01/27/23-20:23:18 and batch size: 512 started
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] Loading bytes from /workspace/conversion/trt-fp16/model.plan
[I] The trtexec command being run: ['trtexec', '--loadEngine=/workspace/conversion/trt-fp16/model.plan', '--exportOutput=/tmp/tmp9pd4lew3', '--useCudaGraph', '--shapes=s_cat__0:512x192x1,k_cat__2:512x192x2,k_cont__3:512x192x1,target__6:512x192x1', '--loadInputs=s_cat__0:/tmp/tmpzj1g07cf,k_cat__2:/tmp/tmpwnbyyizo,k_cont__3:/tmp/tmp62bz_7it,target__6:/tmp/tmpvsie41h7']
[01/27/2023-20:23:34] [W] * GPU compute time is unstable, with coefficient of variance = 3.53653%.
[01/27/2023-20:23:34] [W]   If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.
[32m2023-01-27 20:23:34[0m [1;34mINFO    [0m [34mNavigator API:[0m Performance profiling result for trtexec-runner-N0-01/27/23-20:23:18 and batch size: 512:
Batch: 512
Request count: 291
Throughput: 48363.0250 [infer/sec]
Avg Latency: 10.5866 [ms]
Std Latency: 0.0000 [ms]
p50 Latency: 10.5132 [ms]
p90 Latency: 0.0000 [ms]
p95 Latency: 0.0000 [ms]
p99 Latency: 12.5056 [ms]

In [ ]: