NVIDIA

Fine-Tuning COVID-19 CT Scan Classifier

Resource

NVIDIA

Fine-Tuning COVID-19 CT Scan Classifier

Example Jupyter Notebook Sample Code to Fine-Tune Clara Train COVID-19 CT Scan Classification Pretrained Model with Custom Dataset

Back to File Browser

CovidCT-ScanClassifier.ipynb

Fine-Tuning a COVID CT-Scan Classification Model

NVIDIA Clara is a healthcare application framework for AI-powered imaging, genomics, and the development and deployment of smart sensors. NVIDIA Clara Imaging is a subset of these libraries that focus on medical imaging.

This Notebook uses the Clara Train SDK, NVIDIA’s domain optimized application framework that accelerates deep learning training and inference for medical imaging use cases, to fine tune a previously trained classifier to be able to classify data with different characteristics to the ones present in the data the model was trained on.

Clara Train is a framework that includes two main libraries; AI-Assisted Annotation (AIAA), which enables medical viewers to rapidly create annotated datasets suitable for training, and a Training Framework, a TensorFlow based framework to kick start AI development with techniques like transfer learning, federated learning and AutoML. Clara Train utilizes a concept called MMAR (Medical Model ARchive) that describes a model, configuration, transforms, and data associated with the model.

The Clara CT-Scan Covid Classifier is used as a base model in this Jupyter notebook. This model is developed by NVIDIA researchers in conjunction with the NIH. It was trained and evaluated on a global dataset with thousands of experimental cohorts collected from across the globe. The model achieved an accuracy of greater than 90% on a test set consisting of more than one thousand CT images collected across the globe. The model requires two inputs, a CT scan image and a lung segmentation image, to guide the model to focus on the lung area. A CT scan, or computed tomography scan, is a 3D medical imaging procedure that uses computer-processed combinations of many X-ray measurements taken from different angles to produce cross-sectional (tomographic) images, also known as slices. Before training, the raw data is preprocesed to be in Hounsfield units and to be in a specific orientation.

The intent of this notebook is to showcase the features of the CLARA Train SDK, not to produce a research quality model.

The Clara CT-Scan Covid Classifier is for Research Use Only. The recommendations produced by the software should not be solely or primarily relied upon to diagnose or treat COVID-19 by a Healthcare Professional. This research use only software has not been cleared or approved by FDA or any regulatory agency.

In this Jupyter Notebook, we will perform several steps.

Markdown Monster icon

The first step is to set up the environment, inspect the base model and the data. The data is downloaded through obtaining the Kaggle Key and using it to download the Kaggle data used in this application. Some of this data will be held back from being used in training, so that it can instead be used for testing the application.

Note the application set up assumes that the application and its supporting materials/requirements were uploaded to an Azure Compute Cluster through the use of the NGC-AzureML Quick Launch Toolkit, as described here.

This notebook can be used without using the NGC-AzureML Quick Launch Toolkit but the following steps would need to be done manually

Access to a GPU-supported Compute Cluster/VM were to run this notebook
The Clara Train SDK image has to be running on the compute resource
The Clara CT-Scan Covid Classifier has to be preloaded
The new Data has to be preloaded

Step 1: Set Up

1.1 Medical Model ARchive

Clara manages models using the MMAR (Medical Model ARchive) format, which is a directory structure that contains the configuration, commands and the base model. NGC provides many models to get started, including a pre-trained Clara CT-Scan Covid Classifier which is used as a starting point for this notebook. It is assumed that the model was uploaded by the NGC-AzureML Quick Launch Toolkit into the workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/ folder of the Compute Cluster

An NVIDIA Clara MMAR, has six main subfolders:

Folder	Description
commands	Where all the scripts are stored
config	Points to files where the user tells the system which data is to be used, what parameters are to be used, etc
docs	Contains the license and a reference to more information about the model
resources	Where the log files are stored
models	The default location where the trained models are stored
docs	The default location where inference results are stored

Please Navigate through the MMAR structure and familiarize yourself with it

We will be using as base model the Clara CT-Scan Covid Classifier and so all the files on the MMAR refer to the data and the configurations used to train that model. Through this application we would use simple functions contained on file TransformInputData.py to adapt the original files to refer to the new data and context

1.2 Setting Datasets

We will use this classifier to classify data that has not been preprocessed, unlike in the case of the data the model was trained on.

For this example labelled data from two sources are used:

CT-scans from COVID patients on the COVID-19 CT scans Kaggle Database
CT-scans from non-COVID CT-scans from The Cancer Imaging Archive.

NGC hosts many public datasets that can be uploaded to a Compute Cluster using the NGC-AzureML Quick Launch Toolkit, which is the case for the dataset from The Cancer Imaging Archive. For the Kaggle dataset, the data needs to be loaded programatically.

Obtaining and Uploading Kaggle Credentials/Key

To download data programmatically from Kaggle an API is provided that requires the Users credentials, in the form of username and a key that needs to be generated at the Kaggle site

The user needs to:

Create a Kaggle profile (https://www.kaggle.com/)
Navigate to its profile home page
Click on its profile icon (a duck by default)
Choose “My Account” on the dropdown list.
Once in the “My Account” page, the user should click the “Create New API Token” that would download a file called “kaggle.json” in the users “Downloads” folder.

Markdown Monster icon

This file, “kaggle.json”, should then be uploaded manually into the workspaceblobstore/clara/ folder in the Compute Cluster by using the upload icon (squared in Red) on the Jupyter Lab:

Markdown Monster icon

Loading Libraries

In [1]:

import TransformInputData as tid
import nibabel as nib
import matplotlib.pyplot as plt
import os
import numpy as np
from sklearn.metrics import average_precision_score

The TransformInputData library contains a series of simple functions that transform the input data into the format expected by the Clara Train SDK. It also has functions that adapt the Clara MMAR configuration files from the base classifier to the desired specs for the new classifier, including labels, learning rate, and number of epochs.

Installing the Kaggle API

In [2]:

!pip install kaggle

Collecting kaggle
  Downloading kaggle-1.5.6.tar.gz (58 kB)
[K     |████████████████████████████████| 58 kB 3.7 MB/s eta 0:00:011
[?25hCollecting urllib3<1.25,>=1.21.1
  Downloading urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
[K     |████████████████████████████████| 118 kB 14.1 MB/s eta 0:00:01
[?25hRequirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from kaggle) (1.12.0)
Requirement already satisfied: certifi in /usr/local/lib/python3.6/dist-packages (from kaggle) (2020.4.5.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.8.0)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.23.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from kaggle) (4.36.1)
Collecting python-slugify
  Downloading python-slugify-4.0.1.tar.gz (11 kB)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->kaggle) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->kaggle) (3.0.4)
Collecting text-unidecode>=1.3
  Downloading text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
[K     |████████████████████████████████| 78 kB 23.2 MB/s eta 0:00:01
[?25hBuilding wheels for collected packages: kaggle, python-slugify
  Building wheel for kaggle (setup.py) ... [?25ldone
[?25h  Created wheel for kaggle: filename=kaggle-1.5.6-py3-none-any.whl size=72857 sha256=7db58f6c4516cb398bf43e1013a213a21a4ff792477bf50c7ebec49e0550f82f
  Stored in directory: /root/.cache/pip/wheels/01/3e/ff/77407ebac3ef71a79b9166a8382aecf88415a0bcbe3c095a01
  Building wheel for python-slugify (setup.py) ... [?25ldone
[?25h  Created wheel for python-slugify: filename=python_slugify-4.0.1-py2.py3-none-any.whl size=6768 sha256=6ebb6c367fb3858adbc102624374a4a6ea60ffc031605540d6b442008302a3aa
  Stored in directory: /root/.cache/pip/wheels/72/e6/db/122611605e60148f54ee2abaca98b2bbeafc6e22486a867bad
Successfully built kaggle python-slugify
Installing collected packages: urllib3, text-unidecode, python-slugify, kaggle
  Attempting uninstall: urllib3
    Found existing installation: urllib3 1.25.8
    Uninstalling urllib3-1.25.8:
      Successfully uninstalled urllib3-1.25.8
Successfully installed kaggle-1.5.6 python-slugify-4.0.1 text-unidecode-1.3 urllib3-1.24.3
[33mWARNING: You are using pip version 20.0.2; however, version 20.2.2 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.[0m

Downloading the zip file

In [3]:

if not os.path.exists('/root/.kaggle/'):
    !mkdir /root/.kaggle/
!cp kaggle.json /root/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
!kaggle datasets download -d "andrewmvd/covid19-ct-scans"

Downloading covid19-ct-scans.zip to /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara
100%|██████████████████████████████████████▊| 1.03G/1.03G [00:14<00:00, 106MB/s]
100%|██████████████████████████████████████| 1.03G/1.03G [00:16<00:00, 67.2MB/s]

Unzipping the file into the data folder

In [4]:

tid.unzippedfile('experiments/kaggleset/','covid19-ct-scans.zip')

1.3 Inspect the data

A CT scan, or computed tomography scan, is a 3D medical imaging procedure that uses computer-processed combinations of many X-ray measurements taken from different angles to produce cross-sectional (tomographic) images.

The following line instructs the Notebook where to find the data:

In [5]:

kaggledata_dir='experiments/kaggleset/'

These functions help to visualize the data:

In [6]:

def get_ct_scan_data(filename):
    ct_scan_file = '{0}ct_scans/{1}'.format(kaggledata_dir,filename)
    lung_mask_file = '{0}lung_mask/{1}'.format(kaggledata_dir,filename)
    lung_mask_file  = lung_mask_file.replace('org_covid-19-pneumonia-','').replace('org_','').replace('-dcm','')
    lung_mask_file  = lung_mask_file.replace('LIDC-IDRI-','').replace('_1.nii.gz','.nii.gz')
    ct_scan_data = nib.load(ct_scan_file)
    lung_mask_data = nib.load(lung_mask_file)
    return ct_scan_data, lung_mask_data 

def show_ct_scan_data(ct_scan_data,lung_mask_data,num):
    plt.imshow(ct_scan_data.get_fdata()[:,:,num])
    plt.show()
    plt.imshow(lung_mask_data.get_fdata()[:,:,num])

Testing data

As there are only 40 examples in order to fine tune the model, only three examples will be taken as a test set that would not be used in the fine tuning mechanism. For this example, the studies held back will be the following::

Example	Description
coronacases_org_004.nii	A positive example with characteristics similar resulting from preprocessing the data
radiopaedia_org_covid-19-pneumonia-14_85914_0-dcm.nii	A positive example not in Hounsfield units and also not in anatomically correct orientation
LIDC-IDRI-0032_1.nii.gz	A negative example

The test labels are then [1,1,0], corresponding to two positive examples followed by a negative one

In [7]:

testing=['coronacases_org_004.nii','radiopaedia_org_covid-19-pneumonia-14_85914_0-dcm.nii','LIDC-IDRI-0032_1.nii.gz']
y_test=np.array([1.0,1.0,0.0])
y_test

Out[7]:

array([1., 1., 0.])

Loading the data from the first example:

In [8]:

ct_scan_data, lung_mask_data = get_ct_scan_data(testing[0])
ct_scan_data.shape

Out[8]:

(512, 512, 270)

We can observe that the data is, in fact, in three dimensions:

First testing image (A positive example)

As we will be plotting in two dimensions, we need to specify which exact X-Ray (num) we want to analyze. Here, we will look at slice 50:

In [9]:

show_ct_scan_data(ct_scan_data,lung_mask_data,num=50)

In [42]:

y_pred[1] = pred_labels[testing[1]]['COVID']
results[testing[1]]

Out[42]:

'No COVID = 0.025162333999999998; COVID = 0.97483766'

There are two images; the first is the original CT-Scan slice, and the second is the segmented Lung Mask. If we inspect another slice in the same CT-Scan (Slice 100), we get the following:

In [10]:

show_ct_scan_data(ct_scan_data,lung_mask_data,num=100)

Review Image Set 2 (Positive Example, Different Units)

In this example, we will view another CT exam, but the units are not the same as what the original model is expecting. This study uses pixel values; the model uses Hounsfield units (measuring radiodensity of the areas scanned). We will perform the same steps as before – look at the 3D aspect of the data, look at slice 50, and look at slice 100

In [11]:

ct_scan_data, lung_mask_data = get_ct_scan_data(testing[1])
ct_scan_data.shape

Out[11]:

(630, 401, 110)

In [12]:

show_ct_scan_data(ct_scan_data,lung_mask_data,50)

In [13]:

show_ct_scan_data(ct_scan_data,lung_mask_data,100)

Notice that the images in the second example look a bit different than the ones in the first example. This is because the second example is not in Hounsfield units.

Review Image Set 3 (Negative Example)

We will perform the same steps as before – look at the 3D aspect of the data, look at slice 50, and look at slice 100.

In [14]:

ct_scan_data, lung_mask_data = get_ct_scan_data(testing[2])
ct_scan_data.shape

Out[14]:

(512, 512, 249)

In [15]:

show_ct_scan_data(ct_scan_data,lung_mask_data,50)

In [16]:

show_ct_scan_data(ct_scan_data,lung_mask_data,100)

1.4 Data Indexing

The data index file Clara train uses, stored as JSON, describes the data to be used and where to find it, as part of the training run. It categorizes this data into a “training” set and a “validation” set. It also indicates which data is tagged for which labels. The result is a JSON file that Clara Train will use for the training run.

Notice that we generate two files, “experiments/covid19_3d_ct_classification-v2/config/data_kaggle_train.json” and “experiments/covid19_3d_ct_classification-v2/config/data_kaggle_infer.json” both relative to the workspaceblobstore/clara/ folder. The first folder is used for training with a 80-20 validation split and the second one contains the three files left for testing.

In [17]:

inputFile = 'experiments/kaggleset/metadata.csv'
negativesFile =  'experiments/kaggleset/Ct-Scan-Negatives.txt'
dataFile = 'experiments/covid19_3d_ct_classification-v2/config/data_kaggle_train.json'
testingFile='experiments/covid19_3d_ct_classification-v2/config/data_kaggle_infer.json'
testing_full = ['ct_scans/{0}'.format(x) for x in testing]
validation_size=0.20
tid.transformCTScanMetadataToJSON(inputFile,negativesFile,dataFile,testingFile,kaggledata_dir,testing_full,validation_size)

/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py:189: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)

Step 2: Classify/Infer on Test Data (with the original model)

We want to infer the labels of the data we left out for testing using the base model as it is. Clara has an infer.sh command to do so. This step has four parts:

Update the configuration files so that Clara could find the data we want to make inferences on
Update the infer command to point to the new configuration files
Run the infer.sh command
Evaluate the results

2.1 Update Configuration File

All Clara commands require a properly cnfigured “environment.json” config file

“environment.json” contains general configuration about the MMAR, including the root directory of the dataset established in the previous step

As we are using the MMAR from the base model, all its configuration files refer to the data the base model was trained with. So, the “environment.json” must be updated to refer to the new data index file and the new data source (generated in the previous step). The original “environment.json” is adapted into the new “infer_environment.json"

In [18]:

inputFile ='experiments/covid19_3d_ct_classification-v2/config/environment.json'
outputFile = 'experiments/covid19_3d_ct_classification-v2/config/infer_environment.json'
tid.adaptJSONFile(inputFile,outputFile,['DATASET_JSON','DATA_ROOT'],[testingFile,kaggledata_dir])

2.2 Update Infer.sh Command

The “infer,sh” command needs to be updated to refer to the updated “infer_environment.json” config file

The original “infer.sh” is used to create an updated version “new_infer,sh”

In [19]:

inputFile = 'experiments/covid19_3d_ct_classification-v2/commands/infer.sh'
outputFile='experiments/covid19_3d_ct_classification-v2/commands/new_infer.sh'
oldValues=["$MMAR_ROOT/config/dataset_0.json","config/environment.json"]
newValues=[testingFile,"config/infer_environment.json"]
tid.adaptScript(inputFile,outputFile,oldValues,newValues)

2.3 Executing the inference procedure

In [20]:

!experiments/covid19_3d_ct_classification-v2/commands/new_infer.sh

MMAR_ROOT set to /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/..
2020-08-24 21:11:41.623891: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[43101,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 65f2bbe182cf4de3b0542d8b3ccfb74b000000

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
Using TensorFlow backend.
2020-08-24 21:11:44,145 - nvmidl.utils.train_conf - INFO - Automatic Mixed Precision status: Disabled
Previously evaluated: 0 ; To be evaluated: 3
2020-08-24 21:11:45.862676: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2593990000 Hz
2020-08-24 21:11:45.865684: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d5c160 executing computations on platform Host. Devices:
2020-08-24 21:11:45.865709: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-08-24 21:11:45.868127: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-24 21:11:51.717291: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1ddca50 executing computations on platform CUDA. Devices:
2020-08-24 21:11:51.717327: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:11:51.717340: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:11:51.717350: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:11:51.717359: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:11:51.719071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:11:51.720133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 3a94:00:00.0
2020-08-24 21:11:51.721217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 6362:00:00.0
2020-08-24 21:11:51.722274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 8945:00:00.0
2020-08-24 21:11:51.722311: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:11:51.722413: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:11:51.722454: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:11:51.722491: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:11:51.725291: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:11:51.726961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:11:51.727021: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:11:51.735542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2020-08-24 21:11:51.735587: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:11:53.913704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 21:11:53.913758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2020-08-24 21:11:53.913774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N 
2020-08-24 21:11:53.913783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N 
2020-08-24 21:11:53.913789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N 
2020-08-24 21:11:53.913797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N 
2020-08-24 21:11:53.919710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14889 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 1a4c:00:00.0, compute capability: 7.0)
2020-08-24 21:11:53.921361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14889 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-16GB, pci bus id: 3a94:00:00.0, compute capability: 7.0)
2020-08-24 21:11:53.923269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 14889 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-16GB, pci bus id: 6362:00:00.0, compute capability: 7.0)
2020-08-24 21:11:53.924753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 14889 MB memory) -> physical GPU (device: 3, name: Tesla V100-PCIE-16GB, pci bus id: 8945:00:00.0, compute capability: 7.0)
2020-08-24 21:12:00.546696: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:12:00.754847: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Batch 1 / 3: 8.03s; pre-process: 4.11s; infer: 3.92s; post-process: 0.00s
Batch 2 / 3: 1.62s; pre-process: 1.57s; infer: 0.05s; post-process: 0.00s
Batch 3 / 3: 4.39s; pre-process: 4.34s; infer: 0.05s; post-process: 0.00s


Total Inference Time: 4.018452882766724s
2020-08-24 21:12:08,073 - nvmidl.utils.train_conf - INFO - Total Evaluation Time 24.60403060913086

2.4 Inspecting the inference results

Since we did not change the default parameters on infer.sh, the system stored the inferred predictions in the default folder and file: eval/preds_model.csv.

The following lines retrieve the probabilities produced by the “new_infer,sh” command and estimates the predicted labels (1:COVID, 0:NO COVID).

In [21]:

labels=['No COVID', 'COVID']
predictions_file = 'experiments/covid19_3d_ct_classification-v2/eval/preds_model.csv'
results,pred_labels = tid.processCTScanPredictions(predictions_file,labels,'COVID')

Estimating predicted labels for the three testing examples

In [22]:

y_pred = np.zeros(len(testing))

Inspecting the result for the first testing case (A positive one)

In [23]:

y_pred[0] = pred_labels[testing[0]]['COVID']
results[testing[0]]

Out[23]:

'No COVID = 0.01203339; COVID = 0.98796654'

The model correctly classified the example as belonging to a COVID case.

Inspecting the result for the second testing case (A positive one)

In [24]:

y_pred[1] = pred_labels[testing[1]]['COVID']
results[testing[1]]

Out[24]:

'No COVID = 0.99991715; COVID = 8.286651400000001e-05'

The model misclassifies the example as a non-COVID case, presumably because the example is not in Hounsfield units.

Inspecting the result for the third testing case (A negative one)

In [25]:

y_pred[2] = pred_labels[testing[2]]['COVID']
results[testing[2]]

Out[25]:

'No COVID = 0.99940354; COVID = 0.000596442'

The model correctly classified the example as NOT belonging to a COVID case.

2.5 Computing the Average COVID Classification Precision over all examples

The predicted labels are then used along the true labels to compute the testing examples average precision score using the sklearn.metrics.average_precision_score function.

In [26]:

print('Expected Labels')
print(y_test)
print("Predicted Labels")
print(y_pred)
average_precision = average_precision_score(y_test, y_pred)
print('Average Precision')
print(average_precision)

Expected Labels
[1. 1. 0.]
Predicted Labels
[1. 0. 0.]
Average Precision
0.8333333333333333

Notice that the average precision is not as high as expected because some instances of the new data are not preprocessed

FortunatelyFortunately, CLARA Clara has a mechanism to account for mistmaches between new data and the data a model was trained with.

That mechanism is called fine tune, where the parameters of the original model are fine tuned to account for the peculiarities new data.

Step 3: Fine-Tune

Clara has a train_finetune.sh command that executes the fine-tuning mechanism. This step has four parts:

Update the configuration files so that Clara could find the data we want to fine-tune the model on
Update the train_finetune.sh command to point to the new configuration file
Execute the train_finetune.sh command
Export the tuned model to be able to use for runing inferences with it

3.1 Update Configuration Files

The “environment.json” must be updated to refer to the data index file and the new data source, in addition, the original “environment.json” is adapted into the new “new_environment.json

In [27]:

inputFile ='experiments/covid19_3d_ct_classification-v2/config/environment.json'
outputFile = 'experiments/covid19_3d_ct_classification-v2/config/new_environment.json'
tid.adaptJSONFile(inputFile,outputFile,['DATASET_JSON','DATA_ROOT'],[dataFile,kaggledata_dir])

There is a “config_train.json” in the configuration folder, which contains all the training parameters including, the training framework, the pre-transforms necessary to perform on ingestion and the validation framework to monitor the progression of the model training. Choosing the right configuration is important, as it will be most likely be an iterative process to find the right parameters. You can use the AutoML features to tune some of these parameters automatically to speed up the training time. In this case the original “config_train.json” config file is adapted into config file “new_config_train.json” changing the output_batch_size to 2.

In [28]:

inputFile ='experiments/covid19_3d_ct_classification-v2/config/config_train.json'
outputFile = 'experiments/covid19_3d_ct_classification-v2/config/new_config_train.json'
output_batch_size = 2
tid.adaptCTScanJSONTrainConfigFile(inputFile,outputFile,output_batch_size)

3.2 Update train_finetune.sh Command

The “train_finetune.sh” command needs to be updated to refer to the updated “new_environment.json” and “new_config_train.json” config files. Given that we do not have much data to do the fine tuning the numbers of epochs is reduced to 30

The original “train_finetune,sh” is used to create an updated version “new_train_finetune,sh”

In [29]:

inputFile = 'experiments/covid19_3d_ct_classification-v2/commands/train_finetune.sh'
outputFile='experiments/covid19_3d_ct_classification-v2/commands/new_train_finetune.sh'
numepochs = 30
oldValues=["/workspace/Data/COVID/COVID_Test_Data","${MMAR_ROOT}/config/dataset_0.json","config/environment.json","config/config_train.json","epochs=1"]
newValues=["experiments/kaggleset",dataFile,"config/new_environment.json","config/new_config_train.json","epochs={0}".format(numepochs)]
tid.adaptScript(inputFile,outputFile,oldValues,newValues)

3.3 Execute Fine-tune Script

In [30]:

!experiments/covid19_3d_ct_classification-v2/commands/new_train_finetune.sh

MMAR_ROOT set to /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/..
2020-08-24 21:14:45.540435: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[43689,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 65f2bbe182cf4de3b0542d8b3ccfb74b000000

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
Using TensorFlow backend.
2020-08-24 21:14:47,441 - TrainConfiger - INFO - DETERMINISM IS ON
2020-08-24 21:14:47,448 - nvmidl.utils.train_conf - INFO - Automatic Mixed Precision status: Enabled
Number of samples: 28
Data Property: {'task': 'classification', 'num_channels': 1, 'num_label_channels': 1, 'data_format': 'channels_last', 'label_format': [2], 'crop_size': [192, 192, 64], 'num_data_dims': 3}
deterministic transforms: 3; non-deterministic transforms: 7
[KerasPipeline] Data Property: {'task': 'classification', 'num_channels': 1, 'num_label_channels': 1, 'data_format': 'channels_last', 'label_format': [2], 'crop_size': [192, 192, 64], 'num_data_dims': 3}
[KerasPipeline] No. Items: 28
Number of samples: 8
Data Property: {'task': 'classification', 'num_channels': 1, 'num_label_channels': 1, 'data_format': 'channels_last', 'label_format': [2], 'crop_size': [192, 192, 64], 'num_data_dims': 3}
[KerasPipeline] Data Property: {'task': 'classification', 'num_channels': 1, 'num_label_channels': 1, 'data_format': 'channels_last', 'label_format': [2], 'crop_size': [192, 192, 64], 'num_data_dims': 3}
[KerasPipeline] No. Items: 8
net_config: {'pretrain_weight_name': [], 'pretrain_weight_url': None, 'data_format': 'channels_last'}
Fitting with single gpu
2020-08-24 21:15:22.206150: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2593990000 Hz
2020-08-24 21:15:22.208805: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1bbeec0 executing computations on platform Host. Devices:
2020-08-24 21:15:22.208841: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-08-24 21:15:22.211525: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-24 21:15:28.769040: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1c3f7b0 executing computations on platform CUDA. Devices:
2020-08-24 21:15:28.769080: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:15:28.770081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:15:28.770113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:15:28.772072: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:15:28.773903: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:15:28.774653: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:15:28.776565: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:15:28.777625: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:15:28.781605: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:15:28.783251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-08-24 21:15:28.783292: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:15:29.138017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 21:15:29.138072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-08-24 21:15:29.138081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-08-24 21:15:29.139963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14959 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 1a4c:00:00.0, compute capability: 7.0)
2020-08-24 21:15:37.036659: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:15:37.078483: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1331] No whitelist ops found, nothing to do
2020-08-24 21:15:42,310 - SupervisedFitter - INFO - RESTORING ALL VARS from /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../models/model.ckpt
2020-08-24 21:15:49.707936: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:15:49.716100: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1331] No whitelist ops found, nothing to do
2020-08-24 21:16:37.538754: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:16:37.539016: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1331] No whitelist ops found, nothing to do
Requested train epochs: 30; iterations: 14; previous global epoch: 1494
compute initial best val metric for restored session
2020-08-24 21:17:17.626919: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:17:17.659084: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1837] Converted 292/3733 nodes to float16 precision using 124 cast(s) to float16 (excluding Const and Variable casts)
2020-08-24 21:17:19.362001: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:17:19.644558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
initial best val metric: -1.2013486623764038
/usr/local/lib/python3.6/dist-packages/scipy/ndimage/interpolation.py:611: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)
2020-08-24 21:18:56.532006: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:18:56.777788: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1837] Converted 981/20311 nodes to float16 precision using 251 cast(s) to float16 (excluding Const and Variable casts)
Epoch: 1/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 0.7829  time: 125.54s        
Epoch: 1/30, Iter: 2/14 [==                  ]  train_accuracy: 0.2500  train_loss: 3.8880  time: 0.25s        
Epoch: 1/30, Iter: 3/14 [====                ]  train_accuracy: 0.1667  train_loss: 3.4915  time: 0.23s        
Epoch: 1/30, Iter: 4/14 [=====               ]  train_accuracy: 0.1250  train_loss: 4.1850  time: 0.24s        
Epoch: 1/30, Iter: 5/14 [=======             ]  train_accuracy: 0.2000  train_loss: 3.5445  time: 0.25s        
Epoch: 1/30, Iter: 6/14 [========            ]  train_accuracy: 0.1667  train_loss: 4.1575  time: 0.22s        
Epoch: 1/30, Iter: 7/14 [==========          ]  train_accuracy: 0.2857  train_loss: 3.5637  time: 2.36s        
Epoch: 1/30, Iter: 8/14 [===========         ]  train_accuracy: 0.2500  train_loss: 3.9915  time: 1.61s        
Epoch: 1/30, Iter: 9/14 [============        ]  train_accuracy: 0.3333  train_loss: 3.5481  time: 3.20s        
Epoch: 1/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4000  train_loss: 3.1934  time: 0.28s        
Epoch: 1/30, Iter: 11/14 [===============     ]  train_accuracy: 0.3636  train_loss: 3.5200  time: 3.60s        
Epoch: 1/30, Iter: 12/14 [=================   ]  train_accuracy: 0.3750  train_loss: 3.4797  time: 3.64s        
Epoch: 1/30, Iter: 13/14 [==================  ]  train_accuracy: 0.3462  train_loss: 3.7599  time: 0.29s        
Epoch: 1/30, Iter: 14/14 [====================]  train_accuracy: 0.3571  train_loss: 3.6675  time: 0.26s        
This epoch: 171.98s; per epoch: 171.98s; elapsed: 171.98s; remaining: 4987.46s; best metric: -1.2013486623764038 at epoch 0
Epoch: 1/30, train_accuracy: 0.3571  train_loss: 3.6675  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.4139  val_time: 0.23s    
New best val metric: -0.41387006640434265
Saving model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt
2020-08-24 21:20:14.765058: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1881] Running auto_mixed_precision graph optimizer
2020-08-24 21:20:14.773199: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1331] No whitelist ops found, nothing to do
Epoch: 2/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0008  time: 0.42s        
Epoch: 2/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 0.4207  time: 0.23s        
Epoch: 2/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 1.1812  time: 0.84s        
Epoch: 2/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 2.5849  time: 1.49s        
Epoch: 2/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 2.1995  time: 0.23s        
Epoch: 2/30, Iter: 6/14 [========            ]  train_accuracy: 0.5833  train_loss: 1.8331  time: 10.55s        
Epoch: 2/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 1.9980  time: 2.20s        
Epoch: 2/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5625  train_loss: 1.8326  time: 0.34s        
Epoch: 2/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 1.6291  time: 5.21s        
Epoch: 2/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6500  train_loss: 1.4692  time: 0.22s        
Epoch: 2/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6364  train_loss: 1.5152  time: 0.39s        
Epoch: 2/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.6965  time: 0.32s        
Epoch: 2/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6154  train_loss: 1.5661  time: 0.29s        
Epoch: 2/30, Iter: 14/14 [====================]  train_accuracy: 0.5714  train_loss: 1.9246  time: 3.52s        
This epoch: 26.26s; per epoch: 99.12s; elapsed: 198.24s; remaining: 2775.32s; best metric: -0.41387006640434265 at epoch 1
Epoch: 2/30, train_accuracy: 0.5714  train_loss: 1.9246  mean_accuracy: 0.8750  valid_mean_neg_loss: -1.4193  val_time: 0.21s    
Epoch: 3/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 7.1512  time: 0.23s        
Epoch: 3/30, Iter: 2/14 [==                  ]  train_accuracy: 0.2500  train_loss: 5.3234  time: 4.27s        
Epoch: 3/30, Iter: 3/14 [====                ]  train_accuracy: 0.1667  train_loss: 5.0443  time: 0.55s        
Epoch: 3/30, Iter: 4/14 [=====               ]  train_accuracy: 0.1250  train_loss: 5.5650  time: 0.38s        
Epoch: 3/30, Iter: 5/14 [=======             ]  train_accuracy: 0.2000  train_loss: 5.1234  time: 0.51s        
Epoch: 3/30, Iter: 6/14 [========            ]  train_accuracy: 0.2500  train_loss: 4.4175  time: 4.87s        
Epoch: 3/30, Iter: 7/14 [==========          ]  train_accuracy: 0.2857  train_loss: 3.8975  time: 0.23s        
Epoch: 3/30, Iter: 8/14 [===========         ]  train_accuracy: 0.3125  train_loss: 3.6204  time: 0.44s        
Epoch: 3/30, Iter: 9/14 [============        ]  train_accuracy: 0.3333  train_loss: 3.5358  time: 0.47s        
Epoch: 3/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4000  train_loss: 3.1824  time: 0.50s        
Epoch: 3/30, Iter: 11/14 [===============     ]  train_accuracy: 0.4091  train_loss: 3.1124  time: 0.37s        
Epoch: 3/30, Iter: 12/14 [=================   ]  train_accuracy: 0.4583  train_loss: 2.8531  time: 0.54s        
Epoch: 3/30, Iter: 13/14 [==================  ]  train_accuracy: 0.4231  train_loss: 3.1173  time: 0.44s        
Epoch: 3/30, Iter: 14/14 [====================]  train_accuracy: 0.4286  train_loss: 3.1253  time: 0.30s        
This epoch: 14.12s; per epoch: 70.79s; elapsed: 212.36s; remaining: 1911.22s; best metric: -0.41387006640434265 at epoch 1
Epoch: 3/30, train_accuracy: 0.4286  train_loss: 3.1253  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0493  val_time: 16.42s    
New best val metric: -0.04928049445152283
Saving model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt
Epoch: 4/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.1907  time: 0.28s        
Epoch: 4/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.2922  time: 0.22s        
Epoch: 4/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 2.9666  time: 0.22s        
Epoch: 4/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 2.4265  time: 0.23s        
Epoch: 4/30, Iter: 5/14 [=======             ]  train_accuracy: 0.3000  train_loss: 2.3546  time: 0.22s        
Epoch: 4/30, Iter: 6/14 [========            ]  train_accuracy: 0.3333  train_loss: 2.3303  time: 0.24s        
Epoch: 4/30, Iter: 7/14 [==========          ]  train_accuracy: 0.3571  train_loss: 2.4653  time: 0.24s        
Epoch: 4/30, Iter: 8/14 [===========         ]  train_accuracy: 0.3750  train_loss: 2.5226  time: 0.22s        
Epoch: 4/30, Iter: 9/14 [============        ]  train_accuracy: 0.3333  train_loss: 3.0214  time: 0.23s        
Epoch: 4/30, Iter: 10/14 [==============      ]  train_accuracy: 0.3000  train_loss: 3.4354  time: 0.24s        
Epoch: 4/30, Iter: 11/14 [===============     ]  train_accuracy: 0.3182  train_loss: 3.3459  time: 0.23s        
Epoch: 4/30, Iter: 12/14 [=================   ]  train_accuracy: 0.3333  train_loss: 3.2731  time: 0.24s        
Epoch: 4/30, Iter: 13/14 [==================  ]  train_accuracy: 0.3462  train_loss: 3.1596  time: 0.26s        
Epoch: 4/30, Iter: 14/14 [====================]  train_accuracy: 0.3571  train_loss: 3.1420  time: 0.22s        
This epoch: 3.30s; per epoch: 53.91s; elapsed: 215.65s; remaining: 1401.74s; best metric: -0.04928049445152283 at epoch 3
Epoch: 4/30, train_accuracy: 0.3571  train_loss: 3.1420  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0438  val_time: 20.01s    
New best val metric: -0.043783072382211685
Saving model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt
Epoch: 5/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0008  time: 0.26s        
Epoch: 5/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 0.8774  time: 0.23s        
Epoch: 5/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 2.9804  time: 0.24s        
Epoch: 5/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 2.2355  time: 0.22s        
Epoch: 5/30, Iter: 5/14 [=======             ]  train_accuracy: 0.6000  train_loss: 2.5122  time: 0.23s        
Epoch: 5/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 3.2882  time: 0.22s        
Epoch: 5/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 2.9311  time: 0.22s        
Epoch: 5/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5625  train_loss: 2.5651  time: 0.29s        
Epoch: 5/30, Iter: 9/14 [============        ]  train_accuracy: 0.5556  train_loss: 2.3956  time: 0.22s        
Epoch: 5/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5500  train_loss: 2.4458  time: 0.22s        
Epoch: 5/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.4837  time: 0.22s        
Epoch: 5/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5417  train_loss: 2.4138  time: 0.22s        
Epoch: 5/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 2.4929  time: 0.29s        
Epoch: 5/30, Iter: 14/14 [====================]  train_accuracy: 0.5357  train_loss: 2.4052  time: 0.38s        
This epoch: 3.47s; per epoch: 43.82s; elapsed: 219.12s; remaining: 1095.61s; best metric: -0.043783072382211685 at epoch 4
Epoch: 5/30, train_accuracy: 0.5357  train_loss: 2.4052  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0092  val_time: 19.50s    
New best val metric: -0.009176086634397507
Saving model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt
Epoch: 6/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 4.7206  time: 0.37s        
Epoch: 6/30, Iter: 2/14 [==                  ]  train_accuracy: 0.2500  train_loss: 3.4808  time: 0.26s        
Epoch: 6/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 3.2149  time: 0.22s        
Epoch: 6/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 2.9118  time: 0.22s        
Epoch: 6/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 2.3296  time: 0.24s        
Epoch: 6/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 2.0881  time: 0.24s        
Epoch: 6/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 1.7899  time: 0.24s        
Epoch: 6/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 1.5792  time: 0.22s        
Epoch: 6/30, Iter: 9/14 [============        ]  train_accuracy: 0.6667  train_loss: 1.4060  time: 0.23s        
Epoch: 6/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 1.5932  time: 0.36s        
Epoch: 6/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5909  train_loss: 1.5645  time: 0.24s        
Epoch: 6/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.6011  time: 0.24s        
Epoch: 6/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5769  train_loss: 1.6227  time: 0.22s        
Epoch: 6/30, Iter: 14/14 [====================]  train_accuracy: 0.5714  train_loss: 1.7530  time: 0.22s        
This epoch: 3.52s; per epoch: 37.11s; elapsed: 222.64s; remaining: 890.57s; best metric: -0.009176086634397507 at epoch 5
Epoch: 6/30, train_accuracy: 0.5714  train_loss: 1.7530  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0032  val_time: 20.48s    
New best val metric: -0.0032119210809469223
Saving model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt
Epoch: 7/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 7.0019  time: 0.29s        
Epoch: 7/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.5050  time: 0.22s        
Epoch: 7/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 2.5923  time: 0.23s        
Epoch: 7/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 1.9444  time: 0.24s        
Epoch: 7/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 1.5564  time: 0.21s        
Epoch: 7/30, Iter: 6/14 [========            ]  train_accuracy: 0.6667  train_loss: 1.7433  time: 0.22s        
Epoch: 7/30, Iter: 7/14 [==========          ]  train_accuracy: 0.6429  train_loss: 1.9829  time: 0.21s        
Epoch: 7/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 1.8593  time: 0.26s        
Epoch: 7/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 1.9493  time: 0.23s        
Epoch: 7/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 1.9789  time: 0.25s        
Epoch: 7/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5909  train_loss: 1.9689  time: 0.23s        
Epoch: 7/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 2.1068  time: 0.22s        
Epoch: 7/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6154  train_loss: 1.9448  time: 0.23s        
Epoch: 7/30, Iter: 14/14 [====================]  train_accuracy: 0.6429  train_loss: 1.8060  time: 0.23s        
This epoch: 3.29s; per epoch: 32.28s; elapsed: 225.93s; remaining: 742.34s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 7/30, train_accuracy: 0.6429  train_loss: 1.8060  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0441  val_time: 19.61s    
Epoch: 8/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0016  time: 0.32s        
Epoch: 8/30, Iter: 2/14 [==                  ]  train_accuracy: 1.0000  train_loss: 0.0372  time: 0.26s        
Epoch: 8/30, Iter: 3/14 [====                ]  train_accuracy: 1.0000  train_loss: 0.0254  time: 0.33s        
Epoch: 8/30, Iter: 4/14 [=====               ]  train_accuracy: 1.0000  train_loss: 0.0192  time: 0.24s        
Epoch: 8/30, Iter: 5/14 [=======             ]  train_accuracy: 1.0000  train_loss: 0.0156  time: 0.25s        
Epoch: 8/30, Iter: 6/14 [========            ]  train_accuracy: 1.0000  train_loss: 0.0154  time: 0.21s        
Epoch: 8/30, Iter: 7/14 [==========          ]  train_accuracy: 0.8571  train_loss: 0.9714  time: 0.22s        
Epoch: 8/30, Iter: 8/14 [===========         ]  train_accuracy: 0.7500  train_loss: 1.7314  time: 0.22s        
Epoch: 8/30, Iter: 9/14 [============        ]  train_accuracy: 0.7778  train_loss: 1.5392  time: 0.22s        
Epoch: 8/30, Iter: 10/14 [==============      ]  train_accuracy: 0.7000  train_loss: 2.1057  time: 0.23s        
Epoch: 8/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6818  train_loss: 2.2096  time: 0.22s        
Epoch: 8/30, Iter: 12/14 [=================   ]  train_accuracy: 0.7083  train_loss: 2.0255  time: 0.44s        
Epoch: 8/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6923  train_loss: 2.0244  time: 0.46s        
Epoch: 8/30, Iter: 14/14 [====================]  train_accuracy: 0.6786  train_loss: 2.1193  time: 0.42s        
This epoch: 4.04s; per epoch: 28.75s; elapsed: 229.97s; remaining: 632.41s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 8/30, train_accuracy: 0.6786  train_loss: 2.1193  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0252  val_time: 23.96s    
Epoch: 9/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0711  time: 0.30s        
Epoch: 9/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.1362  time: 0.24s        
Epoch: 9/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 4.4293  time: 0.40s        
Epoch: 9/30, Iter: 4/14 [=====               ]  train_accuracy: 0.2500  train_loss: 5.0515  time: 0.25s        
Epoch: 9/30, Iter: 5/14 [=======             ]  train_accuracy: 0.4000  train_loss: 4.0414  time: 0.24s        
Epoch: 9/30, Iter: 6/14 [========            ]  train_accuracy: 0.3333  train_loss: 4.0260  time: 0.22s        
Epoch: 9/30, Iter: 7/14 [==========          ]  train_accuracy: 0.3571  train_loss: 3.9524  time: 0.21s        
Epoch: 9/30, Iter: 8/14 [===========         ]  train_accuracy: 0.4375  train_loss: 3.4585  time: 0.22s        
Epoch: 9/30, Iter: 9/14 [============        ]  train_accuracy: 0.3889  train_loss: 3.7075  time: 0.22s        
Epoch: 9/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4500  train_loss: 3.3370  time: 0.27s        
Epoch: 9/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5000  train_loss: 3.0337  time: 0.24s        
Epoch: 9/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5417  train_loss: 2.7810  time: 0.23s        
Epoch: 9/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 2.7341  time: 0.26s        
Epoch: 9/30, Iter: 14/14 [====================]  train_accuracy: 0.5357  train_loss: 2.7390  time: 0.27s        
This epoch: 3.59s; per epoch: 25.95s; elapsed: 233.56s; remaining: 544.97s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 9/30, train_accuracy: 0.5357  train_loss: 2.7390  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.1543  val_time: 24.28s    
Epoch: 10/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 2.3055  time: 0.32s        
Epoch: 10/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.3836  time: 0.24s        
Epoch: 10/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.5893  time: 0.41s        
Epoch: 10/30, Iter: 4/14 [=====               ]  train_accuracy: 0.7500  train_loss: 1.1922  time: 0.22s        
Epoch: 10/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 1.6737  time: 0.22s        
Epoch: 10/30, Iter: 6/14 [========            ]  train_accuracy: 0.7500  train_loss: 1.3949  time: 0.23s        
Epoch: 10/30, Iter: 7/14 [==========          ]  train_accuracy: 0.7143  train_loss: 1.4109  time: 0.21s        
Epoch: 10/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 2.1227  time: 0.22s        
Epoch: 10/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 2.1304  time: 0.23s        
Epoch: 10/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5500  train_loss: 2.6120  time: 0.22s        
Epoch: 10/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.5036  time: 0.23s        
Epoch: 10/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5417  train_loss: 2.6001  time: 0.22s        
Epoch: 10/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 2.4643  time: 0.25s        
Epoch: 10/30, Iter: 14/14 [====================]  train_accuracy: 0.5714  train_loss: 2.2883  time: 0.24s        
This epoch: 3.46s; per epoch: 23.70s; elapsed: 237.02s; remaining: 474.04s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 10/30, train_accuracy: 0.5714  train_loss: 2.2883  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.3203  val_time: 24.26s    
Epoch: 11/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 1.5960  time: 0.29s        
Epoch: 11/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 1.5374  time: 0.23s        
Epoch: 11/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 2.6855  time: 0.43s        
Epoch: 11/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 2.8673  time: 0.21s        
Epoch: 11/30, Iter: 5/14 [=======             ]  train_accuracy: 0.4000  train_loss: 2.4733  time: 0.26s        
Epoch: 11/30, Iter: 6/14 [========            ]  train_accuracy: 0.4167  train_loss: 2.3229  time: 0.26s        
Epoch: 11/30, Iter: 7/14 [==========          ]  train_accuracy: 0.4286  train_loss: 2.3584  time: 0.22s        
Epoch: 11/30, Iter: 8/14 [===========         ]  train_accuracy: 0.3750  train_loss: 2.9494  time: 0.22s        
Epoch: 11/30, Iter: 9/14 [============        ]  train_accuracy: 0.3333  train_loss: 3.4149  time: 0.22s        
Epoch: 11/30, Iter: 10/14 [==============      ]  train_accuracy: 0.3500  train_loss: 3.3161  time: 0.21s        
Epoch: 11/30, Iter: 11/14 [===============     ]  train_accuracy: 0.3636  train_loss: 3.3008  time: 0.21s        
Epoch: 11/30, Iter: 12/14 [=================   ]  train_accuracy: 0.3333  train_loss: 3.6008  time: 0.22s        
Epoch: 11/30, Iter: 13/14 [==================  ]  train_accuracy: 0.3462  train_loss: 3.5508  time: 0.21s        
Epoch: 11/30, Iter: 14/14 [====================]  train_accuracy: 0.3214  train_loss: 3.8111  time: 0.22s        
This epoch: 3.41s; per epoch: 21.86s; elapsed: 240.43s; remaining: 415.29s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 11/30, train_accuracy: 0.3214  train_loss: 3.8111  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.8038  val_time: 25.09s    
Epoch: 12/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0016  time: 0.29s        
Epoch: 12/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 0.8009  time: 0.23s        
Epoch: 12/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.7469  time: 0.40s        
Epoch: 12/30, Iter: 4/14 [=====               ]  train_accuracy: 0.7500  train_loss: 1.3105  time: 0.23s        
Epoch: 12/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 1.4930  time: 0.22s        
Epoch: 12/30, Iter: 6/14 [========            ]  train_accuracy: 0.6667  train_loss: 1.8531  time: 0.22s        
Epoch: 12/30, Iter: 7/14 [==========          ]  train_accuracy: 0.6429  train_loss: 2.1100  time: 0.21s        
Epoch: 12/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6875  train_loss: 1.8464  time: 0.22s        
Epoch: 12/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 1.8212  time: 0.22s        
Epoch: 12/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 1.8669  time: 0.22s        
Epoch: 12/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6364  train_loss: 1.6972  time: 0.21s        
Epoch: 12/30, Iter: 12/14 [=================   ]  train_accuracy: 0.6250  train_loss: 1.7560  time: 0.23s        
Epoch: 12/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6154  train_loss: 1.6951  time: 0.22s        
Epoch: 12/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 1.8346  time: 0.23s        
This epoch: 3.37s; per epoch: 20.32s; elapsed: 243.81s; remaining: 365.71s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 12/30, train_accuracy: 0.6071  train_loss: 1.8346  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0409  val_time: 24.08s    
Epoch: 13/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 6.9531  time: 0.28s        
Epoch: 13/30, Iter: 2/14 [==                  ]  train_accuracy: 0.2500  train_loss: 5.3057  time: 0.23s        
Epoch: 13/30, Iter: 3/14 [====                ]  train_accuracy: 0.1667  train_loss: 5.8291  time: 0.42s        
Epoch: 13/30, Iter: 4/14 [=====               ]  train_accuracy: 0.2500  train_loss: 4.7686  time: 0.23s        
Epoch: 13/30, Iter: 5/14 [=======             ]  train_accuracy: 0.3000  train_loss: 4.1536  time: 0.22s        
Epoch: 13/30, Iter: 6/14 [========            ]  train_accuracy: 0.3333  train_loss: 4.0180  time: 0.22s        
Epoch: 13/30, Iter: 7/14 [==========          ]  train_accuracy: 0.2857  train_loss: 4.4696  time: 0.22s        
Epoch: 13/30, Iter: 8/14 [===========         ]  train_accuracy: 0.3750  train_loss: 3.9110  time: 0.25s        
Epoch: 13/30, Iter: 9/14 [============        ]  train_accuracy: 0.3333  train_loss: 4.2792  time: 0.23s        
Epoch: 13/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4000  train_loss: 3.8621  time: 0.22s        
Epoch: 13/30, Iter: 11/14 [===============     ]  train_accuracy: 0.4091  train_loss: 3.8269  time: 0.21s        
Epoch: 13/30, Iter: 12/14 [=================   ]  train_accuracy: 0.3750  train_loss: 3.9056  time: 0.59s        
Epoch: 13/30, Iter: 13/14 [==================  ]  train_accuracy: 0.3846  train_loss: 3.8104  time: 0.38s        
Epoch: 13/30, Iter: 14/14 [====================]  train_accuracy: 0.3929  train_loss: 3.6701  time: 0.28s        
This epoch: 4.00s; per epoch: 19.06s; elapsed: 247.81s; remaining: 324.05s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 13/30, train_accuracy: 0.3929  train_loss: 3.6701  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0272  val_time: 24.01s    
Epoch: 14/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.6203  time: 0.26s        
Epoch: 14/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.0087  time: 0.22s        
Epoch: 14/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 3.1470  time: 0.43s        
Epoch: 14/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 4.1283  time: 0.21s        
Epoch: 14/30, Iter: 5/14 [=======             ]  train_accuracy: 0.3000  train_loss: 4.7473  time: 0.22s        
Epoch: 14/30, Iter: 6/14 [========            ]  train_accuracy: 0.2500  train_loss: 4.6168  time: 0.22s        
Epoch: 14/30, Iter: 7/14 [==========          ]  train_accuracy: 0.2143  train_loss: 4.2651  time: 0.21s        
Epoch: 14/30, Iter: 8/14 [===========         ]  train_accuracy: 0.2500  train_loss: 4.0010  time: 0.21s        
Epoch: 14/30, Iter: 9/14 [============        ]  train_accuracy: 0.2778  train_loss: 3.9447  time: 0.22s        
Epoch: 14/30, Iter: 10/14 [==============      ]  train_accuracy: 0.3000  train_loss: 3.8977  time: 0.21s        
Epoch: 14/30, Iter: 11/14 [===============     ]  train_accuracy: 0.2727  train_loss: 3.7012  time: 0.21s        
Epoch: 14/30, Iter: 12/14 [=================   ]  train_accuracy: 0.2500  train_loss: 3.8682  time: 0.21s        
Epoch: 14/30, Iter: 13/14 [==================  ]  train_accuracy: 0.2692  train_loss: 3.6534  time: 0.22s        
Epoch: 14/30, Iter: 14/14 [====================]  train_accuracy: 0.2500  train_loss: 3.8956  time: 0.22s        
This epoch: 3.30s; per epoch: 17.94s; elapsed: 251.10s; remaining: 286.98s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 14/30, train_accuracy: 0.2500  train_loss: 3.8956  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.6057  val_time: 24.61s    
Epoch: 15/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 3.2258  time: 0.27s        
Epoch: 15/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 1.6134  time: 0.23s        
Epoch: 15/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.0763  time: 0.44s        
Epoch: 15/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 1.1118  time: 0.23s        
Epoch: 15/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 0.8897  time: 0.24s        
Epoch: 15/30, Iter: 6/14 [========            ]  train_accuracy: 0.6667  train_loss: 1.0750  time: 0.21s        
Epoch: 15/30, Iter: 7/14 [==========          ]  train_accuracy: 0.6429  train_loss: 1.4002  time: 0.22s        
Epoch: 15/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 1.6201  time: 0.21s        
Epoch: 15/30, Iter: 9/14 [============        ]  train_accuracy: 0.5556  train_loss: 2.0216  time: 0.22s        
Epoch: 15/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 1.8195  time: 0.22s        
Epoch: 15/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5909  train_loss: 1.9685  time: 0.22s        
Epoch: 15/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.9605  time: 0.24s        
Epoch: 15/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5769  train_loss: 1.9239  time: 0.25s        
Epoch: 15/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 1.7866  time: 0.33s        
This epoch: 3.55s; per epoch: 16.98s; elapsed: 254.66s; remaining: 254.66s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 15/30, train_accuracy: 0.6071  train_loss: 1.7866  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.4463  val_time: 24.50s    
Epoch: 16/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 7.0585  time: 0.28s        
Epoch: 16/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.5296  time: 0.24s        
Epoch: 16/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 4.7693  time: 0.22s        
Epoch: 16/30, Iter: 4/14 [=====               ]  train_accuracy: 0.2500  train_loss: 5.2846  time: 0.25s        
Epoch: 16/30, Iter: 5/14 [=======             ]  train_accuracy: 0.3000  train_loss: 4.7315  time: 0.24s        
Epoch: 16/30, Iter: 6/14 [========            ]  train_accuracy: 0.4167  train_loss: 3.9574  time: 0.22s        
Epoch: 16/30, Iter: 7/14 [==========          ]  train_accuracy: 0.4286  train_loss: 3.5206  time: 0.24s        
Epoch: 16/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5000  train_loss: 3.0806  time: 0.22s        
Epoch: 16/30, Iter: 9/14 [============        ]  train_accuracy: 0.5556  train_loss: 2.7388  time: 0.24s        
Epoch: 16/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5000  train_loss: 3.1783  time: 0.22s        
Epoch: 16/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.8895  time: 0.22s        
Epoch: 16/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5417  train_loss: 2.8941  time: 0.24s        
Epoch: 16/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 2.9385  time: 0.21s        
Epoch: 16/30, Iter: 14/14 [====================]  train_accuracy: 0.5357  train_loss: 2.9319  time: 0.29s        
This epoch: 3.32s; per epoch: 16.12s; elapsed: 257.98s; remaining: 225.73s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 16/30, train_accuracy: 0.5357  train_loss: 2.9319  mean_accuracy: 0.8750  valid_mean_neg_loss: -0.4076  val_time: 24.79s    
Epoch: 17/30, Iter: 1/14 [=                   ]  train_accuracy: 0.0000  train_loss: 3.2047  time: 0.33s        
Epoch: 17/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 1.6027  time: 0.23s        
Epoch: 17/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 2.2743  time: 0.33s        
Epoch: 17/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 3.4942  time: 0.24s        
Epoch: 17/30, Iter: 5/14 [=======             ]  train_accuracy: 0.3000  train_loss: 4.1901  time: 0.23s        
Epoch: 17/30, Iter: 6/14 [========            ]  train_accuracy: 0.4167  train_loss: 3.4920  time: 0.22s        
Epoch: 17/30, Iter: 7/14 [==========          ]  train_accuracy: 0.4286  train_loss: 3.5065  time: 0.21s        
Epoch: 17/30, Iter: 8/14 [===========         ]  train_accuracy: 0.4375  train_loss: 3.4876  time: 0.22s        
Epoch: 17/30, Iter: 9/14 [============        ]  train_accuracy: 0.4444  train_loss: 3.3543  time: 0.22s        
Epoch: 17/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4000  train_loss: 3.6882  time: 0.24s        
Epoch: 17/30, Iter: 11/14 [===============     ]  train_accuracy: 0.4545  train_loss: 3.3529  time: 0.22s        
Epoch: 17/30, Iter: 12/14 [=================   ]  train_accuracy: 0.4167  train_loss: 3.6686  time: 0.21s        
Epoch: 17/30, Iter: 13/14 [==================  ]  train_accuracy: 0.4231  train_loss: 3.4386  time: 0.22s        
Epoch: 17/30, Iter: 14/14 [====================]  train_accuracy: 0.3929  train_loss: 3.6841  time: 0.24s        
This epoch: 3.39s; per epoch: 15.37s; elapsed: 261.37s; remaining: 199.87s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 17/30, train_accuracy: 0.3929  train_loss: 3.6841  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0213  val_time: 24.36s    
Epoch: 18/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0019  time: 0.29s        
Epoch: 18/30, Iter: 2/14 [==                  ]  train_accuracy: 1.0000  train_loss: 0.0029  time: 0.21s        
Epoch: 18/30, Iter: 3/14 [====                ]  train_accuracy: 0.8333  train_loss: 0.4154  time: 0.40s        
Epoch: 18/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 1.2040  time: 0.23s        
Epoch: 18/30, Iter: 5/14 [=======             ]  train_accuracy: 0.6000  train_loss: 1.6820  time: 0.22s        
Epoch: 18/30, Iter: 6/14 [========            ]  train_accuracy: 0.6667  train_loss: 1.4018  time: 0.22s        
Epoch: 18/30, Iter: 7/14 [==========          ]  train_accuracy: 0.6429  train_loss: 1.6797  time: 0.21s        
Epoch: 18/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 1.7444  time: 0.22s        
Epoch: 18/30, Iter: 9/14 [============        ]  train_accuracy: 0.6667  train_loss: 1.5506  time: 0.22s        
Epoch: 18/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 2.0411  time: 0.22s        
Epoch: 18/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.0995  time: 0.21s        
Epoch: 18/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.9248  time: 0.21s        
Epoch: 18/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5769  train_loss: 1.8654  time: 0.22s        
Epoch: 18/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 1.7325  time: 0.21s        
This epoch: 3.33s; per epoch: 14.71s; elapsed: 264.69s; remaining: 176.46s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 18/30, train_accuracy: 0.6071  train_loss: 1.7325  mean_accuracy: 0.8750  valid_mean_neg_loss: -0.9192  val_time: 24.26s    
Epoch: 19/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0007  time: 0.28s        
Epoch: 19/30, Iter: 2/14 [==                  ]  train_accuracy: 1.0000  train_loss: 0.0154  time: 0.23s        
Epoch: 19/30, Iter: 3/14 [====                ]  train_accuracy: 1.0000  train_loss: 0.0351  time: 0.37s        
Epoch: 19/30, Iter: 4/14 [=====               ]  train_accuracy: 0.7500  train_loss: 1.5618  time: 0.22s        
Epoch: 19/30, Iter: 5/14 [=======             ]  train_accuracy: 0.8000  train_loss: 1.2510  time: 0.23s        
Epoch: 19/30, Iter: 6/14 [========            ]  train_accuracy: 0.7500  train_loss: 1.4680  time: 0.23s        
Epoch: 19/30, Iter: 7/14 [==========          ]  train_accuracy: 0.6429  train_loss: 1.9070  time: 0.23s        
Epoch: 19/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6875  train_loss: 1.7047  time: 0.22s        
Epoch: 19/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 2.2932  time: 0.24s        
Epoch: 19/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 2.1231  time: 0.23s        
Epoch: 19/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5909  train_loss: 2.0169  time: 0.31s        
Epoch: 19/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.9200  time: 0.45s        
Epoch: 19/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5769  train_loss: 1.8274  time: 0.30s        
Epoch: 19/30, Iter: 14/14 [====================]  train_accuracy: 0.5714  train_loss: 1.9351  time: 0.54s        
This epoch: 4.09s; per epoch: 14.15s; elapsed: 268.78s; remaining: 155.61s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 19/30, train_accuracy: 0.5714  train_loss: 1.9351  mean_accuracy: 0.8750  valid_mean_neg_loss: -1.3824  val_time: 24.14s    
Epoch: 20/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0008  time: 0.29s        
Epoch: 20/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.5535  time: 0.31s        
Epoch: 20/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 2.8519  time: 0.38s        
Epoch: 20/30, Iter: 4/14 [=====               ]  train_accuracy: 0.5000  train_loss: 3.0426  time: 0.23s        
Epoch: 20/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 3.1320  time: 0.23s        
Epoch: 20/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 3.1952  time: 0.24s        
Epoch: 20/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 2.7445  time: 0.22s        
Epoch: 20/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5000  train_loss: 3.3049  time: 0.22s        
Epoch: 20/30, Iter: 9/14 [============        ]  train_accuracy: 0.5556  train_loss: 2.9391  time: 0.22s        
Epoch: 20/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5000  train_loss: 3.3616  time: 0.22s        
Epoch: 20/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 3.0613  time: 0.23s        
Epoch: 20/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 2.8063  time: 0.22s        
Epoch: 20/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5769  train_loss: 2.8650  time: 0.45s        
Epoch: 20/30, Iter: 14/14 [====================]  train_accuracy: 0.5714  train_loss: 2.7731  time: 0.28s        
This epoch: 3.75s; per epoch: 13.63s; elapsed: 272.53s; remaining: 136.27s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 20/30, train_accuracy: 0.5714  train_loss: 2.7731  mean_accuracy: 0.8750  valid_mean_neg_loss: -1.2174  val_time: 24.28s    
Epoch: 21/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.4010  time: 0.31s        
Epoch: 21/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.0125  time: 0.24s        
Epoch: 21/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.3595  time: 0.43s        
Epoch: 21/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 1.2197  time: 0.22s        
Epoch: 21/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 1.0537  time: 0.24s        
Epoch: 21/30, Iter: 6/14 [========            ]  train_accuracy: 0.5833  train_loss: 2.0701  time: 0.23s        
Epoch: 21/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 2.7249  time: 0.22s        
Epoch: 21/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5000  train_loss: 2.4773  time: 0.22s        
Epoch: 21/30, Iter: 9/14 [============        ]  train_accuracy: 0.5000  train_loss: 2.3756  time: 0.22s        
Epoch: 21/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5500  train_loss: 2.1381  time: 0.22s        
Epoch: 21/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.2542  time: 0.22s        
Epoch: 21/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5000  train_loss: 2.6549  time: 0.21s        
Epoch: 21/30, Iter: 13/14 [==================  ]  train_accuracy: 0.4615  train_loss: 3.0023  time: 0.21s        
Epoch: 21/30, Iter: 14/14 [====================]  train_accuracy: 0.4643  train_loss: 2.8580  time: 0.22s        
This epoch: 3.43s; per epoch: 13.14s; elapsed: 275.96s; remaining: 118.27s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 21/30, train_accuracy: 0.4643  train_loss: 2.8580  mean_accuracy: 0.7500  valid_mean_neg_loss: -1.3023  val_time: 24.32s    
Epoch: 22/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 0.7940  time: 0.29s        
Epoch: 22/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 0.6862  time: 0.22s        
Epoch: 22/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 2.8217  time: 0.41s        
Epoch: 22/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 2.7357  time: 0.23s        
Epoch: 22/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 2.1888  time: 0.22s        
Epoch: 22/30, Iter: 6/14 [========            ]  train_accuracy: 0.4167  train_loss: 3.0196  time: 0.22s        
Epoch: 22/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 2.6358  time: 0.23s        
Epoch: 22/30, Iter: 8/14 [===========         ]  train_accuracy: 0.4375  train_loss: 3.2019  time: 0.22s        
Epoch: 22/30, Iter: 9/14 [============        ]  train_accuracy: 0.3889  train_loss: 3.5593  time: 0.21s        
Epoch: 22/30, Iter: 10/14 [==============      ]  train_accuracy: 0.3500  train_loss: 3.9208  time: 0.22s        
Epoch: 22/30, Iter: 11/14 [===============     ]  train_accuracy: 0.3182  train_loss: 4.2078  time: 0.22s        
Epoch: 22/30, Iter: 12/14 [=================   ]  train_accuracy: 0.3333  train_loss: 4.1361  time: 0.21s        
Epoch: 22/30, Iter: 13/14 [==================  ]  train_accuracy: 0.3462  train_loss: 3.8763  time: 0.22s        
Epoch: 22/30, Iter: 14/14 [====================]  train_accuracy: 0.3214  train_loss: 4.0902  time: 0.22s        
This epoch: 3.37s; per epoch: 12.70s; elapsed: 279.33s; remaining: 101.58s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 22/30, train_accuracy: 0.3214  train_loss: 4.0902  mean_accuracy: 0.7500  valid_mean_neg_loss: -0.8253  val_time: 24.56s    
Epoch: 23/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.5063  time: 0.31s        
Epoch: 23/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.0441  time: 0.24s        
Epoch: 23/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 2.0298  time: 0.35s        
Epoch: 23/30, Iter: 4/14 [=====               ]  train_accuracy: 0.7500  train_loss: 1.5226  time: 0.23s        
Epoch: 23/30, Iter: 5/14 [=======             ]  train_accuracy: 0.8000  train_loss: 1.2182  time: 0.22s        
Epoch: 23/30, Iter: 6/14 [========            ]  train_accuracy: 0.8333  train_loss: 1.0153  time: 0.22s        
Epoch: 23/30, Iter: 7/14 [==========          ]  train_accuracy: 0.7857  train_loss: 1.2712  time: 0.25s        
Epoch: 23/30, Iter: 8/14 [===========         ]  train_accuracy: 0.7500  train_loss: 1.1985  time: 0.22s        
Epoch: 23/30, Iter: 9/14 [============        ]  train_accuracy: 0.6667  train_loss: 1.8652  time: 0.21s        
Epoch: 23/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6500  train_loss: 1.8120  time: 0.25s        
Epoch: 23/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6364  train_loss: 1.7035  time: 0.22s        
Epoch: 23/30, Iter: 12/14 [=================   ]  train_accuracy: 0.6667  train_loss: 1.5683  time: 0.21s        
Epoch: 23/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6923  train_loss: 1.4477  time: 0.21s        
Epoch: 23/30, Iter: 14/14 [====================]  train_accuracy: 0.6786  train_loss: 1.6058  time: 0.24s        
This epoch: 3.40s; per epoch: 12.29s; elapsed: 282.73s; remaining: 86.05s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 23/30, train_accuracy: 0.6786  train_loss: 1.6058  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0473  val_time: 24.40s    
Epoch: 24/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 2.3390  time: 0.30s        
Epoch: 24/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.7233  time: 0.22s        
Epoch: 24/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.8158  time: 0.37s        
Epoch: 24/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 2.2701  time: 0.30s        
Epoch: 24/30, Iter: 5/14 [=======             ]  train_accuracy: 0.6000  train_loss: 2.4002  time: 0.24s        
Epoch: 24/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 3.1826  time: 0.23s        
Epoch: 24/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 2.8487  time: 0.22s        
Epoch: 24/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5000  train_loss: 2.6676  time: 0.22s        
Epoch: 24/30, Iter: 9/14 [============        ]  train_accuracy: 0.5000  train_loss: 2.5156  time: 0.23s        
Epoch: 24/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5500  train_loss: 2.2641  time: 0.22s        
Epoch: 24/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5000  train_loss: 2.7109  time: 0.22s        
Epoch: 24/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5417  train_loss: 2.4851  time: 0.23s        
Epoch: 24/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5000  train_loss: 2.8139  time: 0.27s        
Epoch: 24/30, Iter: 14/14 [====================]  train_accuracy: 0.5000  train_loss: 2.7434  time: 0.24s        
This epoch: 3.51s; per epoch: 11.93s; elapsed: 286.24s; remaining: 71.56s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 24/30, train_accuracy: 0.5000  train_loss: 2.7434  mean_accuracy: 0.7500  valid_mean_neg_loss: -1.0198  val_time: 24.24s    
Epoch: 25/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.6369  time: 0.28s        
Epoch: 25/30, Iter: 2/14 [==                  ]  train_accuracy: 0.2500  train_loss: 5.3843  time: 0.22s        
Epoch: 25/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 4.3561  time: 0.22s        
Epoch: 25/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 4.1173  time: 0.24s        
Epoch: 25/30, Iter: 5/14 [=======             ]  train_accuracy: 0.4000  train_loss: 3.5306  time: 0.26s        
Epoch: 25/30, Iter: 6/14 [========            ]  train_accuracy: 0.4167  train_loss: 3.5107  time: 0.22s        
Epoch: 25/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 3.0093  time: 0.23s        
Epoch: 25/30, Iter: 8/14 [===========         ]  train_accuracy: 0.4375  train_loss: 3.4769  time: 0.25s        
Epoch: 25/30, Iter: 9/14 [============        ]  train_accuracy: 0.4444  train_loss: 3.4639  time: 0.22s        
Epoch: 25/30, Iter: 10/14 [==============      ]  train_accuracy: 0.4500  train_loss: 3.4370  time: 0.22s        
Epoch: 25/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5000  train_loss: 3.1247  time: 0.27s        
Epoch: 25/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5000  train_loss: 3.1564  time: 0.23s        
Epoch: 25/30, Iter: 13/14 [==================  ]  train_accuracy: 0.4615  train_loss: 3.4432  time: 0.22s        
Epoch: 25/30, Iter: 14/14 [====================]  train_accuracy: 0.5000  train_loss: 3.1973  time: 0.23s        
This epoch: 3.32s; per epoch: 11.58s; elapsed: 289.56s; remaining: 57.91s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 25/30, train_accuracy: 0.5000  train_loss: 3.1973  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0218  val_time: 24.48s    
Epoch: 26/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 2.8623  time: 0.26s        
Epoch: 26/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 1.4317  time: 0.22s        
Epoch: 26/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 2.0371  time: 0.23s        
Epoch: 26/30, Iter: 4/14 [=====               ]  train_accuracy: 0.6250  train_loss: 1.7207  time: 0.24s        
Epoch: 26/30, Iter: 5/14 [=======             ]  train_accuracy: 0.6000  train_loss: 1.8028  time: 0.23s        
Epoch: 26/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 2.7047  time: 0.24s        
Epoch: 26/30, Iter: 7/14 [==========          ]  train_accuracy: 0.4286  train_loss: 3.3295  time: 0.25s        
Epoch: 26/30, Iter: 8/14 [===========         ]  train_accuracy: 0.3750  train_loss: 3.6416  time: 0.23s        
Epoch: 26/30, Iter: 9/14 [============        ]  train_accuracy: 0.4444  train_loss: 3.2371  time: 0.22s        
Epoch: 26/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5000  train_loss: 2.9135  time: 0.22s        
Epoch: 26/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5455  train_loss: 2.6487  time: 0.22s        
Epoch: 26/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 2.4289  time: 0.23s        
Epoch: 26/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6154  train_loss: 2.2421  time: 0.21s        
Epoch: 26/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 2.2878  time: 0.26s        
This epoch: 3.27s; per epoch: 11.26s; elapsed: 292.83s; remaining: 45.05s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 26/30, train_accuracy: 0.6071  train_loss: 2.2878  mean_accuracy: 0.8750  valid_mean_neg_loss: -0.1389  val_time: 25.07s    
Epoch: 27/30, Iter: 1/14 [=                   ]  train_accuracy: 1.0000  train_loss: 0.0008  time: 0.31s        
Epoch: 27/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 3.5950  time: 0.22s        
Epoch: 27/30, Iter: 3/14 [====                ]  train_accuracy: 0.3333  train_loss: 3.6214  time: 0.40s        
Epoch: 27/30, Iter: 4/14 [=====               ]  train_accuracy: 0.5000  train_loss: 2.7162  time: 0.21s        
Epoch: 27/30, Iter: 5/14 [=======             ]  train_accuracy: 0.6000  train_loss: 2.1732  time: 0.26s        
Epoch: 27/30, Iter: 6/14 [========            ]  train_accuracy: 0.5833  train_loss: 1.9371  time: 0.26s        
Epoch: 27/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5000  train_loss: 2.5058  time: 0.23s        
Epoch: 27/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5000  train_loss: 2.6120  time: 0.23s        
Epoch: 27/30, Iter: 9/14 [============        ]  train_accuracy: 0.4444  train_loss: 3.0084  time: 0.25s        
Epoch: 27/30, Iter: 10/14 [==============      ]  train_accuracy: 0.5000  train_loss: 2.7076  time: 0.23s        
Epoch: 27/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5000  train_loss: 2.6641  time: 0.23s        
Epoch: 27/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5000  train_loss: 2.7444  time: 0.22s        
Epoch: 27/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 2.5334  time: 0.22s        
Epoch: 27/30, Iter: 14/14 [====================]  train_accuracy: 0.5357  train_loss: 2.4419  time: 0.22s        
This epoch: 3.50s; per epoch: 10.98s; elapsed: 296.33s; remaining: 32.93s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 27/30, train_accuracy: 0.5357  train_loss: 2.4419  mean_accuracy: 1.0000  valid_mean_neg_loss: -0.0272  val_time: 24.58s    
Epoch: 28/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 1.8467  time: 0.29s        
Epoch: 28/30, Iter: 2/14 [==                  ]  train_accuracy: 0.5000  train_loss: 2.1349  time: 0.23s        
Epoch: 28/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 1.6494  time: 0.37s        
Epoch: 28/30, Iter: 4/14 [=====               ]  train_accuracy: 0.5000  train_loss: 1.5365  time: 0.21s        
Epoch: 28/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 1.3812  time: 0.27s        
Epoch: 28/30, Iter: 6/14 [========            ]  train_accuracy: 0.5833  train_loss: 1.1511  time: 0.22s        
Epoch: 28/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 1.1183  time: 0.22s        
Epoch: 28/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 0.9787  time: 0.26s        
Epoch: 28/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 1.2226  time: 0.22s        
Epoch: 28/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6000  train_loss: 1.2985  time: 0.26s        
Epoch: 28/30, Iter: 11/14 [===============     ]  train_accuracy: 0.5909  train_loss: 1.5083  time: 0.23s        
Epoch: 28/30, Iter: 12/14 [=================   ]  train_accuracy: 0.5833  train_loss: 1.4522  time: 0.23s        
Epoch: 28/30, Iter: 13/14 [==================  ]  train_accuracy: 0.5385  train_loss: 1.8668  time: 0.22s        
Epoch: 28/30, Iter: 14/14 [====================]  train_accuracy: 0.5357  train_loss: 1.9917  time: 0.21s        
This epoch: 3.45s; per epoch: 10.71s; elapsed: 299.79s; remaining: 21.41s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 28/30, train_accuracy: 0.5357  train_loss: 1.9917  mean_accuracy: 0.8750  valid_mean_neg_loss: -0.2523  val_time: 24.24s    
Epoch: 29/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 1.7552  time: 0.32s        
Epoch: 29/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 0.8981  time: 0.23s        
Epoch: 29/30, Iter: 3/14 [====                ]  train_accuracy: 0.6667  train_loss: 1.5682  time: 0.41s        
Epoch: 29/30, Iter: 4/14 [=====               ]  train_accuracy: 0.7500  train_loss: 1.1779  time: 0.22s        
Epoch: 29/30, Iter: 5/14 [=======             ]  train_accuracy: 0.7000  train_loss: 1.6689  time: 0.22s        
Epoch: 29/30, Iter: 6/14 [========            ]  train_accuracy: 0.6667  train_loss: 1.8476  time: 0.24s        
Epoch: 29/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 2.2352  time: 0.22s        
Epoch: 29/30, Iter: 8/14 [===========         ]  train_accuracy: 0.5625  train_loss: 2.2515  time: 0.23s        
Epoch: 29/30, Iter: 9/14 [============        ]  train_accuracy: 0.6111  train_loss: 2.0014  time: 0.23s        
Epoch: 29/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6500  train_loss: 1.8014  time: 0.23s        
Epoch: 29/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6364  train_loss: 1.7735  time: 0.25s        
Epoch: 29/30, Iter: 12/14 [=================   ]  train_accuracy: 0.6667  train_loss: 1.6258  time: 0.22s        
Epoch: 29/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6154  train_loss: 2.0508  time: 0.22s        
Epoch: 29/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 2.1284  time: 0.22s        
This epoch: 3.46s; per epoch: 10.46s; elapsed: 303.25s; remaining: 10.46s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 29/30, train_accuracy: 0.6071  train_loss: 2.1284  mean_accuracy: 0.7500  valid_mean_neg_loss: -1.0636  val_time: 24.70s    
Epoch: 30/30, Iter: 1/14 [=                   ]  train_accuracy: 0.5000  train_loss: 3.3924  time: 0.27s        
Epoch: 30/30, Iter: 2/14 [==                  ]  train_accuracy: 0.7500  train_loss: 1.7295  time: 0.25s        
Epoch: 30/30, Iter: 3/14 [====                ]  train_accuracy: 0.5000  train_loss: 3.5250  time: 0.43s        
Epoch: 30/30, Iter: 4/14 [=====               ]  train_accuracy: 0.3750  train_loss: 4.3272  time: 0.23s        
Epoch: 30/30, Iter: 5/14 [=======             ]  train_accuracy: 0.5000  train_loss: 3.4619  time: 0.22s        
Epoch: 30/30, Iter: 6/14 [========            ]  train_accuracy: 0.5000  train_loss: 3.2905  time: 0.22s        
Epoch: 30/30, Iter: 7/14 [==========          ]  train_accuracy: 0.5714  train_loss: 2.8206  time: 0.22s        
Epoch: 30/30, Iter: 8/14 [===========         ]  train_accuracy: 0.6250  train_loss: 2.4682  time: 0.23s        
Epoch: 30/30, Iter: 9/14 [============        ]  train_accuracy: 0.6667  train_loss: 2.1940  time: 0.22s        
Epoch: 30/30, Iter: 10/14 [==============      ]  train_accuracy: 0.6500  train_loss: 2.2884  time: 0.24s        
Epoch: 30/30, Iter: 11/14 [===============     ]  train_accuracy: 0.6818  train_loss: 2.0805  time: 0.23s        
Epoch: 30/30, Iter: 12/14 [=================   ]  train_accuracy: 0.6667  train_loss: 2.2131  time: 0.22s        
Epoch: 30/30, Iter: 13/14 [==================  ]  train_accuracy: 0.6538  train_loss: 2.2362  time: 0.22s        
Epoch: 30/30, Iter: 14/14 [====================]  train_accuracy: 0.6071  train_loss: 2.5451  time: 0.22s        
This epoch: 3.44s; per epoch: 10.22s; elapsed: 306.69s; remaining: 0.00s; best metric: -0.0032119210809469223 at epoch 6
Epoch: 30/30, train_accuracy: 0.6071  train_loss: 2.5451  mean_accuracy: 0.7500  valid_mean_neg_loss: -1.5908  val_time: 24.85s    
Saved final model checkpoint at: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model_final.ckpt
Total time for fitting: 1053.77s
Best validation metric: -0.0032119210809469223 at epoch 6
2020-08-24 21:34:12,968 - nvmidl.utils.train_conf - INFO - Total Training Time 1165.5195095539093

Once the training completes, review the statistics that were generated, including training accuracy, training loss, mean accuracy, and the time it took for the fine-tuning to complete.

3.4 Export Model

After tuning the model, the result is not automatically saved nor exported. The trained model can be exported using the export.sh command. This produces the frozen graphs needed for inference and can be used by inference engines, like Clara Deploy, for a workflow deployment. This deployment pipeline can be then connected to medical imaging devices for research purposes.

Step 4 will use the exported model to perform some inference tests; but other uses of the model weights include constructing pipelines can be staged, that includes operators for the various phases of pre-transforms, inference, and creating results that are consumable by the medical imaging ecosystem (e.g., a DICOM-SR, a secondary capture image with burnt-in results, an HL7 or FHIR message). Understanding and mapping the architecture in any given environment is important to know when constructing this pipeline.

In [31]:

inputFile = 'experiments/covid19_3d_ct_classification-v2/commands/export.sh'
outputFile='experiments/covid19_3d_ct_classification-v2/commands/new_export.sh'
oldValues=["$MMAR_ROOT/models"]
newValues=["$MMAR_ROOT/model_finetuned"]
tid.adaptScript(inputFile,outputFile,oldValues,newValues)

Executing the Updated Export Procedure

In [32]:

!experiments/covid19_3d_ct_classification-v2/commands/new_export.sh

2020-08-24 21:38:11.511130: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:38:13.330150: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

Creating a regular frozen graph from Checkpoint at '/mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned' ...
Loaded meta graph file '/mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.ckpt.meta
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-08-24 21:38:14.104049: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-24 21:38:19.271885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:38:19.273001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 3a94:00:00.0
2020-08-24 21:38:19.274071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 6362:00:00.0
2020-08-24 21:38:19.275141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 8945:00:00.0
2020-08-24 21:38:19.275167: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:38:19.275248: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:38:19.275281: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:38:19.275310: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:38:19.277307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:38:19.278344: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:38:19.278392: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:38:19.286743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2020-08-24 21:38:19.303913: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2593990000 Hz
2020-08-24 21:38:19.306543: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x177c7b30 executing computations on platform Host. Devices:
2020-08-24 21:38:19.306567: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-08-24 21:38:19.861907: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x17362df0 executing computations on platform CUDA. Devices:
2020-08-24 21:38:19.861947: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:38:19.861957: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:38:19.861967: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:38:19.861977: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:38:19.863648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:38:19.864753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 3a94:00:00.0
2020-08-24 21:38:19.865826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 6362:00:00.0
2020-08-24 21:38:19.866891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 8945:00:00.0
2020-08-24 21:38:19.866926: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:38:19.866970: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:38:19.866992: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:38:19.867013: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:38:19.867036: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:38:19.867057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:38:19.867077: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:38:19.875269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2020-08-24 21:38:22.051581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 21:38:22.051639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2020-08-24 21:38:22.051660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N 
2020-08-24 21:38:22.051669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N 
2020-08-24 21:38:22.051675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N 
2020-08-24 21:38:22.051682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N 
2020-08-24 21:38:22.057615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14889 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 1a4c:00:00.0, compute capability: 7.0)
2020-08-24 21:38:22.059261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14889 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-16GB, pci bus id: 3a94:00:00.0, compute capability: 7.0)
2020-08-24 21:38:22.060783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 14889 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-16GB, pci bus id: 6362:00:00.0, compute capability: 7.0)
2020-08-24 21:38:22.062869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 14889 MB memory) -> physical GPU (device: 3, name: Tesla V100-PCIE-16GB, pci bus id: 8945:00:00.0, compute capability: 7.0)
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
Striping unused nodes with NVIDIA fix...
WARNING:tensorflow:From utils/strip_unused_lib.py:120: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
4485 ops in the final graph.
Creating TRT-optimized graph ...
WARNING:tensorflow:From apps/export.py:216: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

WARNING:tensorflow:From apps/export.py:217: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

2020-08-24 21:38:47.844356: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 4
2020-08-24 21:38:47.844501: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2020-08-24 21:38:47.846057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:38:47.847105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 3a94:00:00.0
2020-08-24 21:38:47.848146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 6362:00:00.0
2020-08-24 21:38:47.849210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 8945:00:00.0
2020-08-24 21:38:47.849265: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:38:47.849460: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:38:47.849493: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:38:47.849517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:38:47.849545: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:38:47.849570: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:38:47.849590: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:38:47.856865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2020-08-24 21:38:47.857033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 21:38:47.857048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2020-08-24 21:38:47.857058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N 
2020-08-24 21:38:47.857067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N 
2020-08-24 21:38:47.857079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N 
2020-08-24 21:38:47.857086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N 
2020-08-24 21:38:47.861476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14889 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 1a4c:00:00.0, compute capability: 7.0)
2020-08-24 21:38:47.862544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14889 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-16GB, pci bus id: 3a94:00:00.0, compute capability: 7.0)
2020-08-24 21:38:47.863604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 14889 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-16GB, pci bus id: 6362:00:00.0, compute capability: 7.0)
2020-08-24 21:38:47.864689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 14889 MB memory) -> physical GPU (device: 3, name: Tesla V100-PCIE-16GB, pci bus id: 8945:00:00.0, compute capability: 7.0)
2020-08-24 21:38:48.847379: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 2066 ops of 15 different types in the graph that are not converted to TensorRT: AvgPool3D, MaxPool3D, Add, Identity, Mul, Squeeze, Placeholder, NoOp, Const, Switch, Conv3D, Sub, PlaceholderWithDefault, Merge, Mean, (For more information see https://docs.nvidia.com/deeplearning/dgx/tf-trt-user-guide/index.html#supported-ops).
2020-08-24 21:38:49.042331: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:735] Number of TensorRT candidate segments: 0
2020-08-24 21:38:49.043828: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:38:49.104365: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:752] Optimization results for grappler item: tf_graph
2020-08-24 21:38:49.104417: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 3879 nodes (-606), 4903 edges (-606), time = 334.529ms.
2020-08-24 21:38:49.104429: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   layout: Graph size after: 3879 nodes (0), 4903 edges (0), time = 135.951ms.
2020-08-24 21:38:49.104438: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 3879 nodes (0), 4903 edges (0), time = 196.481ms.
2020-08-24 21:38:49.104446: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   TensorRTOptimizer: Graph size after: 3879 nodes (0), 4903 edges (0), time = 373.443ms.
Validating TRT-optimized graph ...
Saving the TRT-optimized graph ...
Frozen File Generated: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.fzn.pb
TRT File Generated: /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/../model_finetuned/model.trt.pb

Step 4: Reclassifying/inference with fine-tuned model

We first run inference on the pre-trained model as is, with new validation data to get a sense for its accuracy with new data. We want to infer the labels of the data we left out for testing using the fine-tuned model. Clara has an infer.sh command to do so that needs to be updated to point to the new model. Same as last time, this step has four parts:

Update the configuration files so that Clara could find the data we want to make inferences on
Update the infer command to point to the new configuration files and to the new model
Run the infer.sh command
Evaluate the results

4.1 Update Configuration File

The “environment.json” must be updated to refer to the data index file, the new data source and the new_model The original “environment.json” is adapted into the new “infer_finetuned_environment.json

In [33]:

inputFile ='experiments/covid19_3d_ct_classification-v2/config/environment.json'
outputFile = 'experiments/covid19_3d_ct_classification-v2/config/infer_finetuned_environment.json'
tid.adaptJSONFile(inputFile,outputFile,['DATASET_JSON','DATA_ROOT','MMAR_EVAL_OUTPUT_PATH','MMAR_CKPT_DIR'],[testingFile,kaggledata_dir,'eval_finetuned','model_finetuned'])

4.2 Update Infer.sh Command

The “infer,sh” command needs to be updated to refer to the updated “infer_finetuned_environment.json” config file We use the original “infer,sh” to create an updated version “finetuned_infer,sh”

In [34]:

inputFile = 'experiments/covid19_3d_ct_classification-v2/commands/infer.sh'
outputFile='experiments/covid19_3d_ct_classification-v2/commands/finetuned_infer.sh'
oldValues=["$MMAR_ROOT/config/dataset_0.json","config/environment.json"]
newValues=[testingFile,"config/infer_finetuned_environment.json"]
tid.adaptScript(inputFile,outputFile,oldValues,newValues)

4.3 Execute finetuned_Infer.sh Command

In [35]:

!experiments/covid19_3d_ct_classification-v2/commands/finetuned_infer.sh

MMAR_ROOT set to /mnt/batch/tasks/shared/LS_root/jobs/tutorialtesta1/azureml/claratest_1598282135_c710b479/mounts/workspaceblobstore/clara/experiments/covid19_3d_ct_classification-v2/commands/..
2020-08-24 21:55:39.022775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[42824,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 65f2bbe182cf4de3b0542d8b3ccfb74b000000

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
Using TensorFlow backend.
2020-08-24 21:55:41,149 - nvmidl.utils.train_conf - INFO - Automatic Mixed Precision status: Disabled
Previously evaluated: 0 ; To be evaluated: 3
2020-08-24 21:55:42.130851: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2593990000 Hz
2020-08-24 21:55:42.133473: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x29ff9b0 executing computations on platform Host. Devices:
2020-08-24 21:55:42.133500: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-08-24 21:55:42.135796: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-24 21:55:48.329207: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2a802a0 executing computations on platform CUDA. Devices:
2020-08-24 21:55:48.329250: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:55:48.329262: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:55:48.329271: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:55:48.329280: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-08-24 21:55:48.331098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 1a4c:00:00.0
2020-08-24 21:55:48.332180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 3a94:00:00.0
2020-08-24 21:55:48.333298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 6362:00:00.0
2020-08-24 21:55:48.334380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 8945:00:00.0
2020-08-24 21:55:48.334421: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:55:48.334542: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:55:48.334585: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-08-24 21:55:48.334621: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-08-24 21:55:48.338633: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-08-24 21:55:48.340767: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-08-24 21:55:48.340826: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-24 21:55:48.349233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2020-08-24 21:55:48.349273: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-08-24 21:55:50.606106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 21:55:50.606163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2020-08-24 21:55:50.606180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N 
2020-08-24 21:55:50.606189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N 
2020-08-24 21:55:50.606196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N 
2020-08-24 21:55:50.606204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N 
2020-08-24 21:55:50.612224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14889 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 1a4c:00:00.0, compute capability: 7.0)
2020-08-24 21:55:50.613974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14889 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-16GB, pci bus id: 3a94:00:00.0, compute capability: 7.0)
2020-08-24 21:55:50.615914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 14889 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-16GB, pci bus id: 6362:00:00.0, compute capability: 7.0)
2020-08-24 21:55:50.617415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 14889 MB memory) -> physical GPU (device: 3, name: Tesla V100-PCIE-16GB, pci bus id: 8945:00:00.0, compute capability: 7.0)
2020-08-24 21:55:57.301291: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-08-24 21:55:57.505595: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Batch 1 / 3: 8.13s; pre-process: 4.16s; infer: 3.96s; post-process: 0.00s
Batch 2 / 3: 1.53s; pre-process: 1.45s; infer: 0.08s; post-process: 0.00s
Batch 3 / 3: 4.38s; pre-process: 4.34s; infer: 0.04s; post-process: 0.00s


Total Inference Time: 4.085515737533569s
2020-08-24 21:56:04,797 - nvmidl.utils.train_conf - INFO - Total Evaluation Time 24.1669602394104

4.4 Inspect Inference Results with the Tuned Model

The system stored the inferred predictions in file: eval_finetuned/preds_model.csv.

The following lines retrieve the probabilities produced by the “finetuned_infer,sh” command and estimates the predicted labels (1:COVID, 0:NO COVID).

In [36]:

labels=['No COVID', 'COVID']
predictions_file = 'experiments/covid19_3d_ct_classification-v2/eval_finetuned/preds_model.csv'
results,pred_labels = tid.processCTScanPredictions(predictions_file,labels,'COVID')

Inference Results on Tuned Model for Image Set 1 (Positive Example)

In [37]:

y_pred[0] = pred_labels[testing[0]]['COVID']
results[testing[0]]

Out[37]:

'No COVID = 2.7362875e-08; COVID = 1.0'

The model correctly classified the example as belonging to a COVID case.

Inference Results on Tuned Model for Image Set 2 (Positive Example, Different Units)

In [38]:

y_pred[1] = pred_labels[testing[1]]['COVID']
results[testing[1]]

Out[38]:

'No COVID = 0.025162333999999998; COVID = 0.97483766'

This is significant; this differs from the original model. With the tuned model, the model correctly classifies the example as a COVID case. The fine tuning has made a difference.

Inference Results on Tuned Model for Image Set 3 (Negative Example)

In [39]:

y_pred[2] = pred_labels[testing[2]]['COVID']
results[testing[2]]

Out[39]:

'No COVID = 0.9962755999999999; COVID = 0.0037243334000000004'

The model correctly classified the example as NOT belonging to a COVID case

4.5 Computing the Average COVID Classification Precision over all examples with the tuned model

In [40]:

print('Expected Labels')
print(y_test)
print("Predicted Labels")
print(y_pred)
average_precision = average_precision_score(y_test, y_pred)
print('Average Precision')
print(average_precision)

Expected Labels
[1. 1. 0.]
Predicted Labels
[1. 1. 0.]
Average Precision
1.0

Notice that the average precision is now much higher, ; so the fine tune mechanism has succeeded to fine tunning the model to account for the peculiarities of the new data.

The purpose of this notebook was to showcase the ease of use of the Clara Train SDK and how it could be used to fine tune a state of the art model, trained with global data, using your own data. The procedure depicted in this notebook is not a rigorous data science experiment, as we only had 40 new examples, but rather to illustrate how to use the tools provided on the Clara Train SDK. Now that you have an environment ready, you can train and fine-tune models, using any of the pre-trained models with your own data. Your platform is ready and equipped with advanced features like federated learning (AMP), automated mixed precision, and AutoML.

This notebook is also meant to be used in conjuction with the AzureML-NGC Set Up Mechanism to exemplify how easy is to set up cool application on AzureML using NVIDIA NGC content

Clara Tutorial

This material was uploaded automatically through the use of the NGC-AzureML Quick Launch Toolkit, as described here. It also was loaded into the Compute Cluster a Jupyter Notebook based self-paced tutorial to go deeper into all the details and extra functionality that Clara offers such as AutoML and Multi-GPU Processing. The start of the tutorial could be found here: workspaceblobstore/clara/MMARs/GettingStarted/GettingStarted.ipynb

In [ ]: