Notebook for Medical Image Segmentation

NVIDIA

Resource

NVIDIA

Notebook for Medical Image Segmentation

3D-UNet medical image segmentation allows for seamless segmentation of 3D volumes, with high accuracy and performance.

Back to File Browser

unet3d-v7.ipynb

Data Preprocessing

Data can be obtained by registering on Brain Tumor Segmentation 2019 dataset website at this address "https://www.med.upenn.edu/cbica/brats-2019/". The data should be downloaded and placed where /data in the container is mounted.

The dataset/preprocess_data.py script will convert the raw data into tfrecord format used for training and evaluation.

In this dastaset, ample multi-institutional routine clinically-acquired pre-operative multimodal MRI scans of glioblastoma (GBM/HGG) and lower grade glioma (LGG), with pathologically confirmed diagnosis and available OS, are provided as the training, validation and testing data for 2019 BraTS challenge. By running the next two cells you can see the folders inside the dataset and the files that you can see inside these folders. the format of images is nii.gz. NIfTI is a type of file format for neuroimaging.

In [1]:

! ls /data/MICCAI_BraTS_2019_Data_Training

HGG  LGG  content.json	data_splits  name_mapping.csv  survival_data.csv

In [2]:

! ls /data/MICCAI_BraTS_2019_Data_Training/HGG/BraTS19_2013_10_1

BraTS19_2013_10_1_flair.nii.gz	BraTS19_2013_10_1_t1ce.nii.gz
BraTS19_2013_10_1_seg.nii.gz	BraTS19_2013_10_1_t2.nii.gz
BraTS19_2013_10_1_t1.nii.gz

check the image

To check the dataset you can use nibabel. By running the next three cells you can install nibabel using pip install, choose an image from dataset and plot the chosen 3d image from dataset using matplotlib. You can check other images in the dataset by changing the address of image in the code.

In [3]:

 pip install nibabel;

Requirement already satisfied: nibabel in /usr/local/lib/python3.6/dist-packages (3.2.1)
Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from nibabel) (1.17.3)
Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.6/dist-packages (from nibabel) (20.4)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging>=14.3->nibabel) (1.13.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging>=14.3->nibabel) (2.4.7)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m
Note: you may need to restart the kernel to use updated packages.

In [6]:

import nibabel as nib
import matplotlib.pyplot as plt


img_arr = nib.load('/data/MICCAI_BraTS_2019_Data_Training/HGG/BraTS19_2013_10_1/BraTS19_2013_10_1_flair.nii.gz').get_data()
# plt.imshow(img_arr)
# plt.pause(3)

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: DeprecationWarning: get_data() is deprecated in favor of get_fdata(), which has a more predictable return type. To obtain get_data() behavior going forward, use numpy.asanyarray(img.dataobj).

* deprecated from version: 3.0
* Will raise <class 'nibabel.deprecator.ExpiredDeprecationError'> as of version: 5.0
  """

In [7]:

def show_plane(ax, plane, cmap="gray", title=None):
    ax.imshow(plane, cmap=cmap)
    ax.axis("off")

    if title:
        ax.set_title(title)
        
(n_plane, n_row, n_col) = img_arr.shape
_, (a, b, c) = plt.subplots(ncols=3, figsize=(15, 5))

show_plane(a, img_arr[n_plane // 2], title=f'Plane = {n_plane // 2}')
show_plane(b, img_arr[:, n_row // 2, :], title=f'Row = {n_row // 2}')
show_plane(c, img_arr[:, :, n_col // 2], title=f'Column = {n_col // 2}')

preprocess the dataset

In the next cell we used preprocess-data script to preprocess the downloaded dataset. The final format of the processed images is tfrecord. To read data efficiently it can be helpful to serialize your data and store it in a set of files (100–200MB each) that can each be read linearly. This is especially true if the data is being streamed over a network. This can also be useful for caching any data-preprocessing. The TFRecord format is a simple format for storing a sequence of binary records. Using tfrecords can speed things up especially if the bottleneck of the training is loading the data.

In [8]:

! python dataset/preprocess_data.py -i /data/MICCAI_BraTS_2019_Data_Training -o /data/preprocessed -v

2021-05-03 18:25:01.647501: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
1/84 tfrecord files created
2/84 tfrecord files created
3/84 tfrecord files created
4/84 tfrecord files created
5/84 tfrecord files created
6/84 tfrecord files created
7/84 tfrecord files created
8/84 tfrecord files created
9/84 tfrecord files created
10/84 tfrecord files created
11/84 tfrecord files created
12/84 tfrecord files created
13/84 tfrecord files created
14/84 tfrecord files created
15/84 tfrecord files created
16/84 tfrecord files created
17/84 tfrecord files created
18/84 tfrecord files created
19/84 tfrecord files created
20/84 tfrecord files created
21/84 tfrecord files created
22/84 tfrecord files created
23/84 tfrecord files created
24/84 tfrecord files created
25/84 tfrecord files created
26/84 tfrecord files created
27/84 tfrecord files created
28/84 tfrecord files created
29/84 tfrecord files created
30/84 tfrecord files created
31/84 tfrecord files created
32/84 tfrecord files created
33/84 tfrecord files created
34/84 tfrecord files created
35/84 tfrecord files created
36/84 tfrecord files created
37/84 tfrecord files created
38/84 tfrecord files created
39/84 tfrecord files created
40/84 tfrecord files created
41/84 tfrecord files created
42/84 tfrecord files created
43/84 tfrecord files created
44/84 tfrecord files created
45/84 tfrecord files created
46/84 tfrecord files created
47/84 tfrecord files created
48/84 tfrecord files created
49/84 tfrecord files created
50/84 tfrecord files created
51/84 tfrecord files created
52/84 tfrecord files created
53/84 tfrecord files created
54/84 tfrecord files created
55/84 tfrecord files created
56/84 tfrecord files created
57/84 tfrecord files created
58/84 tfrecord files created
59/84 tfrecord files created
60/84 tfrecord files created
61/84 tfrecord files created
62/84 tfrecord files created
63/84 tfrecord files created
64/84 tfrecord files created
65/84 tfrecord files created
66/84 tfrecord files created
67/84 tfrecord files created
68/84 tfrecord files created
69/84 tfrecord files created
70/84 tfrecord files created
71/84 tfrecord files created
72/84 tfrecord files created
73/84 tfrecord files created
74/84 tfrecord files created
75/84 tfrecord files created
76/84 tfrecord files created
77/84 tfrecord files created
78/84 tfrecord files created
79/84 tfrecord files created
80/84 tfrecord files created
81/84 tfrecord files created
82/84 tfrecord files created
83/84 tfrecord files created
84/84 tfrecord files created

The result of Preprocessed Script

the next cell shows the result of preprocessed script

In [9]:

! ls /data/preprocessed

volume-0.tfrecord   volume-28.tfrecord	volume-47.tfrecord  volume-66.tfrecord
volume-1.tfrecord   volume-29.tfrecord	volume-48.tfrecord  volume-67.tfrecord
volume-10.tfrecord  volume-3.tfrecord	volume-49.tfrecord  volume-68.tfrecord
volume-11.tfrecord  volume-30.tfrecord	volume-5.tfrecord   volume-69.tfrecord
volume-12.tfrecord  volume-31.tfrecord	volume-50.tfrecord  volume-7.tfrecord
volume-13.tfrecord  volume-32.tfrecord	volume-51.tfrecord  volume-70.tfrecord
volume-14.tfrecord  volume-33.tfrecord	volume-52.tfrecord  volume-71.tfrecord
volume-15.tfrecord  volume-34.tfrecord	volume-53.tfrecord  volume-72.tfrecord
volume-16.tfrecord  volume-35.tfrecord	volume-54.tfrecord  volume-73.tfrecord
volume-17.tfrecord  volume-36.tfrecord	volume-55.tfrecord  volume-74.tfrecord
volume-18.tfrecord  volume-37.tfrecord	volume-56.tfrecord  volume-75.tfrecord
volume-19.tfrecord  volume-38.tfrecord	volume-57.tfrecord  volume-76.tfrecord
volume-2.tfrecord   volume-39.tfrecord	volume-58.tfrecord  volume-77.tfrecord
volume-20.tfrecord  volume-4.tfrecord	volume-59.tfrecord  volume-78.tfrecord
volume-21.tfrecord  volume-40.tfrecord	volume-6.tfrecord   volume-79.tfrecord
volume-22.tfrecord  volume-41.tfrecord	volume-60.tfrecord  volume-8.tfrecord
volume-23.tfrecord  volume-42.tfrecord	volume-61.tfrecord  volume-80.tfrecord
volume-24.tfrecord  volume-43.tfrecord	volume-62.tfrecord  volume-81.tfrecord
volume-25.tfrecord  volume-44.tfrecord	volume-63.tfrecord  volume-82.tfrecord
volume-26.tfrecord  volume-45.tfrecord	volume-64.tfrecord  volume-83.tfrecord
volume-27.tfrecord  volume-46.tfrecord	volume-65.tfrecord  volume-9.tfrecord

Move some of the preprocessed images to a new folder to use for testing the model. We made a folder called "preprocessed_test" and moved some preprocessed image in that folder. Later we will use this test folder for model prediction.

Training using default parameters

After the Docker container is launched, the training of a single fold (fold 0) with the default hyperparameters (for example 1/8 GPUs TF-AMP/FP32/TF32) can be started with:

bash examples/unet3d_train_single{_TF-AMP}.sh <number/of/gpus> <path/to/dataset> <path/to/checkpoint> <batch/size>

For example, to run with 32-bit precision (FP32 or TF32) with batch size 2 on 1 GPU, simply use: bash examples/unet3d_train_single.sh 1 /data/preprocessed /results 2

to train a single fold with mixed precision (TF-AMP) with on 8 GPUs batch size 2 per GPU, use: bash examples/unet3d_train_single_TF-AMP.sh 8 /data/preprocessed /results 2

In [11]:

! bash examples/unet3d_train_single_TF-AMP.sh 4 /data/preprocessed /results 2

2021-05-03 18:37:59.636058: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 18:38:02.497590: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 18:38:02.516929: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 18:38:02.532883: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 18:38:02.541710: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,1]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,3]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,2]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,0]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,0]<stderr>:
[1,0]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,0]<stderr>:
[1,2]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,2]<stderr>:
[1,2]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,2]<stderr>:
[1,1]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,1]<stderr>:
[1,1]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,1]<stderr>:
[1,3]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,3]<stderr>:
[1,3]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 18:38:12.424063: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,0]<stderr>:2021-05-03 18:38:12.424066: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,1]<stderr>:2021-05-03 18:38:12.424070: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,3]<stderr>:2021-05-03 18:38:12.424067: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,1]<stderr>:2021-05-03 18:38:12.735624: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:12.735597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:12.735602: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:12.735602: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:12.740194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:04.0
[1,2]<stderr>:2021-05-03 18:38:12.740254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:04.0
[1,3]<stderr>:2021-05-03 18:38:12.740316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:04.0
[1,0]<stderr>:2021-05-03 18:38:12.740309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:12.740367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:04.0
[1,2]<stderr>:2021-05-03 18:38:12.740390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:12.740437: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:12.740491: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:12.744448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:05.0
[1,0]<stderr>:2021-05-03 18:38:12.744554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:12.744609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:05.0
[1,3]<stderr>:2021-05-03 18:38:12.744723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:05.0
[1,2]<stderr>:2021-05-03 18:38:12.744727: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:12.744820: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:12.744849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:05.0
[1,1]<stderr>:2021-05-03 18:38:12.744940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:12.749125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:06.0
[1,0]<stderr>:2021-05-03 18:38:12.749222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:12.749253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:06.0
[1,2]<stderr>:2021-05-03 18:38:12.749353: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:12.749383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:06.0
[1,3]<stderr>:2021-05-03 18:38:12.749470: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:12.749527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:06.0
[1,1]<stderr>:2021-05-03 18:38:12.749620: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:12.753692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:07.0
[1,0]<stderr>:2021-05-03 18:38:12.753728: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 18:38:12.753879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:07.0
[1,2]<stderr>:2021-05-03 18:38:12.753925: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 18:38:12.753986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:07.0
[1,3]<stderr>:2021-05-03 18:38:12.754023: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 18:38:12.754035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:07.0
[1,1]<stderr>:2021-05-03 18:38:12.754059: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 18:38:13.173223: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 18:38:13.173235: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 18:38:13.173300: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:2021-05-03 18:38:13.173331: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 18:38:13.376405: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 18:38:13.376398: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 18:38:13.376387: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,0]<stderr>:2021-05-03 18:38:13.376406: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,1]<stderr>:2021-05-03 18:38:13.471581: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,2]<stderr>:2021-05-03 18:38:13.471695: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 18:38:13.471729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 18:38:13.471855: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 18:38:13.696813: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,2]<stderr>:2021-05-03 18:38:13.696806: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 18:38:13.696813: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,1]<stderr>:2021-05-03 18:38:13.696872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 18:38:13.829522: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,2]<stderr>:2021-05-03 18:38:13.829629: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,3]<stderr>:2021-05-03 18:38:13.829630: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,1]<stderr>:2021-05-03 18:38:13.829728: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 18:38:13.829866: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 18:38:13.829912: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 18:38:13.829918: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 18:38:13.830009: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 18:38:13.830044: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.830043: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.830044: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.830156: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.834630: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.834691: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.834744: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.834798: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.839008: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.839185: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.839258: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.839384: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.843561: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.843738: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.843815: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.843935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.847972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.848139: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.848225: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.848352: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.852618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.852763: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.852857: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.852973: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.857175: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.857296: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.857375: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.857498: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.861767: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:13.861876: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:13.861971: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:13.862145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:13.866341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,0]<stderr>:2021-05-03 18:38:13.866520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,3]<stderr>:2021-05-03 18:38:13.866631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,1]<stderr>:2021-05-03 18:38:13.866687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,3]<stderr>:2021-05-03 18:38:21.076345: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,0]<stderr>:2021-05-03 18:38:21.076269: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,2]<stderr>:2021-05-03 18:38:21.076313: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,1]<stderr>:2021-05-03 18:38:21.076333: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,3]<stderr>:2021-05-03 18:38:21.094479: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6a39d30 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,3]<stderr>:2021-05-03 18:38:21.094507: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,0]<stderr>:2021-05-03 18:38:21.094466: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6acef80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,0]<stderr>:2021-05-03 18:38:21.094508: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,1]<stderr>:2021-05-03 18:38:21.094449: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5135120 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,1]<stderr>:2021-05-03 18:38:21.094508: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,2]<stderr>:2021-05-03 18:38:21.094545: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53bc6d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,2]<stderr>:2021-05-03 18:38:21.094572: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,1]<stderr>:2021-05-03 18:38:21.955944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:21.956109: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:21.956242: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:21.956220: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:21.960406: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x50d2080 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,1]<stderr>:2021-05-03 18:38:21.960443: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,2]<stderr>:2021-05-03 18:38:21.961311: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53e6bf0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,2]<stderr>:2021-05-03 18:38:21.961341: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,3]<stderr>:2021-05-03 18:38:21.961422: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6a18eb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,3]<stderr>:2021-05-03 18:38:21.961453: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,1]<stderr>:2021-05-03 18:38:21.961452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:21.961519: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x504ca90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,0]<stderr>:2021-05-03 18:38:21.961551: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,2]<stderr>:2021-05-03 18:38:21.961597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:21.961687: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:21.961781: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:21.965772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:05.0
[1,1]<stderr>:2021-05-03 18:38:21.965823: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 18:38:21.965921: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 18:38:21.965977: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,1]<stderr>:2021-05-03 18:38:21.965998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,1]<stderr>:2021-05-03 18:38:21.966018: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,1]<stderr>:2021-05-03 18:38:21.966038: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,1]<stderr>:2021-05-03 18:38:21.966058: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 18:38:21.966158: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:21.966280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:06.0
[1,2]<stderr>:2021-05-03 18:38:21.966321: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 18:38:21.966375: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 18:38:21.966395: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 18:38:21.966409: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,2]<stderr>:2021-05-03 18:38:21.966423: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,2]<stderr>:2021-05-03 18:38:21.966436: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,2]<stderr>:2021-05-03 18:38:21.966451: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 18:38:21.966466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:07.0
[1,3]<stderr>:2021-05-03 18:38:21.966504: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 18:38:21.966520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:04.0
[1,0]<stderr>:2021-05-03 18:38:21.966553: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 18:38:21.966557: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 18:38:21.966560: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:21.966572: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 18:38:21.966585: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 18:38:21.966606: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 18:38:21.966597: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,3]<stderr>:2021-05-03 18:38:21.966612: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 18:38:21.966620: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 18:38:21.966625: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 18:38:21.966633: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 18:38:21.966646: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 18:38:21.966659: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 18:38:21.966673: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 18:38:21.966704: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:21.966753: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:21.970603: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:21.971133: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:21.971444: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:21.971560: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:21.975098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 1
[1,2]<stderr>:2021-05-03 18:38:21.975511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 2
[1,3]<stderr>:2021-05-03 18:38:21.975732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 3
[1,0]<stderr>:2021-05-03 18:38:21.975780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
[1,3]<stderr>:2021-05-03 18:38:22.002285: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 18:38:22.002283: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 18:38:22.002298: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 18:38:22.002310: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 18:38:27.169116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,3]<stderr>:2021-05-03 18:38:27.169173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      3 
[1,3]<stderr>:2021-05-03 18:38:27.169182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 3:   N 
[1,3]<stderr>:2021-05-03 18:38:27.192299: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:27.193458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 18:38:27.194884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:07.0, compute capability: 7.0)
[1,0]<stderr>:2021-05-03 18:38:27.194962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,0]<stderr>:2021-05-03 18:38:27.195014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
[1,0]<stderr>:2021-05-03 18:38:27.195023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
[1,0]<stderr>:2021-05-03 18:38:27.195354: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:27.196478: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 18:38:27.197635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
[1,1]<stderr>:2021-05-03 18:38:27.198283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,1]<stderr>:2021-05-03 18:38:27.198323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      1 
[1,1]<stderr>:2021-05-03 18:38:27.198332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1:   N 
[1,1]<stderr>:2021-05-03 18:38:27.198585: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:27.199786: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 18:38:27.200915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:05.0, compute capability: 7.0)
[1,2]<stderr>:2021-05-03 18:38:27.263863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,2]<stderr>:2021-05-03 18:38:27.263918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      2 
[1,2]<stderr>:2021-05-03 18:38:27.263927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 2:   N 
[1,2]<stderr>:2021-05-03 18:38:27.264214: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:27.265337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 18:38:27.266342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:06.0, compute capability: 7.0)
[1,3]<stderr>:2021-05-03 18:38:28.487791: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:28.491518: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:28.497541: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:28.500490: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:28.500739: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:28.502477: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:28.505385: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:28.505570: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:29.920126: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:29.922718: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:29.952628: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:29.964633: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:30.795267: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:30.797938: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:30.811098: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:30.812049: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:30.812572: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:30.814539: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:30.815369: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:30.815819: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:31.027872: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,2]<stderr>:2021-05-03 18:38:31.058312: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,1]<stderr>:2021-05-03 18:38:31.061669: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,0]<stderr>:2021-05-03 18:38:31.064643: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,3]<stderr>:2021-05-03 18:38:31.090786: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:31.137500: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:31.138026: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:31.143527: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:31.329301: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:31.329554: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:31.355916: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:31.356186: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:31.357090: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:31.357335: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:31.359243: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:31.359492: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:31.524563: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:31.527020: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:31.568595: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:31.568902: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:31.570319: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:31.571836: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:31.572151: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:31.573765: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:31.820889: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:31.891100: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:31.896318: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:31.904754: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.033900: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:32.034266: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.043186: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.045366: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.046580: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.067909: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.075287: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.076604: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.077887: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:32.096224: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:32.096558: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.109605: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:32.110046: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.116612: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:32.117007: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.119140: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.122013: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.123577: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.126353: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.128932: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.130451: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.151124: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.158047: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.160297: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.161909: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.163467: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.167328: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.168800: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.170288: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:32.512871: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:32.517579: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:32.531622: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:32.534550: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:32.537992: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:32.540902: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 19465, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 10385, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 7785, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 22073, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 16089, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 12841, errno = 1
[1,0]<stderr>:2021-05-03 18:38:35.769078: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:35.772143: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.327278: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:37.327732: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.336876: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.339520: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.341018: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.367434: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.376702: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.378315: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.379914: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:37.676428: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:37.682909: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:[f267448f44e0:00855] Read -1, expected 17912, errno = 1
[1,2]<stderr>:[f267448f44e0:00854] Read -1, expected 17913, errno = 1
[1,1]<stderr>:[f267448f44e0:00853] Read -1, expected 17912, errno = 1
[1,3]<stderr>:2021-05-03 18:38:38.923555: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:38:38.930996: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:38.931353: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:38.931385: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:39.079394: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:39.079726: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:38:39.115781: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:39.116116: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:38:39.117188: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:39.117508: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:38:40.178688: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:38:40.207516: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 5131
[1,3]<stderr>:Recognized nodes available for conversion: 3292
[1,3]<stderr>:Total nodes converted: 118
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,3]<stderr>:Whitelisted nodes converted: 138
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,0]<stderr>:2021-05-03 18:38:40.258077: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:38:40.294839: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 5130
[1,0]<stderr>:Recognized nodes available for conversion: 3291
[1,0]<stderr>:Total nodes converted: 118
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,0]<stderr>:Whitelisted nodes converted: 138
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,2]<stderr>:2021-05-03 18:38:40.339302: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:38:40.344186: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:38:40.373960: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 5131
[1,2]<stderr>:Recognized nodes available for conversion: 3292
[1,2]<stderr>:Total nodes converted: 118
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,2]<stderr>:Whitelisted nodes converted: 138
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,1]<stderr>:2021-05-03 18:38:40.378618: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 5131
[1,1]<stderr>:Recognized nodes available for conversion: 3292
[1,1]<stderr>:Total nodes converted: 118
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,1]<stderr>:Whitelisted nodes converted: 138
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 18:38:45.487027: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 3071
[1,0]<stderr>:Recognized nodes available for conversion: 2197
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 99
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,1]<stderr>:2021-05-03 18:38:45.504805: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 3071
[1,1]<stderr>:Recognized nodes available for conversion: 2197
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 99
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,3]<stderr>:2021-05-03 18:38:45.506750: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 3071
[1,3]<stderr>:Recognized nodes available for conversion: 2197
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 99
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 18:38:45.625555: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 3071
[1,2]<stderr>:Recognized nodes available for conversion: 2197
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 99
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 18:38:46.784262: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 18:38:46.784293: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 18:38:46.784315: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 18:38:46.784263: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 18:38:55.616211: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 18:38:55.616106: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 18:38:55.616119: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 18:38:55.616093: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 4849, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 4849, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 4849, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 4585, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 5377, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 8457, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 11497, errno = 1
[1,0]<stderr>:[f267448f44e0:00852] Read -1, expected 7673, errno = 1
[1,1]<stderr>:[f267448f44e0:00853] Read -1, expected 5080, errno = 1
[1,2]<stderr>:[f267448f44e0:00854] Read -1, expected 5081, errno = 1
[1,3]<stderr>:[f267448f44e0:00855] Read -1, expected 5080, errno = 1
[1,3]<stderr>:2021-05-03 18:39:03.082652: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:39:03.091583: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:39:03.093083: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:39:03.093768: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:39:03.227820: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:39:03.232088: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:39:03.232218: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:39:03.236554: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stdout>:DLL 2021-05-03 18:39:03.332976 - (0,) total_loss_ref:0 : 4.109068 
[1,3]<stderr>:2021-05-03 18:39:04.396091: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:39:04.423751: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 5132
[1,3]<stderr>:Recognized nodes available for conversion: 3292
[1,3]<stderr>:Total nodes converted: 118
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,3]<stderr>:Whitelisted nodes converted: 138
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,0]<stderr>:2021-05-03 18:39:04.553055: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:39:04.554239: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:39:04.555775: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:39:04.586237: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 5131
[1,0]<stderr>:Recognized nodes available for conversion: 3291
[1,0]<stderr>:Total nodes converted: 118
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,0]<stderr>:Whitelisted nodes converted: 138
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,1]<stderr>:2021-05-03 18:39:04.587713: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 5132
[1,1]<stderr>:Recognized nodes available for conversion: 3292
[1,1]<stderr>:Total nodes converted: 118
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,1]<stderr>:Whitelisted nodes converted: 138
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,2]<stderr>:2021-05-03 18:39:04.589304: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 5132
[1,2]<stderr>:Recognized nodes available for conversion: 3292
[1,2]<stderr>:Total nodes converted: 118
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,2]<stderr>:Whitelisted nodes converted: 138
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 18:39:06.060449: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 3071
[1,3]<stderr>:Recognized nodes available for conversion: 2197
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 99
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,0]<stderr>:2021-05-03 18:39:06.594362: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 3071
[1,0]<stderr>:Recognized nodes available for conversion: 2197
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 99
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,2]<stderr>:2021-05-03 18:39:06.599544: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 3071
[1,2]<stderr>:Recognized nodes available for conversion: 2197
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 99
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,1]<stderr>:2021-05-03 18:39:06.619175: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 3071
[1,1]<stderr>:Recognized nodes available for conversion: 2197
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 99
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,3]<stderr>:2021-05-03 18:39:09.322442: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:39:09.334730: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:39:09.336270: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:39:09.341508: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:39:09.403949: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:39:09.439426: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:39:09.443353: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:39:09.477398: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:39:24.332338: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332396: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332404: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332410: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332416: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332422: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332427: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332433: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332438: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332444: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,3]<stderr>:2021-05-03 18:39:24.332497: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,3]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,3]<stderr>:2021-05-03 18:39:24.332505: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,3]<stderr>:2021-05-03 18:39:24.332510: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,3]<stderr>:2021-05-03 18:39:24.332515: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,3]<stderr>:2021-05-03 18:39:24.332538: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,3]<stderr>:2021-05-03 18:39:24.332558: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,1]<stderr>:2021-05-03 18:39:24.345588: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345636: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345645: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345651: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345657: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345664: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345670: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345676: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345682: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345688: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,1]<stderr>:2021-05-03 18:39:24.345719: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,1]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,1]<stderr>:2021-05-03 18:39:24.345730: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,1]<stderr>:2021-05-03 18:39:24.345735: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,1]<stderr>:2021-05-03 18:39:24.345740: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,1]<stderr>:2021-05-03 18:39:24.345745: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,1]<stderr>:2021-05-03 18:39:24.345753: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,0]<stderr>:2021-05-03 18:39:24.346551: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346597: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346606: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346613: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346620: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346626: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346633: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346640: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346646: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346653: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,0]<stderr>:2021-05-03 18:39:24.346687: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,0]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,0]<stderr>:2021-05-03 18:39:24.346727: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,0]<stderr>:2021-05-03 18:39:24.346732: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,0]<stderr>:2021-05-03 18:39:24.346738: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,0]<stderr>:2021-05-03 18:39:24.346743: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,0]<stderr>:2021-05-03 18:39:24.346751: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,2]<stderr>:2021-05-03 18:39:24.348314: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348361: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348369: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348376: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348382: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348401: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348407: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348413: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348419: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348425: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,2]<stderr>:2021-05-03 18:39:24.348475: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,2]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,2]<stderr>:2021-05-03 18:39:24.348517: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,2]<stderr>:2021-05-03 18:39:24.348535: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,2]<stderr>:2021-05-03 18:39:24.348540: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,2]<stderr>:2021-05-03 18:39:24.348545: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,2]<stderr>:2021-05-03 18:39:24.348553: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,3]<stderr>:2021-05-03 18:39:51.182677: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,1]<stderr>:2021-05-03 18:39:52.027371: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,2]<stderr>:2021-05-03 18:39:52.091022: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,0]<stderr>:2021-05-03 18:39:52.096749: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,3]<stderr>:2021-05-03 18:40:48.102016: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 18:40:48.130043: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 5131
[1,3]<stderr>:Recognized nodes available for conversion: 3292
[1,3]<stderr>:Total nodes converted: 118
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,3]<stderr>:Whitelisted nodes converted: 138
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,0]<stderr>:2021-05-03 18:40:48.256494: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 18:40:48.264494: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 18:40:48.272504: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 18:40:48.289848: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 5130
[1,0]<stderr>:Recognized nodes available for conversion: 3291
[1,0]<stderr>:Total nodes converted: 118
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,0]<stderr>:Whitelisted nodes converted: 138
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,1]<stderr>:2021-05-03 18:40:48.297939: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 5131
[1,1]<stderr>:Recognized nodes available for conversion: 3292
[1,1]<stderr>:Total nodes converted: 118
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,1]<stderr>:Whitelisted nodes converted: 138
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,2]<stderr>:2021-05-03 18:40:48.306184: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 5131
[1,2]<stderr>:Recognized nodes available for conversion: 3292
[1,2]<stderr>:Total nodes converted: 118
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,2]<stderr>:Whitelisted nodes converted: 138
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 18:40:49.742953: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 3071
[1,3]<stderr>:Recognized nodes available for conversion: 2197
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 99
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 18:40:50.191040: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 3071
[1,2]<stderr>:Recognized nodes available for conversion: 2197
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 99
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,1]<stderr>:2021-05-03 18:40:50.242315: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 3071
[1,1]<stderr>:Recognized nodes available for conversion: 2197
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 99
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 18:40:50.245542: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 3071
[1,0]<stderr>:Recognized nodes available for conversion: 2197
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 99
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,3]<stderr>:2021-05-03 18:40:51.484465: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:40:51.496010: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:40:51.518346: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:40:51.525612: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 18:40:51.569156: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:40:51.606892: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:40:51.653039: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:40:51.662477: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stdout>:DLL 2021-05-03 18:40:51.736556 - (100,) total_loss_ref:0 : 1.495921 
[1,3]<stderr>:2021-05-03 18:41:31.742652: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 18:41:31.743101: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 18:41:31.743401: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 18:41:31.743419: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stdout>:DLL 2021-05-03 18:41:42.743924 - (200,) total_loss_ref:0 : 1.5049638 
[1,0]<stdout>:DLL 2021-05-03 18:43:02.110847 - (300,) total_loss_ref:0 : 1.5299674 
[1,0]<stdout>:DLL 2021-05-03 18:43:53.300892 - (400,) total_loss_ref:0 : 1.4572296 
[1,0]<stdout>:DLL 2021-05-03 18:44:44.159887 - (500,) total_loss_ref:0 : 1.7901969 
[1,0]<stdout>:DLL 2021-05-03 18:45:34.982815 - (600,) total_loss_ref:0 : 1.3826758 
[1,0]<stdout>:DLL 2021-05-03 18:46:25.824182 - (700,) total_loss_ref:0 : 1.3859297 
[1,0]<stdout>:DLL 2021-05-03 18:47:16.509056 - (800,) total_loss_ref:0 : 1.2844148 
[1,0]<stdout>:DLL 2021-05-03 18:48:07.827334 - (900,) total_loss_ref:0 : 0.8467322 
[1,0]<stdout>:DLL 2021-05-03 18:48:59.052272 - (1000,) total_loss_ref:0 : 2.2208543 
[1,0]<stdout>:DLL 2021-05-03 18:49:49.835678 - (1100,) total_loss_ref:0 : 2.8669574 
[1,0]<stdout>:DLL 2021-05-03 18:50:40.510279 - (1200,) total_loss_ref:0 : 1.2450827 
[1,0]<stdout>:DLL 2021-05-03 18:51:31.271824 - (1300,) total_loss_ref:0 : 0.8765604 
[1,0]<stdout>:DLL 2021-05-03 18:52:22.155356 - (1400,) total_loss_ref:0 : 0.6756872 
[1,0]<stdout>:DLL 2021-05-03 18:53:12.955353 - (1500,) total_loss_ref:0 : 1.557979 
[1,0]<stdout>:DLL 2021-05-03 18:54:03.811395 - (1600,) total_loss_ref:0 : 2.0827234 
[1,0]<stdout>:DLL 2021-05-03 18:54:54.508715 - (1700,) total_loss_ref:0 : 1.6375121 
[1,0]<stdout>:DLL 2021-05-03 18:55:45.381461 - (1800,) total_loss_ref:0 : 1.3933635 
[1,0]<stdout>:DLL 2021-05-03 18:56:36.259948 - (1900,) total_loss_ref:0 : 1.5003388 
[1,0]<stdout>:DLL 2021-05-03 18:57:27.089483 - (2000,) total_loss_ref:0 : 1.6277912 
[1,0]<stdout>:DLL 2021-05-03 18:58:17.913735 - (2100,) total_loss_ref:0 : 0.6864653 
[1,0]<stdout>:DLL 2021-05-03 18:59:08.654815 - (2200,) total_loss_ref:0 : 1.6221154 
[1,0]<stdout>:DLL 2021-05-03 18:59:59.450261 - (2300,) total_loss_ref:0 : 0.3663895 
[1,0]<stdout>:DLL 2021-05-03 19:00:50.385605 - (2400,) total_loss_ref:0 : 1.6057699 
[1,0]<stdout>:DLL 2021-05-03 19:01:41.223833 - (2500,) total_loss_ref:0 : 1.3493402 
[1,0]<stdout>:DLL 2021-05-03 19:02:32.143043 - (2600,) total_loss_ref:0 : 1.4090843 
[1,0]<stdout>:DLL 2021-05-03 19:03:22.872373 - (2700,) total_loss_ref:0 : 1.4304377 
[1,0]<stdout>:DLL 2021-05-03 19:04:13.749549 - (2800,) total_loss_ref:0 : 1.5779047 
[1,0]<stdout>:DLL 2021-05-03 19:05:04.590282 - (2900,) total_loss_ref:0 : 1.4108907 
[1,0]<stdout>:DLL 2021-05-03 19:05:55.502304 - (3000,) total_loss_ref:0 : 1.2419345 
[1,0]<stdout>:DLL 2021-05-03 19:06:46.304047 - (3100,) total_loss_ref:0 : 0.6504372 
[1,0]<stdout>:DLL 2021-05-03 19:07:37.113844 - (3200,) total_loss_ref:0 : 2.1104422 
[1,0]<stdout>:DLL 2021-05-03 19:08:27.789997 - (3300,) total_loss_ref:0 : 0.841873 
[1,0]<stdout>:DLL 2021-05-03 19:09:18.574652 - (3400,) total_loss_ref:0 : 1.7877399 
[1,0]<stdout>:DLL 2021-05-03 19:10:09.333588 - (3500,) total_loss_ref:0 : 2.073061 
[1,0]<stdout>:DLL 2021-05-03 19:11:00.250665 - (3600,) total_loss_ref:0 : 1.3677889 
[1,0]<stdout>:DLL 2021-05-03 19:11:50.946519 - (3700,) total_loss_ref:0 : 1.7387836 
[1,0]<stdout>:DLL 2021-05-03 19:12:41.758383 - (3800,) total_loss_ref:0 : 0.38377652 
[1,0]<stdout>:DLL 2021-05-03 19:13:32.758546 - (3900,) total_loss_ref:0 : 0.9498587 
[1,0]<stdout>:DLL 2021-05-03 19:14:23.499415 - (4000,) total_loss_ref:0 : 0.85289997 
[1,3]<stderr>:2021-05-03 19:14:27.080203: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:14:27.080984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:07.0
[1,3]<stderr>:2021-05-03 19:14:27.081042: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 19:14:27.081166: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 19:14:27.081193: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 19:14:27.081210: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 19:14:27.081224: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,3]<stderr>:2021-05-03 19:14:27.081239: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,3]<stderr>:2021-05-03 19:14:27.081254: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 19:14:27.081355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:14:27.082114: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:14:27.082778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 3
[1,3]<stderr>:2021-05-03 19:14:27.082822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,3]<stderr>:2021-05-03 19:14:27.082837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      3 
[1,3]<stderr>:2021-05-03 19:14:27.082843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 3:   N 
[1,3]<stderr>:2021-05-03 19:14:27.082987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:14:27.083702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:14:27.084344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:07.0, compute capability: 7.0)
[1,2]<stderr>:2021-05-03 19:14:27.108661: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:14:27.109348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:06.0
[1,2]<stderr>:2021-05-03 19:14:27.109395: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:14:27.109463: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 19:14:27.109487: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 19:14:27.109503: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,2]<stderr>:2021-05-03 19:14:27.109518: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,2]<stderr>:2021-05-03 19:14:27.109538: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,2]<stderr>:2021-05-03 19:14:27.109553: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 19:14:27.109634: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:14:27.110323: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:14:27.110961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 2
[1,2]<stderr>:2021-05-03 19:14:27.110998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,2]<stderr>:2021-05-03 19:14:27.111004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      2 
[1,2]<stderr>:2021-05-03 19:14:27.111011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 2:   N 
[1,2]<stderr>:2021-05-03 19:14:27.111113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:14:27.111805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:14:27.112489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:06.0, compute capability: 7.0)
[1,1]<stderr>:2021-05-03 19:14:27.127232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:14:27.127934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:05.0
[1,1]<stderr>:2021-05-03 19:14:27.127973: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 19:14:27.128041: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 19:14:27.128065: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,1]<stderr>:2021-05-03 19:14:27.128081: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,1]<stderr>:2021-05-03 19:14:27.128096: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,1]<stderr>:2021-05-03 19:14:27.128111: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,1]<stderr>:2021-05-03 19:14:27.128127: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 19:14:27.128208: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:14:27.129004: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:14:27.129647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 1
[1,1]<stderr>:2021-05-03 19:14:27.129685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,1]<stderr>:2021-05-03 19:14:27.129691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      1 
[1,1]<stderr>:2021-05-03 19:14:27.129697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1:   N 
[1,1]<stderr>:2021-05-03 19:14:27.129804: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:14:27.130509: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:14:27.131167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:05.0, compute capability: 7.0)
[1,3]<stderr>:2021-05-03 19:14:27.168815: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.170527: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.209999: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:27.212197: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.229372: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:27.231520: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.303775: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.364907: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.368993: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.369922: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.385487: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.439799: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:27.441064: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.460947: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:27.462311: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.463260: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.508894: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.509227: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.541061: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.562591: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.563719: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.564085: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.593159: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:27.593622: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.617252: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:27.617771: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.655635: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:27.657082: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.670100: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.678961: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:27.680318: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.719253: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.719596: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.723723: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.727025: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.731178: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.775397: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.804353: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.835104: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:27.835492: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.840277: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.844294: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:14:27.849483: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.863385: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:27.863772: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.868580: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.872683: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:14:27.877836: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:14:27.879078: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:14:27.883767: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 831
[1,3]<stderr>:Recognized nodes available for conversion: 475
[1,3]<stderr>:Total nodes converted: 38
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 23
[1,3]<stderr>:Whitelisted nodes converted: 33
[1,3]<stderr>:Blacklisted nodes blocking conversion: 110
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 19:14:28.009638: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:14:28.015494: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 831
[1,2]<stderr>:Recognized nodes available for conversion: 475
[1,2]<stderr>:Total nodes converted: 38
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 23
[1,2]<stderr>:Whitelisted nodes converted: 33
[1,2]<stderr>:Blacklisted nodes blocking conversion: 110
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,1]<stderr>:2021-05-03 19:14:28.037336: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:14:28.043025: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 831
[1,1]<stderr>:Recognized nodes available for conversion: 475
[1,1]<stderr>:Total nodes converted: 38
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 23
[1,1]<stderr>:Whitelisted nodes converted: 33
[1,1]<stderr>:Blacklisted nodes blocking conversion: 110
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 19:14:28.068329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:14:28.069116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:04.0
[1,0]<stderr>:2021-05-03 19:14:28.069165: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:14:28.069251: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:2021-05-03 19:14:28.069283: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,0]<stderr>:2021-05-03 19:14:28.069305: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 19:14:28.069329: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 19:14:28.069350: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 19:14:28.069370: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 19:14:28.069473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:14:28.070189: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:14:28.070829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
[1,0]<stderr>:2021-05-03 19:14:28.070870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,0]<stderr>:2021-05-03 19:14:28.070879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
[1,0]<stderr>:2021-05-03 19:14:28.070885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
[1,0]<stderr>:2021-05-03 19:14:28.070993: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:14:28.071703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:14:28.072379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
[1,0]<stderr>:2021-05-03 19:14:28.159606: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:28.161763: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:28.507689: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:28.615536: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:28.616884: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:28.744416: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:28.836275: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:28.836956: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.020238: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:29.022652: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.196737: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.282255: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:29.282647: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.287572: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.291755: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.296951: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:14:29.492561: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:14:29.498209: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 831
[1,0]<stderr>:Recognized nodes available for conversion: 475
[1,0]<stderr>:Total nodes converted: 38
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 23
[1,0]<stderr>:Whitelisted nodes converted: 33
[1,0]<stderr>:Blacklisted nodes blocking conversion: 110
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,3]<stderr>:2021-05-03 19:14:29.917678: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 837
[1,3]<stderr>:Recognized nodes available for conversion: 420
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 33
[1,3]<stderr>:Blacklisted nodes blocking conversion: 110
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 19:14:30.033115: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 837
[1,2]<stderr>:Recognized nodes available for conversion: 420
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 33
[1,2]<stderr>:Blacklisted nodes blocking conversion: 110
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,1]<stderr>:2021-05-03 19:14:30.880775: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 837
[1,1]<stderr>:Recognized nodes available for conversion: 420
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 33
[1,1]<stderr>:Blacklisted nodes blocking conversion: 110
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 19:14:31.659649: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 837
[1,0]<stderr>:Recognized nodes available for conversion: 420
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 33
[1,0]<stderr>:Blacklisted nodes blocking conversion: 110
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 122
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,3]<stderr>:2021-05-03 19:15:05.009592: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:15:05.010037: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:15:05.031227: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:15:05.238645: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:15:05.239100: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:15:05.260640: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:15:05.280194: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:15:05.280682: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:15:05.301584: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:15:06.177467: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:15:06.177913: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:15:06.198687: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stdout>:DLL 2021-05-03 19:15:06.622979 - () TumorCore : 0.6864195466041565  PeritumoralEdema : 0.7561293840408325  EnhancingTumor : 0.6733013391494751  MeanDice : 0.7052834232648214  WholeTumor : 0.8617445230484009

Benchmarking

Training Performance Benchmarking

The training performance can be evaluated by using benchmarking scripts, such as:

bash examples/unet3d_{train,infer}_benchmark{_TF-AMP}.sh <number/of/gpus/for/training> <path/to/dataset> <path/to/checkpoint> <batch/size>

This script makes the model run and report the performance. For example, to benchmark training with TF-AMP with batch size 2 on 4 GPUs, use:

bash examples/unet3d_train_benchmark_TF-AMP.sh 4 /data/preprocessed /results 2

In [13]:

! bash examples/unet3d_train_benchmark_TF-AMP.sh 4 /data/preprocessed /results 2

2021-05-03 19:31:51.277730: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:31:53.071617: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:31:53.097450: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 19:31:53.097704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 19:31:53.119685: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,2]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,1]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,3]<stderr>:WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
[1,0]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,0]<stderr>:
[1,0]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,0]<stderr>:
[1,1]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,1]<stderr>:
[1,1]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,1]<stderr>:
[1,2]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,2]<stderr>:
[1,2]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,2]<stderr>:
[1,3]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[1,3]<stderr>:
[1,3]<stderr>:WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[1,3]<stderr>:
[1,3]<stderr>:2021-05-03 19:31:56.574732: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,0]<stderr>:2021-05-03 19:31:56.575203: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,2]<stderr>:2021-05-03 19:31:56.576572: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,1]<stderr>:2021-05-03 19:31:56.581759: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
[1,3]<stderr>:2021-05-03 19:31:56.913702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.915138: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.916313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:04.0
[1,3]<stderr>:2021-05-03 19:31:56.916481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.918263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:04.0
[1,0]<stderr>:2021-05-03 19:31:56.918430: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.919314: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.920268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:05.0
[1,3]<stderr>:2021-05-03 19:31:56.920419: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.922583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:05.0
[1,0]<stderr>:2021-05-03 19:31:56.922682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.923525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:04.0
[1,1]<stderr>:2021-05-03 19:31:56.923685: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.924573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:06.0
[1,3]<stderr>:2021-05-03 19:31:56.924679: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.926798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:06.0
[1,0]<stderr>:2021-05-03 19:31:56.926895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.927304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.927546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:05.0
[1,1]<stderr>:2021-05-03 19:31:56.927645: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.928850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:07.0
[1,3]<stderr>:2021-05-03 19:31:56.928890: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:31:56.931096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:07.0
[1,0]<stderr>:2021-05-03 19:31:56.931129: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:31:56.931323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:04.0
[1,1]<stderr>:2021-05-03 19:31:56.931365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:06.0
[1,1]<stderr>:2021-05-03 19:31:56.931452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.931472: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.932256: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 19:31:56.933656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:07.0
[1,1]<stderr>:2021-05-03 19:31:56.933690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:31:56.933719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:05.0
[1,3]<stderr>:2021-05-03 19:31:56.933824: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 19:31:56.933822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.934179: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 19:31:56.934362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 19:31:56.934810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:06.0
[1,2]<stderr>:2021-05-03 19:31:56.934891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.935654: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 19:31:56.935834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:07.0
[1,2]<stderr>:2021-05-03 19:31:56.935861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:31:56.935969: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,1]<stderr>:2021-05-03 19:31:56.936691: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 19:31:56.937198: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,1]<stderr>:2021-05-03 19:31:56.938013: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 19:31:56.938021: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,3]<stderr>:2021-05-03 19:31:56.938212: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 19:31:56.938338: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 19:31:56.938324: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.938859: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:2021-05-03 19:31:56.938956: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,3]<stderr>:2021-05-03 19:31:56.939385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.939688: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 19:31:56.939886: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 19:31:56.940002: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.940172: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 19:31:56.940544: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 19:31:56.940809: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.941312: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 19:31:56.941978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.942099: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,1]<stderr>:2021-05-03 19:31:56.942280: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 19:31:56.942384: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.943378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.943486: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,2]<stderr>:2021-05-03 19:31:56.944237: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,2]<stderr>:2021-05-03 19:31:56.944432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 19:31:56.944551: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.945445: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.945662: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.947818: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.949239: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.950079: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.950267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.952373: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.953916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.954755: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.954802: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.956895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.958377: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.959209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.959263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.961612: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.963013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.963880: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.963942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:31:56.966152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,2]<stderr>:2021-05-03 19:31:56.967480: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.968070: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.968177: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.970896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:31:56.971480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,1]<stderr>:2021-05-03 19:31:56.971604: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:31:56.973311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:31:56.973690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,2]<stderr>:2021-05-03 19:31:56.974555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0, 1, 2, 3
[1,2]<stderr>:2021-05-03 19:32:03.241763: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,2]<stderr>:2021-05-03 19:32:03.243115: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x516a160 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,2]<stderr>:2021-05-03 19:32:03.243143: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,3]<stderr>:2021-05-03 19:32:03.390749: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,3]<stderr>:2021-05-03 19:32:03.392099: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x551b5a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,3]<stderr>:2021-05-03 19:32:03.392132: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,1]<stderr>:2021-05-03 19:32:03.396446: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,1]<stderr>:2021-05-03 19:32:03.397681: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4f70820 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,1]<stderr>:2021-05-03 19:32:03.397711: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,0]<stderr>:2021-05-03 19:32:03.409914: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
[1,0]<stderr>:2021-05-03 19:32:03.411331: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6ae6580 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
[1,0]<stderr>:2021-05-03 19:32:03.411362: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[1,2]<stderr>:2021-05-03 19:32:04.005594: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.010939: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x50f4540 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,2]<stderr>:2021-05-03 19:32:04.010975: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,2]<stderr>:2021-05-03 19:32:04.011254: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.012954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,2]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,2]<stderr>:pciBusID: 0000:00:06.0
[1,2]<stderr>:2021-05-03 19:32:04.013013: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:32:04.013103: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,2]<stderr>:2021-05-03 19:32:04.013151: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,2]<stderr>:2021-05-03 19:32:04.013175: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,2]<stderr>:2021-05-03 19:32:04.013195: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,2]<stderr>:2021-05-03 19:32:04.013219: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,2]<stderr>:2021-05-03 19:32:04.013240: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 19:32:04.013344: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.015345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.016321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 2
[1,2]<stderr>:2021-05-03 19:32:04.016414: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 19:32:04.040072: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.041513: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x551f2d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,3]<stderr>:2021-05-03 19:32:04.041549: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,3]<stderr>:2021-05-03 19:32:04.041834: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.042882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,3]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,3]<stderr>:pciBusID: 0000:00:07.0
[1,3]<stderr>:2021-05-03 19:32:04.042924: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,3]<stderr>:2021-05-03 19:32:04.043010: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 19:32:04.043041: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,3]<stderr>:2021-05-03 19:32:04.043062: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,3]<stderr>:2021-05-03 19:32:04.043083: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,3]<stderr>:2021-05-03 19:32:04.043104: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,3]<stderr>:2021-05-03 19:32:04.043128: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 19:32:04.043224: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.044655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.045915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 3
[1,3]<stderr>:2021-05-03 19:32:04.045966: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 19:32:04.054216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.056503: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4f74550 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,1]<stderr>:2021-05-03 19:32:04.056551: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,0]<stderr>:2021-05-03 19:32:04.056726: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.056810: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.059192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,1]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,1]<stderr>:pciBusID: 0000:00:05.0
[1,1]<stderr>:2021-05-03 19:32:04.059233: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,1]<stderr>:2021-05-03 19:32:04.059297: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 19:32:04.059322: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,1]<stderr>:2021-05-03 19:32:04.059338: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,1]<stderr>:2021-05-03 19:32:04.059357: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,1]<stderr>:2021-05-03 19:32:04.059372: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,1]<stderr>:2021-05-03 19:32:04.059388: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 19:32:04.059503: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6a23920 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
[1,0]<stderr>:2021-05-03 19:32:04.059535: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
[1,1]<stderr>:2021-05-03 19:32:04.059508: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:32:04.059779: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.061943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:32:04.062277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
[1,0]<stderr>:name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
[1,0]<stderr>:pciBusID: 0000:00:04.0
[1,0]<stderr>:2021-05-03 19:32:04.062326: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:32:04.062386: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:2021-05-03 19:32:04.062408: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
[1,0]<stderr>:2021-05-03 19:32:04.062424: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
[1,0]<stderr>:2021-05-03 19:32:04.062438: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
[1,0]<stderr>:2021-05-03 19:32:04.062452: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
[1,0]<stderr>:2021-05-03 19:32:04.062467: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 19:32:04.062553: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.063538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 1
[1,1]<stderr>:2021-05-03 19:32:04.063588: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,0]<stderr>:2021-05-03 19:32:04.065043: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:32:04.067302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
[1,0]<stderr>:2021-05-03 19:32:04.067352: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
[1,2]<stderr>:2021-05-03 19:32:04.448566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,2]<stderr>:2021-05-03 19:32:04.448625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      2 
[1,2]<stderr>:2021-05-03 19:32:04.448634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 2:   N 
[1,2]<stderr>:2021-05-03 19:32:04.448980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.450455: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,2]<stderr>:2021-05-03 19:32:04.452013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:06.0, compute capability: 7.0)
[1,3]<stderr>:2021-05-03 19:32:04.475203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,3]<stderr>:2021-05-03 19:32:04.475256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      3 
[1,3]<stderr>:2021-05-03 19:32:04.475264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 3:   N 
[1,3]<stderr>:2021-05-03 19:32:04.475576: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.477138: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,3]<stderr>:2021-05-03 19:32:04.478175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:07.0, compute capability: 7.0)
[1,1]<stderr>:2021-05-03 19:32:04.493979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,1]<stderr>:2021-05-03 19:32:04.494027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      1 
[1,1]<stderr>:2021-05-03 19:32:04.494036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1:   N 
[1,1]<stderr>:2021-05-03 19:32:04.494301: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.495398: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,1]<stderr>:2021-05-03 19:32:04.496417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:05.0, compute capability: 7.0)
[1,0]<stderr>:2021-05-03 19:32:04.499230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
[1,0]<stderr>:2021-05-03 19:32:04.499271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
[1,0]<stderr>:2021-05-03 19:32:04.499280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
[1,0]<stderr>:2021-05-03 19:32:04.499554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:32:04.500673: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[1,0]<stderr>:2021-05-03 19:32:04.501711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
[1,2]<stderr>:2021-05-03 19:32:05.249092: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:05.253434: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:05.346122: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:05.351351: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:05.371139: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:05.374143: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:05.376499: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:05.379491: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:06.300245: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:06.379527: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:06.389205: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:06.452746: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:07.147580: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:07.147702: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:07.150316: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:07.151267: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:07.228500: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:07.232116: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:07.244474: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:07.248130: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:07.417102: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,3]<stderr>:2021-05-03 19:32:07.426419: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,2]<stderr>:2021-05-03 19:32:07.486633: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:07.500438: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,3]<stderr>:2021-05-03 19:32:07.503567: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:07.523040: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
[1,1]<stderr>:2021-05-03 19:32:07.593142: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:07.615821: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:07.682127: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:07.682374: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:07.723579: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:07.723860: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:07.818122: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:07.818414: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:07.845797: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:07.846085: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:07.886804: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:07.889532: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:07.944332: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:07.947892: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.037132: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:08.040561: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.061129: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:08.064640: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.213004: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.296047: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.392387: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.403912: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:08.404282: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.410786: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.411975: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.414199: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.415528: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.437191: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.444901: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.446219: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.447463: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.511918: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:08.512367: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.521605: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.524227: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.525807: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.552329: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.561355: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.562873: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.564400: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.613127: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:08.613536: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.623291: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.625992: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.627479: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.633546: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:08.633923: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.642695: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.645378: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.646880: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.654510: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.663652: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.665185: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.666658: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.673447: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.682463: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.683971: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.685546: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:08.730549: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:08.739498: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:08.872034: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:08.879119: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:08.972630: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:08.979750: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:08.995570: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:09.002194: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 17969, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 11889, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 15977, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 13873, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 18793, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 11073, errno = 1
[1,1]<stderr>:[f267448f44e0:01847] Read -1, expected 17916, errno = 1
[1,3]<stderr>:[f267448f44e0:01849] Read -1, expected 17916, errno = 1
[1,2]<stderr>:[f267448f44e0:01848] Read -1, expected 17917, errno = 1
[1,3]<stderr>:2021-05-03 19:32:09.989840: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:09.990324: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:10.005374: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:10.008991: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:10.170156: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:10.170499: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:10.176297: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:10.176640: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:10.179421: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:10.179699: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:10.196870: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:10.197180: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:11.303535: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:11.333818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 5131
[1,2]<stderr>:Recognized nodes available for conversion: 3292
[1,2]<stderr>:Total nodes converted: 118
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,2]<stderr>:Whitelisted nodes converted: 138
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 19:32:11.430234: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:11.449633: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:11.450309: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:11.464734: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 5131
[1,3]<stderr>:Recognized nodes available for conversion: 3292
[1,3]<stderr>:Total nodes converted: 118
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,3]<stderr>:Whitelisted nodes converted: 138
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,0]<stderr>:2021-05-03 19:32:11.484347: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 5131
[1,0]<stderr>:Recognized nodes available for conversion: 3292
[1,0]<stderr>:Total nodes converted: 118
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,0]<stderr>:Whitelisted nodes converted: 138
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,1]<stderr>:2021-05-03 19:32:11.484947: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 5131
[1,1]<stderr>:Recognized nodes available for conversion: 3292
[1,1]<stderr>:Total nodes converted: 118
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,1]<stderr>:Whitelisted nodes converted: 138
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,2]<stderr>:2021-05-03 19:32:16.289540: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 3071
[1,2]<stderr>:Recognized nodes available for conversion: 2197
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 99
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,0]<stderr>:2021-05-03 19:32:16.573235: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 3071
[1,0]<stderr>:Recognized nodes available for conversion: 2197
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 99
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,1]<stderr>:2021-05-03 19:32:16.607513: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 3071
[1,1]<stderr>:Recognized nodes available for conversion: 2197
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 99
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,3]<stderr>:2021-05-03 19:32:16.638928: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 3071
[1,3]<stderr>:Recognized nodes available for conversion: 2197
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 99
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,2]<stderr>:2021-05-03 19:32:17.262741: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,1]<stderr>:2021-05-03 19:32:17.579132: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,3]<stderr>:2021-05-03 19:32:17.631437: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,0]<stderr>:2021-05-03 19:32:17.635471: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
[1,2]<stderr>:2021-05-03 19:32:18.640898: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,1]<stderr>:2021-05-03 19:32:18.917236: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,3]<stderr>:2021-05-03 19:32:18.971596: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:2021-05-03 19:32:19.216685: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 4849, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 13001, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 4857, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 13001, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 4849, errno = 1
[1,0]<stderr>:[f267448f44e0:01846] Read -1, expected 13001, errno = 1
[1,3]<stderr>:[f267448f44e0:01849] Read -1, expected 4436, errno = 1
[1,2]<stderr>:[f267448f44e0:01848] Read -1, expected 4437, errno = 1
[1,1]<stderr>:[f267448f44e0:01847] Read -1, expected 4436, errno = 1
[1,1]<stderr>:2021-05-03 19:32:22.972641: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:22.975707: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:23.002503: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:23.005078: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:23.081760: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:23.086618: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:23.128723: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:23.132167: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:24.333892: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,2]<stderr>:2021-05-03 19:32:24.364219: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 5132
[1,2]<stderr>:Recognized nodes available for conversion: 3292
[1,2]<stderr>:Total nodes converted: 118
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,2]<stderr>:Whitelisted nodes converted: 138
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 19:32:24.407818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,1]<stderr>:2021-05-03 19:32:24.419790: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,3]<stderr>:2021-05-03 19:32:24.444018: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 5132
[1,3]<stderr>:Recognized nodes available for conversion: 3292
[1,3]<stderr>:Total nodes converted: 118
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,3]<stderr>:Whitelisted nodes converted: 138
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,1]<stderr>:2021-05-03 19:32:24.455963: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 5132
[1,1]<stderr>:Recognized nodes available for conversion: 3292
[1,1]<stderr>:Total nodes converted: 118
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,1]<stderr>:Whitelisted nodes converted: 138
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 19:32:24.456091: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1987] Running auto_mixed_precision graph optimizer
[1,0]<stderr>:2021-05-03 19:32:24.490952: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 5132
[1,0]<stderr>:Recognized nodes available for conversion: 3292
[1,0]<stderr>:Total nodes converted: 118
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 46
[1,0]<stderr>:Whitelisted nodes converted: 138
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,2]<stderr>:2021-05-03 19:32:26.198588: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,2]<stderr>:
[1,2]<stderr>:Total processable nodes: 3071
[1,2]<stderr>:Recognized nodes available for conversion: 2197
[1,2]<stderr>:Total nodes converted: 0
[1,2]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,2]<stderr>:Whitelisted nodes converted: 99
[1,2]<stderr>:Blacklisted nodes blocking conversion: 432
[1,2]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,2]<stderr>:
[1,2]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,2]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,2]<stderr>:
[1,2]<stderr>:
[1,3]<stderr>:2021-05-03 19:32:26.480924: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,3]<stderr>:
[1,3]<stderr>:Total processable nodes: 3071
[1,3]<stderr>:Recognized nodes available for conversion: 2197
[1,3]<stderr>:Total nodes converted: 0
[1,3]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,3]<stderr>:Whitelisted nodes converted: 99
[1,3]<stderr>:Blacklisted nodes blocking conversion: 432
[1,3]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,3]<stderr>:
[1,3]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,3]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,3]<stderr>:
[1,3]<stderr>:
[1,1]<stderr>:2021-05-03 19:32:26.519048: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,1]<stderr>:
[1,1]<stderr>:Total processable nodes: 3071
[1,1]<stderr>:Recognized nodes available for conversion: 2197
[1,1]<stderr>:Total nodes converted: 0
[1,1]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,1]<stderr>:Whitelisted nodes converted: 99
[1,1]<stderr>:Blacklisted nodes blocking conversion: 432
[1,1]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,1]<stderr>:
[1,1]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,1]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,1]<stderr>:
[1,1]<stderr>:
[1,0]<stderr>:2021-05-03 19:32:26.575264: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1035] Automatic Mixed Precision Grappler Pass Summary:
[1,0]<stderr>:
[1,0]<stderr>:Total processable nodes: 3071
[1,0]<stderr>:Recognized nodes available for conversion: 2197
[1,0]<stderr>:Total nodes converted: 0
[1,0]<stderr>:Total FP16 Cast ops used (excluding Const and Variable casts): 0
[1,0]<stderr>:Whitelisted nodes converted: 99
[1,0]<stderr>:Blacklisted nodes blocking conversion: 432
[1,0]<stderr>:Nodes blocked from conversion by blacklisted nodes: 614
[1,0]<stderr>:
[1,0]<stderr>:For more information regarding mixed precision training, including how to make automatic mixed precision aware of a custom op type, please see the documentation available here:
[1,0]<stderr>:https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tfamp
[1,0]<stderr>:
[1,0]<stderr>:
[1,2]<stderr>:2021-05-03 19:32:29.290144: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:29.298391: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:29.302031: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:29.303243: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:29.377752: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,0]<stderr>:2021-05-03 19:32:29.405440: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,1]<stderr>:2021-05-03 19:32:29.410461: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,3]<stderr>:2021-05-03 19:32:29.413193: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1343] No whitelist ops found, nothing to do
[1,2]<stderr>:2021-05-03 19:32:43.410930: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.410993: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411001: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411007: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411012: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411018: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411024: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411029: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411035: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411040: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,2]<stderr>:2021-05-03 19:32:43.411067: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,2]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,2]<stderr>:2021-05-03 19:32:43.411073: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,2]<stderr>:2021-05-03 19:32:43.411078: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,2]<stderr>:2021-05-03 19:32:43.411082: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,2]<stderr>:2021-05-03 19:32:43.411087: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,2]<stderr>:2021-05-03 19:32:43.411094: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,0]<stderr>:2021-05-03 19:32:43.559012: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559062: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559070: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559076: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559082: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559088: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559094: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559101: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559107: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559112: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,0]<stderr>:2021-05-03 19:32:43.559142: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,0]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,0]<stderr>:2021-05-03 19:32:43.559149: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,0]<stderr>:2021-05-03 19:32:43.559154: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,0]<stderr>:2021-05-03 19:32:43.559159: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,0]<stderr>:2021-05-03 19:32:43.559163: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,0]<stderr>:2021-05-03 19:32:43.559171: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,3]<stderr>:2021-05-03 19:32:43.576168: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576218: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576227: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576233: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576241: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576247: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576253: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576260: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576266: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576273: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,3]<stderr>:2021-05-03 19:32:43.576305: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,3]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,3]<stderr>:2021-05-03 19:32:43.576316: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,3]<stderr>:2021-05-03 19:32:43.576322: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,3]<stderr>:2021-05-03 19:32:43.576327: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,3]<stderr>:2021-05-03 19:32:43.576332: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,3]<stderr>:2021-05-03 19:32:43.576340: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,1]<stderr>:2021-05-03 19:32:43.594090: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 0: 1321 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594151: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 1: 1320 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594160: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 2: 1321 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594166: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 3: 1322 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594173: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 4: 1321 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594179: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 5: 1320 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594186: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 6: 1320 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594192: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 7: 1320 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594198: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 8: 1322 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594204: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:646] Difference at 9: 1321 vs 1024
[1,1]<stderr>:2021-05-03 19:32:43.594237: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:450] Results mismatch between different convolution algorithms. This is likely a bug/unexpected loss of precision in cudnn.
[1,1]<stderr>:%custom-call.35 = (f16[2,2,2,32,64]{2,1,0,3,4}, u8[0]{0}) custom-call(f16[2,128,128,128,32]{3,2,1,4,0} %slice.6143, f16[2,64,64,64,64]{3,2,1,4,0} %convert.1811), window={size=2x2x2 stride=2x2x2}, dim_labels=b012f_012io->b012f, custom_call_target="__cudnn$convBackwardFilter", metadata={op_type="Conv3DBackpropFilterV2" op_name="gradients/conv3d_transpose_4/conv3d_transpose_grad/Conv3DBackpropFilterV2"}, backend_config="{\"convResultScale\":1}" for 1+TC vs 1
[1,1]<stderr>:2021-05-03 19:32:43.594263: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:133] Device: Tesla V100-SXM2-16GB
[1,1]<stderr>:2021-05-03 19:32:43.594268: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:134] Platform: Compute Capability 7.0
[1,1]<stderr>:2021-05-03 19:32:43.594273: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:135] Driver: 11000 (450.80.2)
[1,1]<stderr>:2021-05-03 19:32:43.594279: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:136] Runtime: <undefined>
[1,1]<stderr>:2021-05-03 19:32:43.594286: E tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:143] cudnn version: 8.0.1
[1,2]<stderr>:2021-05-03 19:33:09.328373: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,1]<stderr>:2021-05-03 19:33:10.849498: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,0]<stderr>:2021-05-03 19:33:10.856839: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,3]<stderr>:2021-05-03 19:33:10.865565: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
[1,0]<stdout>:DLL 2021-05-03 19:33:55.845869 - () throughput_train : 15.701845897069663  latency_train: : 509.6717119216919  Latency_train 90%: : 512.1431652287586  Latency_train 95%: : 512.6164222450053  Latency_train 99%: : 513.5419026323324

Inference Performance Benchmarking

To obtain inference performance with 32-bit precision (FP32 or TF32) with batch size 1, use: bash examples/unet3d_infer_benchmark.sh /data/preprocessed /results 1

In [14]:

! bash examples/unet3d_infer_benchmark.sh /data/preprocessed /results 1

2021-05-03 19:35:35.426331: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[11104,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: f267448f44e0

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
2021-05-03 19:35:39.921035: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2021-05-03 19:35:39.922901: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x570d860 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-05-03 19:35:39.922934: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-05-03 19:35:39.925978: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-05-03 19:35:40.359518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.361008: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5780960 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-05-03 19:35:40.361047: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2021-05-03 19:35:40.361367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.362402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:04.0
2021-05-03 19:35:40.362461: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-05-03 19:35:40.365709: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2021-05-03 19:35:40.367112: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-05-03 19:35:40.367506: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-05-03 19:35:40.370499: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-05-03 19:35:40.371364: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2021-05-03 19:35:40.371634: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-05-03 19:35:40.371778: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.372796: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.373689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2021-05-03 19:35:40.373729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-05-03 19:35:40.770387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-03 19:35:40.770448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2021-05-03 19:35:40.770457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2021-05-03 19:35:40.770763: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.771889: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:35:40.772847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
2021-05-03 19:35:41.191060: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1648] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2021-05-03 19:35:42.734909: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-05-03 19:35:49.060549: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2021-05-03 19:36:22.420580: I tensorflow/compiler/jit/xla_compilation_cache.cc:243] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
DLL 2021-05-03 19:37:48.422457 - () throughput_test : 1.5606345599067808  latency_test: : 640.7692982600286  Latency_test 90%: : 641.021711073765  Latency_test 95%: : 641.0700454423528  Latency_test 99%: : 641.164565985369

Main.py Script

To see the full list of available options for main.py use the -h or --help. By running the next cell you can see how to change execute mode and other parameters of this script. You can perform model training, predicting, evaluating and inferenceing using customized hyperparameters using this script.

In [24]:

! python main.py --help

2021-04-28 20:07:50.657240: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[8298,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: b09e49533fde

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
usage: main.py [-h] --model_dir MODEL_DIR
               [--exec_mode {train,evaluate,train_and_evaluate,predict,debug_train,debug_predict}]
               [--benchmark] [--max_steps MAX_STEPS]
               [--learning_rate LEARNING_RATE] [--log_every LOG_EVERY]
               [--log_dir LOG_DIR] [--loss {dice,ce,dice+ce}]
               [--warmup_steps WARMUP_STEPS]
               [--normalization {instancenorm,batchnorm,groupnorm}]
               [--include_background] [--resume_training] [--augment]
               --data_dir DATA_DIR [--batch_size BATCH_SIZE] [--fold FOLD]
               [--num_folds NUM_FOLDS] [--use_amp] [--use_xla]

UNet-3D

optional arguments:
  -h, --help            show this help message and exit
  --model_dir MODEL_DIR
  --exec_mode {train,evaluate,train_and_evaluate,predict,debug_train,debug_predict}
  --benchmark
  --max_steps MAX_STEPS
  --learning_rate LEARNING_RATE
  --log_every LOG_EVERY
  --log_dir LOG_DIR
  --loss {dice,ce,dice+ce}
  --warmup_steps WARMUP_STEPS
  --normalization {instancenorm,batchnorm,groupnorm}
  --include_background
  --resume_training
  --augment
  --data_dir DATA_DIR
  --batch_size BATCH_SIZE
  --fold FOLD
  --num_folds NUM_FOLDS
  --use_amp, --amp
  --use_xla, --xla

Predict

we can use test dataset and predict as exec-mode to test the model. the result will be saved in model_dir.

python main.py --model_dir /results --exec_mode predict --data_dir /data/path to test data

In [15]:

! python main.py --model_dir /results --exec_mode predict --data_dir /data/preprocessed_test

2021-05-03 19:47:00.780670: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:152: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.19.1-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:178: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

--------------------------------------------------------------------------
[[10922,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: f267448f44e0

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
2021-05-03 19:47:05.006007: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2021-05-03 19:47:05.007558: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5b664c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-05-03 19:47:05.007592: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-05-03 19:47:05.010085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-05-03 19:47:05.415001: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.416501: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5bd97f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-05-03 19:47:05.416556: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2021-05-03 19:47:05.416866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.417911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:04.0
2021-05-03 19:47:05.417950: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-05-03 19:47:05.421044: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2021-05-03 19:47:05.422587: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-05-03 19:47:05.422942: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-05-03 19:47:05.426085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-05-03 19:47:05.426899: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2021-05-03 19:47:05.427107: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-05-03 19:47:05.427230: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.428269: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.429212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2021-05-03 19:47:05.429253: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-05-03 19:47:05.832270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-03 19:47:05.832329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2021-05-03 19:47:05.832338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2021-05-03 19:47:05.832663: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.833752: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-03 19:47:05.834779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14799 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
2021-05-03 19:47:07.250249: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-05-03 19:47:08.602479: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11

Plotting the Result of Prediction

in the next two cells, we chose one of the results from the results folder and plot it.

In [60]:

import numpy as np
data= np.load('/results/vol_3.npy')

In [61]:

from mpl_toolkits import mplot3d
import numpy as np
import matplotlib.pyplot as plt

def show_plane(ax, plane, cmap="gray", title=None):
    ax.imshow(plane, cmap=cmap)
    ax.axis("off")

    if title:
        ax.set_title(title)
        
(n_plane, n_row, n_col) = data.shape
_, (a, b, c) = plt.subplots(ncols=3, figsize=(15, 5))

show_plane(a, data[n_plane // 2], title=f'Plane = {n_plane // 2}')
show_plane(b, data[:, n_row // 2, :], title=f'Row = {n_row // 2}')
show_plane(c, data[:, :, n_col // 2], title=f'Column = {n_col // 2}')

In [ ]: