ContainersNVIDIADeepSeek-R1NVIDIALlama-3.1-Nemotron-70B-InstructNVIDIAPyTorch
CollectionsNVIDIAOmniverse Kit (FB)NVIDIADeepStream SDKNVIDIAOmniverse Kit App Streaming
ModelsNVIDIAStyleGAN3 pretrained modelsNVIDIAPeopleNetNVIDIATrafficCamNet
ResourcesNVIDIARiva Skills Quick Start
Helm ChartsNVIDIAGPU OperatorNVIDIANVIDIA NIM Operator
NVIDIA
NVIDIA
BERT for TensorFlow Jupyter Notebook
Resource
NVIDIA
NVIDIA
BERT for TensorFlow Jupyter Notebook

Jupyter Notebook example for Question Answering with BERT for TensorFlow

BERT Question Answering in TensorFlow with Mixed Precision

Copyright 2021 NVIDIA Corporation. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

!nvidia-smi

Tue Mar 16 00:26:48 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-DGXS...  On   | 00000000:08:00.0 Off |                    0 |
| N/A   37C    P0    39W / 300W |      5MiB / 32508MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

1. Overview

Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.

The original paper can be found here: https://arxiv.org/abs/1810.04805.

NVIDIA's BERT is an optimized version of Google's official implementation, leveraging mixed precision arithmetic and tensor cores on V100 GPUS for faster training times while maintaining target accuracy.

Learning objectives

This notebook demonstrates:

  • Inference on Question Answering (QA) task with BERT Large model
  • The use/download of fine-tuned NVIDIA BERT models from NGC
  • Use of Mixed Precision models for Inference

2. Setup

Pre-Trained NVIDIA BERT TensorFlow Models on NGC

We will be using the following configuration of BERT in this example:

ModelHidden layersHidden unit sizeAttention headsFeedforward filter sizeMax sequence lengthParameters
BERTLARGE24 encoder1024164 x 1024512330M

To do so, we will take advantage of the pre-trained models available on the NGC Model Registry.

Among the many configurations available we will download one of these two:

  • bert_tf_ckpt_large_qa_squad2_amp_384

which are trained on the SQuaD 2.0 Dataset.

We can choose the mixed precision model (which takes much less time to train than the fp32 version) without losing accuracy, with the following flag:

use_mixed_precision_model = True

# bert_tf_ckpt_large_qa_squad2_amp_384
DATA_DIR_FT = '/workspace/bert/data/finetuned_large_model_SQUAD2.0'
!mkdir -p $DATA_DIR_FT
    
!wget --content-disposition -O $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip  \
https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/zip \
&& unzip -n -d $DATA_DIR_FT/ $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip \
&& rm -rf $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip

--2021-03-16 00:19:46--  https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 52.9.83.149, 52.9.91.161
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|52.9.83.149|:443... connected.
HTTP request sent, awaiting response... 302 
Location: https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/models/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T001946Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f987fe14d3094638bb6e6abdbe47c066e0d75ba623cf1b77cf02275f52bd8e12 [following]
--2021-03-16 00:19:46--  https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/models/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T001946Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f987fe14d3094638bb6e6abdbe47c066e0d75ba623cf1b77cf02275f52bd8e12
Resolving s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)... 52.218.250.176
Connecting to s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)|52.218.250.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3672271344 (3.4G) [application/zip]
Saving to: ‘/workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip’

/workspace/bert/dat 100%[===================>]   3.42G  36.8MB/s    in 1m 47s  

2021-03-16 00:21:34 (32.6 MB/s) - ‘/workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip’ saved [3672271344/3672271344]

Archive:  /workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_config.json  
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.data-00000-of-00001  
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.index  
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.meta  
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/tf_bert_squad_1n_fp16_gbs32.190523090758.log  
  inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/vocab.txt

NGC Model Scripts

While we're at it, we'll also pull down some BERT helper scripts from the NGC Model Scripts Registry

# Download BERT helper scripts
!wget -nc --show-progress -O bert_scripts.zip \
     https://api.ngc.nvidia.com/v2/recipes/nvidia/bert_for_tensorflow/versions/1/zip
!mkdir -p /workspace/bert
!unzip -n -d /workspace/bert bert_scripts.zip

--2021-03-16 00:22:16--  https://api.ngc.nvidia.com/v2/recipes/nvidia/bert_for_tensorflow/versions/1/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 52.9.83.149, 52.9.91.161
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|52.9.83.149|:443... connected.
HTTP request sent, awaiting response... 302 
Location: https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/recipes/bert_for_tensorflow/versions/1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T002217Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ed2fba6c87fcfe93220e5ce6bc1207ffb9568596c5a73aefd9d83034434c4cf2 [following]
--2021-03-16 00:22:17--  https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/recipes/bert_for_tensorflow/versions/1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T002217Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ed2fba6c87fcfe93220e5ce6bc1207ffb9568596c5a73aefd9d83034434c4cf2
Resolving s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)... 52.218.250.24
Connecting to s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)|52.218.250.24|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 132136 (129K) [application/zip]
Saving to: ‘bert_scripts.zip’

bert_scripts.zip    100%[===================>] 129.04K  --.-KB/s    in 0.1s    

2021-03-16 00:22:17 (1.25 MB/s) - ‘bert_scripts.zip’ saved [132136/132136]

Archive:  bert_scripts.zip
  inflating: /workspace/bert/.dockerignore  
  inflating: /workspace/bert/CONTRIBUTING.md  
  inflating: /workspace/bert/Dockerfile  
  inflating: /workspace/bert/LICENSE  
  inflating: /workspace/bert/NOTICE  
  inflating: /workspace/bert/README.md  
  inflating: /workspace/bert/__init__.py  
  inflating: /workspace/bert/create_pretraining_data.py  
  inflating: /workspace/bert/data/README.md  
  inflating: /workspace/bert/data/bookcorpus/clean_and_merge_text.py  
  inflating: /workspace/bert/data/bookcorpus/config.sh  
  inflating: /workspace/bert/data/bookcorpus/create_pseudo_test_set.py  
  inflating: /workspace/bert/data/bookcorpus/create_pseudo_test_set.sh  
  inflating: /workspace/bert/data/bookcorpus/preprocessing.sh  
  inflating: /workspace/bert/data/bookcorpus/preprocessing_test_set.sh  
  inflating: /workspace/bert/data/bookcorpus/preprocessing_test_set_xargs_wrapper.sh  
  inflating: /workspace/bert/data/bookcorpus/preprocessing_xargs_wrapper.sh  
  inflating: /workspace/bert/data/bookcorpus/run_preprocessing.sh  
  inflating: /workspace/bert/data/bookcorpus/sentence_segmentation_nltk.py  
  inflating: /workspace/bert/data/bookcorpus/shard_text_input_file.py  
  inflating: /workspace/bert/data/pretrained_models_google/download_models.py  
  inflating: /workspace/bert/data/squad/squad_download.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/config.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/create_pseudo_test_set.py  
  inflating: /workspace/bert/data/wikipedia_corpus/create_pseudo_test_set.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/preprocessing.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_test_set.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_test_set_xargs_wrapper.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_xargs_wrapper.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/remove_tags_and_clean.py  
  inflating: /workspace/bert/data/wikipedia_corpus/run_preprocessing.sh  
  inflating: /workspace/bert/data/wikipedia_corpus/shard_text_input_file.py  
  inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_nltk.py  
  inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_spacy.py  
  inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_spacy_pipe.py  
  inflating: /workspace/bert/extract_features.py  
  inflating: /workspace/bert/gpu_environment.py  
  inflating: /workspace/bert/modeling.py  
  inflating: /workspace/bert/modeling_test.py  
  inflating: /workspace/bert/multilingual.md  
  inflating: /workspace/bert/optimization.py  
  inflating: /workspace/bert/optimization_test.py  
  inflating: /workspace/bert/predicting_movie_reviews_with_bert_on_tf_hub.ipynb  
  inflating: /workspace/bert/requirements.txt  
  inflating: /workspace/bert/run_classifier.py  
  inflating: /workspace/bert/run_classifier_with_tfhub.py  
  inflating: /workspace/bert/run_pretraining.py  
  inflating: /workspace/bert/run_squad.py  
  inflating: /workspace/bert/sample_text.txt  
  inflating: /workspace/bert/scripts/data_download.sh  
  inflating: /workspace/bert/scripts/data_download_helper.sh  
  inflating: /workspace/bert/scripts/docker/build.sh  
  inflating: /workspace/bert/scripts/docker/launch.sh  
  inflating: /workspace/bert/scripts/finetune_inference_benchmark.sh  
  inflating: /workspace/bert/scripts/finetune_train_benchmark.sh  
  inflating: /workspace/bert/scripts/run.sub  
  inflating: /workspace/bert/scripts/run_pretraining.sh  
  inflating: /workspace/bert/scripts/run_squad.sh  
  inflating: /workspace/bert/scripts/run_squad_inference.sh  
  inflating: /workspace/bert/scripts/start_pretraining.sh  
  inflating: /workspace/bert/tokenization.py  
  inflating: /workspace/bert/tokenization_test.py

BERT Config

# Download BERT vocab file
!mkdir -p /workspace/bert/config.qa
!wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt \
    -O /workspace/bert/config.qa/vocab.txt

--2021-03-16 00:22:20--  https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.80.246
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.80.246|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 231508 (226K) [text/plain]
Saving to: ‘/workspace/bert/config.qa/vocab.txt’

/workspace/bert/con 100%[===================>] 226.08K   364KB/s    in 0.6s    

2021-03-16 00:22:21 (364 KB/s) - ‘/workspace/bert/config.qa/vocab.txt’ saved [231508/231508]

%%writefile /workspace/bert/config.qa/bert_config.json
{
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "max_position_embeddings": 512,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

Writing /workspace/bert/config.qa/bert_config.json

Helper Functions

# Create dynamic JSON files based on user inputs
def write_input_file(context, qinputs, predict_file):
    # Remove quotes and new lines from text for valid JSON
    context = context.replace('"', '').replace('\n', '')
    # Create JSON dict to write
    json_dict = {
      "data": [
        {
          "title": "BERT QA",
          "paragraphs": [
            {
              "context": context,
              "qas": qinputs
            }
          ]
        }
      ]
    }
    # Write JSON to input file
    with open(predict_file, 'w') as json_file:
        import json
        json.dump(json_dict, json_file, indent=2)
    
# Display Inference Results as HTML Table
def display_results(predict_file, output_prediction_file):
    import json
    from IPython.display import display, HTML

    # Here we show only the prediction results, nbest prediction is also available in the output directory
    results = ""
    with open(predict_file, 'r') as query_file:
        queries = json.load(query_file)
        input_data = queries["data"]
        with open(output_prediction_file, 'r') as result_file:
            data = json.load(result_file)
            for entry in input_data:
                for paragraph in entry["paragraphs"]:
                    for qa in paragraph["qas"]:
                        results += "<tr><td>{}</td><td>{}</td><td>{}</td></tr>".format(qa["id"], qa["question"], data[qa["id"]])

    display(HTML("<table><tr><th>Id</th><th>Question</th><th>Answer</th></tr>{}</table>".format(results)))

3. BERT Inference: Question Answering

We can run inference on a fine-tuned BERT model for tasks like Question Answering.

Here we use a BERT model fine-tuned on a SQuaD 2.0 Dataset which contains 100,000+ question-answer pairs on 500+ articles combined with over 50,000 new, unanswerable questions.

Paragraph and Queries

In this example we will ask our BERT model questions related to the following paragraph:

The Apollo Program "The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two-man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975."

The paragraph and the questions can be easily customized by changing the code below:

# Create BERT input file with (1) context and (2) questions to be answered based on that context
predict_file = '/workspace/bert/config.qa/input.json'

%%writefile $predict_file
{"data": 
 [
     {"title": "Project Apollo",
      "paragraphs": [
          {"context":"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975.", 
           "qas": [
               { "question": "What project put the first Americans into space?", 
                 "id": "Q1"
               },
               { "question": "What program was created to carry out these projects and missions?",
                 "id": "Q2"
               },
               { "question": "What year did the first manned Apollo flight occur?",
                 "id": "Q3"
               },                
               { "question": "What President is credited with the notion of putting Americans on the moon?",
                 "id": "Q4"
               },
               { "question": "Who did the U.S. collaborate with on an Earth orbit mission in 1975?",
                 "id": "Q5"
               },
               { "question": "How long did Project Apollo run?",
                 "id": "Q6"
               },               
               { "question": "What program helped develop space travel techniques that Project Apollo used?",
                 "id": "Q7"
               },                
               {"question": "What space station supported three manned missions in 1973-1974?",
                 "id": "Q8"
               }
]}]}]}

Writing /workspace/bert/config.qa/input.json

4. Running Question/Answer Inference

To run QA inference we will launch the script run_squad.py with the following parameters:

import os

# This specifies the model architecture.
bert_config_file = '/workspace/bert/config.qa/bert_config.json'

# The vocabulary file that the BERT model was trained on.
vocab_file = '/workspace/bert/config.qa/vocab.txt'

# Initiate checkpoint to the fine-tuned BERT Large model
init_checkpoint = os.path.join('/workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt')

# Create the output directory where all the results are saved.
output_dir = '/workspace/bert/results'
output_prediction_file = os.path.join(output_dir,'predictions.json')
    
# Whether to lower case the input - True for uncased models / False for cased models.
do_lower_case = True
  
# Total batch size for predictions
predict_batch_size = 8

# Whether to run eval on the dev set.
do_predict = True

# When splitting up a long document into chunks, how much stride to take between chunks.
doc_stride = 128

# The maximum total input sequence length after WordPiece tokenization.
# Sequences longer than this will be truncated, and sequences shorter than this will be padded.
max_seq_length = 384

4a. Run Inference

# Ask BERT questions
!python /workspace/bert/run_squad.py \
  --bert_config_file=$bert_config_file \
  --vocab_file=$vocab_file \
  --init_checkpoint=$init_checkpoint \
  --output_dir=$output_dir \
  --do_predict=$do_predict \
  --predict_file=$predict_file \
  --predict_batch_size=$predict_batch_size \
  --doc_stride=$doc_stride \
  --max_seq_length=$max_seq_length

2021-03-16 00:22:33.385548: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:162: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0316 00:22:34.840889 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W0316 00:22:34.841060 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0316 00:22:34.841254 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0316 00:22:34.842063 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0316 00:22:34.972105 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0316 00:22:34.972314 140590680020800 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fdd090fc3a0>) includes params argument, but params are not passed to Estimator.
W0316 00:22:35.720636 140590680020800 estimator.py:1992] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fdd090fc3a0>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fdcff6c1370>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I0316 00:22:35.721187 140590680020800 estimator.py:212] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fdcff6c1370>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I0316 00:22:35.721480 140590680020800 tpu_context.py:220] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0316 00:22:35.721641 140590680020800 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

W0316 00:22:35.721812 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

W0316 00:22:35.723536 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0316 00:22:35.794478 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:***** Running predictions *****
I0316 00:22:35.794595 140590680020800 run_squad.py:1354] ***** Running predictions *****
INFO:tensorflow:  Num orig examples = 8
I0316 00:22:35.794691 140590680020800 run_squad.py:1355]   Num orig examples = 8
INFO:tensorflow:  Num split examples = 8
I0316 00:22:35.794868 140590680020800 run_squad.py:1356]   Num split examples = 8
INFO:tensorflow:  Batch size = 8
I0316 00:22:35.794964 140590680020800 run_squad.py:1357]   Batch size = 8
WARNING:tensorflow:From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0316 00:22:35.795106 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

INFO:tensorflow:Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
I0316 00:22:35.795398 140590680020800 estimator.py:614] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
WARNING:tensorflow:Entity <function input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7fdcff6d63a0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults'
W0316 00:22:35.851542 140590680020800 ag_logging.py:146] Entity <function input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7fdcff6d63a0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults'
WARNING:tensorflow:From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W0316 00:22:35.851733 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

INFO:tensorflow:Calling model_fn.
I0316 00:22:35.867721 140590680020800 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Running infer on CPU
I0316 00:22:35.867883 140590680020800 tpu_estimator.py:3124] Running infer on CPU
WARNING:tensorflow:From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0316 00:22:35.870865 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0316 00:22:35.872210 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

W0316 00:22:35.897977 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

W0316 00:22:39.388511 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

W0316 00:22:39.394683 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

INFO:tensorflow:Done calling model_fn.
I0316 00:22:40.470538 140590680020800 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Graph was finalized.
I0316 00:22:41.255115 140590680020800 monitored_session.py:240] Graph was finalized.
2021-03-16 00:22:41.283573: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198615000 Hz
2021-03-16 00:22:41.286205: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54540d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-16 00:22:41.286256: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-03-16 00:22:41.290408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-03-16 00:22:41.533042: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54339f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-16 00:22:41.533093: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-03-16 00:22:41.537140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: 
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2021-03-16 00:22:41.537196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-16 00:22:41.554704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-03-16 00:22:41.562981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-03-16 00:22:41.565560: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-03-16 00:22:41.581478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-03-16 00:22:41.585296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-03-16 00:22:41.585491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-03-16 00:22:41.590070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-03-16 00:22:41.590119: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-16 00:22:42.007522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-16 00:22:42.007563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212]      0 
2021-03-16 00:22:42.007576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0:   N 
2021-03-16 00:22:42.011538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 31026 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
INFO:tensorflow:Running local_init_op.
I0316 00:22:44.737054 140590680020800 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0316 00:22:44.841894 140590680020800 session_manager.py:502] Done running local_init_op.
2021-03-16 00:22:46.448377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
INFO:tensorflow:Processing example: 0
I0316 00:22:47.535837 140590680020800 run_squad.py:1373] Processing example: 0
INFO:tensorflow:prediction_loop marked as finished
I0316 00:22:47.610052 140590680020800 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:prediction_loop marked as finished
I0316 00:22:47.610238 140590680020800 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:-----------------------------
I0316 00:22:47.610357 140590680020800 run_squad.py:1388] -----------------------------
INFO:tensorflow:0 Total Inference Time = 11.82 Inference Time W/O start up overhead = 2.01 Sentences processed = 8
I0316 00:22:47.610444 140590680020800 run_squad.py:1389] 0 Total Inference Time = 11.82 Inference Time W/O start up overhead = 2.01 Sentences processed = 8
INFO:tensorflow:0 Inference Performance = 3.9767 sentences/sec
I0316 00:22:47.610535 140590680020800 run_squad.py:1392] 0 Inference Performance = 3.9767 sentences/sec
INFO:tensorflow:-----------------------------
I0316 00:22:47.610617 140590680020800 run_squad.py:1393] -----------------------------
INFO:tensorflow:Writing predictions to: /workspace/bert/results/predictions.json
I0316 00:22:47.610741 140590680020800 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json
INFO:tensorflow:Writing nbest to: /workspace/bert/results/nbest_predictions.json
I0316 00:22:47.610820 140590680020800 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json

4b. Display Results:

display_results(predict_file, output_prediction_file)
IdQuestionAnswer
Q1What project put the first Americans into space?Project Mercury
Q2What program was created to carry out these projects and missions?The Apollo program
Q3What year did the first manned Apollo flight occur?1968
Q4What President is credited with the notion of putting Americans on the moon?John F. Kennedy
Q5Who did the U.S. collaborate with on an Earth orbit mission in 1975?Soviet Union
Q6How long did Project Apollo run?1961 to 1972
Q7What program helped develop space travel techniques that Project Apollo used?Gemini missions
Q8What space station supported three manned missions in 1973-1974?Skylab
Click to reveal expected answers to the questions above
IdQuestionAnswer
Q1What project put the first Americans into space?Project Mercury
Q2What program was created to carry out these projects and missions?The Apollo program
Q3What year did the first manned Apollo flight occur?1968
Q4What President is credited with the notion of putting Americans on the moon?John F. Kennedy
Q5Who did the U.S. collaborate with on an Earth orbit mission in 1975?Soviet Union
Q6How long did Project Apollo run?1961 to 1972
Q7What program helped develop space travel techniques that Project Apollo used?Gemini missions
Q8What space station supported three manned missions in 1973-1974?Skylab

5. Custom Inputs

Now that you are familiar with running QA Inference on BERT, you may want to try your own paragraphs and queries.

  1. Copy and paste your context from Wikipedia, news articles, etc. when prompted below
  2. Enter questions based on the context when prompted below.
  3. Run the inference script
  4. Display the inference results
predict_file = '/workspace/bert/config.qa/custom_input.json'
num_questions = 3           # You can configure this number

# Create your own context to ask questions about.
context = input("Paste your context here: ")

Paste your context here: Alan Mathison Turing OBE FRS (/ˈtjʊərɪŋ/; 23 June 1912 – 7 June 1954) was an English[6] mathematician, computer scientist, logician, cryptanalyst, philosopher and theoretical biologist.[7] Turing was highly influential in the development of theoretical computer science, providing a formalisation of the concepts of algorithm and computation with the Turing machine, which can be considered a model of a general-purpose computer.[8][9][10] Turing is widely considered to be the father of theoretical computer science and artificial intelligence.[11] Despite these accomplishments, he was not fully recognised in his home country during his lifetime, due to his homosexuality, and because much of his work was covered by the Official Secrets Act.  During the Second World War, Turing worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain's codebreaking centre that produced Ultra intelligence. For a time he led Hut 8, the section that was responsible for German naval cryptanalysis. Here, he devised a number of techniques for speeding the breaking of German ciphers, including improvements to the pre-war Polish bombe method, an electromechanical machine that could find settings for the Enigma machine.  Turing played a pivotal role in cracking intercepted coded messages that enabled the Allies to defeat the Nazis in many crucial engagements, including the Battle of the Atlantic, and in so doing helped win the war.[12][13] Due to the problems of counterfactual history, it's hard to estimate what effect Ultra intelligence had on the war,[14] but at the upper end it has been estimated that this work shortened the war in Europe by more than two years and saved over 14 million lives.[12]  After the war, Turing worked at the National Physical Laboratory, where he designed the Automatic Computing Engine, which was one of the first designs for a stored-program computer. In 1948, Turing joined Max Newman's Computing Machine Laboratory at the Victoria University of Manchester, where he helped develop the Manchester computers[15] and became interested in mathematical biology. He wrote a paper on the chemical basis of morphogenesis[1] and predicted oscillating chemical reactions such as the Belousov–Zhabotinsky reaction, first observed in the 1960s.  Turing was prosecuted in 1952 for homosexual acts; the Labouchere Amendment of 1885 had mandated that "gross indecency" was a criminal offence in the UK. He accepted chemical castration treatment, with DES, as an alternative to prison. Turing died in 1954, 16 days before his 42nd birthday, from cyanide poisoning. An inquest determined his death as a suicide, but it has been noted that the known evidence is also consistent with accidental poisoning.  In 2009, following an Internet campaign, British Prime Minister Gordon Brown made an official public apology on behalf of the British government for "the appalling way he was treated". Queen Elizabeth II granted Turing a posthumous pardon in 2013. The Alan Turing law is now an informal term for a 2017 law in the United Kingdom that retroactively pardoned men cautioned or convicted under historical legislation that outlawed homosexual acts.[16]  On 15 July 2019 the Bank of England announced that Turing would be depicted on the United Kingdom's new £50 note.

# Get questions from user input
questions = [input("Question {}/{}: ".format(i+1, num_questions)) for i in range(num_questions)]
# Format questions and write to JSON input file
qinputs = [{ "question":q, "id":"Q{}".format(i+1)} for i,q in enumerate(questions)]
write_input_file(context, qinputs, predict_file)

Question 1/3: Who is Alan Turing?
Question 2/3: When was Alan Turing born?
Question 3/3: Which laboratory Turing joined in 1948?

# Ask BERT questions
!python /workspace/bert/run_squad.py \
  --bert_config_file=$bert_config_file \
  --vocab_file=$vocab_file \
  --init_checkpoint=$init_checkpoint \
  --output_dir=$output_dir \
  --do_predict=$do_predict \
  --predict_file=$predict_file \
  --predict_batch_size=$predict_batch_size \
  --doc_stride=$doc_stride \
  --max_seq_length=$max_seq_length

2021-03-16 00:25:20.406731: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:162: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0316 00:25:21.868111 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W0316 00:25:21.868280 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0316 00:25:21.868470 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0316 00:25:21.869253 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0316 00:25:21.998484 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0316 00:25:21.998684 140608677435200 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fe139ce63a0>) includes params argument, but params are not passed to Estimator.
W0316 00:25:22.744968 140608677435200 estimator.py:1992] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fe139ce63a0>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1302d2370>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I0316 00:25:22.745511 140608677435200 estimator.py:212] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1302d2370>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I0316 00:25:22.745804 140608677435200 tpu_context.py:220] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0316 00:25:22.745965 140608677435200 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

W0316 00:25:22.746137 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

W0316 00:25:22.749643 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0316 00:25:22.830683 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:***** Running predictions *****
I0316 00:25:22.830802 140608677435200 run_squad.py:1354] ***** Running predictions *****
INFO:tensorflow:  Num orig examples = 3
I0316 00:25:22.830897 140608677435200 run_squad.py:1355]   Num orig examples = 3
INFO:tensorflow:  Num split examples = 12
I0316 00:25:22.831071 140608677435200 run_squad.py:1356]   Num split examples = 12
INFO:tensorflow:  Batch size = 8
I0316 00:25:22.831164 140608677435200 run_squad.py:1357]   Batch size = 8
WARNING:tensorflow:From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0316 00:25:22.831306 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

INFO:tensorflow:Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
I0316 00:25:22.831603 140608677435200 estimator.py:614] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
WARNING:tensorflow:Entity <function input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7fe1302d1a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults'
W0316 00:25:22.888306 140608677435200 ag_logging.py:146] Entity <function input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7fe1302d1a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults'
WARNING:tensorflow:From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W0316 00:25:22.888502 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

INFO:tensorflow:Calling model_fn.
I0316 00:25:22.904676 140608677435200 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Running infer on CPU
I0316 00:25:22.904836 140608677435200 tpu_estimator.py:3124] Running infer on CPU
WARNING:tensorflow:From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0316 00:25:22.907857 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0316 00:25:22.909193 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

W0316 00:25:22.935186 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

W0316 00:25:26.434307 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

W0316 00:25:26.440835 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

INFO:tensorflow:Done calling model_fn.
I0316 00:25:27.519404 140608677435200 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Graph was finalized.
I0316 00:25:28.309463 140608677435200 monitored_session.py:240] Graph was finalized.
2021-03-16 00:25:28.339586: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198615000 Hz
2021-03-16 00:25:28.341293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5978d70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-16 00:25:28.341321: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-03-16 00:25:28.344109: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-03-16 00:25:28.580121: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8b42f70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-16 00:25:28.580169: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-03-16 00:25:28.583618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: 
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2021-03-16 00:25:28.583675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-16 00:25:28.589475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-03-16 00:25:28.591388: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-03-16 00:25:28.591844: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-03-16 00:25:28.596971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-03-16 00:25:28.597815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-03-16 00:25:28.597983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-03-16 00:25:28.601731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-03-16 00:25:28.601770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-03-16 00:25:28.994633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-16 00:25:28.994679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212]      0 
2021-03-16 00:25:28.994692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0:   N 
2021-03-16 00:25:28.998624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 31026 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
INFO:tensorflow:Running local_init_op.
I0316 00:25:31.745624 140608677435200 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0316 00:25:31.858720 140608677435200 session_manager.py:502] Done running local_init_op.
2021-03-16 00:25:33.689415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
INFO:tensorflow:Processing example: 0
I0316 00:25:34.384404 140608677435200 run_squad.py:1373] Processing example: 0
INFO:tensorflow:prediction_loop marked as finished
I0316 00:25:34.586190 140608677435200 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:prediction_loop marked as finished
I0316 00:25:34.586374 140608677435200 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:-----------------------------
I0316 00:25:34.586491 140608677435200 run_squad.py:1388] -----------------------------
INFO:tensorflow:0 Total Inference Time = 11.76 Inference Time W/O start up overhead = 1.89 Sentences processed = 16
I0316 00:25:34.586577 140608677435200 run_squad.py:1389] 0 Total Inference Time = 11.76 Inference Time W/O start up overhead = 1.89 Sentences processed = 16
INFO:tensorflow:0 Inference Performance = 8.4703 sentences/sec
I0316 00:25:34.586667 140608677435200 run_squad.py:1392] 0 Inference Performance = 8.4703 sentences/sec
INFO:tensorflow:-----------------------------
I0316 00:25:34.586749 140608677435200 run_squad.py:1393] -----------------------------
INFO:tensorflow:Writing predictions to: /workspace/bert/results/predictions.json
I0316 00:25:34.586873 140608677435200 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json
INFO:tensorflow:Writing nbest to: /workspace/bert/results/nbest_predictions.json
I0316 00:25:34.586952 140608677435200 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json

display_results(predict_file, output_prediction_file)
IdQuestionAnswer
Q1Who is Alan Turing?father of theoretical computer science and artificial intelligence
Q2When was Alan Turing born?23 June 1912
Q3Which laboratory Turing joined in 1948?Max Newman's Computing Machine Laboratory at the Victoria University of Manchester

NVIDIA uses cookies to improve your experience on our web site. We and our third-party partners also use cookies and other tools to collect and record information you provide as well as information about your interactions with our websites for performance improvement, analytics, and to assist in marketing efforts. By clicking "Accept All", you consent to our use of cookies and other tools as described in our Cookie Policy. You can manage your cookie settings by clicking on "Manage Settings." By continuing to use this site or by clicking one of the buttons below, you agree to our Terms of Service (which contains important waivers). Please see our Privacy Policy for more information on our privacy practices.