NVIDIA

BERT for TensorFlow Jupyter Notebook

Resource

NVIDIA

BERT for TensorFlow Jupyter Notebook

Jupyter Notebook example for Question Answering with BERT for TensorFlow

Back to File Browser

BERT_Jupyter.ipynb

BERT Question Answering in TensorFlow with Mixed Precision

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

In [1]:

!nvidia-smi

Fri Oct  4 18:14:07 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.145                Driver Version: 384.145                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-DGXS...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   33C    P0    32W / 300W |     11MiB / 16271MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-DGXS...  On   | 00000000:08:00.0 Off |                    0 |
| N/A   33C    P0    32W / 300W |      2MiB / 16273MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-DGXS...  On   | 00000000:0E:00.0 Off |                    0 |
| N/A   32C    P0    31W / 300W |      2MiB / 16273MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-DGXS...  On   | 00000000:0F:00.0 Off |                    0 |
| N/A   31C    P0    32W / 300W |      2MiB / 16273MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

1. Overview

Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.

The original paper can be found here: https://arxiv.org/abs/1810.04805.

NVIDIA's BERT is an optimized version of Google's official implementation, leveraging mixed precision arithmetic and tensor cores on V100 GPUS for faster training times while maintaining target accuracy.

Learning objectives

This notebook demonstrates:

Inference on Question Answering (QA) task with BERT Large model
The use/download of fine-tuned NVIDIA BERT models from NGC
Use of Mixed Precision models for Inference

2. Setup

Pre-Trained NVIDIA BERT TensorFlow Models on NGC

We will be using the following configuration of BERT in this example:

Model	Hidden layers	Hidden unit size	Attention heads	Feedforward filter size	Max sequence length	Parameters
BERTLARGE	24 encoder	1024	16	4 x 1024	512	330M

To do so, we will take advantage of the pre-trained models available on the NGC Model Registry.

Among the many configurations available we will download one of these two:

bert_tf_v2_large_fp32_384
bert_tf_v2_large_fp16_384

which are trained on the SQuaD 2.0 Dataset.

We can choose the mixed precision model (which takes much less time to train than the fp32 version) without losing accuracy, with the following flag:

In [15]:

use_mixed_precision_model = True

In [16]:

if use_mixed_precision_model:
    # bert_tf_v2_large_fp16_384
    !mkdir -p /workspace/bert/data/finetuned_model_fp16
    !wget -nc -q --show-progress -O /workspace/bert/data/finetuned_model_fp16/bert_tf_v2_large_fp16_384.zip \
    https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_v2_large_fp16_384/versions/1/zip
    !unzip -n -d /workspace/bert/data/finetuned_model_fp16/ /workspace/bert/data/finetuned_model_fp16/bert_tf_v2_large_fp16_384.zip 
else:
    # bert_tf_v2_large_fp32_384
    !mkdir -p /workspace/bert/data/finetuned_model_fp32
    !wget -nc -q --show-progress -O /workspace/bert/data/finetuned_model_fp32/bert_tf_v2_large_fp32_384.zip \
    https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_v2_large_fp32_384/versions/1/zip
    !unzip -n -d /workspace/bert/data/finetuned_model_fp32/ /workspace/bert/data/finetuned_model_fp32/bert_tf_v2_large_fp32_384.zip

Archive:  /workspace/bert/data/finetuned_model_fp16/bert_tf_v2_large_fp16_384.zip

NGC Model Scripts

While we're at it, we'll also pull down some BERT helper scripts from the NGC Model Scripts Registry

In [17]:

# Download BERT helper scripts
!wget -nc --show-progress -O bert_scripts.zip \
     https://api.ngc.nvidia.com/v2/recipes/nvidia/bert_for_tensorflow/versions/1/zip
!mkdir -p /workspace/bert
!unzip -n -d /workspace/bert bert_scripts.zip

File ‘bert_scripts.zip’ already there; not retrieving.
Archive:  bert_scripts.zip

BERT Config

In [18]:

# Download BERT vocab file
!mkdir -p /workspace/bert/config.qa
!wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt \
    -O /workspace/bert/config.qa/vocab.txt

File ‘/workspace/bert/config.qa/vocab.txt’ already there; not retrieving.

In [19]:

%%writefile /workspace/bert/config.qa/bert_config.json
{
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "max_position_embeddings": 512,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

Overwriting /workspace/bert/config.qa/bert_config.json

Helper Functions

In [20]:

# Create dynamic JSON files based on user inputs
def write_input_file(context, qinputs, predict_file):
    # Remove quotes and new lines from text for valid JSON
    context = context.replace('"', '').replace('\n', '')
    # Create JSON dict to write
    json_dict = {
      "data": [
        {
          "title": "BERT QA",
          "paragraphs": [
            {
              "context": context,
              "qas": qinputs
            }
          ]
        }
      ]
    }
    # Write JSON to input file
    with open(predict_file, 'w') as json_file:
        import json
        json.dump(json_dict, json_file, indent=2)
    
# Display Inference Results as HTML Table
def display_results(predict_file, output_prediction_file):
    import json
    from IPython.display import display, HTML

    # Here we show only the prediction results, nbest prediction is also available in the output directory
    results = ""
    with open(predict_file, 'r') as query_file:
        queries = json.load(query_file)
        input_data = queries["data"]
        with open(output_prediction_file, 'r') as result_file:
            data = json.load(result_file)
            for entry in input_data:
                for paragraph in entry["paragraphs"]:
                    for qa in paragraph["qas"]:
                        results += "<tr><td>{}</td><td>{}</td><td>{}</td></tr>".format(qa["id"], qa["question"], data[qa["id"]])

    display(HTML("<table><tr><th>Id</th><th>Question</th><th>Answer</th></tr>{}</table>".format(results)))

3. BERT Inference: Question Answering

We can run inference on a fine-tuned BERT model for tasks like Question Answering.

Here we use a BERT model fine-tuned on a SQuaD 2.0 Dataset which contains 100,000+ question-answer pairs on 500+ articles combined with over 50,000 new, unanswerable questions.

Paragraph and Queries

In this example we will ask our BERT model questions related to the following paragraph:

The Apollo Program "The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two-man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975."

The paragraph and the questions can be easily customized by changing the code below:

In [21]:

# Create BERT input file with (1) context and (2) questions to be answered based on that context
predict_file = '/workspace/bert/config.qa/input.json'

In [22]:

%%writefile $predict_file
{"data": 
 [
     {"title": "Project Apollo",
      "paragraphs": [
          {"context":"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975.", 
           "qas": [
               { "question": "What project put the first Americans into space?", 
                 "id": "Q1"
               },
               { "question": "What program was created to carry out these projects and missions?",
                 "id": "Q2"
               },
               { "question": "What year did the first manned Apollo flight occur?",
                 "id": "Q3"
               },                
               { "question": "What President is credited with the notion of putting Americans on the moon?",
                 "id": "Q4"
               },
               { "question": "Who did the U.S. collaborate with on an Earth orbit mission in 1975?",
                 "id": "Q5"
               },
               { "question": "How long did Project Apollo run?",
                 "id": "Q6"
               },               
               { "question": "What program helped develop space travel techniques that Project Apollo used?",
                 "id": "Q7"
               },                
               {"question": "What space station supported three manned missions in 1973-1974?",
                 "id": "Q8"
               }
]}]}]}

Overwriting /workspace/bert/config.qa/input.json

To effectively evaluate the speedup of mixed precision try a bigger workload by uncommenting the following lines:

TODO: Waiting on model scripts repo to fix Windows newlines in squad_download.sh

In [23]:

#!bash /workspace/bert/data/squad/squad_download.sh
#predict_file = '/workspace/bert/data/squad/v2.0/dev-v2.0.json'

4. Running Question/Answer Inference

To run QA inference we will launch the script run_squad.py with the following parameters:

In [24]:

import os

# This specifies the model architecture.
bert_config_file = '/workspace/bert/config.qa/bert_config.json'

# The vocabulary file that the BERT model was trained on.
vocab_file = '/workspace/bert/config.qa/vocab.txt'

# Depending on the mixed precision flag we use different fine-tuned model
if use_mixed_precision_model:
    init_checkpoint = '/workspace/bert/data/finetuned_model_fp16/model.ckpt-8144'
else:
    init_checkpoint = '/workspace/bert/data/finetuned_model_fp32/model.ckpt-8144'

# Create the output directory where all the results are saved.
output_dir = '/workspace/bert/results'
output_prediction_file = os.path.join(output_dir,'predictions.json')
    
# Whether to lower case the input - True for uncased models / False for cased models.
do_lower_case = True
  
# Total batch size for predictions
predict_batch_size = 8

# Whether to run eval on the dev set.
do_predict = True

# When splitting up a long document into chunks, how much stride to take between chunks.
doc_stride = 128

# The maximum total input sequence length after WordPiece tokenization.
# Sequences longer than this will be truncated, and sequences shorter than this will be padded.
max_seq_length = 384

4a. Run Inference

In [25]:

# Ask BERT questions
!python /workspace/bert/run_squad.py \
  --bert_config_file=$bert_config_file \
  --vocab_file=$vocab_file \
  --init_checkpoint=$init_checkpoint \
  --output_dir=$output_dir \
  --do_predict=$do_predict \
  --predict_file=$predict_file \
  --predict_batch_size=$predict_batch_size \
  --doc_stride=$doc_stride \
  --max_seq_length=$max_seq_length

2019-10-04 17:41:20.232821: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W1004 17:41:21.340837 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

W1004 17:41:21.354933 140007069570880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/horovod-0.16.2-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:107: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

W1004 17:41:21.356337 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W1004 17:41:21.356724 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W1004 17:41:21.356826 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W1004 17:41:21.356929 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W1004 17:41:21.357506 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1004 17:41:21.442186 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1004 17:41:21.945942 140007069570880 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1004 17:41:21.946371 140007069570880 estimator.py:1984] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f551bca6048>) includes params argument, but params are not passed to Estimator.
I1004 17:41:21.947021 140007069570880 estimator.py:209] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f55102d5e10>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2), '_cluster': None}
I1004 17:41:21.947435 140007069570880 tpu_context.py:209] _TPUContext: eval_on_tpu True
W1004 17:41:21.947723 140007069570880 tpu_context.py:211] eval_on_tpu ignored because use_tpu is False.
W1004 17:41:21.947842 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

W1004 17:41:21.948863 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

W1004 17:41:21.994768 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

I1004 17:41:21.994851 140007069570880 run_squad.py:1354] ***** Running predictions *****
I1004 17:41:21.994913 140007069570880 run_squad.py:1355]   Num orig examples = 8
I1004 17:41:21.995212 140007069570880 run_squad.py:1356]   Num split examples = 8
I1004 17:41:21.995281 140007069570880 run_squad.py:1357]   Batch size = 8
W1004 17:41:21.995372 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

I1004 17:41:21.995577 140007069570880 estimator.py:612] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
W1004 17:41:22.015170 140007069570880 deprecation.py:323] From /workspace/bert/run_squad.py:776: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.map_and_batch(...)`.
W1004 17:41:22.015273 140007069570880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/data/python/ops/batching.py:273: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.
W1004 17:41:22.016580 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W1004 17:41:22.019920 140007069570880 deprecation.py:323] From /workspace/bert/run_squad.py:750: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
I1004 17:41:22.035059 140007069570880 estimator.py:1145] Calling model_fn.
I1004 17:41:22.035180 140007069570880 tpu_estimator.py:2965] Running infer on CPU
W1004 17:41:22.038202 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1004 17:41:22.039526 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W1004 17:41:22.120327 140007069570880 deprecation.py:323] From /workspace/bert/modeling.py:675: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
W1004 17:41:26.304313 140007069570880 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

I1004 17:41:27.458134 140007069570880 estimator.py:1147] Done calling model_fn.
W1004 17:41:27.706729 140007069570880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py:1354: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I1004 17:41:28.256394 140007069570880 monitored_session.py:240] Graph was finalized.
2019-10-04 17:41:28.261712: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3198235000 Hz
2019-10-04 17:41:28.262294: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f27410 executing computations on platform Host. Devices:
2019-10-04 17:41:28.262310: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-10-04 17:41:28.264070: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1

2019-10-04 17:41:28.720664: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ffe6b0 executing computations on platform CUDA. Devices:
2019-10-04 17:41:28.720686: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:41:28.720691: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:41:28.720696: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:41:28.720701: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:41:28.721833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:07:00.0
2019-10-04 17:41:28.722693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:08:00.0
2019-10-04 17:41:28.723557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:0e:00.0
2019-10-04 17:41:28.724414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:0f:00.0
2019-10-04 17:41:28.724439: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-10-04 17:41:28.725878: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-10-04 17:41:28.727100: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-10-04 17:41:28.727325: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-10-04 17:41:28.728672: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-10-04 17:41:28.729471: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-10-04 17:41:28.732386: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-10-04 17:41:28.739182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-10-04 17:41:28.739210: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-10-04 17:41:30.158485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-04 17:41:30.158518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2019-10-04 17:41:30.158526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y 
2019-10-04 17:41:30.158531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y 
2019-10-04 17:41:30.158538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y 
2019-10-04 17:41:30.158544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N 
2019-10-04 17:41:30.163221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15127 MB memory) -> physical GPU (device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0)
2019-10-04 17:41:30.164546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15138 MB memory) -> physical GPU (device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0)
2019-10-04 17:41:30.165791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15138 MB memory) -> physical GPU (device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-04 17:41:30.167029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15138 MB memory) -> physical GPU (device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0)
2019-10-04 17:41:31.909400: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I1004 17:41:31.948915 140007069570880 session_manager.py:500] Running local_init_op.
I1004 17:41:32.015150 140007069570880 session_manager.py:502] Done running local_init_op.
2019-10-04 17:41:33.244791: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
I1004 17:41:33.821089 140007069570880 run_squad.py:1373] Processing example: 0
I1004 17:41:33.911081 140007069570880 error_handling.py:96] prediction_loop marked as finished
I1004 17:41:33.911292 140007069570880 error_handling.py:96] prediction_loop marked as finished
I1004 17:41:33.911444 140007069570880 run_squad.py:1388] -----------------------------
I1004 17:41:33.911543 140007069570880 run_squad.py:1391] 0 Total Inference Time = 11.92 Inference Time W/O start up overhead = 1.31 Sentences processed = 8
I1004 17:41:33.911634 140007069570880 run_squad.py:1392] 0 Inference Performance = 6.1243 sentences/sec
I1004 17:41:33.911716 140007069570880 run_squad.py:1393] -----------------------------
I1004 17:41:33.911850 140007069570880 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json
I1004 17:41:33.911930 140007069570880 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json

4b. Display Results:

In [26]:

display_results(predict_file, output_prediction_file)

Click to reveal expected answers to the questions above

Id	Question	Answer
Q1	What project put the first Americans into space?	Project Mercury
Q2	What program was created to carry out these projects and missions?	The Apollo program
Q3	What year did the first manned Apollo flight occur?	1968
Q4	What President is credited with the notion of putting Americans on the moon?	John F. Kennedy
Q5	Who did the U.S. collaborate with on an Earth orbit mission in 1975?	Soviet Union
Q6	How long did Project Apollo run?	1961 to 1972
Q7	What program helped develop space travel techniques that Project Apollo used?	Gemini missions
Q8	What space station supported three manned missions in 1973-1974?	Skylab

5. Custom Inputs

Now that you are familiar with running QA Inference on BERT, you may want to try your own paragraphs and queries.

Copy and paste your context from Wikipedia, news articles, etc. when prompted below
Enter questions based on the context when prompted below.
Run the inference script
Display the inference results

In [27]:

predict_file = '/workspace/bert/config.qa/custom_input.json'
num_questions = 3           # You can configure this number

In [28]:

# Create your own context to ask questions about.
context = input("Paste your context here: ")

Paste your context here: Alan Mathison Turing OBE FRS (/ˈtjʊərɪŋ/; 23 June 1912 – 7 June 1954) was an English[6] mathematician, computer scientist, logician, cryptanalyst, philosopher and theoretical biologist.[7] Turing was highly influential in the development of theoretical computer science, providing a formalisation of the concepts of algorithm and computation with the Turing machine, which can be considered a model of a general-purpose computer.[8][9][10] Turing is widely considered to be the father of theoretical computer science and artificial intelligence.[11] Despite these accomplishments, he was not fully recognised in his home country during his lifetime, due to his homosexuality, and because much of his work was covered by the Official Secrets Act.  During the Second World War, Turing worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain's codebreaking centre that produced Ultra intelligence. For a time he led Hut 8, the section that was responsible for German naval cryptanalysis. Here, he devised a number of techniques for speeding the breaking of German ciphers, including improvements to the pre-war Polish bombe method, an electromechanical machine that could find settings for the Enigma machine.  Turing played a pivotal role in cracking intercepted coded messages that enabled the Allies to defeat the Nazis in many crucial engagements, including the Battle of the Atlantic, and in so doing helped win the war.[12][13] Due to the problems of counterfactual history, it's hard to estimate what effect Ultra intelligence had on the war,[14] but at the upper end it has been estimated that this work shortened the war in Europe by more than two years and saved over 14 million lives.[12]  After the war, Turing worked at the National Physical Laboratory, where he designed the Automatic Computing Engine, which was one of the first designs for a stored-program computer. In 1948, Turing joined Max Newman's Computing Machine Laboratory at the Victoria University of Manchester, where he helped develop the Manchester computers[15] and became interested in mathematical biology. He wrote a paper on the chemical basis of morphogenesis[1] and predicted oscillating chemical reactions such as the Belousov–Zhabotinsky reaction, first observed in the 1960s.  Turing was prosecuted in 1952 for homosexual acts; the Labouchere Amendment of 1885 had mandated that "gross indecency" was a criminal offence in the UK. He accepted chemical castration treatment, with DES, as an alternative to prison. Turing died in 1954, 16 days before his 42nd birthday, from cyanide poisoning. An inquest determined his death as a suicide, but it has been noted that the known evidence is also consistent with accidental poisoning.  In 2009, following an Internet campaign, British Prime Minister Gordon Brown made an official public apology on behalf of the British government for "the appalling way he was treated". Queen Elizabeth II granted Turing a posthumous pardon in 2013. The Alan Turing law is now an informal term for a 2017 law in the United Kingdom that retroactively pardoned men cautioned or convicted under historical legislation that outlawed homosexual acts.[16]  On 15 July 2019 the Bank of England announced that Turing would be depicted on the United Kingdom's new £50 note.

In [29]:

# Get questions from user input
questions = [input("Question {}/{}: ".format(i+1, num_questions)) for i in range(num_questions)]
# Format questions and write to JSON input file
qinputs = [{ "question":q, "id":"Q{}".format(i+1)} for i,q in enumerate(questions)]
write_input_file(context, qinputs, predict_file)

Question 1/3: Who is Alan Turing?
Question 2/3: What did he do?
Question 3/3: Where did he work?

In [30]:

# Ask BERT questions
!python /workspace/bert/run_squad.py \
  --bert_config_file=$bert_config_file \
  --vocab_file=$vocab_file \
  --init_checkpoint=$init_checkpoint \
  --output_dir=$output_dir \
  --do_predict=$do_predict \
  --predict_file=$predict_file \
  --predict_batch_size=$predict_batch_size \
  --doc_stride=$doc_stride \
  --max_seq_length=$max_seq_length

2019-10-04 17:42:08.018740: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W1004 17:42:09.129620 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

W1004 17:42:09.143753 139990286526272 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/horovod-0.16.2-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:107: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

W1004 17:42:09.145139 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W1004 17:42:09.145527 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W1004 17:42:09.145627 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W1004 17:42:09.145733 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W1004 17:42:09.146319 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1004 17:42:09.227888 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1004 17:42:09.730068 139990286526272 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1004 17:42:09.730493 139990286526272 estimator.py:1984] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f5133717048>) includes params argument, but params are not passed to Estimator.
I1004 17:42:09.731131 139990286526272 estimator.py:209] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f5127d46320>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2), '_cluster': None}
I1004 17:42:09.731536 139990286526272 tpu_context.py:209] _TPUContext: eval_on_tpu True
W1004 17:42:09.731824 139990286526272 tpu_context.py:211] eval_on_tpu ignored because use_tpu is False.
W1004 17:42:09.731940 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

W1004 17:42:09.733937 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

W1004 17:42:09.784600 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

I1004 17:42:09.784681 139990286526272 run_squad.py:1354] ***** Running predictions *****
I1004 17:42:09.784742 139990286526272 run_squad.py:1355]   Num orig examples = 3
I1004 17:42:09.785042 139990286526272 run_squad.py:1356]   Num split examples = 12
I1004 17:42:09.785112 139990286526272 run_squad.py:1357]   Batch size = 8
W1004 17:42:09.785203 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

I1004 17:42:09.785398 139990286526272 estimator.py:612] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict.
W1004 17:42:09.805274 139990286526272 deprecation.py:323] From /workspace/bert/run_squad.py:776: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.map_and_batch(...)`.
W1004 17:42:09.805377 139990286526272 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/data/python/ops/batching.py:273: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.
W1004 17:42:09.806472 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W1004 17:42:09.809580 139990286526272 deprecation.py:323] From /workspace/bert/run_squad.py:750: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
I1004 17:42:09.825074 139990286526272 estimator.py:1145] Calling model_fn.
I1004 17:42:09.825195 139990286526272 tpu_estimator.py:2965] Running infer on CPU
W1004 17:42:09.828354 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1004 17:42:09.829684 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W1004 17:42:09.911925 139990286526272 deprecation.py:323] From /workspace/bert/modeling.py:675: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
W1004 17:42:14.175340 139990286526272 deprecation_wrapper.py:119] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

I1004 17:42:15.349575 139990286526272 estimator.py:1147] Done calling model_fn.
W1004 17:42:15.603036 139990286526272 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py:1354: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I1004 17:42:16.163598 139990286526272 monitored_session.py:240] Graph was finalized.
2019-10-04 17:42:16.169261: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3198235000 Hz
2019-10-04 17:42:16.169919: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x743d320 executing computations on platform Host. Devices:
2019-10-04 17:42:16.169938: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-10-04 17:42:16.172064: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1

2019-10-04 17:42:16.644922: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7515190 executing computations on platform CUDA. Devices:
2019-10-04 17:42:16.644944: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:42:16.644950: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:42:16.644955: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:42:16.644960: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla P100-DGXS-16GB, Compute Capability 6.0
2019-10-04 17:42:16.646154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:07:00.0
2019-10-04 17:42:16.647045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:08:00.0
2019-10-04 17:42:16.648029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:0e:00.0
2019-10-04 17:42:16.649059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: Tesla P100-DGXS-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.4805
pciBusID: 0000:0f:00.0
2019-10-04 17:42:16.649086: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-10-04 17:42:16.650940: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-10-04 17:42:16.652507: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-10-04 17:42:16.652792: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-10-04 17:42:16.654563: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-10-04 17:42:16.655673: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-10-04 17:42:16.659549: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-10-04 17:42:16.667120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-10-04 17:42:16.667152: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-10-04 17:42:18.080335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-04 17:42:18.080365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 
2019-10-04 17:42:18.080374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y 
2019-10-04 17:42:18.080379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y 
2019-10-04 17:42:18.080384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y 
2019-10-04 17:42:18.080389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N 
2019-10-04 17:42:18.085105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15127 MB memory) -> physical GPU (device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0)
2019-10-04 17:42:18.086382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15138 MB memory) -> physical GPU (device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0)
2019-10-04 17:42:18.087622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15138 MB memory) -> physical GPU (device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-04 17:42:18.088840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15138 MB memory) -> physical GPU (device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0)
2019-10-04 17:42:19.738443: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I1004 17:42:19.776933 139990286526272 session_manager.py:500] Running local_init_op.
I1004 17:42:19.842164 139990286526272 session_manager.py:502] Done running local_init_op.
2019-10-04 17:42:21.089302: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
I1004 17:42:21.652874 139990286526272 run_squad.py:1373] Processing example: 0
I1004 17:42:21.921473 139990286526272 error_handling.py:96] prediction_loop marked as finished
I1004 17:42:21.921690 139990286526272 error_handling.py:96] prediction_loop marked as finished
I1004 17:42:21.921820 139990286526272 run_squad.py:1388] -----------------------------
I1004 17:42:21.921907 139990286526272 run_squad.py:1391] 0 Total Inference Time = 12.14 Inference Time W/O start up overhead = 1.49 Sentences processed = 16
I1004 17:42:21.921993 139990286526272 run_squad.py:1392] 0 Inference Performance = 10.7314 sentences/sec
I1004 17:42:21.922071 139990286526272 run_squad.py:1393] -----------------------------
I1004 17:42:21.922402 139990286526272 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json
I1004 17:42:21.922497 139990286526272 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json

In [31]:

display_results(predict_file, output_prediction_file)

In [ ]: