BERT PyTorch checkpoint (Dist-4L-288D, Pretraining, AMP) | NVIDIA NGC

NVIDIA Deep Learning Examples

BERT PyTorch checkpoint (Dist-4L-288D, Pretraining, AMP)

Model

NVIDIA Deep Learning Examples

BERT PyTorch checkpoint (Dist-4L-288D, Pretraining, AMP)

BERT Distilled 4L-288D PyTorch checkpoint distilled on pretraining dataset using AMP

1 Version

21.11.0_ampSelected

07/07/2022 2:25 PM UTC50.45 MBAccuracy: 00 EpochsBatch Size: 256GPU: V100

architecture

Key	Value
attn_heads	12
task	Pretraining
hidden_size	288
layers	4
type	Dist-4L-288D

performance

Key	Value
loss	0.6395510681051957
rep_loss	0.30611084806291683
att_loss	0.29690841490762276
value_loss	0.03653180576468769

training

Key	Value
batch_size	256
LR	0.0001
training_precision	AMP
max_seq_length	512
steps	6861
dataset	wikipedia+bookcorpus
warmup_proportion	0.1