ResNet v1.5 for MXNet

NVIDIA

Resource

NVIDIA

ResNet v1.5 for MXNet

With modified architecture and initialization this ResNet50 version gives ~0.5% better accuracy than original.

The following sections provide greater details of the dataset, running training and inference, and the training results.

Scripts and sample code

In the root directory, the most important files are:

runner: A wrapper on the train.py script which is the main executable script for training/validation/predicting.
benchmark.py: A script for benchmarking.
Dockerfile: Container to build the container.
fit.py: A file containing most of the training and validation logic.
data.py: Data loading and preprocessing code.
dali.py: Data loading and preprocessing code using DALI.
models.py: The model architecture.
report.py: A file containing JSON report structure and description of fields.

In the scripts directory, the most important files are:

prepare_imagenet.sh: A script that converts raw dataset format to RecordIO format.

Parameters

The complete list of available parameters contains:

Model:
 --arch {resnetv1,resnetv15,resnextv1,resnextv15,xception}
 model architecture (default: resnetv15)
 --num-layers NUM_LAYERS
 number of layers in the neural network, required by
 some networks such as resnet (default: 50)
 --num-groups NUM_GROUPS
 number of groups for grouped convolutions, required by
 some networks such as resnext (default: 32)
 --num-classes NUM_CLASSES
 the number of classes (default: 1000)
 --batchnorm-eps BATCHNORM_EPS
 the amount added to the batchnorm variance to prevent
 output explosion. (default: 1e-05)
 --batchnorm-mom BATCHNORM_MOM
 the leaky-integrator factor controling the batchnorm
 mean and variance. (default: 0.9)
 --fuse-bn-relu FUSE_BN_RELU
 have batchnorm kernel perform activation relu
 (default: 0)
 --fuse-bn-add-relu FUSE_BN_ADD_RELU
 have batchnorm kernel perform add followed by
 activation relu (default: 0)

Training:
 --mode {train_val,train,val,pred}
 mode (default: train_val)
 --seed SEED random seed (default: None)
 --gpus GPUS list of gpus to run, e.g. 0 or 0,2,5 (default: [0])
 --kv-store {device,horovod}
 key-value store type (default: device)
 --dtype {float32,float16}
 precision (default: float16)
 --amp If enabled, turn on AMP (Automatic Mixed Precision)
 (default: False)
 --batch-size BATCH_SIZE
 the batch size (default: 192)
 --num-epochs NUM_EPOCHS
 number of epochs (default: 90)
 --lr LR initial learning rate (default: 0.1)
 --lr-schedule {multistep,cosine}
 learning rate schedule (default: cosine)
 --lr-factor LR_FACTOR
 the ratio to reduce lr on each step (default: 0.256)
 --lr-steps LR_STEPS the epochs to reduce the lr, e.g. 30,60 (default: [])
 --warmup-epochs WARMUP_EPOCHS
 the epochs to ramp-up lr to scaled large-batch value
 (default: 5)
 --optimizer OPTIMIZER
 the optimizer type (default: sgd)
 --mom MOM momentum for sgd (default: 0.875)
 --wd WD weight decay for sgd (default: 3.0517578125e-05)
 --label-smoothing LABEL_SMOOTHING
 label smoothing factor (default: 0.1)
 --mixup MIXUP alpha parameter for mixup (if 0 then mixup is not
 applied) (default: 0)
 --disp-batches DISP_BATCHES
 show progress for every n batches (default: 20)
 --model-prefix MODEL_PREFIX
 model checkpoint prefix (default: model)
 --save-frequency SAVE_FREQUENCY
 frequency of saving model in epochs (--model-prefix
 must be specified). If -1 then save only best model.
 If 0 then do not save anything. (default: -1)
 --begin-epoch BEGIN_EPOCH
 start the model from an epoch (default: 0)
 --load LOAD checkpoint to load (default: None)
 --test-io test reading speed without training (default: False)
 --test-io-mode {train,val}
 data to test (default: train)
 --log LOG file where to save the log from the experiment
 (default: log.log)
 --dllogger-log DLLOGGER_LOG
 file where to save the dllogger log from the
 experiment (default: dllogger_log.log)
 --workspace WORKSPACE
 path to directory where results will be stored
 (default: ./)
 --no-metrics do not calculate evaluation metrics (for benchmarking)
 (default: False)
 --benchmark-iters BENCHMARK_ITERS
 run only benchmark-iters iterations from each epoch
 (default: None)

Data:
 --data-train DATA_TRAIN
 the training data (default: None)
 --data-train-idx DATA_TRAIN_IDX
 the index of training data (default: )
 --data-val DATA_VAL the validation data (default: None)
 --data-val-idx DATA_VAL_IDX
 the index of validation data (default: )
 --data-pred DATA_PRED
 the image on which run inference (only for pred mode)
 (default: None)
 --data-backend {dali-gpu,dali-cpu,mxnet,synthetic}
 set data loading & augmentation backend (default:
 dali-gpu)
 --image-shape IMAGE_SHAPE
 the image shape feed into the network (default: [3,
 224, 224])
 --rgb-mean RGB_MEAN a tuple of size 3 for the mean rgb (default: [123.68,
 116.779, 103.939])
 --rgb-std RGB_STD a tuple of size 3 for the std rgb (default: [58.393,
 57.12, 57.375])
 --input-layout {NCHW,NHWC}
 the layout of the input data (default: NCHW)
 --conv-layout {NCHW,NHWC}
 the layout of the data assumed by the conv operation
 (default: NCHW)
 --batchnorm-layout {NCHW,NHWC}
 the layout of the data assumed by the batchnorm
 operation (default: NCHW)
 --pooling-layout {NCHW,NHWC}
 the layout of the data assumed by the pooling
 operation (default: NCHW)
 --num-examples NUM_EXAMPLES
 the number of training examples (doesn't work with
 mxnet data backend) (default: 1281167)
 --data-val-resize DATA_VAL_RESIZE
 base length of shorter edge for validation dataset
 (default: 256)

DALI data backend:
 entire group applies only to dali data backend

 --dali-separ-val each process will perform independent validation on
 whole val-set (default: False)
 --dali-threads DALI_THREADS
 number of threadsper GPU for DALI (default: 3)
 --dali-validation-threads DALI_VALIDATION_THREADS
 number of threadsper GPU for DALI for validation
 (default: 10)
 --dali-prefetch-queue DALI_PREFETCH_QUEUE
 DALI prefetch queue depth (default: 2)
 --dali-nvjpeg-memory-padding DALI_NVJPEG_MEMORY_PADDING
 Memory padding value for nvJPEG (in MB) (default: 64)
 --dali-fuse-decoder DALI_FUSE_DECODER
 0 or 1 whether to fuse decoder or not (default: 1)

MXNet data backend:
 entire group applies only to mxnet data backend

 --data-mxnet-threads DATA_MXNET_THREADS
 number of threads for data decoding for mxnet data
 backend (default: 40)
 --random-crop RANDOM_CROP
 if or not randomly crop the image (default: 0)
 --random-mirror RANDOM_MIRROR
 if or not randomly flip horizontally (default: 1)
 --max-random-h MAX_RANDOM_H
 max change of hue, whose range is [0, 180] (default:
 0)
 --max-random-s MAX_RANDOM_S
 max change of saturation, whose range is [0, 255]
 (default: 0)
 --max-random-l MAX_RANDOM_L
 max change of intensity, whose range is [0, 255]
 (default: 0)
 --min-random-aspect-ratio MIN_RANDOM_ASPECT_RATIO
 min value of aspect ratio, whose value is either None
 or a positive value. (default: 0.75)
 --max-random-aspect-ratio MAX_RANDOM_ASPECT_RATIO
 max value of aspect ratio. If min_random_aspect_ratio
 is None, the aspect ratio range is
 [1-max_random_aspect_ratio,
 1+max_random_aspect_ratio], otherwise it is
 [min_random_aspect_ratio, max_random_aspect_ratio].
 (default: 1.33)
 --max-random-rotate-angle MAX_RANDOM_ROTATE_ANGLE
 max angle to rotate, whose range is [0, 360] (default:
 0)
 --max-random-shear-ratio MAX_RANDOM_SHEAR_RATIO
 max ratio to shear, whose range is [0, 1] (default: 0)
 --max-random-scale MAX_RANDOM_SCALE
 max ratio to scale (default: 1)
 --min-random-scale MIN_RANDOM_SCALE
 min ratio to scale, should >= img_size/input_shape.
 otherwise use --pad-size (default: 1)
 --max-random-area MAX_RANDOM_AREA
 max area to crop in random resized crop, whose range
 is [0, 1] (default: 1)
 --min-random-area MIN_RANDOM_AREA
 min area to crop in random resized crop, whose range
 is [0, 1] (default: 0.05)
 --min-crop-size MIN_CROP_SIZE
 Crop both width and height into a random size in
 [min_crop_size, max_crop_size] (default: -1)
 --max-crop-size MAX_CROP_SIZE
 Crop both width and height into a random size in
 [min_crop_size, max_crop_size] (default: -1)
 --brightness BRIGHTNESS
 brightness jittering, whose range is [0, 1] (default:
 0)
 --contrast CONTRAST contrast jittering, whose range is [0, 1] (default: 0)
 --saturation SATURATION
 saturation jittering, whose range is [0, 1] (default:
 0)
 --pca-noise PCA_NOISE
 pca noise, whose range is [0, 1] (default: 0)
 --random-resized-crop RANDOM_RESIZED_CROP
 whether to use random resized crop (default: 1)

Command-line options

To see the full list of available options and their descriptions, use the -h or --help command line option:

./runner --help and python train.py --help

./runner acts as a wrapper on train.py and all additional flags will be passed to train.py.

Getting the data

The MXNet ResNet-50 v1.5 script operates on ImageNet 1k, a widely popular image classification dataset from ILSVRC challenge. You can download the images from http://image-net.org/download-images.

The recommended data format is RecordIO, which concatenates multiple examples into seekable binary files for better read efficiency. MXNet provides a tool called im2rec.py located in the /opt/mxnet/tools/ directory. The tool converts individual images into .rec files.

To prepare a RecordIO file containing ImageNet data, we first need to create .lst files which consist of the labels and image paths. We assume that the original images were downloaded to /data/imagenet/raw/train-jpeg and /data/imagenet/raw/val-jpeg.

python /opt/mxnet/tools/im2rec.py --list --recursive train /data/imagenet/raw/train-jpeg
python /opt/mxnet/tools/im2rec.py --list --recursive val /data/imagenet/raw/val-jpeg

Next, we generate the .rec (RecordIO files with data) and .idx (indexes required by DALI to speed up data loading) files. To obtain the best training accuracy we do not preprocess the images when creating the RecordIO file.

python /opt/mxnet/tools/im2rec.py --pass-through --num-thread 40 train /data/imagenet/raw/train-jpeg
python /opt/mxnet/tools/im2rec.py --pass-through --num-thread 40 val /data/imagenet/raw/val-jpeg

Dataset guidelines

The process of loading, normalizing, and augmenting the data contained in the dataset can be found in the data.py and dali.py files.

The data is read from RecordIO format, which concatenates multiple examples into seekable binary files for better read efficiency.

Data augmentation techniques are described in the Default configuration section.

Multi-dataset

In most cases, to train a model on a different dataset, no changes in the code are required, but the dataset has to be converted into RecordIO format.

To convert a custom dataset, follow the steps from Getting the data section, and refer to the scripts/prepare_dataset.py script.

Training process

To start training, run: ./runner -n -b --data-root --dtype

By default, the training script runs the validation after each epoch:

The best checkpoint will be stored in the model_best.params file in the working directory.
The log from training will be saved in the log.log file in the working directory.
The JSON report with statistics will be saved in the report.json file in the working directory.

If ImageNet is mounted in the /data/imagenet/train-val-recordio-passthrough directory, you don't have to specify the --data-root flag.

Inference process

To start validation, run: ./runner -n -b --data-root --dtype --mode val

By default:

The log from validation will be saved in the log.log file in the working directory.
The JSON report with statistics will be saved in the report.json file in the working directory.