With modified architecture and initialization this ResNet50 version gives ~0.5% better accuracy than original.
The following sections provide greater details of the dataset, running training and inference, and the training results.
Scripts and sample code
In the root directory, the most important files are:
runner: A wrapper on thetrain.pyscript which is the main executable script for training/validation/predicting.benchmark.py: A script for benchmarking.Dockerfile: Container to build the container.fit.py: A file containing most of the training and validation logic.data.py: Data loading and preprocessing code.dali.py: Data loading and preprocessing code using DALI.models.py: The model architecture.report.py: A file containing JSON report structure and description of fields.
In the scripts directory, the most important files are:
prepare_imagenet.sh: A script that converts raw dataset format to RecordIO format.
Parameters
The complete list of available parameters contains:
Model:
--arch {resnetv1,resnetv15,resnextv1,resnextv15,xception}
model architecture (default: resnetv15)
--num-layers NUM_LAYERS
number of layers in the neural network, required by
some networks such as resnet (default: 50)
--num-groups NUM_GROUPS
number of groups for grouped convolutions, required by
some networks such as resnext (default: 32)
--num-classes NUM_CLASSES
the number of classes (default: 1000)
--batchnorm-eps BATCHNORM_EPS
the amount added to the batchnorm variance to prevent
output explosion. (default: 1e-05)
--batchnorm-mom BATCHNORM_MOM
the leaky-integrator factor controling the batchnorm
mean and variance. (default: 0.9)
--fuse-bn-relu FUSE_BN_RELU
have batchnorm kernel perform activation relu
(default: 0)
--fuse-bn-add-relu FUSE_BN_ADD_RELU
have batchnorm kernel perform add followed by
activation relu (default: 0)
Training:
--mode {train_val,train,val,pred}
mode (default: train_val)
--seed SEED random seed (default: None)
--gpus GPUS list of gpus to run, e.g. 0 or 0,2,5 (default: [0])
--kv-store {device,horovod}
key-value store type (default: device)
--dtype {float32,float16}
precision (default: float16)
--amp If enabled, turn on AMP (Automatic Mixed Precision)
(default: False)
--batch-size BATCH_SIZE
the batch size (default: 192)
--num-epochs NUM_EPOCHS
number of epochs (default: 90)
--lr LR initial learning rate (default: 0.1)
--lr-schedule {multistep,cosine}
learning rate schedule (default: cosine)
--lr-factor LR_FACTOR
the ratio to reduce lr on each step (default: 0.256)
--lr-steps LR_STEPS the epochs to reduce the lr, e.g. 30,60 (default: [])
--warmup-epochs WARMUP_EPOCHS
the epochs to ramp-up lr to scaled large-batch value
(default: 5)
--optimizer OPTIMIZER
the optimizer type (default: sgd)
--mom MOM momentum for sgd (default: 0.875)
--wd WD weight decay for sgd (default: 3.0517578125e-05)
--label-smoothing LABEL_SMOOTHING
label smoothing factor (default: 0.1)
--mixup MIXUP alpha parameter for mixup (if 0 then mixup is not
applied) (default: 0)
--disp-batches DISP_BATCHES
show progress for every n batches (default: 20)
--model-prefix MODEL_PREFIX
model checkpoint prefix (default: model)
--save-frequency SAVE_FREQUENCY
frequency of saving model in epochs (--model-prefix
must be specified). If -1 then save only best model.
If 0 then do not save anything. (default: -1)
--begin-epoch BEGIN_EPOCH
start the model from an epoch (default: 0)
--load LOAD checkpoint to load (default: None)
--test-io test reading speed without training (default: False)
--test-io-mode {train,val}
data to test (default: train)
--log LOG file where to save the log from the experiment
(default: log.log)
--dllogger-log DLLOGGER_LOG
file where to save the dllogger log from the
experiment (default: dllogger_log.log)
--workspace WORKSPACE
path to directory where results will be stored
(default: ./)
--no-metrics do not calculate evaluation metrics (for benchmarking)
(default: False)
--benchmark-iters BENCHMARK_ITERS
run only benchmark-iters iterations from each epoch
(default: None)
Data:
--data-train DATA_TRAIN
the training data (default: None)
--data-train-idx DATA_TRAIN_IDX
the index of training data (default: )
--data-val DATA_VAL the validation data (default: None)
--data-val-idx DATA_VAL_IDX
the index of validation data (default: )
--data-pred DATA_PRED
the image on which run inference (only for pred mode)
(default: None)
--data-backend {dali-gpu,dali-cpu,mxnet,synthetic}
set data loading & augmentation backend (default:
dali-gpu)
--image-shape IMAGE_SHAPE
the image shape feed into the network (default: [3,
224, 224])
--rgb-mean RGB_MEAN a tuple of size 3 for the mean rgb (default: [123.68,
116.779, 103.939])
--rgb-std RGB_STD a tuple of size 3 for the std rgb (default: [58.393,
57.12, 57.375])
--input-layout {NCHW,NHWC}
the layout of the input data (default: NCHW)
--conv-layout {NCHW,NHWC}
the layout of the data assumed by the conv operation
(default: NCHW)
--batchnorm-layout {NCHW,NHWC}
the layout of the data assumed by the batchnorm
operation (default: NCHW)
--pooling-layout {NCHW,NHWC}
the layout of the data assumed by the pooling
operation (default: NCHW)
--num-examples NUM_EXAMPLES
the number of training examples (doesn't work with
mxnet data backend) (default: 1281167)
--data-val-resize DATA_VAL_RESIZE
base length of shorter edge for validation dataset
(default: 256)
DALI data backend:
entire group applies only to dali data backend
--dali-separ-val each process will perform independent validation on
whole val-set (default: False)
--dali-threads DALI_THREADS
number of threadsper GPU for DALI (default: 3)
--dali-validation-threads DALI_VALIDATION_THREADS
number of threadsper GPU for DALI for validation
(default: 10)
--dali-prefetch-queue DALI_PREFETCH_QUEUE
DALI prefetch queue depth (default: 2)
--dali-nvjpeg-memory-padding DALI_NVJPEG_MEMORY_PADDING
Memory padding value for nvJPEG (in MB) (default: 64)
--dali-fuse-decoder DALI_FUSE_DECODER
0 or 1 whether to fuse decoder or not (default: 1)
MXNet data backend:
entire group applies only to mxnet data backend
--data-mxnet-threads DATA_MXNET_THREADS
number of threads for data decoding for mxnet data
backend (default: 40)
--random-crop RANDOM_CROP
if or not randomly crop the image (default: 0)
--random-mirror RANDOM_MIRROR
if or not randomly flip horizontally (default: 1)
--max-random-h MAX_RANDOM_H
max change of hue, whose range is [0, 180] (default:
0)
--max-random-s MAX_RANDOM_S
max change of saturation, whose range is [0, 255]
(default: 0)
--max-random-l MAX_RANDOM_L
max change of intensity, whose range is [0, 255]
(default: 0)
--min-random-aspect-ratio MIN_RANDOM_ASPECT_RATIO
min value of aspect ratio, whose value is either None
or a positive value. (default: 0.75)
--max-random-aspect-ratio MAX_RANDOM_ASPECT_RATIO
max value of aspect ratio. If min_random_aspect_ratio
is None, the aspect ratio range is
[1-max_random_aspect_ratio,
1+max_random_aspect_ratio], otherwise it is
[min_random_aspect_ratio, max_random_aspect_ratio].
(default: 1.33)
--max-random-rotate-angle MAX_RANDOM_ROTATE_ANGLE
max angle to rotate, whose range is [0, 360] (default:
0)
--max-random-shear-ratio MAX_RANDOM_SHEAR_RATIO
max ratio to shear, whose range is [0, 1] (default: 0)
--max-random-scale MAX_RANDOM_SCALE
max ratio to scale (default: 1)
--min-random-scale MIN_RANDOM_SCALE
min ratio to scale, should >= img_size/input_shape.
otherwise use --pad-size (default: 1)
--max-random-area MAX_RANDOM_AREA
max area to crop in random resized crop, whose range
is [0, 1] (default: 1)
--min-random-area MIN_RANDOM_AREA
min area to crop in random resized crop, whose range
is [0, 1] (default: 0.05)
--min-crop-size MIN_CROP_SIZE
Crop both width and height into a random size in
[min_crop_size, max_crop_size] (default: -1)
--max-crop-size MAX_CROP_SIZE
Crop both width and height into a random size in
[min_crop_size, max_crop_size] (default: -1)
--brightness BRIGHTNESS
brightness jittering, whose range is [0, 1] (default:
0)
--contrast CONTRAST contrast jittering, whose range is [0, 1] (default: 0)
--saturation SATURATION
saturation jittering, whose range is [0, 1] (default:
0)
--pca-noise PCA_NOISE
pca noise, whose range is [0, 1] (default: 0)
--random-resized-crop RANDOM_RESIZED_CROP
whether to use random resized crop (default: 1)
Command-line options
To see the full list of available options and their descriptions, use the -h or --help command line option:
./runner --help and python train.py --help
./runner acts as a wrapper on train.py and all additional flags will be passed to train.py.
Getting the data
The MXNet ResNet-50 v1.5 script operates on ImageNet 1k, a widely popular image classification dataset from ILSVRC challenge. You can download the images from http://image-net.org/download-images.
The recommended data format is
RecordIO, which
concatenates multiple examples into seekable binary files for better read
efficiency. MXNet provides a tool called im2rec.py located in the /opt/mxnet/tools/ directory.
The tool converts individual images into .rec files.
To prepare a RecordIO file containing ImageNet data, we first need to create .lst files
which consist of the labels and image paths. We assume that the original images were
downloaded to /data/imagenet/raw/train-jpeg and /data/imagenet/raw/val-jpeg.
python /opt/mxnet/tools/im2rec.py --list --recursive train /data/imagenet/raw/train-jpeg
python /opt/mxnet/tools/im2rec.py --list --recursive val /data/imagenet/raw/val-jpeg
Next, we generate the .rec (RecordIO files with data) and .idx (indexes required by DALI
to speed up data loading) files. To obtain the best training accuracy we do not preprocess the images when creating the RecordIO file.
python /opt/mxnet/tools/im2rec.py --pass-through --num-thread 40 train /data/imagenet/raw/train-jpeg
python /opt/mxnet/tools/im2rec.py --pass-through --num-thread 40 val /data/imagenet/raw/val-jpeg
Dataset guidelines
The process of loading, normalizing, and augmenting the data contained in the dataset can be found in the data.py and dali.py files.
The data is read from RecordIO format, which concatenates multiple examples into seekable binary files for better read efficiency.
Data augmentation techniques are described in the Default configuration section.
Multi-dataset
In most cases, to train a model on a different dataset, no changes in the code are required, but the dataset has to be converted into RecordIO format.
To convert a custom dataset, follow the steps from Getting the data section, and refer to the scripts/prepare_dataset.py script.
Training process
To start training, run:
./runner -n <number of gpus> -b <batch size per GPU> --data-root <path to imagenet> --dtype <float32 or float16>
By default, the training script runs the validation after each epoch:
- The best checkpoint will be stored in the
model_best.paramsfile in the working directory. - The log from training will be saved in the
log.logfile in the working directory. - The JSON report with statistics will be saved in the
report.jsonfile in the working directory.
If ImageNet is mounted in the /data/imagenet/train-val-recordio-passthrough directory, you don't have to specify the --data-root flag.
Inference process
To start validation, run:
./runner -n <number of gpus> -b <batch size per GPU> --data-root <path to imagenet> --dtype <float32 or float16> --mode val
By default:
- The log from validation will be saved in the
log.logfile in the working directory. - The JSON report with statistics will be saved in the
report.jsonfile in the working directory.