ResNet50 v1.5 for PaddlePaddle

NVIDIA Deep Learning Examples

Resource

NVIDIA Deep Learning Examples

ResNet50 v1.5 for PaddlePaddle

With modified architecture and initialization this ResNet50 version gives ~0.5% better accuracy than original.

The following sections provide greater details of the dataset, running training and inference, and the training results.

Scripts and sample code

To run a non standard configuration use:

# For single GPU evaluation
python -m paddle.distributed.launch --gpus=0 train.py

# For 8 GPUs evaluation
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py

Command-line options:

To find the full list of available options and their descriptions, use the -h or --help command-line option, for example: python [train.py|export_model.py|inference.py] -h

PaddlePaddle RN50v1.5 training script

optional arguments:
  -h, --help            show this help message and exit

Global:
  --output-dir OUTPUT_DIR
                        A path to store trained models. (default: ./output/)
  --run-scope {train_eval,train_only,eval_only}
                        Running scope. It should be one of {train_eval, train_only, eval_only}. (default: train_eval)
  --epochs EPOCHS       The number of epochs for training. (default: 90)
  --save-interval SAVE_INTERVAL
                        The iteration interval to save checkpoints. (default: 1)
  --eval-interval EVAL_INTERVAL
                        The iteration interval to test trained models on a given validation dataset. Ignored when --run-scope is
                        train_only. (default: 1)
  --print-interval PRINT_INTERVAL
                        The iteration interval to show training/evaluation message. (default: 10)
  --report-file REPORT_FILE
                        A file in which to store JSON experiment report. (default: ./report.json)
  --data-layout {NCHW,NHWC}
                        Data format. It should be one of {NCHW, NHWC}. (default: NCHW)
  --benchmark           To enable benchmark mode. (default: False)
  --benchmark-steps BENCHMARK_STEPS
                        Steps for benchmark run, only be applied when --benchmark is set. (default: 100)
  --benchmark-warmup-steps BENCHMARK_WARMUP_STEPS
                        Warmup steps for benchmark run, only be applied when --benchmark is set. (default: 100)
  --model-prefix MODEL_PREFIX
                        The prefix name of model files to save/load. (default: resnet_50_paddle)
  --from-pretrained-params FROM_PRETRAINED_PARAMS
                        A folder path which contains pretrained parameters, that is a file in name --model-prefix + .pdparams. It should
                        not be set with --from-checkpoint at the same time. (default: None)
  --from-checkpoint FROM_CHECKPOINT
                        A checkpoint path to resume training. It should not be set with --from-pretrained-params at the same time. The
                        path provided could be a folder contains < epoch_id/ckpt_files > or < ckpt_files >. (default: None)
  --last-epoch-of-checkpoint LAST_EPOCH_OF_CHECKPOINT
                        The epoch id of the checkpoint given by --from-checkpoint. It should be None, auto or integer >= 0. If it is set
                        as None, then training will start from 0-th epoch. If it is set as auto, then it will search largest integer-
                        convertable folder --from-checkpoint, which contains required checkpoint. Default is None. (default: None)
  --show-config SHOW_CONFIG
                        To show arguments. (default: True)
  --enable-cpu-affinity ENABLE_CPU_AFFINITY
                        To enable in-built GPU-CPU affinity. (default: True)

Dataset:
  --image-root IMAGE_ROOT
                        A root folder of train/val images. It should contain train and val folders, which store corresponding images.
                        (default: /imagenet)
  --image-shape IMAGE_SHAPE
                        The image shape. Its shape should be [channel, height, width]. (default: [4, 224, 224])
  --batch-size BATCH_SIZE
                        The batch size for both training and evaluation. (default: 256)
  --dali-random-seed DALI_RANDOM_SEED
                        The random seed for DALI data loader. (default: 42)
  --dali-num-threads DALI_NUM_THREADS
                        The number of threads applied to DALI data loader. (default: 4)
  --dali-output-fp16    Output FP16 data from DALI data loader. (default: False)

Data Augmentation:
  --crop-size CROP_SIZE
                        The size to crop input images. (default: 224)
  --rand-crop-scale RAND_CROP_SCALE
                        Range from which to choose a random area fraction. (default: [0.08, 1.0])
  --rand-crop-ratio RAND_CROP_RATIO
                        Range from which to choose a random aspect ratio (width/height). (default: [0.75, 1.3333333333333333])
  --normalize-scale NORMALIZE_SCALE
                        A scalar to normalize images. (default: 0.00392156862745098)
  --normalize-mean NORMALIZE_MEAN
                        The mean values to normalize RGB images. (default: [0.485, 0.456, 0.406])
  --normalize-std NORMALIZE_STD
                        The std values to normalize RGB images. (default: [0.229, 0.224, 0.225])
  --resize-short RESIZE_SHORT
                        The length of the shorter dimension of the resized image. (default: 256)

Model:
  --model-arch-name MODEL_ARCH_NAME
                        The model architecture name. It should be one of {ResNet50}. (default: ResNet50)
  --num-of-class NUM_OF_CLASS
                        The number classes of images. (default: 1000)
  --bn-weight-decay     Apply weight decay to BatchNorm shift and scale. (default: False)

Training:
  --label-smoothing LABEL_SMOOTHING
                        The ratio of label smoothing. (default: 0.1)
  --optimizer OPTIMIZER
                        The name of optimizer. It should be one of {Momentum}. (default: Momentum)
  --momentum MOMENTUM   The momentum value of optimizer. (default: 0.875)
  --weight-decay WEIGHT_DECAY
                        The coefficient of weight decay. (default: 3.0517578125e-05)
  --lr-scheduler LR_SCHEDULER
                        The name of learning rate scheduler. It should be one of {Cosine}. (default: Cosine)
  --lr LR               The initial learning rate. (default: 0.256)
  --warmup-epochs WARMUP_EPOCHS
                        The number of epochs for learning rate warmup. (default: 5)
  --warmup-start-lr WARMUP_START_LR
                        The initial learning rate for warmup. (default: 0.0)

Advanced Training:
  --amp                 Enable automatic mixed precision training (AMP). (default: False)
  --scale-loss SCALE_LOSS
                        The loss scalar for AMP training, only be applied when --amp is set. (default: 1.0)
  --use-dynamic-loss-scaling
                        Enable dynamic loss scaling in AMP training, only be applied when --amp is set. (default: False)
  --use-pure-fp16       Enable pure FP16 training, only be applied when --amp is set. (default: False)
  --asp                 Enable automatic sparse training (ASP). (default: False)
  --prune-model         Prune model to 2:4 sparse pattern, only be applied when --asp is set. (default: False)
  --mask-algo {mask_1d,mask_2d_greedy,mask_2d_best}
                        The algorithm to generate sparse masks. It should be one of {mask_1d, mask_2d_greedy, mask_2d_best}. This only
                        be applied when --asp and --prune-model is set. (default: mask_1d)

Paddle-TRT:
  --trt-inference-dir TRT_INFERENCE_DIR
                        A path to store/load inference models. export_model.py would export models to this folder, then inference.py
                        would load from here. (default: ./inference)
  --trt-precision {FP32,FP16,INT8}
                        The precision of TensorRT. It should be one of {FP32, FP16, INT8}. (default: FP32)
  --trt-workspace-size TRT_WORKSPACE_SIZE
                        The memory workspace of TensorRT in MB. (default: 1073741824)
  --trt-min-subgraph-size TRT_MIN_SUBGRAPH_SIZE
                        The minimal subgraph size to enable PaddleTRT. (default: 3)
  --trt-use-static TRT_USE_STATIC
                        Fix TensorRT engine at first running. (default: False)
  --trt-use-calib-mode TRT_USE_CALIB_MODE
                        Use the PTQ calibration of PaddleTRT int8. (default: False)
  --trt-export-log-path TRT_EXPORT_LOG_PATH
                        A file in which to store JSON model exporting report. (default: ./export.json)
  --trt-log-path TRT_LOG_PATH
                        A file in which to store JSON inference report. (default: ./inference.json)
  --trt-use-synthat TRT_USE_SYNTHAT
                        Apply synthetic data for benchmark. (default: False)

Noted that arguments in Paddle-TRT are only available to export_model.py or inference.py.

Dataset guidelines

To use your own dataset, divide it in directories as in the following scheme:

Training images - train/<class id>/<image>
Validation images - val/<class id>/<image>

If the number of classes in your dataset is not 1000, you need to specify it to --num-of-class.

Training process

The model will be stored in the directory specified with --output-dir and --model-arch-name, including three files:

.pdparams: The parameters contain all the trainable tensors and will save to a file with the suffix ".pdparams".
.pdopts: The optimizer information contains all the Tensors used by the optimizer. For Adam optimizer, it contains beta1, beta2, momentum, and so on. All the information will be saved to a file with suffix ".pdopt". (If the optimizer has no Tensor need to save (like SGD), the file will not be generated).
.pdmodel: The network description is the description of the program. It's only used for deployment. The description will save to a file with the suffix ".pdmodel".

The prefix of model files is specified by --model-prefix, which default value is resnet_50_paddle. Model of each epoch would be stored in directory ./output/ResNet50/epoch_id/ with three files by default, including resnet_50_paddle.pdparams, resnet_50_paddle.pdopts, resnet_50_paddle.pdmodel. Note that epoch_id is 0-based, which means epoch_id is from 0 to 89 for a total of 90 epochs. For example, the model of the 89th epoch would be stored in ./output/ResNet50/89/resnet_50_paddle

Assume you want to train the ResNet50 for 90 epochs, but the training process aborts during the 50th epoch due to infrastructure faults. To resume training from the checkpoint, specify --from-checkpoint and --last-epoch-of-checkpoint with following these steps:

Set ./output/ResNet50/49 to --from-checkpoint.
Set --last-epoch-of-checkpoint to 49. Then rerun the training to resume training from the 50th epoch to the 89th epoch.

Example:

# Resume AMP training from checkpoint of 50-th epoch
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --epochs 90 \
  --amp \
  --scale-loss 128.0 \
  --use-dynamic-loss-scaling \
  --data-layout NHWC \
  --model-prefix resnet_50_paddle \
  --from-checkpoint ./output/ResNet50/49 \
  --last-epoch-of-checkpoint 49

We also provide automatic searching for the checkpoint from last epoch. You can enable this by set --last-epoch-of-checkpoint as auto. Noted that if enable automatic searching, --from-checkpoint should be a folder contains chekcpoint files or <epoch_id>/<ckpt_files>. In previous example, it should be ./output/ResNet50.

Example:

# Resume AMP training from checkpoint with automatic searching
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --epochs 90 \
  --amp \
  --scale-loss 128.0 \
  --use-dynamic-loss-scaling \
  --data-layout NHWC \
  --model-prefix resnet_50_paddle \
  --from-checkpoint ./output/ResNet50 \
  --last-epoch-of-checkpoint auto

To start training from pretrained weights, set --from-pretrained-params to ./output/ResNet50/<epoch_id>/<--model-prefix>.

Example:

# Train AMP with model initialization by <./your_own_path_to/resnet_50_paddle>
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --epochs 90 \
  --amp \
  --scale-loss 128.0 \
  --use-dynamic-loss-scaling \
  --data-layout NHWC \
  --model-prefix resnet_50_paddle \
  --from-pretrained-params ./output/ResNet50/<epoch_id>

Make sure:

Resume from checkpoints: Both <--model-prefix>.pdopts and <--model-prefix>.pdparams must be in the given path.
Start from pretrained weights: <--model-prefix>.pdparams must be in the given path.
Don't set --from-checkpoint and --from-pretrained-params at the same time.

The difference between those two is that --from-pretrained-params contain only model weights, and --from-checkpoint, apart from model weights, contain the optimizer state, and LR scheduler state.

--from-checkpoint is suitable for dividing the training into parts, for example, in order to divide the training job into shorter stages, or restart training after infrastructure faults.

--from-pretrained-params can be used as a base for finetuning the model to a different dataset or as a backbone to detection models.

Metrics gathered through both training and evaluation:

[train|val].loss - loss
[train|val].top1 - top 1 accuracy
[train|val].top5 - top 5 accuracy
[train|val].data_time - time spent on waiting on data
[train|val].compute_time - time spent on computing
[train|val].batch_time - time spent on a mini-batch
[train|val].ips - speed measured in images per second

Metrics gathered through training only

train.lr - learning rate

Automatic SParsity training process:

To enable automatic sparsity training workflow, turn on --amp and --prune-mode when training launches. Refer to Command-line options

Note that automatic sparsity (ASP) requires a pretrained model to initialize parameters.

You can apply scripts/training/train_resnet50_AMP_ASP_90E_DGXA100.sh we provided to launch ASP + AMP training.

# Default path to pretrained parameters is ./output/ResNet50/89/resnet_50_paddle
bash scripts/training/train_resnet50_AMP_ASP_90E_DGXA100.sh <pretrained_parameters>

Or following steps below to manually launch ASP + AMP training.

First, set --from-pretrained-params to a pretrained model file. For example, if you have trained the ResNet50 for 90 epochs following Training process, the final pretrained weights would be stored in ./output/ResNet50/89/resnet_50_paddle.pdparams by default, and set --from-pretrained-params to ./output/ResNet50/89.

Then run following command to run AMP + ASP:

python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --from-pretrained-params ./output/ResNet50/89 \
  --model-prefix resnet_50_paddle \
  --epochs 90 \
  --amp \
  --scale-loss 128.0 \
  --use-dynamic-loss-scaling \
  --data-layout NHWC \
  --asp \
  --prune-model \
  --mask-algo mask_1d

Inference process

Inference on your own datasets.

To run inference on a single example with pretrained parameters,

Set --from-pretrained-params to your pretrained parameters.
Set --image-root to the root folder of your own dataset.

Note that validation dataset should be in image-root/val.

Set --run-scope to eval_only.

# For single GPU evaluation
python -m paddle.distributed.launch --gpus=0 train.py \
  --from-pretrained-params <path_to_pretrained_params> \
  --image-root <your_own_dataset> \
  --run-scope eval_only

# For 8 GPUs evaluation
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --from-pretrained-params <path_to_pretrained_params> \
  --image-root <your_own_dataset> \
  --run-scope eval_only

Inference with TensorRT

To run inference with TensorRT for the best performance, you can apply the scripts in scripts/inference.

For example,

Run bash scripts/inference/export_resnet50_AMP.sh <your_checkpoint> to export an inference model.

The default path of checkpoint is ./output/ResNet50/89.

Run bash scripts/inference/infer_resnet50_AMP.sh to infer with TensorRT.

Or you could manually run export_model.py and inference.py with specific arguments, refer to Command-line options.

Note that arguments passed to export_model.py and inference.py should be the same with arguments used in training.