Mask R-CNN for PyTorch

NVIDIA Deep Learning Examples

Resource

NVIDIA Deep Learning Examples

Mask R-CNN for PyTorch

Mask R-CNN is a convolution based network for object instance segmentation. This implementation provides 1.3x faster training while maintaining target accuracy.

Benchmarking

Benchmarking can be performed for both training and inference. Both scripts run the Mask R-CNN model using the parameters defined in configs/e2e_mask_rcnn_R_50_FPN_1x.yaml. You can specify whether benchmarking is performed in FP16, TF32 or FP32 by specifying it as an argument to the benchmarking scripts.

Training performance benchmark

Training benchmarking can performed by running the script:

scripts/train_benchmark.sh <float16/tf32/float32> <number of gpus> <NHWC True/False> <Hybrid dataloader True/False>

Inference performance benchmark

Inference benchmarking can be performed by running the script:

scripts/inference_benchmark.sh <float16/tf32/float32> <batch_size>

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training Accuracy Results

Training accuracy: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the scripts/train.sh training script in the 21.12-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs.

GPUs	Batch size / GPU	BBOX mAP - TF32	MASK mAP - TF32	BBOX mAP - FP16	MASK mAP - FP16	Time to train - TF32	Time to train - mixed precision	Time to train speedup (TF32 to mixed precision)
8	12	0.3765	0.3408	0.3763	0.3417	2.15	1.85	1.16x

Training accuracy: NVIDIA DGX-1 (8x V100 32GB)

Our results were obtained by running the scripts/train.sh training script in the PyTorch 21.12-py3 NGC container on NVIDIA DGX-1 with 8x V100 32GB GPUs.

GPUs	Batch size / GPU	BBOX mAP - FP32	MASK mAP - FP32	BBOX mAP - FP16	MASK mAP - FP16	Time to train - FP32	Time to train - mixed precision	Time to train speedup (FP32 to mixed precision)
8	12	0.3768	0.3415	0.3755	0.3403	5.58	3.37	1.65x

Note: Currently V100 32GB + FP32 + NHWC + Hybrid dataloader causes a slowdown. So for all V100 32GB FP32 runs hybrid dataloader and NHWC are disabled NHWC=False HYBRID=False DTYPE=float32 bash scripts/train.sh

Training loss curves

Loss Curve

Here, multihead loss is simply the summation of losses on the mask head and the bounding box head.

Training Stability Test

The following tables compare mAP scores across 5 different training runs with different seeds. The runs showcase consistent convergence on all 5 seeds with very little deviation.

Config	Seed 1	Seed 2	Seed 3	Seed 4	Seed 5	Mean	Standard Deviation
8 GPUs, final AP BBox	0.3764	0.3766	0.3767	0.3752	0.3768	0.3763	0.0006
8 GPUs, final AP Segm	0.3414	0.3411	0.341	0.3407	0.3415	0.3411	0.0003

Training Performance Results

Training performance: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the scripts/train_benchmark.sh training script in the 21.12-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Performance numbers in images per second were averaged over 500 iterations.

GPUs	Batch size / GPU	Throughput - TF32	Throughput - mixed precision	Throughput speedup (TF32 - mixed precision)	Weak scaling - TF32	Weak scaling - mixed precision

| 1 | 12 | 23 | 24 | 1.04 | 1 | 1 | | 4 | 12 | 104 | 106 | 1.02 | 4.52 | 4.42 | | 8 | 12 | 193 | 209 | 1.08 | 8.39 | 8.71 |

Training performance: NVIDIA DGX-1 (8x V100 32GB)

Our results were obtained by running the scripts/train_benchmark.sh training script in the 21.12-py3 NGC container on NVIDIA DGX-1 with (8x V100 32GB) GPUs. Performance numbers in images per second were averaged over 500 iterations.

GPUs	Batch size / GPU	Throughput - FP32	Throughput - mixed precision	Throughput speedup (FP32 - mixed precision)	Weak scaling - FP32	Weak scaling - mixed precision
1	12	12	16	1.33	1	1
4	12	44	71	1.61	3.67	4.44
8	12	85	135	1.59	7.08	8.44

Note: Currently V100 32GB + FP32 + NHWC + Hybrid dataloader causes a slowdown. So for all V100 32GB FP32 runs hybrid dataloader and NHWC are disabled bash scripts/train_benchmark.sh fp32 <number of gpus> False False To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80GB)

Our results were obtained by running the scripts/inference_benchmark.sh training script in the PyTorch 21.12-py3 NGC container on NVIDIA DGX A100 (1x A100 80GB) GPU.

FP16 Inference Latency

Batch size	Throughput Avg	Latency Avg (ms)	Latency 90% (ms)	Latency 95% (ms)	Latency 99% (ms)
1	23	34.91	33.87	33.95	34.15
2	26	59.31	57.80	57.99	58.27
4	31	101.46	99.24	99.51	99.86
8	31	197.57	193.82	194.28	194.77

TF32 Inference Latency

Batch size	Throughput Avg	Latency Avg (ms)	Latency 90% (ms)	Latency 95% (ms)	Latency 99% (ms)
1	25	31.66	31.03	31.13	31.26
2	28	56.91	55.88	56.05	56.02
4	29	104.11	102.29	102.53	102.74
8	30	201.13	197.43	197.84	198.19

Inference performance: NVIDIA DGX-1 (1x V100 32GB)

Our results were obtained by running the scripts/inference_benchmark.sh training script in the PyTorch 21.12-py3 NGC container on NVIDIA DGX-1 with 1x V100 32GB GPUs.

FP16 Inference Latency

Batch size	Throughput Avg	Latency Avg (ms)	Latency 90% (ms)	Latency 95% (ms)	Latency 99% (ms)
1	19	44.72	43.62	43.77	44.03
2	21	82.80	81.37	81.67	82.06
4	22	155.25	153.15	153.63	154.10
8	22	307.60	304.08	304.82	305.48

FP32 Inference Latency

Batch size	Throughput Avg	Latency Avg (ms)	Latency 90% (ms)	Latency 95% (ms)	Latency 99% (ms)
1	16	52.78	51.87	52.16	52.43
2	17	100.81	99.19	99.67	100.15
4	17	202.05	198.84	199.98	200.92
8	18	389.99	384.29	385.77	387.66

To achieve these same results, follow the steps in the Quick Start Guide.