Models
LPDNet

LPDNet

Logo for LPDNet
Description
Object Detection network to detect license plates in an image of a car.
Publisher
NVIDIA
Latest Version
pruned_v2.2.1
Modified
November 12, 2024
Size
1.7 MB

License Plate Detection (LPDNet) Model Card

Description:

LPDNet detects license plates from an image. This model is ready for commercial use.

References:

Using TAO Pre-trained Models

Model Architecture:

Architecture Type: Convolution Neural Network (CNN)
Network Architecture: DetectNet_v2 ; YOLOv4-tiny

DetectNet_v2 models are based on NVIDIA DetectNet_v2 detector with ResNet18 as feature extractor. This architecture, also known as GridBox object detection, uses bounding-box regression on a uniform grid on the input image. Gridbox system divides an input image into a grid which predicts four normalized bounding-box parameters (xc, yc, w, h) and confidence value per output class.The raw normalized bounding-box and confidence detections needs to be post-processed by a clustering algorithm such as DBSCAN or NMS to produce final bounding-box coordinates and category labels.

YOLOv4-tiny models are based on YOLOv4-tiny detector with cspdarknet_tiny as feature extractor.

Input:

Input Type(s): Images
Input Format(s): Red, Green, Blue (RGB)
Input Parameters: 4D
Other Properties Related to Input:

  • For US license plate

    • Color Images of resolution 640 X 480 X 3 (W x H x C)
    • Channel Ordering of the Input: NCHW, where N = Batch Size, C = number of channels (3), H = Height of images (480), W = Width of the images (640)
    • Input scale: 1/255.0
    • Mean subtraction: None

  • For Chinese license plate: Color Images of resolution 720 X 1168 X 3

    • Color Images of resolution 720 X 1168 X 3 (W x H x C)
    • Channel Ordering of the Input: NCHW, where N = Batch Size, C = number of channels (3), H = Height of images (1168), W = Width of the images (720)
    • Input scale: 1/255.0
    • Mean subtraction: None

Output:

Output Type(s): Label(s), Bounding-Box(es), Confidence Scores
Output Format: Label: Text String(s); Bounding Box: (x-coordinate, y-coordinate, width, height), Confidence Scores: Floating Point
Other Properties Related to Output: Category Label(s): License Plate characters, Bounding Box Coordinates, Confidence Scores

Software Integration:

Runtime Engine(s):

  • TAO - 5.2
  • DeepStream 6.1 or later

Supported Hardware Architecture(s):

  • Ampere
  • Jetson
  • Hopper
  • Lovelace
  • Pascal
  • Turing
  • Volta

Supported Operating System(s):

  • Linux
  • Linux 4 Tegra

Model Version(s):

  • unpruned_v1.0 - ResNet18 based pre-trained model. Intended for training.
  • pruned_v1.0 - ResNet18 deployment models. Contains calibration cache for GPU and DLA. DLA one is required if running inference on Jetson AGX Xavier or Xavier NX DLA.
  • unpruned_v2.0 - CSPDarkNet-Tiny based YOLOv4-Tiny pre-trained model. Intended for training.
  • pruned_v1.0 - CSPDarkNet-Tiny based YOLOv4-Tiny deployment model. Contains calibration cache for INT8 deployment on GPU and DLA.

Training & Evaluation:

Training Dataset:

Data Collection Method by dataset:

  • Automatic/Sensors

Labeling Method by dataset:

  • Human

Properties:
LPRNet model has been trained on two datasets:

  • Internal, proprietary dataset of 45,000 US license plates images taken at various angles and illumination collected from dash and side cameras of a vehicle.
  • 172,000 images from the Chinese City Parking Dataset (CCPD) of a provincial capital of China.

Evaluation Dataset:

Data Collection Method by dataset:

  • Automatic/Sensors

Labeling Method by dataset:

Methodology and KPI

The key performance indicator is the accuracy of license plate detection.The KPI for the evaluation data are reported in the table below.

Version 1.0 models (DetectNet_v2)

Model Dataset Accuracy
usa_unpruned_model NVIDIA 3k LPD eval dataset 98.58%
usa_pruned_model NVIDIA 3k LPD eval dataset 98.46%
ccpd_unpruned_model 14% of CCPD-Base dataset 99.24%
ccpd_pruned_model 14% of CCPD-Base dataset 99.22%

Version 2.0 models (YoloV4-Tiny)

Model Dataset Accuracy
yolov4_tiny_usa_trainable_model NVIDIA 3k LPD eval dataset 99.53%
yolov4_tiny_usa_deployable_model NVIDIA 3k LPD eval dataset 99.61%
yolov4_tiny_ccpd_trainable_model 14% of CCPD-Base dataset 99.99%
yolov4_tiny_ccpd_deployable_model 14% of CCPD-Base dataset 99.98%

Version 2.0 models based on YoloV4-Tiny provides accuracy improvement over the version 1.0 models

Inference:

Engine: Tensor(RT)
Test Hardware:

  • Jetson AGX Xavier
  • Xavier NX
  • Orin
  • Orin NX
  • NVIDIA T4
  • Ampere GPU
  • A2
  • A30
  • L4
  • T4
  • DGX H100
  • DGX A100
  • DGX H100
  • L40
  • JAO 64GB
  • Orin NX16GB
  • Orin Nano 8GB

The inference is run on the provided pruned(deployable) models at INT8 precision. On the Jetson Nano FP16 precision is used. The inference performance runs with trtexec on Jetson Nano, Jetson TX2, AGX Xavier, Xavier NX and NVIDIA T4 GPU. The Jetson devices run at Max-N configuration for maximum system performance. The performance shown below is only for inference of the usa pruned(deployable) model. The end-to-end performance with streaming video data might slightly vary depending on use cases of applications.

Version 2.0 (YoloV4-tiny) models provide higher accuracy for different use cases, but comes at a cost of inference performance. The inference performance for v2.0 model is lower than v1.0 model. For version 1.0 (DetectNet_v2) based models,

Device Precision Batch_size FPS
Nano FP16 1 66
TX2 INT8 1 187
NX INT8 1 461
Xavier INT8 1 913
T4 INT8 1 2748

For Version 2.0 (YOLOv4-tiny) based models,

Device Precision Batch_size FPS
Nano FP16 1 40
TX2 INT8 1 105
NX INT8 1 225
Xavier INT8 1 489
T4 INT8 1 1264

How to use Version 1.0 (Detectnet_v2) based model

These models need to be used with NVIDIA Hardware and Software. For Hardware, the models can run on any NVIDIA GPU including NVIDIA Jetson devices. These models can only be used with Train Adapt Optimize (TAO) Toolkit, DeepStream SDK or TensorRT.

For Version 1.0 (Detectnet_v2) based model, there are four models provided:

  • usa_unpruned.tlt
  • usa_pruned.etlt
  • ccpd_unpruned.tlt
  • ccpd_pruned.etlt

The unpruned models are intended for training and fine-tuning using TAO Toolkit along with the user's dataset of license plates in United States of America or China. High fidelity models can be trained and adapted to the use case. The DetectNet_v2 Jupyter notebook available as a part of TAO CV resources can be used to re-train.

The usa pruned models are intended for easy deployment to the edge using DeepStream SDK or TensorRT. They accept 640x480x3 dimension input tensors and outputs 40x30x12 bbox coordinate tensor and 40x30x3 class confidence tensor.

The ccpd pruned models are intended for easy deployment to the edge using DeepStream SDK or TensorRT. These models accept 720x1168x3 dimension input tensors and outputs 45x73x12 bbox coordinate tensor and 45x73x3 class confidence tensor.

DeepStream provides a toolkit to create efficient video analytics pipelines to capture, decode, and pre-process the data before running inference. DeepStream will then post-process the output bbox coordinate tensor and class confidence tensors with NMS or DBScan clustering algorithm to create appropriate bounding boxes. The sample application and config file to run these models are provided in DeepStream SDK.

The unpruned and pruned models are encrypted and can be decrypted with the following key:

  • Model load key: nvidia_tlt

Please make sure to use this as the key for all TAO commands that require a model load key.

Instructions to use version 1.0 unpruned model with TAO

In order to use these models as a pretrained weights for transfer learning, please use the snippet below as template for the model_config component of the experiment spec file to train a DetectNet_v2 model. For more information on the experiment spec file, please refer to the TAO Toolkit DetectNet_v2 Guide.

  1. For ResNet18
model_config {
  num_layers: 18
  pretrained_model_file: "/path/to/the/model.tlt"
  use_batch_norm: true
  objective_set {
    bbox {
      scale: 35.0
      offset: 0.5
    }
    cov {
    }
  }
  training_precision {
    backend_floatx: FLOAT32
  }
  arch: "resnet"
}

Instructions to deploy version 1.0 models with DeepStream

To create the entire end-to-end video analytics application, deploy these models with DeepStream SDK. DeepStream SDK is a streaming analytics toolkit to accelerate building AI-based video analytics applications. DeepStream supports direct integration of these models into the deepstream sample app.

To deploy these models with DeepStream 6.0, please follow the instructions below:

Download and install DeepStream SDK. The installation instructions for DeepStream are provided in DeepStream development guide. The config files for the purpose-built models are located in:

/opt/nvidia/deepstream/deepstream-6.0/samples/configs/tao_pretrained_models

/opt/nvidia/deepstream is the default DeepStream installation directory. This path will be different if you are installing in a different directory.

You need to create 1 label file and 2 config files.

labels_lpdnet.txt - Label file with 1 class
deepstream_app_source1_trafficcamnet_lpdnet.txt - Main config file for DeepStream app
config_infer_secondary_lpdnet.txt - File to configure inference settings

Create label file labels_lpdnet.txt

echo lpd > labels_lpdnet.txt

Create config file deepstream_app_source1_trafficcamnet_lpdnet.txt

cp deepstream_app_source1_trafficcamnet.txt deepstream_app_source1_trafficcamnet_lpdnet.txt

Modify config file deepstream_app_source1_trafficcamnet_lpdnet.txt. Add below lines in it.

[secondary-gie0]
enable=1
model-engine-file=usa_pruned.etlt_b4_gpu0_int8.engine
gpu-id=0
batch-size=4
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_lpdnet.txt

Create config file config_infer_secondary_lpdnet.txt

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
labelfile-path=<path to labels_lpdnet.txt>
tlt-encoded-model=<path to etao_model>
tlt-model-key=nvidia_tlt
int8-calib-file=<path to calibration cache>
uff-input-dims=3;480;640;0  #For us model, set to 3;480;640;0  For ccpd model, set to 3;1168;720;0
uff-input-blob-name=input_1
batch-size=16
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=1
##1 Primary 2 Secondary
process-mode=2
interval=0
gie-unique-id=2
#0 detector 1 classifier 2 segmentatio 3 instance segmentation
network-type=0
operate-on-gie-id=1
operate-on-class-ids=0
cluster-mode=3
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
input-object-min-height=30
input-object-min-width=40
#enable-dla=1

[class-attrs-all]
pre-cluster-threshold=0.3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

Run deepstream-app:

deepstream-app -c deepstream_app_source1_trafficcamnet_lpdnet.txt

Documentation to deploy with DeepStream is provided in "Deploying to DeepStream" chapter of TAO User Guide.

How to use Version 2.0 (YOLOv4-tiny) based model

These models need to be used with NVIDIA Hardware and Software. For Hardware, the models can run on any NVIDIA GPU including NVIDIA Jetson devices. These models can only be used with Train Adapt Optimize (TAO) Toolkit, DeepStream SDK or TensorRT.

Primary use case intended for these models is detecting license plates in a color (RGB) image. The model can be used to detect license plates from photos and videos by using appropriate video or image decoding and pre-processing.

For YOLOv4-tiny based model, totally there are four models provided:

  • yolov4_tiny_usa_trainable.tlt
  • yolov4_tiny_usa_deployable.etlt
  • yolov4_tiny_ccpd_trainable.tlt
  • yolov4_tiny_ccpd_deployable.etlt

The unpruned models are intended for training and fine-tuning using TAO Toolkit along with the user's dataset of license plates in United States of America or China. High fidelity models can be trained and adapted to the use case. The YoloV4-Tiny Jupyter notebook available as a part of TAO CV resources can be used to re-train.

The usa deployable models are intended for easy deployment to the edge using DeepStream SDK or TensorRT.

The trainable and deployable models are encrypted and can be decrypted with the following key:

  • Model load key: nvidia_tlt

Please make sure to use this as the key for all TAO commands that require a model load key.

Instructions to use version 2.0 trainable model with TAO

In order to use these models as a pretrained weights for transfer learning, please use the snippet below as template for the training_config component of the experiment spec file to train a YOLOv4-tiny model. For more information on the experiment spec file, please refer to the TAO Toolkit YoloV4 Tiny Guide.

  1. Set pretrain_model_path in training_config section
training_config {
  pretrain_model_path: "/path/to/the/yolov4_tiny_usa_trainable.tlt"
  ...
}

Instructions to deploy version 2.0 models with DeepStream

To create the entire end-to-end video analytics application, deploy these models with DeepStream SDK. DeepStream SDK is a streaming analytics toolkit to accelerate building AI-based video analytics applications. DeepStream supports direct integration of these models into the deepstream sample app.

To deploy these models with DeepStream 6.0, please follow the instructions below:

Download and install DeepStream SDK. The installation instructions for DeepStream are provided in DeepStream development guide. The config files for the purpose-built models are located in:

/opt/nvidia/deepstream/deepstream-6.0/samples/configs/tao_pretrained_models

/opt/nvidia/deepstream is the default DeepStream installation directory. This path will be different if you are installing in a different directory.

You need to create 1 label file and 2 config files.

labels_lpdnet.txt - Label file with 1 class
deepstream_app_source1_trafficcamnet_lpdnet.txt - Main config file for DeepStream app
config_infer_secondary_lpdnet.txt - File to configure inference settings

Create label file labels_lpdnet.txt

echo lpd > labels_lpdnet.txt

Create config file deepstream_app_source1_trafficcamnet_lpdnet.txt

cp deepstream_app_source1_trafficcamnet.txt deepstream_app_source1_trafficcamnet_lpdnet.txt

Modify config file deepstream_app_source1_trafficcamnet_lpdnet.txt. Add below lines in it.

[secondary-gie0]
enable=1
model-engine-file=yolov4_tiny_usa_deployable.etlt_b4_gpu0_int8.engine
gpu-id=0
batch-size=4
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_lpdnet.txt

Create config file config_infer_secondary_lpdnet.txt

[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=<path to labels_lpdnet.txt>
tlt-encoded-model=<path to deployable_model>
tlt-model-key=nvidia_tlt
int8-calib-file=<path to calibration cache>
uff-input-dims=3;480;640;0  #For ccpd models, please set to 3;1168;720;0
uff-input-blob-name=Input
batch-size=16
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=1
##1 Primary 2 Secondary
process-mode=2
interval=0
gie-unique-id=2
#0 detector 1 classifier 2 segmentatio 3 instance segmentation
network-type=0
operate-on-gie-id=1
operate-on-class-ids=0
cluster-mode=3
output-blob-names=BatchedNMS
parse-bbox-func-name=NvDsInferParseCustomBatchedNMSTLT
custom-lib-path=deepstream_tao_apps/post_processor/libnvds_infercustomparser_tao.so
#enable-dla=1

[class-attrs-all]
pre-cluster-threshold=0.3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

Build libnvds_infercustomparser_tao.so

git clone -b release/tao3.0  https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps.git
cd deepstream_tao_apps/post_processor/
export CUDA_VER=xy.z                  // xy.z is CUDA version, e.g. 11.4
make
cd -

Run deepstream-app:

deepstream-app -c deepstream_app_source1_trafficcamnet_lpdnet.txt

Documentation to deploy with DeepStream is provided in Deploying to DeepStream section of TAO User Guide.

Technical blogs

Suggested reading

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Promise and the Explainability, Bias, Safety & Security, and Privacy Subcards.