Nova Extrinsics Sensor Calibration Tool

Tool Overview

The Nova Extrinsics Sensor Calibration Tool is a containerized application that provides extrinsic calibration between sensors on a Nova Carter robot. On initial setup of Nova Carter or when recalibration is required, this application is used to generate a calibration file with the relative position of different sensors on the robot.

The calibration file is used for robotics functions that combine information from multiple sensor modalities. This calibration process achieves highly consistent results by using a single calibration target and a mobile device connected to the robot over a web browser, which guides a user through the procedure of placing the calibration target in different positions around the robot.

Visuals of the calibration tool guiding the operator to position the calibration target in different locations.

This tool is self-contained and performs both data collection and calibration optimization on the robot. It does not require external services or internet connection after the container is deployed to the robot. The resulting calibration file is deployed to /etc/nova/calibration/ both in JSON (isaac_calibration.json) and URDF (isaac_calibration.urdf) formats.

Requirements

The Nova Extrinsics Sensor Calibration Tool requires the following components and space considerations:

• Nova Carter robot.
• Calibration target (precisely following description below).
• Mobile device (tablet recommended) attached to the back of the calibration target.
• Access from the mobile device to the robot on the same Wi-Fi network.
• Bright room with diffuse light and open space around the robot and calibration target.

Calibration Target

The checkerboards used for extrinsic calibration target must have the following qualities:

• The material of the board should be a "Sandwich Composite" material with a low-density foam core and hard, laminated surfaces. "Ultraboard" is the recommended product for the calibration process, but any material adhering to the description provided would be sufficient
• Thickness greater than or equal to 3/4" (19mm) to avoid deformation (bending) of the board. The checkerboard should be rigid without any noticeable bending when holding it during operation.
• The dimensions of the checkerboard squares must be accurate, within +/- 1mm from nominal measurement. It is recommended that all new checkerboards have dimensions verified with a ruler or calipers for quality assurance.

The design of the checkerboard (page 1) and printing instructions (page 2) are included in this pdf file. The checkerboard dimensions must follow the description below:

• Outer Dimensions [mm]: 800x1200
• Pattern Dimensions [mm]: 700x1100
• Square Size [mm]: 100x100
• Circle Radius [mm]: 12.5
• Border Width [mm]: 50

Finding a Printing Service:

There are different ways to find a service to print calibration targets, for example:

• Contacting printing services in your area and providing the specifications outlined above.
• Searching with a preferred search engine, for example: "ultra board" 3/4 inch printing <your area>.
• Finding local foam board suppliers for Ultraboard.

Mobile Device

The calibration tool runs on the robot but requires that the operator use a mobile device (tablet recommended) attached to the back of the calibration target for visualization of the tool's web UI. Otherwise, images might look flipped.

• Device specs: 10.5'' screen or larger and a case to attach to the calibration target.
  • Tablet example: Galaxy Tab A8 (SM-X200)
• Web browser: Google Chrome at the latest version available in the app store.
• Wi-Fi requirements: Mobile device needs to be on the same Wi-Fi as the robot being calibrated. Wi-Fi connection needs to be stable and strong.

Hand Holders and Mobile Device case

It is necessary to attach three holders at the back of the calibration target, one for each hand and one for the mobile device (tablet case is sufficient).

Use strong velcro (able to hold 15 lb or more) to attach the mobile device and the holders at the back of the calibration target, as shown in the image below.

NOTE: Do not hold the calibration board by its sides. Any occlusion or modification of the checkerboard borders may lead to a non-optimal or invalid calibration result.

View of the back of the checkerboard with tablet and holders attached with velcro.

Space Requirements

Illumination

You must perform the calibration process in a bright room with diffuse light, to avoid glare on the calibration target.

Open Space Around the Robot and the Calibration Target

During the full calibration process the robot must be on flat ground. 4 m - 6 m of empty space around the robot and 2 m of empty space around the calibration board are required at all times. Space requirements are per camera.

During the calibration process, you must:

• Hold the calibration target by the handles in its back.
• Keep the calibration target away from walls, objects, and body parts at all times.
• Avoid people and dynamic objects between the robot and the calibration board.
• Make sure that the front of the calibration target is not obscured in any way, no matter how small.

Empty space requirement around the robot: 4 m in each direction and 6 m in the direction of the camera currently being calibrated.

Empty space requirement around target board during calibration: 2 m in each direction.

If the app guides you to poses that are close to objects or unreachable, feel free to reposition the robot and keep following the steps in the UI only after the robot is static in its new position. Do not move the robot while following the steps on the UI.

Calibrating a Nova Carter Robot

The calibration process uses a web-based app running on a Nova Carter robot, a calibration target, and a mobile device. Make sure you have all the components before starting the calibration process.

The web-based app supports the calibration of the following tasks:

• 3D Lidar - robot
• 3D Lidar - front stereo camera
• 3D Lidar - rear stereo camera
• 3D Lidar - left stereo camera
• 3D Lidar - right stereo camera

The calibration process involves preparation and steps in the web app.

Deploying the Container and Preparing the Environment

1. Install nova-carter-init by following the instructions in this page.

2. Configure docker by following the instructions in this page.

3. Open a terminal, ssh to the robot and pull the Docker image:

ssh nvidia@<carter ip>
docker pull nvcr.io/nvidia/isaac/nova_extrinsics_sensor_calibration_tool:master-aarch64

4. Create a file named run_calibration_app.sh in the robot and add the following content:

IMAGE_PATH=$1

shift # remove first argument from the argument list

docker run -it --rm --privileged     --network=host \
        --mount type=bind,source=/dev,target=/dev  \
        --mount type=bind,source=/etc/nova,target=/etc/nova \
        --mount type=bind,source=/sys/devices,target=/sys/devices  \
        --mount type=bind,source=/tmp/argus_socket,target=/tmp/argus_socket  \
        --mount type=bind,source=/sys/bus/iio/devices,target=/sys/bus/iio/devices \
        --mount type=bind,source=/mnt/nova_ssd/recordings,target=/mnt/nova_ssd/recordings  \
        --mount type=bind,source=/var/nvidia/nvcam/settings/camera_overrides.isp,target=/var/nvidia/nvcam/settings/camera_overrides.isp  \
        $IMAGE_PATH "$@"

5. Set the Hesai Lidar to FIRST return mode by running the command below. This setting allows the calibration software to receive the Lidar pulses corresponding to the calibration target instead of their potential reflections.

/opt/nvidia/nova/python/venv/bin/python3 -c '
from nova_init.hesai.hesai_utils import *
l = HesaiLidar("192.168.1.201")
print(f"Previous setting: {l.get_return_mode()}")
l.set_return_mode(HesaiReturnMode.FIRST)
if l.get_return_mode() == HesaiReturnMode.FIRST:
    print("SUCCESS. Lidar set to FIRST return mode")'

NOTE: This is a persistent change, make sure to set it back to its default setting (for example, FIRST_STRONGEST) after calibrating the robot.

6. Uncover the LIDAR and cameras from obstructions, such as removable covers and protective stickers on camera lenses.

Calibrating Nova Carter Sensors

The following calibration process must be repeated for:

• Front and rear stereo cameras
• Left and right stereo cameras

1. Verify that the robot is on flat ground and with the required empty space around it.
2. Open a terminal, ssh to the robot, and run the following command to calibrate front and rear stereo cameras and 3D Lidar:

bash run_calibration_app.sh <docker image name> --stereo_cameras front_rear

NOTE: The optional flag --disable-urdf-output disables the URDF export.

3. Open Google Chrome in the mobile device behind the calibration board and navigate to the following address:

https://<device_ip>:3000/calibration.html

If any calibration step fails, you are prompted to re-launch the calibration app (rerun the same command to launch the app again).

4. Open a terminal, ssh to the robot, and run the following command to calibrate left and right stereo cameras and 3D Lidar:

ssh nvidia@<device_ip>
bash run_calibration_app.sh <docker image name> --stereo_cameras left_right

5. Open Google Chrome in the mobile device behind the calibration board and navigate to the following address:

https://<device_ip>:3000/calibration.html

If any calibration step fails, you are prompted to re-launch the calibration app (rerun the same command to launch the app again).

6. Follow the instructions in the mobile device and place the checkerboard in the requested positions by aligning the contour of the red dots with the blue rectangle (dots). For example:

Screenshot of the calibration tool web UI guiding the operator to position the checkerboard towards their right.

You must:

• Hold the calibration target by the handles in its back. It must not be held by its sides.
• Keep the calibration target away from walls, objects, and body parts at all times

Operator correctly rotating the calibration target guided by the calibration tool.

Calibration File Usage and Backup

• The outcomes of the calibration process are two calibration files in JSON and URDF (optional) formats:
  • JSON: /etc/nova/calibration/isaac_calibration.json
  • URDF: /etc/nova/calibration/isaac_calibration.urdf
• It is recommended to backup the calibration files.
• By default both isaac and and isaac-ros applications expect the calibration files to be located in the paths defined above.

Performance

Repeatability

The repeatability error is measured as the transformation between a given sensor calibration and the mean transformation from a set of repeated calibrations for the same sensor. Rotation is expected to be under 2 mrad and translation under 1 cm of error.

Time Duration

Data collection and calibration time is approximately 10-15 minutes per sensor, hence calibrating the 4 stereo cameras in Nova Carter with respect to the 3D Lidar is expected to take approximately 40-60 minutes.

Limitations

Application Designed for Nova Carter

The current version of the extrinsic calibration tool is designed specifically for the Nova Carter robot and its sensors (3D Lidar and stereo cameras). Different robots and sensor configurations are not supported at the moment.

Sensor Modalities and Usability

The current version of the extrinsic calibration tool performs Lidar-robot calibration, and Lidar-camera calibration between the 3D Lidar and all the stereo cameras in the robot. Lidar-robot calibration refers to estimating the relative pose of the 3D Lidar with respect to the ground and includes 3 degrees of freedom: pitch, roll, and translation in Z axis. The calibration takes two steps (both of them include Lidar-robot calibration):

• Front and rear stereo cameras
• Left and right stereo cameras

If any calibration sub-step fails (for example, Lidar-front stereo camera), the operator is directed to restart the calibration process from the beginning. Nominals are provided for the remaining sensors.

Space and Calibration Target

Follow the instructions for space and calibration target requirements to minimize the failure rate of the tool.

Versions

2.1.0 Supporting Lidar-robot calibration, and Lidar-camera calibration for the 4 stereo cameras and 3D Lidar on Nova Carter.

License

Nvidia Isaac ROS Software License.

Third Party Software

• Third party notices
• Third party open source software