Hand-eye Calibration for Medical Robots¶

Description¶

Hand-eye calibration is a well-studied topic in the field of robotics. Applications that involve the use cameras typically require conducting hand-eye calibration beforehand. This project presents a pipeline for conducting hand-eye calibration using a bespoke marker for a medical robot (the first generation da Vinci research kit). The experimental setup is shown in Fig 1.

Fig 1. Experimental setup

The method presented in this project aims to conduct registration between two sets of point cloud using singular value decomposition (SVD). This project aims to instruct students to understand basic knowledge of hand-eye calibration and grasp essential skills in using computer vision libraries such as OpenCV and Point Cloud library in C++. Hand-eye calibration results can be visualised through 2D back projections. Fig 2 and Fig 3 display the back projection of a surgical tool shaft after accurate and inaccurate hand-eye calibrations, respectively.

Fig 2. Overlay with accurate calibration	Fig 3. Overlay with inaccurate calibration

Learning Outcomes¶

• Understand what hand-eye calibration is and prevalent school of thoughts in solving this problem.
• Develop basic skills in writing an object-oriented project in C++.
• Develop basic skills in using computer vision libraries, such as OpenCV and Point Cloud Library.

Requirements¶

Academic¶

• Basic knowledge on robotics (eg. Denavit–Hartenberg (DH) parameters, forward kinematics)
• Hand-eye calibration and fundamental knowledge on computer vision (eg. camera intrinsic matrix, extrinsic matrix, projection geometry)
• Linear algebra (eg. Singular Value Decomposition (SVD)). Detailed information can be referred to docs/Background.md

System¶

• A C++ toolchain along with the necessary development libraries:
  • Eigen library (Eigen 3)
  • OpenCV library (OpenCV 4.6)
  • Point cloud library (PCL 1.11)

Getting Started¶

Background knowledge on robotics and computer vision can be referred to docs/Background.md. The workflow of this project can be referred to docs/Code_Overview.md. docs/EigenLibraryIntro.md, docs/OpenCVLibraryIntro.md and docs/PointCloudLibraryIntro.md contain information on the basic usage of these C++ libraries, and explanations on related fuctions written in this project using these libraries. docs/input.md provides information of input files used in this project.

Project Structure¶

.
├── input
│   ├── IR
│   ├── RGB
│   ├── pcd
│   ├── Acusense_RGB_K.txt
│   ├── ExtrinsicMat.txt
│   ├── q_history.txt
│   ├── pos_act.txt
│   └── pos_des.txt
├── output
├── src
│   ├── main.cc
│   ├── FeatureDetection.cc
│   └── utils.cc
├── include
│   ├── FeatureDetection.hpp
│   └── utils.hpp
├── CMakeLists.txt 
├── docs
│   ├── Background.md
│   ├── Code_Overview.md
│   ├── EigenLibraryIntro.md
│   ├── FeatureDetection.md
│   ├── OpenCVLibrary.md
│   ├── PointClourLibraryIntro.md
│   └── input.md
├── README.md
├── LICENSE.md
├── mkdocs.yml
├── doxide.yaml
└── requirements.txt 

License¶

This project is licensed under the BSD-3-Clause license