1. Introduction

This tutorial explains how to implement eye-in-hand image-based visual servoing (IBVS) with Franka Emika's Panda 7-dof robot equipped with an Intel Realsense camera mounted on its end-effector. It follows Tutorial: Eye-in-hand PBVS with Panda 7-dof robot from Franka Emika that explains also how to setup the robot.

2. Eye-in-hand Image-based visual servoing

At this point we suppose that you follow all the 2. Prerequisites given in Tutorial: Eye-in-hand PBVS with Panda 7-dof robot from Franka Emika.

An example of eye-in-hand image-based visual servoing using Panda robot equipped with a Realsense camera is available in servoFrankaIBVS.cpp.

Attach your Realsense camera to the robot end-effector
Put an Apriltag in the camera field of view
If not already done, follow Tutorial: Camera eye-in-hand extrinsic calibration to estimate $^e{\bf M}_c$ the homogeneous transformation between robot end-effector and camera frame. We suppose here that the file is located in apps/calibration/intrinsic/eMc.yaml.

Now enter in ${VISP_WS}/visp-build/example/servo-franka folder and run servoFrankaIBVS binary using --eMc to locate the file containing the $^e{\bf M}_c$ transformation. Other options are available. Using --help show them:

$ cd example/servo-franka

$ ./servoFrankaIBVS --help

Run the binary activating the plot and using a constant gain:

$ ./servoFrankaIBVS --eMc ../../apps/calibration/intrinsic/eMc.yaml --plot

Use the left mouse click to enable the robot controller, and the right click to quit the binary.

At this point the behaviour that you should observe is the following:

Legend: Example of initial position. The goal is here to center the 4 tag corners in the image.

Legend: Example of final position reached after position-based visual servoing. In green, you can see the trajectories in the image of the tag corners. When the camera extrinsic parameters are well estimated these trajectories are straight lines.

Legend: Corresponding visual-features (x and y coordinates of the corner points in the image plane) and velocities applied to the robot in the camera frame. You can observe an exponential decrease of the visual features.

You can also activate an adaptive gain that will make the convergence faster:

$ ./servoFrankaIBVS --eMc ../../apps/calibration/intrinsic/eMc.yaml --plot --adpative-gain

You can also start the robot with a zero velocity at the beginning introducing task sequencing option:

$ ./servoFrankaIBVS --eMc ../../apps/calibration/intrinsic/eMc.yaml --plot --task-sequencing

And finally you can activate the adaptive gain and task sequencing:

$ ./servoFrankaIBVS --eMc ../../apps/calibration/intrinsic/eMc.yaml --plot --adpative-gain --task-sequencing

To learn more about adaptive gain and task sequencing see Tutorial: How to boost your visual servo control law.

3. Next tutorial

If you want to achieve a physical simulation of a Franka robot, with a model that has been accurately identified from a real Franka robot, like in the next video, we recommend to make a tour on Tutorial: FrankaSim a Panda 7-dof robot from Franka Emika simulator that is available in visp_ros. Here you will find a ROS package that allows to implement position, velocity and impedance control of a simulated Franka robot using ROS and CoppeliaSim.

You can also follow Tutorial: Image-based visual servo (IBVS) that will give some hints on image-based visual servoing in simulation with a free flying camera.