Tutorial: Eye-in-hand PBVS with Panda 7-dof robot from Franka Emika
1. Introduction
This tutorial explains how to implement eye-in-hand position-based visual servoing (PBVS) with Franka Emika's Panda 7-dof robot equipped with an Intel Realsense camera mounted on its end-effector.
The following video shows the resulting robot trajectory when the robot is achieving a position-based visual servoing over an Apriltag target.
2. Prerequisites
2.1. Hardware
We suppose here that you have:
a Panda robot in its research version from Franka Emika that will be controlled throw vpRobotFranka class.
an Intel Realsense D345 camera attached to the robot end-effector. Note that this tutorial should also work with any other Intel Realsense camera.
2.2. Setting up a real-time kernel
In order to control your robot using libfranka, the controller program on the workstation PC must run with real-time priority under a PREEMPT_RT kernel.
Install real-time kernel
This tutorial shows how to proceed to install a real-time kernel from Ubuntu 1.04 to Ubuntu 22.04.
Note
For Ubuntu 22.04, we recommend using the Ubuntu Pro real-time kernel.
Allow a user to set real-time permissions for its processes
After the PREEMPT_RT kernel is installed and running, add a group named realtime and add the user controlling your robot to this group:
$ sudo addgroup realtime
$ sudo usermod -a -G realtime $(whoami)
Afterwards, add the following limits to the realtime group in /etc/security/limits.conf:
@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 102400
The limits will be applied after you log out and in again.
2.3. Install Franka library
Our robot is a Franka Research (FR1 or Panda) robot with a system firmware 4.2.2 version. This firmware supports libfranka 0.9.2. Note that ViSP is known to be compatible with Franka Research 3 (FR3) robot. In that case, you may install libfranka latest release.
2.3.1. libfranka 0.9.2 (Ubuntu)
To install libfranka 0.9.2 compatible with Franka Research (FR1 or Panda) robot, proceed with:
Install vcpkg the C++ Library Manager for Windows.
Install vcpkg in %VISP_WS%/3rdparty folder by clone from github repository or ensure to have the latest version with git pull. To this end open a Command (cmd) line terminal and run:
C:\> cd %VISP_WS%\3rdparty
C:\> git clone https://github.com/microsoft/vcpkg
or if already cloned:
C:\> cd %VISP_WS%\3rdparty\vcpkg
C:\> git pull
Boostrap vcpkg
C:\> cd %VISP_WS%\3rdparty\vcpkg
C:\> bootstrap-vcpkg.bat
Set VCPKG_ROOT environment var accordingly to the path to vcpkg folder using a Command (cmd) line terminal. In our case we set this var using:
C:\> setx VCPKG_ROOT "%VISP_WS%\3rdparty\vcpkg"
To help finding vcpkg executable, add %VCPKG_ROOT% to the Path environment variable. It could be added in a Command (cmd) line terminal with:
C:\> setx Path "%Path%;%VCPKG_ROOT%"
Close all the Command line terminals to ensure that VCPKG_ROOT and Path environment variables are taken into account.
Install dependencies
Open a new Command (cmd) line terminal and install libfranka dependencies using vcpkg:
Help finding libfranka dll files
To help finding libfranka.dll, PocoNet.dll and PocoFoundation.dll, you may update Path environment var with the following line (we recommand to use the "system settings" pannel):
Help finding libpinocchio
To help finding Pinocchio libraries, you may update LD_LIBRARY_PATH env var. The following line could also be added in ${HOME}/.bashrc file.
Follow the steps described in Tutorial: Camera eye-in-hand extrinsic calibration in order to estimate the end-effector to camera transformation. This step is mandatory to control the robot in cartesian in the camera frame.
2.7. Configure and build ViSP
Since you installed new libfranka and librealsense 3rd parties, you need to configure again ViSP with cmake in order that ViSP is able to use these libraries.
Now to help finding ViSP DLLs, add %VISP_WS%\visp-build-vc17\x64\vc17\bin\Release folder to the Path environment variable.
2.8. Configure Ethernet
Our robot controller has by default IP 192.168.1.1. Here we show how to configure a laptop that is connected with an Ethernet cable to the robot controller.
Edit Ethernet connections:
Add a new connexion using "Add" button. Choose the default Ethernet connection type:
Click "Create" button in order to create a new Franka controller connection that has a static IPv4 like 192.168.1.10 and netmask 255.255.255.0:
Click "Save" button.
2.9. Connect to Franka desk
Select the new Ethernet Networks connection named "Franka controller". When the connection is established open a web browser like Firefox or Chromium and enter the address https://192.168.1.1/desk. The first time you will be warned that the connection is not secure. Click "Advanced" and "Add Exception":
Then confirm security exception
When connected, you may release the user-stop button and open brakes:
3. Position-based visual servoing
An example of position-based visual servoing using Panda robot equipped with a Realsense camera is available in servoFrankaPBVS.cpp.
Attach your Realsense camera to the robot end-effector
Put an Apriltag in the camera field of view
If not already done, follow Tutorial: Camera eye-in-hand extrinsic calibration to estimate the homogeneous transformation between robot end-effector and camera frame. We suppose here that the file is located in apps/calibration/intrinsic/eMc.yaml.
Now enter in example/servo-franka folder and run servoFrankaPBVS binary using --eMc to locate the file containing the transformation. Other options are available. Using --help show them:
$ cd example/servo-franka
$ ./servoFrankaPBVS --help
Run the binary activating the plot and using a constant gain:
Franka network exception: libfranka: Connection to FCI refused. Please install FCI feature or enable FCI mode in Desk.
you need to use your navigator to activate FCI. This new security feature was introduced in Franka system between version 4.0.2 and 4.2.2. See known issue: 4.1. libfranka: Connection to FCI refused.
Now you should see new window that shows the image from the camera like in the next image. In this window, use the left mouse click to enable the robot controller, and the right click to quit the binary.
Legend: Example of initial position. The goal is here to bring the RGB frame attached to the tag over the yellow frame corresponding to the desired position of the tag in the camera frame.
Legend: Example of final position reached after position-based visual servoing. In green, you can see the trajectories in the image of the tag corners and tag cog. The latest correspond to the trajectory of the projection in the image of the tag frame origin. The 3D trajectory of this frame is a straight line when the camera extrinsic parameters are well calibrated.
Legend: Corresponding visual-features (translation and orientation of the cdMc homogeneous matrix corresponding to the transformation between the desired camera pose and the current one) and velocities applied to the robot in the camera frame. You can observe an exponential decrease of the visual features.
You can also activate an adaptive gain that will make the convergence faster:
When executing one of the binaries that use libfranka you may encounter the following error like:
$ ./servoFrankaIBVS --ip 192.168.100.2
Franka network exception: libfranka: Connection to FCI refused. Please install FCI feature or enable FCI mode in Desk.
Check if you are connected to the Franka robot or if you specified the right IP using --ip command line option set by
default to 192.168.1.1.
This error occurred with our robot after upgrading the Franka system from 4.0.2 to 4.2.2 version.
The Dashboard shows that after a synchonization using Franka World, our
System version is 4.2.2.
To overcome this error, you need:
First check if FCI is installed:
When FCI is installed, in Desk you should see this image.
Secondly activate FCI. For that, as shown in the next image click on "Activate FCI" in the Desk right column (1). It will open the popup (2) that should remain present all the time you want to execute any binary that needs to communicate with the robot using libfranka.
Behavior when FCI (Franka Controller Interface) is activated.
If you want to achieve a physical simulation of a Franka robot, with a model that has been accurately identified from a real Franka robot, like in the next video, we recommend to make a tour on Tutorial: FrankaSim a Panda 7-dof robot from Franka Emika simulator that is available in visp_ros. Here you will find a ROS package that allows to implement position, velocity and impedance control of a simulated Franka robot using ROS and CoppeliaSim.
the homogeneous transformation between robot end-effector and camera frame. We suppose here that the file is located in apps/calibration/intrinsic/eMc.yaml.
1.15.0