In the context of providing a render-based tracker, this tutorial introduces a new, easy to use renderer based on Panda3D.
This renderer can output:
A color image, with support for textures and lighting
Depth image
Normal maps
In world space
In camera space
It only supports camera models with no distortion.
It is also possible to compute camera clipping values, depending on the pose of an object in the camera frame. This ensures that the depth buffer is as accurate as possible when considering this object.
Below is a set of renders for a textured cube object.
Multi-output rendering is performed via the vpPanda3DRendererSet class, which duplicates the scene across multiple renders and synchronizes changes to objects and the camera. Each Sub renderer implements a specific type of render: geometric (vpPanda3DGeometryRenderer) or color-based (vpPanda3DRGBRenderer) etc. They all inherit from vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.
2. Panda3D installation
2.1 Installation on Ubuntu
It is recommended to install the assimp package before installing Panda3D. This will allow you to load .obj files, which is a common and easily exportable format. Without it, you will be restricted to .bam files, which is a panda specific format. To convert to a .bam file, see 3. Converting a mesh to Panda's file format.
To install assimp, run:
$ sudo apt install libassimp-dev
There are three different ways to install Panda3D.
The easiest way is to download an installer available for Ubuntu browsing the download page. At the time this tutorial was written, the last Panda3D release was 1.10.15 version. Once downloaded you can install Panda with something similar to:
install the produced Debian package (recommended) with:
$ sudo dpkg -i panda3d1.11_1.11.0_amd64.deb
use the Panda3D libraries located in the built folder without installing the Debian package panda3d1.11_1.11.0_amd64.deb, but in that case you need to set LD_LIBRARY_PATH environment var:
Without setting LD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
$ ./tutorial-panda3d-renderer
./tutorial-panda3d-renderer: error while loading shared libraries: libp3dtoolconfig.so.1.11: cannot open shared object file: No such file or directory
The other way is to install Panda3D from source using cmake:
$ mkdir -p $VISP_WS/3rdparty/panda3d
$ cd $VISP_WS/3rdparty/panda3d
$ git clone https://github.com/panda3d/panda3d
$ cd panda3d
At this point you need to patch Panda3D source code to ensure Panda3D is build with -frtti compiler flag as described in 7.4 Link issue: undefined reference to typeinfo section. This could be done by modifying dtool/CompilerFlags.cmake file replacing:
By default, Panda3D is installed in /usr/local/. ViSP is then able to detect Panda3D by using /usr/local/lib/cmake/Panda3D/Panda3DConfig.cmake file.
Depending on how you installed Panda3D, building ViSP may differ:
If you install Panda3D using a .deb file as described in 1. or 2.a sections, you may notice that there is nothing specific to do, just run cmake as usual:
$ cd $VISP_WS/visp-build
$ cmake ../visp
$ make -j$(nproc)
If you prefer rather use Panda3D build libraries as described in 2.a section. without installing the .deb package you can either:
set Panda3D_DIR cmake var to the Panda3D cloned folder:
If you build Panda from source using cmake, just run cmake as usual:
$ cd $VISP_WS/visp-build
$ cmake ../visp
$ make -j$(nproc)
2.2 Installation on macOS
There are three different ways to install Panda3D.
Installers are available for macOS by browsing the download page.
Note
For the latest Panda3D 1.10.15 SDK, there is an Installer for macOS X 10.9+ that is only compatible with the x86_64 architecture. If you are using a Mac M1 or M2, there is no Panda3D SDK available yet for arm64 architecture. The solution is to build Panda3D from source.
Hereafter you will find the instructions to build Panda3D from source on macOS with makepanda.py.
It is recommended to install the assimp package before installing Panda3D. This will allow you to load .obj files, which is a common and easily exportable format. Without it, you will be restricted to .bam files, which is a panda specific format. To convert to a .bam file, see 3. Converting a mesh to Panda's file format. To install assimp, run:
$ brew install assimp
On macOS, you will need to download a set of precompiled third-party packages in order to compile Panda3D. Navigate to Panda3D download page, select the lastest SDK (in our case SDK 1.10.15), and under </> Source Code section, download Thirdparty tools for macOS (in our case panda3d-1.10.15-tools-mac.tar.gz).
Extract third-party tools for macOS from downloaded archive
$ cd ~/Downloads
$ tar xvzf panda3d-1.10.15-tools-mac.tar.gz
Once done clone Panda3D:
$ mkdir -p $VISP_WS/3rdparty/panda3d
$ cd $VISP_WS/3rdparty/panda3d
$ git clone https://github.com/panda3d/panda3d
$ cd panda3d
Move the downloaded third-party tools in Panda3D source code folder
install the produced Panda3D-1.11.0.dmg file (recommended) just by double clicking on it. In the installer window, don't forget to enable the C++ Header Files check box before pressing the installation button. After that you have to set DYLIB_LIBRARY_PATH environment var:
or use the Panda3D libraries located in the built folder without installing .dmg file, but in that case you need to set DYLIB_LIBRARY_PATH environment var:
Without setting DYLD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
$ ./tutorial-panda3d-renderer
dyld[257]: Library not loaded: @loader_path/../lib/libpanda.1.11.dylib
Now to build ViSP with Panda3D support when .dmg file Panda3D-1.11.0.dmg is installed, you can just run cmake as usual. Note that PCL is not compatible with Panda3D, that's why we disable here PCL usage (see 7.2 Segfault: :framework(error): Unable to create window).
$ cd $VISP_WS/visp-build
$ cmake ../visp -DUSE_PCL=OFF
$ make -j$(sysctl -n hw.logicalcpu)
There is also the possibility to build ViSP with Panda3D support without installing the .dmg file
By setting Panda3D_DIR cmake var to the Panda3D cloned folder
Installers are available for Windows browsing the download page.
3. Converting a mesh to Panda's file format
If you did not install assimp before installing Panda, you are fairly restricted in what mesh format you can load with Panda.
Panda's recommended format is .bam, a binary format specific to the engine itself. If you have Python, you can easily convert your model to a .bam file by doing the following:
Installing the panda3D-gltf package: pip install panda3d-gltf
From your favorite 3D modelling software export your model to GLTF (available in blender)
Running the converter tool:
$ gltf2bam source.gltf output.bam
If you encounter issues, check the tool's settings with gltf2bam -h
4. Using Panda3D for rendering
An example that shows how to exploit Panda3D in ViSP to render a color image with support for textures and lighting, a depth image, normals in object space and in camera space is given in tutorial-panda3d-renderer.cpp.
To start rendering, we first instanciate a vpPanda3DRendererSet. This object allows to render multiple modalities (color, depth, etc.) in a single pass. To add different rendering modalities, we will use subclasses that will be registered to the renderer set. Internally, each sub renderer has its own scene: the renderer set synchronizes everything when the state changes (i.e, an object is added, an object is moved or the camera parameters change.)
A panda3D renderer should be instanciated with a vpPanda3DRenderParameters object. This object defines:
The camera intrinsics (see vpCameraParameters): As of now, Only parameters for a distortion free model are supported
The image resolution
The near and far clipping plane values. Object parts that are too close (less than the near clipping value) or too far (greater than the far clipping value) will not be rendered.
The creation of the renderer set can be found below
vpPanda3DGeometryRenderer instances allow to retrieve 3D information about the object: these are the surface normals in the object or camera frame, as well as the depth information.
vpPanda3DRGBRenderer objects perform the traditional color rendering. Lighting interaction can be disable, as is the case for the second renderer (diffuse only).
Once they have been added, a call to vpPanda3DBaseRenderer::initFramework() should be performed. Otherwise, no rendering will be performed and objects will not be loaded.
Here you will find the code used to configure the scene:
We start by loading the object to render with vpPanda3DBaseRenderer::loadObject, followed by v pPanda3DBaseRenderer::addNodeToScene. For the Color-based renderer, we add lights to shade our object. Different light types are supported, reusing the available Panda3D features.
Once the scene is setup, we can start rendering. This will be performed in a loop.
Each frame, we compute the values of the clipping planes, and update the rendering properties. This will ensure that the target object is visible. Depending on your use case, this may not be necessary.
Once this is done, we can call upon Panda3D to render the object with
Note that under the hood, all subrenderers rely on the same Panda3D "framework": calling renderFrame on one will call it for the others.
To use the renders, we must convert them to ViSP images. To do so, each subrenderer defines its own getRender method, that will perform the conversion from a panda texture to the relevant ViSP data type.
For each render type, we start by getting the correct renderer via the vpPanda3DRendererSet::getRenderer, then call its getRender method.
Now that we have the retrieved the images, we can display them. To do so, we leverage utility functions defined beforehand (see the full code for more information). This may be required in cases where the data cannot be directly displayed. For instance, normals are encoded as 3 32-bit floats, but displays require colors to be represented as 8-bit unsigned characters. The same goes for the depth render, which is mapped back to the 0-255 range, although its value unbound.
Finally, we use the snippet below to move the object, updating the scene. To have a constant velocity, we multiply the displacement by the time that has elapsed between the frame's start and end.
The installer calls makepanda/makepandacore.py and enters in SdkLocateMacOSX() function to try to find the latest MacOSX SDK in one of the following folders:
The following linking issue can occur when you mix -fno-rtti and -frtti compiler flags.
[100%] Linking CXX executable catchPanda
/usr/bin/ld: ../../lib/libvisp_ar.so.3.6.1: undefined reference to `typeinfo for GraphicsOutput'
/usr/bin/ld: ../../lib/libvisp_ar.so.3.6.1: undefined reference to `typeinfo for GraphicsBuffer'
/usr/bin/ld: ../../lib/libvisp_ar.so.3.6.1: undefined reference to `typeinfo for TypedObject'
collect2: error: ld returned 1 exit status
ViSP needs to be built with -frtti to avoid build issues in vpIoTools.cpp, while Panda3D enables -frtti when built in Debug mode and enables -fno-rtti otherwise. We cannot have one library built with -frtti and another one with the -fno-rtti flag. The solution here is to patch Panda3D's source code to always enable the -frtti flag. This can be achieved by modifying the dtool/CompilerFlags.cmake file:
1.15.0