It’s wonderful to see AR Applications, running within our physical environment. For an average guy it really looks like magic. But creating that magic was never easy. A professional magician always has to acquire a proven set of tools and techniques in order to deliver a realistic illusion. So as with the developer! For years ARToolkit served as the fundamental set of software package for crafting AR applications. However it becomes more rewarding to combine ARToolkit with a 3D Graphics Rendering Engine so that a one can create his own 3D graphics or even an animation and turn them into Augmented Reality. Therefore the objective of this article is to share my own experience in combing an AR application with a 3D rendering engine.
This application was tested in a Win32 platform. Practically I faced two challenges. The first one was aligning the pose of the marker accurately with the graphic object. The second one was to set the background of the OGRE’s render window into the camera view. It’s in the OGRE’s render window that we display both the camera background and the 3D graphic.
OGRE is a 3D rendering engine. It’s capable of drawing 3D graphics as well as animations in a window. Generally, OGRE provides two options for rendering graphics – i.e. Direct3D and OpenGL. Both OpenGL and Direct3D communicate closely with the underlying video graphics drivers in order to render the output. You can find a file called ogre.cfg getting created when you run an OGRE application for the first time. Usually this file resides in the bin\debug directory of OGRE installation-in my case it’s C:\OgreSDK\OgreSDK_vc10_v1-7-4\bin\debug. Modify this file to set the desired rendering method. It’s up to the developer to choose which rendering method to go with.
I know it’s quite lame (for pro’s ) but let me introduce the ARToolkit a little bit. It’s a set of C++ libraries and header files that facilitates in creating marker-based AR applications. It’s capable of tracking a predefined marker at real-time and calculate the pose at frame rate. By combining ARToolkit with OGRE we can track a marker real-time and apply its pose (position & orientation) to a graphics object, which is being rendered by OGRE.
Let’s see the steps involved in ARToolkit & OGRE briefly.
I started out by first downloading one of the tutorial applications in OGRE. Basically http://www.ogre3d.org/tikiwiki/tiki-index.php is good starting point. The challenge here is to supply the ARToolkit’s transformation matrix to OGRE’s graphics object. This can be achieved by running OGRE and ARToolkit in two separate threads. I used Pthreads from http://sourceware.org/pthreads-win32 .However you may stick to your own flavor perhaps… TBB (http://threadingbuildingblocks.org/) from Intel.
You’ll find out that in most of the OGRE’s tutorial applications, there’s a function named frameRenderingQueued(). We can update the scene for each frame through this function. So that we use the transformation matrix which is a 3 X 4 matrix calculated by ARToolkit and set the graphic object’s position & orientation according to following order.
As with the above manner, we can apply the marker’s pose into the graphic object. Sometimes the pose of the graphic object might not coincide with the marker pattern. For example, when we rotate marker clockwise, the graphic object turns anti-clockwise. This happens due to the misalignments in the axis directions. We can overcome this by inverting such axis. (Probably this would take some time to realize which axis to be inverted – since it’s trial & error.)
The next big challenge is to set the OGRE’s background into the camera view. In OGRE it’s not obvious to directly capture the camera view and set it as background.
First we have to create a Texture through TextureManager class in OGRE. After that we have to create a Material and bind the texture to that Material. Then a rectangle box should be created and attach the material with this Rectangular box. Finally this Rectangular box should be rendered as the background. The tutorial here http://www.ogre3d.org/tikiwiki/tiki-index.php?page=Creating+dynamic+textures&structure=Cookbook would be helpful for more details. I also went through that.
The logic behind the above process is that the data which is confined within this rectangle area need to be filled with the camera’s captured pixels. So we need to acquire the pixel buffer within this box and assign each pixel of this pixel buffer to the value of the data pointer (dereference the pointer) which points to the ARToolkit’s pixel buffer of the new video frame. Obviously this was done inside a loop. In this manner we can set the background of the OGRE render window to the camera’s view at frame rate.
In summary…mentioned here is just one way of creating an AR application with 3D graphics and animations. The libraries/frameworks used for this application are as below.
- OGRE 3D
Here’s an important note to the end. Camera connection was made by ARToolkit through DSVL – DirectShow Video Library. (see here http://www.icg.tugraz.at/Members/daniel/DSVL_doc). Therefore you have to download necessary DSVL dlls in order to function. The pixel format is 32 bit RGB.
In a later post I’ll show how to modify the DSVL connection so that we can capture video from camera in different pixel formats, resolution etc..