In the past months I have played with multi-views auto-stereoscopic display, and thought that it could be a nice feature for GPAC. This is still an on-going work, but results are already interesting. GPAC can now support various output type and view number, but requires OpenGL to do so.
Warning, multi-view rendering is a costly operation. The scene gets rendered multiple times, and you will need a decent graphic card to try that, especially support for VBOs to avoid sending the geometry data repeatedly, and OpenGL shaders for most screen configurations.
The output type is selected in GPAC configuration file by the following keys
[Compositor]
NumViews=2
ReverseViews=no
StereoType=None
InterleaverShader=/path/to/fragmentshader_sourcecode.glsl
ViewDistance=180
CameraLayout=Circular
NumViews specify the number of target views. It may be ignored depending on the stereo output type. ReverseViews instruct GPAC to inverse the views, e.g. ViewX becomes NumViews-ViewX.
StereoType can be one of the follwing:
- None: regular single-view display
- SideBySide: views are rendered side-by-side, from left to right
- TopToBottom: views are rendered top-to-bottom, starting with the left
- Anaglyph (*): views are rendered as anaglyph. Number of views in this mode is forced to 2.
- Columns (*): views are rendered interleaved columns by columns. Number of views in this mode is forced to 2. Left view is mapped to even pixel X coordinate on the display window/screen.
- Rows (*): views are rendered interleaved rows by rows by. Number of views in this mode is forced to 2. Left view is mapped to even pixel Y coordinate on the display window/screen.
- Custom (*): views are rendered by the GLSL fragment shader source code specified by the InterleaverShader option.
(*): Requires OpenGL GLSL shader support.
When shaders are used, each view X in [1, NumViews] is exposed to the shader as:
uniform sampler2D gfViewX
For example, the column interleaving in GPAC is done with a built-in shader whose code is:
uniform sampler2D gfView1;
uniform sampler2D gfView2;
void main(void) {
if ( int( mod(gl_FragCoord.x, 2.0) ) == 0)
gl_FragColor = texture2D(gfView1, gl_TexCoord[0].st);
else
gl_FragColor = texture2D(gfView2, gl_TexCoord[0].st);
}
CameraLayout defines how the camera is positioned in the 3D world. It can take the value OffAxis (default), Circular or Linear – you can get pretty good overviews of this on the web. ViewDistance specifies the nominal viewing distance of the autostereo display. GPAC will use that to compute each view’s camera position in the virtual world. I have started some experiments on automatic calibration of 3D models, but it is not yet fully tested and may results in too high disparity. Don’t worry, this can be adjusted at run-time: in MP4Client, use Alt+shift+UP/Down to modify the inter-occular distance (by default 6.8 cm) and Alt+shift+UP/Down to modify the view distance.
Here are some videos showing some of these modes.
[youtube]http://www.youtube.com/watch?v=_8sZc3dL9ds[/youtube]
Original scene designed during the Triscope project, in which we built an auto-stereoscopic mobile prototype.
[youtube]http://www.youtube.com/watch?v=7bNHkuFBg0c[/youtube]
Anaglyph version of the Triscope demo.
[youtube]http://www.youtube.com/watch?v=6kzK54XiRXw[/youtube]
Side-by-Side version of the Triscope Demo.
Custom interleaving version of the Triscope Demo for a 5 views display.
The demo is written in BIFS, and uses a hardcoded proto (e.g. gpac-specific) node called DepthGroup for the 2D menu. This node allows to specify depth translation and scaling factors to a subtree, which are then cumulated when going down the subtree. We could also have used Transform3D nodes, but we didn’t want to have too many 3D matrices to handle (the original prototype platform was not that powerfull). To declare such a node in a BT file, you need to insert the following code before your scene:
EXTERNPROTO DepthGroup [
exposedField MFNode children []
exposedField SFInt32 _3d_type 4
exposedField SFFloat depth_gain 1.0
exposedField SFFloat depth_offset 0.0
] "urn:inet:gpac:builtin:DepthGroup"
You can ignore the _3d_type field, which was specific to the hardware part of the prototype.
The resulting depth value is scaled by a constant defined in GPAC configuration file:
[Compositor]
DepthScale=100
This is still very preliminary work, automatic scaling is planed in the near future.
Support for SVG and depth has also been investigated, but this work is not yet integrated in GPAC.
For images, we defined a new image format called PNGD. These are regular PNG files where the alpha channel is interpreted as the depth map of the RGB channels. These files can be played by GPAC whether in stereo-mode or not.
For video, we defined a new (not standard yet) MPEG-4 descriptor defining a video stream as carrying the depth map of another video stream. This is in-sync with approaches based on MPEG-C Part 3. You can find a sample BT file showing this here, then use MP4Box to produce the mp4 file.
The playback of video+depth or image+depth is in its early stage in GPAC, you will need a BIG machine and a recent graphics card to get good results. Don’t expect too much from high resolution video yet, they will likely overkill the player. The reason for this ? The video data is sent as as a point sprite, triangle strip elevation grid or vertex buffer, and this costs a lot (this will have to be improved in the future).
The way depth video is rendered is defined by the following option:
[Compositor]
DepthType=None
DepthType take the following value:
- None: video is rendered as a flat rectangle
- Points: video is rendered as a point sprite if GL_ARB_point_parameters extension is supported
- Strips: video is rendered as a triangle strip. You may specify “Strips=V” as an option; this will perform two-pass rendering and V the cut-off value for the depth (expressed between 0 and 255)
- VertexArray: video is rendered as a full vertex array, potentially cached if VBOs are supported.
The height of each vertex is taken from the depth map and multiplied by the DepthScale value.
Note that you don’t need stereo-rendering to play with this:
[youtube]http://www.youtube.com/watch?v=-AuaU1Eq5Xg[/youtube]
When playing depth+video or depth+image, GPAC will automatically switch to OpenGL rendering if the DepthType option is not None.
Thanks for reading, and, happy 3D’ing !!!
