Cogl or OpenGL for 3D clutter scenes?

I'm using Cogl for a project, where I create an intuitive user interface for pilots. The starting point is a 2D background, actually a video through gstreamer, which shows the actual world. An overlay on that video, also called "OnScreenDisplay" or OSD or HUD provides some interesting numbers on plane speed, altitude and whatever else is of interest. In this different app though, I'm also painting in some 3D objects over the real image, which is intended to provide additional information on where things are located. Basically augmented reality. Since this is intended for video piloting, this seems like a very good combination.

In my first implementation I was using OpenGL directly. After some issues related to how to set this up, I had that working and things were showing up quite nicely and were spot on referenced. I then switched to gstreamer + gl extensions (glupload + glimagesink), but found that these were somewhat difficult to get absolutely right. The texture of the video was not as good as it could be due to some mipmapping issues. The gstreamer extension of clutter turned out better and clutter provides some interesting capabilities for painting text and a variety of other things, like bitmaps with animation and cairo for custom drawing.

Unfortunately, clutter uses cogl in the backend (apparently) and this means that the original code for augmented reality objects no longer worked properly. I tried to get this to work by saving opengl states directly and then paint it over the image, but that didn't work out. Although the position seemed correct, the material turned a solid or transparent grey. Due to the complexity of getting this to work properly, I decided to just bite the bullet and do this in cogl entirely.

What I started out with was to define a custom actor that can be plugged into the main render loop. The idea is then that this actor gets access to the state of the application at the right time, so that this decomplicates finding alternative ways to do your painting. If you don't do that, then you'll be painting direct opengl anywhere the application decides you should paint. Usually that's after buffers were flushed already, so you have issues with blending and things just look weird. So when you want to use 3D in clutter, you should use cogl to save yourself the pain. There are absolutely no guarantees if you do use plain old OpenGL and things don't work. I had unexplainable issues related to color, but the rest looked ok to me. That's why I went the cogl way to get more predictable results.

Once you have a custom actor, then override the paint method with this:

static void scene_actor_paint (ClutterActor *actor) {
CoglMatrix mvMatrix, pMatrix;

cogl_get_modelview_matrix( &mvMatrix );
cogl_get_projection_matrix( &pMatrix );

cogl_set_modelview_matrix( &matrix );

cogl_perspective( fov,
    (float)stage_width / (float)stage_height,
    zNear,
    zFar );

// Let pilot know its position and attitude
pilot_setPosition( telemetry.lat, telemetry.lon, telemetry.alt );
pilot_setAttitude( telemetry.pitch, telemetry.roll, telemetry.yaw );

// This rotates the world around the pilot...
pilot_display();

// show some objects
yyyyyyyy_display();
......

cogl_set_modelview_matrix( &mvMatrix );
cogl_set_projection_matrix( &pMatrix );
}

And then in order to render some other object in this scene:

void yyyyyyyy_display() {
   cogl_push_source( material );

   cogl_push_matrix ();
       cogl_translate( home_e, home_n, home_d );
       cogl_rotate( home_hdg, 0.0, 0.0, 1.0 );
       cogl_rotate( home_elev, 1.0, 0.0, 0.0 );

       cogl_polygon( vertices, 12, FALSE );      
   cogl_pop_matrix ();

   cogl_pop_source();
}

So this is how you jump out of the clutter loop:

1. Define a custom actor. I did one in C, another example uses the C++ version. See also here.
   
2. Define some properties that modify how things are rendered and some other general behavior.
3. Override the paint loop. Save the matrices, define your own matrices, call your custom drawing code in 3D (has to be cogl!) and then put the matrices back as you found them.