Cogl or OpenGL for 3D clutter scenes?

I'm using Cogl for a project, where I create an intuitive user interface for pilots. The starting point is a 2D background, actually a video through gstreamer, which shows the actual world. An overlay on that video, also called "OnScreenDisplay" or OSD or HUD provides some interesting numbers on plane speed, altitude and whatever else is of interest. In this different app though, I'm also painting in some 3D objects over the real image, which is intended to provide additional information on where things are located. Basically augmented reality. Since this is intended for video piloting, this seems like a very good combination.

In my first implementation I was using OpenGL directly. After some issues related to how to set this up, I had that working and things were showing up quite nicely and were spot on referenced. I then switched to gstreamer + gl extensions (glupload + glimagesink), but found that these were somewhat difficult to get absolutely right. The texture of the video was not as good as it could be due to some mipmapping issues. The gstreamer extension of clutter turned out better and clutter provides some interesting capabilities for painting text and a variety of other things, like bitmaps with animation and cairo for custom drawing.

Unfortunately, clutter uses cogl in the backend (apparently) and this means that the original code for augmented reality objects no longer worked properly. I tried to get this to work by saving opengl states directly and then paint it over the image, but that didn't work out. Although the position seemed correct, the material turned a solid or transparent grey. Due to the complexity of getting this to work properly, I decided to just bite the bullet and do this in cogl entirely.

What I started out with was to define a custom actor that can be plugged into the main render loop. The idea is then that this actor gets access to the state of the application at the right time, so that this decomplicates finding alternative ways to do your painting. If you don't do that, then you'll be painting direct opengl anywhere the application decides you should paint. Usually that's after buffers were flushed already, so you have issues with blending and things just look weird. So when you want to use 3D in clutter, you should use cogl to save yourself the pain. There are absolutely no guarantees if you do use plain old OpenGL and things don't work. I had unexplainable issues related to color, but the rest looked ok to me. That's why I went the cogl way to get more predictable results.

Once you have a custom actor, then override the paint method with this:

static void scene_actor_paint (ClutterActor *actor) {
CoglMatrix mvMatrix, pMatrix;

cogl_get_modelview_matrix( &mvMatrix );
cogl_get_projection_matrix( &pMatrix );

cogl_set_modelview_matrix( &matrix );

cogl_perspective( fov,
    (float)stage_width / (float)stage_height,
    zNear,
    zFar );

// Let pilot know its position and attitude
pilot_setPosition( telemetry.lat, telemetry.lon, telemetry.alt );
pilot_setAttitude( telemetry.pitch, telemetry.roll, telemetry.yaw );

// This rotates the world around the pilot...
pilot_display();

// show some objects
yyyyyyyy_display();
......

cogl_set_modelview_matrix( &mvMatrix );
cogl_set_projection_matrix( &pMatrix );
}

And then in order to render some other object in this scene:

void yyyyyyyy_display() {
   cogl_push_source( material );

   cogl_push_matrix ();
       cogl_translate( home_e, home_n, home_d );
       cogl_rotate( home_hdg, 0.0, 0.0, 1.0 );
       cogl_rotate( home_elev, 1.0, 0.0, 0.0 );

       cogl_polygon( vertices, 12, FALSE );      
   cogl_pop_matrix ();

   cogl_pop_source();
}

So this is how you jump out of the clutter loop:

1. Define a custom actor. I did one in C, another example uses the C++ version. See also here.
   
2. Define some properties that modify how things are rendered and some other general behavior.
3. Override the paint loop. Save the matrices, define your own matrices, call your custom drawing code in 3D (has to be cogl!) and then put the matrices back as you found them.

W5100 chip garbage output

I've got my hands on an Arduino + W5100 ethernet chip. This allows an Arduino to communicate with other clients over an ethernet connection.

When I compiled the example in Arduino however, there is one noticeable bug that is resolved in the newest ethernet library distribution:

http://code.google.com/p/arduino/issues/detail?id=605&start=200

The other thing is that the example code contains: if ( client == true ) , which can be replaced by a simpler ( if client ).

After these fixes, the W5100 behaves properly. On to the next challenge!

Ocelot install

I've just installed Ocelot and ran into loads of problems. The install process crashed on me right when it was finalizing some updates, so I had to rerun a couple of dpkg configures. Ubuntu didn't even boot into X, nor the normal shell, but I managed to boot into a previous Linux kernel of natty.

The most important thing that could go wrong is probably the '/var/run' to '/run' relocation. The install script is supposed to copy contents of /var/run to /run, contents of /var/lock to '/run/lock' and then delete /var/run and /var/lock altogether and replace them by symlinks.

(i) create directories /run and /run/lock,
(ii) move contents of /var/run into /run and /var/lock into /run/lock,
(iii) delete directories /var/run and /var/lock
(iv) create replacement simlinks; e.g. 'ln -s /run /var/run' and 'ln -s /run/lock /var/lock'

Not doing this gives you problems like 'waiting 60 seconds for network...' and longer boot times, but X never starting. Before you think it's an issue with the graphics driver (which is also likely), make sure the above is correct and sorted first.

Charmed by pythons

In my latest project I'm using Python to construct the basis for a GUI application. Because one of the main design goals is to make this as modular as possible, it is also used to construct an application messaging bus and another object to keep application data in one single place. My experiences so far are extremely positive. I'm used to formally specified languages where there is no possibility to become confused about the meaning of a parameter or what its type is. This makes it slightly easier at times to understand what a parameter is doing, but at the same time removes some of the flexibilities that these programming languages offer. Python seems to be the ultimate mix between form and function, although it takes some time to get used to the idiosyncrasies of this particular language. Once you stop worrying that your application isn't going to be used after three years anyway and that nobody wants to extend your particular piece of code, Python becomes something that you can start to embrace.

What I had to get used to at first:

• How python expects you to indent your code. I set my editors to 4 spaces instead of tabs to make my life easier. Still, you download a snippet of code from the Internet and you end up rewriting tabs as spaces and vice versa.
  
• The same indentation levels is how scope is managed, whereas C and Java use scope braces.
• The ability to simply assign a variable some value and how it persists over time. There is still a gotcha or something to remember here, because sometimes variable assignments are persisted in the instance and not the class. But usually this turns up soon enough.
• Some short-hand notations for iterators over collections, sets, lists, deques and dictionaries. It takes some time to get used to how double braces differ from brackets and from square brackets (they mean different things), but when you know Java and the differences between sets, maps and lists, these notations become rather natural.
• How some declarations or references of C libraries eventually must be interpreted to understand which classes must be instantiated and where C-enumerated types are declared in the python bindings (at least it's consistent!)
  

The awesome thing in python is that it's not just something you run on the command line anymore. We're using this together with the Gtk 2/3 libraries, Clutter and libchamplain. These are highly graphical applications written in C or C++ and python with the GObject bindings give you access to all the functionalities in those classes.

The two coolest things in python is that we now have access to a very clean and empty user interface application that we can enrich using a set of plugins. If you know what a model/view/control (MVC) separation of concerns is, then python definitely knows how to support that. For our data and for our messaging bus, we've created a singleton object in the VM which every object can get to in a very simple way. Any plugin can declare data items that it wants to store and it can itself use the message bus to declare new kinds of signals that other plugins can react to, or it uses the messaging bus to declare interest in messages of other plugins.

This way, the application is 100% modular, but there's still a sense of control onto what kind of data is stored, where it is stored and it warns developers when a plugin wants to get access to data that hasn't been put there in the first place.

The plugins we've defined are all of a specific type and have their specific pre-determined uses. Communication plugins usually run in a separate thread and they're responsible for opening their own sockets. They then receive or send information from/to the system. Using the messaging bus notification signals, they extract information from the model and send this on, or they receive new information from the environment and add this to the model.

At some point though, one needs to be aware that any application can ever do so much. The multi-threadedness is highly governed by the ability of the main thread to keep up with whatever is going on in the environment. That is... in a graphical environment like clutter or gtk you can't update or manage components from just any thread, but you can only do that from the main thread that is running Clutter.main() or Gtk.main(). This usually means you add notifications to the message queue of the main thread, which is only emptied when the main thread becomes idle.

Thus... if you are in an environment where lots of user interactions happen and the UI is never truly idle, the communication message handling may start to delay by quite a bit and you may notice 'halts' in the UI updates from these systems. Because of that, this is not necessarily the way to go for everyone. But this is the best of both worlds really... you can't have blocking sockets in UI thread code, you can't/shouldn't obstruct the general UI thread with system messages (making user interaction choppy) and other considerations like that.

So, the main design concepts of this system are:

• Keep data in one place wherever possible (if multiple plugins use that data, don't copy it for every plugin).
• Allow data to be private to plugins when no other plugin or code uses it.
• Pass in required references to objects that make sense to be externally referenced. Because the use of each plugin is clear, you can also separate these.
• Communication plugins probably need a separate thread for communication handling. Be careful with blocking sockets, because UDP sockets may continue blocking forever. Therefore, TCP sockets may be blocking (as far as you shutdown and close them). UDP sockets should not be blocking.
• Use a special singleton instance for a messaging bus where messages are declared on and where hooks can be inserted. This allows you to manage mbus code in one place and you have a nice intermediate class that passes signals around.
• Do not pass large amounts of data on this message bus. If large, hierarchical pieces of data are manipulated, store them in one place in a model and allow plugins to query them if they are so interested on the receipt of these signals.
• Define what your UI should look like. That is probably the only thing that ends up being highly specific code to the application. But if you have your mbus+model objects defined already (and these are generic), you'll find the main application window is nothing but a 'shell' from which plugin code is run and the logic is defined by what plugins do and which kind of clutter/gtk classes are contained in your widgets.

So yes... I've been slightly charmed by the elegance of python in certain expressions. It's a rather mathematical way of seeing things, but it starts to make sense a lot. The abundance of libraries, extensions and base libraries, most especially its support for Gnome repository bindings for all sorts of purposes make this a very attractive language to program in.

Ritewing Zephyr

So to the right is an example Ritewing Zephyr. I'm working on building my own zephyr and the build log is on my website: http://www.radialmind.org/projects/zephyrbuild. This plane will be a joy to fly. I'm looking forward on having everything done. There are quite a number of videos on them already. Check below for some examples.

Digital Video Broadcasting... how it works...

I'm reading up on Digital Video Broadcasting standards. The DVB standard is for receiving digital video back in your home. There are a couple of subtypes in this main category which distinct themselves by their error correcting facilities (based on what's needed over the medium that they are transmitted), bandwidth, etc. The DVB-S standard is one of the things I'm most interested in. Digital Video Broadcasting is usually done with MPEG-2 transport streams. Suppose you have a video on your computer and a bit of music, along with some information on what else is available on your channel. The transport stream is composed by multiplexing all the information together (rather fast) and creating one larger bitstream that can be transported using either DVB-T (terrestrial), DVB-S (satellite) or DVB-C (cable).

I had thought that analog TV would be more resilient to noise created in the atmosphere, but this is not necessarily the case. If you send a file over the ether composed of 0's and 1's, then any noise or interference in the stream may cause a bit to be misread or misinterpreted or missed. Since the playback of a file is usually dependent on all the bits being read correctly, this is where you may get huge problems already. One or slightly more bits falling over may already cause the entire stream unusable.

Unless.... you add error correction. But this increases the size of the entire stream... How then...? Well, the MPEG-TS doesn't carry nearly as much information as an analog video stream, because analog stuff is not compressed, although in the analog world you can remove some information without significantly reducing the quality of your experience (one example here is mp3). In analog video, this means you can easily reduce a bit of the color in an image, although luminance (that which you'd see as black and white) is far more important for a person's perception of an image.

Back to MPEG-2 however... digital compression standards rely on encoding those things that matter only once, where 'motion' in the video would typically require you to encode a bit more about some spatial event for example. So a green screen that doesn't change a pixel will be very easy to transmit and extremely cheap, whereas a fast-paced action movie may temporarily reduce in quality a bit, because all the parts on screen are in motion all the time.

Let's assume that we have some digitally encoded video+audio and that it is ready for transmission. For transmission in DVB-S and all the error correction abilities we need to have at the receiver side, the huge file is packetized into 187 bytes and then a sync byte attached to the start of this "packet". The interesting thing here is that this file may be rather regular in terms of how one byte and its neighbor relate to one another. One interesting finding is that equal bytes that follow up one another may cause more reception problems at the rx side than a noisy transmission will be, because there's more variation.

For this reason, each byte in the packet, excluding the sync byte, is 'AND-ed' with a pseudo random number generator (a simple one that is). This means that some bits now turn on, others turn off and this machine has a certain period over which it operates. This PRNG is reset after every 8 frames of transmission.

What we're getting now is already an interesting stream of information that's nicely packeted, more resistant to some errors. Each packet is fed through a "Reed Solomon" encoder. This is an error-correcting encoder that has the ability to correct 8 bytes of information from this packet at the rx side. So this is the first stage where we're adding additional information to the stream that is going to help us later on. Reed-Solomon (RS) is also used frequently in other mechanisms, like storage, data transfer for other applications (CD? HD? etc.). Sometimes it's getting replaced by other algorithms like Turbocoder (space missions, etc.) and so on. Just think... the images you're seeing from Space sent by those satellites also use these schemes to ensure no bits get inverted/changed during this transfer.

The next step after the RS is some interleaving. Interleaving is a process where you shuffle parts of one packet with parts of another packet. The reason for doing this is that errors typically occur in bursts, not like 'hit and run' errors. By shuffling the original position of bytes in one packet with another, the deinterleaver relocates the parts to their original position later. If any error burst occurred, the spread of the damage caused by the error burst is much lower (it didn't zero out 3 bytes in a row, but perhaps one). Thus, it makes the signal again more robust against interference and errors.

After the interleaving, another forward error correction scheme is used called "Viterbi encoding". This may in worst case duplicate the number of bits in the transmissions stream. More bits mean higher bandwidth. The challenge is to fit the entire MPEG-2 stream within around 6MHz of RF bandwidth, so both the original MPEG-2 stream bitrate as well as what happens after that is very important. If Viterbi encoding can be less aggressive, slightly more MPEG-2 data can be sent in the channel before it uses all the allocated (planned) bandwidth.

The steps after this are 'baseband shaping' and 'I/Q modulation'. This means that the digital signal is mapped to an analog signal for transmission. Words you'll see here are "constellation". The kind used in DVB-S is quadrature phase shift keying. This means that a sequence of 2 bits is taken together and mapped to some vector 45 degrees in a constellation space. Being this far apart prevents any errors that may occur and you'd typically choose that based on the amount of expected noise in the channel. DVB-C, the cable kind, has so little expected noise that it uses 64 positions in this constellation instead. This means that in theory, it has 16 times more bandwidth.

Different analog video signals occupy 5-8 MHz in bandwidth. Expressed in Mbits/sec, PAL video equates to 216 Mbits/sec, whereas MPEG-2 compressed PAL reduces that to 2.5-6 Mbit/sec. High value means bitrate when there's lots of motion, the other when there's little. Compressed HDTV is higher than that in the order of 12-20 Mbits/sec. However, this is measured against MPEG-2. H.264 encoding is three times more efficient, so this gets HD video back into reach for actual transmissions. The alternative would be to use different modulation techniques or to occupy a larger portion than 6 MHz in the transmission region.

Some issues still arise... DVB-S was specifically created for LOS conditions away from reflective buildings and other interference sources. As soon as DVB-S is used for terrestrial transmissions this may have a huge impact on video quality. Tests so far indicate this is not necessarily the case and I reckon that with the circular polarized antennas that for example FPV fliers are using for their analog video, the multipathing issues that threaten DVB-S may well be reduced to a minimum.

Quad tuning

I made a post to the diydrones website some days ago, explaining some modifications I made to the Arducopter source code. A colleague of mine explained me a couple of things about issues related to aircraft control. The standard Arducopter code has the attitude controller in place and the GPS Hold. I added the velocity controller. As you can see from the diagram, this means that the GPS Hold controller immediately changes the attitude of the quad on the basis of some difference in position. The attitude controller itself just maintains a certain angle setting.

The attitude of the quad may induce a certain acceleration into some direction. When you steer a quad to some location by hand, you don't maintain the angle until you get there, but steer towards the other direction for a brief moment to zero the velocity with the intention to have zero velocity on the intended position.

My experiences with the GPS Hold code in the arducopter are poor. Others have had more success, but I could never find the right settings that made the quad behave correctly in all circumstances.

Because the GPS Hold controller controls the angles directly, but doesn't look at the velocity, it will only zero the velocity after it has passed through the setpoint. This means that with some larger drifts around a setpoint, overshoot cannot be avoided. Aggressive settings then cause overshoot into one direction; the quad then slows down, reverses direction and overshoots the other direction. Thus it oscillates around a position. Higher D-gains help in this regard, but I couldn't get this to calibrate correctly. The I-term does more evil than good and should be used very sparingly.

With the velocity controller in the middle I had more success. The velocity controller is also a better means to control where one is going. Letting go of the sticks means that the quad already attempts to hover around doing nothing. With little wind you'll see that this leads to a near-perfect GPS Hold operation. The GPS Hold code that you do put in than only removes the little offset that does take place due to small disturbances and other inaccuracies due to some dampening filter on the GPS course/speed readings.

In order to calibrate things correctly, start with the last controller going backwards. The attitude of the quad must be maintained with near perfection. Indoors in a large enough area, it should not travel significantly in any direction. If the quad does that, it may indicate:

• most likely cause: too many vibrations in the quad causing the IMU to get slightly confused at times or over time. It may then tilt somewhat into any direction causing speed to build up.
  
• motors not pointing straight up, so that propellers have thrust in the xy-plane.
• Incorrect response of ESC / motor due to incorrect ESC calibration, defect motor, etc.

I cannot stress enough how important it is to remove vibrations as best as you can, because you get much better results that way. In my case, I've flown with the quad in a situation where it was moving about quite a bit in outdoor environments and making a sound like a lawn mower (you know, where the mower blades cut reeds, those kinds of 'graty' sounds). I found out eventually that my bolts appeared to be tightened, but with a proper spanner could still tighten it further by 1/4 turn. This improvement for about 5 bolts resolved the graty sounds entirely and on the next launch, it hummed perfectly. That is how much bolts / nuts may impact stability, the way how motors are balanced and in turn impact the vibrations on the IMU. So make sure that works ok.

To calibrate the attitude controller, set the tx into attitude control mode of course. Then zero I and D and start with the P setting. You're looking for a P-setting that is just high enough to cause the quad to just about oscillate. Then lower the P setting a little notch (this is a relative operation) and work on the I and D terms next. The I-gain has two purposes:

• Increase the speed at which you find your setpoint.
• Resolve any bias that may accumulate in your system.

The bias for example is wind. If your quad already remains very level, in outdoor environments a level quad will drift away slowly on the wind. Suppose that you're activating the GPS Hold controller. Without the I-gain, you'll drift downwind until proportionally speaking, your quad has such an angle that it finds an equilibrium with the wind conditions. The I-gain will start to kick in, increase the angle and the idea is that the build-up of the I-gain over time and the decrease of the P-gain eventually establish a new equilibrium on the exact setpoint. That would be perfect.

The D-gain is there to reduce the speed of approximation towards some setpoint, such that it reduces overshoot of the setpoint (similar to how the GPS Hold working on angles should work).

For the attitude controller, I'm using some suggested values that are not special at all: P=3.4, I=0.015, D=1.2. These are the values for my quad and mine is custom built with relatively large distances between props. It's likely that if you have a smaller quad, you can sustain some more aggressive values.

Soon as the attitude controller is stable, work on the velocity controller. This only has two variables to adjust: P and I. At some point, especially with systems that have low frequency of reads, there's no point to use D-terms anymore. The P-gain for the velocity controller should not be too high to prevent instability. The velocity controller depends on the GPS information and this is basically some complicated piece of hardware nowadays with its own filters, dampeners and other algorithms. It's likely that a high frequency GPS (10Hz) together with doppler shift readings for speed give the best results. I set the P-gain to 0.04, which equates to a 4 degree angle when speed is 1 meter per second. If this is set more aggressive, it's possible that you see a circling motion occur due to the way how ground course is calculated in some GPS's. The I-term is basically determined on the basis of how much 'angle' one would need to compensate for windy conditions (in order to still develop a certain velocity).

Since the velocity controller is already very effective in keeping the quad fixed in place, the GPS Hold controller is just there to resolve any difference in position that still does occur over times in the 20-30 second area. It slowly develops a certain velocity that the quad should have towards the setpoint and slowly retargets the quad towards a certain position. My GPS hold controller only uses a P-setting. An I-term could be added to make it slightly more aggressive, but I never felt a need to do that.