Transparency = f(distance) with OpenGL shaders

I'm working on a product at the moment where some OpenGL code is involved. I've worked on OpenGL before, so it's fun to work on that again. In this work I'm rendering video on a set of quads and these cover the entire screen. There is a virtual reality overlay that renders a virtual tunnel over this image.

I don't want this tunnel to extend indefinitely into the distance and in OpenGL you'd typically use some fog functionality to give a sense of depth. If you have a black background and black fog, you can create the illusion indeed as if the tunnel is disappearing in the background. From there, you'd think to apply an alpha value to fog, so that it doesn't just recolor the pixels, but it would also apply a transparency value. 

Unfortunately, fog doesn't use transparency. So if you have a transparent cube painted in fog, chances are you're going to see outlines in the transparency.

Instead I turned to vertex shaders to make objects fade away into the background. This saves a lot of work in the pre-processing pipeline, where you'd otherwise have to set alpha values on each vertex. Since I use vertex objects (uploaded to the GPU already) to paint my tunnels very quickly and without overhead, that'd mean I'd need to re-upload them every time if position changes.

Although it's initially a bit challenging to work with vertex shaders, as soon as you have a function that loads and compiles them with error detection, they're pretty straight-forward from there. Here's the vertex shader, which gets called first:

varying float fogFactor; 

void main()

{

   //Compute the final vertex position in clip space. 

      gl_Position = ftransform(); 

      // Pass through the front color (textures require something different)

    gl_FrontColor = gl_Color;

    // calculate the fog factor based on EXP2 type fog.

    const float LOG2 = 1.442695;

gl_FogFragCoord = gl_Position.z;

fogFactor = exp2( -gl_Fog.density * 

  gl_Fog.density * 

  gl_FogFragCoord * 

  gl_FogFragCoord * 

  LOG2 );

fogFactor = clamp(fogFactor, 0.0, 1.0);

}

Here's the fragment shader, which gets called later:

varying float fogFactor; 

void main(void) 
{  

   // Actual fragment color is the fogFactor as alpha multiplied by alpha setting

   // in gl_Color.

   gl_FragColor = vec4(vec3(gl_Color), gl_Color.a * fogFactor );

}

So in the end it's very simple. This code doesn't work for textures, where you need to work a little bit differently and you may need to implement your own lighting to get things to work correctly. Those examples are easy to find online.

The OpenGL fog system still needs to be enabled for this bit to work. It uses the configuration set in those parameters to achieve the effect. That also makes this a parametrizable program at runtime, which is good!