Getting ceph and rados running

I finally managed to get a rados gw going (it's a service exposing an S3 like API), so that you can access files on the huge cluster through a webserver. There were some issues where radosgw couldn't connect to ceph, but that was eventually resolved.

radosgw essentially creates new pools in ceph and at startup it does this incrementally. If this pool isn't healthy, apparently updates to it are stalled and processes that use it seem to halt. What I did to remedy it was to set the replication level to 1 for all pools. Here's my output:

# ceph osd lspools
0 data,1 metadata,2 rbd,3 .rgw.root,4 .rgw.control,5 .rgw,6 .rgw.gc,7 .users.uid,8 .users,

When you use this command:

# radosgw-admin user create --uid=johndoe --display-name="John Doe"

It stalls when .users is not set yet and this is created after you rerun.

So a remedy here is to start ceph and rados, keep looking for new pools when you continue on the tutorial and set the replication level to 1. There should be another way to create new pools with the level set correctly from the start.

# ceph osd pool set size 1

Running Ceph on standard Ubuntu 14.04

I'm looking at how to configure and run a simple Ceph cluster on a single machine only for development and integration of some other services. Ceph has grown since last time I checked it and also grown in complexity and outdated documentation.

It is moving towards the use of "ceph-deploy", but on the current version of Ubuntu this was getting me issues with host resolution, even though hosts and hostname are correct.

There's another page that uses the older method of creating a cluster, but this also creates problems when the OSD is to be started ( the one that saves your files ).

It did get me further. The link is here:

http://ceph.com/docs/dumpling/start/quick-start/

So I just followed that guide. When you make an error, you can't just remove the osd directories, because the keyring is copied along and then you get authentication issues. So on an error also remove the mds and mon and just rerun the mkcephfs command again.

I don't have a special partition available to use for ceph, so I just have files in /var/lib/ceph for now. When the service is restarted however, it complains about this:

Error ENOENT: osd.0 does not exist.  create it before updating the crush map

One solution for this is to start the OSD's yourself:

ceph-osd -i 0 -c /etc/ceph/ceph.conf

That'd get you halfway there. You only need to do this once, afterwards the automated start script from ceph will work. The next thing is that ceph health shows issues, which is because the standard replication level is 3. This means you need a minimum of 3 servers to get items replicated and we just configured 2

On my machine, I don't activate replication, so I ran:

# ceph osd pool set data size 1
# ceph osd pool set metadata size 1
# ceph osd pool set rbd size 1

You can query all pools configured:

# ceph osd lspools

The other step is to configure a rados gateway so that it's possible to access files a la Amazon S3 style. There's some sites that claim they know how to do this, but I found this one here:

http://ceph.com/docs/dumpling/start/quick-rgw/

There should be a better way to do this for simple setups.  For real clusters, I think issues should be a little bit easier as not everything is running on the same machine. I think that's causing some things to break here or there.

Transparency = f(distance) with OpenGL shaders

I'm working on a product at the moment where some OpenGL code is involved. I've worked on OpenGL before, so it's fun to work on that again. In this work I'm rendering video on a set of quads and these cover the entire screen. There is a virtual reality overlay that renders a virtual tunnel over this image.

I don't want this tunnel to extend indefinitely into the distance and in OpenGL you'd typically use some fog functionality to give a sense of depth. If you have a black background and black fog, you can create the illusion indeed as if the tunnel is disappearing in the background. From there, you'd think to apply an alpha value to fog, so that it doesn't just recolor the pixels, but it would also apply a transparency value. 

Unfortunately, fog doesn't use transparency. So if you have a transparent cube painted in fog, chances are you're going to see outlines in the transparency.

Instead I turned to vertex shaders to make objects fade away into the background. This saves a lot of work in the pre-processing pipeline, where you'd otherwise have to set alpha values on each vertex. Since I use vertex objects (uploaded to the GPU already) to paint my tunnels very quickly and without overhead, that'd mean I'd need to re-upload them every time if position changes.

Although it's initially a bit challenging to work with vertex shaders, as soon as you have a function that loads and compiles them with error detection, they're pretty straight-forward from there. Here's the vertex shader, which gets called first:

varying float fogFactor; 

void main()

{

   //Compute the final vertex position in clip space. 

      gl_Position = ftransform(); 

      // Pass through the front color (textures require something different)

    gl_FrontColor = gl_Color;

    // calculate the fog factor based on EXP2 type fog.

    const float LOG2 = 1.442695;

gl_FogFragCoord = gl_Position.z;

fogFactor = exp2( -gl_Fog.density * 

  gl_Fog.density * 

  gl_FogFragCoord * 

  gl_FogFragCoord * 

  LOG2 );

fogFactor = clamp(fogFactor, 0.0, 1.0);

}

Here's the fragment shader, which gets called later:

varying float fogFactor; 

void main(void) 
{  

   // Actual fragment color is the fogFactor as alpha multiplied by alpha setting

   // in gl_Color.

   gl_FragColor = vec4(vec3(gl_Color), gl_Color.a * fogFactor );

}

So in the end it's very simple. This code doesn't work for textures, where you need to work a little bit differently and you may need to implement your own lighting to get things to work correctly. Those examples are easy to find online.

The OpenGL fog system still needs to be enabled for this bit to work. It uses the configuration set in those parameters to achieve the effect. That also makes this a parametrizable program at runtime, which is good!

Throwing it all away and starting new

In the previous post I attempted to recover the partitioning of the drive and it looked like I had done it, but there were still issues afterwards apparently, mostly due to rEfit screwing things up (it has its own utility in the startup screen). I went back into the Disk Utility, which attempted a repair and then into gdisk, which caused more issues even then. The disk looked like this:

1 EFI

2 Mac OSX

3 Recovery

4 Recovery

5 Linux boot

6 Linux home

7 swap

It was impossible to zap any recovery and the MBR was a complete mess. I've basically done the following:

1. Recreated my external disk. 20G allocated to the mountain lion installer and the rest as a new partition for Time Machine. I just used Disk Utility and set up 2 partitions, pretty straight-forward. Make sure the partition type in Options is set to GUID partition table.
2. Created the Mac OSX installer partition onto the external drive. That's basically using the "Show package contents", finding the installerESD.img file and copying that into the partition using "Restore" in Disk Utility.
3. Saved my Mac OSX partition into Time Machine. Takes a while. Waited for Time Machine to complete.
4. Restarted the Mac and booting into the external drive with the installer. Press the Option key when rebooting the machine.
5. Completely repartitioned the main drive using the disk utility in two partitions. The first being the main Mac partition that I intend to use mostly as Mac OSX extended (Journaled). The second as free space. Note that there's no specific partition for the Recovery HD here yet. EFI is automatically created.
6. Then I restored the copy from Time Machine into the main partition. Waited for this to finish.
7. Rebooted into the Mac using the main drive (partition just created and loaded up).
8. Now you can recreate the Recovery partition at the end of this partition using this page: http://musings.silvertooth.us/2012/03/restoring-a-lost-recovery-partition-in-lion/ . Some more background info: https://plus.google.com/108724035107725322855/posts/Y33cF3cJR9o
9. Then I installed Ubuntu Linux using a special Mac Linux amd64 installer. The 10.10 version is available here: http://releases.ubuntu.com/oneiric/.
10. 
    

Mountain Lion installed!

Just found out why things were failing. There's an EFI partition on every disk that you're not necessarily seeing. First, you'd probably want to enable debug mode for the "Disk Utility" application:

defaults write com.apple.DiskUtility DUDebugMenuEnabled 1

Then, start Disk Utility, select the Debug Menu and tick "Show all partitions".
This allows you to see EFI partitions there.

The best indication of whether something is wrong or not is done by "diskutil". From the terminal, use:

diskutil list

This will pop up the partitions for your disks. My EFI was borked, as it showed "Microsoft Basic Data" there, but had the right size of 209.7 MB. This is the culprit that didn't allow my install of Mountain Lion to go ahead, plus some other issues with refit and those sorts. I've fixed this in a roundabout way, but you need an 8G or so USB disk or a spare HD (one of those USB types).

First, format the spare disk (disk1, where disk0 is your main startup disk) using the Disk Utility. Just select the spare disk, hit "partition" and make one large partition there. Then in Options, make sure "GUID partition table" is selected and that the type is "Mac OSX Extended (Journaled)" . Click OK to partition the drive. You can then "zero" the main partition as well by erasing it.

If you've downloaded the Mountain Lion Installer from the AppStore, this should be in your Applications folder. Don't open it to run it, but rightclick and "Show Package Contents". Then find the InstallESD.img file that you need in the "Contents/SharedSupport" folder there. This is the file you want to restore to the drive.

In Disk Utility, select the entire spare disk and select "Restore". Then drag the installESD.img file from the finder window into the source and the spare disk into the Destination. Hit Restore to transfer the image. What you need on the spare disk is the image, but also a prepended EFI partition on the spare too. This will be needed to copy the contents of the EFI to the main startup disk after we recreate it.

Now you can reboot the Mac. Keep the "Option" key pressed to reboot into this spare drive. What this does is that it frees up your main drive for some editing you need to perform.

Once you're finished booting into the spare mountain lion setup, open a terminal from the Utilities menu. In this terminal, verify your main startup disk is disk0, I'm assuming that for the following commands:

diskutil list

Verify that your main startup disk partitions are there. Verify that your spare disk has both the normal volume and the EFI partition too for copying that later. Do not continue if that isn't there. There's probably another mounted disk which is the install image contents.

gpt -r show disk0
diskutil unmountDisk disk0
gpt remove -i 1 disk0
diskutil unmountDisk disk0
gpt add -b 40 -i 1 -s 409600 -t C12A7328-F81F-11D2-BA4B-00A0C93EC93B
diskutil unmountDisk disk0
gpt -r show disk0
diskutil list

You should now see your EFI partition back on the main drive.

The final step is to copy the contents of the spare drive EFI into the new EFI, assuming disk1 is the spare and disk0 is the main drive:

dd if=/dev/disk1s1 of=/dev/disk0s1

Let's reboot back onto the main startup disk to fix up some more issues.

On my drive, the MBR or partition scheme (whatever that is) was out of sync with the GPT table or whatever. Now you'd like to run "Verify Disk" on the main drive to verify it's all ok. This probably calls for a repair to sync things back up, set some other things back to what they should be. Once that's done, you should finally be able to install Mountain Lion on your main startup disk.

Note that I do have an Ubuntu & rEfit setup that caused this issue probably. I didn't care about Ubuntu being corrupted, so didn't pay any attention there. Later on I'll probably reinstall refit and verify things a bit better before continuing.

Hope that helps. If anything else is wrong, please don't write, because I'm far from a Mac expert.

My main issue with the install was that the installer complained about "Mountain Lion cannot boot from this disk".

Edit: After careful review I noticed there still were issues and Ubuntu still didn't like booting up. I've fixed that in a new post.

I'm one of those victims unable to upgrade from Apple Lion (10.7) to Mountain Lion (10.8). I'm working with a Mac Mini and have a dual-boot configuration. At first, the installer wouldn't let me upgrade at all because the RecoveryHD was missing. Through the help of a blog, I could download the Recovery installer from Apple (1.0 version) and then with a couple of terminal commands, I could recreate this in the current partition. In my setup, I moved my entire disk from what I had to a new SSD, so somewhere along the way the RecoveryHD got lost probably. Anyway, after that was created, the installer seems happy to prepare the install process and reboot. That's where the problems start. When the system reboots, I see the big "X" install process starting up and then it suddenly comes up with a dialog saying that mountain lion cannot be installed to the target location. That's when you find yourself in the purgatory between the installation process and nothing else, because not even EFI works at that point.

I've tried to slightly reduce the size of my "MacintoshHD" partition by 256MB, no luck. I've tried using the terminal to remove the EFI partition, because in my setup it shows as "Microsoft Basic" of some kind, which should actually be EFI. No luck due to "Resource busy".

In the end, at the top left in the Apple command, you can select the "disk" to use to restart the system. Select your regular, trustworthy MacintoshHD there and you're back into your regular EFI startup screen, where you can select either Apple or Ubuntu.

Some things are seriously wrong in the update process and I'm not sure I want to use the current software to do my updates. I'm going to try to write a report about this experience to see if Apple can redo some of this process or "auto-fix" some of the issues. I don't have anything valuable in the Ubuntu partitions that I care about (it's re-creatable in 1.5 hours or so), but the Apple philosophy is that things should just "work", and they don't without a proper indication of what failed.

So yeah... not a standard Apple as it's a dual-boot, but certainly using their hardware and some of the indications could be a bit more informative as to why things failed.

I've just upgraded my graphics card on the 2 year old beast and switched in a 1-yr old GTX 580 from Asus. I'm on a PCI-E 16x 2.x, so figured that the 3.x would yet be a bit of a waste of money and my primary concern is CUDA anyway and the number of cores is enough. This card will take me 2 years in the future anyway. The biggest issue when installing was figuring out the power connections. After the card was plugged in, the monitor showed "no signal". Fearing I was dealt a wrong deck of "cards", I tried switching the supply, but nothing. Then I figured out that the 6-pin PCI-E connectors also had a floppy 2-pin on the side. I stuck that in on both ends and the card magically worked. I don't know why the 8-pin to 2x 6-pin is supplied, but I'm not using that at all.

So my first test here on Linux was to see if the drivers work. OpenGL is active and it's all dandy.

Started up Blender and made a Cycles render with the CPU. This took 22 seconds for a complicated scene on the CPU. Then activated "GPU render" mode and the same window rendered in like 3 seconds. Fantastic difference when you consider the amount of time it may take for an animation, as this will cut down the rendering time a factor 8 or so.

Nice card, I'm happy. Now to wait to see if the card manages to stay on and perform well for the next couple of days. I saw on forums people sometimes get issues when playing particular games after a couple of days.