ilkka.github.com

Packaging a Qt app for OS X

2011-10-19T20:06:27Z

I participated in BarCamp Tampere 2 recently, and one of the many very interesting presentations was Ville Ranki talking about Siilihai, a web forum reader app intentionally very reminiscent of newsreaders of old. Siilihai scrapes the content from web forums and presents it as threaded conversations so that it’s easier to read and follow conversations. Parsers can be written pretty easily for most forum software, and while forum content is only accessed locally by the client, it stores generated message IDs in the siilihai.com service so that whatever messages you read in one client are marked read in any other clients you might use as well. The service is also used as a repository for forum parsers.

Anyway, as the event was held at our office, I grabbed a Mac that was handy so that I could try out Siilihai for myself. Getting it built was simple as it is written using Qt, but I wanted to create a distributable package (an app bundle in OS X parlance) so that all the other beret wearers could enjoy it too. This, however, turned out to be a bit more involved than I’d thought.

The anatomy of an app

An app bundle is pretty self-contained. Apart from basic system libraries that can be assumed to be available on every Mac, apps usually ship all their libraries inside the app bundle together with the runnable binary. Some space may be wasted this way if several apps bundle the same libraries, but generally it seems pretty useful. At least it’s simple to deploy applications when they can simply be copied into place.

Siilihai uses a couple of shared libraries of its own, and naturally Qt. Since Qt is not installed by default in OS X, I had to include all of these in the app bundle. Bundling Qt is relatively simple using the macdeployqt tool included in the Qt SDK, but it only really takes care of the whole process if the application is a single binary that is only dependent on Qt. Siilihai’s custom libraries required a bit more love.

Rewiring and rewriting

For an app that uses dynamic libraries to work, the runtime linker must be able to find the libraries when the app is run. In Unix-style OSes this is generally accomplished by installing the libraries into standard paths such as /usr/lib. The dynamic linker is preconfigured to look in these directories when it is looking for library files.

If the libraries are for some reason installed into a nonstandard path, that path can also be configured as an “rpath” in the app binary itself. This is just another way of telling the dynamic linker where to look, but in a per-app fashion.

These concepts are valid for OS X .dylib files as well, with some differences. Using the command line utility otool we can examine this search path information. Let’s see the output for otool itself:

$ otool -L `which otool`
/usr/bin/otool:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
        version 159.0.0)

We can see that otool depends on a single dylib, libSystem. Now, my knowledge isn’t broad enough to tell if the actual path to the lib is hardcoded in the binary, or if the output is due to the dynamic linker knowing to look in /usr/lib, but in the context of this discussion it doesn’t really matter. Let’s look at the output for libSystem.B.dylib next:

$ otool -L /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
        version 159.0.0)
        /usr/lib/system/libcache.dylib (compatibility version 1.0.0, current
        version 47.0.0)
        /usr/lib/system/libcommonCrypto.dylib (compatibility version 1.0.0,
        current version 55010.0.0)
        /usr/lib/system/libcompiler_rt.dylib (compatibility version 1.0.0,
        current version 6.0.0)
        /usr/lib/system/libcopyfile.dylib (compatibility version 1.0.0,
        current version 85.1.0)
        ...

We see that libSystem is linked against a whole bunch of other system libraries, but also that the otool output lists the library itself. As far as I can tell, this is because the library knows its own path, something that needs to be taken into account later. If you know better, please leave a comment.

The consequence is that because I was going to put siilihai’s libraries in a nonstandard place (inside the app bundle), I need to edit the library locations stored in the app binary so that the libraries get found by the linker. Siilihai’s libraries also link to each other, so I have to do the same thing to them too.

If this all sounds complicated, that is because it is. In software it soon pays to automate even the simplest of tasks, and here there is definitely a lot to gain, so I wrote a script to build Siilihai and perform all these magic tricks for me.

Scripture

You can see the whole script in github, but I’ll explain the most relevant bits here.

First, the script tries to be clever about allowing you to run it from whatever directory and still finding all the sources and whatnot. For this it first locates the script directory:

SCRIPT=$(python -c 'import os,sys;print os.path.realpath(sys.argv[1])' $0)
SCRIPTDIR=$(dirname $SCRIPT)

Using GNU readlink, the first assignment would be the much simpler SCRIPT=$(readlink -f $0), but it doesn’t, hence the Python. Anyway, now that the script knows where it is, by extension it also knows where the Siilihai client source is. It assumes that libsiilihai’s sources are in the same directory.

The script creates a temporary directory and proceeds to do normal shadow builds of the lib and the client. The lib build generates exclusively .dylib files while the client generates both .dylibs and an app bundle. The next step is to copy all the .dylibs into the app bundle. It doesn’t really matter where, but I decided to put them in the /Contents/Frameworks directory since I happen to know that’s where the Qt libs will end up too.

status "Copying libsiilihai dylibs into client bundle"
BUNDLEDIR=$CLIENTBUILDDIR/src/reader/siilihai.app
FWKDIR=$BUNDLEDIR/Contents/Frameworks
mkdir -p "$FWKDIR"
for LIB in "$LIBBUILDDIR"/src/*.dylib "$CLIENTBUILDDIR"/src/common/*.dylib \
        "$CLIENTBUILDDIR"/src/parsermaker/*.dylib; do
        cp -a "$LIB" "$FWKDIR"

Now we use install_name_tool to set the ID of each library after it is copied:

LIBBASENAME=$(basename $LIB)
LIB=$FWKDIR/$LIBBASENAME
install_name_tool -id "@executable_path/../Frameworks/$LIBBASENAME" \
        "$LIB"

We set the ID to be the relative path inside the app bundle from the executable binary (which resides in /Contents/MacOSX) to the library. This is the same rpath that will be set in the executable later.

Since all the Siilihai libraries are linked against Qt, and the Qt frameworks will be deployed into the app bundle too, we need to change their rpaths in the libraries. At this point they still point at e.g. the Qt SDK used to build the libs. We grep the framework names from otool output, generate new paths for them and change the paths in the libs:

QTDEPS=$(otool -L $LIB|egrep 'Qt.*\.framework'| \
        egrep -v '@executable_path'|awk '{print $1}'|tr '\n' ' ')
for DEP in $(echo $QTDEPS); do
        NEWDEP=$(echo $DEP|sed 's%.*gcc/lib/%%')
        install_name_tool -change "$DEP" \
                "@executable_path/../Frameworks/$NEWDEP" "$LIB"
done

Note that we ignore any Qt frameworks that might already have working paths with the egrep -v '@executable_path' bit. Now when the Qt frameworks get copied into /Contents/Frameworks, the Siilihai libs will get linked against them at runtime. The libs are dependent on each other too, though, so we need another round of dependency path fixing, this time picking anything with libsiilihai in the name:

DEPS=$(otool -L $LIB|egrep 'libsiilihai'|egrep -v '@executable_path' \
        |awk '{print $1}'|tr '\n' ' ')
for DEP in $(echo $DEPS); do
        install_name_tool -change "$DEP" \
                "@executable_path/../Frameworks/$DEP" "$LIB"
done

That concludes the library ID and path fixing loop. Now the only thing remaining is to run the macdeployqt tool. This tool finds all the Qt frameworks the executable is linked against and copies them into the bundle. It also strips them and fixes all the IDs and rpaths. As an additional bonus, since we’ve already copied Siilihai’s own .dylibs into the bundle, it also fixes their paths in the executable. The result is a self-contained app bundle that can be run in any reasonably modern Mac. Currently I believe the bundle is compatible with Intel Macs running OS X 10.5 or later.

Further development

One rather major problem with the script so far is that it leaves the app bundle using the default app icon, while it should use the Siilihai icon from the repo. This is something I’ll have to find out how to do. Other than that, the only thing I’ve thought of is support for tagging the repos when successful builds are made, and settings some sort of version metadata, should bundles support something like that.

The experience does leave me with the feeling that if I was to write an app that I wanted to deploy onto Macs, I would just link everything into a monolithic binary. Given today’s storage space yadda yadda etc., you get the point.

Dgh, The Debian/Ubuntu Downgrade Helper

2011-05-25T19:39:00Z

Ever had the bright idea of upgrading, say, your Ubuntu machine to use some snazzy new software from a PPA, only to have it blow up in your face? The grim realization that there’s no easy downgrade path, instead you’ll have to find every single package the PPA upgraded and downgrade them by hand to keep software from breaking? Me too!

I recently had this exact situation with the gnome3-team PPA. It worked like a charm in my work computer, but when I tried it on my Nokia Booklet, it totally failed, probably thanks to the infamous GMA500 graphics hardware. Anyway, after a bit of hand-wringing I realized that:

I had to downgrade, there was stuff on the machine I didn’t want to wipe,
I had been in this same situation before, and
probably others had, too.

So I wrote a little tool to help, and I’m glad to say it worked perfectly. The tool is dgh, and it helps by finding the packages that look like they need to be downgraded. It does this by looking at apt-cache policy output, and finding packages whose installed version is not available in any repository. Usage is simple, but requires some manual work (for now):

Remove the offending package archive from apt’s sources
Run apt-get update
Produce apt-cache policy output for all installed packages, e.g.:

    <pre><code>dpkg --get-selections| \
egrep '\binstall'|awk '{print $1}'| \
xargs env LANG=C apt-cache policy| \
tee packages.list</code></pre>

Run dgh on the output:

    <pre><code>$ dgh packages.list</code></pre>

Dgh will report packages it thinks are downgrade candidates. Note that it will probably report all manually installed packages too, so be careful when downgrading.

I have a few enhancements I want to do before I call it 1.0, like not requiring a pre-created policy output file and perhaps even performing the downgrade by itself, but it’s usable now, just grab the code or gem install dgh directly from rubygems.

Qt Quick 2 QML Scene Graph GLSL fragment shader tutorial

2011-03-04T11:33:50Z

I had been hearing a lot about Scene Graph and Qt Quick 2 making it possible to implement whatever effects one wanted, to really make apps gorgeous, and still stay in the la-la-land of QML UI coding. What I hadn’t had was a chance to try it out myself, until recently that is. It’s been the most fun I remember having in a while, coding-wise.

As usual though, documentation is a bit lacking. The effects are implemented using OpenGL Shader Language (GLSL), and googling does reveal some tutorials for that, but still the jump from those mainly C/C++ OpenGL -oriented write-ups to Qt Quick 2 isn’t altogether straightforward. The docs that I found most helpful were:

So, having studied these and banged my head against the wall, I was pretty excited when I got my first fragment shaders working on both Ubuntu and the N900. I then decided that I’d do a tutorial-style write-up to maybe make it easier for other people to get started. This is not a real GLSL tutorial because I’m really not qualified to write something like that. I’m aiming to tell you what I know, and give tips on how to apply other GLSL tutorials to a Qt Quick context. Here’s a short-n-shaky video of what I managed and what this tutorial will produce:

Qt Quick 2 Scenegraph + GLSL fragment shaders from Ilkka Laukkanen on Vimeo.

Building Qt with Scene Graph

To get all the scene graph goodness working, you need to build a special version of Qt. There are two ways to accomplish this that I know of:

Qt master + scene graph master (as documented in Zchydem’s post)
The qml-team/qtquick2 branch from the Qt staging repo.

I’d actually recommend the second method because it’s a simpler, almost-one-step process, and because it’s the method I know will work for the N900. So go ahead and clone, checkout and build:

$ git clone git://gitorious.org/qt/staging.git
$ cd staging
$ git checkout -b qtquick2 origin/qml-team/qtquick2
$ ./configure -developer-build -opensource -confirm-license -fast \
     -nomake demos -nomake examples \
     && nice make -j5

I guess you don’t need -developer-build if going this route, but using it is a habit I’ve gotten into, because you never know when you might need to build something that uses those private headers. Also, you might want to scale the number of make jobs according to your hardware.

When that completes, you will find the qmlscene binary in staging/bin. It’s a viewer for Scene Graph QML files, pretty much comparable to qmlviewer.

Running Qt Quick 2 files

I ran into a strange problem on my Ubuntu machine, where having the LANG environment variable set would cause the shaders not to work correctly. I therefore suggest you run everything in a sanitized environment (here QTDIR is the staging directory from the build instructions above):

$ env -i LD_LIBRARY_PATH=$QTDIR/lib DISPLAY=:0 $QTDIR/bin/qmlscene \
    whatever.qml

First steps in Qt Quick 2

For writing the actual code, I built the master branch of Qt Creator. It does have syntax highlighting for GLSL, but apparently that only works when putting the shaders in separate files, and that’s not supported for plain QML UI projects. I didn’t get any further into that yet, but will surely explore further later. For now, I just put the shaders inline into my QML. But I’m getting ahead of myself.

Writing Qt Quick 2 code is pretty much exactly as writing Qt Quick 1 code, unsurprisingly. Of course we import QtQuick 2.0 in the beginning instead of QtQuick 1.x, but other than that everything looks the same. Let’s go ahead and create a new “Qt Quick UI” project. What we get is this:

import Qt Quick 1.0

Rectangle {
    width: 360
    height: 360
    Text {
        anchors.centerIn: parent
        text: "Hello World"
    }
    MouseArea {
        anchors.fill: parent
        onClicked: {
            Qt.quit();
        }
    }
}

Change the import to version 2.0, run it using qmlscene and it should work as expected. Now the fun parts begin.

Remove the Text element, get a nice fun square-ish PNG image from somewhere, and add Image, Item and Text elements like this:

import QtQuick 2.0

Rectangle {
    width: 512
    height: 512

    MouseArea {
        anchors.fill: parent
        onClicked: {
            Qt.quit();
        }
    }

    Image {
        id: kitty
        source: "kitty.png"
    }

    Item {
        id: scrollercontainer
        anchors.fill: parent

        Text {
            id: scroller
            text: "Hello QtQuick world!"
            color: "white"
            font.pixelSize: 80
            anchors.verticalCenter: parent.verticalCenter
        }
    }
}

Still pretty basic stuff: we’ve got our Image and Item as children of the main Rectangle. The reason why the Text is inside the Item rather than as a direct child of the root is that we’re going to use the use the Item to clip the possibly long text to a manageable size, because a long text possibly wouldn’t fit into a texture.

One interesting thing here is that the Z-ordering or placement of the items doesn’t really have much bearing on how the end result is going to look, apart from the size determining the effective size of the textures we’re going to animate in the end. So, bigger items → more detailed textures.

Sourcery

Next, let’s add ShaderEffectSource items for both the image and the text. Note that we use the scrollercontainer element as the source for the scroller shader, not the actual text element. This is the clipping bit.

    ShaderEffectSource {
        id: kitty_source
        sourceItem: kitty
    }

    ShaderEffectSource {
        id: scroller_source
        sourceItem: scrollercontainer
    }

As far as I can tell, a ShaderEffectSource is an invisible item that acts as a binding between a shader effect (that is, a ShaderEffectItem) and the source element. Pretty much like a normal Qt Quick Binding element.

The first effect

Okay, sources are set up so now let’s set up the first effect. For that we’re going to create a ShaderEffectItem. It is a visible item and it will be actual visual representation of the original, now-invisible image. We’re going to make this the background trippy kitty from the video at the top, so we’re going to set it to fill the parent. The real work is being done by the GLSL code, i.e. the contents of the fragmentShader property:

    ShaderEffectItem {
        id: kitty_effect
        anchors.fill: parent

        property variant source : kitty_source

        fragmentShader: "
        varying highp vec2 qt_TexCoord0;
        uniform lowp sampler2D source;
        void main() {
            gl_FragColor = texture2D(source, qt_TexCoord0);
        }"
    }

The shader above is a simple pass-through operation, in that it displays the source image unaltered. A bit like a C program, the actual work is done in the main() function. What the shader effectively does is set the color of a “fragment” (i.e. pixel, I guess), and it performs this feat by giving a value to the gl_FragColor variable. gl_FragColor is a four-component vector in RGBA format, with all the components ranging from 0 to 1.

To display the original image, the filter needs to get the corresponding pixel value from the source image. It does this by using the texture2D() function. The function takes two parameters: the texture and a two-component vector, again with both components between 0 and 1, signifying the (x, y) point in the texture where to get the value from.

Before the main() function is a couple of declarations. The declarations in GLSL have four parts: the first keyword is a storage qualifier, the second is a precision qualifier, the third is the data type and fourth is the name. Here I have to admit that the storage qualifiers are pretty much a mystery to me: as far as I understand, varying is for communication between vertex and fragment shaders, and uniform is for values that stay the same over a primitive. The qualifiers here are basically just copied from other examples and tutorials, but these seem to work. The precision qualifiers are only for OpenGL ES, but I couldn’t find any information on them in the GLES 2.x specification. What I did notice is that the runtime I built for N900 requires them for everything, or the shaders don’t work. I guess they might have some performance impact but haven’t explored it.

Both of the declarations act as an input to the shader. qt_TexCoord0 is apparently provided by the scene graph runtime, and is, unsurprisingly, the input texture coordinate. source is for getting pixels from the source item. This works by way of having a property by the same name in the ShaderEffectItem. That property has type variant and gets the correct ShaderEffectSource as its default value. This is how QML communicates with the shader.

All right, that works but isn’t interesting. But since we’re setting all components of the output color, we can do whatever we want to the original pixels or even replace them. So let’s do something more:

Some motion

    ShaderEffectItem {
        id: kitty_effect
        anchors.fill: parent

        property real angle : 0.0
        PropertyAnimation on angle {
            to: 360.0
            duration: 800
            loops: Animation.Infinite
        }

        property variant source : kitty_source

        fragmentShader: "
        varying highp vec2 qt_TexCoord0;
        uniform lowp sampler2D source;
        uniform highp float angle;
        void main() {
            highp float texAngle = 0.0;
            if (qt_TexCoord0.x != 0.0 || qt_TexCoord0.y != 0.0) {
                texAngle = atan(qt_TexCoord0.y - 0.5, qt_TexCoord0.x - 0.5);
            }
            gl_FragColor = vec4(sin(texAngle + radians(angle)),
                                sin(texAngle + radians(angle - 120.0)),
                                sin(texAngle + radians(angle - 240.0)),
                                1.0)
                           * texture2D(source, qt_TexCoord0);
        }"
    }

We’ve added a few things here. The first is an animated angle property that runs through a full circle in degrees and a corresponding float variable declaration in the shader. We’re using the angle to create a color spinner that we’re going to overlay on the source image.

To calculate the correct color to overlay on any given pixel, we note that it depends on the angle of an imagined line drawn from the center of the image to the pixel. This is simply a conversion from Cartesian to polar coordinates, which is accomplished with an arc tan operation. Because the source texture coordinates range from 0 to 1, however, the resultant angles will be from 0 to 90. To get angles that go all around, we shift the coordinates by -0.5.

Then we compose the value for gl_FragColor with a multiplication operation: we multiply the original texture pixel with a four-component vector created with the vec4 function. Because we want the color components to change gradually, we take their values from a sine function of the angle we calculated earlier. We apply different offsets (120 and 240 degrees) to the different components so that we get different colors at different directions, and we also apply the animated offset from the QML angle property. Note that sin takes a radian input, so we convert from degrees to radians. Finally, while many 3D coding tutorials advocate a coding style where the decimal point and zero are always written even when not necessary, in shaders it seems to be required whenever dealing in floating-point numbers.

These enhancements result in a fun little primary color spinner overlaid on the source image.

Increase the trippy

I want more trippy. To increase the trippy, I want to twist up the spinner so it’s hypnotic!

    ShaderEffectItem {
        ...
        fragmentShader: "
        ...
        void main() {
            highp float texAngle = 0.0;
            if (qt_TexCoord0.x != 0.0 || qt_TexCoord0.y != 0.0) {
                texAngle = atan(qt_TexCoord0.y - 0.5, qt_TexCoord0.x - 0.5);
            }
            highp float skew = sqrt(pow(qt_TexCoord0.x - 0.5, 2.0)
                                    + pow(qt_TexCoord0.y - 0.5, 2.0))
                               * 10.0;
            highp vec4 colorwheel = vec4(sin(texAngle + radians(angle) - skew),
                                   sin(texAngle + radians(angle - 120.0) - skew),
                                   sin(texAngle + radians(angle - 240.0) - skew),
                                   1.0);
            highp vec4 texpixel = texture2D(source, qt_TexCoord0);
            gl_FragColor = colorwheel * texpixel;
        }"
    }

Roight, the skew factor, or how far we twist up the spinner is relative to the distance from the center of the spinner, so we calculate the value of skew with Pythagoras’ formula, with a suitable constant multiplier to get the effect we want. We apply the skew as an offset to the sine function argument in the colorwheel vector, and thus it acts as a delay that increases linearly as the distance from the center of the item increases.

Finally for extra trippy let’s make the background wavy! This is easy-peasy at this point, all we need to do is throw the texture source coordinates through sine and cosine functions before using texture2D to get the original image pixel:

    ShaderEffectItem {
        ...
        fragmentShader: "
        ...
        void main() {
            ...
            highp float wavefactor = 0.03;
            highp float wave_x = qt_TexCoord0.x + wavefactor
                           * sin(radians(angle + qt_TexCoord0.x * 360.0));
            highp float wave_y = qt_TexCoord0.y + wavefactor
                           * cos(radians(angle + qt_TexCoord0.y * 360.0));
            highp vec4 texpixel = texture2D(source, vec2(wave_x, wave_y));
            gl_FragColor = colorwheel * texpixel;
        }"
    }

The scroller

That’s pretty trippy. All it needs now is – a scrolling message. That’s how I feel at least. So let’s add some text to the scroller element and make it scroll.

        Text {
            id: scroller
            text: "Hello QtQuick world! Hello QtQuick world! " +
		"Hello QtQuick world! Hello QtQuick world! " +
		"Hello QtQuick world! Hello QtQuick world! " +
		"Hello QtQuick world! Hello QtQuick world! "
            color: "white"
            font.pixelSize: 80
            anchors.verticalCenter: parent.verticalCenter
            PropertyAnimation on x {
                from: scrollercontainer.width
                to: -scroller.width
                duration: 800 * (scroller.width / 100)
                loops: Animation.Infinite
            }
        }

Multi-line string literals are a bit clunky without support for here-doc syntax, but this way we get a suitably long string to test the idea. A bog-standard PropertyAnimation moves the text from right to left across the parent element, with a speed that’s dependent on the width of the string itself, so that it stays constant with varying strings. Trippy and reminiscent of old-school demo stuff, I like.

All done for now

That’s it for now. I’ll try to find the time to keep working on this stuff a bit more. The need to build the qtquick2 staging branch is a bit of a hindrance since it seems it’s impossible to build it on OS X versions prior to the upcoming Lion, due to a OpenGL version issue. Since my workhorse home computer is a Mac, this puts a bit of a crimp on toying with scenegraph.

Migrating a live Ubuntu install onto LVM2

2011-02-27T11:56:40Z

My only home computer is, and has been for a while, a Nokia Booklet. It’s a nice ultra-portable fanless laptop that gets about 6-7 hours of use per charge, and is in my opinion one of the most stylish products ever shipped by Nokia. Granted, it’s not too powerful, only sporting a dual-core Intel Atom Z530 running at 1.6 GHz, and it has one major flaw from a Linux user’s standpoint in that it has the almost infamous GMA500 graphics hardware courtesy of the “Poulsbo” chipset. Support nowadays is okay, in that I run Maverick on it at native panel resolution, and can actually pretty consistently put the machine to sleep with uswsusp and have it wake up correctly, but I can’t for example run the snazzy new gnome-shell on it, or use compiz.

Anyway, I used to have the original Windows 7 install on a second partition, but it soon became apparent that I would find no use for it and decided to use the whole disk for Ubuntu. I didn’t want to reinstall, however, and unfortunately hadn’t originally put it on LVM, so the exercise would be a bit more substantial than simply growing a logical volume.

This, then, was my plan of action:

Nuke the windows partition, make a LVM partition in it’s place, and create a volume group and logical volume with it, then create a new ext4 fs on the logical volume.
Copy over the running Ubuntu installation onto the new filesystem.
Update grub to locate the new-old Ubuntu on the LVM filesystem.
Boot from the logical volume.
Add the old Ubuntu partition to the volume group and grow the logical volume and filesystem onto that.

Simple! There are a couple of glaring holes in the plan but we’ll handle those as we get to ‘em.

Nuke ‘em high

The first step was simple: first, just delete the windows partition and create a new partition in its place with the partition type set to 0x8e for Linux LVM. Then we need to tell LVM about this new partition, or “physical volume” in LVM terminology, by running pvcreate on it (I’m running all these commands as root):


# pvcreate /dev/sda1

Then we create a volume group, which is a container of physical volumes, and itself acts as a container for logical volumes:


# vgcreate vg-0 /dev/sda1

Finally, create a logical volume that fills up the new volume group:


# lvcreate -n root -l 100%VG vg-0

Now there’s a brand spanking new logical volume called root on the volume group vg-0. As evidence of this, there is a new device node for it at /dev/vg-0/root. Next, create the filesystem:


# mkfs.ext4 /dev/vg-0/root

Replica

Now the new filesystem can be mounted and the running installation copied onto it. There might be some cases where copying from a live fs might be a bad idea, but generally as long as you don’t do anything substantial at the same time, you shouldn’t have too much trouble.

I used rsync to do the actual copying, but any similar tool could be used as long as it correctly preserves all file attributes. What’s just as important is to skip all the special pseudofilesystems and the mountpoint itself:


# mount /dev/vg-0/root /mnt
# rsync -a / --exclude=/{dev,proc,sys,var/run,var/lock} /mnt | \
  tee transfer.log; \
  ogg123 /usr/share/sounds/ubuntu/stereo/system-ready.ogg

I used tee to capture the output from rsync and save it in a logfile, and made it play a sound when it finished so I wouldn’t have to keep checking. It’s always a good idea to do a dry run first, and rsync has the -n flag for that. For me the command complained about not being able to copy all file attributes, but even after trawling the verbose logs I couldn’t find any specific file-related errors, so I decided to ignore that message and went ahead with the real copy. All told the process took a couple of hours for 20+ GB.

Rub-a-dub-dub

I haven’t played with grub a whole lot, but after looking at how the thing was configured on Ubuntu, I tentatively came to the conclusion that grub (version 2 that is) should find the copied Ubuntu from the logical volume and add it to the boot menu automagically. So I updated grub.cfg:


# update-grub

The output hinted at what a look at /boot/grub/grub.cfg confirmed, i.e. that grub had indeed found the copy. However, I happened to notice that the root settings for the LVM kernels were incorrect, and had the UUIDs of my old root partition. I didn’t think much of it, expecting them to be set correctly as I ran update-grub from the new partition, and simply fixed them manually.

At this point I rebooted, selected the latest kernel from the logical volume and found Ubuntu running nicely. I used fdisk to delete the old root partition and created a new LVM partition in its place, and shutdown the machine, deciding to call it a day.

This was a near-catastrophic error, as it turns out. Well, catastrophic in a relative sense.

Disco inferno

Booting in the morning, I was greeted by a friendly grub message saying “ERROR: no such disk”, and a grub-rescue> prompt that didn’t know a single command, not even help.

After thinking about it for a few moments, my error was glaringly obvious: grub on the MBR was still looking for it’s config file from the old root partition, which I had nuked, because I had never reinstalled it, only recreated the config file. And even if I had reinstalled it, it wouldn’t have booted, because I had only updated the grub.cfg file on the old root, not on the new one. Facepalm.

Well, thankfully I happened to have another Ubuntu machine available, so I quickly created a bootable USB stick and started fixing things. The default Ubuntu live image doesn’t have the userspace LVM tools, but as long as there’s network connectivity, one can just install them on the running ramdisk, so that’s no problem.

The first order of business was to get the new root mounted using the live image:


# apt-get install -y lvm2
# vgchange -a y vg-0
# mount /dev/vg-0/root /mnt

The vgchange command makes the volume group active, this is required so that the logical volume can be mounted. Now the mounted root has all the tools required to reinstall and reconfigure grub, so the easiest method is to chroot into the mount and fix it from there. For this to work, the most important pseudofilesystems need to be bind-mounted into the chroot:


# for d in dev/pts dev proc sys; do mount -B /$d /mnt/$d; done

Now we can chroot into the volume, reinstall grub and rebuild the configuration:


# chroot /mnt
# grub-install /dev/sda
# update-grub

Easy peasy. Now for a reboot…

…and the same error message.

I did the same boot from USB stick, install lvm, mount root rumba again and started scratching my head and looking at files. In the end it was a simple comment on some forum somewhere that tipped me off as to what the actual problem was. I forget where (sorry) but someone said that the format for the parameter for the set root command of grub was the same as the volume node name in the /dev/mapper directory. Now, update-grub had put entries like set root='(vg-0-root)' in the grub.cfg, but actually the dash in the volume group name gets kinda-sorta escaped, so the node is in fact named /dev/mapper/vg--0-root. It appears there is a bug in update-grub that causes this behavior.

As manually fixing the file would have just been a one-off solution, I renamed the volume group (which was thankfully very easy):


# umount /mnt
# vgchange -a n vg-0
# vgrename vg-0 vg0
# vgchange -a y vg0
# mount /dev/vg0/root /mnt

The volume group needs to be deactivated to rename it, but other than that it’s straightforward. Now I grub-installed and update-grub‘d again and confirmed that this time it generated correct config files. Time to wipe off that cold sweat and lean back.

Grow your own

All that then remained was to add the old root’s physical volume to the volume group, then use up that space. For this, you first need to initialize the partition for LVM, then extend the volume group onto it, then grow the logical volume, and finally grow the filesystem to match the new volume size. Sounds complicated but is just a few simple steps really (/dev/sda6 is the old root partition here):


# pvcreate /dev/sda6
# vgextend vg0 /dev/sda6
# lvextend -l 100%VG vg0/root
# resize2fs /dev/vg0/root

And that’s it! One reboot later I’m here typing this and feeling a bit smug for shooting myself in the leg, then managing to bandage it up without doing any more damage in the process.

Jitsu is a meta build system for Ninja

2011-02-13T20:19:50Z

A while back I hit upon Ninja, a very fast and very simple build system that is apparently used for building Chromium. Can’t remember where I first heard of it, might’ve been Hacker News or just somebody’s tweet, but having been interested in build systems for years, at least since setting up Linux builds from scratch for a big piece of software while working on my M.Sc. thesis, I immediately started playing around with it.

Ninja is indeed fast, but by design the buildfiles are hard to write. Ninja sources include a patch to Gyp, a higher-level build system also somehow (not sure how exactly) related to Chromium, but the support wasn’t very complete, and somehow I just didn’t find Gyp appealing. I therefore did what anyone in my circumstances would’ve done, and started writing a “meta build system” myself.

It’s called Jitsu, and since I really like Ruby, that’s what I wrote it in. It’s at version 0.1.2 now and available from rubygems.org, and as of now it can build (or rather, generate ninja buildfiles for) C++ binaries and static libraries. Builds are specified as build.jitsu files in YAML format, which the tool processes into build.ninja files that can then be used by Ninja. I’ll keep trying to use it in my own projects so hopefully it’ll keep on improving from here on out. Let’s see where it will go, I haven’t really delineated the scope yet, to e.g. just doing C++ builds.

If you are interested, install the gem, maybe grab the source and give it a spin. Let me know what you think. Also see the documentation.