Packaging a Qt app for OS X

I participated in BarCamp Tampere 2 recently, and one of the many very interesting presentations was Ville Ranki talking about Siilihai, a web forum reader app intentionally very reminiscent of newsreaders of old. Siilihai scrapes the content from web forums and presents it as threaded conversations so that it’s easier to read and follow conversations. Parsers can be written pretty easily for most forum software, and while forum content is only accessed locally by the client, it stores generated message IDs in the siilihai.com service so that whatever messages you read in one client are marked read in any other clients you might use as well. The service is also used as a repository for forum parsers.

Anyway, as the event was held at our office, I grabbed a Mac that was handy so that I could try out Siilihai for myself. Getting it built was simple as it is written using Qt, but I wanted to create a distributable package (an app bundle in OS X parlance) so that all the other beret wearers could enjoy it too. This, however, turned out to be a bit more involved than I’d thought.

The anatomy of an app

An app bundle is pretty self-contained. Apart from basic system libraries that can be assumed to be available on every Mac, apps usually ship all their libraries inside the app bundle together with the runnable binary. Some space may be wasted this way if several apps bundle the same libraries, but generally it seems pretty useful. At least it’s simple to deploy applications when they can simply be copied into place.

Siilihai uses a couple of shared libraries of its own, and naturally Qt. Since Qt is not installed by default in OS X, I had to include all of these in the app bundle. Bundling Qt is relatively simple using the macdeployqt tool included in the Qt SDK, but it only really takes care of the whole process if the application is a single binary that is only dependent on Qt. Siilihai’s custom libraries required a bit more love.

Rewiring and rewriting

For an app that uses dynamic libraries to work, the runtime linker must be able to find the libraries when the app is run. In Unix-style OSes this is generally accomplished by installing the libraries into standard paths such as /usr/lib. The dynamic linker is preconfigured to look in these directories when it is looking for library files.

If the libraries are for some reason installed into a nonstandard path, that path can also be configured as an “rpath” in the app binary itself. This is just another way of telling the dynamic linker where to look, but in a per-app fashion.

These concepts are valid for OS X .dylib files as well, with some differences. Using the command line utility otool we can examine this search path information. Let’s see the output for otool itself:

$ otool -L `which otool`
/usr/bin/otool:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
        version 159.0.0)

We can see that otool depends on a single dylib, libSystem. Now, my knowledge isn’t broad enough to tell if the actual path to the lib is hardcoded in the binary, or if the output is due to the dynamic linker knowing to look in /usr/lib, but in the context of this discussion it doesn’t really matter. Let’s look at the output for libSystem.B.dylib next:

$ otool -L /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
        version 159.0.0)
        /usr/lib/system/libcache.dylib (compatibility version 1.0.0, current
        version 47.0.0)
        /usr/lib/system/libcommonCrypto.dylib (compatibility version 1.0.0,
        current version 55010.0.0)
        /usr/lib/system/libcompiler_rt.dylib (compatibility version 1.0.0,
        current version 6.0.0)
        /usr/lib/system/libcopyfile.dylib (compatibility version 1.0.0,
        current version 85.1.0)
        ...

We see that libSystem is linked against a whole bunch of other system libraries, but also that the otool output lists the library itself. As far as I can tell, this is because the library knows its own path, something that needs to be taken into account later. If you know better, please leave a comment.

The consequence is that because I was going to put siilihai’s libraries in a nonstandard place (inside the app bundle), I need to edit the library locations stored in the app binary so that the libraries get found by the linker. Siilihai’s libraries also link to each other, so I have to do the same thing to them too.

If this all sounds complicated, that is because it is. In software it soon pays to automate even the simplest of tasks, and here there is definitely a lot to gain, so I wrote a script to build Siilihai and perform all these magic tricks for me.

Scripture

You can see the whole script in github, but I’ll explain the most relevant bits here.

First, the script tries to be clever about allowing you to run it from whatever directory and still finding all the sources and whatnot. For this it first locates the script directory:

SCRIPT=$(python -c 'import os,sys;print os.path.realpath(sys.argv[1])' $0)
SCRIPTDIR=$(dirname $SCRIPT)

Using GNU readlink, the first assignment would be the much simpler SCRIPT=$(readlink -f $0), but it doesn’t, hence the Python. Anyway, now that the script knows where it is, by extension it also knows where the Siilihai client source is. It assumes that libsiilihai’s sources are in the same directory.

The script creates a temporary directory and proceeds to do normal shadow builds of the lib and the client. The lib build generates exclusively .dylib files while the client generates both .dylibs and an app bundle. The next step is to copy all the .dylibs into the app bundle. It doesn’t really matter where, but I decided to put them in the /Contents/Frameworks directory since I happen to know that’s where the Qt libs will end up too.

status "Copying libsiilihai dylibs into client bundle"
BUNDLEDIR=$CLIENTBUILDDIR/src/reader/siilihai.app
FWKDIR=$BUNDLEDIR/Contents/Frameworks
mkdir -p "$FWKDIR"
for LIB in "$LIBBUILDDIR"/src/*.dylib "$CLIENTBUILDDIR"/src/common/*.dylib \
        "$CLIENTBUILDDIR"/src/parsermaker/*.dylib; do
        cp -a "$LIB" "$FWKDIR"

Now we use install_name_tool to set the ID of each library after it is copied:

LIBBASENAME=$(basename $LIB)
LIB=$FWKDIR/$LIBBASENAME
install_name_tool -id "@executable_path/../Frameworks/$LIBBASENAME" \
        "$LIB"

We set the ID to be the relative path inside the app bundle from the executable binary (which resides in /Contents/MacOSX) to the library. This is the same rpath that will be set in the executable later.

Since all the Siilihai libraries are linked against Qt, and the Qt frameworks will be deployed into the app bundle too, we need to change their rpaths in the libraries. At this point they still point at e.g. the Qt SDK used to build the libs. We grep the framework names from otool output, generate new paths for them and change the paths in the libs:

QTDEPS=$(otool -L $LIB|egrep 'Qt.*\.framework'| \
        egrep -v '@executable_path'|awk '{print $1}'|tr '\n' ' ')
for DEP in $(echo $QTDEPS); do
        NEWDEP=$(echo $DEP|sed 's%.*gcc/lib/%%')
        install_name_tool -change "$DEP" \
                "@executable_path/../Frameworks/$NEWDEP" "$LIB"
done

Note that we ignore any Qt frameworks that might already have working paths with the egrep -v '@executable_path' bit. Now when the Qt frameworks get copied into /Contents/Frameworks, the Siilihai libs will get linked against them at runtime. The libs are dependent on each other too, though, so we need another round of dependency path fixing, this time picking anything with libsiilihai in the name:

DEPS=$(otool -L $LIB|egrep 'libsiilihai'|egrep -v '@executable_path' \
        |awk '{print $1}'|tr '\n' ' ')
for DEP in $(echo $DEPS); do
        install_name_tool -change "$DEP" \
                "@executable_path/../Frameworks/$DEP" "$LIB"
done

That concludes the library ID and path fixing loop. Now the only thing remaining is to run the macdeployqt tool. This tool finds all the Qt frameworks the executable is linked against and copies them into the bundle. It also strips them and fixes all the IDs and rpaths. As an additional bonus, since we’ve already copied Siilihai’s own .dylibs into the bundle, it also fixes their paths in the executable. The result is a self-contained app bundle that can be run in any reasonably modern Mac. Currently I believe the bundle is compatible with Intel Macs running OS X 10.5 or later.

Further development

One rather major problem with the script so far is that it leaves the app bundle using the default app icon, while it should use the Siilihai icon from the repo. This is something I’ll have to find out how to do. Other than that, the only thing I’ve thought of is support for tagging the repos when successful builds are made, and settings some sort of version metadata, should bundles support something like that.

The experience does leave me with the feeling that if I was to write an app that I wanted to deploy onto Macs, I would just link everything into a monolithic binary. Given today’s storage space yadda yadda etc., you get the point.