One of the greatest things about most Linux distributions is the availability of pre-compiled software packages that you can, usually, through some very quick commands, download, install and get up and running in very short order. If you are old enough to remember “dependency hell,” you’ll nod sagely in recalling the associated frustration of your software application not being able to access the additional programming it requires to function. If you are too young to really know about that, be glad that the dependency solver tends to sort the long list of things you may need to make a package work. However, installing a piece of software on a Linux distro, a seemingly innocuous process, is not as easy as you think.
Chasing after the easy install
Where this daunting situation starts, as most things with open source, is with someone who wants to install a piece of software. Maybe they intend to deploy it on many machines, maybe they want a reproducible build, or maybe the build is just weird and esoteric enough that getting it to work is cumbersome, long and complex and they don’t want everyone else to suffer through that pain. (On a side note, huge shout out to Tom “Spot” Callaway for the plethora of kinds of packages he maintains.) The reasoning may vary, but the end goal is the same: They want an easy-to-install installable for their chosen distro.
So this someone first tries their package manager and with dismay, finds that the package isn’t available. Here, they have a few options. 1) They can compile it themselves and get it working (which may lead to a few other things needing to be compiled and installed), or 2) they can choose to take the plunge and properly package the software.
Usually, the packaging process is fairly straightforward. With most software you figure out the dependencies for both building and installing, make note of them in the package specification, run make && make install and then list what files should be installed. If everything goes well, out pops a package you can install! (Note: run make && make install is a bit of an oversimplification, depending on your packaging standard. It’ll usually involve various wrapper calls to set up different variables for the build, so it’s standard for that distro).
Fun fact: It’s never quite that easy.
The tricks of the trade
Packaging has a well-earned reputation for being cumbersome, thankless and finicky. Building code in a reasonably repeatable manner on a variety of architectures, and even potentially distributions, is fundamentally hard. If you think about this as a combinatorics problem, you have one package by A architectures by D distros by B build flags. Obviously, that can get complicated fast, and I’m already ignoring gcc vs. clang, different versions of libraries, etc. The package maker ends up in a cycle of slowly ironing out build issues until they find the overall incantation to make everything work. Sometimes they have to turn off parallel building or add additional linker libraries or remove some. Sometimes they need an entire complex script that figures out some weird issue and passes it into the build or it’s just a matter of changing a file’s permission. Needless to say, it can take a while, and sometimes even involves creating custom patches for the code base that the upstream project may not be interested in maintaining. A packager’s job is part build engineer, part programmer and a lot of stubborn persistence to get to the other side with a final package.
Containers, packages, whaaaa?
Yet, I can hear you all nearly saying it: Packaging is hard, so why not make it easier on ourselves and just containerize it? Time for another fun fact: Containers are packaging! Containers are packaging that’s somewhat freed, at least for now, from the more stringent constraints that a single distribution puts on it. This makes a whole lot of things easier and faster to do, but it can also lead to far more complicated systems to make these builds work for the long term.
Whatever the path on packaging you choose, take a moment to think about all the individuals and teams that sit down and make these packages possible. There are entire infrastructure teams dedicated to making systems to build packages for distros — individuals who do the legwork to make a package build and the folks who stick with the package and maintain it for the long haul. Packaging may be hard, but it’s also something we all want and enjoy having, so next time you see someone who’s doing the work on packaging, let them know you appreciate their efforts!
Stay tuned to the Open Source Blog and follow us on Twitter for more deep dives into the world of open source contributing.
