jro.codegrinder.com

I, recently, decided to move some of my home rails sites to my aging PPC-based mac mini. I’m still running Tiger on it because I’m lazy, don’t need any new features, and know how macports & fink work in Tiger. I ran through the normal, install fink, ruby package,… routine. But, when I got to compiling some of my gems, I got this error:

/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: flag: -undefined dynamic_lookup can't be used with MACOSX_DEPLOYMENT_TARGET environment variable set to: 10.1

Which seems pretty strange since I’m obviously running on 10.4. And, while I don’t know why the system is getting it wrong, a quick work around is to set the environment variable yourself.

export MACOSX_DEPLOYMENT_TARGET=10.4

Oddly, it doesn’t happen on my intel macs, so I suspect that there’s a bug somewhere in the PPC fink stack that just doesn’t effect all compilations. /shrug.

Better late than never. This is the first “real” article in the Packaging Software series. It’s long, but hopefully it’ll help someone out. Enjoy!

So what’s an RPM?

In it’s distilled form, a big ball of files that hopefully make up something useful, like software. But wait, there’s more! It’s not just a file format for installing, uninstalling, updating, verifying software, it’s also the system for creating those bundles and doing all the installing, etc. Confused? What we have is two parts: The files commonly referred to as RPM, aka the software bundles, and rpm set of commands for creating and managing those bundles.

History and the Soap Opera

RPM was birthed by Red Hat as the Red Hat Package Manager, later becoming the RPM Package Manager as it became building block of more and more non-Red Hat Linux distributions. It’s grown to be used in innumerable Linux distros, large and small, popular and not.

I don’t want to get the drama wrong, so I’m just going to quote Wikipedia

As of the 31st of May, 2007, there are two versions of RPM in development. The first is led by long-time (since 1999) RPM maintainer, Jeff Johnson (‘JBJ’), who continued his development efforts after departing Red Hat. It has combined recently with efforts of OpenPKG and added a new website rpm5 tine of the fork. The ‘JBJ’ variant is used by Mandriva, PLD Linux Distribution and supported by OpenPKG. Jeff Johnson formerly worked for Red Hat before a controversy.

The other effort is led by Red Hat recent hire (2007) staff member Panu Matilainen. The Red Hat RPM tine of the fork which issued its first major code revision in July 2007, is used by Fedora, Red Hat Enterprise Linux, Novell’s openSUSE and CentOS. Matilainen has also worked on an apt-like front end for package management for RPM previously; apt-get is a Debian Linux management tool.

The displaced former RPM website content is archived by its long time maintainer at oldrpm web archive. On December 14, 2006, Red Hat decided to take complete control of editorial content at the formerly community maintained website which content was maintainted in an ‘open to the community’ process manner.

Does it matter to you? Probably not, just be aware there are currently two major forks of RPM and there may be slight differences in the API standards or tools that are available.

The simple things

So let’s do something useful. I’m working on a Red Hat system, so if you’re not it’s possible things will be a little different.

What do we have installed?

List them all

Good for grep’ing through when you don’t know what you’re looking for.

[jro@localhost ~]$ rpm -qa 
libgcc-3.4.6-8.i386
filesystem-2.3.0-1.i386
glibc-common-2.3.4-2.36.i386
bzip2-libs-1.0.2-13.EL4.3.i386
...

Find a specific package

Good when you know it’s there, but just need the version.

[jro@localhost ~]$ rpm -q rmt
rmt-0.4b39-3.EL4.2.i386

Find out what RPM owns a file

Good for.. where the hell did that come from?

[jro@localhost ~]$ rpm -qf /etc/snmp/snmpd.conf
net-snmp-5.1.2-11.EL4.10.i386

Get some basic info on an installed package.

[jro@localhost ~]$ rpm -qi hdparm
Name        : hdparm                       Relocations: (not relocatable)
Version     : 5.7                               Vendor: Red Hat, Inc.
Release     : 2                             Build Date: Tue 21 Sep 2004 05:23:51 AM CDT
Install Date: Wed 27 Jun 2007 04:10:31 PM CDT      Build Host: tweety.build.redhat.com
Group       : Applications/System           Source RPM: hdparm-5.7-2.src.rpm
Size        : 64075                            License: BSD
Signature   : DSA/SHA1, Wed 05 Jan 2005 05:18:55 PM CST, Key ID 219180cddb42a60e
Packager    : Red Hat, Inc. 
Summary     : A utility for displaying and/or setting hard disk parameters.
Description :
Hdparm is a useful system utility for setting (E)IDE hard drive
parameters. For example, hdparm can be used to tweak hard drive
performance and to spin down hard drives for power conservation.

List all the files installed as part of a package

Good for where the hell did that config file get put?

[jro@localhost ~]$ rpm -ql rmt
/etc/rmt
/sbin/rmt
/usr/share/man/man8/rmt.8.gz

Installing and Upgrading RPMs

Installing a single RPM

[jro@localhost ~]$ rpm -i nginx-0.5.31-1.vp.i386.rpm
error: can't create transaction lock on /var/lock/rpm/transaction

Hrmm.. maybe I need to do this as root

[jro@localhost ~]$ sudo rpm -i nginx-0.5.31-1.vp.i386.rpm
Password: *****
[jro@localhost ~]$

Yep. Not a lot of output tho

[jro@localhost ~]$ sudo rpm -iv nginx-0.5.31-0.vp.i386.rpm
Preparing packages for installation...
nginx-0.5.31-0.vp

Better.

[jro@localhost ~]$ sudo rpm -ivh nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [100%]

There we go.

Installing multiple packages at the same time

[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm
error: Failed dependencies:
        nginx = 0.5.31 is needed by nginx-contrib-0.5.31-0.vp.i386
[jro@localhost ~]$ sudo rpm -ivh nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [100%]
[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx-contrib          ########################################### [100%]

So our first package required another, and installing them one by one worked well enough but it’s a pain.

[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [ 50%]
   2:nginx-contrib          ########################################### [100%]

It’s way easier just to put them all together on one line. Or go c-razy and use file globbing ( rpm -ivh *.rpm ).

Upgrading RPMs

[jro@localhost ~]$ sudo rpm -Uvh nginx-contrib-0.5.31-1.vp.i386.rpm nginx-0.5.31-1.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [ 50%]
   2:nginx-contrib          ########################################### [100%]

It’s not too much different than installing, just that the -i turns into a -U. HOWEVER, you can also use the -U to install a package. So why bother using -i? It’s simple. Don’t. Always use -U, unless you specifically don’t want to upgrade a package you might have installed. It sounds silly, but you might end up in that situation one day if you have an RPM that restarts a service, etc.

Uninstalling.. er Erasing RPMs

[jro@localhost ~]$ sudo rpm -e nginx
Password:
error: Failed dependencies:
        nginx = 0.5.31 is needed by (installed) nginx-contrib-0.5.31-1.vp.i386
[jro@localhost ~]$ sudo rpm -e nginx-contrib nginx
[jro@localhost ~]$                                       

Pretty much what you’d expect at this point.

Useful Magic

See the architecture of a package

If you’re running a 64-bit machine, it’s VERY useful to know which packages are 32-bit and which are 64-bit. As on some distros it’s possible to have both versions installed, and then things start to get very confusing.

[jro@localhost ~]$ rpm -qa | head -5
libgcc-3.4.6-8
filesystem-2.3.0-1
glibc-common-2.3.4-2.36
bzip2-libs-1.0.2-13.EL4.3
ethtool-1.8-4
[jro@localhost ~]$ echo "%_query_all_fmt         %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros
[jro@localhost ~]$ rpm -qa | head -5
libgcc-3.4.6-8.i386
filesystem-2.3.0-1.i386
glibc-common-2.3.4-2.36.i386
bzip2-libs-1.0.2-13.EL4.3.i386
ethtool-1.8-4.i386

We’ll talk more about the rpmmacros files when we get to building packages, but I recommend adding this even if you’re not running a system with packages from differing architectures.

See the changelog for a package

Installed package

[jro@localhost ~]$ rpm -q --changelog nginx 
* Wed Sep 19 2007  0.5.31-1.vp

- add condrestart

* Wed Sep 19 2007  0.5.31-0.vp

- upgrade to latest stable

* Mon Sep 17 2007  0.5.26-3.vp
....

Package not yet installed

[jro@localhost ~]$ rpm -qp nginx-0.5.26-0.vp.i386.rpm --changelog
* Mon Jul 02 2007  0.5.26-0.vp

- new nginx version - 0.5.26
- start fork of package

* Wed May 23 2007  0.5.20-0

- new nginx version - 0.5.20

* Thu Apr 26 2007  0.5.19-0

Muck with the output format of RPM without making it semi-permanent

[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}\n" | head -3
libgcc
filesystem
glibc-common
[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}_%{VERSION}\n" | head -3
libgcc_3.4.6
filesystem_2.3.0
glibc-common_2.3.4
[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}_%{VERSION}_%{DISTRIBUTION}\n" | head -3
libgcc_3.4.6_Red Hat (RHEL-4)
filesystem_2.3.0_Red Hat Linux
glibc-common_2.3.4_Red Hat (RHEL-4)

It’s not super intuitive, but thankfully there’s some info in the rpm man page, and it can be handy when writing scripts to report on your packages, create lists to mirror systems from, etc.

Resources

• The RPM Guide at The Fedora Project - This used to be a book that’s no longer in print. This is the best resource outside of looking at what other people have done.
• The RPM HOWTO at The Linux Documentation Project - It’s a bit dated but there’s still some relevant information there.
• RPM.org wiki - Links to other resources.

I just submitted an nginx package to fink after waiting about a month to define which files were documentation. Hopefully, it’ll be validated and added quickly, but if you need it now head over to sourceforge to grab the .info and .patch. The packaging tutorial over at finkproject should have all the info you’d need to build it for yourself. As well as teach you how simple it is to get started working on packages for fink.

I should have really taken better notes as I got setup using fink, but here’s a braindump of what I needed for rails to work for me. Note, rmagick and me are not near as angry with each other as we were under macports.

Install Fink

Go Here - You should be able to figure out the install.

Set it to unstable

# Edit /sw/etc/fink.conf
# set Trees line
Trees: local/main stable/main stable/crypto unstable/main unstable/crypto

Edit Jan 21, 2008: You’ll probably want to do a fink update-all and/or fink -dv selfupdate here.

Install fink packages

$> sudo fink install mysql mysql-client  mysql15-dev mysql15-shlibs
....
$> sudo fink install ruby ruby18 ruby18-dev ruby18-shlibs rubygems-rb18
....

Edit Jan 21, 2008: If you want rmagick, you’ll need to fink install imagemagick or imagemagick-nox .. it will take a while

Make sure you’re using the fink ruby

You may need to open a new shell or update your $PATH but this what it’ll look like if it’s correct

jro@fireant:~/Desktop$ which ruby
/sw/bin/ruby

Time for some gems

$> sudo gem update --system
....
$> sudo gem update
....
$> sudo gem install mysql
....
$> sudo gem install -y rails
....
$> sudo gem install rmagick -- --disable-htmldoc
....

Edit Jan 21, 2008: You’ll probably want to setup daemonic to start mysql on system startup.

sudo daemonic enable mysql

Hopefully there aren’t too many typos/mistakes due to my poor notes. If you find any, throw up a comment and I’ll patch the post.

The only software I’m really missing right now in fink is nginx, but I’ll have more on that soon. :)

I’ve been a pretty happy user of MacPorts/DarwinPorts for the past couple years, and all was good. I could install all the UNIX goodness to run a rails stack and have most of the network/security apps I was used to from the Linux world. But, recently I stuck in a dynamic library situation with ImageMagick and the rmagick ruby gem that was causing me a lot of grief. So, I removed ports and my gems and started over, but still no dice.

In a perfect world, I’d have rolled back to the last working ImageMagick package and go on my happy way. Unfortunately, that doesn’t seem to be an option with MacPorts if you don’t have the old versions still compiled on your system. Of course, I didn’t because I naively decided to start from a blank slate (see above). Ultimately, I decided to try the out Fink before digging too deeply into how to fix the linking issue.

I’d used Fink for about a month when I first dipped my toe into the Mac waters, so I had some idea how the system worked. It’s based on some Debian tools like dpkg and apt-get, and offers both binary and source packages. Much like Debian, I had to immediately switch to unstable to get any sort of useful ruby install. Also, unless you can live in Candy Land and only need stable tree, I’d avoid using apt-get no matter how much your Debian/Ubuntu love makes you want to. The “fink” command line is really what you want.

And my favorite bit, is that I can rollback packages! Check it out:

jro@fireant:~$ fink dumpinfo -fallversions zsh
allversions:
 b      4.2.6-1001
        4.3.4-2

jro@fireant:~$ fink install zsh
.....compile compile compile.. this is unstable after all.....
jro@fireant:~$ fink dumpinfo -fallversions zsh
allversions:
 b      4.2.6-1001
 bi     4.3.4-2

jro@fireant:~$ fink install zsh-4.2.6-1001
The following package will be installed or updated:
 zsh
/sw/bin/apt-get-lockwait -q --ignore-breakage --download-only install zsh=4.2.6-1001
Reading Package Lists...
Building Dependency Tree...
The following packages will be DOWNGRADED
  zsh 
0 packages upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0  not upgraded.
Need to get 2087kB of archives. After unpacking 1524kB will be freed.
Do you want to continue? [Y/n] 
......
...... downgrade...install..yay!....
......

jro@fireant:~$ fink dumpinfo -fallversions zsh
allversions:
 bi     4.2.6-1001
 b      4.3.4-2

jro@fireant:~$ fink install zsh-4.3.4-2   
The following package will be installed or updated:
 zsh
Reading buildlock packages...
/sw/bin/dpkg-lockwait -i /sw/fink/dists/unstable/main/binary-darwin-i386/shells/zsh_4.3.4-2_darwin-i386.deb
(Reading database ... 30667 files and directories currently installed.)
Preparing to replace zsh 4.2.6-1001 (using .../zsh_4.3.4-2_darwin-i386.deb) ...
Unpacking replacement zsh ...
Setting up zsh (4.3.4-2) ...

jro@fireant:~$ fink dumpinfo -fallversions zsh
allversions:
 b      4.2.6-1001
 bi     4.3.4-2

From a mailing list entry by William Davis, ::

sudo rm -rf \
/opt/local \
/Applications/MacPorts \
/Library/Tcl/macports1.0 \
/Library/LaunchDaemons/org.macports.* \
/Library/StartupItems/DarwinPortsStartup

Why? Well if like me you run into some strange OS X dynamic linking issues and want to reinstall all your ports, or if you want to try out fink and make sure you don’t end up hitting your old ports applications.

You’ll often hear SysAdmins advocate writing a script for any task you expect to repeat more than once, and I couldn’t agree more. Over the years, I’ve often found myself taking it a step further, and packaging anything I expect to install more than once, or install in any production environment. There are a lot of benefits, but mostly I find the added server consistency and conveniences of the package management tools make up for any extra effort required in bundling it together.

Of course you could do the same thing with your own custom scripts, but I don’t like reinventing the wheel and I really like the extra integration of using the same system as my OS. So what makes a good packaging system? Here’s my list:

• Versioning
• Robust dependencies
• Able to upgrade and downgrade packages
• Pre-install, Post-install scripting
• Binary packages - my preference, definitely not a hard requirement

With that in mind, consider this the kick-off to a series on effectively building and mixing your own packages with your OS distribution of choice. I’ll be focusing on RPMs as the package type on RedHat / Fedora Linux distributions, but the patterns should work well with other RPM based distributions, and probably apply to other packaging systems as well.

Series index

(will turn into links as series is written)

1. Introduction to RPM
2. Building Basic RPMs
3. Working RPM management software
4. Patterns: Building configuration packages
5. Patterns: Replacing vendor RPMs
6. Patterns: Building RPMS for “trouble” software
7. Conclusions?