jro.codegrinder.com

I’m back, and still surviving

The Orbitz Years

When I started in the NOC at Orbitz, I really only expected to stay a year or two cutting my teeth before finishing school and moving on to something new. Well, that year turned into six, and that finishing school thing never really happened. I was, however, lucky enough to get to know some truly awesome people, and work in three rather different positions within the company.

In the NOC, I recall meeting Ryan on one of my first days at the company and chatting about the Enlightenment window manager while the rest of the team moved the NOC down a floor. We got on pretty well, both being from down-state Illinois, and spent a fair bit of time trolling around Chicago as we learned our way around. Shenanigans included a few visits to Defcon, many rounds of drunken chess, and being asked if we were FBI agents while Wardriving.., er Warsitting?.

After the NOC, I was able to move into a position as a Systems Engineer attached to the development team. I learned about managing a large number of workstations, and dealing with the requests from a large number of developers. LDAP for authentication, RPM + apt, and a local Jabber server really really helped me from going insane. This is about the time I saw the light for properly packaging software for local use. Mmmmm RPMDEBFINKPKG-stuff.

Moving on, I landed in the Information Security department with Justin, Ed, and Laz. We had great fun, and I was able to really spend some time with Ruby, visit Mauritius errr.. audit a vendor in Mauritius, and meet real live Sarbanes-Oxley auditors (shudder).

There were, really, too many great folks to talk about my experiences with each, but it was a wonderful experience all around. I will however note that Laz is still procrastinating in starting his blog. Laz I know you worked at SGI during the Jurassic period, but seriously, stop contemplating, start writing.

Viewpoints

In May of 2007, I left Orbitz to start a new adventure at Viewpoints. It’s been a great experience so far. We’re a small team (an awesome one), we’re a Ruby on Rails site, and I’ve been able to learn a lot. Awesome!

So 10 years in tech?

I’m very happy with where I’ve landed at Viewpoints, and I don’t see myself freaking out and changing careers any time soon. But, it has been interesting to look back over the last 10 years and contemplate where I’ll be 10 years from now. It seems quite likely as I get a bit older that my next real adventures won’t have anything to do with my career or career aspirations, but that’s a thought for another day.

As of a day or two ago, I have survived for 28 years of life. Of those, I’ve spent roughly the last decade working in technology. I’m not normally a huge birthday person, so maybe it’s been the recent onslaught of “your parents music” at work, but for some reason it strikes me as a milestone. So, I thought I’d share a few memorable points and people from the last 10 years of being a geek.

The first “real” job

I’d worked some farming, and food industry jobs before, but right after I graduated high school in May of 1998, my friend Joe secured me a job at the school district he was working and graduated from. I think we reported sometime in June to Jim, where we spent the first month or two of the summer lashing cat5 cables together, making ethernet patch cords and crawling through crawlspaces and air-ducts in up-to century old school buildings. Moving into maintaining the hundreds of windows95 clients for the school district, I was tasked with building some kind of mirroring script using “rdist” (for DOS/Win). It was sort of like Norton Ghost, but command-line, rather rudimentary, and free to educators at the time. It did, however, allow us build a lab of 30 computers in something like a day, rather than week once the “master” image for that lab was finished. That was really my first push into Systems Administration. After that I setup an authentication system for the school’s self-hosted ISP, managed Fool-Proof configs (a totally awesome and amusing way to lock down Windows95 machines from student tampering), and started helping Corey with managing email, DNS and the other “core” services that were on the Linux server in the house.

Amazingly, it looks like the Hoopeston School District is still sporting a web design based somewhat off of one I did a decade ago. In some ways I’m not sure I’d ever design anything like it again, but I am proud of it’s longevity. Check it out here, if you dare!

All in all, at eighteen and working with Linux, I really did feel very lucky to be doing what I was doing.

Net66

Around the start of fall semester in ‘99 I needed to look for some work more local to the university; I just wasn’t able to keep the kind of hours I needed to with the high school being an hour away and the cost of school was an increasing burden. My friend Pete and I applied for Support Tech positions at Net66 (now gone), which was then a small ISP owned by Dennis. I got the position and spent a year or so working with Dennis, Marsita, Bobby, Alan, and a whole mess of other folks that rotated in and out, or were a part of Dennis’s other company (Suburban Express)

Dennis, himself, is really what defined my experience at Net66. Much more-so than my hung-over morning shifts on the phones, late night Saganaki and coding sessions with Alan, annoying Marsita by being late to a shift after falling asleep in a class (more often than you think), and even more than George the Cat.

Dennis started Suburban Express while in school, so that he could have an excuse to explain his business school grades falling to.. B’s. It’s a niche bus service that take kids from their central Illinois campus to the Chicago suburbs where they live. And, he’s been terribly successful at running it, and keeping competitors away, including a price battle with GreyHound and various other upstarts.

So, Dennis decides one day to start and ISP because all the other ones in town suck, and he orders a few custom build fancy fancy servers that will run BSDi, thinking he can get them setup in a weekend. It takes Dennis a week to get Apache setup. And, really, that’s one of the things I’ve come to admire most about him; was simply that when he sees an opportunity, he starts working on it for real. Not just thinking, but doing. He’s had some false starts along the way, a newspaper to compete with the monopolized college rag, etc; but his determination keeps him moving forward.

The sale(s)

Dennis sold Net66 to Advancenet in 2000, and I sort of went along with the company. This was after finishing my last two weeks making flyers for Suburban Express, in what used to be O’Malleys (of Don Mclean - American Pie / UIUC midnight drinking song fame). At Advancenet, I was able to meet some folks that would become important to my life, Dave, and very briefly the girl that I would marry, Cindy. Though, neither of us knew that at that time. Unfortunately for her, (or fortunately, depending) she worked near the front of the office which I did my damnedest to avoid, because that’s where the sales folks and those that took messages from customers were. And both groups created more and more work for me. It got to be so bad after a while that I couldn’t get a haircut without being called or paged, and ultimately had to leave school at UIUC for a bit.

I stayed at AdvanceNet for about a year all told, until I bought a car that was a little too expensive for my income at the time. On a lark, I put out some feelers based on hotjobs.com postings from local tech companies. Within days I had an offer at Orbitz.com that I had no idea what to do with. The salary was near double, so I figured I didn’t really have a choice to make and I took it.

While at Orbitz, Advancenet was bought by another company Egix who I did a bit of consulting for to help transition some of the ISP authentication systems that I had been spending a lot of time on. I was pretty naive about it all, and either through my fault or H&R Block’s ended up creating a couple thousand dollar tax bill for myself. OUCH!

To Be

(continued)

for that last post. I was moving this blog do a different virtual host and wanted to test it out, but didn’t have anything real to say, and well “I’m sorry”.

With that out of the way. It’s time to put some energy back into this. But, rather than a bunch of empty promises that I’ll write something every day for the next 30 days or some other such non-sense, you’ll just have to wait and see. I do plan to revisit an old post/tutorial on RPM, so be sure to check it out for a refresher. Also, if you have any feedback on the content, length, or frequency that you find most interesting or useful in a blog, I’d love to hear it in the comments.

seems to now report status to the command line in kilobits rather than Kilobytes. Sneaky.. sneaky.

I caught this Ruby Inside post about using ruby as an applescript replacement earlier today, and started reading the associated links, and searching for something I could try to do with ruby and applescript that might actually make my life a little easier.

Thanks to my friend Dave, I’ve been giving twitter a 2nd chance. So, I created a tiny script to let me tweet from the command line. It’ll either take arguments and use them as the input, or standard input. I’m sure I could have done it without using applescript or the wonderful Twitterrific, but that wouldn’t have been as much fun. This way I get Twitterrific’s niceties, and besides I always have it running anyway.

#!/usr/bin/env ruby

require 'rubygems'
require 'appscript'

str = ARGV.join(" ")
if str.length == 0
 $stdin.each {|l| str << l}
end

Appscript.app('Twitterrific').post_update(str[0..139])

Usage:

jro@fireant:~$ tweet Having a great old time
jro@fireant:~$ tweet @someone that sounds like fun!
jro@fireant:~$ uptime | tweet 

Frequently mentioned on Lifehacker, Mozy Home is an incredibly economical network backup option at $5 a month. The online backup service will store an unlimited amount of data, and has Mac and Windows clients. The big caveat there is that you don’t have unlimited throughput, so it can still take a while to backup a large fileserver, say. In that case it’s especially important to be able to manage it effectively.

I’m currently using an old PPC-based Mac Mini, running Tiger, with a couple (or more) USB attached drives, as my home fileserver. For a long time I ran a Linux box with an internal RAID5 disk array, but with the advent of mozy and 5$ a month offsite backup, I’ve made the switch. It hasn’t been completely painless, however, so here are a couple notes about effectively using Mozy on a headless Mac.

• You still probably need VNC or that fancy new Leopard desktop-sharing feature, especially to install.

• Mozy has a command line interface! Enable SSH on your Mac and check this out

bombe:~ jro$ /Applications/Mozy.app/Contents/Resources/MozyBackup help
Commands: start|cancel|auth|rescan|status|tempdir

bombe:~ jro$ /Applications/Mozy.app/Contents/Resources/MozyBackup status
Status: Backing up files...

Preparing (0.0 B/s): 
   70.9% 4925415586 of 6943326336

 Sending   (46.3 KB/s): blah_blah_20061012.tgz
    0.9% 87851008 of 9257768960

You can probably guess what most of the commands do. It’s great being able to start and cancel a backup on a whim, say when I upload more data to the server, or stop it when it’s messing with my VOIP call quality. I definitely recommend still having Mozy backup on a regular schedule, even if you’re doing a fair bit by hand. You never know if you’ll forget!

• Mozy has a very useful log.

bombe:~ jro$ sudo less /Library/Logs/Mozy.log

2008-03-15 12:43:55.229 MozyBackup[31] (backup) Starting backup
2008-03-15 12:43:55.231 MozyBackup[31] (backup) Checking for update
2008-03-15 12:43:56.821 MozyBackup[31] (backup) Connecting to r4165.mozy.client.ut2.berkeleydata.com (66.133.112.25)
2008-03-15 12:43:57.277 MozyBackup[31] (backup) Version is up-to-date (0.9.1.0)
...
2008-03-15 12:46:01.028 MozyBackup[31] (backup) Building backup set
2008-03-15 12:46:03.278 MozyBackup[31] (backup) Processing files in scan cache
2008-03-15 12:46:09.704 MozyBackup[31] (backup) Processing files in manifest but not in scan cache
2008-03-15 12:46:19.569 MozyBackup[31] (backup) Preparing 4 files (6943326336 bytes)
2008-03-15 12:46:19.570 MozyBackup[31] (backup) Unlocking cache...
2008-03-15 12:46:19.570 MozyBackup[31] (backup) Unlocked cache
2008-03-15 12:46:19.570 MozyBackup[31] (prepare) Starting
2008-03-15 12:46:19.575 MozyBackup[31] (send) Starting
2008-03-15 12:46:19.575 MozyBackup[31] (send) Getting files from queue
2008-03-15 12:51:12.245 MozyBackup[31] (send) Offering 1 files (0 are removes)

• Mozy can use your your encryption key. If you don’t want the guys at EMC/Mozy to know what kind of ahem data you’re storing on their machines, you can configure Mozy to use your own encryption key. This definitely increases the security of your data, just make sure you don’t lose that key, or you’ll be in a world of hurt come restore time.

I, recently, decided to move some of my home rails sites to my aging PPC-based mac mini. I’m still running Tiger on it because I’m lazy, don’t need any new features, and know how macports & fink work in Tiger. I ran through the normal, install fink, ruby package,… routine. But, when I got to compiling some of my gems, I got this error:

/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: flag: -undefined dynamic_lookup can't be used with MACOSX_DEPLOYMENT_TARGET environment variable set to: 10.1

Which seems pretty strange since I’m obviously running on 10.4. And, while I don’t know why the system is getting it wrong, a quick work around is to set the environment variable yourself.

export MACOSX_DEPLOYMENT_TARGET=10.4

Oddly, it doesn’t happen on my intel macs, so I suspect that there’s a bug somewhere in the PPC fink stack that just doesn’t effect all compilations. /shrug.

I hope everyone had a safe and happy time over the holidays. I’m just getting back to normal life after a month of craziness, and I need to figure out my resolutions, or lack of, for the new year. What are you doing differently or better this year?

Every ruby nerd in the world will be blogging about this. I’m no exception. Ruby on Rails 2.0 is out and represents a mountain of progress. Many thanks to the rails core team and everyone that’s assisted in bringing this to us, the masses.

• Riding Rails
• Ryan’s Scraps
• 37 signals blog

Better late than never. This is the first “real” article in the Packaging Software series. It’s long, but hopefully it’ll help someone out. Enjoy!

So what’s an RPM?

In it’s distilled form, a big ball of files that hopefully make up something useful, like software. But wait, there’s more! It’s not just a file format for installing, uninstalling, updating, verifying software, it’s also the system for creating those bundles and doing all the installing, etc. Confused? What we have is two parts: The files commonly referred to as RPM, aka the software bundles, and rpm set of commands for creating and managing those bundles.

History and the Soap Opera

RPM was birthed by Red Hat as the Red Hat Package Manager, later becoming the RPM Package Manager as it became building block of more and more non-Red Hat Linux distributions. It’s grown to be used in innumerable Linux distros, large and small, popular and not.

I don’t want to get the drama wrong, so I’m just going to quote Wikipedia

As of the 31st of May, 2007, there are two versions of RPM in development. The first is led by long-time (since 1999) RPM maintainer, Jeff Johnson (‘JBJ’), who continued his development efforts after departing Red Hat. It has combined recently with efforts of OpenPKG and added a new website rpm5 tine of the fork. The ‘JBJ’ variant is used by Mandriva, PLD Linux Distribution and supported by OpenPKG. Jeff Johnson formerly worked for Red Hat before a controversy.

The other effort is led by Red Hat recent hire (2007) staff member Panu Matilainen. The Red Hat RPM tine of the fork which issued its first major code revision in July 2007, is used by Fedora, Red Hat Enterprise Linux, Novell’s openSUSE and CentOS. Matilainen has also worked on an apt-like front end for package management for RPM previously; apt-get is a Debian Linux management tool.

The displaced former RPM website content is archived by its long time maintainer at oldrpm web archive. On December 14, 2006, Red Hat decided to take complete control of editorial content at the formerly community maintained website which content was maintainted in an ‘open to the community’ process manner.

Does it matter to you? Probably not, just be aware there are currently two major forks of RPM and there may be slight differences in the API standards or tools that are available.

The simple things

So let’s do something useful. I’m working on a Red Hat system, so if you’re not it’s possible things will be a little different.

What do we have installed?

List them all

Good for grep’ing through when you don’t know what you’re looking for.

[jro@localhost ~]$ rpm -qa 
libgcc-3.4.6-8.i386
filesystem-2.3.0-1.i386
glibc-common-2.3.4-2.36.i386
bzip2-libs-1.0.2-13.EL4.3.i386
...

Find a specific package

Good when you know it’s there, but just need the version.

[jro@localhost ~]$ rpm -q rmt
rmt-0.4b39-3.EL4.2.i386

Find out what RPM owns a file

Good for.. where the hell did that come from?

[jro@localhost ~]$ rpm -qf /etc/snmp/snmpd.conf
net-snmp-5.1.2-11.EL4.10.i386

Get some basic info on an installed package.

[jro@localhost ~]$ rpm -qi hdparm
Name        : hdparm                       Relocations: (not relocatable)
Version     : 5.7                               Vendor: Red Hat, Inc.
Release     : 2                             Build Date: Tue 21 Sep 2004 05:23:51 AM CDT
Install Date: Wed 27 Jun 2007 04:10:31 PM CDT      Build Host: tweety.build.redhat.com
Group       : Applications/System           Source RPM: hdparm-5.7-2.src.rpm
Size        : 64075                            License: BSD
Signature   : DSA/SHA1, Wed 05 Jan 2005 05:18:55 PM CST, Key ID 219180cddb42a60e
Packager    : Red Hat, Inc. 
Summary     : A utility for displaying and/or setting hard disk parameters.
Description :
Hdparm is a useful system utility for setting (E)IDE hard drive
parameters. For example, hdparm can be used to tweak hard drive
performance and to spin down hard drives for power conservation.

List all the files installed as part of a package

Good for where the hell did that config file get put?

[jro@localhost ~]$ rpm -ql rmt
/etc/rmt
/sbin/rmt
/usr/share/man/man8/rmt.8.gz

Installing and Upgrading RPMs

Installing a single RPM

[jro@localhost ~]$ rpm -i nginx-0.5.31-1.vp.i386.rpm
error: can't create transaction lock on /var/lock/rpm/transaction

Hrmm.. maybe I need to do this as root

[jro@localhost ~]$ sudo rpm -i nginx-0.5.31-1.vp.i386.rpm
Password: *****
[jro@localhost ~]$

Yep. Not a lot of output tho

[jro@localhost ~]$ sudo rpm -iv nginx-0.5.31-0.vp.i386.rpm
Preparing packages for installation...
nginx-0.5.31-0.vp

Better.

[jro@localhost ~]$ sudo rpm -ivh nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [100%]

There we go.

Installing multiple packages at the same time

[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm
error: Failed dependencies:
        nginx = 0.5.31 is needed by nginx-contrib-0.5.31-0.vp.i386
[jro@localhost ~]$ sudo rpm -ivh nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [100%]
[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx-contrib          ########################################### [100%]

So our first package required another, and installing them one by one worked well enough but it’s a pain.

[jro@localhost ~]$ sudo rpm -ivh nginx-contrib-0.5.31-0.vp.i386.rpm nginx-0.5.31-0.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [ 50%]
   2:nginx-contrib          ########################################### [100%]

It’s way easier just to put them all together on one line. Or go c-razy and use file globbing ( rpm -ivh *.rpm ).

Upgrading RPMs

[jro@localhost ~]$ sudo rpm -Uvh nginx-contrib-0.5.31-1.vp.i386.rpm nginx-0.5.31-1.vp.i386.rpm
Preparing...                ########################################### [100%]
   1:nginx                  ########################################### [ 50%]
   2:nginx-contrib          ########################################### [100%]

It’s not too much different than installing, just that the -i turns into a -U. HOWEVER, you can also use the -U to install a package. So why bother using -i? It’s simple. Don’t. Always use -U, unless you specifically don’t want to upgrade a package you might have installed. It sounds silly, but you might end up in that situation one day if you have an RPM that restarts a service, etc.

Uninstalling.. er Erasing RPMs

[jro@localhost ~]$ sudo rpm -e nginx
Password:
error: Failed dependencies:
        nginx = 0.5.31 is needed by (installed) nginx-contrib-0.5.31-1.vp.i386
[jro@localhost ~]$ sudo rpm -e nginx-contrib nginx
[jro@localhost ~]$                                       

Pretty much what you’d expect at this point.

Useful Magic

See the architecture of a package

If you’re running a 64-bit machine, it’s VERY useful to know which packages are 32-bit and which are 64-bit. As on some distros it’s possible to have both versions installed, and then things start to get very confusing.

[jro@localhost ~]$ rpm -qa | head -5
libgcc-3.4.6-8
filesystem-2.3.0-1
glibc-common-2.3.4-2.36
bzip2-libs-1.0.2-13.EL4.3
ethtool-1.8-4
[jro@localhost ~]$ echo "%_query_all_fmt         %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros
[jro@localhost ~]$ rpm -qa | head -5
libgcc-3.4.6-8.i386
filesystem-2.3.0-1.i386
glibc-common-2.3.4-2.36.i386
bzip2-libs-1.0.2-13.EL4.3.i386
ethtool-1.8-4.i386

We’ll talk more about the rpmmacros files when we get to building packages, but I recommend adding this even if you’re not running a system with packages from differing architectures.

See the changelog for a package

Installed package

[jro@localhost ~]$ rpm -q --changelog nginx 
* Wed Sep 19 2007  0.5.31-1.vp

- add condrestart

* Wed Sep 19 2007  0.5.31-0.vp

- upgrade to latest stable

* Mon Sep 17 2007  0.5.26-3.vp
....

Package not yet installed

[jro@localhost ~]$ rpm -qp nginx-0.5.26-0.vp.i386.rpm --changelog
* Mon Jul 02 2007  0.5.26-0.vp

- new nginx version - 0.5.26
- start fork of package

* Wed May 23 2007  0.5.20-0

- new nginx version - 0.5.20

* Thu Apr 26 2007  0.5.19-0

Muck with the output format of RPM without making it semi-permanent

[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}\n" | head -3
libgcc
filesystem
glibc-common
[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}_%{VERSION}\n" | head -3
libgcc_3.4.6
filesystem_2.3.0
glibc-common_2.3.4
[jro@localhost ~]$ rpm -qa --queryformat "%{NAME}_%{VERSION}_%{DISTRIBUTION}\n" | head -3
libgcc_3.4.6_Red Hat (RHEL-4)
filesystem_2.3.0_Red Hat Linux
glibc-common_2.3.4_Red Hat (RHEL-4)

It’s not super intuitive, but thankfully there’s some info in the rpm man page, and it can be handy when writing scripts to report on your packages, create lists to mirror systems from, etc.

Resources

• The RPM Guide at The Fedora Project - This used to be a book that’s no longer in print. This is the best resource outside of looking at what other people have done.
• The RPM HOWTO at The Linux Documentation Project - It’s a bit dated but there’s still some relevant information there.
• RPM.org wiki - Links to other resources.