I have long used MacPorts as my open source package manager, but I’ve had issues recently with certain ports not being updated or being out of date. So I was really interested in Homebrew.  The fact that it is all on github, open and actively developed really appealed to me. After getting it up and running, I wanted to port my package update notifier to use Homebrew. Doing so was really quite easy. Here’s what I came up with, which is also in my dotfiles on github.

#!/bin/bash
#
# Notify of Homebrew updates via Growl on Mac OS X
#
# Author: Chris Streeter http://www.chrisstreeter.com
# Requires: Growl Notify Extra to be installed. Install with
#   brew install growlnotify
 
TERM_APP='/Applications/Terminal.app'
BREW_EXEC='/usr/local/bin/brew'
GROWL_NOTIFY='/usr/local/bin/growlnotify'
GROWL_TITLE="Homebrew Update(s) Available"
GROWL_ARGS="-n 'Homebrew' -d $GROWL_NOTIFY -a $BREW_EXEC"
 
$BREW_EXEC update 2>&1 > /dev/null
outdated=`$BREW_EXEC outdated | tr ' ' '\n'`
 
if [ -z "$outdated" ] ; then
    if [ -e $GROWL_NOTIFY ]; then
        # No updates available
        $GROWL_NOTIFY $GROWL_ARGS -m '' -t "No Homebrew Updates Available"
    fi
else
    # We've got an outdated formula or two
 
    # Nofity via growl
    if [ -e $GROWL_NOTIFY ]; then
        lc=$((`echo "$outdated" | wc -l`))
        outdated=`echo "$outdated" | tail -$lc`
        message=`echo "$outdated" | head -5`
        if [ "$outdated" != "$message" ]; then
            message="Some of the outdated formulae are:
$message"
        else
            message="The following formulae are outdated:
$message"
        fi
        # Send to growlnotify
        echo "$message" | $GROWL_NOTIFY $GROWL_ARGS -s -t $GROWL_TITLE
    fi
fi

Assuming the script is at ~/bin/brew-update-notifier, you can install the script to a crontab by running sudo crontab -e, then adding the line 0 12 * * * /Users/<username>/bin/brew-update-notifier to the end of the file (substituting <username> for your username, or wherever you’ve put the script). I’ve chosen to run the script every day at noon because my computer is likely to be on and connected to a network.

My primary computer is a Macbook Pro. Since it is also the computer that all my photos and important data is on, I don’t want to lose any of it. Every day I make sure to plug in my 1 TB external hard drive that Time Machine mirrors the contents of drive to. So at all times, I’ve got a backup of my drive and a history of the files. I also have another external drive that I keep at my parent’s house in their safe that is filled with Time Machine backups. So if my house blows up, I have something to fall back on. The problem with that disk though is that it is several months out of date. And that sucks because my data changes all the time.

But thats not all for my laptop. I also make use of CrashPlan. At home, I also have a Linux box with a 4 hard drives in a RAID 5 setup, so if one drive dies, I don’t lose any data, I can just replace the drive. So I have CrashPlan backup my laptop to my Linux box onto it’s RAID storage, and this can happen anywhere my laptop has Internet access. Currently, CrashPlan is configured to run every night.

My email is hosted in the cloud, but I still run hourly backups onto the RAID storage in my linux box using mbsync. I also run backups of my Google Docs every day using gdatacopier (again, saving to my Linux box).

I used to also have Mozy, but decided to scale back costs for a while. Though I would really like to use it again. For some important things, I’ll stick them on Dropbox, which maintains a backup copy in the cloud. However, storage on Dropbox is much more limited than something like Mozy.

Right now, one single point of failure is my Linux box. The drives are getting old and should be replaced soon (the MTTF is way overdue) and I’m relying too much on them these days. And I really should have a standby drive in the array in case of multiple disk failures. The plus side is that there really isn’t anything on that computer that is not somewhere else, or I can’t redownload. But it would be a huge hassle.

I’ve really tried to create a good, easy solution for backups without any single point of failure. Though there are definitely several areas that I need to improve on. You never need the backup until you need it. And at that point, if you don’t have the backup, you may find you’ve lost years of irreplaceable memories. I know that to me, my photos are irreplaceable. I also have all the documents I created from sixth grade all the way through college. And if I lost those, there’s no bringing them back (there’s just no way to recreate the awesome resume I wrote in sixth grade – “Expected graduation from Middle School, 1998″, “experience with many tools and their uses”, “fast runner” – classic).

If you own a Mac, there is no excuse, just buy a cheap external hard drive and plug it in and use Time Machine. Windows users should just buy a Mac. Linux users… well, write some perl script that rsync’s your files somewhere and put it in cron or write some other solution (I mean really, you’re using Linux, you should be able to figure something out). Or look at CrashPlan. Hopefully, if you aren’t backing up your files, you’ll start.

#!/bin/bash
#
# Notify of MacPort updates via Growl on Mac OS X
#
# Author: Chris Streeter http://www.chrisstreeter.com
# Requires: Growl Notify Extra to be installed (but fails gracefully). Info
#       about how to get the extra is at http://growl.info/extras.php
 
TERM_APP='/Applications/Terminal.app'
PORT_EXEC='/opt/local/bin/port'
GROWL_NOTIFY='/usr/local/bin/growlnotify'
GROWL_TITLE="MacPort Update(s) Available"
GROWL_ARGS="-n 'MacPorts' -d $GROWL_NOTIFY -a $PORT_EXEC"
 
$PORT_EXEC sync 2>&1 > /dev/null
outdated=`$PORT_EXEC outdated`
 
if echo $outdated | grep -q 'No installed ports are outdated.' ; then
    if [ -e $GROWL_NOTIFY ]; then
        # No updates available
        $GROWL_NOTIFY $GROWL_ARGS -m '' -t "No MacPort Updates Available"
    fi
else
    # We've got an outdated port or two
 
    # Nofity via growl
    if [ -e $GROWL_NOTIFY ]; then
        lc=$((`echo "$outdated" | wc -l` - 1))
        outdated=`echo "$outdated" | tail -$lc | cut -d " " -f 1`
        message=`echo "$outdated" | head -5`
        if [ "$outdated" != "$message" ]; then
            message="Some of the outdated packages are:
$message"
        else
            message="The following packages are outdated:
$message"
        fi
        # Send to growlnotify
        echo "$message" | $GROWL_NOTIFY $GROWL_ARGS -s -t $GROWL_TITLE
    fi
fi

Assuming the script is at ~/bin/port-update-notifier, you can install the script to a crontab by running sudo crontab -e, then adding the line 0 12 * * * /Users/<username>/bin/port-update-notifier to the end of the file (substituting <username> for your username, or wherever you’ve put the script). I’ve chosen to run the script every day at noon because my computer is likely to be on and connected to a network.

I saw this as an opportunity for improvement. I also saw this an opportunity to learn about Google App Engine. So I wrote a python application that lets you schedule email reminders for the Contacts Birthday Calendar. In addition, you can select a specific hour in a specific time-zone to send the reminders at as well. The application use’s the Google App Engine user authentication and the Google Data AuthSub permission request API to get upcoming events for display and email reminders.

This gave me some good experience with Google App Engine’s scheduled tasks and Task Queues, as well as a chance to hone up on my Python. Without further ado, I present GCalendar Reminders. Feel free to use it to send yourself emails using the security of Google App Engine.

I then started looking into PHP clients as our current environment is mostly PHP. We require that the client support consistent hashing, and, from a quick search, a couple turned up. PRedis seemed to offer the most potential, and after some quick tests, also seemed to offer the greatest performance. So I set up a more elaborate benchmark of the the client and server package.

My test setup involved using 5 servers with between 2 and 5 enabled at a time on the clients (ie. I disabled up to 3 of the servers in the client configurations). For performance, I configured the servers to never write to disk, though periodically syncing to disk should not cause too much of a performance loss. In fact performance was most greatly affected by forcing an fsync after every write. I then had 9 other client boxes running the same code base, with all 9 enabled for each test.

Each client would start a master PHP process that forked 20, 30 or 40 child processes to simulate greater and greater load. Each forked PHP process then did 10,000 SETs on random keys with 4 byte payloads (early tests showed that payload size didn’t drastically affect the results). I was using the PHP 4.2.6 branch of the PRedis client, and had optimized it a bit so that it did fewer counts of the consistent hash array. I made the optimizations based on some results after profiling the code. I then had the master PHP process on each box repeat the test 5 times to help to average the test results.

I used dsh to start the test simultaneously on all the clients, timing how long it took the dsh process to start and finish executing. This was the amount of time it took to execute (5 repeats) * (9 client boxes) * (20, 30 or 40 client processes / box) * (10,000 requests) = X total requests. I then graphed the results below.

Graphing Requests / Second With Redis and PRedis

This ended up showing that with 9 clients, going up to 4 servers was the point of diminishing returns. Adding the fifth server (with that client box count) did not increase throughput, but rather, the throughput went down. The reason for this is most likely a combination of network interface contention on each client and higher overhead from consistent hashing. So with 9 clients, I found that the sweet spot, with my setup, was 4 servers. By adding more clients, along with more servers, the throughput would increase, instead of the decrease I was starting to see.

First, a description of my situation. I run all my email through GMail. I enjoy the interface and the fact that it is a cloud service; I can access my email seamlessly on my phone, my home computer, my work computer, some other computer, etc. However, I don’t want to lose all that information. Google is great, but who is to say that something terrible won’t happen and some (or all) of my mail is lost? So I wanted to setup some sort of backup. And then once I got that setup, make it automated.

At home, I run an Ubuntu box, that I just upgraded to 9.04, Jaunty Jackalope. This machine primarily serves as a media box, hosting video that streams to my Tivo off the 1.5TB RAID 5 array. I also use it as a network mounted TimeMachine box as well. Since I have extra storage on it, I figured I’d get something to sync my mail over IMAP periodically, and then I have a nice little backup.

After some searching, I came across two sites that had instructions using the utility mbsync (formerly isync). I found that following the instructions worked pretty well, though I had to customize the patch provided to get it to work with the version provided by Ubuntu.  And then I thought I’d detail my steps here for others to see.

1. First, enable IMAP in your GMail account.
2. Install the dependencies for mbsync:

   sudo apt-get install libc6 libdb4.8 libdb-dev libdb4.8-dev libssl0.9.8 libssl-dev

3. Get the source for mbsync on Ubuntu with:

   sudo apt-get source isync

   This will download the source for mbsync and create a directory (in your current directory) named isync-1.0.4 (Yes, I know it is called isync. That is a legacy name).

4. Now the normal mbsync install works just fine. However, it doesn’t support recursive directories. I use GMail labels setup to provide a hierarchy using forward slashes. It turns out that the fix to get mbsync to support directories like this is really easy to do and someone went through the trouble to get it to work. However, it doesn’t quite work on the version of the code packaged with Ubuntu. So I had to modify the patch a bit. The patch I created can be found here. Download it and put it somewhere on your system. To apply the patch, change directories into the isync-1.0.4 directory that was just downloaded and run:

   patch -p1 < /path/to/patch/file/recursive_imap_ubuntu.patch

   Once you’ve patched the directory, configure (./configure), build (make) and install (sudo make install).

5. Now you should have an executable mbsync in your path. So it is time to start preparing to do the initial sync. Choose a place to store your backups. I chose

   /export/backups/mail/

6. We want to connect to Google securely which means you’ll need the latest SSL certificates. To get those, use the openssl client on your machine. Run:

   openssl s_client -connect imap.gmail.com:993 -showcerts

   which should show two blocks of

   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----

   in the output. You’ll want to take each block (including the BEGIN/END CERTIFICATE lines), and put each of them into their own file. I put the first one in a file gmail.crt and the second one in the file google.crt (since the first signs imap.gmail.com which is signed by Google Internet Authority, the second certificate).

7. The second certificate, the one for the Google Internet Authority, is signed by Equifax. So we’ll need Equifax’s certificate also. An as it turns out, Ubuntu has a copy of Equifax’s certificate already sitting in the repositories. Just run

   sudo apt-get install ca-certificates

   to get the latest CA certificates. After installing the CAs, Equifax’s CA sits at /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt, which we’ll need in the configuration file in the next step.

8. Now we can write the configuration file we are going to use. Here is a copy of mine:

   IMAPAccount gmail
   Host imap.gmail.com
   User yourusername@gmail.com
   UseIMAPS yes
   CertificateFile /export/backups/mail/gmail.crt
   CertificateFile /export/backups/mail/google.crt
   CertificateFile /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt 
   
   IMAPStore gmail-remote
   Account gmail
   
   MaildirStore gmail-local
   Path /export/backups/mail/gmail/
   Inbox /export/backups/mail/gmail/Inbox
   
   

   Channel gmail
   Master :gmail-remote:
   Slave :gmail-local:
   # Exclude everything under the internal [Gmail] folder, except the interesting folders
   Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
   Create Slave
   Sync Pull
   SyncState *

   Check out the Patterns line. That is where you would include or exclude various labels. All lables are stored at the root of the hierarchy, with the special directory [Gmail] having things like ‘Sent Mail’, ‘Spam’, ‘Starred’, etc in it. I wanted to exclude all the items in the [Gmail] directory except for the ones listed. The ‘*’ at the beginning includes all other labels. You will also want to change the Path and Inbox lines to point to your mail location, as well as the first two CertificateFile lines. Also, be sure to enter your actual GMail login on the User line. Now save this file somewhere. Note: saving it as ~/.mbsyncrc will cause it to be automatically loaded when mbsync is run, meaning you don’t need to specify which config file with the -c option.

9. Now go ahead and test it out by listing the labels in your account with the command mbsync -l -c /path/to/the/configfile.rc gmail. Running it will look like this and ask you for your password:

   [streeter@scout]:~$ mbsync -l -c ~/.mbsyncrc gmail
   Reading configuration file /home/streeter/.mbsyncrc
   Resolving imap.gmail.com... ok
   Connecting to 209.85.199.109:993... ok
   Connection is now encrypted
   Logging in...
   Password (yourusername@gmail.com@imap.gmail.com):
   Channel gmail
   lists/code
   bills
   archive/cron
   archive/classes
   archive/work
   [Gmail]/Starred
   [Gmail]/Sent Mail
   [Gmail]/All Mail
   INBOX
   @followup
   [streeter@scout]:~$

   If you see something like this, then it worked! Now just go ahead and start your first mail download with mbsync -c /path/to/the/configfile.rc gmail And then you get to wait while it finishes which can take a while depending on how much mail you have.

You can go an extra step and save your password in the file. If you add the line Pass yourpasswordhere right after the User line, you won’t be asked for your password and can then setup a cronjob to automatically keep mail locally on a schedule. However, your password for your Google Account then is stored in cleartext on your machine. So only do this is you absolutely know what you are doing and who can access the machine.

Hopefully this becomes useful for people.

Update 4/18/2011:

I just updated the directions slightly for Ubuntu 10.10 Maverick Meerkat. For Maverick, I had to also install libssl-dev to get it to work with recursive directories.