The issue was that I got exception tracebacks when sourcing the /usr/local/bin/virtualenvwrapper.sh script upon bash initialization. However, the exception happens while Python is running it’s atexit handlers. So the stack was only 3 levels deep. I’m running virtualenv and virtualenvwrapper on my Ubuntu 9.04 server.
Here’s the output of the exception that I got:

[streeter@mail]:~$ source /usr/local/bin/virtualenvwrapper.sh
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/handlers.py", line 72, in emit
self.doRollover()
File "/usr/lib/python2.6/logging/handlers.py", line 129, in doRollover
os.rename(self.baseFilename, dfn)
OSError: [Errno 13] Permission denied
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/handlers.py", line 71, in emit
if self.shouldRollover(record):
File "/usr/lib/python2.6/logging/handlers.py", line 145, in shouldRollover
self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1508, in shutdown
h.flush()
File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
self.stream.flush()
ValueError: I/O operation on closed file
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1508, in shutdown
h.flush()
File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
self.stream.flush()
ValueError: I/O operation on closed file
[streeter@mail]:~$

As you can see, not a lot to go with. So it seems that virtualenvwrapper does some logging, and it looks like the logging module wants to rename that logfile. Digging even deeper, the logging module wants to rotate the logfile because it got too big. Well, where is the log file? Turns out that virtualenvwrapper sets up a logfile called hook.log at the root of the WORKON_HOME environment variable. My WORKON_HOME is set to /usr/local/pythonenv and lo and behold, that directory was not writeable by my logged in user.  Thats an easy fix. Just change the permissions.

I then started looking into PHP clients as our current environment is mostly PHP. We require that the client support consistent hashing, and, from a quick search, a couple turned up. PRedis seemed to offer the most potential, and after some quick tests, also seemed to offer the greatest performance. So I set up a more elaborate benchmark of the the client and server package.

My test setup involved using 5 servers with between 2 and 5 enabled at a time on the clients (ie. I disabled up to 3 of the servers in the client configurations). For performance, I configured the servers to never write to disk, though periodically syncing to disk should not cause too much of a performance loss. In fact performance was most greatly affected by forcing an fsync after every write. I then had 9 other client boxes running the same code base, with all 9 enabled for each test.

Each client would start a master PHP process that forked 20, 30 or 40 child processes to simulate greater and greater load. Each forked PHP process then did 10,000 SETs on random keys with 4 byte payloads (early tests showed that payload size didn’t drastically affect the results). I was using the PHP 4.2.6 branch of the PRedis client, and had optimized it a bit so that it did fewer counts of the consistent hash array. I made the optimizations based on some results after profiling the code. I then had the master PHP process on each box repeat the test 5 times to help to average the test results.

I used dsh to start the test simultaneously on all the clients, timing how long it took the dsh process to start and finish executing. This was the amount of time it took to execute (5 repeats) * (9 client boxes) * (20, 30 or 40 client processes / box) * (10,000 requests) = X total requests. I then graphed the results below.

Graphing Requests / Second With Redis and PRedis

This ended up showing that with 9 clients, going up to 4 servers was the point of diminishing returns. Adding the fifth server (with that client box count) did not increase throughput, but rather, the throughput went down. The reason for this is most likely a combination of network interface contention on each client and higher overhead from consistent hashing. So with 9 clients, I found that the sweet spot, with my setup, was 4 servers. By adding more clients, along with more servers, the throughput would increase, instead of the decrease I was starting to see.

First, a description of my situation. I run all my email through GMail. I enjoy the interface and the fact that it is a cloud service; I can access my email seamlessly on my phone, my home computer, my work computer, some other computer, etc. However, I don’t want to lose all that information. Google is great, but who is to say that something terrible won’t happen and some (or all) of my mail is lost? So I wanted to setup some sort of backup. And then once I got that setup, make it automated.

At home, I run an Ubuntu box, that I just upgraded to 9.04, Jaunty Jackalope. This machine primarily serves as a media box, hosting video that streams to my Tivo off the 1.5TB RAID 5 array. I also use it as a network mounted TimeMachine box as well. Since I have extra storage on it, I figured I’d get something to sync my mail over IMAP periodically, and then I have a nice little backup.

After some searching, I came across two sites that had instructions using the utility mbsync (formerly isync). I found that following the instructions worked pretty well, though I had to customize the patch provided to get it to work with the version provided by Ubuntu.  And then I thought I’d detail my steps here for others to see.

1. First, enable IMAP in your GMail account.
2. Install the dependencies for mbsync:

   sudo apt-get install libc6 libdb4.8 libdb-dev libdb4.8-dev libssl0.9.8 libssl-dev

3. Get the source for mbsync on Ubuntu with:

   sudo apt-get source isync

   This will download the source for mbsync and create a directory (in your current directory) named isync-1.0.4 (Yes, I know it is called isync. That is a legacy name).

4. Now the normal mbsync install works just fine. However, it doesn’t support recursive directories. I use GMail labels setup to provide a hierarchy using forward slashes. It turns out that the fix to get mbsync to support directories like this is really easy to do and someone went through the trouble to get it to work. However, it doesn’t quite work on the version of the code packaged with Ubuntu. So I had to modify the patch a bit. The patch I created can be found here. Download it and put it somewhere on your system. To apply the patch, change directories into the isync-1.0.4 directory that was just downloaded and run:

   patch -p1 < /path/to/patch/file/recursive_imap_ubuntu.patch

   Once you’ve patched the directory, configure (./configure), build (make) and install (sudo make install).

5. Now you should have an executable mbsync in your path. So it is time to start preparing to do the initial sync. Choose a place to store your backups. I chose

   /export/backups/mail/

6. We want to connect to Google securely which means you’ll need the latest SSL certificates. To get those, use the openssl client on your machine. Run:

   openssl s_client -connect imap.gmail.com:993 -showcerts

   which should show two blocks of

   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----

   in the output. You’ll want to take each block (including the BEGIN/END CERTIFICATE lines), and put each of them into their own file. I put the first one in a file gmail.crt and the second one in the file google.crt (since the first signs imap.gmail.com which is signed by Google Internet Authority, the second certificate).

7. The second certificate, the one for the Google Internet Authority, is signed by Equifax. So we’ll need Equifax’s certificate also. An as it turns out, Ubuntu has a copy of Equifax’s certificate already sitting in the repositories. Just run

   sudo apt-get install ca-certificates

   to get the latest CA certificates. After installing the CAs, Equifax’s CA sits at /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt, which we’ll need in the configuration file in the next step.

8. Now we can write the configuration file we are going to use. Here is a copy of mine:

   IMAPAccount gmail
   Host imap.gmail.com
   User yourusername@gmail.com
   UseIMAPS yes
   CertificateFile /export/backups/mail/gmail.crt
   CertificateFile /export/backups/mail/google.crt
   CertificateFile /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt 
   
   IMAPStore gmail-remote
   Account gmail
   
   MaildirStore gmail-local
   Path /export/backups/mail/gmail/
   Inbox /export/backups/mail/gmail/Inbox
   
   

   Channel gmail
   Master :gmail-remote:
   Slave :gmail-local:
   # Exclude everything under the internal [Gmail] folder, except the interesting folders
   Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
   Create Slave
   Sync Pull
   SyncState *

   Check out the Patterns line. That is where you would include or exclude various labels. All lables are stored at the root of the hierarchy, with the special directory [Gmail] having things like ‘Sent Mail’, ‘Spam’, ‘Starred’, etc in it. I wanted to exclude all the items in the [Gmail] directory except for the ones listed. The ‘*’ at the beginning includes all other labels. You will also want to change the Path and Inbox lines to point to your mail location, as well as the first two CertificateFile lines. Also, be sure to enter your actual GMail login on the User line. Now save this file somewhere. Note: saving it as ~/.mbsyncrc will cause it to be automatically loaded when mbsync is run, meaning you don’t need to specify which config file with the -c option.

9. Now go ahead and test it out by listing the labels in your account with the command mbsync -l -c /path/to/the/configfile.rc gmail. Running it will look like this and ask you for your password:

   [streeter@scout]:~$ mbsync -l -c ~/.mbsyncrc gmail
   Reading configuration file /home/streeter/.mbsyncrc
   Resolving imap.gmail.com... ok
   Connecting to 209.85.199.109:993... ok
   Connection is now encrypted
   Logging in...
   Password (yourusername@gmail.com@imap.gmail.com):
   Channel gmail
   lists/code
   bills
   archive/cron
   archive/classes
   archive/work
   [Gmail]/Starred
   [Gmail]/Sent Mail
   [Gmail]/All Mail
   INBOX
   @followup
   [streeter@scout]:~$

   If you see something like this, then it worked! Now just go ahead and start your first mail download with mbsync -c /path/to/the/configfile.rc gmail And then you get to wait while it finishes which can take a while depending on how much mail you have.

You can go an extra step and save your password in the file. If you add the line Pass yourpasswordhere right after the User line, you won’t be asked for your password and can then setup a cronjob to automatically keep mail locally on a schedule. However, your password for your Google Account then is stored in cleartext on your machine. So only do this is you absolutely know what you are doing and who can access the machine.

Hopefully this becomes useful for people.

Update 4/18/2011:

I just updated the directions slightly for Ubuntu 10.10 Maverick Meerkat. For Maverick, I had to also install libssl-dev to get it to work with recursive directories.