I’m running DD-WRT v24-sp1 on my Linksys WRT54GL v1.1 with an Airport Express running the latest 6.3 firmware. On the WRT54GL, I set the wireless security mode to WPA2 Personal and the WPA Algorithm set to TKIP. On the Airport Express, I told it to join my existing network with wireless security of WPA/WPA2 Personal. Doing all this, and I can now stream music to my stereo.

I’m sure someone else will find this information useful.

I saw this as an opportunity for improvement. I also saw this an opportunity to learn about Google App Engine. So I wrote a python application that lets you schedule email reminders for the Contacts Birthday Calendar. In addition, you can select a specific hour in a specific time-zone to send the reminders at as well. The application use’s the Google App Engine user authentication and the Google Data AuthSub permission request API to get upcoming events for display and email reminders.

This gave me some good experience with Google App Engine’s scheduled tasks and Task Queues, as well as a chance to hone up on my Python. Without further ado, I present GCalendar Reminders. Feel free to use it to send yourself emails using the security of Google App Engine.

I then started looking into PHP clients as our current environment is mostly PHP. We require that the client support consistent hashing, and, from a quick search, a couple turned up. PRedis seemed to offer the most potential, and after some quick tests, also seemed to offer the greatest performance. So I set up a more elaborate benchmark of the the client and server package.

My test setup involved using 5 servers with between 2 and 5 enabled at a time on the clients (ie. I disabled up to 3 of the servers in the client configurations). For performance, I configured the servers to never write to disk, though periodically syncing to disk should not cause too much of a performance loss. In fact performance was most greatly affected by forcing an fsync after every write. I then had 9 other client boxes running the same code base, with all 9 enabled for each test.

Each client would start a master PHP process that forked 20, 30 or 40 child processes to simulate greater and greater load. Each forked PHP process then did 10,000 SETs on random keys with 4 byte payloads (early tests showed that payload size didn’t drastically affect the results). I was using the PHP 4.2.6 branch of the PRedis client, and had optimized it a bit so that it did fewer counts of the consistent hash array. I made the optimizations based on some results after profiling the code. I then had the master PHP process on each box repeat the test 5 times to help to average the test results.

I used dsh to start the test simultaneously on all the clients, timing how long it took the dsh process to start and finish executing. This was the amount of time it took to execute (5 repeats) * (9 client boxes) * (20, 30 or 40 client processes / box) * (10,000 requests) = X total requests. I then graphed the results below.

Graphing Requests / Second With Redis and PRedis

This ended up showing that with 9 clients, going up to 4 servers was the point of diminishing returns. Adding the fifth server (with that client box count) did not increase throughput, but rather, the throughput went down. The reason for this is most likely a combination of network interface contention on each client and higher overhead from consistent hashing. So with 9 clients, I found that the sweet spot, with my setup, was 4 servers. By adding more clients, along with more servers, the throughput would increase, instead of the decrease I was starting to see.

First, a description of my situation. I run all my email through GMail. I enjoy the interface and the fact that it is a cloud service; I can access my email seamlessly on my phone, my home computer, my work computer, some other computer, etc. However, I don’t want to lose all that information. Google is great, but who is to say that something terrible won’t happen and some (or all) of my mail is lost? So I wanted to setup some sort of backup. And then once I got that setup, make it automated.

At home, I run an Ubuntu box, that I just upgraded to 9.04, Jaunty Jackalope. This machine primarily serves as a media box, hosting video that streams to my Tivo off the 1.5TB RAID 5 array. I also use it as a network mounted TimeMachine box as well. Since I have extra storage on it, I figured I’d get something to sync my mail over IMAP periodically, and then I have a nice little backup.

After some searching, I came across two sites that had instructions using the utility mbsync (formerly isync). I found that following the instructions worked pretty well, though I had to customize the patch provided to get it to work with the version provided by Ubuntu.  And then I thought I’d detail my steps here for others to see.

1. First, enable IMAP in your GMail account.
2. Install the dependencies for mbsync:

   sudo apt-get install libc6 libdb4.2 libssl0.9.8

3. Get the source for mbsync on Ubuntu with:

   sudo apt-get source isync

   This will download the source for mbsync and create a directory (in your current directory) named isync-1.0.4 (Yes, I know it is called isync. That is a legacy name).

4. Now the normal mbsync install works just fine. However, it doesn’t support recursive directories. I use GMail labels setup to provide a hierarchy using forward slashes. It turns out that the fix to get mbsync to support directories like this is really easy to do and someone went through the trouble to get it to work. However, it doesn’t quite work on the version of the code packaged with Ubuntu. So I had to modify the patch a bit. The patch I created can be found here. Download it and put it somewhere on your system. To apply the patch, change directories into the isync-1.0.4 directory that was just downloaded and run:

   patch -p1 < /path/to/patch/file/recursive_imap_ubuntu.patch

   Once you’ve patched the directory, configure (./configure), build (make) and install (sudo make install).

5. Now you should have an executable mbsync in your path. So it is time to start preparing to do the initial sync. Choose a place to store your backups. I chose /export/backups/mail/
6. We want to connect to Google securely which means you’ll need the latest SSL certificates. To get those, use the openssl client on your machine. Run:

   openssl s_client -connect imap.gmail.com:993 -showcerts

   which should show two blocks of

   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----

   in the output. You’ll want to take each block (including the BEGIN/END CERTIFICATE lines), and put each of them into their own file. I put the first one in a file gmail.crt and the second one in the file google.crt (since the first signs imap.gmail.com which is signed by Google Internet Authority, the second certificate).

7. The second certificate, the one for the Google Internet Authority, is signed by Equifax. So we’ll need Equifax’s certificate also. An as it turns out, Ubuntu has a copy of Equifax’s certificate already sitting in the repositories. Just run

   sudo apt-get install ca-certificates

   to get the latest CA certificates. After installing the CAs, Equifax’s CA sits at /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt, which we’ll need in the configuration file in the next step.

8. Now we can write the configuration file we are going to use. Here is a copy of mine:

   IMAPAccount gmail
   Host imap.gmail.com
   User yourusername@gmail.com
   UseIMAPS yes
   CertificateFile /export/backups/mail/gmail.crt
   CertificateFile /export/backups/mail/google.crt
   CertificateFile /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt
   
   IMAPStore gmail-remote
   Account gmail
   
   MaildirStore gmail-local
   Path /export/backups/mail/gmail/
   Inbox /export/backups/mail/gmail/Inbox
   
   Channel gmail
   Master :gmail-remote:
   Slave :gmail-local:
   # Exclude everything under the internal [Gmail] folder, except the interesting folders
   Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
   Create Slave
   Sync Pull
   SyncState *

   Check out the Patterns line. That is where you would include or exclude various labels. All lables are stored at the root of the hierarchy, with the special directory [Gmail] having things like ‘Sent Mail’, ‘Spam’, ‘Starred’, etc in it. I wanted to exclude all the items in the [Gmail] directory except for the ones listed. The ‘*’ at the beginning includes all other labels. You will also want to change the Path and Inbox lines to point to your mail location, as well as the first two CertificateFile lines. Also, be sure to enter your actual GMail login on the User line. Now save this file somewhere. Note: saving it as ~/.mbsyncrc will cause it to be automatically loaded when mbsync is run, meaning you don’t need to specify which config file with the -c option.

9. Now go ahead and test it out by listing the labels in your account with the command mbsync -l -c /path/to/the/configfile.rc gmail. Running it will look like this and ask you for your password:

   [streeter@scout]:~$ mbsync -l -c ~/.mbsyncrc gmail
   Reading configuration file /home/streeter/.mbsyncrc
   Resolving imap.gmail.com... ok
   Connecting to 209.85.199.109:993... ok
   Connection is now encrypted
   Logging in...
   Password (yourusername@gmail.com@imap.gmail.com):
   Channel gmail
   lists/code
   bills
   archive/cron
   archive/classes
   archive/work
   [Gmail]/Starred
   [Gmail]/Sent Mail
   [Gmail]/All Mail
   INBOX
   @followup
   [streeter@scout]:~$

   If you see something like this, then it worked! Now just go ahead and start your first mail download with mbsync -c /path/to/the/configfile.rc gmail And then you get to wait while it finishes which can take a while depending on how much mail you have.

You can go an extra step and save your password in the file. If you add the line Pass yourpasswordhere right after the User line, you won’t be asked for your password and can then setup a cronjob to automatically keep mail locally on a schedule. However, your password for your Google Account then is stored in cleartext on your machine. So only do this is you absolutely know what you are doing and who can access the machine.

Hopefully this becomes useful for people.

To study for the finals, I worked for about 45 minutes the other day trying to get an applescript to work correctly in Adium, my AIM client. The goal was to be able to pass the applescript call, in Adium, arguments, and get a live countdown til some time. However, I was having a real hard time figuring this out. I eventually gave up, and hard coded the dates into the shell script. Which isn’t as cool and versatile as I had originally planned. In fact, it wasn’t any cooler than the one I had before, just that it worked better. Now I can specify any date, not just one that is in the current month. My current away message has been of the sorts: “4 days 11 hours 22 minutes left until summer”, the time my last final ends. So, if anybody does read this, and is also talks to me on AIM, know this: if I have my away message up, I’m probably there. I sign off when I leave (usually). And, if I’m idle, that also means I’m probably there. With Adium, you can set your own custom idle, and you can still talk to people maintaining your idle and awayness. Adium is really cool and I’d actually like to try to get into some of the development. They’re pro mac, develop an app I frequently use and enjoy, and would give me valuable experience in Cocoa, and a large project. But thats probably a while away. Anyways, time to begin studying again.