A couple of days ago, I found this piece on Digg showing how to back up Gmail messages on a local computer using fetchmail. While the instructions found there are mostly Cygwin-specific, fetchmail has its roots on Unix and those instructions can easily be adapted to Unix-based OS’s.

I decided to try it out and later found that it’s a waste of space to back up everything on my local computer. A lot of my messages on my Gmail account are from mailing lists and those are already available on the mailing list archives. What I really needed to do was to back up only important messages and discard the rest. For that, I needed procmail.

The instructions here are geared towards Gnu/Linux and specifically Ubuntu (or any Debian-based distro). I also assume that you have already enabled POP in your Gmail account. We start by installing the packages we need:

$ sudo aptitude install fetchmail procmail

That should pull in all the packages needed to run both programs. I already have postfix installed on my local computer as my SMTP server. So the instructions below are for postfix:

1. Edit /etc/postfix/main.cf and make sure the following lines are present:

   mailbox_command = procmail -a “$EXTENSION”
   home_mailbox = Mail/inbox/
   

   Save the configuration file and then restart/reload postfix. The settings above tell postfix to invoke procmail for local mail delivery instead of delivering it to your local mailbox. Your local mailbox is specified in the second configuration line above. It tells postfix that you want your mail to arrive in your ~/Main/inbox/ in Maildir format (note the trailing slash).

   Those familiar with postfix will notice that the home_mailbox option is no longer needed as mailbox_command takes care of local mail delivery. I added it there to make it easy to switch off procmail processing. Simply comment out mailbox_command and restart postfix when you want to turn of processing and delivery using procmail.

2. Create your ~/.fetchmailrc configuration file:

   poll pop.gmail.com with proto POP3 and options no dns
   user &#8216;<full GMAIL ADDRESS>&#8216; there with password &#8216;<gmail PASSWORD>&#8216; is &#8216;nimrod&#8217; here options ssl sslcertck sslcertpath &#8216;/etc/ssl/certs/&#8217;
   limit 5242880 limitflush
   

   Substitute your FULL GMAIL ADDRESS and GMAIL PASSWORD above. You may notice that it is a slightly modified version of the config file found in the lifehacker article. The second line is the one that specifies our authentication options to Gmail. In this case, I have opted to enable SSL (or TLS) authentication. The third line tells fetchmail to discard messages larger than 5MB (explained below).

   Make sure you set the correct permissions for your ~/.fetchmailrc. Do:

   $ chmod 0600 ~/.fetchmailrc
   

3. Create your ~/.procmailrc. In my case I only wanted to save messages that were directly sent to my GMail address. Anything else is discarded. So in my ~/.procmailrc I have the following (with my actual email address removed.):

   # Download only stuff directly sent to my email address…
   :0:
   * (^TO|^CC).*my\.email\.address@gmail\.com
   Mail/inbox/
   
   # Everything else goes into the bottomless pit…
   # If you don’t want unwanted emails to vanish without a trace, change /dev/null below
   # to point to another Maildir or mbox in your home directory.
   :0:
   * !(^TO|^CC).*my\.email\.address@gmail\.com
   /dev/null
   

   You will need to replace my\.email\.address@gmail\.com with your actual email address. Dots in your email address need to be escaped with a backslash if they appear in a condition line. Read more about the .procmailrc file by typing:

   $ man procmailrc
   

   To see some examples of procmail recipes:

   $ man procmailex
   

4. Now all that’s left for you to do is to create your Maildir mailbox:

   $ mkdir -p ~/Mail/inbox/{cur,new,tmp}
   

5. Run fetchmail:

   $ fetchmail
   

It should start downloading your email now. fetchmail will download almost everything. I noticed that for some reason the number of messages left to download as indicated by fetchmail do not always equal the number of messages left on the Gmail server. But you can keep polling your account by repeatedly entering fetchmail at the command line or saying:

$ fetchmail –silent –daemon 120

This will put fetchmail into daemon mode (background process) and have it poll your Gmail account every 2 minutes (120 secs).

limit and limitflush

I have setup my fetchmail configuration in such a way that very large messages are not downloaded. I have one message that is about 13MB in size and Gmail seems to close the connection near the end of the download. I found the message and it’s one of those email forwards with a very large attachment. Deleting it from my inbox did not seem to cause it to disappear in the POP3 messages seen by fetchmail. Because download fails for this message fetchmail keeps seeing this message and attempts to download it. This causes fetchmail to get stuck attempting to download this message. It seems that downloading messages larger than 10MB causes the Gmail server to timeout if your connection is slow.

The limit option sets the maximum size of a message in octets. The option I specified above sets this to 5MB. limitflush tells fetchmail to “flush” messages larger than limit. This prevents fetchmail from getting stuck with a very large message if you put it into daemon mode where you will probably have no idea what it’s doing.

You can adjust limit to your prefered message size limit or remove both options if you are sure all your messages are smaller than 10MB or if you have a very fast Internet connection.

Notes

Recently, Ubuntu seems to have switched from postfix to exim4-daemon-light as the default SMTP server. At least that’s as far as I can tell from what’s installed on my portable computer (clean install of Ubuntu Edgy Eft).

You need to follow a different set of instructions in order to use procmail if your setup uses exim4 instead of postfix.

For The “Under-privileged” Among Us

For those who do not have the luxury of having administrator or sudo privileges, worry not: you can still use procmail if your local SMTP server allows QMail-style email aliases using dot-forward or ~/.forward files. Create your ~/.procmail file as above. Then simply create a file ~/.forward with the following contents (yes, those quotes belong there):

&#8220;|IFS=&#8217; &#8216; &#038;&#038; exec /usr/bin/procmail -f- || exit 75 <username>&#8221;

Replace USERNAME above with your username.

This entry was posted on Monday, February 12th, 2007 at 10:59 am and is filed under System Administration, Ubuntu Linux, Yes, I do have a Life.. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.