Monthly Archives: May 2008
Python Commons release
UDP Prober release
Mailing List Filter, Python Commons releases
Gmail and IMAP
Update 1/19/09: the Gmail IMAP semantics were updated such that marking a message is deleted is the proper way to delete messages, and not moving it to the Trash.
The fact that Gmail’s labels manifest themselves as mailboxes when accessed over IMAP results in highly unconventional IMAP behavior that client developers and users should be aware of. For instance, it’s well known that to delete a message over Gmail IMAP, the user must move the message to the “[Gmail]/Trash” mailbox rather than relying on the usual “\Deleted” flagging. But what exactly are the semantics here? What if we just copy the message to the Trash? Does trashing a message cause its labels to be lost? Is trashing from IMAP identical to trashing from the web frontend? Etc. Here are some answers I’ve gathered in putting together a simple mail filter.
Some background: IMAP servers are capable of storing flags for each message. IMAP specifies a pre-defined set of system flags that have special semantics—these include “\Seen,” “\Answered,” “\Deleted,” “\Flagged,” and so forth. (Users can also define their own custom flags, called keywords, which do not begin with backslashes—Gmail labels are similar to these. Also, flags lie outside of the familiar RFC822 representations of emails, so you won’t find them in message headers.)
Deleting a message from a mailbox usually means adding the “\Deleted” flag to a message. (Many servers automatically move the message to a Trash mailbox when it is flagged with “\Deleted,” and vice-versa.) However, in Gmail, flagging a message with “\Deleted” in a certain mailbox only removes the label represented by that mailbox. (E.g., flagging a message in “Inbox” only removes the “Inbox” label.) To truly delete a message, it must be copied into the Gmail trash mailbox (called “[Gmail]/Trash”). Furthermore, once a message is moved to this mailbox, it is immediately removed from all other mailboxes which it once belonged to, even though Gmail still retains the message’s labels internally (such that I don’t believe you can determine over IMAP the labels associated with a message that is in the Trash). When the selected mailbox is “[Gmail]/All Mail,” nothing happens, even though the server replies with an “OK” and an updated flag set that includes “\Deleted.” Try to fetch the message again (even from a different connection), and it will still be in All Mail with no “\Deleted” in its flag set.
Note that although most IMAP clients implement message “moving” by copying the message to the destination folder, then flagging the message with “\Deleted,” inspecting the message in the Trash via Gmail’s web frontend reveals that it retains all labels. This includes the one corresponding to the mailbox from which it was removed (even if it was the “Inbox”); I’m guessing this is due to the ordering of these two operations (the COPY operation coming first removes the message from the mailbox and thus preempts the subsequent STORE +FLAGS operation). This is not the same as what happens when the message is moved to Trash via the web frontend: if the message was in the Inbox, deleting it removes sends it to the Trash without its Inbox label. However, other labels are retained. To further confound the situation, restoring a message in the web frontend is “Move to Inbox,” even though the message may not have originated from the Inbox.
Similarly, the IMAP COPY command does not actually create a copy of a message when the destination is a separate folder, but instead adds the corresponding label to the message. Future modifications to the message from either mailbox affect the same message.
The Recommended IMAP client settings in Gmail’s documentation suggest configuring clients (at least Thunderbird) such that “When I delete a message > Mark it as deleted.” Since this means adding the “\Deleted” flag to a message, this only has the effect of removing the label represented by this mailbox, whereas the user likely wanted to have the message moved to the Trash so that it will eventually be purged. A message that is removed this way from the Inbox, for instance, will become indistinguishable from an Archived message. A slightly better solution is to leave Thunderbird’s setting to its default (“Move it to the Trash folder”), so that the messages are isolated and one can later perform the actual deletion from within Gmail. To actually have deleted messages moved to the “[Gmail]/Trash” mailbox, follow these instructions.
It’s unclear why Gmail chose to represent labels with mailboxes rather than keywords; this may have been attributable to usability results (most users may simply prefer the more familiar notion of mailboxes/folders when working over IMAP) and mainstream client support for keywords (or lack thereof).
Gmail IMAP uses a naming scheme wherein labels/mailboxes that have system names (Drafts, All Mail, etc.) are isolated in distinct namespaces. I don’t know the reason why this namespace isolation is desirable, but here’s how it seems to work. The Gmail “system labels” all appear as sub-mailboxes of a “[Gmail]” mailbox when viewed over IMAP:
- [Gmail]/Sent Mail
- [Gmail]/All Mail
Things are reversed when viewed from the Gmail web frontend. The system labels are not prefixed with “[Gmail]/”, but IMAP mailboxes that have the same names as any of the system labels are prefixed with “[Imap]/”, e.g. “[Imap]/Trash.” What if you try to create a label called “Trash”? You can’t—Gmail gives you the error, “System specific names are not allowed. Please try another name.”
The one exception to all this is the “Inbox” system label/mailbox, whose name is just “Inbox” from either IMAP or the Gmail web frontend.
Otherwise, normal (non-system) labels and their corresponding IMAP mailbox share identical names. Subdirectories correspond to labels of the full path (using the “/” path separator).
The amount of detailed information on Gmail IMAP is surprisingly sparse.
- Some notes
- A rant with some details
And, of course, details about the IMAP4 revision 1 protocol can be found in RFC 3501.
ExoTech and exokernels
There was a startup created based on the ideas that came out of the exokernel research in the mid-90’s called ExoTech. They tried to make high-performance video-on-demand server appliances. I’m not sure what happened, but they didn’t take off.
As for exokernels themselves, the dramatic performance gains that these systems yielded back then were gradually dampened by evolutionary optimizations to mainstream operating systems. That, along with the established use of mainstream systems, and with the myriad real and complex application workloads that don’t yield the same idealized performance graphs as those found in various papers, has largely cast exokernels into the realm of academia, despite other arguable advantages including flexibility, reliability, security, and so forth. Their fate is one that is mostly shared by microkernels.
Thanks to Russ for this discussion.
The sql.mit.edu service performs nightly
/mit/sql/backup/. This could have spared me some grief when I wiped out my database.
sipb-xen and Athena 10
SIPB has a nifty new project called sipb-xen, wherein users can easily create and use virtual machines. This has the potential to be released as a general-purpose open-source tool for virtual machine management, as the only other open-source “competition” at this point appears to be Red Hat’s Virtual Machine Manager. About 10 SIPB members are currently actively working on this.
debathena is also the future of Athena.