Gmail and IMAP

Update 1/19/09: the Gmail IMAP semantics were updated such that marking a message is deleted is the proper way to delete messages, and not moving it to the Trash.

The fact that Gmail’s labels manifest themselves as mailboxes when accessed over IMAP results in highly unconventional IMAP behavior that client developers and users should be aware of. For instance, it’s well known that to delete a message over Gmail IMAP, the user must move the message to the “[Gmail]/Trash” mailbox rather than relying on the usual “\Deleted” flagging. But what exactly are the semantics here? What if we just copy the message to the Trash? Does trashing a message cause its labels to be lost? Is trashing from IMAP identical to trashing from the web frontend? Etc. Here are some answers I’ve gathered in putting together a simple mail filter.

Some background: IMAP servers are capable of storing flags for each message. IMAP specifies a pre-defined set of system flags that have special semantics—these include “\Seen,” “\Answered,” “\Deleted,” “\Flagged,” and so forth. (Users can also define their own custom flags, called keywords, which do not begin with backslashes—Gmail labels are similar to these. Also, flags lie outside of the familiar RFC822 representations of emails, so you won’t find them in message headers.)

Deleting a message from a mailbox usually means adding the “\Deleted” flag to a message. (Many servers automatically move the message to a Trash mailbox when it is flagged with “\Deleted,” and vice-versa.) However, in Gmail, flagging a message with “\Deleted” in a certain mailbox only removes the label represented by that mailbox. (E.g., flagging a message in “Inbox” only removes the “Inbox” label.) To truly delete a message, it must be copied into the Gmail trash mailbox (called “[Gmail]/Trash”). Furthermore, once a message is moved to this mailbox, it is immediately removed from all other mailboxes which it once belonged to, even though Gmail still retains the message’s labels internally (such that I don’t believe you can determine over IMAP the labels associated with a message that is in the Trash). When the selected mailbox is “[Gmail]/All Mail,” nothing happens, even though the server replies with an “OK” and an updated flag set that includes “\Deleted.” Try to fetch the message again (even from a different connection), and it will still be in All Mail with no “\Deleted” in its flag set.

Note that although most IMAP clients implement message “moving” by copying the message to the destination folder, then flagging the message with “\Deleted,” inspecting the message in the Trash via Gmail’s web frontend reveals that it retains all labels. This includes the one corresponding to the mailbox from which it was removed (even if it was the “Inbox”); I’m guessing this is due to the ordering of these two operations (the COPY operation coming first removes the message from the mailbox and thus preempts the subsequent STORE +FLAGS operation). This is not the same as what happens when the message is moved to Trash via the web frontend: if the message was in the Inbox, deleting it removes sends it to the Trash without its Inbox label. However, other labels are retained. To further confound the situation, restoring a message in the web frontend is “Move to Inbox,” even though the message may not have originated from the Inbox.

Similarly, the IMAP COPY command does not actually create a copy of a message when the destination is a separate folder, but instead adds the corresponding label to the message. Future modifications to the message from either mailbox affect the same message.

The Recommended IMAP client settings in Gmail’s documentation suggest configuring clients (at least Thunderbird) such that “When I delete a message > Mark it as deleted.” Since this means adding the “\Deleted” flag to a message, this only has the effect of removing the label represented by this mailbox, whereas the user likely wanted to have the message moved to the Trash so that it will eventually be purged. A message that is removed this way from the Inbox, for instance, will become indistinguishable from an Archived message. A slightly better solution is to leave Thunderbird’s setting to its default (“Move it to the Trash folder”), so that the messages are isolated and one can later perform the actual deletion from within Gmail. To actually have deleted messages moved to the “[Gmail]/Trash” mailbox, follow these instructions.

It’s unclear why Gmail chose to represent labels with mailboxes rather than keywords; this may have been attributable to usability results (most users may simply prefer the more familiar notion of mailboxes/folders when working over IMAP) and mainstream client support for keywords (or lack thereof).

Naming

Gmail IMAP uses a naming scheme wherein labels/mailboxes that have system names (Drafts, All Mail, etc.) are isolated in distinct namespaces. I don’t know the reason why this namespace isolation is desirable, but here’s how it seems to work. The Gmail “system labels” all appear as sub-mailboxes of a “[Gmail]” mailbox when viewed over IMAP:

  • [Gmail]/Drafts
  • [Gmail]/Sent Mail
  • [Gmail]/All Mail
  • [Gmail]/Spam
  • [Gmail]/Starred
  • [Gmail]/Trash

Things are reversed when viewed from the Gmail web frontend. The system labels are not prefixed with “[Gmail]/”, but IMAP mailboxes that have the same names as any of the system labels are prefixed with “[Imap]/”, e.g. “[Imap]/Trash.” What if you try to create a label called “Trash”? You can’t—Gmail gives you the error, “System specific names are not allowed. Please try another name.”

The one exception to all this is the “Inbox” system label/mailbox, whose name is just “Inbox” from either IMAP or the Gmail web frontend.

Otherwise, normal (non-system) labels and their corresponding IMAP mailbox share identical names. Subdirectories correspond to labels of the full path (using the “/” path separator).

See Also

The amount of detailed information on Gmail IMAP is surprisingly sparse.

And, of course, details about the IMAP4 revision 1 protocol can be found in RFC 3501.

Follow me on Twitter for stuff far more interesting than what I blog.

  • Tom

    I really dislike Gmail’s IMAP “solution”. The tags system may work well in their web interface, but it’s unwieldy in IMAP.

  • Tom

    I really dislike Gmail’s IMAP “solution”. The tags system may work well in their web interface, but it’s unwieldy in IMAP.

  • Chris R.

    I see no problem w/ Gmail’s /Deleted behavior and T-bird configuration recommendation. In the Gmail model of nearly unlimited storage, there’s virtually no reason to actually delete a message, so they’re optimizing for the common case of removing labels vs. the infrequent case of actually deleting a message, which seems perfectly reasonable to me.

  • Chris R.

    I see no problem w/ Gmail’s /Deleted behavior and T-bird configuration recommendation. In the Gmail model of nearly unlimited storage, there’s virtually no reason to actually delete a message, so they’re optimizing for the common case of removing labels vs. the infrequent case of actually deleting a message, which seems perfectly reasonable to me.

  • Pingback: Web 2.0 Announcer()

  • iquanyin

    my problem is this: gmail is trashing messages i put into “all mail” (randomly trashing, doesnt always do it) and it’s also replacing–in the trash–mail i have deleted, over and over.

    in fact, that’s how i come to be reading this. i’ve read the threads on both issues over at the goole help groups. they’re open. no support there. you’d think this would be high priority.

    the trash issues happen on iphone in native mail client, on phone using gmail’s web client via both the gmail app (appstore) and safari, and using desktop pc as well. i’m frustrated. i’ve tried both options of configuring trash behavior offered for mobile users, tried switching to pop, then back to imap (might as well). i’ve read google’s riff on labels vs. folders and up to now thought labels was a fine idea–flexible, spacesaving. not so sure at the moment. anyway, thanks for your details! might help me figure out a way to fix (or minimize) the problem. today: deciding how to best back up mail (i use zoho plus all the big players, but have no pc of my own. webmail’s my sole resource right now).

  • iquanyin

    my problem is this: gmail is trashing messages i put into “all mail” (randomly trashing, doesnt always do it) and it’s also replacing–in the trash–mail i have deleted, over and over.

    in fact, that’s how i come to be reading this. i’ve read the threads on both issues over at the goole help groups. they’re open. no support there. you’d think this would be high priority.

    the trash issues happen on iphone in native mail client, on phone using gmail’s web client via both the gmail app (appstore) and safari, and using desktop pc as well. i’m frustrated. i’ve tried both options of configuring trash behavior offered for mobile users, tried switching to pop, then back to imap (might as well). i’ve read google’s riff on labels vs. folders and up to now thought labels was a fine idea–flexible, spacesaving. not so sure at the moment. anyway, thanks for your details! might help me figure out a way to fix (or minimize) the problem. today: deciding how to best back up mail (i use zoho plus all the big players, but have no pc of my own. webmail’s my sole resource right now).

  • If you ever used Facebook or shopped on-line since Fall of 2009, you may be infected! It seems like Santa Claus has determined that every user on Facebook is naughty, and for Holiday they are all receiving a personal computer virus compared to the regular lump of coal. The spammers are at it once more on the well-liked Facebook social network. In respect to Panda Security, the new personal computer virus assault will involve the posting of a movie on a user?s wall posts by ***SantA*** and is implied to be a Christmas greeting. When you click on the phony video clip player, this computer gets contaminated with a variant of the Koobface worm, Koobface.GK. Click here or go to http://registrycleanerwindows7.com now for a FREE scan to determine if your system is infected!

  • Yes, gmail's imap is not nice for me either

  • Google just added a new feature in Google Labs for advanced IMAP controls. Access it by going to 'settings' in Gmail and clicking on 'labs':

    Choose which labels show up in IMAP, turn off message auto-expunging, or trash messages when they're deleted from IMAP

  • asansorkiralama

    very good.. 🙂

  • Have you updated this post with the new Google updates?

  • I too have problems with gmails IMAP. Glad I'm not alone.

  • Blodman

    Thank you so much for this information, I see you guys put a lot of work into this blog, I enjoy every post here and learn… thank you Admin.

  • I do not have too much information about it. However , it looks that i can use it for my job. Anyway , if I have a problem , I post a new one 🙂

  • ashokeban

    Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that “The content of your post is awesome” Great work.

    Addiction Treatment
    Addiction Treatment Florida
    Drug Rehab

  • I really dislike Gmail’s IMAP “solution”.

  • gmail’s imap is not nice for me either

  • Anonymous

    Thank you for sharing this information. The article you posted is very awesome. I found the post very useful as well as interesting. I will come back to read some more.
    Movers

  • Riventiger

    Thanks for the information its great to see places like this with such interesting reading material.

    Regards   
    registry cleaner guru
    http://registryscleaner.com

  • BradlyLeasor

    A good study in
    Gmail and IMAP. But would you please tell me, how do you do your study before writing this information? Maybe do you learn from other references, doing some surveys, researches or interviewing people?

    Thanks and regards,

    Bradly Leasor
    Tweet Adder

  • Niels

    I created separate account subfolders in my
    [Gmail]/Drafts[Gmail]/Sent Mail

    which resulted in gmail showing them as:
    [Gmail]/Drafts/address1.com[Gmail]/Drafts/address2.com[Gmail]/Sent Mail/address1.com[Gmail]/Sent Mail/address2.com… and as a result were unreadable in the small sidebar because of the long (and ugly) label names.

    I expected the usual behavior like so:
    [Gmail]
        Drafts
              address1.com
              address2.com
        Sent

              address1.com

              address2.com

    I solved the issue by adding a NEW label [Gmail]; afterwards it reorded the subfolders correctly.

  • Which blog you have shared really informative and interesting , I like it .