Category Archives: computing

256-color xterm

I somehow managed to never discover this until today. I was chatting with Ryan about myriad topics from spatial programming abstractions to distributed system consistency, but by far the bit of information that would most immediately and dramatically change my life forever was xterm-256color. Simply export TERM=xterm-256color, and bam—you now have a rainbow of colors in Vim and Emacs. It should work right out of the box these days—it works for me in GNOME Terminal and Putty. Fantastic.

Recovering files using TestDisk

I recently had a hard disk go bad. It had a single NTFS partition where the partition was no longer recognized by Windows or Linux.

I managed to recover (I think) all of my files by reading the data off of the disk using TestDisk. It discovered my partition without issue, and the actual file recovery went relatively smoothly.

I believe the way TestDisk was primarily designed to work was by attempting to actually repair the partition. This should be fine if you’re working on a copy of the raw disk bits (you can probably use dd for Windows for this, or write a program that accesses ), but I’d generally be averse to making any further changes to a bad disk. I chose to recover my data just by copying the files directly out. You can do this as follows:

  • go to (your disk) > Intel > Analyse > (your dynamic partition)
  • highlight your partition
  • press P to list files
  • press H to hide deleted files
  • press C to copy the current directory (the directory from which you started TestDisk)

One thing to watch out for when using this approach is to first make sure you press H to hide deleted files. Otherwise, you end up recovering deleted files as well, and sometimes these deleted files (for whatever reason—probably because my data’s sufficiently corrupted) end up pointing back into the root of the file system, leading to cycles in the tree and thus an infinitely recursive file copying procedure.

Re-enabling desktop effects in Windows 7

Every once in a while, Windows 7’s Desktop Window Manager (DWM) spontaneously goes funky on me and loses its effects (transparency, blurring, shadows).

When this happens, you can get your effects back by going to Services and restarting the “Desktop Window Manager Session Manager” service, or by going to an Administrative command prompt and running:

net stop uxsms
net start uxsms

Windows 7 non-impressions

I’ve been using the free beta release of Windows 7 on my Lenovo X41 Tablet, and during this time I’ve found that I really don’t need that many local applications installed. It’s nice and simple, and everything runs smoothly and quickly (most notably, suspend/hibernate). No IBM/Lenovo software accessories or manually installed device drivers to set up or get in the way. I think the only thing which I could imagine missing is the IBM accelerometer-based hard disk protector.

I installed the system by blindly following these instructions to create a bootable USB from the DVD ISO under Windows XP (most instructions assume Windows Vista or Windows 7). The only hitch during setup was that the system doesn’t have out-of-box support for my Intel PRO/Wireless 2200BG network adapter; I needed to run Windows Update over an Ethernet cable first. Another annoyance, but unrelated to Windows 7, was that I had to defragment my (NTFS-formatted) partition some ten times over before I could ntfsresize it (with an Ubuntu live USB) to make space for Windows 7.

Here’s the shortlist of applications that I actually installed:

  • Chrome: “mostly-FLOSS” browser of choice
  • Flash: not FLOSS, but your alternative is to live under a rock
  • PuTTY: standard SSH client
  • Sumatra PDF: a PDF viewer that’s much lighter weight than Acrobat

And that’s it! No Java. No IM or email client—these run on my main desktop, but even if they didn’t, I’d just use Gmail/Meebo/etc. No IDE or dev tools—I do most of my development over SSH. No office suite—I just don’t use these applications much anymore, and probably also because I’ve moved most of my document authoring to LaTeX or Pandoc or what have you. I imagine I’ll be crawling back to Powerpoint 2007 once I have to crank out another presentation, though—it’s just easy, fast, and pretty.

I didn’t even need to download the Consolas font, since it’s included in Windows 7.

Over time, I can imagine myself also installing the following:

  • Filezilla: use it mainly for SFTP
  • TightVNC: so I can connect to my desktop for apps like Thunderbird
  • VideoLAN Client: because it somehow manages to play any video I throw at it, but until I actually need to watch a movie on my laptop, I usually find myself watching Hulu and Youtube
  • Vuze: downloading media wirelessly may be safer, and Vuze is still the best BT client with features that matter (faster downloads, resilient tracking, search); goes well with PeerGuardian

I thought I’d have more to say about Windows 7 by this point, but I honestly still haven’t experienced it that deeply. (The fact that it’s out of my way is just as well a Good Thing, but where’s the fun in that?) That, plus the fact that I don’t have many local apps, lets me realize first-hand that the significance of the laptop/netbook OS is becoming marginalized, and that Chrome OS/Android/Linux really could become relevant. The paradox is that native mobile device apps—like those written for the iPhone OS—are popular and preferred over their web counterparts, and I think that has more to do with generally less-connected usage of these devices and the not-quite-as-slick UI of most web apps.

I elected to try Windows 7 primarily for fun, but also to keep at least some part of myself familiar with the dominant desktop OS of the world, and lastly because it just has better tablet support, battery life, and suspend/hibernate compared to my prior experiences with Linux on the laptop.

When you release shift too early

$ diff /tmp/list.txt ~/Desktop/list.txt
$ rm !!;1
rm: cannot remove `diff': No such file or directory
bash: 1: command not found

The semi-colon should’ve been a colon, which selects just word 1 from the previous line. Instead, I deleted everything. Of course that’s what I intended to do, you piece of crap.

I usually use trash-cli and don’t bother deleting most files in /tmp. duplicity helped me get my files back, thankfully.

While on this: freedesktop.org’s trash specification leaves a few things to be desired:

  • Consistent support for trashing on mounts besides the one containing your home directory.
  • Automatic erasing of old files on space depletion.

Practical media downloading

A random assortment of tips for getting stuff with some mix of expedience, safety, and broad coverage.

  • For single tracks, start with Skreemr. It doesn’t manage to find that much, unfortunately, but it’s a quick check. Other similar services include BeeMP3 and SeeqPod, which have (IMO) clumsier UIs. Once you’ve checked with Skreemr, just try Googling for the artist, title, and “mp3” – this usually works, esp. for popular tracks, but is less slick than Skreemr since you’ll probably land at a site where you need to satisfy some captcha, run a gauntlet of ads or gaudy web designs, wait some number of seconds, find a broken link, and retry with the next few hits.
  • Alternatively, download full albums. Even if you’re after just a single track, if you can’t find it, you may be able to find its album. You can do this by Googling for the artist and album name alongside the names of popular “private” file-sharing services (“rapidshare | megaupload | …”). These file hosting services are actually good for finding large media files in general, including movies and TV shows. I have a little script for making these kinds of searches.
  • You can always resort to BitTorrent or a file-sharing network if you’re feeling a bit more promiscuous. If you’re on Windows, one of the better file-sharing clients is Shareaza, which supports Gnutella, ed2k, and “Gnutella2.” For BitTorrent, Shareaza’s support wasn’t that great when I tried. I’ve heard good things about uTorrent, which is probably the most popular client on Windows, but I’ve only ever used the cross-platform Vuze (formerly Azureus). Vuze sports an awesome search interface that lets you breezily grab torrents from some of the popular sites out there including btjunkie and mininova.
  • When engaging in promiscuous file-sharing, try to operate from a WLAN for which logs are not kept (e.g., StataCenter). For a general public WLAN, you can fiddle with your MAC address and minimize concurrent traffic from the same host (esp. anything that could identify you).
  • You can use an IP filter (“firewall”) to prevent communication with certain hosts. BlueTack maintains popular blocklists. A paper from UCR called “P2P: Is Big Brother Watching You?” (Ars Technica article) concludes that users will exchange data with blocklisted users 100% of the time, and that blocking just 5 IPs reduces this to 1%.
  • Blocklist managers are optimized to filter large numbers/ranges of IPs. PeerGuardian 2 and moblock are popular blocklist managers for Windows and Linux, respectively.
  • Should you be simply unable to find the song anywhere but a streaming source, such as Songza, Last.fm, MySpace, the artist’s website, etc., then just capture your system audio output while playing it from the streaming source. (Streaming audio quality tends to be lower, though.)
  • Find iTunes shares and public file shares (CIFS/SMB, FTP, etc.) on your LAN. Might work well if you’re in something like a dorm or frat setting, depending on the network configuration. I don’t know if there’s some working continuation of myTunes or if ourTunes still works, but those are places to start looking for iTunes pulling.
  • For TV shows, another possibility is to just watch them streamed from the web. You can find a lot on YouTube (though certain videos might just be around for short windows of time). Sidereel is a community that organizes links to these videos into shows and episodes.

Updated 7/21/2009: added note on Googling for single tracks.

Google Chrome first impressions

Google Chrome looks promising, having innovated in a number of fundamental areas – performance, stability/security, Gears, clean UI, etc. Features I’d most like to see:

  • Ability to install custom Certificate Authorities, Client Certificates, and so on.
  • Keyboard breeziness: a separate type-as-you-find for links only, single-key shortcuts, spatial link navigation, etc.
  • Ability to disable the Most Visited panel in newly created tabs (or have manually set panels).
  • Proper handling of web application-specific shortcut keys (e.g. the ctrl-shift-left/right tab-switching in Remember The Milk).

Less urgently, I’m also waiting for Chrome to get extensions, or at least many more features: profile synchronization, ad-blocking, bookmark service integration, custom styling/scripting, gestures, developer/debugging tools, etc.

Gmail and IMAP

Update 1/19/09: the Gmail IMAP semantics were updated such that marking a message is deleted is the proper way to delete messages, and not moving it to the Trash.

The fact that Gmail’s labels manifest themselves as mailboxes when accessed over IMAP results in highly unconventional IMAP behavior that client developers and users should be aware of. For instance, it’s well known that to delete a message over Gmail IMAP, the user must move the message to the “[Gmail]/Trash” mailbox rather than relying on the usual “\Deleted” flagging. But what exactly are the semantics here? What if we just copy the message to the Trash? Does trashing a message cause its labels to be lost? Is trashing from IMAP identical to trashing from the web frontend? Etc. Here are some answers I’ve gathered in putting together a simple mail filter.

Some background: IMAP servers are capable of storing flags for each message. IMAP specifies a pre-defined set of system flags that have special semantics—these include “\Seen,” “\Answered,” “\Deleted,” “\Flagged,” and so forth. (Users can also define their own custom flags, called keywords, which do not begin with backslashes—Gmail labels are similar to these. Also, flags lie outside of the familiar RFC822 representations of emails, so you won’t find them in message headers.)

Deleting a message from a mailbox usually means adding the “\Deleted” flag to a message. (Many servers automatically move the message to a Trash mailbox when it is flagged with “\Deleted,” and vice-versa.) However, in Gmail, flagging a message with “\Deleted” in a certain mailbox only removes the label represented by that mailbox. (E.g., flagging a message in “Inbox” only removes the “Inbox” label.) To truly delete a message, it must be copied into the Gmail trash mailbox (called “[Gmail]/Trash”). Furthermore, once a message is moved to this mailbox, it is immediately removed from all other mailboxes which it once belonged to, even though Gmail still retains the message’s labels internally (such that I don’t believe you can determine over IMAP the labels associated with a message that is in the Trash). When the selected mailbox is “[Gmail]/All Mail,” nothing happens, even though the server replies with an “OK” and an updated flag set that includes “\Deleted.” Try to fetch the message again (even from a different connection), and it will still be in All Mail with no “\Deleted” in its flag set.

Note that although most IMAP clients implement message “moving” by copying the message to the destination folder, then flagging the message with “\Deleted,” inspecting the message in the Trash via Gmail’s web frontend reveals that it retains all labels. This includes the one corresponding to the mailbox from which it was removed (even if it was the “Inbox”); I’m guessing this is due to the ordering of these two operations (the COPY operation coming first removes the message from the mailbox and thus preempts the subsequent STORE +FLAGS operation). This is not the same as what happens when the message is moved to Trash via the web frontend: if the message was in the Inbox, deleting it removes sends it to the Trash without its Inbox label. However, other labels are retained. To further confound the situation, restoring a message in the web frontend is “Move to Inbox,” even though the message may not have originated from the Inbox.

Similarly, the IMAP COPY command does not actually create a copy of a message when the destination is a separate folder, but instead adds the corresponding label to the message. Future modifications to the message from either mailbox affect the same message.

The Recommended IMAP client settings in Gmail’s documentation suggest configuring clients (at least Thunderbird) such that “When I delete a message > Mark it as deleted.” Since this means adding the “\Deleted” flag to a message, this only has the effect of removing the label represented by this mailbox, whereas the user likely wanted to have the message moved to the Trash so that it will eventually be purged. A message that is removed this way from the Inbox, for instance, will become indistinguishable from an Archived message. A slightly better solution is to leave Thunderbird’s setting to its default (“Move it to the Trash folder”), so that the messages are isolated and one can later perform the actual deletion from within Gmail. To actually have deleted messages moved to the “[Gmail]/Trash” mailbox, follow these instructions.

It’s unclear why Gmail chose to represent labels with mailboxes rather than keywords; this may have been attributable to usability results (most users may simply prefer the more familiar notion of mailboxes/folders when working over IMAP) and mainstream client support for keywords (or lack thereof).

Naming

Gmail IMAP uses a naming scheme wherein labels/mailboxes that have system names (Drafts, All Mail, etc.) are isolated in distinct namespaces. I don’t know the reason why this namespace isolation is desirable, but here’s how it seems to work. The Gmail “system labels” all appear as sub-mailboxes of a “[Gmail]” mailbox when viewed over IMAP:

  • [Gmail]/Drafts
  • [Gmail]/Sent Mail
  • [Gmail]/All Mail
  • [Gmail]/Spam
  • [Gmail]/Starred
  • [Gmail]/Trash

Things are reversed when viewed from the Gmail web frontend. The system labels are not prefixed with “[Gmail]/”, but IMAP mailboxes that have the same names as any of the system labels are prefixed with “[Imap]/”, e.g. “[Imap]/Trash.” What if you try to create a label called “Trash”? You can’t—Gmail gives you the error, “System specific names are not allowed. Please try another name.”

The one exception to all this is the “Inbox” system label/mailbox, whose name is just “Inbox” from either IMAP or the Gmail web frontend.

Otherwise, normal (non-system) labels and their corresponding IMAP mailbox share identical names. Subdirectories correspond to labels of the full path (using the “/” path separator).

See Also

The amount of detailed information on Gmail IMAP is surprisingly sparse.

And, of course, details about the IMAP4 revision 1 protocol can be found in RFC 3501.

Totally shady: AcetoneISO

AcetoneISO is popular for its ability to e.g. work with DAA files, and usually suggested as an alternative to PowerISO for Linux (the free version of which has limitations, such as max file size).

But I was looking at Acetone’s convert script and found that it just downloads and runs a poweriso binary, which apparently must have the limitations removed (based on the experiences shared in forums).

Whatever, it’s useful. DAA has no reason to exist anyway.