Category Archives: web development

Functional reactive programming for the web, or: where’s my Lunascript?!

Update: there’s some good discussion on Reddit.

Even when using the latest and greatest frameworks and disciplines, writing fast, highly interacting web applications involves a lot of accidental complexity:

First you need server code to figure out what data the browser needs. Hopefully you have an ORM layer, but you still need to carefully structure your code to minimize your backend dispatches, and you need to carefully keep that in sync with your front-end code lest you don’t fetch enough data or hurt performance by fetching too much. If it’s a Web 2.0-style app, you re-implement a ton of that server-side code in JavaScript, once for creating the page and then again as controller code for keeping it up to date and consistent. And when the user changes something, you bottle that up — typically in a custom over-the-wire format — and send it as an XHR to the server. The server has to de-serialize it into different data structures in a different language, notify the persistence store, figure out what other clients care about the change, and send them a notification over your Comet pipe, which is handled by yet more JavaScript controller code. Offline support? More code.

Strike a chord? That’s the motivation setting the stage for [Lunascript], a reactive programming framework for web apps. What?

Continue reading

Making sense of OpenID, OAuth, OpenSocial, Google Friend Connect, Facebook Connect, and more

Last Thursday I dropped in to the Google SIPB hackathon, where I got a chance to chat with several Googlers in the Cambridge office about the whole ecosystem of decentralized identity and social networking services. I had actually previously spent a bit of time searching for a high-level map laying out how these various services related to each other, strictly out of curiosity, but never really found anything that was succinct, clear, and free of BS. There also seems to be a lot of contradictory information and general confusion. Along with the recent news expecting similar service stacks from Twitter, it seems timely to share all the things I’ve been learning.

The executive summary:

  • OpenID: authentication; use one login across many sites
  • OpenID Attribute Exchange: a key-value store protocol for OpenID
  • OAuth: authorization and authentication; grant sites specific access to your account
  • OAuth WRAP: a simpler OAuth that leverages PKI instead of its own signature scheme
  • OpenSocial: a standard API into social networks
  • Google Friend Connect: an OpenSocial router, plus a bunch of other less-important stuff
  • Facebook Platform: all the above (and a bit more), for the Facebook stack
  • Facebook Connect: establish links between Facebook and third-party user account systems
  • Portable Contacts: just the slice of OpenSocial that deals with contacts

Continue reading

Web Sockets tutorial with simple Python server

The Landscape: HTML5

HTML5 is an emerging and in-flux client-side standard for developing web applications. It’s really more of a rich client platform specification than just a markup language, including the following slew of new features:

  • canvas for vector graphics
  • video and audio for multimedia
  • local offline storage
  • drag and drop operations
  • Web Socket API for bidirectional client-server communications
  • GeoLocation API
  • standard WYSIWYG HTML editor component
  • Web Workers API for message-passing processes
  • webcam and microphone access
  • 3D graphics rendering engine
  • and more…

A lot of this effort is about wrapping up and building into browsers native support for various (proprietary) technologies already in widespread use on the Web, such as Flash video/webcam/mic and Google Gears offline/drag-and-drop. Others are about cleaning up things for which there currently exist various hacks, and Web Sockets fall into this category.

Introducing Web Sockets

This Chromium blog post contains a nice introduction to Web Sockets:

The Web Sockets API enables web applications to handle bidirectional communications with server-side process in a straightforward way. Developers have been using XMLHttpRequest (“XHR”) for such purposes, but XHR makes developing web applications that communicate back and forth to the server unnecessarily complex. XHR is basically asynchronous HTTP, and because you need to use a tricky technique like long-hanging GET for sending data from the server to the browser, simple tasks rapidly become complex. As opposed to XMLHttpRequest, Web Sockets provide a real bidirectional communication channel in your browser. Once you get a Web Socket connection, you can send data from browser to server by calling a send() method, and receive data from server to browser by an onmessage event handler. A simple example is included below.

In addition to the new Web Sockets API, there is also a new protocol (the “web socket protocol”) that the browser uses to communicate with servers. The protocol is not raw TCP because it needs to provide the browser’s “same-origin” security model. It’s also not HTTP because web socket traffic differers from HTTP’s request-response model. Web socket communications using the new web socket protocol should use less bandwidth because, unlike a series of XHRs and hanging GETs, no headers are exchanged once the single connection has been established. To use this new API and protocol and take advantage of the simpler programming model and more efficient network traffic, you do need a new server implementation to communicate with — but don’t worry. We also developed pywebsocket, which can be used as an Apache extension module, or can even be run as standalone server.

(The mentioned technique of a long-hanging GET is also known as Comet.)

Chrome is presently the only browser that has Web Sockets, and only in the Dev channel releases. Firefox and Safari/WebKit support are under way, according to the implementation status page.

The Web Sockets Protocol

The protocol has the client and server do an HTTP-style handshake, where all text is in UTF-8 and newlines include a carriage return and newline. After this, arbitrary data can be sent back and forth, but delimited in frames, which begin with a 0x00 byte and end with a 0xff byte. Contrast this with the byte stream abstraction presented by raw TCP—having the system hand you whole frames frees the application from having to manually buffer and parse messages out of the stream (which the browser may be able to do more efficiently).

As for how this mixes with browser security policies, the basic gist is that the same-origin policy no longer applies. Requiring the Web Socket to communicate only with the same origin (same host and port as where the HTML/Javascript came from) would be a barrier to deployment because it would require the httpd to additionally speak Web Sockets. (That said, the default port for Web Sockets is in fact port 80.) More generally, this prevents all cross-site communication, which is critical to many classes of applications such as mash-ups and widget dashboards.

But the protocol does require the browser to send the origin information to the server, the server to validate this by echoing the origin, and finally the client to validate that the server echoed this. According to the protocol specs, the response must include the exact same origin, location, and protocol as the request, where:

  • the origin is just the (protocol, host, port) triplet (,
  • the location is the target of the request (ws://, and
  • the protocol is an arbitrary string used to identify the exact application-level protocol expected.

(Note that the origin is different from the Referrer, which includes the full resource path, thus leading to privacy concerns. I hope to write more on the Origin header in a broader context and client-side web security in general soon.)

Example Client and Server

To give you a flavor of how to write a complete end-to-end web application using Web Sockets, the following is a simple client and server application where the server sends two messages down to the client, “hello” and “world.” This example is from my sandbox.

The client-side API for Web Sockets is very simple. The example client just connects to a server on port 9876 and alerts the user of each new message. Just to make this a wholesome HTML5 experience, we’ll write everything in XHTML5 (yes, there exists a stricter, XML flavor of HTML5 for those who preferred XHTML over HTML tag soup):

<!DOCTYPE html>
<html xmlns="">
    <title>Web Socket Example</title>
    <meta charset="UTF-8">
      window.onload = function() {
        var s = new WebSocket("ws://localhost:9876/");
        s.onopen = function(e) { alert("opened"); }
        s.onclose = function(e) { alert("closed"); }
        s.onmessage = function(e) { alert("got: " +; }
      <div id="holder" style="width:600px; height:300px"></div>

Now for the server, which is written in Python. It sends the two messages after a one-second delay before each. Note that the server is hard-coding the response to expect a locally connected client; this is for simplicity and clarity. In particular, it requires that the client is being served from localhost:8888. A real server would parse the request, validate it, and generate an appropriate response.

#!/usr/bin/env python

import socket, threading, time

def handle(s):
  print repr(s.recv(4096))
HTTP/1.1 101 Web Socket Protocol Handshake\r
Upgrade: WebSocket\r
Connection: Upgrade\r
WebSocket-Origin: http://localhost:8888\r
WebSocket-Location: ws://localhost:9876/\r
WebSocket-Protocol: sample
  '''.strip() + '\r\n\r\n')

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 9876));
while 1:
  t,_ = s.accept();
  threading.Thread(target = handle, args = (t,)).start()

To run the above, start the Web Socket server (./ and start a web server on port 8888 serving index.html:

./ &
python -m SimpleHTTPServer 8888

Further Exploration

For a more complete application (that’s still reasonably simple), I threw together a real-time X-Y scatter plotting application called Real-Time Plotter. It plots some number of data sources and supports streaming to multiple clients.

The Python server listens for data sources on port 9876. It expects a stream of text, where the first line is the name of the data source and each subsequent line contains a space-separated x-y pair of floating point numbers in the series to be plotted. It listens also on port 9877 for Web Socket clients. A simple data source that issues a random y-value per second can be started from the shell using netcat:

  echo 'my data source'
  while true ; do
    echo "${i}000 $RANDOM"
    sleep 1
} | nc localhost 9876

The client page uses Web Sockets to connect to the server and fetch historical data, as well as start streaming new data. Plotting is done using Flot, a jQuery library for generating decent-looking plots. For throttling when the server is streaming new points quickly, the client only fetches new data (by sending an empty frame) after a complete redraw; the server responds by sending a batch of all new points since the last fetch. (Note: the server’s pump routine currently treats the x values as millisecond timestamps and only issues a single point per second, but this can be easily tweaked/removed.)

Web Sockets can also be used over TLS. This is done by using wss: instead of ws: in the URL, and this defaults to the HTTPS port 443.

Web form i18n

By default, if you submit a web form containing certain non-ASCII (Unicode) characters, the browser sends ambiguous encodings of these characters, such that completely different text appears indistinguishable to the server.

An Example

For instance, if you enter the Chinese character for “one” into some text box in a web form:

the raw characters that the server receives off the socket are:


which is the URL encoding of the NCR of that Unicode character:


The NCR is just an HTML entity that refers to the numeric ID of a Unicode character; this character’s numeric ID is 19968.

Now if you instead wrote that NCR literally into the same text box:


the raw characters that the server receives off the socket are again:


Notice that it is impossible for the server to determine whether the user had typed in a Unicode character or the NCR thereof. Ideally, we want the server to see the direct URL encoding of the Unicode character (not its NCR):


Note: we always need URL encoding because that’s how HTML form data is transmitted. HTTP is a (mostly) ASCII protocol, and form data also needs to be delimited (key=value&key=value&... means that at least = and & characters must be URL encoded).

Unsuitable Charsets

Why is the browser sending the URL encoding of the NCR instead of the URL encoding of the original character? Because the browser is using an unsuitable character set. By default, the browser uses an ASCII charset, meaning it assumes the server only handles ASCII characters. When the user enters a non-ASCII character, the browser won’t be able to send this. However, instead of failing, it silently makes an effort to capture the character in ASCII by sending its NCR.

This implicit and ambiguous encoding behavior is AFAICT undefined, but all browsers I’ve tried seem to do this. More relevant information can be found in the HTML 4 specs on forms.

Specifying Charsets

One way to specify a suitable charset (e.g., UTF-8) that the browser should use for submitting data is to add

accept-charset="UTF-8" enctype="multipart/form-data"

to your form element. According to the specs:

The content type “multipart/form-data” should be used for submitting forms that contain files, non-ASCII data, and binary data.

This means that your form data doesn’t need get URL encoded. If Leaving it out, so that the URL encoding of the UTF-8 encoding is sent over the wire, is fine.

Another way is to leave the form alone and to specify the charset either in the HTTP response header:

Content-type: text/html; charset=utf-8

or in the HTML page head:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

These specify the page encoding, and the form submission encoding should default to the page encoding:

The default value for this attribute is the reserved string “UNKNOWN”. User agents may interpret this value as the character encoding that was used to transmit the document containing this FORM element.

Specifying a Unicode encoding for the entire page also allows you to display Unicode characters, as opposed to just accepting them in form input. More details on specifying the page encoding are in the HTML specs on charsets.

Notice that the charset specifications are actually specifying a particular encoding for that charset (e.g., UTF-8 for the Unicode character set). In Python, this data can be decoded using bytes.decode('utf-8') (str.decode('utf-8') also works in Python 2), which returns a unicode (or str in Python 3).

The moral of the story: to make sure your application is prepared to work with Unicode properly, you will always want to specify a Unicode encoding like UTF-8 in either your form’s accepted-charset or in your page’s Content-Type.

I was able to find a good discussion of this topic in this Bugzilla ticket.