Fast, native-C Protocol Buffers from Python

As of the latest release of Protocol Buffers (2.3), protoc --py_out generates only pure Python code. While PB can generate fast parsing and serialization code for C++, this isn’t made available to Python, and manually wrapping generated code amounts to very tedious maintenance work. This is a recurring feature request from the discussion group, but pure Python code was a higher priority because of certain client requirements—namely (according to the team), AppEngine.

Luckily, native code is slated for PB 2.4, and has been available in the svn trunk, so you’ve been able to use fast PBs for a while now. (We’ve been using r352 for some time and have not seen any problems.) The PB team has been understandably reluctant to specify any release date, but in my thread Kenton Varda does mention early 2011 as a rough estimate.

I haven’t seen this documented anywhere else yet, so hopefully this will be useful to others.

How To Do It Fast

After installing the new PB library and rebuilding your PBs with protoc --py_out=..., you need to set the environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp before running your Python program in order to opt in to using the C++ implementation, or else PB defaults to the Python implementation.

That’s it! At least, this will use generic C++ code in the PB runtime library that dynamically parses/serializes messages. (Notice that we haven’t generated any C++ code yet.)

How fast is it? I wrote a simple program to get a sense of the performance increase in our application:

nruns = 1000
nwarmups = 100

xs = ... # your protobufs

def ser():
  return [x.SerializeToString() for x in xs]

def parse(ys):
  for y in ys: pb.Email().ParseFromString(y)

t = timeit.Timer(lambda:None)
t.timeit(nwarmups)
print 'noop:', t.timeit(nruns) / nruns

t = timeit.Timer(ser)
t.timeit(nwarmups)
print 'ser:', t.timeit(nruns) / nruns / len(xs)

ys = ser()
t = timeit.Timer(lambda: parse(ys))
t.timeit(nwarmups)
print 'parse:', t.timeit(nruns) / nruns / len(xs)

print 'msg size:', sum(len(y) for y in ys) / len(ys)

This gave the following times on my desktop, in seconds:

$ python sandbox/pbbench.py out.ini
ser: 0.000434461673101
parse: 0.000602062404156
msg size: 10730

$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp \
> python sandbox/pbbench.py out.ini
ser: 2.86788344383e-05
parse: 7.63910810153e-05
msg size: 10730

This shows ~15x and ~8x speedups on serialization and parsing, respectively. Not bad! But we can go faster.

How To Do It Faster

Now we actually generate a C++ implementation specialized for your PBs, so we no longer use runtime reflection. First, add a C extension to your Python project, e.g. by modifying setup.py:

setup(
    ...
    ext_modules=[Extension('podpb',
sources=['cpp/podpb.c','cpp/main.pb.cc'], libraries=['protobuf'])],
    ...
    )

Generate the main.pb.c with protoc --cpp_out=cpp, and create podpb.c as follows to set up an empty Python C module:

#include <Python.h>

static PyMethodDef PodMethods[] = {
  {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
initpodpb(void)
{
  PyObject *m;

  m = Py_InitModule("podpb", PodMethods);
  if (m == NULL)
    return;
}

Now python setup.py build should build everything. Just import the C module (podpb in our case) into your project, and the PB runtime library will automagically use the C++ implementation.

We now have ~68x and ~13x speedups. Hot dog.

$ PYTHONPATH=build/lib.linux-x86_64-2.6/:$PYTHONPATH \
> PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp \
> python sandbox/pbbench.py out.ini
ser: 6.39575719833e-06
parse: 4.55250144005e-05
msg size: 10730

See Also

I had this post lying around and completely forgot about it. In the meantime, the good folks at connex.io and Greplin have both released their own native Python implementations, cypb and fast-python-pb, respectively. cypb was announced on the PB mailing list and might be performing comparably, but still needs to get to “usable” state. fast-python-pb supports only string, int32, int64, double, and sub message members for now. I don’t know much else about these projects.

You can also check out my original thread on the PB mailing list where I learned about this.

Follow me on Twitter for stuff far more interesting than what I blog.

  • We have used python in our auto leads website. This website providing quality >a href=”http://www.autoleadsnetwrok.net”>auto leads with a highest conversion rates. Python will help  us build a system that can store auto leads in out http://www.autoleadsnetwork.net database

  • Guest

    Speed Up Your Torrent Downloads, Get a Seedbox

    A seedbox is BitTorrent jargon for a dedicated high-speed server, used exclusively for torrent transfers. With a seedbox you’ll be able to download and upload faster than you ever imagined. Additionally, you can manage your torrents through a browser from anywhere, anytime.

    With a seedbox, you don’t even need to use a BitTorrent client on your home computer – your worries about the RIAA or MPAA spying on you are over. No more DMCA notices or warning letters from your ISP – and more importantly, no lawsuit letters will be coming either.

    Super Ratio, Super Privacy, SuperSeedBox.co.uk

  • There is no doubt that Python is much more robust platform than C++. And your examples only solidify that fact.

  • Thanks! But it’s important to note that you need to have PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp set in the environment when building the python library with setup.py. Otherwise you’ll get errors like

    ImportError: cannot import name _net_proto2___python

    when importing the protobuf c extensions.

  • Ankur

    This isnt working for me as expected.
    Without the C extensions, the speedups are about 3x and 1.5 x respectively for Serialize and ParseFromString, However, creating new message is much slower when using the cpp implementation.

    Building a c extension and importing isnt causing any change in performance that makes me suspect that the C++ implementation specialized for my PBs is not being used at all!

    Any suggestions as to what might be going wrong?

  • Andrew Fung

    Would happen to be able to share the original source for this? I’m new to Python, and I’m struggling to get the C extensions working. Thanks!

  • That’s pretty much all the source code there is to it in the original post—but I would hazard that the info in this post is likely out of date, and now that this is officially supported I would turn more toward the official docs (or other more recent resources)!

  • AAA