Saturday, May 30, 2009

Top gear-- how fast can we drive Python?

In order to get an idea of how well my wrapper is performing, I needed to get some idea of the theoretical limit in performance I could expect with Python. Specifically, I needed to get an idea of how fast I could generate upcalls into Python from an external library via a thread that Python knows nothing about.

Depending on how you use it, LBM can spawn a couple of threads for your process, one of which appears to be in charge of performing upcalls into application code upon receipt of messages. As these threads are outside the set spawned by Python itself, I was concerned as to whether there would be significant cost in Python bestowing the GIL to an alien thread for the upcall. So I figured I'd create a trivial extension module that performs upcalls in a couple of different ways to see how fast we can go.

The test is composed of three parts: the main line code which sets everything in motion and reports performance stats, a simple extension that does upcalls as fast as possible, and a class on which the upcalls will be performed. The method which is the upcall target does nothing, thus giving me a reasonable baseline for comparison.

I decided to have two variables that I'd modify to see what differences emerge:

  • First, I'll vary what thread we generate upcalls from. The test extension module will be able to perform upcalls from a Python-spawned thread, a thread spawned outside of Python (a raw pthread), and from the main Python thread itself.

  • Second, we'll vary the kind of object we upcall to. I'll implement a pure Python class, and then a second class that will be implemented as a Pyrex extension type. There will be a callback method on each that will do nothing but return.
Here's the test program's main, which includes the pure Python upcall target:

import time
import cUpcaller

class PyUpcallTarget(object):
def __init__(self):
self.kind = "python upcall target"

def callback(self):

def doit():
limit = 5000000
targets = [PyUpcallTarget(), cUpcaller.CUpcallTarget()]
ucThreadsDict = {cUpcaller.WITH_PYTHON_THREAD:"python thread",
cUpcaller.WITH_ALIEN_THREAD:"alien thread",
cUpcaller.WITH_CALLING_THREAD:"calling thread"}
waysToCall = ucThreadsDict.keys()
for target in targets:
for callHow in waysToCall:
upcaller = cUpcaller.Upcaller(callHow, target.callback, limit)
print ("Timing for %s from a %s"
% (target.kind, ucThreadsDict[callHow]))
start = time.time()
elapsed = time.time() - start
print (" %d upcalls took %f secs, averaging %f upcalls/sec"
% (limit, elapsed, limit / elapsed))

if __name__ == "__main__":
And the extension Pyrex code, which includes the extension type upcall target:

import threading
cdef extern from "pthread.h" nogil:
ctypedef unsigned long int pthread_t
cdef enum:
__SIZEOF_PTHREAD_ATTR_T = 256 #the value here isn't important
cdef union pthread_attr_t:
char __size[__SIZEOF_PTHREAD_ATTR_T]
long int __align
int pthread_create(pthread_t *__newthread, pthread_attr_t *__attr,
void *(*__start_routine) (object), object __arg)
int pthread_join(pthread_t tid, void **valPtr)


cdef class Upcaller

cdef int upcaller1(object theUpcaller) with gil:
cdef int result
cdef Upcaller ucRouter
ucRouter = <upcaller> theUpcaller
result = ucRouter.routeUpcall()
return result

cdef void *upcaller1Agent(object theUpcaller) nogil:
cdef int quit
quit = 0
while quit == 0:
quit = upcaller1(theUpcaller)
return NULL

cdef class Upcaller:
cdef callback
cdef callHow
cdef upcallThread
cdef pthread_t alienThread
cdef public long upcallLimit
cdef public long upcallCount
def __init__(self, callHow, callback, limit):
self.callback = callback
self.callHow = callHow
self.upcallThread = None
self.upcallLimit = limit
self.upcallCount = 0

def go(self):
cdef int callResult
cdef pthread_t *alienThread
if self.callHow == WITH_PYTHON_THREAD:
self.upcallThread = threading.Thread(target=self._hurtEm,
elif self.callHow == WITH_CALLING_THREAD:
elif self.callHow == WITH_ALIEN_THREAD:
alienThread = &self.alienThread
with nogil:
callResult = pthread_create(alienThread, NULL,
upcaller1Agent, self)
if callResult != 0:
raise Exception("failed to start pthread")
raise Exception("unrecognized callHow value")

def _hurtEm(self):
with nogil:

cdef int routeUpcall(self):
#indicate being all done by returning 1
self.upcallCount += 1
if self.upcallCount > self.upcallLimit:
return 1
return 0

def join(self):
if self.callHow == WITH_PYTHON_THREAD:
elif self.callHow == WITH_CALLING_THREAD:
pass #we blocked in go() so there's nothing to join
else: #must be WITH_ALIEN_THREAD
with nogil:
pthread_join(self.alienThread, NULL)

cdef class CUpcallTarget:
cdef public char *kind

def __init__(self):
self.kind = "c ext upcall target"

def callback(self):
A few words about the Pyrex code:

  • The second line which starts “cdef extern from ...” tells Pyrex a couple of things: first, that the following declarations can be found in the pthread.h file and therefore Pyrex will need to generate a #include for that header, and second that any functions listed in this section are to be called without the GIL. This acts as a flag to Pyrex that it's acceptable for invoke the function inside a with nogil: block.

  • The pair of functions “upcaller1()” and “upcaller1Agent()” serve as the stand-ins for the glue code to the external library and the external library itself. The upcaller1() function includes the “with gil” suffix on the function definition to tell Pyrex to generate code that acquires the GIL upon entry to the function. An analog to this function would be the upcall target for LBM in the real extension and would acquire the GIL for each upcall, making it safe to subsequently interact with Python objects. The upcaller1Agent() function in essence acts as the whole of the LBM library; it does whatever it does, and when it needs to call out to user code (for instance, to delivery a recently arrived message), it activates the extension callback function upcaller1(). Since upcaller1Agent() is a stand-in for LBM, it is marked as “nogil” to indicate that the GIL cannot be held when calling this function.

  • I probably could have gotten a bit more speed using a straight function rather than a bound method for the upcall target, but since I planned on doing away with the old “client data pointer”, a method on an object seemed like a reasonable choice. Anyway, we're really looking for a ballpark figure here, as a real implementation isn't bound to get anywhere near this performance.
I ran the test program five times and averaged the results, which are shown in the following table. These rates are upcalls/sec:

Pure Python upcall target
Python C extension class upcall target
Upcalling thread known to Python
Upcalling thread unknown to Python
Upcalls from main thread

The test host contains an Athlon 64 X2 dual core 3800+ processor and 3 GB of RAM.

Pretty interesting numbers, to be sure. The good news is that a user of the LBM extension would have some options if they needed better performance; it's pretty clear that by turning your callback objects into Pyrex extension types would give you a significant boost in performance (an interesting data point for Pyrex use in general, too). The bad news is the performance hit encountered when the upcalls are performed by a thread created outside of Python's knowledge (that is, not using threading but rather a raw pthread). I understand Java suffers from a similar effect with alien threads calling up to Java via the JNI.

This of course raises an API extension request for 29West. It would be great to expose an optional interface for the user to plug in their own “thread factory”. The default factory would simply be a call to pthread() to start up a thread of control at some identified function. However, a user-supplied factory could use whatever means it desired to spawn a thread. In the case of Python, it would be a simple matter to create a new thread with “threading” and have it invoke a “nogil” function that would then execute the 29West thread entry point. This way Python won't have to do all the work it otherwise must face whenever an alien thread tries to acquire the GIL, and thus allow such code to run much faster.

Now I have some idea of what to aspire to.

Sunday, May 24, 2009

Opt-out-- simplifying LBM options handling P3

Now that I've described an option value abstraction and help in managing the sea of available options, today's post will cover the facilities to apply those options to LBM objects in a simplified way.

What I want is a single small set of entry points that will be available on all objects that can take options. This is in contrast to the LBM API itself, which has some 36 functions for dealing with options across a variety of objects. The functions allow you to set or get options either as fundamental data types or as a string (when sensible to do so). So each object that has options has two setters (one for fundamental data and one for strings), and two getters (again, for data and strings).

Not all of the objects that have getter/setter functions are “operational” objects; that is, not all are directly involved in the business of processing messages. Many of the operational objects in the C API have a corresponding “attribute” object upon which options can be set, and the attribute object can then be used when the associated target object is created in order to set a whole batch of options at once.

While the names of these functions conform to a pretty straightforward algorithm, for example lbm_event_queue_setopt() and lbm_event_queue_str_setopt() for setting options on event queues, there’s really no need to propagate these distinctions up through an object interface since they can easily be abstracted away. What I wanted was to be able to distill down the interface to just a single pair of getter/setters for each object that takes options.

Given the above, I decided to create a base class in Pyrex that established the option handling interface protocol and have the other extension types that front LBM objects subclass this base. The subclasses would then implement the specific machinery for setting options on their underlying LBM object.

Here’s the base option handling class:

cdef class OptionMgr:
cdef object _setOptionWithStr(self, char *coptName,
char *coptValue)
cdef object _setOptionWithObject(self, char *coptName,
void *optRef, lbmh.size_t optLen)
cdef object _getOption(self, char *coptName, void *optRef,
lbmh.size_t *optLen)
cdef object _getOptionStr(self, char *coptName, char *optValue,
lbmh.size_t *optLen)
This class defines the internal protocol which all extension types layered over LBM objects must implement in order to provide access to all of the functions that LBM makes available. The implementations of these methods are where the distinction between the various LBM option setting/getting functions are made.

The OptionMgr class in the .pyx file establishes the interface presented to Python itself:

cdef class OptionMgr:
cdef object _setOptionWithStr(self, char *coptName,
char *coptValue):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _setOptionWithObject(self, char *coptName,
void *optRef,
lbmw.lbmh.size_t optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _getOption(self, char *coptName, void *optRef,
lbmw.lbmh.size_t *optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _getOptionStr(self, char *coptName, char *optValue,
lbmw.lbmh.size_t *optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

def setOption(self, opt, optValue=None):
The opt argument may either be a non-unicode string containing
the name of an LBM option, or it may be an instance of an
Option subclass.

If opt is a string, then optValue must be present and must
also be a string containing the desired option value. If opt
is an instance of an Option subclass, then optValue can be
left out (if supplied it is ignored).
cdef int result
cdef char *coptName
cdef char *coptValue
cdef void *coptRef
cdef lbmw.lbmh.size_t coptLen
cdef Option copt

if type(opt) == str:
if optValue is None or type(optValue) is not str:
raise LBMWException("If opt is a string then optValue "
"must be supplied as a string")
coptName = opt
coptValue = optValue
calling, result = self._setOptionWithStr(coptName,
elif typecheck(opt, Option):
copt =
A few words about various Pyrex constructs and some other extension stuff are in order here:

  • The construction cdef Option copt is Pyrex's way to allow you to define variables of specific Python extension types. While many cases you can treat an extension type just like any other Python object, accessing methods in this manner is slower than if you specify the variable is of a particular extension type. When using cdef, you can directly and efficiently access C members. Also, this is the only way you can get at the object's methods that were defined with cdef.
  • The construction <SomeClass> is Pyrex notation for a cast. This allows you to use a generic Python object in a context that requires a specific extension type. You can also do straight C casts with the angle brackets as well.
  • The typecheck() function is a safer version of isinstance() that can be used within extensions. Apparently isinstance() can be fooled occasionally, and typecheck() works properly in those circumstances. This function causes Pyrex to use the Python C API to check the type of the provided object.
  • The wrapper defines two exceptions, LBMFailure and LBMWException. LBMFailure is used when the underlying LBM libraries return a failure code. The extension raises this exception and includes the name of the LBM function that returned the failure. LBMWException is used when the extension code itself encountered a failure; the encountered failure isn't associated with the LBM libraries themselves.
  • There's lots of taking data out of Python objects and storing it into local C variables; this is because of the way that Pyrex generates code and temporary Python objects. Not every type requires an explicitly declared local C variable, but to avoid uncertainly I decided to create explicit local copies since Pyrex would be doing it anyway.
The result of all this is that OptionMgr contributes only two Python methods to any extension type that derive from it, setOption() and getOption(). Both can work with either strings or Option instances for setting option values. Any extension class that wants to play in this game needs to implement the four option setting/getting methods, each of which are very brief. As an example of this, here's a a pair of methods from the Python EventQueue extension type that implement some of the internal protocol of OptionMgr:

cdef object _setOptionWithObject(self, char *coptName, void *optRef,
lbmh.size_t optLen):
cdef int result
with nogil:
result = lbmh.lbm_event_queue_attr_setopt(&self.eqAttrs,
coptName, optRef,
retval = result
return ("lbm_event_queue_attr_setopt", retval)

cdef object _getOption(self, char *coptName, void *optRef,
lbmh.size_t *optLen):
cdef int result
with nogil:
result = lbmh.lbm_event_queue_attr_getopt(&self.eqAttrs,
coptName, optRef,
retval = result
return ("lbm_event_queue_attr_getopt", retval)
A word about the with nogil: block; this is another Pyrex construction that tells Pyrex to generate code to release the GIL for the execution of the statements in the block. Generally you'll want to do this when calling out to the library that the extension is wrapping so that other Python threads can run. When the block is complete the method will once again hold the GIL.

Earlier versions of OptionMgr passed the actual Option instance into these methods, however each wound up re-implementing a batch of boilerplate code that acquired all the individual fields needed to make the function call, and so the internal interface got refactored to not refer to any Python objects at all. It made each function much shorter.

What's it like to work with? Well, here's a snippet of an IPython session working the extension type build on top of the above “event queue attributes” LBM object. The session has been edited to reformat it a bit and remove a stack trace that isn't terribly helpful to understanding what's going on.

tom@slim-linux:~/workspace/lbmw/src/lbmw$ ipython

Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
Type "copyright", "credits" or "license" for more information.
IPython 0.8.4 -- An enhanced Interactive Python.

In [1]: from lbmw import core

In [2]: from lbmw.option import eventQueueOpts as eqo

In [3]: eqc = core.EventQueueConfig()

In [4]: eqc.setOption(eqo

In [5]: opt = eqo.queue_objects_purged_on_close.getOptionObj()

In [6]: opt.setValue(0)
Out[6]: <lbmw.coption.IntOpt object at 0x8baeedc>

In [7]: eqc.setOption(opt)

In [8]: eqc.setOption("queue_delay_warning", 1)
LBMWException Traceback (most recent call last)


LBMWException: If opt is a string then optValue must be
supplied as a string

In [9]: eqc.setOption("queue_delay_warning", "1")
As you can see, all the flexibility provided by the underlying LBM libraries hasn't been lost, and a few different ways of interacting with options has been provided. Line 4 shows how the setValue() method of option returns the option itself, allowing you to allocate, set the value, and then set the option in a single statement. Lines 5, 6, and 7 shows how this might be broken out into multiple statements. Line 8 demonstrates an exception that is raised within the extension itself. Finally, line 9 shows that you can still use strings to set option values.

One of the other differences with LBM is the use of exceptions; every LBM function call requires you to examine a return code to determine if the function completed successfully. Python exceptions allow a much more succinct way to express the same operations with no loss of semantics.

Often, new ideas come up when you describe an interface to someone, and a couple of things have occurred to me while working on these posts:

  • Legal value checking would be helpful. It would be nice if you got an exception when you tried to set a value on an option and it was outside the allowable range of values. The current classes would support this nicely: the Option subclasses could by default test to ensure that the value provided fit into the underlying C variable, but could allow an option check function to be specified at construction time that could further restrict legal values. The further restriction function could be part of OptObjectKey and passed down into the Option when it is created.
  • Communicating legal values would be nice. The most helpful form would be some sort of structured information that informs you what legal boundaries are for a given option. This starts to get tricky when you consider options that are actually multi-field structures; trying to do this generally is probably not worth the effort. It may be enough to simply provide a string that contains a text description of the values accepted by the option.
Armed with a means to configure the LBM objects I want to interact with, I can now start to work on building these parts of the extension. Onward!

Friday, May 22, 2009

Opt-out-- simplifying LBM options handling P2

In the previous post, I laid out my plans for simplifying option handling in the Python wrapper I'm creating for 29West's LBM low-latency messaging product. I laid out three goals in that post: to hide option data type details when possible, to provide assistance in actually using the options, and finally to minimize the number of entry points involved to actually get/set options. In that post I introduced the use of Pyrex as the tool I selected to help generate the C extension code for Python, and as a aid in talking about Pyrex's use I showed the definition of the basic classes that will hold option values.

This post will focus on the second goal, namely providing assistance in selecting the proper option value class and in general providing support for navigating through the wide-variety of options available to use with LBM objects.

While LBM defines a large number of options (184 uniquely named options in the version that I'm using), the reality is that only a subset of these apply depending on the kind of transport you select and the kind of objects you create. The object that the option applies to is the “scope” of the option. Options may apply to more than one object, and hence may be in more than one scope.

However, even though the number of relevant options is significantly reduced once you make these choices, weeding out the options you care about from those you don't is a laborious process involving combing through the documentation. And then you still need to select the correct data types to use when dealing with the options.

Another issue that rubs me a bit the wrong way with the LBM option API is that all options are identified by a string name. There are no symbolic #defines for these names, so even in C you don't know that you have a typo until run time. I suppose I have no business griping about this kind of thing since I'm doing Python, but I still think we can raise the bar a bit in this space to help users avoid late discovery of typos.

I'm guessing here that the simplest version of the solution is pretty obvious: establish a dictionary somewhere whose keys are option name strings and whose values are the appropriate subclasses of Option; something like this:

def getOptionObj(key):
Return an option object suitable for containing the kind
of data required for the named option
if isinstance(key, OptObjectKey):
optName = key.optName
raise OptionException("unknown key type:%s" % str(type(key)))
optClass = _optionNameKeyToOptionClassMap.get(key)
if optClass:
inst = optClass(optName)
inst = None
return inst
It's a reasonable start; it addresses one of the key goals, namely to hide the details of the specific data types needed when setting options. You of course still need to know if the data values are integers, floats, strings, etc, but specific types are no longer important. What's still missing is help sorting out which options apply to which LBM objects, and providing a means to catch option name typos earlier.

For this, I introduced the notion of an option object key, OptObjectKey, that performs a few different functions to address these issues. The primary function, however, is to be able to use instances of the class as a way to provide a concrete object that can be referenced elsewhere when a user needs an Option subclass instance. The idea here is to provide an object that can be assigned to a variable whose name matches the option's name, thus giving IDEs something to suggest as users type code, and something that will raise an error upon import if there's a typo. In addition, the key object provides additional functionality to help sort out

The best way to explain how this works is by looking at the class and seeing some examples of how it's used. First, the class:

class OptObjectKey(object):
instancesByScope = {}
emptySet = frozenset()
def __new__(cls, optName, scopeName):
self = _optionKeyInstances.get(optName)
if self is None:
self = super(OptObjectKey, cls).__new__(cls, optName,
_optionKeyInstances[optName] = self
return self

def __init__(self, optName, scopeName):
#since we reuse instances, check to see if self
#already has optName as an attribute
if not hasattr(self, "optName"):
self.optName = optName
self.scopes = set([scopeName])
self.instancesByScope.setdefault(scopeName, set()).add(self)

def __hash__(self):
return hash(self.optName)

def __eq__(self, other):
if type(other) == type(""):
result = False
elif isinstance(other, OptObjectKey):
result = self.optName == other.optName
result = False
return result

def getOptionObj(self):
return getOptionObj(self)

def getOptionClass(self):
return _optionNameKeyToOptionClassMap.get(self)

def getAllScopes(cls):
return list(cls.instancesByScope.keys())

def getOptionsForScope(cls, scopeName):
return cls.instancesByScope.get(scopeName, cls.emptySet)
Instances of this class aren't directly created by modules wishing to publish information about options; a factory function takes care of creating these instances and mapping them to the proper underlying Option subclass:

def addOptionMapping(optNameStr, optType, scopeName):
#we always create the OptObjectKey instance
#since it records the scope information even
#if we've seen this option already
optObjKey = OptObjectKey(optNameStr, scopeName)
optFactoryType = _optionNameKeyToOptionClassMap.get(optNameStr)
if optFactoryType is None:
_optionNameKeyToOptionClassMap[optNameStr] = optType
_optionNameKeyToOptionClassMap[optObjKey] = optType
return optObjKey
Notice that when we add an option mapping we add entries to _optionNameKeyToOptionClassMap using both the option's string name and the OptObjectKey object as keys. This allows the user to use either the string name or the key object (if they have it) when calling getOptionObj(). Of course, getOptionObj() now needs to be changed:

def getOptionObj(key):
Return an option object suitable for containing the
kind of data required for the named option
if type(key) == str:
optName = key
elif isinstance(key, OptObjectKey):
optName = key.optName
raise OptionException("unknown key type:%s" % str(type(key)))
optClass = _optionNameKeyToOptionClassMap.get(key)
if optClass:
inst = optClass(optName)
inst = None
return inst
The OptObjectKey class maintains an internal map of scope names to the set of OptObjectKey instances that belong to each scope. This provides a means for dynamically seeing what options fall within a scope. OptObjectKey instances themselves keep a list of which scopes they apply to, thus providing a means to cross-reference options and scopes.

Publishing option mappings is now a matter of calling addOptionMapping(). Here are some examples from

scope = “eventQueue”
queue_cancellation_callbacks_enabled = addOptionMapping(
coption.IntOpt, scope)
queue_delay_warning = addOptionMapping("queue_delay_warning",
coption.ULongIntOpt, scope)
The return value of addOptionMapping() is an OptObjectKey instance that can subsequently act as a factory for generating appropriate Option subclasses for that option, or as a key to getOptionObj() for doing the same. The getOptionObject() function still takes option string names in case that's a more convenient form for the user, but can risk a runtime error due to a typo.

But if only OptObjectKeys will be used to look up options, IDEs can provide a lot of help. Here's how it can look in Eclipse with Pydev:

And here's how IPython helps out:

Further, the OptObjectKey class can be interrogated for the options in a scope as shown below:

These latter capabilities not only make it easier for the developer, but lend themselves nicely to creating a GUI tool for managing options for a variety of purposes, config file generation dynamic option tweaking being two examples.

The final step here is to address the large number of option getting/setting functions in LBM, and that's what I'll cover in the next post.

UPDATE, 8:42PM the same day

It occurs to me that a couple of the screen grabs I included are kind of "so what-- that's what Pydev/IPython/whatever is supposed to do," and that's all true.

When I originally was working on this post, I wasn't explicitly setting instances to variables with the same name as the option. I thought I'd be terribly clever and format up assignment statements and then exec them dynamically in a for loop in order to create the option variables, something like:

newOptKey = addOptionMapping("some_option",
SomeOptionSubclass, "someScope")
exec "%s = newOptKey" % newOptKey.optName
Well, it worked just fine, but the IDEs weren't up to knowing how clever I was; they were only parsing the modules, not executing them, and so tools like Pydev could never tell me anything about my option variables when I'd type Ctrl-space after a module name.

So while I was working on the post, I changed the code to be a regular assignment statement so that the IDEs will give me proper hints, but I forgot how pedestrian that was in relation to the capabilities of IDEs. I was still in that "see, dynamically generated!" mindset, and so I included the screen grabs in my misplaced enthusiasm. I did say I might occasionally wind up with a bit of egg on my face...

Wednesday, May 20, 2009

Opt-out-- simplifying LBM options handling P1

I want to write about the part of the extension that simplifies option handling, but since it’s a pretty straightforward exercise, I’ll also use this as an opportunity to provide a drive-by tutorial in the use of Pyrex for the development of Python extensions, the tool I’m using in binding Python to the LBM library.

Prior to this project I had only used SWIG for building extensions, but I decided to try Pyrex this time around. I have to say, unless I need to expose a library to multiple languages, I’ll probably continue to use Pyrex going forward. Besides being Python-only, its only big drawback is that it doesn’t do any automatic header file parsing, one of the real time-savers with SWIG. But Pyrex really shines in not only letting you write your extension in a dialect of Python itself (rather than in C), but also has direct support for generating extension types. These are types that are implemented in C just like Python lists and dicts, and therefore have the commensurate speed advantages. Being able to create these easily is a real advantage, and I would highly recommend Pyrex for anyone looking to wrap a C library for use in Python.

There were three areas related to options handling I want to address in the extension: first, eliminate the need to provide the address of a variable of the proper type, cast as a void *, when specifying option values, second, assistance in dealing with the extensive set of options supplied by LBM across the various objects upon which options can be set, and third, reduce the large number of function calls available for setting and getting options across LBM objects.

First off, let's deal with the simpler matter of abstracting option values. In the native LBM C API, getting and setting options can be done either with a char * (where possible) or a void pointer to variable of the correct type for the particular option. From the Python perspective, the LBM API expresses too many details of the underlying implementation for a lot of these options, especially when dealing with various integer values. Further, the majority of options involve a single value, something handled nicely by a common option abstraction. There are a few options that deal with structures so these will need to be handled a bit outside of the standard protocol, but that's not the end of the world.

Given the above, I decided to create a base class in Pyrex that established the base option value interface protocol and have the specific option extension types derive from this base. In Pyrex, extension type definitions that are to be shared by different parts of an extension go into a .pxd file, and their implementations go into a .pyx file with the same base file name. Pyrex allows you to define two kinds of methods on extension types: methods which will be visible to Python programs that use the extension type, and methods that are only visible to other Pyrex extensions that are familiar with the actual underlying C structures that the extension deals with. Pyrex-only methods are defined in the .pxd file, while Python-accessible methods are defined in the corresponding .pyx.

This is the Pyrex extension type base class that defines common protocol for all option value objects, as defined in the .pxd file:

cdef class Option:
cdef void *option
cdef readonly optName
cdef void *optRef(self)
cdef int optLen(self)
cdef _setValue(self, value)
cdef toString(self)
The Pyrex reserved word “cdef” has a few uses in in that language: before “class” it identifies a class that should be turned into an extension type. Before the declaration of a variable it identifies what will become an attribute in instances of the extension type, or a local variable in a method or function. And before a method declaration or implementation it identifies methods that will be only accessible from extensions that import the .pxd file. Notice how you can freely mix Python object references (self, value) with C variable declarations; Pyrex takes care of managing each properly. You can also specify argument and return types of a function or method (Pyrex defaults to a Python object).

The above Option class names two data items in the base class, a void * that will refer to the actual underlying option storage attribute, and a read-only variable the holds the option name. The remaining declarations define extension-only methods on this class for getting a direct void pointer to the data item managed by the object, the item's length a low-level mechanism to set its value, and a method to render the option as a string (for calling by the standard __str__ magic method).

The code that implements the class is in the corresponding .pyx file:

cdef class Option:
The actual option structure's address must be assigned to "option".
The derived class should create an attribute named "_option" of the
correct type and implement __cinit__() to set option of the address
of _option.
def __init__(self, optName):
self.optName = optName

def name(self):
return self.optName

def setValue(self, value):
return self

cdef toString(self):
return self.optName

def __str__(self):
return self.toString()

#this next 4 methods are common for many subclasses and so
#are collected into an include file with the .pxi extension
cdef _setValue(self, value):
self._option = value

cdef int optLen(self):
return sizeof(self._option)

cdef void *optRef(self):
return self.option

def getValue(self):
return self._option
The code comments mention a “__cinit__” method. This is a special initialization method available to extension types ( that's invoked before the more familiar __init__ method. At __cinit__ invocation time, you can be sure that all C variables you declared to comprise your extension type's instance attributes have been allocated and initialized to zero, and you are free to perform any dynamic initialization you require. Each derived class is required to implement this method and assign the address of the _option attribute to the option attribute. A couple of examples will illustrate this.

Here are two derived class that handle specific types of option values. First, the entries in the .pxd file:

cdef class UIntOpt(Option):
cdef unsigned int _option

cdef class InAddrOpt(Option):
cdef lbmw.lbmh.in_addr _option
And the corresponding implementations from the .pyx file:

cdef class UIntOpt(Option):
def __cinit__(self, optName):
self.option = &self._option

cdef toString(self):
ps = "%s:%d" % (self.optName, self._option)
return ps

def __str__(self):
return self.toString()

include "coption.pxi"

cdef class InAddrOpt(Option):
def __cinit__(self, optName):
self.option = &self._option

cdef toString(self):
ps = "%s:%d" % (self.optName, self._option.s_addr)
return ps

def __str__(self):
return self.toString()

cdef _setValue(self, value):
self._option.s_addr = value

cdef int optLen(self):
return sizeof(self._option)

cdef void *optRef(self):
return self.option

def getValue(self):
return self._option.s_addr
Notice that UIntOpt class includes a file coption.pxi; this file contains the implementation for the _setValue, optRef, optLen and getValue methods for simple option data types. In Pyrex, the include directive is a simple textual inclusion of one file into another at the current scoping level, and gives us a way to bring in duplicate code in situations where inheritance and overriding won't work (the reason why inheritance won't work here is a bit too subtle to cover in this post). These standard methods aren't used in the InAddrOpt class as the option being managed is a struct and special code is needed to set and get the value of the option.

So far, this doesn't look terribly interesting. If this was all there was to it, it really doesn't get any closer to the design goals I stated earlier, namely that implementation details should be hidden unless they're important. If users had to deal with only these classes, they'd still need to know which class to instantiate for each particular option they wanted to use, thus still expressing the implementation details of the option.

Hiding this bit of detail is the business of the next portion of the option handing system in which the wrapper provides assistance to the user in using the proper type of option and knowing which LBM objects that option can be used with. That's the topic of the next post.

Monday, May 18, 2009

Bipolar-- the highs and lows of wrapping

One of the benefits of an exploratory project of this nature is that because you don’t have a fixed design path to follow you have ample opportunity to refactor when you have one of those “Ooo! That would be a interesting approach!” moments.

But in my mind development needs to be informed by an overall view, a philosophy that drives the shape of what you’re creating. This helps encourage consistency and symmetry, qualities that a good API must have.

I needed such a set of design signposts for the extension module I was creating. Of particular concern was the establishing the “level” that the extension was to express. As befitting a product whose primary goal is high performance and low latency, the C API for 29West is fairly low level. There’s lots of fine-grained distinctions regarding integer sizes in function call arguments (signed/unsigned, short/int/long), typedefs of functions taking pointers to typedefs containing function pointers which take other typedefs as arguments…this sort of thing.

While understandable in a C API, a lot of this isn’t the kind of thing one expects or wants to encounter in a Python binding. An implicit goal of most Python modules is to provide a reasonably high-level abstraction on the semantics they’re implementing, do lots of useful things for you, and give you hooks to override default behavior.

But too high an abstraction can make the power of the underlying library unavailable, or worse, morph the semantics of the API into something alien to users of the C API. I think there’s a strong argument that extension writers shouldn’t be looking for an opportunity to overhaul an API (unless that API is particularly horrid), but largely just impedance-match the existing API to the target language via the extension. We want the concepts in the extension familiar enough that a 29West C programmer can look at it and understand what’s going on, while a Python programmer can pick it up and use it to the full advantage available due to Python.

Given this, I developed a loose set of design goals for the extension:
  • Shrink the interface in size. As I mentioned before, 29West’s C API is big, a consequence of their approach to creating an object-like C interface. This approach requires a uniquely named C function for each operation on every object, sort of pre-mangled C++ method names. Where possibly, I wanted to collapse multiple functions into a single polymorphic method call whose implementation would take care of the required differentiation. This way the interface becomes a good deal smaller and easier to keep in your head.

  • Hide implementation details unless they are important. This is an important tenant of Python itself. There’s no reason that users should have to worry about the number of bytes needed to represent various data items; only the allowable range of values matter. This, coupled with the mandate from the point above, will allow the Python API to provide a polymorphic interface that does the right thing in various circumstances, and only require the user concern themselves with data types when necessary.

  • Minimize the number of new objects introduced. Each new object represents a new concept that needs to be fit into the overall semantic framework that the 29West API already represents. Additions to this set of concepts should be done with care.

  • Automate repetitive tasks. Kind of another standard Python design goal, but well worth making explicit.

  • Replace C idioms with Python idioms. An obvious goal, but in a lot of ways this becomes the most subtle mandate to fulfill.

  • Strive for speed. While I recognize that Python is unlikely to match C’s speed, that’s no reason for complacency. 29West is predicated on high performance, and so this extension should make best efforts to run as quickly as possible.
With these design goals in mind, the following initial implementation goals fell out:
  • Provide an objective interface. This one is obvious, and easy to achieve. The 29West API is well structured to facilitate this, with each function in the C API looking a lot like a method of a class. Taking advantage of this helps with partitioning the available functions into smaller, more manageable sets.

  • Reduce the number of methods using keyword arguments. The 29West C API contains similarly named but operationally different versions of a number of functions, I suspect because C provides no mechanism for overloading function arguments. There’s no need to expose all of these flavors in Python; I’ll simply enable their use by allowing the user to supply the optional keyword arguments.

  • Hide specific data types in the options setting interfaces. From a Python user’s perspective, I don’t want to have to worry about which option calls use shorts, ints, longs, or signed/unsigned integers. I want to set a port, I’m a network programmer, I understand the ranges available for port values, let me just supply a number. I want the user to be able to not worry so much about data types when setting option values, but still have the right thing happen. Having said that, it’s probably worthwhile having some code that guards against out of range values.

  • Collapse all the option handling calls into just a few methods. Most “objects” in the C API have various options that can be set on them. These options can be set either directly on the object or on an optional associated attribute object that is used when creating the target object. Again, most likely due to the restrictions imposed by C, this has resulted in a thicket of get value/set value functions that manipulate the options on these objects. I wanted to see these all collapse down into a handful of method calls.

  • Client data pointers go away. These are needed in C but are really unnecessary in objective languages—the callback itself can encompass the data that would otherwise be referenced by a void *. This makes all method and callback signatures a little simpler. Bound methods and closures are two handy Python mechanisms that would support this mandate.

  • Few limitations on the kinds of objects that can be used for callbacks. Lots of kinds of things in Python are “callable”; besides functions, there are static/class methods, bound methods, callable instance objects, even classes themselves. I didn’t want to limit the user to just one of these mechanisms for implementing callbacks.

  • Use exceptions rather than return codes. Return codes are a standard C idiom, but Python’s support of exceptions provides a mechanism for more streamlined code. Exceptions let us write a block of code in which we don’t have to check every operation for success, making the code more succinct and easier to read, while not losing any information when something does go wrong. The downside to exceptions (in general) is that they can rob a program of performance.

  • Avoid the standard wrapping approach. Frequently, creating a Python extension module is handled in a two-tier approach: the underlying C API is directly mapped through the Python C API into Python space, and then a pure-Python module provides the objective face for the extension. This would work, but would probably run much more slowly than I’d like. In the case of the extensions for 29West, it would be much better if the end-user Python classes were implemented as extension types directly in C; these would run much faster.

  • Simplify object lifecycle management. LBM objects have dependency relationships between them at runtime, and because of this they have to cleaned up according to a specific algorithm. It seems that it would be helpful to the user if the extension could provide some assistance in cleanup operations, making it simpler to dispose of unneeded objects and their dependents.
The reality here is that some of these implementation goals are the result of those “Ooo!” moments coming along during implementation. The important point is that they all can be traced back to the design goals for the system. This isn't to say that every implementation detail must be traced back to a design goal; good ideas can be left behind if the goals are treated as a straitjacket. But when considering next steps, the design goals provide a way to frame your thinking in order to progress in a way that keeps the extension consistent.

Exposition-- all yarns need one

In the wake of the recent turmoil in the financial markets, I found myself with, shall we say, extra time on my hands. So after several reflective walks with a cigar, tending to all the pressing items in the garden, and new job hunting (not to mention a healthy dose of time-wasting on the internet), I decided to look for a project that would keep me occupied in my off-hours and afford me an opportunity to learn some new, marketable skills. This is when I considered taking a look at 29West’s low latency messaging products.

Several jobs ago, I worked with Matt Meinel, 29West’s global director of business development, at an investment bank both here in the UK and in Chicago. Some time after we parted ways, Matt contacted me to introduce me to 29West’s technology. Now, I’ve done a lot of middleware and messaging in my career and have even lead the creation of a commercial market data distribution product, so I had grown somewhat fatigued with the messaging space. But I found the ideas embodied by the 29West technology so intriguing that it rekindled my interest in messaging. And when I considered the arms race that was and still is raging around latency amongst the participants in electronic trading, I figured that gaining some experience with 29West would be a good way to broaden my experience in a marketable way. So I gave Matt a call and got access to the product suite.

After spending some time with the documentation and examples, I felt I wanted to start tinkering with the technology. The question was, in what form? Coming up with a reasonable project that exposes you to many aspects of a product is always a dicey affair. Often you do something that is reasonable in scope but winds up not touching a lot of the product’s API (and 29West’s product has a big API), or you contrive something that takes a lot of the product’s capabilities into account, but your project becomes so enormous that it takes on a life of its own, almost like you’re building a new product rather than trying to get experience with an existing one. I needed a project that would avoid these pitfalls but still give me broad exposure to the 29West API.

That’s when it occurred to me to develop a Python extension for the 29West technology. As a project, it had a lot of the features I was looking for:

  • Wrapping the API would entail touching all of its parts.
  • No higher level application need be created (except for examples); exposing the semantics of the API would be the end in itself.
  • I would be encouraged to think critically about the API in order to put a more “pythonic” face on how the API was exposed.
  • Having a Python extension for 29West would provide me a more productive springboard for some of the other projects I had considered.
But Python? It’s not the first language choice that springs to mind when you’re considering the low-latency messaging space. But the goal wasn’t white-hot performance; for that, nothing but the C API would suffice. Besides, this was to be an exercise of exploration and understanding, not necessarily for creating a product. And I knew from my experience with Python that while I probably wasn’t going to get it to drive trading on electronic exchanges, it would perform well enough for a large number of applications that someone would consider to use 29West for. Given all of the above, a Python extension to 29West seemed a fine project.

Which brings us to this blog, a development diary of sorts. It has occurred to me that someone might actually find hearing about this exercise useful, and so I’ve decided to make the project’s progress public, even if I occasionally wind up with a bit of egg on my face. What I intend to do is discuss goals, designs, tools, and implementations of the code I create to wrap the 29West libraries and make them available via Python. Such an exercise can’t avoid entailing a bit of a critique of the underlying product, and so I want to recognize 29West for voicing no objections regarding this endeavor.

I hope you find something valuable in the posts to come.