Showing posts with label Pyrex. Show all posts
Showing posts with label Pyrex. Show all posts

Sunday, May 24, 2009

Opt-out-- simplifying LBM options handling P3

Now that I've described an option value abstraction and help in managing the sea of available options, today's post will cover the facilities to apply those options to LBM objects in a simplified way.

What I want is a single small set of entry points that will be available on all objects that can take options. This is in contrast to the LBM API itself, which has some 36 functions for dealing with options across a variety of objects. The functions allow you to set or get options either as fundamental data types or as a string (when sensible to do so). So each object that has options has two setters (one for fundamental data and one for strings), and two getters (again, for data and strings).

Not all of the objects that have getter/setter functions are “operational” objects; that is, not all are directly involved in the business of processing messages. Many of the operational objects in the C API have a corresponding “attribute” object upon which options can be set, and the attribute object can then be used when the associated target object is created in order to set a whole batch of options at once.

While the names of these functions conform to a pretty straightforward algorithm, for example lbm_event_queue_setopt() and lbm_event_queue_str_setopt() for setting options on event queues, there’s really no need to propagate these distinctions up through an object interface since they can easily be abstracted away. What I wanted was to be able to distill down the interface to just a single pair of getter/setters for each object that takes options.

Given the above, I decided to create a base class in Pyrex that established the option handling interface protocol and have the other extension types that front LBM objects subclass this base. The subclasses would then implement the specific machinery for setting options on their underlying LBM object.

Here’s the base option handling class:

cdef class OptionMgr:
cdef object _setOptionWithStr(self, char *coptName,
char *coptValue)
cdef object _setOptionWithObject(self, char *coptName,
void *optRef, lbmh.size_t optLen)
cdef object _getOption(self, char *coptName, void *optRef,
lbmh.size_t *optLen)
cdef object _getOptionStr(self, char *coptName, char *optValue,
lbmh.size_t *optLen)
This class defines the internal protocol which all extension types layered over LBM objects must implement in order to provide access to all of the functions that LBM makes available. The implementations of these methods are where the distinction between the various LBM option setting/getting functions are made.

The OptionMgr class in the .pyx file establishes the interface presented to Python itself:

cdef class OptionMgr:
cdef object _setOptionWithStr(self, char *coptName,
char *coptValue):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _setOptionWithObject(self, char *coptName,
void *optRef,
lbmw.lbmh.size_t optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _getOption(self, char *coptName, void *optRef,
lbmw.lbmh.size_t *optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

cdef object _getOptionStr(self, char *coptName, char *optValue,
lbmw.lbmh.size_t *optLen):
retval = lbmw.lbmh.LBM_FAILURE
return ("not_implemented", retval)

def setOption(self, opt, optValue=None):
"""
The opt argument may either be a non-unicode string containing
the name of an LBM option, or it may be an instance of an
Option subclass.

If opt is a string, then optValue must be present and must
also be a string containing the desired option value. If opt
is an instance of an Option subclass, then optValue can be
left out (if supplied it is ignored).
"""
cdef int result
cdef char *coptName
cdef char *coptValue
cdef void *coptRef
cdef lbmw.lbmh.size_t coptLen
cdef Option copt

if type(opt) == str:
if optValue is None or type(optValue) is not str:
raise LBMWException("If opt is a string then optValue "
"must be supplied as a string")
coptName = opt
coptValue = optValue
calling, result = self._setOptionWithStr(coptName,
coptValue)
elif typecheck(opt, Option):
copt =
A few words about various Pyrex constructs and some other extension stuff are in order here:

  • The construction cdef Option copt is Pyrex's way to allow you to define variables of specific Python extension types. While many cases you can treat an extension type just like any other Python object, accessing methods in this manner is slower than if you specify the variable is of a particular extension type. When using cdef, you can directly and efficiently access C members. Also, this is the only way you can get at the object's methods that were defined with cdef.
  • The construction <SomeClass> is Pyrex notation for a cast. This allows you to use a generic Python object in a context that requires a specific extension type. You can also do straight C casts with the angle brackets as well.
  • The typecheck() function is a safer version of isinstance() that can be used within extensions. Apparently isinstance() can be fooled occasionally, and typecheck() works properly in those circumstances. This function causes Pyrex to use the Python C API to check the type of the provided object.
  • The wrapper defines two exceptions, LBMFailure and LBMWException. LBMFailure is used when the underlying LBM libraries return a failure code. The extension raises this exception and includes the name of the LBM function that returned the failure. LBMWException is used when the extension code itself encountered a failure; the encountered failure isn't associated with the LBM libraries themselves.
  • There's lots of taking data out of Python objects and storing it into local C variables; this is because of the way that Pyrex generates code and temporary Python objects. Not every type requires an explicitly declared local C variable, but to avoid uncertainly I decided to create explicit local copies since Pyrex would be doing it anyway.
The result of all this is that OptionMgr contributes only two Python methods to any extension type that derive from it, setOption() and getOption(). Both can work with either strings or Option instances for setting option values. Any extension class that wants to play in this game needs to implement the four option setting/getting methods, each of which are very brief. As an example of this, here's a a pair of methods from the Python EventQueue extension type that implement some of the internal protocol of OptionMgr:

cdef object _setOptionWithObject(self, char *coptName, void *optRef,
lbmh.size_t optLen):
cdef int result
with nogil:
result = lbmh.lbm_event_queue_attr_setopt(&self.eqAttrs,
coptName, optRef,
optLen)
retval = result
return ("lbm_event_queue_attr_setopt", retval)

cdef object _getOption(self, char *coptName, void *optRef,
lbmh.size_t *optLen):
cdef int result
with nogil:
result = lbmh.lbm_event_queue_attr_getopt(&self.eqAttrs,
coptName, optRef,
optLen)
retval = result
return ("lbm_event_queue_attr_getopt", retval)
A word about the with nogil: block; this is another Pyrex construction that tells Pyrex to generate code to release the GIL for the execution of the statements in the block. Generally you'll want to do this when calling out to the library that the extension is wrapping so that other Python threads can run. When the block is complete the method will once again hold the GIL.

Earlier versions of OptionMgr passed the actual Option instance into these methods, however each wound up re-implementing a batch of boilerplate code that acquired all the individual fields needed to make the function call, and so the internal interface got refactored to not refer to any Python objects at all. It made each function much shorter.

What's it like to work with? Well, here's a snippet of an IPython session working the extension type build on top of the above “event queue attributes” LBM object. The session has been edited to reformat it a bit and remove a stack trace that isn't terribly helpful to understanding what's going on.

tom@slim-linux:~/workspace/lbmw/src/lbmw$ ipython

Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
Type "copyright", "credits" or "license" for more information.
IPython 0.8.4 -- An enhanced Interactive Python.

In [1]: from lbmw import core

In [2]: from lbmw.option import eventQueueOpts as eqo

In [3]: eqc = core.EventQueueConfig()

In [4]: eqc.setOption(eqo
.queue_cancellation_callbacks_enabled
.getOptionObj()
.setValue(1))

In [5]: opt = eqo.queue_objects_purged_on_close.getOptionObj()

In [6]: opt.setValue(0)
Out[6]: <lbmw.coption.IntOpt object at 0x8baeedc>

In [7]: eqc.setOption(opt)

In [8]: eqc.setOption("queue_delay_warning", 1)
--------------------------------------------------------
LBMWException Traceback (most recent call last)

<TRACEBACK SNIPPED>

LBMWException: If opt is a string then optValue must be
supplied as a string

In [9]: eqc.setOption("queue_delay_warning", "1")
As you can see, all the flexibility provided by the underlying LBM libraries hasn't been lost, and a few different ways of interacting with options has been provided. Line 4 shows how the setValue() method of option returns the option itself, allowing you to allocate, set the value, and then set the option in a single statement. Lines 5, 6, and 7 shows how this might be broken out into multiple statements. Line 8 demonstrates an exception that is raised within the extension itself. Finally, line 9 shows that you can still use strings to set option values.

One of the other differences with LBM is the use of exceptions; every LBM function call requires you to examine a return code to determine if the function completed successfully. Python exceptions allow a much more succinct way to express the same operations with no loss of semantics.

Often, new ideas come up when you describe an interface to someone, and a couple of things have occurred to me while working on these posts:

  • Legal value checking would be helpful. It would be nice if you got an exception when you tried to set a value on an option and it was outside the allowable range of values. The current classes would support this nicely: the Option subclasses could by default test to ensure that the value provided fit into the underlying C variable, but could allow an option check function to be specified at construction time that could further restrict legal values. The further restriction function could be part of OptObjectKey and passed down into the Option when it is created.
  • Communicating legal values would be nice. The most helpful form would be some sort of structured information that informs you what legal boundaries are for a given option. This starts to get tricky when you consider options that are actually multi-field structures; trying to do this generally is probably not worth the effort. It may be enough to simply provide a string that contains a text description of the values accepted by the option.
Armed with a means to configure the LBM objects I want to interact with, I can now start to work on building these parts of the extension. Onward!

Friday, May 22, 2009

Opt-out-- simplifying LBM options handling P2

In the previous post, I laid out my plans for simplifying option handling in the Python wrapper I'm creating for 29West's LBM low-latency messaging product. I laid out three goals in that post: to hide option data type details when possible, to provide assistance in actually using the options, and finally to minimize the number of entry points involved to actually get/set options. In that post I introduced the use of Pyrex as the tool I selected to help generate the C extension code for Python, and as a aid in talking about Pyrex's use I showed the definition of the basic classes that will hold option values.

This post will focus on the second goal, namely providing assistance in selecting the proper option value class and in general providing support for navigating through the wide-variety of options available to use with LBM objects.

While LBM defines a large number of options (184 uniquely named options in the version that I'm using), the reality is that only a subset of these apply depending on the kind of transport you select and the kind of objects you create. The object that the option applies to is the “scope” of the option. Options may apply to more than one object, and hence may be in more than one scope.

However, even though the number of relevant options is significantly reduced once you make these choices, weeding out the options you care about from those you don't is a laborious process involving combing through the documentation. And then you still need to select the correct data types to use when dealing with the options.

Another issue that rubs me a bit the wrong way with the LBM option API is that all options are identified by a string name. There are no symbolic #defines for these names, so even in C you don't know that you have a typo until run time. I suppose I have no business griping about this kind of thing since I'm doing Python, but I still think we can raise the bar a bit in this space to help users avoid late discovery of typos.

I'm guessing here that the simplest version of the solution is pretty obvious: establish a dictionary somewhere whose keys are option name strings and whose values are the appropriate subclasses of Option; something like this:

def getOptionObj(key):
"""
Return an option object suitable for containing the kind
of data required for the named option
"""
if isinstance(key, OptObjectKey):
optName = key.optName
else:
raise OptionException("unknown key type:%s" % str(type(key)))
optClass = _optionNameKeyToOptionClassMap.get(key)
if optClass:
inst = optClass(optName)
else:
inst = None
return inst
It's a reasonable start; it addresses one of the key goals, namely to hide the details of the specific data types needed when setting options. You of course still need to know if the data values are integers, floats, strings, etc, but specific types are no longer important. What's still missing is help sorting out which options apply to which LBM objects, and providing a means to catch option name typos earlier.

For this, I introduced the notion of an option object key, OptObjectKey, that performs a few different functions to address these issues. The primary function, however, is to be able to use instances of the class as a way to provide a concrete object that can be referenced elsewhere when a user needs an Option subclass instance. The idea here is to provide an object that can be assigned to a variable whose name matches the option's name, thus giving IDEs something to suggest as users type code, and something that will raise an error upon import if there's a typo. In addition, the key object provides additional functionality to help sort out

The best way to explain how this works is by looking at the class and seeing some examples of how it's used. First, the class:

class OptObjectKey(object):
instancesByScope = {}
emptySet = frozenset()
def __new__(cls, optName, scopeName):
self = _optionKeyInstances.get(optName)
if self is None:
self = super(OptObjectKey, cls).__new__(cls, optName,
scopeName)
_optionKeyInstances[optName] = self
return self

def __init__(self, optName, scopeName):
#since we reuse instances, check to see if self
#already has optName as an attribute
if not hasattr(self, "optName"):
self.optName = optName
self.scopes = set([scopeName])
else:
self.scopes.add(scopeName)
self.instancesByScope.setdefault(scopeName, set()).add(self)

def __hash__(self):
return hash(self.optName)

def __eq__(self, other):
if type(other) == type(""):
result = False
elif isinstance(other, OptObjectKey):
result = self.optName == other.optName
else:
result = False
return result

def getOptionObj(self):
return getOptionObj(self)

def getOptionClass(self):
return _optionNameKeyToOptionClassMap.get(self)

@classmethod
def getAllScopes(cls):
return list(cls.instancesByScope.keys())

@classmethod
def getOptionsForScope(cls, scopeName):
return cls.instancesByScope.get(scopeName, cls.emptySet)
Instances of this class aren't directly created by modules wishing to publish information about options; a factory function takes care of creating these instances and mapping them to the proper underlying Option subclass:

def addOptionMapping(optNameStr, optType, scopeName):
#we always create the OptObjectKey instance
#since it records the scope information even
#if we've seen this option already
optObjKey = OptObjectKey(optNameStr, scopeName)
optFactoryType = _optionNameKeyToOptionClassMap.get(optNameStr)
if optFactoryType is None:
_optionNameKeyToOptionClassMap[optNameStr] = optType
_optionNameKeyToOptionClassMap[optObjKey] = optType
return optObjKey
Notice that when we add an option mapping we add entries to _optionNameKeyToOptionClassMap using both the option's string name and the OptObjectKey object as keys. This allows the user to use either the string name or the key object (if they have it) when calling getOptionObj(). Of course, getOptionObj() now needs to be changed:

def getOptionObj(key):
"""
Return an option object suitable for containing the
kind of data required for the named option
"""
if type(key) == str:
optName = key
elif isinstance(key, OptObjectKey):
optName = key.optName
else:
raise OptionException("unknown key type:%s" % str(type(key)))
optClass = _optionNameKeyToOptionClassMap.get(key)
if optClass:
inst = optClass(optName)
else:
inst = None
return inst
The OptObjectKey class maintains an internal map of scope names to the set of OptObjectKey instances that belong to each scope. This provides a means for dynamically seeing what options fall within a scope. OptObjectKey instances themselves keep a list of which scopes they apply to, thus providing a means to cross-reference options and scopes.

Publishing option mappings is now a matter of calling addOptionMapping(). Here are some examples from eventQueueOpts.py:

scope = “eventQueue”
queue_cancellation_callbacks_enabled = addOptionMapping(
“queue_cancellation_callbacks_enabled",
coption.IntOpt, scope)
queue_delay_warning = addOptionMapping("queue_delay_warning",
coption.ULongIntOpt, scope)
The return value of addOptionMapping() is an OptObjectKey instance that can subsequently act as a factory for generating appropriate Option subclasses for that option, or as a key to getOptionObj() for doing the same. The getOptionObject() function still takes option string names in case that's a more convenient form for the user, but can risk a runtime error due to a typo.

But if only OptObjectKeys will be used to look up options, IDEs can provide a lot of help. Here's how it can look in Eclipse with Pydev:



And here's how IPython helps out:



Further, the OptObjectKey class can be interrogated for the options in a scope as shown below:



These latter capabilities not only make it easier for the developer, but lend themselves nicely to creating a GUI tool for managing options for a variety of purposes, config file generation dynamic option tweaking being two examples.

The final step here is to address the large number of option getting/setting functions in LBM, and that's what I'll cover in the next post.

UPDATE, 8:42PM the same day

It occurs to me that a couple of the screen grabs I included are kind of "so what-- that's what Pydev/IPython/whatever is supposed to do," and that's all true.

When I originally was working on this post, I wasn't explicitly setting instances to variables with the same name as the option. I thought I'd be terribly clever and format up assignment statements and then exec them dynamically in a for loop in order to create the option variables, something like:

newOptKey = addOptionMapping("some_option",
SomeOptionSubclass, "someScope")
exec "%s = newOptKey" % newOptKey.optName
Well, it worked just fine, but the IDEs weren't up to knowing how clever I was; they were only parsing the modules, not executing them, and so tools like Pydev could never tell me anything about my option variables when I'd type Ctrl-space after a module name.

So while I was working on the post, I changed the code to be a regular assignment statement so that the IDEs will give me proper hints, but I forgot how pedestrian that was in relation to the capabilities of IDEs. I was still in that "see, dynamically generated!" mindset, and so I included the screen grabs in my misplaced enthusiasm. I did say I might occasionally wind up with a bit of egg on my face...

Wednesday, May 20, 2009

Opt-out-- simplifying LBM options handling P1

I want to write about the part of the extension that simplifies option handling, but since it’s a pretty straightforward exercise, I’ll also use this as an opportunity to provide a drive-by tutorial in the use of Pyrex for the development of Python extensions, the tool I’m using in binding Python to the LBM library.

Prior to this project I had only used SWIG for building extensions, but I decided to try Pyrex this time around. I have to say, unless I need to expose a library to multiple languages, I’ll probably continue to use Pyrex going forward. Besides being Python-only, its only big drawback is that it doesn’t do any automatic header file parsing, one of the real time-savers with SWIG. But Pyrex really shines in not only letting you write your extension in a dialect of Python itself (rather than in C), but also has direct support for generating extension types. These are types that are implemented in C just like Python lists and dicts, and therefore have the commensurate speed advantages. Being able to create these easily is a real advantage, and I would highly recommend Pyrex for anyone looking to wrap a C library for use in Python.

There were three areas related to options handling I want to address in the extension: first, eliminate the need to provide the address of a variable of the proper type, cast as a void *, when specifying option values, second, assistance in dealing with the extensive set of options supplied by LBM across the various objects upon which options can be set, and third, reduce the large number of function calls available for setting and getting options across LBM objects.

First off, let's deal with the simpler matter of abstracting option values. In the native LBM C API, getting and setting options can be done either with a char * (where possible) or a void pointer to variable of the correct type for the particular option. From the Python perspective, the LBM API expresses too many details of the underlying implementation for a lot of these options, especially when dealing with various integer values. Further, the majority of options involve a single value, something handled nicely by a common option abstraction. There are a few options that deal with structures so these will need to be handled a bit outside of the standard protocol, but that's not the end of the world.

Given the above, I decided to create a base class in Pyrex that established the base option value interface protocol and have the specific option extension types derive from this base. In Pyrex, extension type definitions that are to be shared by different parts of an extension go into a .pxd file, and their implementations go into a .pyx file with the same base file name. Pyrex allows you to define two kinds of methods on extension types: methods which will be visible to Python programs that use the extension type, and methods that are only visible to other Pyrex extensions that are familiar with the actual underlying C structures that the extension deals with. Pyrex-only methods are defined in the .pxd file, while Python-accessible methods are defined in the corresponding .pyx.

This is the Pyrex extension type base class that defines common protocol for all option value objects, as defined in the .pxd file:

cdef class Option:
cdef void *option
cdef readonly optName
cdef void *optRef(self)
cdef int optLen(self)
cdef _setValue(self, value)
cdef toString(self)
The Pyrex reserved word “cdef” has a few uses in in that language: before “class” it identifies a class that should be turned into an extension type. Before the declaration of a variable it identifies what will become an attribute in instances of the extension type, or a local variable in a method or function. And before a method declaration or implementation it identifies methods that will be only accessible from extensions that import the .pxd file. Notice how you can freely mix Python object references (self, value) with C variable declarations; Pyrex takes care of managing each properly. You can also specify argument and return types of a function or method (Pyrex defaults to a Python object).

The above Option class names two data items in the base class, a void * that will refer to the actual underlying option storage attribute, and a read-only variable the holds the option name. The remaining declarations define extension-only methods on this class for getting a direct void pointer to the data item managed by the object, the item's length a low-level mechanism to set its value, and a method to render the option as a string (for calling by the standard __str__ magic method).

The code that implements the class is in the corresponding .pyx file:

cdef class Option:
"""
The actual option structure's address must be assigned to "option".
The derived class should create an attribute named "_option" of the
correct type and implement __cinit__() to set option of the address
of _option.
"""
def __init__(self, optName):
self.optName = optName

def name(self):
return self.optName

def setValue(self, value):
self._setValue(value)
return self

cdef toString(self):
return self.optName

def __str__(self):
return self.toString()

#this next 4 methods are common for many subclasses and so
#are collected into an include file with the .pxi extension
cdef _setValue(self, value):
self._option = value

cdef int optLen(self):
return sizeof(self._option)

cdef void *optRef(self):
return self.option

def getValue(self):
return self._option
The code comments mention a “__cinit__” method. This is a special initialization method available to extension types (http://docs.cython.org/docs/special_methods.html) that's invoked before the more familiar __init__ method. At __cinit__ invocation time, you can be sure that all C variables you declared to comprise your extension type's instance attributes have been allocated and initialized to zero, and you are free to perform any dynamic initialization you require. Each derived class is required to implement this method and assign the address of the _option attribute to the option attribute. A couple of examples will illustrate this.

Here are two derived class that handle specific types of option values. First, the entries in the .pxd file:

cdef class UIntOpt(Option):
cdef unsigned int _option

cdef class InAddrOpt(Option):
cdef lbmw.lbmh.in_addr _option
And the corresponding implementations from the .pyx file:

cdef class UIntOpt(Option):
def __cinit__(self, optName):
self.option = &self._option

cdef toString(self):
ps = "%s:%d" % (self.optName, self._option)
return ps

def __str__(self):
return self.toString()

include "coption.pxi"

cdef class InAddrOpt(Option):
def __cinit__(self, optName):
self.option = &self._option

cdef toString(self):
ps = "%s:%d" % (self.optName, self._option.s_addr)
return ps

def __str__(self):
return self.toString()

cdef _setValue(self, value):
self._option.s_addr = value

cdef int optLen(self):
return sizeof(self._option)

cdef void *optRef(self):
return self.option

def getValue(self):
return self._option.s_addr
Notice that UIntOpt class includes a file coption.pxi; this file contains the implementation for the _setValue, optRef, optLen and getValue methods for simple option data types. In Pyrex, the include directive is a simple textual inclusion of one file into another at the current scoping level, and gives us a way to bring in duplicate code in situations where inheritance and overriding won't work (the reason why inheritance won't work here is a bit too subtle to cover in this post). These standard methods aren't used in the InAddrOpt class as the option being managed is a struct and special code is needed to set and get the value of the option.

So far, this doesn't look terribly interesting. If this was all there was to it, it really doesn't get any closer to the design goals I stated earlier, namely that implementation details should be hidden unless they're important. If users had to deal with only these classes, they'd still need to know which class to instantiate for each particular option they wanted to use, thus still expressing the implementation details of the option.

Hiding this bit of detail is the business of the next portion of the option handing system in which the wrapper provides assistance to the user in using the proper type of option and knowing which LBM objects that option can be used with. That's the topic of the next post.