Opt-out-- simplifying LBM options handling P3

Now that I've described an option value abstraction and help in managing the sea of available options, today's post will cover the facilities to apply those options to LBM objects in a simplified way.

What I want is a single small set of entry points that will be available on all objects that can take options. This is in contrast to the LBM API itself, which has some 36 functions for dealing with options across a variety of objects. The functions allow you to set or get options either as fundamental data types or as a string (when sensible to do so). So each object that has options has two setters (one for fundamental data and one for strings), and two getters (again, for data and strings).

Not all of the objects that have getter/setter functions are “operational” objects; that is, not all are directly involved in the business of processing messages. Many of the operational objects in the C API have a corresponding “attribute” object upon which options can be set, and the attribute object can then be used when the associated target object is created in order to set a whole batch of options at once.

While the names of these functions conform to a pretty straightforward algorithm, for example lbm_event_queue_setopt() and lbm_event_queue_str_setopt() for setting options on event queues, there’s really no need to propagate these distinctions up through an object interface since they can easily be abstracted away. What I wanted was to be able to distill down the interface to just a single pair of getter/setters for each object that takes options.

Given the above, I decided to create a base class in Pyrex that established the option handling interface protocol and have the other extension types that front LBM objects subclass this base. The subclasses would then implement the specific machinery for setting options on their underlying LBM object.

Here’s the base option handling class:

cdef class OptionMgr:
    cdef object _setOptionWithStr(self, char *coptName,
                                  char *coptValue)
    cdef object _setOptionWithObject(self, char *coptName,
                                     void *optRef, lbmh.size_t optLen)
    cdef object _getOption(self, char *coptName, void *optRef,
                           lbmh.size_t *optLen)
    cdef object _getOptionStr(self, char *coptName, char *optValue,
                              lbmh.size_t *optLen)

This class defines the internal protocol which all extension types layered over LBM objects must implement in order to provide access to all of the functions that LBM makes available. The implementations of these methods are where the distinction between the various LBM option setting/getting functions are made.

The OptionMgr class in the .pyx file establishes the interface presented to Python itself:

cdef class OptionMgr:
    cdef object _setOptionWithStr(self, char *coptName,
                                  char *coptValue):
        retval = lbmw.lbmh.LBM_FAILURE
        return ("not_implemented", retval)
    
    cdef object _setOptionWithObject(self, char *coptName,
                                     void *optRef,
                                     lbmw.lbmh.size_t optLen):
        retval = lbmw.lbmh.LBM_FAILURE
        return ("not_implemented", retval)

    cdef object _getOption(self, char *coptName, void *optRef,
                           lbmw.lbmh.size_t *optLen):
        retval = lbmw.lbmh.LBM_FAILURE
        return ("not_implemented", retval)
        
    cdef object _getOptionStr(self, char *coptName, char *optValue,
                              lbmw.lbmh.size_t *optLen):
        retval = lbmw.lbmh.LBM_FAILURE
        return ("not_implemented", retval)
    
    def setOption(self, opt, optValue=None):
        """
        The opt argument may either be a non-unicode string containing
        the name of an LBM option, or it may be an instance of an
        Option subclass.
        
        If opt is a string, then optValue must be present and must
        also be a string containing the desired option value. If opt
        is an instance of an Option subclass, then optValue can be
        left out (if supplied it is ignored).
        """
        cdef int result
        cdef char *coptName
        cdef char *coptValue
        cdef void *coptRef
        cdef lbmw.lbmh.size_t coptLen
        cdef Option copt
        
        if type(opt) == str:
            if optValue is None or type(optValue) is not str:
                raise LBMWException("If opt is a string then optValue "
                                    "must be supplied as a string")
            coptName = opt
            coptValue = optValue
            calling, result = self._setOptionWithStr(coptName,
                                                     coptValue)
        elif typecheck(opt, Option):
            copt = opt
            coptName = copt.optName
            coptRef = copt.optRef()
            coptLen = copt.optLen()
            calling, result = self._setOptionWithObject(coptName,
                                                   coptRef, coptLen)
        else:
            raise LBMFailure(msg="unknown type for optVal:%s" %
                                 str(type(optValue)))
        
        if result == lbmw.lbmh.LBM_FAILURE:
            raise LBMFailure(fn=calling)

    def getOption(self, optName, asString=False):
        """
        optName may be a string or a Option subclass
        If asString is True:
         return a new Python string containing the option's value
        else:
            if optName is a string:
                create a new Option instance to hold the result,
                method return value is the new instance
            elif optName is a Option:
                reuse optName to hold the result,
                method return value is the original instance
            else:
                raise exception
        """
        cdef int result
        cdef char *coptName
        cdef Option opt
        cdef void *optRef
        cdef lbmw.lbmh.size_t optLen
        cdef char optVal[lbmw.lbmh.LBM_MSG_MAX_TOPIC_LEN+1]
        
        if typecheck(optName, Option):
            coptName = optName.optName
            if not asString:
                opt = optName
        elif typecheck(optName, string):
            coptName = optName
            if not asString:
                opt = lbmw.option.getOptionObj(optName)
        else:
            LBMWException("This method only supports strings and "
                          "Option instances for optName")
        if asString:
            optLen = lbmw.lbmh.LBM_MSG_MAX_TOPIC_LEN
            calling, result = self._getOptionStr(coptName, optVal,
                                                 &optLen)
            if result == lbmw.lbmh.LBM_FAILURE:
                raise LBMFailure(fn=calling)
            retval = optVal
        else:
            if opt is None:
                raise LBMWException("unrecognized option: %s" %
                                    str(optName))
            optRef = opt.optRef()
            optLen = opt.optLen()
            calling, result = self._getOption(coptName, optRef, 
                                              optLen)
            if result == lbmw.lbmh.LBM_FAILURE:
                raise LBMFailure(fn=calling)
            retval = opt
        return retval

A few words about various Pyrex constructs and some other extension stuff are in order here:

• The construction cdef Option copt is Pyrex's way to allow you to define variables of specific Python extension types. While many cases you can treat an extension type just like any other Python object, accessing methods in this manner is slower than if you specify the variable is of a particular extension type. When using cdef, you can directly and efficiently access C members. Also, this is the only way you can get at the object's methods that were defined with cdef.
• The construction <SomeClass> is Pyrex notation for a cast. This allows you to use a generic Python object in a context that requires a specific extension type. You can also do straight C casts with the angle brackets as well.
• The typecheck() function is a safer version of isinstance() that can be used within extensions. Apparently isinstance() can be fooled occasionally, and typecheck() works properly in those circumstances. This function causes Pyrex to use the Python C API to check the type of the provided object.
• The wrapper defines two exceptions, LBMFailure and LBMWException. LBMFailure is used when the underlying LBM libraries return a failure code. The extension raises this exception and includes the name of the LBM function that returned the failure. LBMWException is used when the extension code itself encountered a failure; the encountered failure isn't associated with the LBM libraries themselves.
• There's lots of taking data out of Python objects and storing it into local C variables; this is because of the way that Pyrex generates code and temporary Python objects. Not every type requires an explicitly declared local C variable, but to avoid uncertainly I decided to create explicit local copies since Pyrex would be doing it anyway.

The result of all this is that OptionMgr contributes only two Python methods to any extension type that derive from it, setOption() and getOption(). Both can work with either strings or Option instances for setting option values. Any extension class that wants to play in this game needs to implement the four option setting/getting methods, each of which are very brief. As an example of this, here's a a pair of methods from the Python EventQueue extension type that implement some of the internal protocol of OptionMgr:

cdef object _setOptionWithObject(self, char *coptName, void *optRef,
                                 lbmh.size_t optLen):
    cdef int result
    with nogil:
        result = lbmh.lbm_event_queue_attr_setopt(&self.eqAttrs,
                                                  coptName, optRef,
                                                  optLen)
    retval = result
    return ("lbm_event_queue_attr_setopt", retval)
    
cdef object _getOption(self, char *coptName, void *optRef,
                       lbmh.size_t *optLen):
    cdef int result
    with nogil:
        result = lbmh.lbm_event_queue_attr_getopt(&self.eqAttrs,
                                                  coptName, optRef,
                                                  optLen)
    retval = result
    return ("lbm_event_queue_attr_getopt", retval)

A word about the with nogil: block; this is another Pyrex construction that tells Pyrex to generate code to release the GIL for the execution of the statements in the block. Generally you'll want to do this when calling out to the library that the extension is wrapping so that other Python threads can run. When the block is complete the method will once again hold the GIL.

Earlier versions of OptionMgr passed the actual Option instance into these methods, however each wound up re-implementing a batch of boilerplate code that acquired all the individual fields needed to make the function call, and so the internal interface got refactored to not refer to any Python objects at all. It made each function much shorter.

What's it like to work with? Well, here's a snippet of an IPython session working the extension type build on top of the above “event queue attributes” LBM object. The session has been edited to reformat it a bit and remove a stack trace that isn't terribly helpful to understanding what's going on.

tom@slim-linux:~/workspace/lbmw/src/lbmw$ ipython

Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49) 
Type "copyright", "credits" or "license" for more information.
IPython 0.8.4 -- An enhanced Interactive Python.

In [1]: from lbmw import core

In [2]: from lbmw.option import eventQueueOpts as eqo

In [3]: eqc = core.EventQueueConfig()

In [4]: eqc.setOption(eqo
                      .queue_cancellation_callbacks_enabled
                      .getOptionObj()
                      .setValue(1))

In [5]: opt = eqo.queue_objects_purged_on_close.getOptionObj()

In [6]: opt.setValue(0)
Out[6]: <lbmw.coption.IntOpt object at 0x8baeedc>

In [7]: eqc.setOption(opt)

In [8]: eqc.setOption("queue_delay_warning", 1)
--------------------------------------------------------
LBMWException          Traceback (most recent call last)

<TRACEBACK SNIPPED>

LBMWException: If opt is a string then optValue must be
  supplied as a string

In [9]: eqc.setOption("queue_delay_warning", "1")

As you can see, all the flexibility provided by the underlying LBM libraries hasn't been lost, and a few different ways of interacting with options has been provided. Line 4 shows how the setValue() method of option returns the option itself, allowing you to allocate, set the value, and then set the option in a single statement. Lines 5, 6, and 7 shows how this might be broken out into multiple statements. Line 8 demonstrates an exception that is raised within the extension itself. Finally, line 9 shows that you can still use strings to set option values.

One of the other differences with LBM is the use of exceptions; every LBM function call requires you to examine a return code to determine if the function completed successfully. Python exceptions allow a much more succinct way to express the same operations with no loss of semantics.

Often, new ideas come up when you describe an interface to someone, and a couple of things have occurred to me while working on these posts:

• Legal value checking would be helpful. It would be nice if you got an exception when you tried to set a value on an option and it was outside the allowable range of values. The current classes would support this nicely: the Option subclasses could by default test to ensure that the value provided fit into the underlying C variable, but could allow an option check function to be specified at construction time that could further restrict legal values. The further restriction function could be part of OptObjectKey and passed down into the Option when it is created.
• Communicating legal values would be nice. The most helpful form would be some sort of structured information that informs you what legal boundaries are for a given option. This starts to get tricky when you consider options that are actually multi-field structures; trying to do this generally is probably not worth the effort. It may be enough to simply provide a string that contains a text description of the values accepted by the option.

Armed with a means to configure the LBM objects I want to interact with, I can now start to work on building these parts of the extension. Onward!