Python Standard Library Implementation of Partial

Submitted by matt_eaton on Sun, 04/23/2017 - 12:57 PM

Tags

Python

Recently, I was working on some Python code where I needed to keep track of an ordered list of functions to where I could call these functions again at any given time based upon a numeric index. Think of this situation like a list keeping track of an object by index, but instead, I wanted to keep track of a functions by index. My first thought was to try and create some sort of generic object that could manage all of these functions and the specific ordering that I needed. However, after giving this a bit of consideration I thought a far more concise approach would be for me to let the Python Standard Library implementation of partial do the heavy lifting and create a new callable object for each function I needed and then just store each callable object in a list or a dictionary and reference them specifically by an index order.

The implementation of my solution is straightforward and described in greater detail in an example that I created for this post below. The real interesting thing was the Python 3.6 Standard Library Implementation of partial itself and the code running in CPython making that class possible. So, what is the partial class anyways? Partial is a small class implemented in the Functools module for making objects callable and treating them almost like functions essentially. The real power of partial comes from abstracting the need to have an object present to call an instance method. At any given point in your program you can create a partial callable to a function and pick that callable up later on a call that function even though your custom object may have gone out of scope. Pretty interesting, right? I thought so, and the code in the Python 3.6 Standard Library running partial is even more interesting, and essentially, the motivation for this post.

In this blog post I thought I would right a brief overview of some of the more interesting parts of the Python 3.6 Standard Library Implementation of partial and the code under the hood that makes it all possible!

Lets dive in!

Partials

The partial class was added to the functools module back in September of 2006 when Python 2.5 was first released. Although the code has changed a lot over the years the standard implementation of partial largely remains the same and that is to return new objects that are callable references to functions. Giving the Python developer a clean an concise way to call functions from different levels of scope.

Compatibility

The code below is an example that I put together for this post and is written for Python 3.6 and above. Python 2.* implementations are available for partial, but will not work with my example below. Specifically, in my example, I used input() to gather STDIN from the user, which is a Python 3.* feature, and raw_input() will have to be used instead for Python 2.*. The majority of the code that is defined in my implementation of the partial class was taken directly from the functools module in the Python 3.6 Standard Library. This example also contains f-strings and this is a feature that is only compatible with Python 3.6 and above. When in doubt, always test your code with a Python vitural environment so you are aware the backwards compatibility and any operating system constrainsts placed on certain dependencies.

Partial Example

The example that I put together is a blend of Python 3.6 core and vanilla Python 3.* example code. I tried to make it as descriptive and straightforward as possible by including all of the executed code in my example and not importing anything that could possibly take away from the context of what I am trying to explain.

To properly demonstrate my situation I needed a custom object to manage properties and instance methods and I also needed a list of instructions to ask the user questions based upon the data they were interacting with. So, I created the Employee object with setters and getters for employee name and number and I created a list of instructions to ask the user information when the program was executed. This illustrated my need for the partial class because the control flow allowed me to create callable objects and manage them in a specific order in a dictionary. In doing this I can reference these callable objects by a loop index and call an employee method based upon the current index the program aligns with.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
import os, sys
 
# Instead of importing from reprlib.py, I added this function in for clarity
def recursive_repr(fillvalue='...'):
    'Decorator to make a repr function return fillvalue for a recursive call'
 
    def decorating_function(user_function):
        repr_running = set()
 
        def wrapper(self):
            key = id(self), get_ident()
            if key in repr_running:
                return fillvalue
            repr_running.add(key)
            try:
                result = user_function(self)
            finally:
                repr_running.discard(key)
            return result
 
        # Can't use functools.wraps() here because of bootstrap issues
        wrapper.__module__ = getattr(user_function, '__module__')
        wrapper.__doc__ = getattr(user_function, '__doc__')
        wrapper.__name__ = getattr(user_function, '__name__')
        wrapper.__qualname__ = getattr(user_function, '__qualname__')
        wrapper.__annotations__ = getattr(user_function, '__annotations__', {})
        return wrapper
 
    return decorating_function
 
# Purely functional, no descriptor behaviour
class partial:
 
    __slots__ = "func", "args", "keywords", "__dict__", "__weakref__"
 
    # --- Python usage ---
    # __new__ creates a new instance of partial
    # __new__ is not the factory __init__ though, pay close attention to that
    # For a deep dive into partials use of __new__ checkout the C source 
    # behind this code in the _functoolsmodule.c module
    # 
    # --- C Soucre ---
    # static PyObject *
    # partial_new(PyTypeObject *type, PyObject *args, PyObject *kw)
    #
    # Also, checkout __new__ in typeobject.c
    #
    # static PyObject *
    # type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds) { ... }
    #
    def __new__(*args, **keywords):
        if not args:
            raise TypeError("descriptor '__new__' of partial needs an argument")
        if len(args) < 2:
            raise TypeError("type 'partial' takes at least one argument")
        cls, func, *args = args
 
        if not callable(func):
            raise TypeError("the first argument must be callable")
        args = tuple(args)
 
        # Assign instance variables by determining their avialability
        if hasattr(func, "func"):
            args = func.args + args
            # This is pretty awesome, the passed in func keywords are copied
            # The func keywords is then updated with the instance keywords
            # Keywords is then overwritten with the temporary keywords
            # The memory is then released for temporary keywords
            tmpkw = func.keywords.copy()
            tmpkw.update(keywords)
            keywords = tmpkw
            del tmpkw
            func = func.func
        # Following the PEP 8 standard to use cls instead of self
        self = super(partial, cls).__new__(cls)
        self.func = func
        self.args = args
        self.keywords = keywords
        return self
 
    # --- Python usage ---
    # __call__ gives the object returned from the partial the callable functionality
    # For example a = partial(func)
    # can now be called like this: a(f) because of __call__
    #
    # --- C Soucre ---
    # For a deep dive into partials use of __call__ checkout the C source 
    # behind this code in the _functoolsmodule.c module
    #
    # static PyObject *
    # partial_call(partialobject *pto, PyObject *args, PyObject *kwargs) { ... }
    #
    # Also, checkout __call__ checkout in typeobject.c
    #
    # static PyObject *
    # type_call(PyTypeObject *type, PyObject *args, PyObject *kwds) { ... }
    #
    def __call__(*args, **keywords):
        if not args:
            raise TypeError("descriptor '__call__' of partial needs an argument")
        self, *args = args
        newkeywords = self.keywords.copy()
        newkeywords.update(keywords)
        # Here is what makes the passed in func callable
        return self.func(*self.args, *args, **newkeywords)
 
    # --- Python usage ---
    # __repr__ is used to create an informal string representation of an instance.
    # Sometimes, a string cannot be created on a partial so <...> would be used.
    # 
    # One thing to note, is that this method uses f-string, which is a Python 3.6
    # feature only.  This allows formatted string literals
    #
    # Notice the @recursive_repr() decorator, this is represented above
    #
    # --- C Soucre ---
    # For a deep dive into partials use of __repr__ checkout the C source 
    # behind this code in the _functoolsmodule.c module
    # static PyObject *
    # partial_repr(partialobject *pto) { ... }
    #
    # Also, checkout __call__ checkout in typeobject.c
    #
    # static PyObject *
    # object_repr(PyObject *self) { ... }
    #
    # static PyObject *
    # type_repr(PyTypeObject *type) { ... }
    #
    @recursive_repr()
    def __repr__(self):
        # Build a string represenation of this instance
        qualname = type(self).__qualname__
        args = [repr(self.func)]
        args.extend(repr(x) for x in self.args)
        # This only works with python 3.6 and above because of f-strings
        args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items())
 
        # If called to debug, a string representation of this instance will
        # be returned, very much like __str__ 
        if type(self).__module__ == "functools":
            return f"functools.{qualname}({', '.join(args)})"
        return f"{qualname}({', '.join(args)})"
 
    # --- Python usage ---
    # In case an instance needs to be pickled (serialized) and a specific
    # variable cannot be serialized down properly, __reduce__ will handle this
    # for us by dealing with this failure gracefully.
    #
    # --- C Soucre ---
    # For a deep dive into partials use of __reduce__ checkout the C source 
    # behind this code in the _functoolsmodule.c module
    #
    # static PyObject *
    # partial_reduce(partialobject *pto, PyObject *unused) {...}
    #
    def __reduce__(self):
        return type(self), (self.func,), (self.func, self.args,
               self.keywords or None, self.__dict__ or None)
 
    # --- Python usage ---
    # In case an instance needs to be pickled (serialized) __setstate__
    # can help setup the state of an object before it is pickled so
    # that the instance variables matche the state variables
    #
    # --- C Soucre ---
    # For a deep dive into partials use of __setstate__ checkout the C source 
    # behind this code in the _functoolsmodule.c module
    #
    # static PyObject *
    # partial_setstate(partialobject *pto, PyObject *state) { ... }
    #
    def __setstate__(self, state):
        if not isinstance(state, tuple):
            raise TypeError("argument to __setstate__ must be a tuple")
        # Check that the state variables match the instances variables
        if len(state) != 4:
            raise TypeError(f"expected 4 items in state, got {len(state)}")
        func, args, kwds, namespace = state
        if (not callable(func) or not isinstance(args, tuple) or
           (kwds is not None and not isinstance(kwds, dict)) or
           (namespace is not None and not isinstance(namespace, dict))):
            raise TypeError("invalid partial state")
 
        args = tuple(args) # just in case it's a subclass
        if kwds is None:
            kwds = {}
        elif type(kwds) is not dict: # XXX does it need to be *exactly* dict?
            kwds = dict(kwds)
        if namespace is None:
            namespace = {}
 
        self.__dict__ = namespace
        self.func = func
        self.args = args
        self.keywords = kwds
 
try:
    from _functools import partial
except ImportError:
    pass
 
 
class Employee():
 
    # Initialize a new instance of the employee
    def __init__(self):
        self.__employee_name = ""
        self.__employee_number = -1
    # Setter for employee name
    def set_employee_name(self, name):
        if len(name) > 0:
            self.__employee_name = name
            return True
        else:
            return False
    # Getter for employee name
    def get_employee_name(self):
        return self.__employee_name
    # Setter for employee number
    def set_employee_number(self, number):
        if self.validate_numeric(number):
            self.__employee_number = number
            return True
        else:
            return False
    # Getter for employee number
    def get_employee_number(self):
        return self.__employee_number
    # Utility validation method
    def validate_numeric(self, num):
        if len(num) == 0:
            return False
        try:
            x = int(num)
            if x < 1:
                return False
            return True
        except ValueError:
            return False
 
 
def main():
    employee = Employee()
    instructions = [
        "Please enter an employee name: ",
        "Please enter an employee number: "
    ]
 
    # Usage of our partial class from Python core above!
    func_dict = {
        0: partial(employee.set_employee_name),
        1: partial(employee.set_employee_number)
    }
	index = 0
	# Process user input
    while index < len(instructions):
        # Value comes off standard output as a string
        input_value = input(instructions[index])
        flag = func_dict[index](input_value)
        if flag:
            index += 1
        else:
            print("Not a valid input, please try again.")
 
    print("Employee name: " + employee.get_employee_name())
    print("Employee number: " + employee.get_employee_number())
 
# Execute the main function
if __name__ == '__main__':
    main()

Under the Hood in the Python Standard Library

Under the hood, in the Python 3.6 Standard Library, the partial class is essentially made up of two critical pieces defined as __new__ and defined as __call__. These two methods are where the majority of the heavy lifting occurs and the rest of the implementation for partial is pretty much scaffolding setup to allow the class to be serialized and be represented as a string in some capacity if needed. So, let's focus on __new__ and __call__, because that is where the magic happens.

__new__ Method

In looking at __new__ implementation, this method is really in charge of performing two different tasks. First, creating a new instance of the object in memory so it can be returned when the partial class returns. Second, copying all of the arguments and keywords passed into the instance and setting up that instance to be referenced as a function with arguments attached to it. This instance is then returned and new callable object reference that is returned is the one that is made available through your program.

# Partial implementation of __new__ from Python 3.6 core.
 
#
# __new__ creates a new instance of partial
# __new__ is not the factory __init__ though, pay close attention to that
#
def __new__(*args, **keywords):
    # Perform error checking before anything is setup
    if not args:
        raise TypeError("descriptor '__new__' of partial needs an argument")
    if len(args) < 2:
        raise TypeError("type 'partial' takes at least one argument")
    # Copying arguments and defining a class variable and func variable
    cls, func, *args = args
 
    if not callable(func):
        raise TypeError("the first argument must be callable")
    args = tuple(args)
 
    # Assign instance variables by determining their avialability
    if hasattr(func, "func"):
        args = func.args + args
        # This is pretty awesome, the passed in func keywords are copied
        # The func keywords is then updated with the instance keywords
        # Keywords is then overwritten with the temporary keywords
        # The memory is then released for temporary keywords
        tmpkw = func.keywords.copy()
        tmpkw.update(keywords)
        keywords = tmpkw
        del tmpkw
        func = func.func
    # Following the PEP 8 standard to use cls instead of self
    self = super(partial, cls).__new__(cls)
    self.func = func
    self.args = args
    self.keywords = keywords
    # Returns a new copy of the instnace
    return self

__call__ Method

In looking at the __call__ implementation, it is very concise with what is happening. Not a lot of setup or building occurring here because a new instance of the partial object already created in memory. This method is really used to make the returned object callable. You probably remember this from the functools documentation, but all partial objects are callable when created and this method make that possible.

In this example there is some basic error checking to make sure that arguments are setup before anything occurs. After the error checking occurs, a double pointer that is representing the keyword dictionary is passed into the object and is copied and merged with the newkeywords variable. This ensures that a copy of the keywords that was originally passed in to the instance is not altered. The last step in passing back the func instance variable and this is where the magic happens because this is what makes the returned object callable. The pointers for args and keywords that is passed back in this function points back to the original arguments and keywords that were used when the partial object was created and allow the new object to used the proper function and argument references when the object is called as a function later in your program. Pretty cool, huh!

# Partial implementation of __call__ from Python 3.6 core.
 
#
# __call__ gives the object returned from the partial the callable functionality
# For example a = partial(func)
# can now be called like this: a(f) because of __call__
#
def __call__(*args, **keywords):
    # Basic error checking before the method is executed
    if not args:
        raise TypeError("descriptor '__call__' of partial needs an argument")
    # Defining variables for self and arguments
    self, *args = args
    # Copying and merging the existing dictionary of keywords
    newkeywords = self.keywords.copy()
    newkeywords.update(keywords)
    # Here is what makes the passed in func callable
    # Notice the double pointer **newkeywords which simply points back to 
    # the *keywords pointer.
    return self.func(*self.args, *args, **newkeywords)

CPython

Under the hood of the functools module there is also another layer written in C, commonly referred to as the CPython source code. This code is the actual C code that is compiled down into the Python interpreter when Python code is executed. I have taken a brief snippet from the _functoolsmodule.c module to illustrate all this is going on under the hood when a partial object is newly created in a program. A lot of this code below is outside the scope of this post but there are a few brief points that I would like to point out about it. First and foremost, notice how everything in this implementation looks very similar to what is happening in the Python code in functools? Once you look past the memory management and the pointers that can cause distraction, a lot of the same functionality is happening to error check the type passed into the instance to create dynamically allocated objects for the arguments and the keywords. This is just like the Python code and sitting in the Standard Library. Next, is the method signature for __new__() and for static PyObject * partial_new(). These signatures are roughly the same and the goal of each is to result in the same callable object being returned from the new instance. Very interesting if you ask me and very approachable once the Python sitting above it is understood.

/* partial object **********************************************************/
 
typedef struct {
    PyObject_HEAD
    PyObject *fn;
    PyObject *args;
    PyObject *kw;
    PyObject *dict;
    PyObject *weakreflist; /* List of weak references */
    int use_fastcall;
} partialobject;
 
static PyTypeObject partial_type;
 
static PyObject *
partial_new(PyTypeObject *type, PyObject *args, PyObject *kw)
{
    PyObject *func, *pargs, *nargs, *pkw;
    partialobject *pto;
 
    if (PyTuple_GET_SIZE(args) < 1) {
        PyErr_SetString(PyExc_TypeError,
                        "type 'partial' takes at least one argument");
        return NULL;
    }
 
    pargs = pkw = NULL;
    func = PyTuple_GET_ITEM(args, 0);
    if (Py_TYPE(func) == &partial_type && type == &partial_type) {
        partialobject *part = (partialobject *)func;
        if (part->dict == NULL) {
            pargs = part->args;
            pkw = part->kw;
            func = part->fn;
            assert(PyTuple_Check(pargs));
            assert(PyDict_Check(pkw));
        }
    }
    if (!PyCallable_Check(func)) {
        PyErr_SetString(PyExc_TypeError,
                        "the first argument must be callable");
        return NULL;
    }
 
    /* create partialobject structure */
    pto = (partialobject *)type->tp_alloc(type, 0);
    if (pto == NULL)
        return NULL;
 
    pto->fn = func;
    Py_INCREF(func);
 
    nargs = PyTuple_GetSlice(args, 1, PY_SSIZE_T_MAX);
    if (nargs == NULL) {
        Py_DECREF(pto);
        return NULL;
    }
    if (pargs == NULL) {
        pto->args = nargs;
    }
    else {
        pto->args = PySequence_Concat(pargs, nargs);
        Py_DECREF(nargs);
        if (pto->args == NULL) {
            Py_DECREF(pto);
            return NULL;
        }
        assert(PyTuple_Check(pto->args));
    }
 
    if (pkw == NULL || PyDict_GET_SIZE(pkw) == 0) {
        if (kw == NULL) {
            pto->kw = PyDict_New();
        }
        else if (Py_REFCNT(kw) == 1) {
            Py_INCREF(kw);
            pto->kw = kw;
        }
        else {
            pto->kw = PyDict_Copy(kw);
        }
    }
    else {
        pto->kw = PyDict_Copy(pkw);
        if (kw != NULL && pto->kw != NULL) {
            if (PyDict_Merge(pto->kw, kw, 1) != 0) {
                Py_DECREF(pto);
                return NULL;
            }
        }
    }
    if (pto->kw == NULL) {
        Py_DECREF(pto);
        return NULL;
    }
 
    pto->use_fastcall = _PyObject_HasFastCall(func);
 
    return (PyObject *)pto;
}

Well, I hope you enjoyed my overview into Python 3.6 Standard Library Implementation of partial. Please let me know what you thought about the post or if there are any corrections that I can make to explain something better. Please check out the source code on my GitHub page here, and follow me on Twitter for more of my Python thoughts and inspirations.

Member for

3 years 9 months

Matt Eaton

Long time mobile team lead with a love for network engineering, security, IoT, oss, writing, wireless, and mobile. Avid runner and determined health nut living in the greater Chicagoland area.

Twitter

Github