Pybind11 Notes :: Yeefea的记事本

Usage⌗

The following two lines should be added to the top of your source code.

#include <pybind11/pybind11.h>
namespace py = pybind11;

Bindings for a simple function⌗

// add is the native function
int add(int i, int j) {
    return i + j;
}

// PYBIND11_MODULE initialize the module
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring

    m.def("add", &add, "A function that adds two numbers");
}

Then use g++ to build the extension:

g++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example$(python3-config --extension-suffix)

As described here, it is prefered to build extensions with setuptools.

Keyword arguments⌗

Use py::arg("name") and its shorter notation _a to specify the name of the argument.

m.def("add", &add, "A function which adds two numbers",
      py::arg("i"), py::arg("j"));
// or 
m.def("add", &add, "A function which adds two numbers", "i"_a, "j"_a);

Default arguments⌗

m.def("add1", &add, py::arg("i") = 1, py::arg("j") = 2);
// or
m.def("add2", &add, "i"_a=1, "j"_a=2);

Exporting variables⌗

Use the attr function to register a variable in a module.
C++ built-in types are automatically converted to Python types when assigned as attributes.
C++ built-in types can be explicitly convered using the function py::cast.

PYBIND11_MODULE(example, m) {
    m.attr("the_answer") = 42;
    py::object world = py::cast("World");
    m.attr("what") = world;
}

Terminology of references⌗

When you develop Python extensions using C or C++, you should deal with references carefully. Any mistake in reference counting may cause problems such as memory leak and crash of the Python interpreter. Some debugging tools are mentioned in this documentation, but sometimes it is difficult to debug the programs.

Basically, there are three kinds of references in Python to identify the owner of the reference and whose job it is to clean it up. They are the following:

“New” references occur when a PyObject is constructed, for example when creating a new list.

long a = 10;
PyObject *pA, *pB, *r;
pA = PyLong_FromLong(a);        /* pA: New reference. */
Py_DECREF(pA);                  /* pA: Released. */

“Stolen” references occur when composing a PyObject, for example appending a value to a list. “Setters” in other words.

PyObject *r;
PyObject *v;
r = PyTuple_New(3);         /* New reference. */
v = PyLong_FromLong(1L);    /* New reference. */
/*
PyTuple_SetItem "steals" the new reference v.
This is fine, because PyTuple_SetItem DOES NOT increase the reference count.
*/
PyTuple_SetItem(r, 0, v);
v = PyLong_FromLong(2L);
PyTuple_SetItem(r, 1, v);
PyTuple_SetItem(r, 2, PyUnicode_FromString("three"));

“Borrowed” references occur when a PyObject is passed to a function, for example when passing a list to a function.

PyObject *pLast;
pLast = PyList_GetItem(pList, PyList_Size(pList) - 1);
fprintf(stdout, "Ref count was: %zd\n", pLast->ob_refcnt);

Here are some pitfalls which should be avoided:

Unexpected Py_DECREF calls

v = PyLong_FromLong(1L);    /* New reference. */
PyTuple_SetItem(r, 0, v);   /* r takes ownership of the reference. */
Py_DECREF(v);               /* Now we are interfering with r's internals. */

Once v has been passed to PyTuple_SetItem then your v becomes a borrowed reference, and you can’t call Py_DECREF on it.

Leaked “New” references

static PyObject *
list_append_one_to_four(PyObject *list) {
    for (int i = 400; i < 405; ++i) {
        PyList_Append(list, PyLong_FromLong(i));  // Since PyList_Append increases the reference count, the memory of PyLong objects will not be released.
    }
    Py_RETURN_NONE;
}

The above problem can be fixed as follows:

static PyObject *
list_append_one_to_four(PyObject *list) {
    PyObject *temporary_item = NULL;

    for (int i = 400; i < 405; ++i) {
        /* Create the object to append to the list. */
        temporary_item = PyLong_FromLong(i);
        /* temporary_item->ob_refcnt == 1 now */
        /* Append it. This will increment the reference count to 2. */
        PyList_Append(list, temporary_item);
        /* Decrement our reference to it leaving the list having the only reference. */
        Py_DECREF(temporary_item);
        /* temporary_item->ob_refcnt == 1 now */
        temporary_item =  NULL; /* Good practice... */
    }
    Py_RETURN_NONE;
}

Call functions while holding “borrowed” references

static PyObject *pop_and_print_BAD(PyObject *pList) {
    PyObject *pLast;

    pLast = PyList_GetItem(pList, PyList_Size(pList) - 1);
    fprintf(stdout, "Ref count was: %zd\n", pLast->ob_refcnt);
    do_something(pList);  // this removes every item in the list
    fprintf(stdout, "Ref count now: %zd\n", pLast->ob_refcnt);
    PyObject_Print(pLast, stdout, 0);
    fprintf(stdout, "\n");
    Py_RETURN_NONE;
}

The above problem can be fixed by increasing the reference count of the list:

static PyObject *pop_and_print_BAD(PyObject *pList) {
    PyObject *pLast;

    pLast = PyList_GetItem(pList, PyList_Size(pList) - 1);
    Py_INCREF(pLast);       /* Prevent pLast being deallocated. */
    /* ... */
    do_something(pList);
    /* ... */
    Py_DECREF(pLast);       /* No longer interested in pLast, it might     */
    pLast = NULL;           /* get deallocated here but we shouldn't care. */
    /* ... */
    Py_RETURN_NONE;
}

Functions⌗

Return value policies⌗

In pybind11, return value policies can be passed to the module_::def() and class_::def()f functions, which are defined for controlling the ways of managing the memory and the lifecycle of Python objects. The default policy is return_value_policy::automatic. The policies are described as follows:

return_value_policy::take_ownership

Reference an existing object (i.e. do not create a new copy) and take ownership. Python will call the destructor and delete operator when the object’s reference count reaches zero. This is equivalent to the “New references” behavior mentioned above.

return_value_policy::copy

Create a new copy of the returned object, which will be owned by Python. This policy is comparably safe because the lifetimes of the two instances are decoupled.

return_value_policy::move

Use std::move to move the return value contents into a new instance that will be owned by Python. This policy is comparably safe because the lifetimes of the two instances (move source and destination) are decoupled.

return_value_policy::reference

Reference an existing object, but do not take ownership. The C++ side is responsible for managing the object’s lifetime and deallocating it when it is no longer used.

return_value_policy::reference_internal

Indicates that the lifetime of the return value is tied to the lifetime of a parent object, namely the implicit this, or self argument of the called method or property. Internally, this policy works just like return_value_policy::reference but additionally applies a keep_alive<0, 1> call policy (described in the next section) that prevents the parent object from being garbage collected as long as the return value is referenced by Python. This is the default policy for property getters created via def_property, def_readwrite, etc.

return_value_policy::automatic

This policy falls back to the policy return_value_policy::take_ownership when the return value is a pointer. Otherwise, it uses return_value_policy::move or return_value_policy::copy for rvalue and lvalue references, respectively. See above for a description of what all of these different policies do. This is the default policy for py::class_-wrapped types.

return_value_policy::automatic_reference

As above, but use policy return_value_policy::reference when the return value is a pointer. This is the default conversion policy for function arguments when calling Python functions manually from C++ code (i.e. via handle::operator()) and the casters in pybind11/stl.h. You probably won’t need to use this explicitly.

Additional call policies⌗

Call policies can be specified to indicate dependencies between parameters(using py::keep_alive) or ensure a certin state for the function call(using py::call_guard).

Keep alive This pilicy is required when the C++ object is any kind of container and another object is being added to the container. keep_alive<Nurse, Patient> indicates that the argument with index Patient should be kept alive at least until the argument with index Nurse is freed by the garbage.
Index 0 refers to the return value
Index 1 refers to the implicit this pointer
Regular arguments begin at index 2

For example:

//  The lifetime of the newly added element is bound to the underlying container. '1' refers to the list object and '2' refers to the first regular argument.
py::class_<List>(m, "List")
    .def("append", &List::append, py::keep_alive<1, 2>());

    py::class_<Nurse>(m, "Nurse")
    .def(py::init<Patient &>(), py::keep_alive<1, 2>());

Call guard The py::call_guard<T> policy allows any scope guard type T to be placed around the function call. Multiple call guards can be specified as py::call_guard<T1, T2, T3...> For example:

m.def("foo", foo, py::call_guard<T>());
// is equivalent to 
m.def("foo", [](args...) {
    T scope_guard;
    return foo(args...);
});

The call guard policy is very useful in combination with gil_scoped_release. Here is another example:

class PyAnimal : public Animal {
public:
    /* Inherit the constructors */
    using Animal::Animal;

    /* Trampoline (need one for each virtual function) */
    std::string go(int n_times) {
        /* Acquire GIL before calling Python code */
        py::gil_scoped_acquire acquire;

        PYBIND11_OVERRIDE_PURE(
            std::string, /* Return type */
            Animal,      /* Parent class */
            go,          /* Name of function */
            n_times      /* Argument(s) */
        );
    }
};

PYBIND11_MODULE(example, m) {
    py::class_<Animal, PyAnimal> animal(m, "Animal");
    animal
        .def(py::init<>())
        .def("go", &Animal::go);

    py::class_<Dog>(m, "Dog", animal)
        .def(py::init<>());
    // call guard
    m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>());
    // or explicit scope guard
    m.def("call_go", [](Animal *animal) -> std::string {
        /* Release GIL before calling into (potentially long-running) C++ code */
        py::gil_scoped_release release;
        return call_go(animal);
    });
}

Python objects as arguments⌗

The wrapper classes of Python types, such as tuple, list and dict , can be used as parameters of functions. A complete list of wrapper classes can be found here.

For example, a C++ function can take a Python dict as an argument:

void print_dict(const py::dict& dict) {
    for (auto item : dict)
        std::cout << "key=" << std::string(py::str(item.first)) << ", "
                  << "value=" << std::string(py::str(item.second)) << std::endl;
}

Accepting *args and **kwargs⌗

Use py::args and py::kwargs as parameters.

void generic(py::args args, const py::kwargs& kwargs) {
    /// .. do something with args
    if (kwargs)
        /// .. do something with kwargs
}

/// Binding code
m.def("generic", &generic);

// do not define keyword arguments like this
// m.def("generic", &generic, py::arg("args"), py::arg("kwargs));

Default arguments⌗

Use py::arg_v to specify a default value and the preview of the default argument.

py::class_<MyClass>("MyClass")
    .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)"));  // human readable signature

It is possible to pass a null pointer value as a default argument.

py::class_<MyClass>("MyClass")
    .def("myFunction", py::arg("arg") = static_cast<SomeType *>(nullptr));

Pybind11 Notes

Usage⌗

Bindings for a simple function⌗

Keyword arguments⌗

Default arguments⌗

Exporting variables⌗

Terminology of references⌗

Functions⌗

Return value policies⌗

Additional call policies⌗

Python objects as arguments⌗

Accepting *args and **kwargs⌗

Default arguments⌗

Referencecs⌗