Python, being both easy and powerful, has become one of the most popular programming languages. Nonetheless, it sometimes lacks the much-valued speed of statically typed and precompiled programming languages like C and Java.
Why is Python Slow?
As you may know, Python is generally implemented through an interpreter. This can cause the code execution to be rather slow compared to languages, like C and Java, with compiled implementations and whose source code is compiled in advance into machine/byte code. However, this topic is beyond the scope of the article.
How Can You Speed Up Python Code?
Unless you have to perform computationally heavy operations, Python’s speed is not usually a problem. This is where C extensions come in handy.
C extensions are a way to code functions in C, compile them into a Python module and use them in your source code as a normal Python library.
Many popular modules are actually written in C or C++ (e.g. numpy, pandas, tensorflow…) for better performance and/or low-level functionalities.
A few quick disclaimers:
- C extensions do not work for Python implementations other than Cpython. This should not be a problem since Cpython is the default one.
- It is recommended you have basic knowledge of the C programming language. However, if you only know Python you should be able to follow along without any problems.
How to Build a C Extension
As an example, let’s implement the classic fib(n) function. fib(n) takes in a number, n, and returns the corresponding number in the Fibonacci sequence. Then we will compare the performance of the Python and the C version.
First of all, you need the Python C API Python.h. It’s a C header file that contains everything that’s needed to interface with Python.
Python API installation
- On Linux you usually have to install the
python-devorpython3-devpackage, if not already present. (Note that on some distros the package name may be different) - By default, if installed with the default installer, Windows should come with Python.
- MacOs should also come with Python. If it does not,
brew reinstall pythonshould do the trick.
Now, open the code editor of your choice and create the C module file. It should be named by convention — something along the lines of module_name.c, although you can name it whatever you want. Here we’ll call it c_module.c.
Before you start coding your extension, you need to include some core definitions and declarations.
| import c_module | |
| print(c_module.c_fib(5)) | |
| # output: 5 |
It is recommended that you put these lines at the beginning of your file for compatibility purposes.
Since in Python everything is an object, our c_fib(n) function should return one, precisely a PyObject pointer (defined in Python.h).
| // pure C function that will be called recursively | |
| int fib(int n) | |
| { | |
| if (n <= 1) | |
| return n; | |
| return fib(n-1) + fib(n-2); | |
| } | |
| // function that will be called from Python code | |
| // wraps around the pure C fib function | |
| PyObject* c_fib(PyObject* self, PyObject* args) | |
| { | |
| int n; | |
| PyArg_ParseTuple(args, “i“, &n); | |
| n = fib(n); | |
| return PyLong_FromLong(n); | |
| } |
Then it’s necessary to declare which functions to export from the module to make them accessible to Python.
| // array containing the module’s methods’ definitions | |
| // put here the methods to export | |
| // the array must end with a {NULL} struct | |
| PyMethodDef module_methods[] = | |
| { | |
| {“c_fib“, c_fib, METH_VARARGS, “Method description“}, | |
| {NULL} // this struct signals the end of the array | |
| }; | |
| // struct representing the module | |
| struct PyModuleDef c_module = | |
| { | |
| PyModuleDef_HEAD_INIT, // Always initialize this member to PyModuleDef_HEAD_INIT | |
| “c_module“, // module name | |
| “Module description“, // module description | |
| –1, // module size (more on this later) | |
| module_methods // methods associated with the module | |
| }; | |
| // function that initializes the module | |
| PyMODINIT_FUNC PyInit_c_module() | |
| { | |
| return PyModule_Create(&c_module); | |
| } |
Defining the module methods
Every exported method is represented as a struct comprised of:
- The exported method name (in this case “
c_fib”) - The actual method to be exported (
c_fib) - The type of arguments the method takes (in this case
METH_VARARGS). This is from the documentation onMETH_VARARGS: “This is the typical calling convention, where the methods have the typePyCFunction. The function expects two `PyObjectvalues. The first one is theselfobject for methods; for module functions, it is the module object. The second parameter (often calledargs`) is a tuple object representing all arguments.”* - A
const char*describing the method
Defining the module
The module is represented as a struct, as shown in the code above. It’s self-documenting, except for the m_size argument, which we set to -1. From the documentation:
Setting
m_sizeto-1means that the module does not support sub-interpreters, because it has global state.
Module initialization function
The PyMODINIT_FUNC gets called when the module is imported, initializing it. Note that the function name must start with PyInit_ and end with your module’s name, hence PyInit_c_module().
These are just a few of the Python API’s functionalities. For more information and features visit the API’s documentation page.
Compile Your Extension to a Python Module
Once the C code is complete, you have to compile it to a Python module. Luckily there are a bunch of built-in tools that allow you to do exactly that.
Create a Python script, traditionally named setup.py, and insert the following code:
| # import tools to create the C extension | |
| from distutils.core import setup, Extension | |
| module_name = ‘c_module’ | |
| # the files your extension is comprised of | |
| c_files = [‘c_module.c’] | |
| extension = Extension( | |
| module_name, | |
| c_files | |
| ) | |
| setup( | |
| name=module_name, | |
| version=‘1.0’, | |
| description=‘The package description’, | |
| author=‘Nicholas Obert’, | |
| author_email=‘nchlsuba@gmail.com’, | |
| url=‘https://my.web.site/some_page’, | |
| ext_modules=[extension] | |
| ) |
This script has many functionalities, but we’ll be using only the build and install commands. For more info, take a look at the documentation or just run the script with the help flag:
python3 setup.py --help
From the command line run the following:
python3 setup.py build
This will create a directory named build and put the compiled libraries in it. When done, run this:
python3 setup.py install
This will install the just-built libraries on your system, making them accessible from anywhere.
Note that you might need root/admin privileges in order to do that. You don’t have to install it system-wide, but if you skip the install process you’ll have to use relative imports in order to use the extension.
Use C Extensions Inside a Python Program
Inside your Python file, import the newly created module using the name you chose. In our case, it is c_module:
| import c_module | |
| print(c_module.c_fib(5)) | |
| # output: 5 |
As you can see, the extension can be used the same way as any other Python module would.
How Does It Compare With the Plain Python Version?
Now let’s compare the c_fib function with its plain Python counterpart. We’ll use the built-in time module:
| import c_module | |
| from time import time | |
| # Python fib version using recursion | |
| def py_fib(n): | |
| if (n <= 1): | |
| return n | |
| return py_fib(n–1) + py_fib(n–2) | |
| n = 5 | |
| # C test | |
| t = time() | |
| c_res = c_module.c_fib(n) | |
| c_time = time() – t | |
| # Python test | |
| t = time() | |
| py_res = py_fib(n) | |
| py_time = time() – t | |
| print(f’Input: {n}\n{py_res=}, {py_time=}\n{c_res=}, {c_time=}‘) |
Output:
Input: 5
py_res=5, py_time=5.245208740234375e-06
c_res=5, c_time=1.6689300537109375e-06
As expected, the C function is faster.
Note that you might get different timings on a different machine, but the C version of the same code will always be faster.
Let’s now try with some larger numbers:
Input: 10
py_res=55, py_time=3.147125244140625e-05
c_res=55, c_time=2.6226043701171875e-06
Input: 30
py_res=832040, py_time=0.40490126609802246
c_res=832040, c_time=0.004115581512451172
Input: 40
py_res=102334155, py_time=50.17047834396362
c_res=102334155, c_time=0.4414968490600586
The C version clearly outperforms the Python one when it comes to large numbers. If you just have to do a few simple calculations, it might not be worth implementing them in C as the performance difference would be minimal. However, given a really time-consuming operation or a function that has to be repeated many times, Python’s speed might not be enough.
This is where C extensions really shine. You can leave all the heavy lifting to C, but still use Python as your main language.
Real-Life Use-Cases for C Extensions
Say you have to perform some heavy calculations, be it for a cryptographic algorithm, deep learning model training, or processing large amounts of data. C extensions can relieve the burden from Python’s vertebrae and speed up your application.
What if you instead wanted to build a low-level interface or work directly on memory from Python? C extensions are the way to go, given you know how to work with raw pointers.
What about optimizing an already-existing Python application that performs badly but you don’t want to (or can’t) rewrite it in another language? C extensions are the answer.
Or what if you’re just an optimization diehard who wants his code to run as fast as possible, but you still want some high-level abstractions for networking, a GUI, and so on. In this case, C extensions are definitely your best friend.
Time is the thing we always wish we had more of. Invest it wisely.
Conclusion
Whether you are a Python developer addicted to performance and efficiency, someone who likes mixing in different technologies, or you just want to experiment with something new, C extensions for Python are a great addition to your developer toolbox. Not only do they provide you with virtually free performance, but they can extend the functionalities of Python, saving it from the obsolete technologies stack.