Speed up Python code with Pyjion
Microsoft’s Pyjion compiler for Python reaches stable version 1.0
What is Pyjion
The most common way to speed up your Python code is to use CPython, the official Python interpreter. It compiles code into an intermediary language called
bytecode and evaluates this code whenever your code is executed. This allows lots of flexibility, but has a downside. Simple calculations where CPUs are very efficient, like arithmetic of integers and floating point numbers are significantly slower in CPython than other interpreters.
In practice, Python developers adopt tools like Cython to make numerical operations more efficient by compiling code ahead of time, using C extensions. This is an effective solution for performance, but isn’t ideal for rapid iteration or experimentation, especially in Jupyter notebooks.
This is where Pyjion comes to help. Pyjion is a Python 3.10 package that analyses Python code and optimizes it on the fly, using a JIT, or “Just-in-Time” compiler. It can be used in a program simply by importing the Pyjion library and enabling it. Everything that runs after that statement is JIT-compiled.
How to pronounce
“Pyjion” is pronounced like the word “pigeon”. @DinoV wanted a name that had something with “Python” – the “Py” part – and something with “JIT” – the “JI” part – and have it be pronounceable.
- Profile Guided JIT Compiler
- Native 64-bit float and integer support
- Small, fast compiler
- Windows, macOS and Linux
- Intel and ARM CPU support
- Builtin IL and ASM disassembler
- Support for native debugging and profiling tools
Pyjion compiles native CPython bytecode into machine code. Without Pyjion, CPython uses a master evaluation loop (called the
frame evaluation loop) to iterate over opcodes. The Pyjion compiler has 3 main stages:
- Build a “stack table” of the abstract types at each opcode position
- Compile CPython opcodes into IL (ECMA335 CIL) instructions
- Emit the CIL opcodes into the .NET EE compiler to convert to native machine code/assembly
Benchmarks conducted by the project’s maintainers show that Pyjion is about two to three times faster than regular Python for real-world work.
Some of the optimizations allow speedups of up to 10 times. Regular arithmetic, always a good candidate for being JIT-optimized, can be an order of magnitude faster.
Some Python features, like
with statements or
async..await statements are not yet implemented. Both are on the roadmap. Pyjion includes a middleware layer that allows WSGI applications to run on Pyjion. Long-running applications such as web apps are good candidates for JIT acceleration.
After following the installation steps, Pyjion is just a Python module that you can import into your Python environment:
After importing Pyjion, enable the JIT using the
Any Python code you write or import after enabling Pyjion will be JIT compiled. You don’t need to execute functions in any special API, it’s completely transparent:
You can also execute Pyjion against any script or module:
Or, for an existing Python module:
How Pyjion compares to:
PyPy is an implementation of Python with its own JIT. The biggest difference compared to Pyjion is that PyPy doesn’t support all C extension modules without modification unless they use CFFI or work with the select subset of CPython’s C API that PyPy does support. Pyjion also aims to support many JIT compilers while PyPy only supports their custom JIT compiler.
Pyston is an implementation of Python using LLVM as a JIT compiler. Compared to Pyjion, Pyston has partial CPython C API support but not complete support. Pyston also only supports LLVM as a JIT compiler.
Numba is a JIT compiler for “array-oriented and math-heavy Python code”. This means that Numba is focused on scientific computing while Pyjion tries to optimize all Python code. Numba also only supports LLVM.
IronPython is an implementation of Python that is implemented using .NET. While IronPython tries to be usable from within .NET, Pyjion does not have a compatibility story with .NET. This also means IronPython cannot use C extension modules while Pyjion can.
Site Nuitka: https://nuitka.net/pages/overview.html
Site Shedskin: https://github.com/shedskin/shedskin
Both Nuitka and Shedskin are Python-to-C++ transpilers, which means they translate Python code into equivalent C++ code. Being a JIT, Pyjion is not a transpiler.