When it comes to python as a language, I've talked about CPython as the reference implementation. It's what the language steering committee uses to push out new language features. Even so, CPython is not the only implementation available to run python code. In this article I'll be looking at the current state of various implementations of the python language.
PyPy
PyPy is one of the most active and up to date implementation of the python language. The main differentiator is that core code is written in python and C integration is mostly handled via cffi. Foreign Function Interface ( FFI )
is a method for one language to call another. As an example:
from cffi import FFI
ffi = FFI()
ffi.cdef("""
typedef long time_t;
time_t time(time_t *);
void srand48(long seedval);
long mrand48(void);
""")
time_t_zero = ffi.new("long *", 0)
lib = ffi.dlopen('libc.so.6')
lib.srand48(lib.time(time_t_zero))
random_number = lib.mrand48()
print(random_number)
cdef lets us define the signature for the C data types and functions we wish to call. new lets a c data type be bound to a named python variable. dlopen() then loads the library containing the methods we want to call (in this case 'libc.so.6' which is the C library). From there calls are made to srand48() and time().
There is also a nice performance gain via Just In Time (JIT) compilation for long running scripts. Running apache benchmark against my HTTP servers tutorial code:
$ ab -r -n 5000 http://127.0.0.1/index.html
cpython 3.9.16
= First Run =
Requests per second: 3237.33 [#/sec] (mean)
= Second Run =
Requests per second: 3237.42 [#/sec] (mean)
pypy 3.9.16
= First Run =
Requests per second: 1945.44 [#/sec] (mean)
= Second Run =
Requests per second: 4557.27 [#/sec] (mean)
The first run allows the JIT to optimize making the second run substantially more performant. JSON loads are performant as well:
import cProfile
import json
with open('large-file.json', 'r') as json_fp:
with cProfile.Profile() as pr:
pr.enable()
json.load(json_fp)
pr.disable()
pr.print_stats()
$ wget https://raw.githubusercontent.com/json-iterator/test-data/0bce379832b475a6c21726ce37f971f8d849513b/large-file.json
$ python json_load.py
16 function calls in 0.153 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.153 0.153 __init__.py:274(load)
1 0.000 0.000 0.106 0.106 __init__.py:299(loads)
1 0.000 0.000 0.015 0.015 codecs.py:319(decode)
1 0.000 0.000 0.106 0.106 decoder.py:332(decode)
1 0.106 0.106 0.106 0.106 decoder.py:343(raw_decode)
1 0.015 0.015 0.015 0.015 {built-in method _codecs.utf_8_decode}
1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 {method 'enable' of '_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 {method 'end' of 're.Match' objects}
2 0.000 0.000 0.000 0.000 {method 'match' of 're.Pattern' objects}
1 0.032 0.032 0.047 0.047 {method 'read' of '_io.TextIOWrapper' objects}
1 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
$ pypy json_load.py
10 function calls in 0.093 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.093 0.093 __init__.py:280(load)
1 0.000 0.000 0.053 0.053 __init__.py:305(loads)
1 0.008 0.008 0.014 0.014 codecs.py:319(decode)
1 0.006 0.006 0.006 0.006 {built-in function _codecs.utf_8_decode}
1 0.053 0.053 0.053 0.053 {built-in function _pypyjson.loads}
1 0.000 0.000 0.000 0.000 {built-in function isinstance}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 {method 'enable' of '_lsprof.Profiler' objects}
1 0.026 0.026 0.040 0.040 {method 'read' of '_io.TextIOWrapper' objects}
1 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
I think that PyPy is definitely an interesting alternative to CPython, especially if you want to look at the inner workings of the language. It does mean that you may have to consider C extension performance and compatibility in some cases.
Iron Python
Iron Python is a distribution which focuses on Python on the .NET runtime. The trickier part is that while it's somewhat more up to date it doesn't work that well with virtual environments. Iron Python's strong selling point is the ability to access the .NET runtime from within Python. Taking some Windows form code:
import clr
clr.AddReference("System.Drawing")
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import Application, Form
class MainForm(Form):
def __init__(self):
self.Text = "Test Application"
self.Name = "TestApplication"
self.Height = 800
self.Width = 600
myform = MainForm()
Application.Run(myform)
It works out just nicely:
Another very valuable feature is the ability to create an executable of Python code:
> ipyc /target:winexe .\window_test.py
IronPython Compiler for IronPython 3.4.1 (3.4.1.0)
Input Files:
.\window_test.py
Output:
window_test
OutputPath:
.
Target:
WindowApplication
Platform:
AMD64
Threading:
STA
While there is also a standalone compilation mode to embed runtime DLLs into the executable, the way it handles code loading is something Windows Defender is not quite too thrilled about:
> ipyc /target:winexe /standalone .\window_test.py
IronPython Compiler for IronPython 3.4.1 (3.4.1.0)
Input Files:
.\window_test.py
Output:
window_test
OutputPath:
.
Target:
WindowApplication
Platform:
AMD64
Threading:
STA
compiling...
Generating stand alone executable
Embedded Microsoft.Scripting 1.3.4.0
Embedded IronPython 3.4.1.0
Embedded Microsoft.Dynamic 1.3.4.0
Embedded IronPython.Modules 3.4.1.0
Embedded System.Memory 4.0.1.2
Embedded System.Runtime.CompilerServices.Unsafe 4.0.4.1
Saved to .\window_test.exe
Operation did not complete successfully because the file contains a virus or potentially unwanted software.At line:1 char:1
+ .\window_test.exe
+ ~~~~~~~~~~~~~~~~~.
The non-standalone version works, but requires the runtime DLLs, any standard library modules, and any site packages to be included as well. This leads to some rather interesting workarounds.
All and all I think this would make an interesting tool for being a glue language between python and the .NET runtime. It could allow gradual conversion of code between python and another .NET language such as C#. The ability to work in a CLI manner might have trouble competing with the more native Powershell scripting solution.
However, the Python version it works against is still a python 3 release but not something newer such as python 3.9 and above. Virus scanner issues with standalone executables being fixed in ipyc, along with a way to properly tie in standard library and site packages would really help Iron Python out. Code completion is also somewhat of a challenge when dealing with .NET classes and types.
MicroPython
MicroPython is a lightweight version of Python with a more stripped down standard library. While it can certainly run on modern systems, the ideal usage is interfacing with micro-controllers and other resource constrained systems. Their main site offers a number of development boards to work with, and the Raspberry Pi Pico supports it as well.
Despite being part of embedded development it's very active in terms of updates. As of writing this the last commit was 2 days ago (and an actual functional commit). The GitHub repository mentions it works off of Python 3.4 with a few ported features. There's also a reference list of ports for those looking into what kind of development board to work with.
RustPython
RustPython is meant to provide a python interpreter backed by Rust language code. The project is still in an experimental phase, though I found it to at least handle the web server code that was used for pypy's benchmarks. It's still a bit on the rough side in terms of performance though:
Requests per second: 1080.04 [#/sec] (mean)
There is also the ability to work in web assembly, as showcased by their python in the browser demo. The base python it runs against, 3.11, is also very up to date. While it's still in an experimental state I think more development on it can really see it to being a very viable CPython alternative, especially the web assembly functionality. If you're looking to contribute to a Rust project I highly recommend taking a look at their contribution section of the README.
Jython
Next is Python on the Java Virtual Machine (JVM), Jython. Much like Iron Python it's main selling point is the ability to integrate with Java much like a glue language. There is a downside though as Jython only supports Python 2.7 at the moment. This makes adoption difficult for those who are used to python 3 development, especially since python 2 is already EOL. That said, there is an in development branch for Jython support of python 3. It's still in a very experimental state and the 2.7 supported version is better for practical applications.
Despite the version shortcomings the ability to add dynamic type support in Java is extremely valuable. Not to mention a very extensive set of documentation on Jython usage. As an example of swing ui:
from javax.swing import JButton, JFrame
frame = JFrame('Hello, Jython!',
defaultCloseOperation = JFrame.EXIT_ON_CLOSE,
size = (300, 300)
)
def change_text(event):
print 'Clicked!'
button = JButton('Click Me!', actionPerformed=change_text)
frame.add(button)
frame.visible = True
Now even with all this there's still some some considerations:
- jar building on modern java versions is a rough task due to the maven-jython plugin being outdated
- This also means bringing in other java packages may be difficult
- Locked to python 2 means a drop in the amount of usable pip modules since many have dropped python 2 support when it went EOL
I think that if it's something done within the scope of the base Java runtime libraries it's not too bad. Python 3 support along with an updated maven jython plugin would certainly make this a more viable solution for Java and python integration.
Stackless Python
Stackless is a currently Python 3.9 supported CPython fork with a number of interesting features related to microthreads and scheduling. Unfortunately the maintainability of it is unknown at this point. Given how much the Python internals change it's not too surprising with the outcome.
Conclusion
This concludes our look at various python implementations in 2023. The python landscape is pretty big so I might have missed some. I think PyPy definitely is one to look out for. MicroPython is definitely one to look out for if you're a dev kit enthusiast. It would be interesting to see an Iron Python on a later version of python and a more streamlined executable package system in the standard library/site packages inclusion area (that Windows Defender doesn't yell at).
Rust Python looks interesting but will need some stability improvements before being taken on a more primary role in alternative implementations. The rest will be a time will tell. Seeing Jython support of python 3 come to fruition would be nice to see, and we'll see what happens with the future of Stackless.
Latest comments (0)