Home / Compyler
Compyler is a native x86 python compiler I worked on in 2006 before my interest moved on to other things. Instead of targeting C, compyler targets native x86. It even includes its own assembler which generates native windows COFF files (but no linker so you still need a toolchain to generate a complete executable).
The online version of this readme is at http://members.verizon.net/~olsongt/compyler/index.html , and the code distribution is located at http://members.verizon.net/~olsongt/compyler/compyler.rar.
[Apparently my provider decided that a .rar file is text/html, so you'll need to right-click, save on Firefox.]
No I wasn't trying to get a faster program. I was primarily interested in generating python apps with small footprints. Secondarily, I considering marketing and selling the program as a obfusticater since that request seemed to come up once every two weeks on comp.lang.python. Luckily, that didn't work out so I decided to release the program under a BSD-style license.
The program is basically abandonware. I don't really have any interest in doing anything more with this in my spare time, and I doubt anyone would pay me to work on it. That being said, I made a reasonable amount of progress. Several test cases of python code will successfully compile and execute. The most complex piece of code to run is pystones.py (from the standard python distro) but it runs excessively slow. It looks like I don't have reference counting evened out. There is a testcase that can generate a primitive .pyd file that can be imported by python that works, but generated .exes currently throw an exception when run.
Right now it is windows only. pyasm (the component that generates the assembly) will run on linux, but it doesn't generate either elf files or something that a GNU assembler can understand.
The basic approach is to use python to generate normal python bytecode, and then transliterate the bytecode into x86 assembly. For example, the RETURN_VALUE bytecode gets translated to the instructions POP EAX and JMP exit.
There are three basic components that make everything work.
pyasm is a dynamic assembler for python. It can generate either coff files as binary output to create an executable, or dynamically compile code, load it to memory, and bind it to a python variable for real time execution.
The files are located at site-packages/pyasm. There is some more documentation under the 'doc' subdirectory there.
xpython is a slightly modified version of the python interpreter. It primarily contains two modifications. There are two new builtin objects called xcodeobject and xfunctionobject. These run the compiled code, and are hooked into ceval. The pertinent files are where you'd expect them to be in the python source tree.
compyler performs the actual translation of python code to x86 code. It is located under site-packages/compyler. The bulk of the work is being done in codeObjectAssembler.py.
The best places to start or the test and benchmark subdirectories under site-packages/compyler. test/test_all.py will run all of the unit tests. test_exe.bat and test_pyd.bat will (attempt) to generate a stand-alone .exe and .pyd. All python code needs to be run with the modified python interpreter listed above, and the batch files will probably require some environment-specific configuration (basically the path to your vcvars32.bat file is). The benchmark directory has a batch file that will run pystones.py to compare against a standard python app.
codeObjectAssembler.py has a variable 'debug' that can be set to different levels to control the amount of diagnostic information.
Okay, so if you've actually gotten this far, things seem to be running, and you're interested in trying to polish things up, here's where I'd start.
Try to correct the refcounts. This seems to be the reason pytstone runs so poorly. If the refcounts get straightened out, it'll (hopefully) run favorably compared to python. There is also a test that does a fibonacci sequence in the test dir that blows up memory.
Update xpython to python 2.5. I don't think there were any bytecode changes introduced, and since compyler starts with the bytecode you should be able to get all new language features for free.
I was unable to get the stub code to bootstrap an exe funtioning correctly. Start looking at about line 168 in codeObjectAssembler.
Make some files like 'compyle' and 'makepyd' as a front end interface. Right now we're using the api.
Right now pyasm generates coff files. You could either generate binary elf files, or generate actual assembly to hand off to the GNU assembler.
The ultimate test for any self respecting compiler. Get compyler to the point where it can generate a pyd of itself.
The code is released under a standard BSD-style license.
-Grant
olsongt at verizon dot net
August 10, 2007