Compiling a Python extension in C, with the Zig toolchain

Increased number of ziglang downloads

ziglang is a Python package that contains the Zig compiler/toolchain, and that can be downloaded and installed with the usual pip and uv utilities.

Yesterday the number of downloads of this package sharply increased:

The most likely cause for this, is that, yesterday evening, I changed the package ruamel.yaml,/ from being dependent on pre-compiled wheels provided in ruamel.yaml.clib to being dependent on ruamel.yaml.clibz, which depends on ziglang and setuptools-zig, during building.

ruamel.yaml.clibz and setuptools-zig, did see the same sharp increase:

ruamel.yaml is downloaded, on average, three million times a day. I made the change early evening (CET), and the download information seems to be updated once every 24 hours, so for the first full day of availability, the numbers should be higher, even though this is Januari 1st. There is daily fluctuation, but even during the days of Christmas 2025 ruamel.yaml still got over two million downloads.

Background

I have been developing and maintaining many Python modules and packages over the last 25 years, several of which as C extensions (the original, 2007, ordereddict implementation in C for the Python 2 series), and, since 2020, several extensions written in Zig.

Some of these packages are open source, and the most successful one, measuring in number of downloads from the Python package download repository PyPI, is ruamel.yaml. This is a YAML loader and dumper for Python, that specifically tries to preserve comments, and other readability aspects, of files containing YAML documents, where other YAML parsers drop these. The package is therefore often used to minimize diffs of automated YAML changes.

After slightly less than ten years of being available, it passed the billion downloads mark in 2024.

The initial problem

The ruamel.yaml package is a 2014 fork from PyYAML, and like that package has a C/Cython extension for speed-up of tokenizing (which is currently not used in the comment preserving mode of ruamel.yaml). That extension essentially required me to provide pre-compiled downloads (i.e. Python wheels), especially for Windows and MacOS users.

These wheels made every version update of the package, take several hours to make, even if that update involved only a Python change, or even a documentation update. That this took several hours was caused by: having to rely on macOS VM running on an Intel machine at a rented facility; having to rely on Appveyor for the Windows builds in their CI setup; supporting multiple Python versions at any time (including same versions with different Unicode widths and/or separate 32/64bit support); having to deal with Docker containers with C building environments, so a Linux build would run on older systems as well; etc.

Some years ago, I split the C/Cython code of in a seperate package ruamel.yaml.clib, which only saw a few updates every year after that. Primarily to add the yearly new supported Python version. The latest (and hopefully last) release of ruamel.yaml.clib, required sixty different wheels to be generated.

At least, because of the above, making a change to the Python code and pushing out a version is relatively painless (creating and uploading, one compressed tarball and one generic wheel).

More problems

Loris Cro brought up the problematic issue of distributing Python packages as binaries, when he “interviewed’ me at SYCL 2024. At which time I (only) pointed out the problem of the trust required, and the potential danger of, downloading such pre-compiled binaries in the form of wheels.

Initially unbeknowst to me, Loris later expanded upon this at PyCon Italia (More people should watch this talk. Python users, but especially extension developers), showing that the increased download volume caused by all the binary packages on PyPI, is probably unmaintainable.

A solution for the binary wheels problem

One solution to the “wheels” problem, is to use Zig’s ease of getting a complete, compile and link, toolchain for compiling (C and/or Zig) extensions for Python. Loris proposes this in his talk (Did I mention that more people should watch that talk?), and this is also something I intended to enable, when uploading the initial version of the setuptools-zig package to PyPI back in 2020. That package is an extension to the Python build tools, the ones generating the files that can be uploaded to PyPI, and uses the zig compiler (initially 0.7), which had to be pre-installed, and available in the PATH, on your computer, to compile C and Zig extensions (examples available in the README).

Since then, the zig toolchain has become available as a (binary wheels) package ziglang on PyPI, and by combining ziglang and setuptools-zig as a build (only) dependency, you can get both installed for anyone who wants to install your package, while you provide the package only in source form. I knew I could use this for ruamel.yaml.clib, but had not taken the time to do so, and test this out.

It must be said that some of the problems mentioned can be solved by using Zig as a cross compiler. Wheels are essentially zip files with a .whl extension specific, specific file contents and layout, and ab-normalised filenames for the the package and its contents, so that you have to deal with filename clashes between namespace and non-namespace packages. (If only someone would come up, and push, a slogan like “Namespaces are one honking great idea – let’s do more of those!”, so that import abc.def and import abc_def would not naturally result in wheels with the same name `abc_def-0.0.1-py3-none-any.whl).

The final straw(s)

Python has had a minor version release every October for the last few years. This year, for Python 3.14, I found that Appveyor, that I had been using for Windows wheel generation, had not been updated to support my CI.

So I made the move to Github CI, which by now also supports Windows as a target. Although I am a Mercurial user, I used hg-git to push releases to github.

A few weeks later the github shit-storm hit about, the brilliant idea to charge users for using their self-hosted runner. This came just after Andrew Kelley had announced the move to Codeberg.

Andrew obviously has a much better sense of timing, and direction, that I do.

And then there was an issue requesting to provide wheels for free-threading (experimental in Python 3.13, standard in 3.14). I had looked at this and somehow bungled something, making it look much harder than it needed to be. But this would add another twenty or so wheels and redoing the sixty already provided, because this required source changes

If at first you don’t succeed …

After a second try, it seemed that I should be able to make setuptools-zig to provide binaries. It took quite a few steps, and iterations, and testing, though to make this work, in a way that I trusted it would work for others. As some of these changes were found only after testing on other systems that I daily use for development (my M1 Macbook and x86-64 based Linux), this meant repeated if changes would not interfere with plastforms already tested.

naming the output

setuptools-zig calls zig build-lib. Using --name doesn’t get you the required (for Python), so this would require moving/renaming after compilation.

It turned out that using -femit-bin= works much nicer, but currently doesn’t support relative directory names (so setuptools-zig provides the absolute path).

need to link against libpythonX.Y

On Windows (only), you need to link against libpython. This is probably documented somewhere.

debugging a setuptools plugin

That the python packaging environment is often “improved” in a backwards incompatible way, and in general developer unfriendly, is something I already knew for a long time.

It turns out that build extensions are even harder to debug, that I expected, due to the fact that some of the compiler output is gobbled up by the build process, even when printed to stdout. That of course makes it look clean when you install a packaged, but required me to (optionally) write debugging output to a file.

executable without execution bits set

Testing installation on the Unifi UDM (Linux alpine on aarch64), installation would not work, because the ziglang package would be downloaded, but the zig command in there, would have permissions 666, at the time of setuptools-zig being called to do the compilation.

If you install ziglang by hand on that platform (i.e. not as part of the build requirements), and check the executable, it does have the execution bits set.

I now have setuptools-zig change the execution bits of zig, if necessary, before invocation.

devpi and local caching

I have been using devpi as a local package repository for a long time, partly to distribute the 200+ packages I maintain, that are not open source.

For developing a build requirement, like setuptools-zig, using devpi seems a requirement (at least I don’t know of another way to get the packages for build requirements into an environment if these are not already fully developed on PyPI).

And one of the first things I do when setting up a (new) server for myself, or for a client is installing Docker and the devpi server (such a server always runs Linux).

The devpi server allows you to re-upload the same version of a package (PyPI understandably doesn’t). And pip and uv do seem to handle caching multiple, same numbered, versions of a package. But I found it is easy to miss a failed tool update and upload to devpi. And sometimes it seems the caches are confused, and certainly I was because of unchanged version numbers.

So I made sure I had the commands for clearing a package from the cache, or clearing the whole cache at hand, and invoke them at the slightest suspicion of saved changes not making it into the build process.

Testing

Although the move to using setuptools-zig and ziglang reduces the time spent because I don’t have to provide 60 (or 80) wheels. There is still an equally sized multi-dimension vector of combinations of Python versions, platforms, processors etc that needs to be tested.

I cannot test every point in this multi-dimension, and e.g. I have not RISC-V processor on which to test at all.

I proceeded to test all combinations on my laptop macOS (aarch64) and Linux (“normal” and musl) under Docker, and did the same on the server at my home (Linux x86-64, docker for musl). This could relatively easy be automated.

For the other dimensions, I tried to cover at least one Python version (the pre-installed one, otherwise Python 3.14). This was a time consuming manual process.

This has to be compared to the production of 60 differnt wheels, not all of which I could test on my own either.

Changes to the C source

I only made those changes to the Cythonn source (and its derived C source) necessary for the name change of the target loadable library. The name of the loadable library was made to allow for testing and a smoother transition.

Transitioning from ruamel.yaml.clib to ruamel.yaml.clibz

I changed the name loadable library name, so that I could make a version of ruamel.yaml, 0.18.17, that would still install ruamel.yaml.clib (i.e. the old library, but would prefer to load the new loadable library if it were installed.

This allowed me to install from PyPI using:

python -m pip install ruamel.yaml ruamel.yaml.clibz

and test the version compiled with zig without affecting other users.

The final changeover was made by changing the dependency, and reversing the loading preference for ruamel.yaml version 0.19.0.

That means, if needed, you can install the old loadable library and have ruamel.yaml use it. You can do so even if the zig toolchain would preclude installing ruamel.yaml for some reason, using the --no-deps option:

python -m pip install --no-deps ruamel.yaml ruamel.yaml.clib

After that it was just realising I was not going to do broader testing anytime soon and releasing the changes to PyPI.

Afterthoughts

Some things I probably would have done different in hindsight, thoughts after looking at the download numbers and others. In no particular order:

Thanks

This change to ruamel.yaml’s dependencies would not have been possible without Zig, and the years of excellent work by Andrew Kelley and the team around him.

A special thanks to Loris Cro, for daring to articulate a problem he perceived at a Python conference, even though he did not much more than describe a solution. I hope that ruamel.yaml.clibz provides a possible solution template for other packages on PyPI. (Did I mention you should watch the video of this talk?)