defunct

bounding brokenness

Civilising global state, take 2

A few months ago, I wrote a post about sane global state via parameters. Now Racket has parameters, but most other languages don’t. How important parameters are for a dynamic language was driven home to me while working on getting the Pymake build system working on our tinderboxes.

Pymake is a Python reimplementation of the venerable GNU Make, with two big advantages over it:

  1. Parallel GNU Make (at least the MSYS variant) is extremely buggy and prone to deadlocks on Windows, which means the only option is to run serially (-j1). Pymake doesn’t have any issues, so build times are much faster on machines with enough cores.
  2. The Mozilla build system uses recursive Make. Each recursive call spawns another Make process, and process spawning is rather expensive on Windows. Pymake performs all its recursion within the same process, which speeds build times on Windows even further.

Pymake also provides a few bonus features, one of which is the ability to run Python files “natively”, meaning within the same process. This helps speed up builds even more by avoiding spawning extra Python processes. Of course, such scripts have to be written with care to avoid trampling over Pymake itself.

One of the scripts for which this feature was enabled was pythonpath.py. By default, Python only loads modules from the current directory and its system directories. pythonpath.py lets us specify additional directories to load modules from. (Yes, virtualenv is a much better solution, and we’re going to switch to it soon.)

To do its work pythonpath.py must modify important global variables like sys.argc and sys.argv. Since it doesn’t undo these modifications, it causes hard-to-debug problems down the line. To fix such issues, one would need to consider every eventuality: returns, exceptions, etc. Not only that, one might need to consider modifications by others as well.

As a result, we had to disable this feature for it, slowing down builds by a bit.

If those variables were parameters, one could simply wrap the modifications up in a parameterize and expect things to Just Work. Since everyone would be doing the same thing, things continue to Just Work no matter what is run.