The small spin-off project from some work I did at Mozilla became far more popular than I expected, netting over 600 downloads last year. Pretty neat!

As of yesterday, Pymake has been enabled on all Firefox Win32 builders. This includes mozilla-central, mozilla-inbound, try, and all project branches (except elm). Clobber build times (this includes Try builds!) should go down by 45 minutes to an hour, resulting in faster turnaround for developers and lower load on the infrastructure.

Pymake has now become the official way to build Firefox 18 and above on Windows. If you’re on Windows and you haven't moved to Pymake yet, you should: it's really simple. I recommend setting up an alias to the in-tree Pymake in your MSYS profile.

GNU Make will still work for now, but we won't have tier 1 continuous integration for it (at least for Firefox — Thunderbird's still on GNU Make, but I’m hoping to switch those builds to Pymake soon). Win64 builds are also on GNU Make right now, but those aren't tier 1.

A few notes:

• If your project branch is seeing build failures on Windows:

  1. make sure you're tracking an up-to-date mozilla-central: the last couple of fixes landed earlier this week and more fixes will be coming down the line
  2. try setting a clobber for Win32 builds on your branch.
  3. ask in #developers or #pymake -- perhaps someone might be able to help you out
  4. if all else fails, file a bug with releng similar to this one.

• While try builds are much faster now, we also lose the ability to build Firefox 15-17 on them. I spent a bit of time investigating workarounds but didn't get anywhere, unfortunately. Update 3/9: Simply add this patch to your queue before pushing to try.

• This also means that you need to pushing an up-to-date mozilla-central to try, otherwise you will see failures on Windows.

• One out of every 20 builds or so is failing with a strange error in dom/bindings/test -- this seems to be corruption caused by a race condition that isn't being properly handled somewhere but honestly shouldn't be happening in the first place. If you see an error in dom/bindings similar to the one in this log, please retrigger that build. The error's being tracked in this bug.

I'd like to say thanks to everyone who helped push it over the finish line: ted, khuey, coop, gps, glandium, catlee, bsmedberg, bhearsum, and anyone else if I missed them (sorry!).

Pymake is a Python reimplementation of the venerable GNU Make, with two big advantages over it:

1. Parallel GNU Make (at least the MSYS variant) is extremely buggy and prone to deadlocks on Windows, which means the only option is to run serially (-j1). Pymake doesn't have any issues, so build times are much faster on machines with enough cores.
2. The Mozilla build system uses recursive Make. Each recursive call spawns another Make process, and process spawning is rather expensive on Windows. Pymake performs all its recursion within the same process, which speeds build times on Windows even further.

Pymake also provides a few bonus features, one of which is the ability to run Python files "natively", meaning within the same process. This helps speed up builds even more by avoiding spawning extra Python processes. Of course, such scripts have to be written with care to avoid trampling over Pymake itself.

One of the scripts for which this feature was enabled was pythonpath.py. By default, Python only loads modules from the current directory and its system directories. pythonpath.py lets us specify additional directories to load modules from. (Yes, virtualenv is a much better solution, and we're going to switch to it soon.)

To do its work pythonpath.py must modify important global variables like sys.argc and sys.argv. Since it doesn't undo these modifications, it causes hard-to-debug problems down the line. To fix such issues, one would need to consider every eventuality: returns, exceptions, etc. Not only that, one might need to consider modifications by others as well.

As a result, we had to disable this feature for it, slowing down builds by a bit.

If those variables were parameters, one could simply wrap the modifications up in a parameterize and expect things to Just Work. Since everyone would be doing the same thing, things continue to Just Work no matter what is run.

Once you spend a lot of time working with Mozilla code, you sometimes forget other projects don't follow such obviously important rules. Case in point: Ruby. A default Ubuntu install builds Ruby out of the box. Of course, when you then try to do anything remotely useful:

% gem install heroku

Ruby fails with a cryptic no such file to load -- zlib (LoadError).

Turns out Ubuntu doesn't come with the zlibg1 dev library, which means the Ruby build system assumes you don't care about zlib support and happily builds without it.

Great, so you installed the library and built Ruby again, and gem actually worked. Now, you try to log in to Heroku:

% heroku login

... and Ruby fails with yet another no such file to load -- net/https error. At least the error message is slightly less cryptic this time, since it tells you to apt-get install libopenssl-ruby. Which means you need to install the library and rebuild Ruby a third time.

God knows how many more libraries the build system's assumed I don't care about.