bounding brokenness

Pymake has been enabled on Win32 Firefox builders

[crossposting from; please continue the discussion there. Google Groups link.]

As of yesterday, Pymake has been enabled on all Firefox Win32 builders. This includes mozilla-central, mozilla-inbound, try, and all project branches (except elm). Clobber build times (this includes Try builds!) should go down by 45 minutes to an hour, resulting in faster turnaround for developers and lower load on the infrastructure.

Pymake has now become the official way to build Firefox 18 and above on Windows. If you’re on Windows and you haven’t moved to Pymake yet, you should: it’s really simple. I recommend setting up an alias to the in-tree Pymake in your MSYS profile.

GNU Make will still work for now, but we won’t have tier 1 continuous integration for it (at least for Firefox — Thunderbird’s still on GNU Make, but I’m hoping to switch those builds to Pymake soon). Win64 builds are also on GNU Make right now, but those aren’t tier 1.

A few notes:

  • If your project branch is seeing build failures on Windows:

    1. make sure you’re tracking an up-to-date mozilla-central: the last couple of fixes landed earlier this week and more fixes will be coming down the line
    2. try setting a clobber for Win32 builds on your branch.
    3. ask in #developers or #pymake — perhaps someone might be able to help you out
    4. if all else fails, file a bug with releng similar to this one.
  • While try builds are much faster now, we also lose the ability to build Firefox 15-17 on them. I spent a bit of time investigating workarounds but didn’t get anywhere, unfortunately. Update 3/9: Simply add this patch to your queue before pushing to try.

  • This also means that you need to pushing an up-to-date mozilla-central to try, otherwise you will see failures on Windows.

  • One out of every 20 builds or so is failing with a strange error in dom/bindings/test — this seems to be corruption caused by a race condition that isn’t being properly handled somewhere but honestly shouldn’t be happening in the first place. If you see an error in dom/bindings similar to the one in this log, please retrigger that build. The error’s being tracked in this bug.

I’d like to say thanks to everyone who helped push it over the finish line: ted, khuey, coop, gps, glandium, catlee, bsmedberg, bhearsum, and anyone else if I missed them (sorry!).