bounding brokenness

MozMill in Thunderbird: looking back and moving forward

The Thunderbird team started using MozMill a couple of years ago to test UI interactions. From its humble beginnings with a couple of not-really-tests to over 400 tests covering a variety of UI interactions with tinderbox coverage on all 3 major platforms, the Thunderbird MozMill test suite has matured into an important and fairly reliable indicator of frontend code correctness.

It’s not always been like this, though – the test harness has had several major issues over the time it has existed.
  • Random test failures. These used to be very common before we made a concerted effort to fix them around the beginning of last year. A lot of them were bugs in the tests themselves, but there were at least a few subtle bugs lurking in the code. Making predicate timeouts count as failures helped a lot in tightening things up. A few random failures still remain but they’re infrequent enough that we don’t have to make fixing them a top priority.
  • Linux support. We got MozMill tinderboxes for Windows and Mac up pretty early, but Linux tests were broken for the longest while. The tests ran in a different order, the wrong part of the UI ended up receiving any keystrokes made… it made Linux frontend developers’ lives a pain, and it took a rather large patch from the wonderful asuth to finally fix things for good. MozMill tinderboxes for Linux were finally turned on some time early this year.
It’s still not as good as it can be, though. Some of the issues that remain are
  • IMAP, SMTP and NNTP support. We’ve had fakeservers for all the protocols we support for a while, but because of the way they work they’re limited to xpcshell tests for now. Thus one can’t write a frontend test that focuses on issues with our UI with IMAP servers (and we have plenty of those). This should be fixed right after jcranmer moves the fakeservers out of the main process, and I look forward to seeing the first IMAP tests soon.
  • Outdated MozMill versions. The MozMill developers have introduced subtle API changes over time, which means we haven’t been able to move to more recent versions as easily as we’d hoped to. (Not blaming them, though, since MozMill was still pretty immature when we decided to start using it.)
  • External dependencies. Imagine you’re a new developer who’d like to get started with your first MozMill test. You discover that you need to install a list of packages through easy_install or similar. You can’t just easy_install mozmill, either: since we don’t work with the latest version of MozMill, you need to specify the URL to each package you need. If you need a different version of MozMill for something else, you need to learn about and use virtualenv. All this is far more troublesome than it should be.

    Well, good news, everyone! We’re soon going to move MozMill to within the tree as part of a larger patch to upgrade it to the latest version. (The move was spurred by the fact that we didn’t want to check the requisite test fixes into older branches and all our branches were tested by the same builders so had to have the same version of MozMill installed. But let’s not focus on that. :) ) Once the patch is in the tree, you’ll be able to run MozMill tests without installing anything else.
MozMill’s future is bright: more developers will be able to write tests for more of our code faster and better.