defunct

New version of ntfsutils

2013-01-25T22:30:00-08:00

I’ve released a new version of the ntfsutils library for Python.

The small spin-off project from some work I did at Mozilla became far more popular than I expected, netting over 600 downloads last year. Pretty neat!

Pymake has been enabled on Win32 Firefox builders

2012-09-01T07:30:00-07:00

[crossposting from mozilla.dev.platform; please continue the discussion there. Google Groups link.]

As of yesterday, Pymake has been enabled on all Firefox Win32 builders. This includes mozilla-central, mozilla-inbound, try, and all project branches (except elm). Clobber build times (this includes Try builds!) should go down by 45 minutes to an hour, resulting in faster turnaround for developers and lower load on the infrastructure.

Pymake has now become the official way to build Firefox 18 and above on Windows. If you’re on Windows and you haven’t moved to Pymake yet, you should: it’s really simple. I recommend setting up an alias to the in-tree Pymake in your MSYS profile.

GNU Make will still work for now, but we won’t have tier 1 continuous integration for it (at least for Firefox — Thunderbird’s still on GNU Make, but I’m hoping to switch those builds to Pymake soon). Win64 builds are also on GNU Make right now, but those aren’t tier 1.

A few notes:

If your project branch is seeing build failures on Windows:
1. make sure you’re tracking an up-to-date mozilla-central: the last couple of fixes landed earlier this week and more fixes will be coming down the line
2. try setting a clobber for Win32 builds on your branch.
3. ask in #developers or #pymake – perhaps someone might be able to help you out
4. if all else fails, file a bug with releng similar to this one.
While try builds are much faster now, we also lose the ability to build Firefox 15-17 on them. ~~I spent a bit of time investigating workarounds but didn’t get anywhere, unfortunately.~~ Update 3/9: Simply add this patch to your queue before pushing to try.
This also means that you need to pushing an up-to-date mozilla-central to try, otherwise you will see failures on Windows.
One out of every 20 builds or so is failing with a strange error in dom/bindings/test – this seems to be corruption caused by a race condition that isn’t being properly handled somewhere but honestly shouldn’t be happening in the first place. If you see an error in dom/bindings similar to the one in this log, please retrigger that build. The error’s being tracked in this bug.

I’d like to say thanks to everyone who helped push it over the finish line: ted, khuey, coop, gps, glandium, catlee, bsmedberg, bhearsum, and anyone else if I missed them (sorry!).

Civilising global state, take 2

2012-08-04T13:30:00-07:00

A few months ago, I wrote a post about sane global state via parameters. Now Racket has parameters, but most other languages don’t. How important parameters are for a dynamic language was driven home to me while working on getting the Pymake build system working on our tinderboxes.

Pymake is a Python reimplementation of the venerable GNU Make, with two big advantages over it:

Parallel GNU Make (at least the MSYS variant) is extremely buggy and prone to deadlocks on Windows, which means the only option is to run serially (-j1). Pymake doesn’t have any issues, so build times are much faster on machines with enough cores.
The Mozilla build system uses recursive Make. Each recursive call spawns another Make process, and process spawning is rather expensive on Windows. Pymake performs all its recursion within the same process, which speeds build times on Windows even further.

Pymake also provides a few bonus features, one of which is the ability to run Python files “natively”, meaning within the same process. This helps speed up builds even more by avoiding spawning extra Python processes. Of course, such scripts have to be written with care to avoid trampling over Pymake itself.

One of the scripts for which this feature was enabled was pythonpath.py. By default, Python only loads modules from the current directory and its system directories. pythonpath.py lets us specify additional directories to load modules from. (Yes, virtualenv is a much better solution, and we’re going to switch to it soon.)

To do its work pythonpath.py must modify important global variables like sys.argc and sys.argv. Since it doesn’t undo these modifications, it causes hard-to-debug problems down the line. To fix such issues, one would need to consider every eventuality: returns, exceptions, etc. Not only that, one might need to consider modifications by others as well.

As a result, we had to disable this feature for it, slowing down builds by a bit.

If those variables were parameters, one could simply wrap the modifications up in a parameterize and expect things to Just Work. Since everyone would be doing the same thing, things continue to Just Work no matter what is run.

Build System on BugsAhoy

2012-06-30T11:25:00-07:00

Josh Matthews’ amazing BugsAhoy now has support for the build system. All open bugs labelled [good first bug] and with a listed mentor in the Core::Build Config and MailNews Core::Build Config components are listed. If you’re interested in one of the bugs, post a comment there or find us in #pymake on irc.mozilla.org.

Assuming makes an ass out of everyone, or how Ruby's build system sucks

2012-05-15T05:30:00-07:00

At Mozilla, our build system has a firm rule we only grudgingly violate: explicit is better than implicit. What that means is that if we depend on a library foo and we don’t find it on the machine we’re building on, we fail instead of silently assuming the user doesn’t want to build in support for foo. If the user really wants that, she would need to pass in a --disable-foo configure flag saying so. This means we know exactly what we’re shipping as binaries, and users know exactly what to expect.

Once you spend a lot of time working with Mozilla code, you sometimes forget other projects don’t follow such obviously important rules. Case in point: Ruby. A default Ubuntu install builds Ruby out of the box. Of course, when you then try to do anything remotely useful:

% gem install heroku

Ruby fails with a cryptic no such file to load -- zlib (LoadError).

Turns out Ubuntu doesn’t come with the zlibg1 dev library, which means the Ruby build system assumes you don’t care about zlib support and happily builds without it.

Great, so you installed the library and built Ruby again, and gem actually worked. Now, you try to log in to Heroku:

% heroku login

… and Ruby fails with yet another no such file to load -- net/https error. At least the error message is slightly less cryptic this time, since it tells you to apt-get install libopenssl-ruby. Which means you need to install the library and rebuild Ruby a third time.

God knows how many more libraries the build system’s assumed I don’t care about.

no-www considered harmful

2012-05-08T07:38:00-07:00

I remember reading about the no-www movement years ago. At that time it struck me as a cool thing to do, but I now know that no-www is a really bad idea in general. That’s because the DNS system lets you assign CNAME records to subdomains, including the www subdomain, but only A or AAAA records to root domains. In other words, you can say that www.example.com is the same as test.example.org, but you can only say that example.com is the same as a particular IP address.

What’s the benefit of this? Well, if you use a CNAME you can let the owner of test.example.org deal with anycast and routing issues, but if you use A records you have to deal with them yourself. If you do decide to deal with them yourself, you’ll be frustrated when your DNS updates take a while to propagate.

More generally, CNAME is an example of the most powerful law of computer science in action, and A isn’t.

All problems in computer science can be solved by another level of indirection. David Wheeler

See Heroku’s documentation on the matter.

Switching to Octopress

2012-05-07T07:35:00-07:00

I finally made the switch to Octopress today. I’ve managed to migrate all my Blogger posts to it, and set up HTTP 301 redirects so that people hitting my old posts and feeds aren’t lost.

I’m hosting my blog on Heroku, and the theme I’m using is the default Octopress theme (which is actually really nice) with a few tweaks. I’ve also played around with OpenType features, which only work in IE 10 on Windows 8, Firefox 15 and above on all platforms, and Chrome 16 and above on Windows and Linux. (Firefox 14 and below do support OpenType features, but with a different syntax I’m too lazy to support.)

Web fonts and Windows

Windows’ GDI ClearType has a really bad time dealing with most web fonts, including the ones I’m using for this site. DirectWrite does much better, but only works in IE 9 and above and in Firefox with hardware acceleration. Thus I made the call to disable web fonts for body text on Windows and use the similar-looking system font Constantia instead. I’ve left web fonts enabled for titles though.

Civilising global state

2012-05-02T13:58:00-07:00

It seems to be universally agreed that global mutable state is considered harmful. It makes dependencies less clear, means that functions are harder to test, introduces races, has to be handled separately when exceptions occur, and so on.

Yet anyone who has programmed for a while knows that there are always situations where global state is useful and convenient. Perhaps adding it to every single function call’s signature is a pain; perhaps you need to set up a callback and need some data there but can’t create a closure for it; perhaps you’d like to look up the stack for security reasons; perhaps you simply don’t want to refactor 90% of your code. (You might think you’re avoiding the pitfalls of global state by using singletons or static variables, but you’d be wrong.)

Funnily enough, we’ve had a solution to these use cases since the dawn of programming languages: dynamic scope. Dynamically looking up the stack for variable bindings is thread-safe, exception-safe and remarkably less error-prone than global state. It has also quite rightfully been mostly abandoned in favour of lexical scope.

But what if you could get all the great properties of dynamic scope with lexically scoped variables? That is what Racket achieves with parameters. Racket parameters

are explicitly declared as such, so you wouldn’t confuse them with regular variables
hold their value within blocks where they’re defined, called parameterize blocks
can be nested safely and work as you would expect them to
importantly for a Scheme dialect, don’t interfere with tail call optimization
are, by their very structure, thread-safe and exception-safe — it is always beautiful to see the semantics you want be a natural, obvious consequence of the syntax you’ve created

So the next time you’re tempted to use a global variable to solve a problem, don’t blame yourself; instead, blame the language you’re using for providing insufficient abstractions.

Transferring apps and data from one Android device to another

2012-04-22T10:01:00-07:00

My Nexus S’s screen has been having some issues with touch lately, so I’ve decided to send it in for repairs. I’ve really come to depend on a smartphone, though, so I went out and bought another Android phone to tide me over until my Nexus S is fixed: an HTC One V.

Now my phone had a lot of data, none of which I wanted to lose. Google’s cloud-based backup service unfortunately doesn’t do much in my experience, and the total amount of data I had (over 2.5 GB) was pretty large anyway, so I decided it’d be best to use my computer to do a full backup and restore.

Full backups, restores and data transfers have always been a big issue with Android. Android 2.x requires you to root your phone and then use an application like Titanium Backup, which is honestly ridiculous. Android 4.0 is actually a lot better if you’re willing to use the command line a bit. adb for Android 4.0 has working “backup” and “restore” commands, no rooting required.

Here’s my setup: I wanted to transfer all my applications, settings and data from an encrypted Nexus S running Android 4.0 to an unencrypted HTC One V, also running Android 4.0. The instructions should work for any pair of Android 4.0 devices.

I first made sure I had all the prerequisites. You essentially need most of an Android development environment on your computer.
- Download the latest version of the Android SDK for your platform, then run the SDK Manager and make sure you have the latest version of the Android SDK Tools and Android SDK Platform Tools.
- If you’re on Windows, you’ll also need the ADB USB drivers for both devices. The Nexus S drivers can be downloaded via the SDK Manager, while the HTC One V drivers are part of HTC Sync. (The HTC drivers are a pain to install: you need to force-install the “My HTC” drivers from C:\Program Files (x86)\HTC\HTC Driver\Driver Files\Win7_x64. Ridiculous.) For other OSes, see these instructions.
- If you’re on Windows, let the HTC Sync installer install the prerequisites (including drivers), but do not install HTC Sync itself. It comes with its own, older adb that interacts badly with the SDK’s adb.
I connected the Nexus S to my computer and turned on USB debugging on the phone. Then, from the platform-tools directory inside the SDK folder, I issued the following command:
```
adb backup -all -apk -noshared -nosystem -f nexus-s-backup.ab
```
See this guide for a complete list of options for the backup command.
On my Nexus S, a prompt (screenshot to the right) showed up asking me to enter my encryption password. I did so and moved ahead.
I now connected the HTC phone to my computer and turned on USB debugging. There’s one more thing I needed to do: in Settings -> Developer Options, set the Desktop Backup Password to the encryption password from step 3.
I then issued this command from the platform-tools directory:
```
adb restore nexus-s-backup.ab
```
Another prompt (screenshot to the right) showed up asking me to type in my encryption password, even though this phone wasn’t actually encrypted. (Very confusing!). I found I had to type the encryption password from my Nexus S into both fields – this was the only way the restore worked.
I finally copied the SD card’s contents manually. That’s about it. Most apps were restored properly, and whichever apps I downloaded from the ~~Android Market~~ Google Play Store showed up as installed there. A few didn’t make the cut: most notably Google’s 2-step Authenticator. I’m not sure why.

More complex than necessary, but I’m happy I didn’t have to redownload much and didn’t lose any settings or game progress. I hope for everyone’s sake that future versions of Android are better at it.

OS X mouse strangeness

2012-04-19T02:45:00-07:00

So I decided to upgrade my Apple notebook’s OS X install to Lion earlier this week. Due to an unfortunate series of events, partly my fault, mostly Apple’s, my MBR was corrupted and my Windows install became unbootable. So I was stuck with OS X for a few days until I had the time to fix my Windows install.

I’ve been running Windows and Linux with mouse acceleration turned off ever since I started using a high-precision mouse (it’s ugly, but really precise and comfortable to use), and I’ve come to appreciate it a lot. OS X’s acceleration curve is very different from Windows, which is fine, but the real problem is that there’s no built-in way to turn it off. Luckily, there’s a $0.99 app on the Mac App Store which lets you disable acceleration. That, and bumping up the resolution of the mouse a notch, made things better.

Things still felt strange, though, and once I fixed my Windows install and booted into it I realized why. OS X has severe input lag with the mouse. According to this blog post, the lag (which has been confirmed to exist by an Apple engineer) exists on all OS X installs, and on all mice and touchpads, and is 32 milliseconds. Now, 32 milliseconds might not sound like much, but I can easily tell the difference between 30 frames per second (33 ms/frame) and 60 FPS (17 ms/frame) while playing a game.

Windows and Linux do not have this problem: the input lag is less than the screen’s refresh rate, so the pointer is locked on to the mouse. Both feel significantly more responsive as a result.

If you’re using OS X now and are sensitive to such things, I suggest switching away until it’s fixed. You’ll be delighted you did.

Offered without comment

2012-04-07T07:22:00-07:00

“The bitterness, futility, and self-righteousness of most moral arguments can now be explicated. In a debate about abortion, politics, consensual incest, or what my friend did to your friend, both sides believe that their positions are based on reasoning about the facts and issues involved (the wag-the-dog illusion). Both sides present what they take to be excellent arguments in support of their positions. Both sides expect the other side to be responsive to such reasons (the wag-the-other-dog’s-tail illusion). When the other side fails to be affected by such good reasons, each side concludes that the other side must be closed minded or insincere. In this way the culture wars over issues such as homosexuality and abortion can generate morally motivated players on both sides who believe that their opponents are not morally motivated.”

– Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review. 108, 814-834. Link (PDF).

A pure Python library to access NTFS features

2012-04-04T02:33:00-07:00

I’ve written a small Python library to access Microsoft NTFS hard links and directory junctions that’s

Pure Python: no C or pywin32
Easy-to-use: the API follows Python’s own link APIs closely
BSD licensed: thanks to the Mozilla Foundation.

You can get it on GitHub.

Update 20/4: Thanks to Roman Lacko the package is now on PyPI. To install, run pip install ntfsutils.

If your build breaks with a PLY "invalid syntax" error on "dictionary"...

2012-03-13T23:38:00-07:00

… make sure there aren’t files called xpidllex.py and xpidlyacc.py in $topsrcdir/xpcom/idl-parser.

Removing them, and whatever .pyc files are in there, and then doing a clobber build should fix things up.

Paper voting is broken

2012-03-10T15:49:00-08:00

An interesting link popped up on Hacker News today: the Wombat voting system. It’s part of a long line of electronic voting systems (including Helios, created by Ben Adida, now at Mozilla) that aim to use cryptography to ensure the privacy and integrity of votes.

Of course, Hacker News, being the contrarian, negative community that it is, immediately shot it down by claiming that ”we don’t need computers in voting”, that such systems attempt to ”fix a problem which don’t exist”, and that such systems alienate ”common people without IT education”.

It’s sad to see people – on Hacker News of all places – being Luddites.

Paper voting is fundamentally broken, and will continue to be as the world’s population increases. The reason’s simple: it doesn’t scale. The last Indian general election had over 417 million people casting votes. 417 million. Imagine the sheer amount of time and resources it would take to count 417 million votes, and the possibility of error that might happen with improperly marked ballots, etc.

Fortunately, Indian elections haven’t had paper voting for a while. They’ve instead had standalone voting machines that keep a tally of votes for each candidate and reveal them on counting day. However, the election commission of India expects citizens to trust these machines on their word, and it’s been shown that these machines are actually quite vulnerable to fraud.

So, yes, the problem exists, and yes, we need computers to solve it. I’m glad to see brilliant minds hard at work on solutions. Democracy’s far too important to let voting systems not be end-to-end auditable by anyone with the requisite knowledge.

One last note: do people feel alienated when they check their email, type in their credit card number on a shopping site, or operate their bank accounts over the Internet without knowing all the mathematics and engineering making sure no one else is reading their data? The way crypto, and scientific knowledge in general, has always worked is that you either (a) have domain knowledge that lets you verify what is being said, or (b) trust others who do.

Dissent

2012-03-08T02:52:00-08:00

Over the last two days, I’ve read a number of blog posts I disagreed with. A couple of them allowed me to comment, so I did so.

On one of them, my comments are still up.

On the other, which adopted a roughly antipodal view, my comments were summarily deleted and I was told I was “derailing and silencing” the conversation by accusing the author of being “overemotional”, “oversensitive” and “taking things too personally”.

Before today, if you’d asked me which post that would have happened with, I’d have picked the first without a second thought.

Guess I have a lot to learn.

Dr. Seuss and the halting problem

2012-02-01T04:26:00-08:00

The unsolvability of the halting problem is a well-known fundamental result which says that it’s impossible to write an algorithm that determines if an arbitrary program halts given some input. It also implies a lot of other things – including, for example, that there’s no general way to determine whether a particular variable in a program has a particular value at a point, and famously, that every consistent formal system powerful enough to do arithmetic in will have unprovable statements.

I’m not going to talk much about it, though – instead I’m going to direct you to this extraordinary proof, written in verse, in the style of Dr. Seuss. It’s called Scooping the Loop Snooper, and it was written by Geoffrey K. Pullum, who you might also know from posts at the Language Log. Wow.

Important upcoming change: starting Gecko 12, pre-Windows 7 SDKs will no longer be supported

2012-01-26T09:57:00-08:00

tl;dr: if you’re on an outdated Windows build configuration, upgrade.

I just checked in a patch to mozilla-inbound to drop support for SDKs prior to the Windows 7 SDK. Depending on when you last updated your build configuration, your Mozilla build might break.

Everything’s documented on the Windows SDK versions page on MDN; here’s a quick summary of what you need to do:

If you’re on Windows 2000: Upgrade to Windows XP, Vista or 7.
If you’re on Windows XP SP1 or below: Update to the newest service pack.
If you have an older SDK (Vista or 2003): Install the Windows 7 SDK instead. You can either download it separately or use Visual C++ 10 Pro, which comes with it.
If you use Visual C++ 8 (2005) Express or Visual C++ 7.1 (VS2003): Upgrade to Visual C++ 10 (2010; recommended; pro or express) or Visual C++ 9 with Service Pack 1 (2008; pro or express).

MSVC9+ opt builds currently broken due to a compiler bug

2012-01-25T03:36:00-08:00

Optimized builds with Microsoft Visual Studio 2008 (VC9) and above are currently broken because of what we believe is a compiler bug. More information and workaround in bug 718541.

C# 5.0 to have fresh bindings per iteration

2012-01-18T02:08:00-08:00

Straight from Eric Lippert. Great news. Now if only ECMAScript Harmony included a fix for this bug…

More experiences with an Apple notebook

2012-01-14T04:55:00-08:00

It’s now been eight months since I got my first Apple notebook. A few months ago I wrote about my initial opinion, where I was pretty sure it would be my last Apple notebook too.

Since then I’ve had the chance to use it in a variety of situations. Spoiler: none of what I’ve seen has improved my opinion of it one bit.

The decision to make the outer body out of aluminium is literally shocking. If the notebook isn’t plugged in to a grounded socket (for instance, if I’m using the plug that comes with the power brick BY DEFAULT), I’m liable to get electric shocks if I touch the casing. I received a couple of shocks, a mild one and a jolting one, before I realized what was happening. Electrical common sense is that if the outer surface is electrically conducting, it MUST be grounded properly. Having an ungrounded plug by default, or even having one in the first place, is inexcusable. (Update: I’ve had several people complain to me about this, and one person also complain about his plastic macbook’s screws shocking him several times. I’m clearly not the only one with this issue.)
The Wi-Fi reception is the worst I’ve ever seen in a laptop, and only slightly better than the reception my Nexus S with its puny little antenna gets. Friends tell me it’s because the aluminium casing acts as a Faraday cage and attenuates the signal. The “unibody” marketing’s clearly far more important to Apple than shipping a working product. (Update: guess who says metal has a “very high” potential to interfere with wireless connections?)
The original power adapters were T-shaped. However, presumably because Apple didn’t like the look of and subsequently didn’t include the strain-relieving flexes found on all other cables, they were easily frayed. To “fix” this, they started using L-shaped adapters. Of course, what it now means is that depending on the way I insert it, either the power cord blocks the Ethernet port or it gets subjected to strain if I tilt the notebook back.
There’s no VGA, DVI or HDMI port, so I need to carry around a set of three dongles everywhere I go. There’s plenty of space on the left side, too, so that’s not an excuse.
The lack of working sleep is more annoying than I thought it would be. Amazingly, the EFI equivalent to the POST takes almost as long as Windows resuming from hibernation. A few people seem to be working on getting Windows to boot via EFI, and my hopes are mostly pinned on that.

My iPod nano media player, the only other Apple product I own, is actually well-designed (save for the fact that I need to keep the piece of crap called iTunes around on my computer, even though I don’t need to use it.) This is not. This stinks of form-over-function failure.

Edit (25/1): Two inline updates.