If anyone knows the joys and sorrows of managing software development projects, it would be Linus Torvalds, creator of the world's most popular open-source software program: the Linux operating system. For more than 20 years, Torvalds has been directing thousands of developers to improve the open source OS. He and I sat down to talk about effective techniques in running large-scale distributed programming teams – and the things that don’t work, too.
“The first thing is thinking that you can throw things out there and ask people to help,” when it comes to open-source software development, he says. “That's not how it works. You make it public, and then you assume that you'll have to do all the work, and ask people to come up with suggestions of what you should do, not what they should do. Maybe they'll start helping eventually, but you should start off with the assumption that you're going to be the one maintaining it and ready to do all the work.”
Torvalds continues, “If you start off with some ‘kumba-ya feeling’ where you think people from all the world are going to come together to make a better world by working together on your project, you probably won't be going very far.”
“The other thing—and it's kind of related—that people seem to get wrong is to think that the code they write is what matters,” says Torvalds. Most software development managers have seen this one. “No, even if you wrote 100% of the code, and even if you are the best programmer in the world and will never need any help with the project at all, the thing that really matters is the users of the code. The code itself is unimportant; the project is only as useful as people actually find it.”
I'll add at this point that this isn't just a programmer problem. I've seen entire companies get locked into the idea that “perfecting” the program was everything. They then neglected what the users wanted from the program, supporting the users and so on. Most of us who've been in the business for a while have seen this cycle play out over and over again.
Expanding on that second point, Torvalds says that's why the Linux kernel team is “so very anal about the whole ‘no regressions’ thing, for example. Breaking the user experience in order to ‘fix’ something is a totally broken concept; you cannot do it. If you break the user experience, you may feel that you have ‘fixed’ something in the code, but if you fixed it by breaking the user, you just violated that second point; you thought the code was more important than the user. Which is not true.”
Torvalds concludes, “Way too many projects seem to think that the code is more important than the user, and they break things left and right, and they don't apologize for it, because they feel that they are ‘fixing’ the code and doing the right thing.”
To that I can only add “Amen!”
On the Importance of Development Tools
I also asked Torvalds about Software Configuration Management (SCM) tools like his ownGit version control system. He replied, “I don't think tools are all that fundamentally important.”
“Now, what is important is that there's a good workflow for the project, and tools can certainly help with that,” said Torvalds. “But most projects don't necessarily really need tools. There's a lot of projects that simply don't have enough changes to really require any tools at all for their work flow; if you only have a few hundred patches per release, you can maintain those just about any way you want, including entirely by hand.”
Linux is a different story of course. “For the kernel, we have thousands of patches flying around every release, and a release roughly every three months, and so for us the tools really are very important,” he says. “But I still don't think it was all that big a mistake to just do tar-balls and patches for the first few years of development; it was a much smaller project back then, and it took several years for the lack of tools to really become a problem.”
Besides, “Some tools encourage workflows that are actively detrimental, and I think CVS [Concurrent Versions System, a version control system] for example has caused a lot of projects to have the notion of a 'commit cabal,'” Torvalds continues. “I personally tend to think tar-balls and patches are actually preferable to that – if only because they make all developers 'equal,' and you don't get the kind of model where certain people have 'commit access,' and the rest are second-class citizens. Sometimes it's better that everybody is a second class citizen than that some people have an easier time at it.”
Torvalds, I should note, knows CVS well and has hated it for years. As he said in a Google Talk in 2007, “I hate CVS with a passion.”
Torvalds continues, “Much more important than the tools is the people. The maintainers, and the mindset.”
Keeping People On Track
And how do these people work together today? I asked Torvalds about the role of theLinux Kernel Mailing List (LKML) in the process. He replied, “I think Linux used to ‘happen’ more on LKML than it does these days. The signal-to-noise ratio and just the pure volume of LKML means that most developers simply don't have time to really read LKML—at best they scan subject lines. As a result, these days I'd argue that most of the real development happens within the sandbox of single developers, and then email on more of a person-to-person scale is actually how things really get done.”
That said, “That doesn't mean that LKML isn't important; it means that LKML has become the ‘public band’ of all those individual email threads,” Torvalds adds. “So what ends up happening is that you have maybe four or five people involved in a discussion about their work, but LKML stays cc'd on the whole thing. That turns what would otherwise be a purely private discussion into something where others can jump in.”
Here's how it works, “A lot of people actually don't really ‘read LKML;’ they often auto-archive it, but then react to certain keywords or, more often, [to] key people being involved in the discussion.”
“It also acts as a kind of archiving notion,” Torvalds continues, “so that people can refer to it later, and a lot of bug reports end up being found by Googling for them. If somebody raises an issue, it may well be some odd hardware problem, but if Google shows that it's been raised several times on LKML in the past, that starts to indicate that it may be obscure, but it's certainly not some totally isolated issue.”
“So I think LKML is really quite important, but no, it's not how we keep people ‘on track,’” he says. “All the developers tend to be pretty self-motivating, and they all have sane ideas (well, the core ones do by definition – because that's how they became core developers, by showing that they had good taste and high motivation). It's important simply because that ‘public part’ of the discussions are still important, even if in practice it's often a pretty small core in any particular discussion. Things are simply different when they happen in the open,” concludes Torvalds.
On Delegating – and Staying Sane
Once, Linux was a solo project. It now has thousands of committers and contributors. I then asked, “How much delegating do you these days? Any thoughts on how to delegate to keep one's sanity and the workflow flowing?”
“If there's one thing I've learnt, it is that you have to learn to let go and not try to control people and the code,” he says. “If you don't think somebody else can do it on their own without your oversight, you might as well give up immediately as a maintainer.”
He continues, “Yes, I often get involved in small details, but it's not because I don't trust people or don't delegate. It's because some small detail ends up being brought up to me. Either it's a bug (and they are almost all just silly small details that got overlooked), or it's just some workflow issue that bothers me (like me complaining about the developer names not showing up properly in the logs earlier today to one sub-maintainer).”
Still, says Torvalds, “Those details have to be occasional details, not the kind of ‘look over the shoulder of the developer to check everything he does.’ I trust sub-maintainers to do the right thing 99% of the time. And then, very occasionally, I end up complaining loudly about something.” Say, for example, on how the open-source GNOME desktop is, or rather isn't, moving forward.
So, there you have it. That's some of the ways Torvalds does it. And, if you think you know better, ask yourself: Have I created a world-class operating system that runs most supercomputers, stock-exchanges, and websites like Google? If your answer's no, I'd re-read his answers and take a long hard think about how you've been managing your own projects.