better than it is

An interesting isotope is detected in the CRU report fall-out plume. Apart from the very high concentrations of concern-troll, tone-troll, and pure drivel, there is something worth learning from.

For this reason, many software professionals encountering science software for the first time may be horrified. How, they ask, can we rely on this crude software, developed in primitive conditions – by amateurs, working with such poor tools and such poor understanding of the field? This is a common reaction to GISTEMP, and is exactly the reaction which many critics have had, some very publicly, to the software published with the CRU emails. Such critics do have a point. Science software should be better than it is. Scientists should be provided with more training, and more support. But consider the uses to which science software is put. Most
software written by scientists:

* consists of tiny programs;
* which will only ever be run a small number of times;
* over the course of a few weeks as it is being developed;
* by the scientist who wrote it;
* on data gathered by that scientist’s team;
* concerning a scientific field in which that scientist is expert;
* to perform data processing on which that scientist is expert; and will be discarded, never to be used again, as soon as the paper containing the results is accepted for publication.

There are hardly any scientists today who don’t do some programming of some sort; there’s not much science that doesn’t involve churning through really big data sets. As a result, there’s a lot of it about. Which reminds me of this Eric Sink post from 2006, about the distinctions between “me-ware, us-ware, and them-ware”. Me-ware is software that you write and only you use; us-ware is software that is used by the same organisation that produces it; them-ware is software that is produced by a software company or open-source project for the general public.

There’s a gradient of difficulty; the further from you the end-user is, the less you know about their needs. On the other hand, if you’re just trying to twiddle the chunks to fit through the ChunkCo Chunkstrainer without needing to buy a ChunkCo Hyperchunk, well, although you know just how big they are, you’re unlikely to spend time building a pretty user interface or doing code reviews. Which only matters up to a point; nobody else would bother solving your problem.

But this can bite you on the arse, which is what happened to the climate researchers. It’s fair to say that if you’re processing a scientific data set, what actually matters is the data, or the mathematical operation you want to do to it. You won’t get the paper into Nature because you hacked up a really elegant list comp or whatever; they won’t refuse it because the code is ugly. Anyone who wants to replicate your results will probably roll their own.

This is OK, but the failure mode is when the political equivalent of Brian Coat comes snooping around your #comments or lack of them. Perhaps I should tidy up the Vfeed scripts while I’m at it.


  1. The Great Simpleton

    Except that we are expected to change the whole world’s economy effectively making us poorer based on Vfeed or the average scientist’s work.

    When AGW became the major political and economic driving force it has become the scientists involved had a duty to sort out their code and more importantly base data.

  2. Actually, most of those small programs that scientists write are doing the _same thing_ (or almost the same thing) as the small programs that all the other scientists write. Lots of redundant effort there. Life would be nicer if there were some standardised science-doing computing tools around: cat and sed for ugly datasets, as it were.

  3. Tom

    “Except that we are expected to change the whole world’s economy effectively making us poorer based on Vfeed or the average scientist’s work.”

    This would only be true if the margin of error of all these little programs was in the same direction. There’s no reason to regard this as true – policy is *always* a balance of probabilities thing.

    One counter-example – the 1960s London Ringways highway engineers used off-the-shelf car trip prediction methods from the USA to come up with the necessary scaling for their designs. I suspect if they’d rolled their own the resulting plans would have been rather less ludicrous (there was one section in West London predicted to need about 12 lanes each way, for instance, busier than any LA freeway).

  4. duaneg

    This is all very pertinent and under-appreciated. The key point is that throw-away single-use software is invariably rubbish, no matter who it is written by.

    Anyone, even the very best software developers, would be embarrassed to show you the quick hacks that glue their local systems together. Diversity of build and running environments requires more robust software, which takes time, effort and experience.

    This is just one of the reasons why custom in-house software is invariably far worse than equivalent OTS alternatives and why proprietary OTS software is usually worse (at least in software engineering terms) than equivalent FOSS.

    There are some excellent science and engineering tools out there, but from what I’ve seen and heard most software in the field is in-house shitware of the worst kind. All too frequently written in head-thumping spaghetti FORTRAN.

    Scientists and engineers could really do with dedicated, trained and experienced programmers on their teams, but how is that supposed to happen? The traffic is all the other way. The guy sitting next to me at work used to be a physicist at CERN. I’ve lost count of the number of science, engineering and maths refugees I’ve worked with.

  5. Cian

    A lot of science software is written in Python these days. There are some really good frameworks and libraries. iPython (which is a fantastic hacker tool) was written for and by scientists. There’s also some excellent statistical programming languages that were developed by statisticians and social scientists. Also there is some very good, well written, general purpose OSS software out there written by scientists. They’re probably as obsessive about their tools as any GNU zealot – freedom not being an abstraction to them. Certain fields have always seemed to churn out good programmers. Astronomy being one for some strange reason, whereas I’ve never heard of a chemist who could program a damn.

    There are also some interesting paradigms emerging from science. One of them being literate programming, where the code is literally embedded in the documentation, rather than the other way round. Scientists will use this to generate papers which include the code, and the results. Its a very elegant solution to a number of problems.

    This is more a story about single-use software, which as duaneg says is rubbish. It takes time to write good code, and why waste time if you don’t have to. Of course, what is also underappreciated is how much software, written by supposed professionals, has essentially grown out of a one-time hack that was really useful…

    I don’t have a problem with the software being crap if nobody else is ever going to use it. But isn’t some of this science code being used to prove stuff? That seems more problematic. How can the scientist be certain there aren’t any bugs which distort the results? And how can any other scientist really trust the outcome?

    Cian

  6. Just to restore some context: the passage you quote is from Nick Barnes’ submission to the Independent Climate Change Email Review Committee.

    Nick also takes aim at some of the misinformed criticism of CRU on the basis of the “HARRY READ ME” file. From later in Nick’s submission:

    This is a very common problem: that no one person knows how a whole system operates. People leave, change jobs or careers, die, or simply forget. This appears to be the background to the infamous HARRY READ ME file published with the CRU emails: it appears to be a very detailed and frank record of a painstaking effort to work out and document the operation of a complex piece of software which has been developed over several years by several different people. Despite all our much-vaunted expertise and skill, this problem is also commonplace in the software industry. I myself have written broadly similar documents to HARRY READ ME, for very similar reasons—although rarely at such great length.

    Cian asks, “How can the scientist be certain there aren’t any bugs which distort the results? And how can any other scientist really trust the outcome?”

    No-one can be certain that there aren’t any bugs which distort the results of a computer program, just as no-one can be certain that a complex experiment was really performed correctly. The way that scientists come to trust the outcome is the usual way that any result becomes trustworthy in science: by critical scrutiny, multiple independent replications, and success in predicting and explaining new phenomena.

    (Disclosure: I’m a colleague of Nick’s, and I contributed a small amount of code to Clear Climate Code.)

  7. yorksranter

    Gareth: thanks!

    Everyone else: thanks for outstanding thread. If you can keep it up a few days more I’ll be back in the blogosphere…

  8. Cian

    Gareth:

    I don’t think we’re really in disagreement. I’d totally agree with Nick’s submission and it was kind of the point I was making. I’ve also written documents like it, and I’ve written plenty of crap code because I was against a deadline and didn’t think it would ever get reused. Sometimes that assumption was correct…

    My only possible disagreement is that I think quite a lot of scientists are very good programmers. There are some really nice tools and techniques emerging from the scientific community through the FOSS community. For example: literate programming. Actually, in my experience the average professional programmer is not great at their job.

    No-one can be certain that there aren’t any bugs which distort the results of a computer program, just as no-one can be certain that a complex experiment was really performed correctly.

    No you can’t, but well written code can at least be read and inspected by others. Whereas a ‘write only’ piece of Perl code, or an entry for the Obfusticated-C competition is basically a black box. It doesn’t matter if its a convenience script whose purpose can easily be described in a paper (and replicated by whoever), it kind of does matter if its a complex piece of climate modelling code and the paper is basically reporting on its outputs.

    Obviously this is a general point as I know next to nothing about the quality of the software in this particular instance.

    The way that scientists come to trust the outcome is the usual way that any result becomes trustworthy in science: by critical scrutiny, multiple independent replications, and success in predicting and explaining new phenomena.

    And Clear Climate Code looks like an excellent contribution to this. I didn’t know that it existed, but I’m glad it does. Maybe this can serve as a model for how computer scientists can support scientists?

  9. On improving the quality of scientific software: the aim ofSoftware Carpentry seems to be to try and give best practice teaching to make programming at least workmanlike.

    That being said, how sophisticated does scientific code need to be? Most of the stuff I wrote as an engineering student simply cranked through a dataset and spat out files of output; although the code would probably have given a computer scientist hives (written in FORTRAN!), it worked well enough for my purposes.

  10. Nick Barnes

    Indeed, kudos to Software Carpentry, who I have previously mentioned on the Clear Climate Code blog.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s



%d bloggers like this: