I was troubled by Dr. John Cook's recent post on the different programming styles of scientists and programmers. It felt like an exaggerated dichotomy to me: the "cowboy" scientist who tosses off throw-away scripts, and the cube-dwelling programmer implementing a serious product. Dr. Cook was trying hard to think of this dichotomy as a cultural distance, based on his experience working with and trying to understand the needs of both groups. However, I think he missed the point that joins scientists and programmers: reproducibility.
As the article mentions, programmers understand that their code needs to produce predictable results. They set up tests to verify this. Entire fields of computer science research revolve around proving code correctness. As a developer of numerical algorithms, I think of my code as an experimental apparatus for testing my hypothesis about the superiority of one algorithm or optimization over another. I can't say I've always done my best to make my code work that way, but I'm working on it.
I think many scientists haven't yet learned that code is really just experimental apparatus, and the reason for this in part may be that modern scientists are not rewarded for reproducing "known" results. Programmers often are -- why else did Google+ follow Facebook, which in turn followed MySpace? Now, a lot of code really is throw-away analysis scripts. This is probably what Dr. Cook means by the following: "Programmers need to understand that sometimes a program really only
needs to run once, on one set of input, with expert supervision." However, simulations are not like this. A simulation code is an experimental apparatus. It need not
be permanent, but it should be constructed well enough to make its
results trustworthy. Just as importantly, it should be described and
documented sufficiently well that other scientists can replicate its
results.
The real problem is not a cultural difference between scientists and programmers: the real problem is reproducibility.
Subscribe to:
Post Comments (Atom)
4 comments:
I had second thoughts after I wrote the "run one time" line. I understand the common lack of reproducibility in science and I didn't want to imply that I approve of that. My point was that programmers may want to design software to do more than it really needs to do, maybe because their attention is immediately drawn to what they understand.
Here are some blog posts I wrote about reproducibility.
Thanks for clarifying your point! I didn't mean to sound so critical of your article in my comment and the post. You make a good point that programmers ("software _engineers_") tend to over-_engineer_ software. Perhaps that's only because they enjoy writing code, whereas some scientists really hate it!
Thanks for the links to reproducibility articles! I think about this issue relating to performance and accuracy of numerical approximation algorithms. It's good to have a slightly different perspective (analysis of experiments) which is nevertheless related, and from which I can learn.
This post reminds me of this proposal for an academic open-source license:
http://matt.might.net/articles/crapl/
I've seen that proposal. I like that it encourages releasing code, and encourages authors not to make premature judgments about the usefulness of their code. That's a good start (some possibly broken code is better than no code).
Post a Comment