Experimental good practice
Like hitting piñatas, scientific experiments need blindfolds. Image: Juergen. CC-BY.I once sent some samples to a biostratigrapher, who immediately asked for the logs to go with the well. 'Fair enough,' I thought, 'he wants to see where the samples are from'. Later, when we went over the results, I asked about a particular organism. I was surprised it was completely absent from one of the samples. He said, 'oh, it’s in there, it’s just not important in that facies, so I don’t count it.' I was stunned. The data had been interpreted before it had even been collected.
I made up my mind to do a blind test next time, but moved to another project before I got the chance. I haven’t ordered lab analyses since, so haven't put my plan into action. To find out if others already do it, I asked my Twitter friends:
Anyone ever heard of a blinded experiment (e.g. in the lab) in geoscience? Gathering evidence (or lack of it)...
— Matt Hall (@kwinkunks) August 20, 2012
Randomized, blinded, controlled testing should be standard practice in geoscience. I mean, if you can randomize trials of government policy, then rocks should be no problem. If there are multiple experimenters involved, like me and the biostrat guy in the story above, perhaps there’s an argument for double-blinding too.
Designing a good experiment
What should we be doing to make geoscience experiments, and the reported results, less prone to bias and error? I'm no expert on lab procedure, but for what it's worth, here are my seven Rs:
- Randomized blinding or double-blinding. Look for opportunities to fight confirmation bias. There’s some anecdotal evidence that geochronologists do this, at least informally — can you do it too, or can you do more?
- Regular instrument calibration, per manufacturer instructions. You should be doing this more often than you think you need to do it.
- Repeatability tests. Does your method give you the same answer today as yesterday? Does an almost identical sample give you the same answer? Of course it does! Right? Right??
- Report errors. Error estimates should be based on known problems with the method or the instrument, and on the outcomes of calibration and repeatability tests. What is the expected variance in your result?
- Report all the data. Unless you know there was an operational problem that invalidated an experiment, report all your data. Don’t weed it, report it.
- Report precedents. How do your results compare to others’ work on the same stuff? Most academics do this well, but industrial scientists should report this rigorously too. If your results disagree, why is this? Can you prove it?
- Release your data. Follow Hjalmar Gislason's advice — use CSV and earn at least 3 Berners-Lee stars. And state the license clearly, preferably a copyfree one. Open data is not altruistic — it's scientific.
Why go to all this trouble? Listen to Richard Feynman:
The first principle is that you must not fool yourself, and you are the easiest person to fool.
Thank you to @ToriHerridge, @mammathus, @volcan01010 and @ZeticaLtd for the stories about blinded experiments in geoscience. There are at least a few out there. Do you know of others? Have you tried blinding? We'd love to hear from you in the comments!
Matt Hall
In Thursday's post, I went on to look at how I might have blinded the biostrat experiment.
Reader Comments (2)
Typically when we process a seismic survey we are given some wells by the client company, but the client company always keeps a couple wells from us to use as a kind of blind test in their own shop.
@Chris: I think that's smart, and fairly common practice, but I do wonder about Catch 22 — did you reduce bias or over-fitting of models at the expense of a better result? The best practice might be to give you all the wells, but then you systematically blind the process to each (or each pair or whatever), and see how stable the result is. There may be no other way to know where the 'tipping point' of over-fitness is.