xkcd #3117: Replication Crisis
Title text:
Maybe encouraging the publication of null results isn’t enough–maybe we need a journal devoted to publishing results the study authors find personally annoying.
Transcript:
Transcript will show once it’s been added to explainxkcd.com
Source: https://xkcd.com/3117/
You must log in or register to comment.
There really is an xkcd for everything.
is there an xkcd about there being an xkcd about everything though?
I once experienced this firsthand. The postdoc kept coming by, trying to find out how we were messing up something he had written a paper on. Eventually, he tried to reproduce it himself. No one could reproduce it, not even him.
I liked once reading an article that showed that with some major findings from important scientists that were later shown to have a wrong value, it wasn’t that a second study promptly “snapped” to the correct values. Instead, over time, subsequent studies incrementally moved to the right value.
On one hand, this is good in that the process does ultimately work, and we got to the right value, though it could take quite some years.
On the other hand, this is embarrassing, because it suggests that people doing follow-up studies to a prestigious person second-guess their own results (“Doctor So-and-So can’t possibly be wrong…it must be me in error”) and aren’t willing to report the full deviation, so they’ll bang on an experiment until they get a value that isn’t that far off and report that.
I can’t seem to find reference to it in the explainxkcd Wikipedia articles, but I remember being intrigued.
On the other hand, this is embarrassing, because it suggests that people doing follow-up studies to a prestigious person second-guess their own results
Some people have started distinguishing between “science”, i.e. the scientific method, and "the science’, i.e. the total collective body of results.
“Science” is precious and pure. It’s never right or wrong, it just approaches correctness as it progresses.
“The science” is always inherently suspect since that’s how “science” works, but it’s frequently treated as indisputable fact. This is problematic for a number of reasons, and the replication crisis is at the top of that list.
The most famous version of this might be Millikan’s oil drop experiments to measure the mass of an electron. His notebook is full of qualitative judgments of his measured values and which ones to include in the final determination. The mass of an electron settled down pretty smoothly
Replication failed successfully.
Or failed replications replicated successfully (could be useful, but was originally marked as failed to replicate)
