Try entering “failure to replicate” in a google search (or better still, let me do that for you) and you will find no shortage of hits. You can even find a reproducibility initiative. Nature has a whole set of articles on the topic. If you live in New Zealand you have probably not escaped the coverage in the news about the botulism bacteria that never was, and you might be among those puzzled about how a lab test could be so “wrong”.
Yet, for scientists working in labs, this issue is commonplace.
Most scientists will acknowledge that reproducing someone else’s published results isn’t always easy. Most will also acknowledge that there they would receive little recognition for replicating someone else’s results. They may even add that the barriers to publish negative results are also too high. The bottom line is that there is little incentive to encourage replication, more so in a narrowing and highly competitive funding ecosystem.
However, some kind of replication happens almost on a daily basis in our labs as we adopt techniques described by others and try to adapt them to our own studies. A lot of time and money can be wasted when the original article does not provide enough detail on the materials and methods. Sometimes authors (consciously or unconsciously) do not articulate explicitly domain-specific tacit knowledge about their procedures, something which may not be easy to resolve. But in other cases, articles just simply lack enough detail about what specific reagents were used in an experiment, like a catalog number, and this is something may be able to fix more easily.
Making explicit the experiment’s reagents would should be quite straightforward, but apparently it is not, at least according to the new study published in PeerJ*. Vasilevsky and her colleagues surveyed articles in a number of journals and from different disciplines and recorded how well documented the raw materials used in the experiments were described. In other words, could anyone, relying solely on information provided in the article, be sure they would be buying the exact same chemical?
Simple enough? Yeah, right.
What their data exposed was a rather sad state of affairs. Based on their sample they concluded that the reporting of “unique identifiers” for laboratory materials is rather poor and they could only unambiguously identify 56% of the resources. Overall, just a little over half of the articles don’t give enough information for proper replication. Look:
But not all research papers are created equal. A breakdown by research discipline and by type of resource shows that some areas or types of reagents do better than others. Papers in immunology, for example tend to report better than papers in neuroscience.
So, could journals for immunology be better quality or have higher standards than the journals for neuroscience?
The authors probably knew we would ask that, and they beat us to the punch.
(Note: Apparently, the IF does not seem to matter when it comes to the quality of reporting on materials**. )
What I found particularly interesting was that whether a journal had good guidelines on reporting didn’t seem to make much of a difference. It appears the problem is more deeply rooted and these seeping through the submission, peer review and editorial process. How come neither authors, reviewers or editors are making sure that the reporting guidelines are followed? (Which in my opinion beats the purpose of having them there in the first place!)
I am not sure I perform myself too much above average (I must confess I am too scared to look!). As authors we may be somewhat blind to how well (or not) we articulate our findings because we are too embedded in the work, missing things that may be obvious to others. Peer reviewers and editors tend to pick up on our blind spots much better than us. Yet apparently a lot that still does not get picked up. Peer-reviewers don’t seem to be picking up on these reporting issues, perhaps they make assumptions based on what is standard in their their particular field of work. Editors may not detect what is missing because they are relying on the peer-review process to identify reporting shortcomings especially when the work is outside their field of expertise. But while I can see how not getting it right can happen, I also see the need to get it right.
While I think all journals should have clear guidelines for reporting materials (the authors developed a set of guidelines that can be found here), Vasilevsky and her colleagues showed that having them in place was not necessarily enough. Checklists similar to those put out by Nature [pdf] to help authors, reviewers and editors might help to minimise the problem.
I would, of course, love to see this study replicated. In the meantime I might give a go at playing with the data.
*Disclosure: I am an academic editor, author and reviewer for PeerJ and obtained early access to this article.
** no, I will not go down this rabbit hole
Vasilevsky et al. (2013), On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ 1:e148; DOI 10.7717/peerj.148
Imagine you drive into a motel in Gatlinburg TN, and see behind an open room door 2 guys setting up cameras pointing at the beds while two young women peek from the parking lot. Well, if it was in the mid ’90’s it might have been Drs Moiseff and Copeland setting up the equipment before venturing into Elkmont in the Smoky Mountains to study the local fireflies. (And one of the two women would have been me.)
Andy Moiseff and Jon Copeland started studying the population of fireflies in the Smoky Mountains National Park after learning from Lynn Faust, who had grown up in the area, that they produced their flashes in a synchronous pattern.
In the species they are studying (Photinus carolinus) the males produce a series or bursts of rhythmic flashes that are followed by a ‘quiet period’. But what is particularly interesting about this species is that nearby males do this in synchrony with each other. If you stand in the dark forest, what you see is groups of lightning bugs beating their lights together in the dark night pumping light into the forest in one of nature’s most beautiful displays.
Females flash in a slightly different manner and, as far as I know, they don’t do it synchronously either with other females nor with the males. One interesting thing in Elkmont is that there are several species of fireflies, and you can pretty much tell them apart by their flashing patterns. But as useful as this is for us biologists (since it avoids having to go through extensive testing for species determination), the question still remained of whether the flashing patterns played a biological role.
And this is what Moiseff and Copeland addressed in their latest study published in Science. They put females in a room where LEDs controlled by a computer simulated individual male fireflies. The LEDs were made to flash with different degrees of synchronisation and they looked at the responses of the females. They found that while the females responded to synchronous flashes of the LEDs, they really didn’t seem to respond when the flashes were not synchronous. Even more, they responded better to many LEDs but not much to a single one. What this means, is that if you are a male of Photinus carolinus, you better play nice with your mates if you want to get the girl.
What *I* want to know is how this behaviour is wired in the brain. At first hand, this seems like a rather complex behaviour, but in essence all that it seems to require is a series of if/then computations, which should not be too hard to build (at least not from an ‘electronic circuit’ point of view). But Bjoern Brembs reminded me of a basic concept in neuroscience: brains are evolved circuits, not engineered circuits. So, Andy and Jon, how *do* they do it?
Original article: Moiseff, A., & Copeland, J. (2010). Firefly Synchrony: A Behavioral Strategy to Minimize Visual Clutter Science, 329 (5988), 181-181 DOI: 10.1126/science.1190421
I am not sure why, but this week appeared to be filled with news about science to share. All of these are brought to you by the magic of Open Access or the efforts of people in the web to make science accesible to everyone.
I would normally not include articles published in Nature here, but this week David Winter from The Atavism pointed me to this one: “Complete Khoisan and Bantu genomes from southern Africa” by Stephan C. Schuster and a group of collaborators. The authors open their paper stating that
“The genetic structure of the indigenous hunter-gatherer peoples of southern Africa, the oldest known lineage of modern human, is important for understanding human diversity.”
The study has been published under a creative commons licence (http://creativecommons.org/licenses/by-nc-sa/3.0/) and the data has also been released here. I dont know whether Nature will ever move to a full Open Access format, but I think it is worth acknowledging that at least some of their material is made available withouth a subscription. To read a full review of the article, you can visit David Winter’s blog.
PLoS One (which yes is a fully Open Access journal) published an article on the cognition behind spontaneous string pulling in New Caledonian Crows, by Alex Taylor, Felipe Medina, Jennifer Holzhaider, Lindsay Hearne, Gavin Hunt, and Russell D. Gray. New Caledonian crows are better known for their ability to manufacture tools both from materials that they would normally find in the environment as well as some they would not. New Caledonian crows can solve rather complex puzzles, and for the most part, it has been assumed that this reflected some ‘higher’ cognitive ability that require building a cognitive scenario and imagination. In this study, the authors subjected crows to a series of tests, and conclude that:
Our findings here raise the possibility that string pulling is based on operant conditioning mediated by a perceptual-motor feedback cycle rather than on ‘insight’ or causal knowledge of string ‘connectivity’.
The finalists for best of Research Blogging are out and there is no shortage of interesting stuff to look into. Also out is the Open Laboratory 2009. This is a great collection of science blog posts that is really worth your money. So go on now, go get yourself a copy…
And if all that geekiness was still not enough, then you are in luck.
Next week will see Global Ignite Week: Ignite talks in 65 cities and 5 continents (and yes, there is one in Wellington on Tuesday). Ignites are a great presentation format (well, unless you are a speaker since they are really really hard to do well!). If you have not heard one before, there are plenty on YouTube Ignite Channel.
One more (an last). If you want to know everything there is to know about <ahem!> me :), thanks to the magic of Bora Zivkovic and The Blog Around the Clock, now you can. There should be a warning or disclaimer before I lead you to this link.
Full Disclaimer: I am an academic editor for PLoS One and I collaborate with the group behind the New Caledonian Crow Study
Congratulations to our fellow Sciblings who made it to the finals for the Research Blogging Awards 2010.
- Misc.ience (by Aimee Whitcroft) was nominated for best blog in Chemistry, Physics or Astronomy.
- The Atavism (by David Winter) was nominated for best lay-level blog.
Congratulations to you both (and if I have failed in identifying a sciblogger, please let me know!
Funny how we are really good, for the most part, at knowing where sounds are coming from. And it is funny since the ear provides the brain with no direct information about the actual relationship in space of different sound sources. Instead, the brain makes use of what happens to the sound as it reaches both ears by virtue of, well, being a sound wave and that we have two ears separated in space.
Imagine a sound coming from the front, the sound will arrive to the two ears at the same time. But if it is coming from the right it will arrive to the right ear first, and to the left ear a wee later. This ‘time difference‘ will depend on the speed of sound in air and how far apart our ears are. Even more, as the sound source moves from the far right to the front of the head those time differences will become smaller and smaller, until they are zero at the front. If one could put one microphone in each ear, one could reliably predict where the sound comes from by measuring that time difference. And this is exactly what a group of neurons in the brain does.
Easy enough? Not quite.
The way the brain works is that things on the left side of our body are mapped on the right side of our brains, and things on the right side of our bodies are mapped on the left side of our brains. So the ‘time comparison’ neurons on the right side of the brain deal mainly with sound from coming from the left (and neurons dealing with the sound from the right are on the left side of the brain). But to do the time comparison these neurons need to get the information from both ears, not just from only one side!
This raises this conundrum: the neural path that the information from the left ear needs to travel to get to the same (left) side of the brain will inevitably be shorter than the path travelled by information coming from the other side of the head. So how does the brain overcome this mis-match?
And here is where having paid attention at school during the “two trains travelling at the same speed leave two different stations blah blah blah” math problem finally pays off. When a sound comes from the front, the information arrives to each of the ears at the same time. The information also arrives to the first station in the brain (nucleus magnocellularis) at the same time. But time comparison neurons need information from both ears, and the path that the information needs to travel from the right side to the time comparison neurons in nucleus laminaris on the left side (red arrow in figure 1) is longer than the path from the same side (blue arrow in figure 1).
However, when you look into an actual brain, things are not so straight-forward (sorry for the pun). The axons from nucleus magnocellularis that go to the time comparison neurons on the same side of the brain take a rather roundabout route (as in figure 2). And for long we assumed that such roundabout way was enough to make signals from the left and right sides to arrive at about the same time.
Easy enough? Not quite
When Seidl, Rubel and Harris actually measured the length of the axons (red and blue) they found that there was no way that the information could arrive at about the same time and that the system could not work in the biological range. But this problem could be overcome (back to the old school problem) by having the two trains (action potentials rather) travel at different speeds. And this is something that neurons in the brain can relatively easily do in two ways: One is to change the girth or diameter of the axon. The other is to regulate how they are myelinated. Myelin forms a discontinuous insulating wrap around the axon, which is interrupted at what is called the Nodes of Ranvier. The closer the Nodes of Ranvier are, the slower the action potential travels down the axon.
What the group found was that both axon diameter and myelination pattern were different in the direct (blue) and crossed (red) axons. When they now calculated how long it would take for the action potential from both sides to reach the time comparison neurons in nucleus laminaris, adjusting speed for the differences in the two axons, they found that yup, that pretty much solved the problem.
Easy enough? Quite
Like the authors say:
The regulation of these axonal parameters within individual axons seems quite remarkable from a cell biological point of view, but it is not unprecedented.
But remarkable indeed, considering that this regulation needs to adjust to a very high degree of temporal precision. I have always used the train analogy when I lecture about sound localisation, and always assumed equal speed on both sides. Seidl, Rubel and Harris’ work means I will have to redo my slides to incorporate differences in speed. Hope my students don’t end up hating me!
Seidl, A., Rubel, E., & Harris, D. (2010). Mechanisms for Adjusting Interaural Time Differences to Achieve Binaural Coincidence Detection Journal of Neuroscience, 30 (1), 70-80 DOI: 10.1523/JNEUROSCI.3464-09.2010