Originally posted September 23, 2013
What makes a scientific publication good?
As scientists, are we capable of discerning good papers from bad papers?
These questions were the focus of a recent conversation I had on twitter with Stuart Cantrill, editor at Nature Chemistry, and bloggers extraordinaire, Chemjobber and See Arr Oh. This conversation started with the observation that a recent paper, containing what might be evidence of scientific misconduct, was generating a lot of buzz on-line. Chemists were rightly upset about this alleged impropriety. But, the incident highlighted a different aspect of scientific publishing to me. Precisely:
We, as a scientific community, generate many thousands of research articles a year. In my ideal world, research publications should be an ongoing conversation to better understand our world. Most of that conversation does occur in the literature. For a healthy discipline, shouldn’t that conversation be happening outside of the literature as well? This dialogue certainly happens at conferences and within research groups and in general-interest articles. Shouldn’t these conversations also be taking place in the on-line world as well? Between blogs and social media, there are some powerful methods for enabling real-time communication with scientists all over the world! Shouldn’t some topics be so vital and interesting to our field that we want to discuss them immediately? Even if they aren’t within our chosen sub-discipline? Shouldn’t we collectively be talking about some research as much as we discuss research misconduct?
With these topics on my mind, I asked Stu, (paraphrasing) “As an editor, can you recognize a paper that should garner broader discussion, and how do you effectively share *good* papers?” We went on to have a broader conversation about how any of us (seriously, we’re all experts in chemistry here) recognize a “worthy” paper. Finally, in a fit of genius, Stu asked, “If we were all given the same 10year old copy of JACS1, would we pick out the same ‘top 5′ papers?” Stu’s hypothesis for this question, which the rest of us agreed with, was: I doubt it!
So we set forth to find out! We sent out emails to a broad range of chemists (grad students, professors, editors, journalists, professionals) asking if they’d like to be involved in our experiment. Could they (we) choose the best papers from a 10 year old issue of JACS?
Best or good or important are judgements that are entirely subjective! So we asked four very specific questions:
1) Which three papers in the issue do YOU think are the most ‘significant’ (your own definition of ‘significant’ is what is important here)?
2) Without looking up the numbers, which three papers do you think will have been cited the most to-date?
3) Which three papers would you most want to point out to other chemists?
4) Which three papers would you want to shout about from the rooftops (i.e., tell anybody about, not just chemists)?
None of these questions really get at what it means to be important. They are just ways to describe how we perceive importance. Even question number 2, which evokes an easily retrieved metric, doesn’t necessarily prove a paper’s “worth”. But, are we (scientists! chemists!) even able to figure out which articles would get cited the most?
Our esteemed panel of experts have weighed in with their thoughts. How well would you do in the JACS Challenge? We’ll be back in one week with our respondents answers along with an analysis of their answers and thoughts.
But, first, we’d love to hear from you! How would you answer these four questions?! We have set up a survey on Survey Monkey in order to compile more answers. We plan on using your anonymous responses to try to glean some generalized trends about the way chemists view a study’s “importance” for a publication that we are preparing. We’d love to hear your thoughts on how you tried to analyze these papers here in the comments of this post!
Are you chemist-enough? You’ve got the questions! You’ve got the issue of JACS! Put your expertise on the line!!
We’re looking forward to hearing from you!
Survey link: HERE
– See Arr Oh, Chemjobber, Stu, and Matt
1This could have just as easily involved an issue of Angewandte Chemie or other journal. We chose JACS because it was the first journal that popped up (in considering journals that is a standard of quality and broad applicability among all chemists) and also because “The JACS Challenge” had such a nice ring to it!