Studying the studies: A flood of fraudulent scientific papers puts industry claims at risk

This article was originally published in Nutrition Business Journal’s Ingredient Science and Innovation Issue.

The first time it happened to me was about a year ago. Popping into PubMed to double-check a citation, I saw beside the title one word: Retracted. Huh? I couldn’t find any more information explaining why, but this paper I had been citing I could no longer cite. Fortunately, it wasn’t especially consequential, affecting only a minor subpoint of messaging that was fairly easy to alter. But then, a few months later, it happened again. I’d seen some articles about fraud, but now I needed to really look into this.

I didn’t have to dig that hard. Early this past spring, the wave of science fraud reached tidal strength, and all those quiet stories suddenly became front-page news. John Wiley & Sons, a global publication house with revenues north of $2 billion, announced on May 19, 2024, that it was closing 19 journals it acquired as part of its 2021 purchase of Egypt-based Hindawi publishing company, citing large-scale research fraud. The Wall Street Journal picked up the story, reporting that Wiley had retracted over 11,300 papers in the last two years (compare that to a global retraction rate of 40 papers per year in 2000 and 1,000 per year in 2013, as reported by The Guardian). And observers think this could be just the tip of the iceberg.

On top of that, in July we had the particularly irksome, if not finally vindicating, retraction by BMC Medicine of the 2013 DNA barcoding paper by Steven Newmaster, PhD, whose work was questioned at the time but still managed to create massive problems for the supplement industry when the New York attorney general used his findings to support a lawsuit against major retailers, charging that they sold products that did not contain the herbs proclaimed on the labels.

By the beginning of this summer, New York Magazine, of all places, ran a piece titled, “Why Science Fraud is Suddenly Everywhere.” 

The culprits

A primary source of fraudulent science is the individual unethical author. There are more of these than I would have thought, and they are being exposed as interested parties begin to look harder. A handful have been at predominant universities like Harvard and Stanford, but the list includes establishments around the world. The “publish or perish” paradigm has morphed into an ideology of publishing just for the sake of padding one’s curriculum vitae. Couple that with journals that make money from the publishing, and you have an environment ripe for dishonesty.

Even more insidious, corrupt operations termed “paper mills” have burst on the scene in the last 15 years. For a fee of between $1,000–$2,000, academics can buy a fully written, spurious scientific paper and get it published in a reputable journal with their name as author. These mills are staffed by young scientists who can’t find other jobs in the field and are concentrated specifically in a few countries: Egypt, China, India, Japan, Italy and, most predominantly, Iran. The outfits are cunning, posing as guest editors for special issues that often have more lax review, or cutthroat, submitting to many journals at once to maximize chances of publication (generally a highly unacceptable practice). In some countries, their activity is legal and out in the open. 

Perhaps most ominous, both of these culprits are about to cash in a lottery ticket in the form of AI.  While AI-generated writing is still in the early stages and often bears tell-tale signs of its computer origins, it’s easy to envision a point in the near future when one can ask AI to generate a full fabricated study report that will be hard to distinguish from the real thing.

Fraud sleuths

While the sheer number of fraudulent papers has, in fact, multiplied, part of their recent visibility reflects the efforts of concerned volunteer academics who comb the literature in their fields for papers with abnormalities and report them to the journal of publication. In addition, there is a growing (albeit still halting) action on the part of the publishers to follow through with retractions and “Expressions of Concern,” the latter of which are issued when pivotal questions about a paper go unresolved.

One champion of scientific integrity comes from our own industry: Alan Gaby, MD, author of the foundational textbook Nutritional Medicine, former faculty member at Bastyr University and contributing editor to multiple alternative medicine journals. In the course of his writing career, he has reviewed, by his own estimation, over 50,000 journal articles. About 20 years ago, he started noticing inconsistencies. In 2022, he published the article, “Is There an Epidemic of Research Fraud in Natural Medicine?” in the journal Integrative Medicine, in which he enumerated the types of inconsistencies he had been seeing. It is an instructive read; see the sidebar for a taste. Also from our industry, Erik Goldman, editor-in-chief of Holistic Primary Care, recently published an in-depth look into the anatomy of a fraudulent study and the conditions that spawn it in his article, “Fraudulent Research Floods Nutrition Field ­Corroding Credibility.”

Another important crusader is Retraction Watch, a news source founded in 2010 and run by two volunteers: Ivan Oransky, journalism professor at New York University, and Adam Marcus, editorial director at Medscape. They put out a weekly summary of retractions and associated stories; have a database of retracted papers searchable by field; and track other areas of interest, including papers and peer reviews with evidence of ChatGPT writing, retracted COVID papers (438 at last count, including several papers testing ivermectin) and a list of the top 30 authors with the most retractions (No. 1 goes to Dr. Joachim Boldt, with 220). 

Implications for our industry

Some of the Hindawi journals closed by Wiley will be familiar to science-types in our industry: Evidence-Based Complementary and Alternative Medicine, Advances in Preventive Medicine, International Journal of Chronic Diseases, to name a few. These days, my antennae go up whenever I come across an article published in one of these journals. Sometimes, when I’m deep into reviewing a study design or statistical analysis that doesn’t quite come together, I’m interrupted by a panicked thought: Wait! What if this whole study is fake!

The ramifications of this new reality are deep. There are considerations to be made inside each company and by us as an industry as we move forward. At the very least, there is another layer of scrutiny that has to be developed when reviewing journal articles as proof-of-concept or to substantiate claims. Then there is the need to look backward and assess prior analyses. 

How the industry communicates this to stakeholders in concentric circles moving outward must also be thought through. Will it be long before sellers are asked, “Can you assure me that these studies are real?” Branded ingredients purveyors who have conducted their own research may have a new value-add to tout; companies using borrowed science may have some extra homework to do.

At the most global level, this could be an opportunity for our industry to step forward and say, “We’re aware of this. We take scientific integrity seriously, and this is how we’re on top of it,” rather than pretending it isn’t happening and letting it become more fodder for critics.

How all this activity takes shape is in the early stages of evolution. Some pieces are best done independently and some collectively across the industry. I encourage executives and industry leaders to start a dialog in their inner and wider circles. I envision pulling together invested parties to generate best practices and resources that can be shared. 

Lastly, this: science fraud brings credibility into question across many industries. We are not the only ones grappling with its repercussions, and there are learnings to be gathered from all over the world.  Tools are emerging, such as those housed by the STM Integrity Hub, called The Papermill Alarm and the Problematic Paper Screener, as well as frameworks for assessing research integrity, such as RIGID (Research Integrity in Guidelines and evIDence synthesis) from medical researchers at Monash University in Australia and the Research Integrity Assessment (RIA), put together by a group of researchers in Germany.

It’s still early enough, and we don’t want to let science fraud get ahead of us. Let’s get a conversation going before the next wave of headlines about science fraud focuses on claims in the supplement industry.

For more insights into the ever-evolving supplement industry, subscribe to Nutrition Business Journal.

Risa Schulman helps nutraceutical companies evaluate scientific studies and strategies at Tap~Root. She will be speaking on an Oct. 30 panel at SupplySide West titled “The real news about fake science: How a flood of fraudulent research is threatening ingredient innovation in the AI age.”