New software makes science more replicable

Every field in science and engineering employs highly specialized equipment: surface area analyzers, nanosizers, recording membrane osmometers. This equipment is often incredibly specific, designed for just a single function, but it’s essential for performing accurate, replicable research. And without the ability to replicate the research of other scientists, the validity of science itself falls apart.

But if everyone agrees on the need for specialized equipment in research, shouldn’t the same mindset apply to every aspect of the scientific process? Why is it, then, that when writing scientific reports, most researchers still use the same program that came with their first computer?

“Most people write their papers in Word,” says John Kitchin, professor of Chemical Engineering, “which is not a good scientific publishing environment. For example, Word has no way to log your data as you go or record exactly how it was analyzed. It isn’t practical to see where the data in a figure came from. So when researchers are making their supporting information file—which contains much of the data and analysis that explains just how the study was done—they’re forced to reconstruct what they think they did from memory.”

Kitchin, whose research focuses mainly on software development for modeling materials and solving problems in engineering, saw this unfortunate trend in his own scientific report writing, and decided to do something about it. That’s why he created SCIMAX: a software specifically designed for the purpose of writing scientific reports.

SCIMAX is an open-source, fully integrated program that combines the function of Word—namely, the narrative portion of the report—with everything else that one might need when writing a research paper. The data, the code, the bibliography—there’s a place in the program for everything, already perfectly formatted and ready for input. And this is key, because doing these things in other programs often leads to serious problems.

“Recently,” says Kitchin, “I saw a paper where the author had copied their MATLAB code into Word. Even that is better than what many people do. But because they copied it and didn't run it from Word, they forgot to copy two of the files that you need to actually run it. So suddenly, the code in the paper—the same code they claim is what helped them run their study—flat out doesn’t work.”

It may sound like semantics—whether you use one program or another to write your reports—but according to Kitchin it actually has wide-spread implications for the integrity of science.

If our research isn’t replicable, I can tell you anything is true, and you just have to take my word for it.

John Kitchin, Professor, Chemical Engineering

“So when I write a paper,” says Kitchin, “I can tell you in the paper that I did something, and I can tell you that it worked, but without the accurate dataset, and in many cases the actual code used to analyze it, there’s no way for you to take the paper and replicate my work. And that’s a fundamental failing on our part. Because if our research isn’t replicable, it can’t be considered true scientific research at all. If our research isn’t replicable, I can tell you anything is true, and you just have to take my word for it.”

According to Kitchin, this is the true goal of SCIMAX—not just to make report writing easier, but to increase the level of integrity in scientific research by increasing replicability. The more research teams that adopt the software, the more accurate their reports will be, and the more others can build on and reuse our scientific research to solve the major problems that affect our world.