By Martin Michaelis and Mark Wass. A lack of data replicability has been blamed for a lack of progress in many disciplines, including the development of anti-cancer therapies. This lack of data quality has been attributed by many commentators to poor research practices and resulted in calls for better research standardisation. However, our most recent study shows a high level of variability in a dataset that was produced in a world-leading research environment under highly standardised conditions. Hence, our findings suggest that the inherent variability in biological systems is higher than suspected, challenge the usefulness of simple data standardisation, and warrant more research into this subject.
There is an intense debate in the sciences and many other academic disciplines about a so-called ‘reproducibility crisis’ or ‘replication crisis’. This is fuelled by reports from researchers, who fail to obtain the same results, when repeating experiments described and published by other researchers in peer-reviewed scientific journals.
The consequences of a low level of data replicability are considered by many as highly damaging to the scientific progress and the translation of research finding into applications. For example, limited data replicability has been blamed for the high failure rates of new drugs during clinical development, i.e. drug testing in human patients. More than 90% of drug candidates that enter clinical trials in humans fail at this latest phase of drug testing.
It sounds surprising, but despite the great importance of this topic and the attention that it generates within and beyond the academic world, there is actually very little research and data on this subject. For example, there is no established understanding of what level of data replicability is technically feasible in a certain model system.
Many comments are based on perceptions and opinions rather than on evidence, and commentators propose that a lack of replicability indicates a lack of proper research standards. However, there is so far very limited evidence suggesting that the quality and robustness of data can be substantially improved by adherence to more stringent standards. In particular, evidence on the technically feasible level of replicability of scientific experiments, a key prerequisite for addressing data replicability in an informed and meaningful way, is extremely scarce.
We have now addressed this question by analysing the variability in the largest existing dataset (at least as far as we know, if there is a larger one, we will be happy to learn about this) containing results of experiments that have been replicated multiple times, often over decades. These data have been thankfully provided by the NCI60 project, which is run by the US National Cancer Institute (NCI). The NCI60 project has repeatedly tested many thousands of chemical compounds for activity against a panel of 60 cancer cell lines since the 1980s, following the highest standards and making all raw data available for analysis.
The variability among results of the same experiment (testing a given drug in a given cell line) was much higher than expected, reaching up to a more than a 10 billion-fold difference. For all compounds that were tested at least 100 times in the same cell line, there was an at least five-fold difference between the highest and the lowest value. This is highly significant, given that even a two-fold dose increase is not possible when anti-cancer drugs are used to treat patients. Traditional cytotoxic chemotherapeutics are administered at the maximum tolerated dose, which means any dose increase would result in intolerable toxicity. So-called targeted drugs, which are more specific towards cancer cells, are administered at the maximum biological dose, at which the target is completely inhibited and at which a further dose increase will not further increase specific anti-cancer effects.
Since the NCI60 project is a world-leading research environment that follows the highest possible research standards, the high variability in the NCI60 data seems to be much more driven by the intrinsic variability of biological systems than previously anticipated. This also means that simple research standardisation will not solve the issues associated with data replication.
We hope that these surprising results act as an eye-opener. When we started this work, we would have never expected such a high level of variability, which shows that we ourselves had unrealistic expectations of the technically feasible level of replicability in biological systems. Hopefully, our findings will inform and inspire the research needed to improve the robustness and meaningfulness of research data and, in turn, also the development of therapies for cancer and other diseases.
More information on the study:
Title: Large inherent variability in data derived from highly standardised cell culture experiments
Authors: Mark Wass, Martin Michaelis – University of Kent; Ian Reddin, Tim Fenton – University of Southampton.
Journal: Pharmacological Research (Pharmacol Res. 2023 Jan 18;188:106671. doi: 10.1016/j.phrs.2023.106671. PMID: 36681368).
Links:
https://doi.org/10.1016/j.phrs.2023.106671
https://www.sciencedirect.com/science/article/pii/S1043661823000270