Nowadays it is very easy to access a wide spectrum of information on various diseases via online media. While most people are aware that they shouldn’t believe everything they read online and have learned to look for sources like scientific papers, rating the quality of these papers can be somewhat difficult.
Clinical evidence proven by randomized, controlled trials as well as real-world studies is the gold standard for treatment decisions. Especially for conditions that are not well studied yet or for which only limited data on potential treatments exist, as for Long COVID, both patients and Health Care Professionals rely on ongoing research and recent publications. However, not all scientific studies hold up to the same standard and publishing processes can be confusing for people who are not familiar with the quality control mechanisms for scientific publications. Moreover, a scientific layout can be used to make non-validated information appear scientifically accurate. Such claims can thereby be quickly taken up by the media and broadcasted to a wide audience with no quality control in place.
Therefore, it is important for everybody to have a basic understanding of how scientific publications should be handled, how you can find out whether a study is trustworthy or not and how well-assessed results can be distinguished from bold claims that are not backed up by data. Here is a list of criteria that you can use to find out if a study is credible or not and which data you should look for in a publication on the results of a clinical trial.
Not all scientific data can be trusted equally. Sometimes it might be legitimate to “judge the book by its cover”.
To evaluate whether the research paper you are about to read meets some standard quality criteria, there are a few things, you can check even BEFORE reading the paper:
- Is the paper published in a peer-reviewed journal?
High impact scientific journals use a peer-reviewing process to ensure the quality of their publications. Before a paper gets published, it is submitted to several peer reviewers (experts in the respective field) who assess the quality of the manuscript. This process is either double blinded (the peer reviewer does not know who authored the paper and the author does not know who did the peer review) or at least single blinded (the author does not know who did the peer review). The experts may request further data, if they are not convinced by the results presented or changes to the manuscript if it is not clear enough. Only if all expert peer-reviewers agree that the presented content is worth publishing, the paper can proceed to publication. Journals which do not require a peer-review process are generally less trustworthy and can contain non-validated data.
- Is it a preprint?
Besides submitting to a peer-reviewed journal, it is possible to publish manuscripts on a preprint platform. This means that the presented data has not been reviewed by other experts. It is possible that a paper gets declined by renowned journals due to lack of quality but still exists as a preprint.
- How many authors are involved?
A clinical trial involves a lot of work, often several different institutions are involved. If a paper only has one or two authors it is most likely rather an opinion, a summary of recent data (literature review) or a summary and comparison of recent data (review and meta-analysis). The latter can give valuable information on how different treatments available on the market compare to each other.
- Where and when was the study conducted?
Is the study up to date? Standards for clinical trials are currently evolving, an older study might not hold up to current quality standards anymore. Origin, ethnicity, and environmental factors can have a strong impact on the efficacy of certain treatments. The characteristics of the study participants should be comparable to you/your patients to draw conclusions.
- Is the respective trial registered at clinicaltrials.gov or a national registry?
A trial has to fulfill certain quality criteria to register at a national or international registry.
- Is there any conflict of interest from the researchers?
How was the study financed and are the authors involved with anyone who could benefit financially from the presented study outcome? Clinical trials are very expensive and therefore rely on the financial support from pharmaceutical companies. To prevent any possible exploitation and ensure complete objectivity, the communication in pharmaceutical industry is highly regulated by national and international authorities.
How is the study structured and how does the study design reflect everyday clinical practice?
If the study you are looking at seems trustworthy to you based on the above-mentioned criteria, you can go on to evaluating the content presented to you. Here’s what to consider when analyzing the design of the study:
- How many patients were involved?
A study which only includes a hand full of people, does not allow conclusions for a broad patient audience. However, the number of people included is highly dependent on how many people are affected by the condition that is being researched.
- What kind of patients were involved?
To draw conclusions, the patient population has to be comparable to you/your patients.
- Were patients randomized?
If two or more treatments are compared in a study it is important to know if the treatments were assigned randomly. If not, there could be a selection bias.
- Was the study blinded?
When comparing two or more treatments it is relevant to know if these patients and investigators knew which treatment they will receive (open label) or not (single or double blinded).
- How long was the study duration?
The study period should be long enough to assess potential safety concerns and to ensure a benefit over time.
- For trials comparing different treatments: Is the chosen comparator suitable and are the patient characteristics in both groups comparable?
Especially for conditions where effective treatments exist, it might be necessary to compare a new treatment to the current standard of care rather than to placebo. The characteristics of patients in the compared groups must be considered. Ideally, they are similar, if not a statistical adjustment of the results based on the baseline characteristics might be necessary.
Does the study have a clear hypothesis and are the presented methods suitable to assess this hypothesis?
The methods used in a study are crucial to evaluate the presented results. If the methodology is unclear, biased, or flawed the obtained results cannot be trusted.
- Is the hypothesis clear?
The authors need to state what the expected outcome of the study was.
- Is the study methodology suitable to assess the hypothesis?
The results can only support the hypothesis if the outcomes are well chosen.
- Who defined the outcomes?
It is a good sign if the outcomes are defined by an independent expert board.
- Which outcomes were defined?
It should be defined what was measured and explained why the results will be able to support or debunk the hypothesis.
Are the results supporting the hypothesis?
The results should be presented objectively and the discussion should explain why these results are able to support the afore mentioned hypothesis.
- Do the results allow the conclusions that are made?
Sometimes conclusions are drawn from ambiguous data. The difference between causality and correlation needs to be clear.
- Are all results included in the analysis and the discussion?
If some results do not appear in the discussion, this is a hint, that the outcomes were not as expected. Moreover, if only a fraction of the patients enrolled in the study are analyzed, it is likely that some patients were censored as they didn’t meet the envisioned results or that many patients dropped out of the study (e.g., due to toxicities or lack of efficacy).
- Can the results be generalized to the clinical practice?
This depends a lot on the patient population. If the patients had to fulfill a lot of very specific criteria to meet the presented result, the treatment might not be applicable to clinical practice.
- Are the results objectively presented?
The results should be stated without any rating. If results are not presented objectively, there is a great potential for bias of the authors.
- Does the discussion state whether prespecified endpoints have been met?
At the beginning of a trial the hypothesis and expected outcome should be defined. Conclusively, the discussion of a scientific publication should include a statement on whether the expected outcomes were met, if the hypothesis is correct and conclusively, if the study is positive or not.
- Do the researchers disclose possible limitations of the study in the discussion?
If the discussion includes a section on the limitations of a study this is a sign that the researchers are aware of potential flaws and are objectively presenting the results, inviting the reader to evaluate the outcome.
Are the study results statistically significant?
As clinical trials usually deal with a great amount of individual patient data, a statistical analysis of the obtained results is indispensable. Statistics can be difficult to understand, however, there is a few simple things to check for:
- Which statistical methods were used?
For people who are not familiar with statistical methods it is still recommendable to check whether a statistical analysis was performed. Health Care Professionals should invest the time to understand standard statistical methods and the meaning of statistical significance.
- What do odds ratios and confidence intervals mean?
Odds ratios are often used for safety outcomes and show how different treatments compare to each other regarding side effects. An odds ratio of 1 means that the occurrence of a certain side effect was the same in both groups. An odds ratio below 1 means the side effect occurred less frequent and an odds ratio above 1 means the side effect occurred more frequently in the assessed treatment group compared to the comparator (e.g., placebo). A confidence interval shows the interval that contains 95% of the results. If the confidence interval is very broad, the results varied a lot from patient to patient. Regarding efficacy this could mean, that in some patients the respective drug was very effective while in others no benefit was seen.
- Was statistical significance demonstrated?
It should be stated in the discussion if statistical significance was reached. Treatments with a very low risk of side effects, could still be considered in practice if statistical significance is not met but the researchers could show that the benefits most likely outweigh the risks and that no harm is done if the treatment would not be effective.
Altea Long COVID is committed to make ongoing research available for everyone to understand. We hope that this guide will help you to evaluate the content of scientific studies more easily and welcome you to discuss new publications with other patients and Health Care Professionals in our Community.
