Can Bias and Balance Coexist? [INFOGRAPHIC]

I love going to conferences. There's so much to learn and so many experts to learn from. Recently, I learned that that not everyone understands that not all research is created equal.

Take low-calorie sweeteners for example—much has been made of their role in health recently. Conflicting research findings have given media much to discuss and therefore given consumers much to consider. But is the evidence on low-calorie sweeteners really conflicting? Let’s take a look.

Research Design: Different Strokes for Different Folks

not-all-evidence-created-equalStudies are designed with different goals in mind. Some intend to increase understanding of underlying science thereby advancing the scientific frontier. Some evaluate product safety and benefits. Others inform science-based public health policies. Depending on the goal, data assessment will differ to ensure alignment with stated objectives and methodologies employed will be fit-for-purpose.  For instance, studies that test different outcomes (e.g., having study endpoints that measure blood pressure versus mortality can yield quite different results) may provide insights into important distinctions such as incidence versus severity. Studies also use different methods to capture data (e.g., 24 hour recall versus administered feedings) which, in this example, provides a window into understanding possible consumption patterns versus assessing product safety. Some published studies are observational in nature whereas others are randomized controlled trials (RCT) – the former introducing constraints on how the results may be interpreted and the kinds of conclusions that can be drawn.  Also, unfortunately, some journal articles get confused for bona fide research when they are in fact only a commentary. And, more recently, there’s the issue of publishing in “pay to play” journals versus traditional peer-reviewed varieties. When reviewing the body of literature on a subject, study design is of the utmost importance in determining the quality and strength of the study. The strongest evidence in the literature is found in RCTs and those using scientifically validated and standardized testing protocols.

Systematic reviews (SR) and meta-analyses (MA)

Newer forms of evidence evaluation have emerged in recent decades. Researchers can conduct evidence-based reviews by examining collectively a large number of qualified studies on a single topic to help compile and collectively analyze large bodies of evidence. In doing this, researchers get down to the most relevant conclusions to illuminate where the science nets out. Check more detail on SR and MA in the textbox below. 

Recently, I was on a panel where the latest published evidence (one meta-analysis containing observational and RCT data and one systematic review) on body weight and the use of low-calorie sweeteners was reviewed.

Not every study contained in the SR and MA showed the same effect on weight. Some studies showed more benefits and some showed less—such is the nature of these types of reviews. But the net results of these data showed a benefit.  In other words, the latest evidence-based reviews (one containing 24 studies and the other containing 20 studies) showed that low-calorie sweeteners can effectively reduce calorie consumption, thus have a positive effect on weight.

Bias and Balance Can Coexist

In presenting these results, to the casual observer it can appear biased toward showing an effect in favor of low-calorie sweeteners’ positive impact on body weight, especially if one is predisposed to believe otherwise. Case in point: a question to our panel from the audience asked why studies that showed negative results were not presented. Why wasn’t the other side to the story discussed?

Fair question, surely the research on low-calorie sweeteners isn’t all positive.

balance-in-science-scaleBut that’s precisely the point. The SR and MA presented didn’t just include studies that were all positive. Even with the inclusion of studies with neutral or negative results, the cumulative results of the SR and MA were positive. So to answer the question from the audience, both sides of the evidence were presented—the balance of evidence was contained within the SR and MA.

Perhaps it was expected that certain studies given attention in the media would be given the same attention in the reviews and on our panel. They weren’t. Perhaps the anticipated studies simply lacked sufficient quality to meet inclusion criteria. They did.

Unfortunately, not all research is created equal. And this is why SR and MA can be such a powerful tool—they allow experts to filter out the noise, distill the highest quality data, and serve up objective conclusions.

In developing official US Dietary Guidelines and global authoritative statements on low-calorie sweeteners, only the highest quality of evidence should be considered. More and more, these recommendations are informed by SR and MA. While this is a good thing, SR and MA are only as good as the quality of studies that are “put through the ringer” and are contingent on the question being asked. Like all research, MA and SR are biased. Fortunately, when done well, they just happen to be biased toward balance.


Additional references

Sign Up to Get Regular Servings of FACTS

Imagine you actually had a resource that broke down the sensationalism about food, agriculture, and nutrition into real, science-based information.

  • Join more than 50,000 mythbusters out there fighting against bad information on food
  • Get no-nonsense, easy-to-understand nutrition and safety insights
  • Read Q&As with experts explaining the latest studies, debates, and news stories
  • Be empowered to make your own decisions about your diet
5 + 3 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.