Of Mice and Media: A Credulous Response to an Iffy Sweetener Study
- A study Sept. 17 in Nature showing a possible link between artificial sweeteners and conditions that can lead to weight gain and diabetes raises interesting questions, but is flawed in many ways.
- Several better-designed studies show positive effects on weight management when people incorporate artificial sweeteners into their diets.
- The media's extensive coverage of the story fits a documented pattern of paying the most attention to the least-reliable studies.
Here we go again. Researchers with a study of dubious relevance and so-so design said "jump," and many media said "how high?"
Now, please bear with me here. This is a rather long post, but it's an important one.
The scientific brouhaha du jour involves a study published Sept. 17 in the journal Nature that purports to show a link between consumption of non-caloric artificial sweeteners (NAS) and glucose intolerance, a condition that often leads to type 2 diabetes. If you'd rather not read everything that follows, the bottom line is that the study is probably wrong, and is directly contradicted by many studies of far better design and credibility.
The researchers, led by Dr. Eran Elinav of the Weizmann Institute of Science in Israel, divided 20 mice into five groups: One was given plain water, one was given sugar water, and three were given water sweetened by either saccharin, aspartame, or surcalose. If you're keeping score, that's a grand total of four mice in each group from which to discern significance.
While the results indicated higher blood glucose levels in the latter three groups, the differences among the three NAS themselves were quite substantial. The researchers never explained why this might be.
I'll come back to the mice in a bit.
The researchers then looked at data from 381 people and found a correlation between higher blood sugar levels and NAS consumption. There are a number of problems here too.
First, consumption patterns were self-reported, which limits the reliability of the data. This is what's known as a "cohort" study, a type whose findings are less reliable than randomized control trials, the gold standard of scientific research. Also, the study corrected only for body mass index, but for no other lifestyle factors such as quality of overall diet, smoking, alcohol consumption, or sedentariness. As you will find out below, when you adjust for these "confounding variables," the supposed causation disappears like a wisp of smoke in a hurricane.
The researchers must have known that too, because they then embarked upon an actual human trial in hopes of finding a causal relationship. Trials involve "interventions," or directing test subjects to follow a predefined plan. (For more about evaluating scientific studies, check out our recent brochure here.) In this case, the subjects, who didn't normally consume NAS, were given controlled but high doses of saccharin over the course of (just) one week, with indications that higher blood glucose levels occurred.
Did I mention that the human trial involved seven people? Seven. And that three of them, nearly half, showed no appreciable changes in blood sugar? Blockbuster stuff, right?
The authors posit a correlation between NAS consumption and changes in the beneficial bacteria in our gut, which they further believe cause glucose intolerance; yet those three aforementioned human subjects also showed no such changes in gut flora. Elinav et al didn't offer an explanation why NAS could cause those microbial changes in the first place, why only some people might be affected, and why the levels of some types of bacteria went up, while others went down.
One expert says the study as a whole is undercut by the lack of involvement by a chemist, pointing out that grouping chemically disparate substances that have only one thing in common—in this case, sweetness—and trying to draw meaningful inferences makes as much sense as testing a group of food ingredients that all happen to be yellow.
Elinav said the study “calls for reassessment of today’s massive, unsupervised consumption of these substances.” This isn't science; it's editorializing. First of all, other credible experts have stated that the findings don't provide "sufficient evidence" of harmful effects. And second, while "massive" is a subjective term, "unsupervised" is simply false. Low-calorie sweeteners are "thoroughly tested and carefully regulated by U.S. and international regulatory authorities, as well as scientific organizations, to ensure the safety of foods, beverages and other products that contain them."
The study concludes by observing that the "increase in [artificial sweetener] consumption coincides with the dramatic increase in the obesity and diabetes epidemics."
"Coincides" is an interesting choice of verbs, because the noun form of that word is "coincidence." If that's what you base your diet on, then you'd better stop eating cheese, because there's an even closer correlation between le fromage and the risk of strangling to death in your bed sheets.
So what does the broader body of science have to say about the likelihood that NAS consumption correlates with obesity and other precursors to diabetes? A lot, actually, and not in a flattering way to the Elinav study.
Exhibit A: A study published in the New England Journal of Medicine (De Ruyter et al, 2012) followed 641 normal-weight children who were randomly assigned a daily sugar-free or sugar-sweetened beverage for a period of 18 months. The result: “[N]oncaloric beverages reduced weight gain and fat accumulation in normal-weight children.”
It’s hard to get more reliable results than a double-blind, placebo-controlled trial like this one.
Exhibit B: A 2012 study in the American Journal of Clinical Nutrition (Tate et al) divided 314 overweight and obese adults into three groups. One group substituted water for caloric beverages, one substituted diet beverages, and one made no substitutions. All three groups attended monthly sessions where they received weight-loss information.
All three groups lost weight after six months. But which group lost the most? The one that drank diet beverages.
Exhibit C: In a clinical trial (Maersk et al, 2011) looking into risks of metabolic syndrome (a potential precursor of diabetes), 47 overweight people without diabetes were randomly assigned to four groups. Each would drink one liter per day of one of four beverages: regular cola, semi-skim milk, aspartame-sweetened diet cola, or water.
The result: “No significant differences in the accumulation of VAT [visceral adipose tissue, or fat found around organs], liver fat, and muscle fat were found between the milk, diet cola, and water groups during the 6-mo intervention.” Interestingly, the water group experienced an increase in muscle fat, as opposed to the milk and diet cola groups, who lost muscle fat.
“The suggestions from observational studies that intake of diet cola can result in obesity, type 2 diabetes, and the metabolic syndrome are not supported by the findings of our current intervention,” the researchers concluded. “[T]he effect of diet cola on fatness, ectopic fat, and metabolic factors is mainly neutral and very similar to that of water.”
By the way, that study—along with others—also found that aspartame may help reduce blood pressure.
Exhibit D: A cohort study (Mozaffarian et al, 2011) in the New England Journal of Medicine followed a total of 120,877 men and women for periods up to 20 years, between 1986 and 2006. (Again, the Elinav cohort discussed above was just 381 people over the course of just one week.)
The researchers looked at changes in consumption of several different types of food and beverages to see which ones were related to weight change. Except these scientists controlled for a broad range of variables among the test subjects, including smoking, alcohol consumption, amount of sleep, TV watching, and—a big one—physical activity.
And VOILA! There’s that wisp of smoke. As it turns out, foods contributing to weight loss include vegetables, nuts, whole grains, fruits, and yogurt. But following right behind those are diet sodas. Increased consumption of many other foods (potato chips, sweets/desserts, sugar-sweetened beverages, etc.), on the other hand, correlated with weight gain.
But now back to the mice. Virtually none of the commentary around the study mentions how unreliable animal studies in general, and rodent studies in particular, are when attempting to gauge human outcomes. To wit:
“[J]ust how often do animal tests predict side effects in humans? Surprisingly, although it is central to the legitimacy of animal testing, only a dozen or so scholars over the past 30 years have explored this question. The results, such as they are, have been somewhat discouraging. One of the scientists, Ralph Heywood, stated in 1989 that ‘there is no reliable way of predicting what type of toxicity will develop in different species to the same compound.’ The concordance between man and animal toxicity tests, he said, assessing three decades of studies on the subject, was somewhere below 25 percent. "Toxicology," concluded Heywood, ‘is a science without a scientific underpinning.’”
"Mice take the blame for one of the most uncomfortable truths in translational research. Even after animal studies suggest that a treatment will be safe and effective, more than 80% of potential therapeutics fail when tested in people. Animal models of disease are frequently condemned as poor predictors of whether an experimental drug can become an effective treatment."
Do you want to know where that final article was published? The same place the Elinav study appeared: the journal Nature. (Physician, heal thyself!)
Certain steps can be taken to improve the quality of animal-study results, such as randomization—which was indeed the case with the Elinav study. But by the researchers’ own admission, the study wasn’t a “blinded” design, which introduces the potential for experimenter biases and skewed outcomes. Other confounders include using only animals of the same sex. All of the mice in the NAS study were male.
This isn't to say that animal trials as a whole are invalid or worthless, just that those substantial limitations have been all but ignored in the discussion. Wouldn't it have been just as accurate to say, "This study has a 75 to 80 percent chance of being wrong"?
This would all be a harmless exercise if several influential media hadn't jumped on it like a wolf on a steak, and without providing the necessary context. "Jumped on" is a bit of an understatement, really. The study so far has received 2.8 billion media impressions. Without delving into what that PR term of art really means, suffice it to say that it is equivalent to the potential readership of 176 different stories on the website of The New York Times.
Some reports have acknowledged limitations such as the study’s small sample sizes. But despite even the authors’ assurance that the study doesn’t warrant changes in NAS consumption, articles and TV segments have featured so-called medical experts who said the study caused them to stop consuming NAS, with tut-tutting TV anchors following suit. Not a word about potential negative consequences of such an action.
Then there's this especially enlightened take in the Huffington Post. The author concedes it's possible that people who consume higher amounts of low-calorie sweeteners are more likely to be overweight or obese for a reason that makes sense to most of us who didn't just fall off the turnip truck: Overweight people are more likely to be on diets, ergo, more likely to use low-calorie sweeteners.
"Duh," you might reply. A logical conclusion would be that being fat causes people to use low-calorie sweeteners, not that using low-calorie sweeteners cause us to be fat. Regardless, the author then draws false equivalence by implying that a small handful of dubious studies sits precisely balanced on a scale across from a mountain of evidence rebutting any ill effects. Apparently some people haven't heard of Occam's razor.
You know what else can produce glucose intolerance? Excessive consumption of any calories including caloric sweeteners—i.e., sugars. If the result of this study is more people consuming excessive amounts of sugars, it will have been a major disservice to public health.
But maybe the media's focus on one somewhat shaky study is the exception and not the rule, right? Wrong.
A report in January 2014 (Selvaraj et al, 2014) published in PLoS One asked the following question: “Media Coverage of Medical Journals: Do the Best Articles Make the News?” The answer is a resounding “no.”
In short, major media organizations (in this case, the five biggest U.S. newspapers) have a “systematic bias” that leads them to devote heavier coverage to less reliable observational/cohort studies, and far less attention to the more rigorous randomized control trials. “Man bites dog” sells papers.
The coverage in high-impact medical journals, of course, was just the opposite—but they don’t attract nearly as many eyeballs as the major traditional media.
Remember the Mozaffarian study that confirmed what we all know intuitively, i.e., if you eat too many desserts you’ll probably gain weight, whereas consumption of diet soda correlates with weight loss? It supports the solid advice most medical and health professionals have been giving for years. But how newsworthy was it?
Turns out, pretty newsworthy, but at roughly 820 million media impressions, far short of Elinav. But rather than focusing on the benefits of diet soda, almost every one of the roughly 400 stories dedicated most of their attention to heaping opprobrium on potato chips.
The IFIC Foundation’s annual Food and Health Survey has consistently found that far more people would like to hear advice about what to eat, rather than what not to eat. It’s no wonder why so many of us feel a bit adrift when it comes to maintaining a healthful diet.
Those other dog-bites-man studies referred to above trailed far behind in media coverage. The de Ruyter and Tate studies each coincidentally netted about 1.3 million media impressions, while the Maersk study didn’t even register a blip on the media radar, further underscoring the conclusion in the Selvaraj study, that the best studies aren’t the ones that make the most news.
There's little doubt that the Elinav et al research raises some interesting questions. And there’s little doubt that other researchers will see if they can indeed prove the causal relationship that’s implied.
But if (when) that doesn’t happen—and if you were to set an over-under of 2.8 billion media impressions on that particular story—then I’d bet my life savings on the “under.” Dog-bites-man news just doesn’t get ratings.
The epidemic of “single-study syndrome” that has broken out among reporters, and their tendency to be seduced by iffy science, are dark blotches on the media. What should have been treated more like a big nothing-burger with an extra helping of skepticism was instead served up with warmed-over hysteria, making journalists look hungry for readers and viewers, but starved of facts.
Imagine you actually had a resource that broke down the sensationalism about food, agriculture, and nutrition into real, science-based information.
- Join the tens of thousands of mythbusters out there fighting against bad information on food
- Get no-nonsense, easy-to-understand nutrition and safety insights
- Read Q&As with experts explaining the latest studies, debates, and news stories
- Be empowered to make your own decisions about your diet