Organic vs. Conventional Crops: Analyzing a Meta-Analysis

Consumers have long been aware of the relationship between nutrition and health. With the advent of environmental concerns, rates of organic farming have increased, offering consumers choices of conventional or organic products. As such, there have been efforts to compare the nutrition content of organic and conventionally grown crops.

A variety of studies have emerged, and in an effort to combine and better understand these studies, meta-analyses have been conducted. To date, three high-profile systematic reviews have been performed, with the most recent being published in July 2014. Titled "Higher antioxidant and lower cadmium concentrations and lower incidence of pesticide residues in organically grown crops: a systematic literature review and meta-analyses," the publication from Barański et al. has received attention from the media but also encountered a great deal of scientific scrutiny.

So what is the overall verdict of the study? Was the research conducted in a scientifically sound manner? With the International Food Information Council Study Evaluation Checklist, one can methodically evaluate the hypothesis, design, methods, and analyses to help critically assess the results and conclusions from any research project.

 

Q1. Do the title and abstract reflect the study?
                 Yes                  No. View Results Skeptically
A1. Yes. The title described the main findings, and the abstract elaborated on these key findings, and offered brief interpretations as to what the results may mean
 

Q2. Is the study useful, novel, and/or relevant to humans?

                 Yes                  No. View Results Skeptically

A2. Yes. The study compared the macronutrient, fiber, and antioxidant content between organic and conventional crops. In addition, the study examined levels of toxic metals and pesticides in organic and conventional crops. All of these endpoints have been shown to play an active role in human health.

 

Q3. Is the hypothesis clearly stated?

                  Yes                  No. View Results Skeptically

A3. No. The paper never outlined a specific hypothesis. However, since the study is a meta-analysis (i.e., a method to analyze data from multiple studies to draw conclusions from previous studies), this type of research is not usually hypothesis-driven. 

 

Q4. Was the study methodology described in detail?

                          Yes                          

No.

Do the authors cite a paper for the methods?
If no, view results skeptically.

A4. Yes. The study methodology was carefully described and referenced. In addition, the authors detailed their rationale for choosing their methods to analyze the data. 

 

Q5. Are the methods valid, accurate and reliable?

                 Yes                  No. View Results Skeptically

A5. Somewhat. The methods utilized by Barański et al.  may skew the results since the  “overall reliability” of the outcome parameters (n=35) were either low (n=14), moderate (n=19), or good (n=3). Criteria used to establish “overall reliability”  should include detailed study design, proper organic practice, and appropriate statistical analyses. Without these criteria, the “reliability” of the study decreases. Inclusion of less reliable studies has the potential to skew the results, create confusing conclusions, and weaken the significance of the findings. To circumvent this issue, the authors performed  “sensitivity analyses” and found that the less reliable work did not skew the data, yet  it still is unclear why these less reliable studies were included. 

 

Q6. Does the analysis of the results make sense?

                 Yes                  No. View Results Skeptically

A6. No. The authors’ analysis of pesticide levels in organic and conventional crops was flawed. The authors only targeted ten  studies (of which nine had already been used in previously published meta-analysis) that are mainly specific to conventional farming and did not examine residues pertinent to organic agriculture. This indicates that the authors introduced a level of bias by skewing the analysis toward conventional related pesticides. 

 

Q7. Are the conclusions supported by the data?

                 Yes                  No. View Results Skeptically

A7. No. The authors’ conclusion of pesticide and toxic metals in conventional crops was incorrect. While the authors found that pesticide residues from conventional crops were four times higher than organic crops, the authors concluded that “ the data available did not allow for scientifically robust comparisons of the concentrations of pesticides”. However, this was a major conclusion of the paper and may mislead readers into thinking that occurrence and concentrations of pesticide residues were measured, even though it was not possible in this analysis. In addition, another main conclusion of the paper was that conventional crops had higher levels of cadmium, a toxic metal. What the authors do not mention is that daily exposure to cadmium through food is  0.0004mg/kg, or roughly ten times lower than the level of cadmium needed to cause kidney damage. Again, the authors fail to mention  concentrations, so implying that eating conventional foods exposes an individual to a toxic metal such as cadmium lacks relevant context. Lastly, while the authors identified  significantly higher antioxidant levels in organic crops, there is no mention if these differences would be nutritionally relevant. This may cause the reader to think that consuming organic crops will have health benefits related to increased antioxidant consumption. 

 

Q8. Are there conflicts of interest?
(personal, academic, financial, conflicts of commitment)

      Yes. Compare findings to the totality of evidence.                              No                            

A8. No. However, the study was partially funded by Sheepdrove Trust, a British charity that supports organic farming research. This does not discredit the findings but it is something to be aware of when reviewing the data and comparing the conclusions the totality of the science. 

 

Q9. Does the study fit into the totality of evidence?

Examine individual study findings versus the totality of evidence on the topic.

A9. Somewhat. There are only handful of studies published that compared the nutritional content between organic and conventional crops in a systematic manner. However, these previously published reviews have shown that there are minimal nutritional differences between organic and conventional crops. (1, 2) As such, this type of research is still in its infancy, so future controlled and well-designed studies should be performed to examine this question.

So how did the article from Barański rate? As you can tell, the study had some flaws regarding the methods, analysis, and conclusions. What was most troubling was that the main conclusion was that organic crops offer more nutritional value and less exposure to pesticides and toxic metals. However, as outlined above, these conclusions may not be accurate and can mislead the reader into thinking that organic crops are the only option for consumption.

Because organic food products can be substantially higher in price than conventional food products, this may be disheartening for consumers who cannot afford the organic option. Scientific experts support this conclusion as well, with Carl K Winter, PhD, food toxicologist at the University of California at Davis stating in a Huffington Post article that “if you limit the amount of fruits and vegetables you eat because you're concerned about pesticides, you're doing yourself much more harm than good."

According to the CDC, the majority of Americans do not meet the daily requirement for fruit and vegetable intake, so it is imperative that consumers strive to meet these requirements regardless of whether the crops are organic or conventionally farmed. Speaking to The Telegraph, Professor Richard Mithen, PhD, lead researcher at the Institute of Food Research, agreed, stating that “to improve public health we need to encourage people to eat more fruit and vegetables, regardless of how they are produced.”

In sum, the most important takeaway from this research is that a diet rich in fruits and vegetables, either organic or conventional, promotes health and reduces the risk of a variety of chronic diseases. 

Megan Meyer is a Microbiology and Immunology Doctoral Student, University of North Carolina at Chapel Hill, and an intern at International Food Information Council and Foundation.

References

1.         Baranski, M. et al. Higher antioxidant and lower cadmium concentrations and lower incidence of pesticide residues in organically grown crops: a systematic literature review and meta-analyses. The British journal of nutrition, 1-18 (2014).

2.         Dangour, A.D. et al. Nutritional quality of organic foods: a systematic review. Am J Clin Nutr 90, 680-685 (2009).

3.         Smith-Spangler, C. et al. Are organic foods safer or healthier than conventional alternatives?: a systematic review. Annals of internal medicine 157, 348-366 (2012).