Meta-Analysis

In the pyramid of evidence-based research, the meta analysis holds the top position. It is the strongest, most powerful form of causal evidence. If a meta-analysis concludes that something does not work, a single clinical trial cannot override those findings. How does a single form of research wield such power? It has everything to do with the methodology. Let’s start with the basics: 

What does Meta-Analysis mean? 

One of the most pervasive misconceptions about a meta-analysis is that the term meta simply means large, therefore a meta-analysis is an incredibly large study. 60 participants? Meh, average. 2,000? Definitely meta. The problem is that’s not what the term meta means, nor does size have anything to do with a meta-analysis.

 The term meta refers to something that is more comprehensive. It’s highly organized; more robust. A meta-analysis, therefore, is a study that is more robust or comprehensive. Researchers define it as a study of studies. The more formal definition is: 

a quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance

In plain language, a meta-analysis is a specific type of research methodology in which scientists study entire past studies rather than the individuals within those studies. In a clinical trial, we enroll participants, gather data, and then analyze the data. Each datapoint represents an individual person. In a meta-analysis, we enroll entire studies, gather data from those studies, then analyze that data. Each datapoint represents a whole study. Which means that a meta-analysis can’t be conducted until several clinical trails (of any size) have been completed. Without completed clinical trials, there is nothing to enroll into a meta-analysis. But before we dive into how these studies are conducted and how to interpret them, we have to first look at why we need meta-analyses. 

Why conduct a meta-analysis?

We’ve all been there. You’re looking for evidence on an herb or an oil. You read a study on PubMed that finds the herb is effective. But wait. Then you read one that says it is not effective. Now what? Do we tally up the total studies and see which side has more? If there are 3 that show it works and 4 that say it doesn’t, do we conclude it doesn’t? What do we do if another study is published saying it works? Now it’s a tie. Do we just go with the biggest study? Whatever the biggest study found is THE answer? Or do we try it statistically? The smallest p-value is the final authority? 

Conflicting findings are the fodder for heated debates, both professionally and personally. And as the total amount of scientific evidence began to grow in the late 1900s, scientists started having a serious problem with making evidence-based decisions that truly reflected the entire body of evidence and not just their favorite studies. That’s when the meta-analytic methodology was developed. This is a specialized way of compiling all of the studies, regardless of what they found, and examining the entirety of the science on a topic. Hence the “comprehensive” definition of meta. 

With a meta-analysis, researchers can combine all of these studies into a single, robust analysis that takes into consideration things such as study size and duration. This cuts through the noise of conflicting results in individual clinical trials, delivering a clear and precise answer to important research questions. Hence the elevated status of a meta-analysis in the pyramid of evidence. 

How to conduct a meta-analysis

Conducting a meta-analysis correctly is an advanced statistical process. The general overview, however, is relatively simple to grasp. The first step is to identify a research question. This needs to be specific. It can’t be vague such as “does elderberry work?” It needs to be precise: “to what extent does elderberry reduce duration of upper respiratory symptoms?” (For the answer to that one, click here.) 

Next, it’s time to set some parameters regarding what studies can enroll in this study of studies. In the case of the elderberry analysis, are we going to also include studies that used elderberry in combination with other herbs such as echinacea? Will we set time limits for when these studies were conducted? Are we focusing on a specific demographic population? Just like any study, a meta-analysis has inclusion criteria. We have to let the research question guide the development of these criteria. 

Now it’s time to enroll clinical studies. This seems rather straightforward: head to PubMed and enter some search terms. But to avoid getting biased results, the search has to be thorough and exhaustive. It typically takes 2-5 researchers several months to reach the point where they can conclusively say they have exhausted all avenues. At this point, they’ve typically reviewed several thousand scientific publications to see if they can identify any studies that meet the inclusion criteria. We typically search for both published and unpublished work so the search also typically involves emailing other researchers in the field to inquire about any projects in the works or recently completed. The thousands of emails and manuscripts are evaluated and narrowed down to a total of around 5-20 studies. 

The next steps are the boring part for readers. All the data available in these studies will have to be extracted, checked for accuracy, and coded. To ensure accuracy, this is typically done by two separate people who then compare results. Effect sizes–the size of the results in each study–are calculated and analyzed. Finally, we arrive at a comprehensive result that answers a broad question. We get an effect size that is more accurate than what is found in a single study, and we are able to answer these complex research questions. 

Next the researchers may want to look at moderator variables. These are factors such as age or sex that may moderate the effects of the intervention. For example, are probiotics more effective at treating colic in breastfed babies than in formula fed babies? Finding these answers can unlock new scientific discoveries that improve the way we use natural products for health and well-being. If we know men and women respond to an intervention differently, we can modify protocols accordingly and get better results. 

Interpreting a meta-analysis

So the next time you find conflicting evidence about a topic and you turn to a meta-analysis to settle the score, here are some things you’ll need to know. 

• Clinical trials are highly specific when it comes to inclusion criteria and dosing and other factors. By definition, a meta-analysis combines apples and oranges–clinical trials with a variety of ages and doses–into a single study. This is part of the magic of a meta-analysis; this combination approach answers broad research questions that single studies can’t answer. Conversely, there will be times when something only works in a very small subset of the general public and meta-analyses that combine highly diverse factors will not reflect these smaller variations. This is where moderator analyses come into play–so don’t ignore those! 
• A clinical trial is only as good as its data. A meta-analysis is only as good as the studies it includes. Part of the analytical process includes evaluating each of the studies for quality (what scientists call bias). If the researcher points out that the studies suffer from many forms of bias–i.e. poor quality–interpret the results accordingly. Unfortunately, the natural health field has a lot of junk science out there, so meta-analyses require special expertise to account for these failings. 
• A meta-analysis can only include studies that exist at the time of analysis. If you see an analysis from 2020 and a trial from 2021, the trial is not going to be reflected in that analysis. For fields (like many natural health fields) with very little research, a meta-analysis from 3-5 years ago may not be the most current source of information. 

Finding meta-analyses

Our team routinely conducts meta-analyses prior to designing landmark clinical trials. By conducting a meta-analysis, we can discover factors that are critical for trial development and we can boost the quality of our work. You can find results from our meta-analyses in scientific manuscripts by searching PubMed or in lay terminology throughout our website. Some of our most popular findings include Elderberry for Upper Respiratory Symptoms and Bergamot for Anxiety.