Skeptical of new health study? Great! (Part 2)

If you haven’t seen it already, Skeptical of that new study? Part 1 was published here.

In that post I go over the first four of the five (plus a bonus) steps of understanding a health study:

Step 1 – Understanding the results (what do they actually say/mean vs what is “read into” them).
Step 2 – What does my gut say? Do I tend to agree or disagree with the results? (Acknowledging and reserving judgment until you have the chance to understand and think about the new study).
Step 3 – What question did the study try to answer? (HUGELY important because the whole study design and critique are based on the foundational question being asked).
Step 4 – How good was the study design for answering that question? (Of all the different ways to research the answer to a question, which one did they choose?).

The study being used as an example of this process is this recently published one: Bread Affects Clinical Parameters and Induces Gut Microbiome-Associated Personal Glycemic Responses (1)

Let’s get on to part 2, where we dive into some scientific methodology! (Are you as excited as I am???) 🙂

Step 4 – How good was the study design for answering that question?

In breaking down studies into these four types, I loosely followed this Rough guide to types of scientific evidence. It’s one of my favourite infographics on the whole internet, ever. Let’s look at the different types of studies, in order of increasing strength of evidence. Starting with the (useful, but) weak evidence of cell and animal studies.

Study type 1 – Cell and Animal Studies (In vitro, ex vivo, in vivo preclinical)

• In vitro – Take a sample from a person and do experiments with it in the lab (e.g. blood, or biopsy);
• Ex vivo – Expose a person to something first, then take a sample and test it in the lab;
• In vivo – Use animals.

FUN FACT: “Clinical” studies are those done in people; while “pre-clinical” are done in animals.

Cell and animal studies are weak in terms of basing health recommendations on. But they are very useful for understanding the cellular and biochemical mechanisms of a process.

For example, if we see that people who tend to eat more of a certain food end up getting more of a certain condition, wouldn’t it be great to understand how our enzymes, DNA, cells and tissues respond to extracts of that food?

Of course!

But, by understanding a biochemical process, does this mean we can now go and recommend that everyone eats/avoids a certain food?

Of course not!

See what I’m sayin’ here? Cell and animal studies provide (useful, but) weak evidence when it comes to human health.

Study type 2 – Observational

When you observe, you don’t change anything. You’re not intervening in anyone’s life choices, you simply ask lots of good questions. Almost all nutrition studies fall into this category. There are lots of ways to do observational studies.

You can:

• Take people with a certain issue and interview them. Ask them questions to figure out if something you asked them may have contributed to their issue. This is called a “case study.” Put lots of these together into a series and you get a “case series.”
• Let’s take a “case study” to the next level. Enter the “case-control study.” Here we take a pre-determined number of people with a certain issue, and the same number of people without that issue. Ask them questions to see if you can find differences between the two groups. It’s possible that one or more of those differences may have contributed or prevented their issue. This is called a “case-control” study because you broke a number of people up into two groups – the “cases” who have the issue, and the “controls” who don’t.
• Take a group of people without an issue and follow them for a while (usually years). Periodically ask them questions and have tests done. Then at the end, see how many people got certain conditions. Then look back at the answers they gave over the course of the years, and compare those answers to the people who did not get those conditions. This is called a “cohort” study because you’re actually following a cohort of people.

Correlation does not equal causation

If you’ve heard “correlation does not equal causation” it’s based on these types of observational studies. When you just observe/ask people about things and look for connections, you’re finding correlations. Things that tend to happen at the same time as other things that happened. They coincide with each other. These correlations can show you increased risks of something occurring when you do something else. It’s statistics. So when you hear that ‘people who do x have a y% increased risk of z’; THIS is correlation.

You can not find the cause of something by observationally looking at what people do and then using statistics to calculate risk.

Here is a short video from Sherry Nouraini discussing a real example of “correlation does not equal causation.” Sherry Nouraini, PhD is a Social Media Strategist, Marketing Instructor, lecturer, and Climate Change Communicator who I happened to connect with on social media around Earth Day when people marched for science (which I wrote about here). (BTW, that post shared pictures from when I lived in Newfoundland, as well as talks about the very first clinical trial ever done). #nerdalert

So, what value does all of the data from these observational studies have?

Of course, when multiple types of observational studies point in the same direction as corresponding cell and animal studies, this evidence may very well point to a cause. It’s also a great way to do nutrition studies because it’s nearly impossible to control every food, macro and micronutrient a bunch of people eat for weeks and months, which is how an experimental clinical study is done.

But we just can’t be 100% sure of the cause because correlation does not equal causation.

Study type 3 – Clinical Trials (Interventional/Experimental)

A clinical trial happens when people agree to change something in their lives. They are part of an experiment. They are allowing an “intervention.” Whether that is taking a new medicine or supplement, or eating a certain amount of a certain food for a certain number of days/weeks.

This is why clinical trial results are much stronger than observational studies when it comes to human health. People actually do something different. We’re not just asking questions here like when we’re observing, we’re actually changing something.

That’s why it’s an “experiment.”

Of course, these studies can come in many sizes and durations. Ideally, they’re randomized (people are randomly assigned to groups) and blinded.

FUN FACT: “Blinding” a study means that the participant doesn’t know if they’re getting the actual treatment or a placebo. “Double-blinding” means that neither the participant, nor the person speaking to the participant, know whether they’re getting the actual treatment or a placebo.

For the most part, the bigger and longer a study is, the better. But we always have to go back to step 3 – What question did the study try to answer? If it’s a very preliminary study of something we don’t know much about, we’re not going to invest in a huge clinical trial just yet. Plus, some questions make no sense to have short studies, and some make no sense having long studies.

So, these aren’t hard and fast rules.

But when it comes to approving a new drug or medical device, these are absolutely necessary in one shape or another.

This is why informed skepticism is a superpower!

Study type 4 – Systematic Reviews and Meta-Analyses

A review study, such as a systematic review or a meta-analysis actually looks at a bunch of studies to try to get a bigger picture of the evidence. All of the clinical evidence. Not just one or two, but several studies.

It’s how the Cochrane Collection got started. To try to make sense of conflicting information, a bunch of similar studies are objectively reviewed to find overall trends. I talk about these in a bit more detail in this post.

Unless I’m looking for a specific study, or I’m already an expert in the field, when researching a topic I ALWAYS filter my PubMed results for review studies first. I want a general overview of a topic, so I don’t want to start with a detail.

Understanding the study type of this bread study

Let’s remember the question being asked for this study:

…”comparing the effects of traditionally milled and prepared whole-grain sourdough bread and industrial white bread made from refined wheat on multiple clinical and disease markers and on the composition and function of the gut microbiome.”

So, because of all of the proposed differences between ‘traditionally milled an prepared whole-grain sourdough bread’ and ‘industrial white bread’, they wanted to measure levels of a number of things in the blood, as well as the gut microbiome to see if there was a difference in any of these things based on the type of bread the people ate.

Study design – Crossover

The study design used was an interventional clinical trial. It used the “crossover” method which means they took two groups of people and had each of them do both interventions.

In this case, one group of 10 people ate white bread (50 g of carbohydrates) for a week, then two weeks of their regular diet. Then those same people ate the sourdough bread (50 g of carbohydrates) every day for a week. They “crossed over” from one bread to the other.

At the same time, the other group did the exact opposite (sourdough for a week, 2 weeks regular diet, white bread for a week).

Subjects were instructed to consume 50 g of available carbohydrates from bread every morning of the week (145 g and 110 g of sourdough and white bread, respectively, given in the form of standardized meals, STAR Methods) and to supplement the rest of their regular diet with additional bread of the same type.

Study design – One week; 2 weeks “washout;” then one more week – Figure 1a

Each intervention period was one week. This isn’t long in terms of how long studies can be, however, they’re measuring short term responses. Things like gut microbes and blood levels of sugar, lipids, nutrients, etc. These don’t necessarily need weeks, months or years to be affected. In fact, measuring gut microbes and blood sugar months after eating a piece of bread can be pretty meaningless, wouldn’t you say?

Controlling for carbohydrate intake of daily bread

Because they were measuring blood sugar, they controlled for the carbohydrate amount. This means that each morning the people had to eat at least 50 g of carbohydrates from the specific bread. This meant 145 g of sourdough and 110 g of white bread. This makes sense because we know that blood sugar rises based on grams of carbohydrate intake, not grams of bread. They were allowed to eat more of that bread throughout the day, as long as they measured how much.

Matching carbohydrate intake from week to week

The reason for that was to control for another variable: carbs between the first week of bread and second week of bread.

On the first intervention period, additional bread was consumed ad libitum; while on the second intervention period, available carbohydrates of supplemented bread were matched to the first intervention period (Figure 1A).

This means that if they ate 65 g of carbs from one bread on day 1 of the first week, they needed to eat 65 g of the other bread on day 1 of the other week.

…consumption of available carbohydrates was purposely matched between the intervention periods

Controlling for differences between people

This is one of the strengths of a crossover study. Each person actually acts as a “control” for themself because they try both interventions on themselves.

So, if one person has a huge blood sugar spike from those 50 g of sourdough carbohydrates, they can compare their own personal blood sugar spike when they have 50 g of white bread.

Controlling for reactions to wheat

Another variable they controlled for was…

Subjects were also instructed not to consume additional wheat products during the intervention periods, including bread that was not supplied to them, pasta, and any other wheat-based products

So, they couldn’t eat other wheat products during those weeks they were eating one bread or the other. This controls for any reactions they have to other wheat products.

Blood tests – Figure 1b

They measured 20 variables, including blood glucose, weight, blood fats, mineral status, etc. You can see them in figure 1 b. The numbers show that, while there were small differences between baseline (the pre-experiment measurements, indicated by the dashes in the middle of the lines).

They then compared the changes from eating one bread or another. That difference is the middle dots of those lines.

You can see that those middle dots are still very close to those middle dashes.

The lines coming out of each dot is the 95% confidence interval. This is a statistic that calculates when we’re 95% confident of our answer.

As you can see the baseline dashes are still within range of those lines coming from the dots.

BOTTOM LINE: The differences between the blood test results from when people at the white bread vs the sourdough bread were not statistically significant. We are not confident that there was any difference on those parameters when people at the white bread or the sourdough bread.

Clinical significance

None of these 20 things changed to the point where they were clinically significant. In other words, there was no health impact.

FUN FACT: According to Wikipedia, “Clinically significant” is the practical importance of a treatment effect – whether it has a real genuine, palpable, noticeable effect on daily life.

So, while the white bread slightly increased iron levels, and the sourdough bread sightly increased calcium levels), none of these changes were clinically significant. The changes were small enough to make someone deficient or not, or to get an illness or not, or to change their life in any way. They were simply test results.

This goes back to the first bullet in the highlights that says:

• Crossover trial shows no differential clinical effect of white versus sourdough bread.
  • People were given white and/or sourdough bread. The type of bread didn’t seem to affect people differently (no differential clinical effect).

Is one week really enough to see the blood test results?

What about the length of the study? I mean, is seven days long enough to test for blood levels of sugars, etc.?

The authors state here:

Notably, we found that a single week of bread consumption resulted in changes to multiple clinical variables and risk factors (Figures 2A–2K) that were statistically significant, albeit numerically small and not necessarily clinically significant.

This means that they found blood levels of nutrients, lipids, etc changed enough to be “significant.” So, when someone ate a bread, it changed their blood levels of certain things. That was statistically significant. Before eating bread and after eating bread was statistically significant.

But, the differences in those changes between the breads were not significant.

And, the actual clinical significance (e.g. health impact) was not there. So, yes, the blood glucose, etc. changes, but does that mean anything for someone’s health…not in this short trial.

This is where they got one of their highlights:

• The glycemic response to the two types of bread varies greatly across people.
  • Individuals had different blood sugar levels (glycemic response) from the bread.

Is one week really enough to see the gut microbe results?

What about the microbes? Is 7-days a reasonable amount of time to test for microbial changes (in stool samples)?

Several studies indicated that even short-term dietary interventions, whether animal-based, plant-based (David et al., 2014), or involving barley bread consumption (Kovatcheva-Datchary et al., 2015), result in significant, rapid, and reproducible alterations to the gut microbiome.

So, yes, 7-days is expected to be long enough to see changes in the gut microbiome.

How the 7-days of bread affected the gut microbes

But…they only saw minor changes.

“our results demonstrate that even though the intervention performed here was radical enough to significantly change clinical variates, microbiota composition underwent only minor alterations, demonstrating that the gut microbiota is resilient to some types of nutritional changes.”

What they found was that it didn’t seem to matter which bread people ate. A person who ate the white bread had similar blood test results and microbiota when they ate the sourdough bread.

Remember earlier when we talked about the strength of using a crossover study? It’s that each person acts as their own “control” because they get both treatments (at different times, with a “washout” period between those times).

This goes to their second highlight:

• The microbiome composition was generally resilient to dietary intervention of bread.
  • The type of bread didn’t affect their gut microbes (resilient to dietary intervention).

Interpersonal variability = Everybody’s different

The people reacted differently to other people, so there was interpersonal variability. But, they acted the same as themself, regardless of the bread they ate.

The lack of differential treatment effects between white and sourdough bread implies that either the two breads exert similar effects within each individual or, more intriguingly, that the effect of each bread is person specific such that, averaged across subjects in the cohort, bread type does not affect the overall average.

A fascinating suggestion they made is that because they found interpersonal variability, but not intrapersonal variability (because of the crossover design), that this can explain a lot of the contradictory results found when studying effects of foods on people.

“our results suggest that such personalized glycemic responses should be measured and accounted for when evaluating the effects of food. We hypothesize that this may explain several contradictions in the literature regarding the effects of different foods.”

Can they predict blood responses based on gut microbes?

Based on their results, they created a model to try to predict the blood response to bread based on the gut microbes. Guess what? It was accurate!

“…demonstrating that the glycemic response-inducing bread can be accurately classified for each subject using only microbiome data.”

Of course, this is correlation. People with certain gut microbes are more likely to have certain blood marker levels when they eat bread. We really don’t know how this happens, and more studies are needed. But, don’t you think it’s fascinating?

And for their final highlight:

• Microbiome-based classifier accurately predicts glycemic response-inducing bread type
  • And that there was a strong connection between someone’s microbiome and their blood sugar levels after eating bread.

Summary of conclusions

Now we’re back to the summary of their conclusions that we talked about in part 1:

• Crossover trial shows no differential clinical effect of white versus sourdough bread.
  • People were given white and/or sourdough bread. The type of bread didn’t seem to affect people differently (no differential clinical effect).
• The microbiome composition was generally resilient to dietary intervention of bread.
  • The type of bread didn’t affect their gut microbes (resilient to dietary intervention).
• The glycemic response to the two types of bread varies greatly across people.
  • Individuals had different blood sugar levels (glycemic response) from the bread.
• Microbiome-based classifier accurately predicts glycemic response-inducing bread type
  • And that there was a strong connection between someone’s microbiome and their blood sugar levels after eating bread.

Are you with me?

It didn’t say:

• That one type of bread is better or worse.
• That bread is good or bad for you.
• Anything about wheat or how it’s processed.
• etc.

[convertkit form=5058159]

Step 5 – Was the research biased?

As I said at the beginning, we all have biases. Especially if we feel like we’re an expert in an area, or have an opinion about something.

This includes scientists.

But, that’s the beauty of the scientific method. Over the centuries, more and more things are being found and done to remove bias. For example, the idea of “blinding” and even “double-blinding” clinical studies to control for the placebo effect. Or having a study “peer reviewed” by three different researchers before publishing it.

Not to mention the ever important, and seriously lacking, confirmation studies.

The whole purpose of science is to understand how things work. Literally, it’s understanding nature.

How does our digestive system work? What causes certain diseases?

Science is not perfect. At all.

It is, however, in my very strong opinion, the absolute BEST way humans have developed to learn how things work. It’s the best way to make decisions, especially when it comes to public health.

There are lots of ways bias can sneak into a study. Here is a great infographic that outlines many of them. It lists many ways science is either done, or represented badly. For example:

Science done badly:

• Conflicts of interest
• Unsupported conclusions
• Problems with sample size
• Unrepresentative samples used
• No control group
• No blind testing
• Selective reporting of data
• Unreplicable results
• Non peer-reviewed material

Science represented badly:

• Sensationalized headlines
• Misinterpreted results
• Correlation and causation
• Unsupported conclusions
• Unrepresentative samples used
• Non peer-reviewed material (this is SO common in published health books and documentaries – beware!)

Was the research biased in this bread study?

First stop is the “conflicts of interest” section which is published in studies now.

E.S. and E.E. are paid scientific consultants for DayTwo Inc. DayTwo analyzes your microbiome to predict blood sugar responses to thousands of different foods. High blood sugar is linked to energy dips, excessive hunger, weight gain and increased risk of obesity and diabetes.

Do you think this is a bias? Well, it sure could be, but even if research is paid for by a company, that doesn’t mean it wasn’t done to the highest standards or was unethical. Money is one of many things that can influence a study’s outcome. (See the others above).

But now that you know – you can dig further, or take it with a grain of salt.

Step 6 – Super-complex other things – BONUS

There is just SO much to add here, and that’s why in order to become a scientist and do research, you need more than a Bachelor’s of science. That gives you a great foundation, for sure! But at that point you’ve mostly learned theory and did a few dozen pre-conceived lab experiments. You barely touch the method of actually designing and conducting research until in graduate school for Master’s, PhD, post-doc, and beyond.

Some of the things that I will include here are:

• Deeply analyzing the specific methodology;
• Randomization – How were people randomized?;
• The questions that were asked and the way they were asked;
• What variables were controlled for?;
• Lab testing methods;
• Critiquing the type of statistics used (have you heard of “p-hacking?”);
• etc.

Conclusion

I hope that these two blog posts have given you a greater understanding and ability to be a more informed skeptic of health studies!

Signing off and toasting: To better quality skepticism of health studies!

Over to you

What do you think? Do you have a new tool to critique health studies? Do you want me to dig deeper into an area?

Let me know in the comments and I’ll personally reply!

References

Further reading

Of course, I stand on the shoulders of giants. Don’t we all?

Here are links to some excellent further reading that I highly recommend if you’re interested in diving deeper into this.

How to read and understand scientific research

• Understanding Health Research
  • A tool for making sense of health studies
• What does a scientific paper look like?
• 9 Questions to Help You Make Sense of Scientific Research
• Evaluating Evidence
• A Rough Guide to Spotting Bad Science
• A Rough Guide to Types of Scientific Evidence
• The #1 thing for holistic professionals to increase (scientific) credibility

Science of nutrition

• Why nutrition science is so confusing. [Infographic]
  • 9 reasons eating well isn’t as straightforward as we’d like it to be.

• Processed Food vs Real Food:
  • Why Nutrition Science is So Confusing (and what to do about it)

• I asked 8 researchers why the science of nutrition is so messy.
  • Here’s what they said.

• What have we learned about nutrition in the past 5 years?