Skeptical of new health study? Great! (Part 1)

So, you’re skeptical of a new health study? This is great!

Two weeks ago I published an article on “3 solid reasons to research health blog posts.”

Let’s say you’re doing that all-important credibility-building research. You found a study.

Now what?

Now you need to figure out what it means, and if it’s useful for you. Right?

I’ve broken down this process into five steps, plus a bonus step. (This post is part 1, here’s the link to part 2).

These steps will give you some guidance on how to approach a study. They’re an overview. In fact, I could (and would love to) write an entire book, book series, or course on how to understand and critique health studies.

So, throw your thoughts and questions into the comments below. I’ll try to help and respond to each one. You may even give me fodder for part 2, or another blog post, book, or course. 🙂

Where I’m coming from

To be 100% honest, I welcome a respectful discussion on this topic.  I know there are a lot of concerns with medical science, and that’s part of what drew me from being a toxicologist, product safety specialist, pharmacovigilance expert; to studying holistic nutrition.

BUT, I see the best in both approaches to health. Both medical science and holistic health freaking rock at certain things!

And both totally suck at others!

My big picture “why” is public health. The way I see it, we need both approaches to health. We need to respect both medical science and holistic health approaches. They serve different people in different stages of health.

And we both need to get off our high horses to refer to each other when necessary. And ditch the egos to admit where we actually don’t know things.  I fully respect humility and transparency. No matter what your position is.

In fact, late last year I wrote an entire post on the #1 thing holistic professionals need to increase their credibility.

Want to know what that one thing is?

SPOILER ALERT:  It’s looking at the balance of the evidence. Putting that one new study into context with the rest of what we actually know on the subject.

Don’t focus on one study, one group of patients/clients, or a handful of expert opinions (yes, I’m talking about documentaries).

What study are we looking at?

The best way for me to walk you through how to review a study is with an example. So, let me choose a study that is current, and has raised some waves in the health world this week:

Bread Affects Clinical Parameters and Induces Gut Microbiome-Associated Personal Glycemic Responses (1)

I’ll go through each of the five steps, and apply it to this study.

Are you ready?

Step 1 – Understanding the results

Let’s admit it. Scientists don’t speak English (or whatever language you speak). They speak “science-ese.” And not only “science-ese,” but specialized versions of it, like “nutrition-ese,” “toxicology-ese,” “physiology-ese,” and “microbiology-ese.”

Amirite?

I am fortunate to understand these languages, because I studied them for years. It doesn’t come easy. And that’s part of the problem.

BTW, I don’t speak “electrochemistry-ese,” “astrophysics-ese,” or “geology-ese,” etc.

So, one thing some scientific journals are doing now to help people understand the results of the studies is to have a summary of the study in…

Wait for it…

…

English! (Or whatever language the study is published in).

What a revelation!

Now, if you’re reading a study that doesn’t have this results summary in English, then you should slowly and carefully read the results. They are found near the end of the study, right before the “Discussion.” Or, if you only have an abstract of the study (not the entire study) it will usually be the last few sentences.

I am not kidding that the results need to be read slowly and carefully. I speak those science-ese languages, and I often read a sentence; read it again out loud; think about it for a minute; read it again. Until I have an “aha” moment and get it.

As I said, it’s unfortunate that this language does not come easy, but with patience and perseverance, it can come!

So, Step 1 is understanding the results of the study.

Understanding the results of this bread study

So, here is an image of the four-bullet summary of this study written in English:

It basically says that:

• People were given white and/or sourdough bread. The type of bread didn’t seem to affect people differently (no differential clinical effect).
• The type of bread didn’t affect their gut microbes (resilient to dietary intervention).
• Individuals had different blood sugar levels (glycemic response) from the bread.
• And that there was a strong connection between someone’s microbiome and their blood sugar levels after eating bread.

Are you with me?

It didn’t say:

• That one type of bread is better or worse.
• That bread is good or bad for you.
• Anything about wheat or how it’s processed.
• etc.

The meaning and implications of the results are discussed in the “Discussion” part of the study. This is after the results. So, right now, let’s focus on what they did, how they did it, and what they found first.

No extrapolation. No assumptions. No implications.

Just results.

Ok, next step…

Step 2 – What does my gut say? Do I tend to agree or disagree with the results?

The next step is a “gut check.” Think for a second whether you agree with the study results or not.

And if you’re being skeptical, then you probably don’t instantly agree.

Great!

Have you ever heard of “confirmation bias?”

Well, if you have a gut reaction (for or against), then you probably already have an opinion on the topic. This is a universal thing. Pretty much everybody has it on certain topics to some degree.

And, me too!

I like to think that I’m open to new information. I mean, coming from a hard-core science background and eventually becoming a holistic nutritionist has some degree of openness to new and conflicting information, right? Some degree of dropping pre-conceived opinions on a topic. If you want to know my health philosophy, and why I don’t fit in, then I wrote it all out here.

So, your gut reaction and opinion is something to notice, acknowledge, (not judge), and put on the backburner for a few minutes while you do some skeptical inquiry.

Do I agree with the results of this bread study

Honestly, I’m neither here nor there with this. The microbiome is a hotbed of research for a few reasons. Mostly because new technology is allowing us to profile multiple microbial species in a short time.(2) Also because this technology is relatively new, we really, truly don’t know that much about the microbiome yet. New studies on gut microbiota (microbes that live in our gut) and probiotics (beneficial microbes we can consume) are coming out every day. I find it fascinating. And I’m reserving judgment until science has explained more.

There is a Human Microbiome Project underway.(3) They’re looking in three different areas:

• Pregnancy and Preterm Birth,
• Onset of IBD,
• Onset of Type II Diabetes

Here’s what I do believe though, gut microbes are a lot more important to our health than we ever thought before! How important?…We really don’t know just yet.

And when it comes to the bread being sourdough or white, I really don’t care about that. It doesn’t bother me either way. Yes, white bread uses refined grains that have been stripped of the bran, fibre, and most nutrients. And yes, those nutrients are often returned to the grain by the process of fortification. But I personally am not as philosophically opposed to that as others are.

When it comes to sourdough, again, I really don’t care too much. Yes, the flour is less processed, has more of its natural nutrients, and has some level of fermentation. I really don’t know if any of those tiny amounts in bread can significantly affect human health.

Also, I really don’t know whether to expect a difference in the gut microbes between white and sourdough bread. I mean, those microbes that ferment the sourdough bread are already killed in the baking process. Apparently they reduce phytates and make the minerals more absorbable. I really haven’t seen any evidence on this. (If you have some, feel free to share it with me in the comments).

FUN FACT: Diets based on processed foods (or shall I say “ultra-processed food-like products”) are bad for everybody. Full stop. No question.(4) That’s not what this study is about, though.

Which brings us to Step 3…

[convertkit form=5058159]

Step 3 – What question did the study try to answer?

This is answered in the introduction of the study. The introduction gives a brief summary of the current knowledge on that topic. It talks about what they want to study, and why they want to study it. It can also reveal their hypothesis, or what they think they might find based on what they already know.

If you only have access to the abstract, and not the whole study, then this should be summarized in the first few sentences of the abstract.

In order to study something scientifically, the best way to remove variables is by being specific and looking at just a couple of things. The more things you try to test at the same time, the more “confounding” there can be. These are factors (variables) that can affect each other. And the more interactions there are, the less you know which result is from which factor.

What question did this bread study try to answer?

In this study, we performed a randomized crossover trial with 20 healthy subjects comparing the effects of traditionally milled and prepared whole-grain sourdough bread and industrial white bread made from refined wheat on multiple clinical and disease markers and on the composition and function of the gut microbiome.

The researchers wanted to know if there would be a difference in disease markers (e.g. blood sugar levels, mineral levels, blood lipids, etc.) and gut microbes in people after eating “traditionally milled and prepared whole-grain sourdough bread” compared with eating “industrial white bread made from refined wheat.”

That was their question.

So, now that you know what they wanted to study…

Step 4 – How good was the study design for answering that question?

Of course, this depends on the question. As I just mentioned, the more specific the question, the better.

And this is different from the holistic approach. Holistically looking at someone takes into account their physical and mental connections. And looks at all systems as being interrelated.

I’m not saying this isn’t true. Studies show that between 60% & 80% of doctor visits may have a stress component.(5) You don’t need to convince me there is a mind-body connection.

But the way to study things objectively is by looking at one or two tiny things at a time. And there are multiple ways to study those tiny things.

Let’s group types of studies into four “buckets.” Study type 1 (lowest quality) remotely and theoretically may possibly apply to human health on some level. Maybe. All the way to study type 4 (highest quality) which is very applicable to human health, because it’s based on many human health studies.

And there are so many combinations and permutations of these studies. Don’t forget, some types of studies cannot be done because they’re unethical. Some studies are too expensive to do on a large scale. Perfect studies are hard to come by. So we need to take each one and critique it and see if and how it provides us one piece into understanding this infinitely complex thing called human health.

As for nutrition, most studies are in the type 2 (observational) category. This is why there are so many conflicting opinions when it comes to health and nutrition. You can’t run real nutrition experiments by forcing people to eat a specific amount of specific foods for days, weeks and years. Food is an inherent part of our daily habits and culture. It’s not like popping a pill (or sugar pill) every day for a few months. That can be added to our daily routines. But having all foods pre-measured for every meal and snack every day. How?

In breaking down studies into these four types, I loosely followed this Rough guide to types of scientific evidence. It’s one of my favourite infographics on the whole internet, ever.

Here’s the link to part 2…

In the meantime, I’ve collected some recommended reading for you. They’re just below the references here.

One list is to understand health science in general. The other links to many articles that talk about why nutrition science is not an exact science.

Signing off and toasting: To better quality skepticism of health studies!

Over to you

What do you think? Do you have a new tool to critique health studies? Do you want me to dig deeper into an area?

Let me know in the comments and I’ll personally reply!

References

(2) Arnold, J.W., Roach, J., M. Andrea Azcarate-Peril, M.A. (2016).  Emerging Technologies for Gut Microbiome Research. 24(11), p887–901. DOI: http://dx.doi.org/10.1016/j.tim.2016.06.008 
http://www.cell.com/trends/microbiology/fulltext/S0966-842X(16)30071-3

(3) NIH Human Microbiome Project 2. About the Human Microbiome
http://ihmpdcc.org/overview/

(4) Ludwig, D.S. (2011). Technology, diet, and the burden of chronic disease. JAMA. 05(13), 1352-3. doi: 10.1001/jama.2011.380.
https://static01.nyt.com/packages/pdf/opinion/Technology_Diet.pdf

(5) Nerurkar A, Bitton A, Davis RB, Phillips RS, Yeh G. When Physicians Counsel About Stress: Results of a National Study. JAMA Intern Med. 2013;173(1):76-77. doi:10.1001/2013.jamainternmed.480
LINK:  http://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1392494

Further reading

Of course, I stand on the shoulders of giants. Don’t we all?

Here are links to some excellent further reading that I highly recommend if you’re interested in diving deeper into this.

How to read and understand scientific research

• Understanding Health Research
  • A tool for making sense of health studies
• What does a scientific paper look like?
• 9 Questions to Help You Make Sense of Scientific Research
• Evaluating Evidence
• A Rough Guide to Spotting Bad Science
• A Rough Guide to Types of Scientific Evidence
• The #1 thing for holistic professionals to increase (scientific) credibility

Science of nutrition

• Why nutrition science is so confusing. [Infographic]
  • 9 reasons eating well isn’t as straightforward as we’d like it to be.

• Processed Food vs Real Food:
  • Why Nutrition Science is So Confusing (and what to do about it)
• I asked 8 researchers why the science of nutrition is so messy.
  • Here’s what they said.

• What have we learned about nutrition in the past 5 years?