What's healthy, what's not? Fix the p-values and we might, maybe know

Long, long ago, we Americans hear that "fat was bad."

Then more research said, maybe just saturated fats.

Now, with more consensus (for now), it's that trans fats are definitely bad.

Meanwhile contrarians like Gary Taubes and Nina Teicholz are claiming that not only are we eating too many carbs, adn that more fats as well as proteins may be good, but specifically, that more saturated fats are good.

Teicholz is the latest contrarian to deliberately overreact and not only criticize but demonize more traditional research.

This shift seemed like a good idea at the time, but it brought many potential health problems in its wake. In those early clinical trials, people on diets high in vegetable oil ... were more likely to die from violent accidents and suicides."

Teicholz then futher undercuts her own credibility by claiming the possibility, even the likelihood, of causal correlation from vegetable oils changing brain chemistry.

Beyond that, it looks like there's a BUNCH of errors and problems behind the latest research Teicholz cites to support her claims. Outright errors run through the filter of meta-analysis is a sure prescription for bad interpretation.

Also, I've generally found that the likes of Tabues and Teicholz fail to distinguish between simple and complex carbohydrates. It's true that the likes of Ancel Keys, along with all of his  research ethics issues in leading the crusade against saturated fats, also did the same. But, that was 60 years ago. There's no excuse for that today.

The real problem? The p-value of 0.05 in medical research is too high, just as it is in social sciences, and is almost surely a leading contributor to the problem of replicability in psychology and sociology.

I understand why it was set as high as it is, compared to the 0.0001 in modern natural sciences. Researchers wanted a loose value so as to not screen out potentially lifesaving, or health- or mental health-saving treatments and protocols.

However, our accuracy of research, plus the number of researchers, especially in health and medicine, plus the compiled history of past research, all tell us that we don't need such loose standards today. If anything, they tell us that such loose standards may even be harmful.

They confuse people on what really is best medical practice.

In addition, they give contrarian diet writers like the above more loopholes to write contrarian books.

And, especially in mental health, but also elsewhere, the loose p-value gives Big Pharma more room to make dubious claims about the effectiveness of many prescriptions.

And, as a combination of Nos. 2 and 3, they cause the US as a whole to waste a bunch of medical spending. 

That said, I'm not going to argue that the current "medical establishment," minus the Big Pharma part, doesn't have some financial vested interests itself. Are they, relative to the overall size of the medical issues involved, of the same degree of financial interest as Big Pharma or contrarians, though? I don't think so.

And, I just mentioned people like Gary Taubes. The folks like Joseph Mercola use the looseness of p-values, and other, related issues, to drive Mack trucks laden with gold through them on the way to their personal Fort Knoxes. These fixes alone won't shut up Mercola, but they'll force the likes of him to rely on studies that are not only more clearly fringe ones, but outdated ones, too.

So, a simple suggestion?

Cut the p-value in health/medicine tighter. Say, 0.03 instead of 0.05. That's still plenty loose enough to not screen out "edgy" but true findings.

The feds, via the National Institutes of Health and the National Institutes of Mental health, have the power to make this happen. Stop funding research that don't adopt the tighter standard.

Will tighter p-values alone make a huge difference?   Maybe not a huge difference, but, they will make a difference. The feds could also stop funding most research that uses meta-analysis, and tighten up other data-related issues, many of which, such as confidence levels, are themselves at least loosely correlated with p-values. All of that together would make a big difference indeed.

And, changing what we readily can on statistical tightness, starting with p-values, would be a signal to the health and medicine fields on the one hand, and social science on the other, that it's a new day in the research world. The tighter standards would signal a need for tighter research in general. And, the requirement of special justification for the use of meta-analysis would signal that the days of cherry-picking are over.

The PLoS antidepressants study, the ‘looseness’ of medical research statistics and ‘faith’ in meta-analysis

Way too loose of p-values for false positives in studies, in medicine (and social sciences) compared to natural sciences, is one reason to not read too much into any individual study that claims antidepressants are ineffective, like the Public Library of Science meta-analysis of individual studies did.

P-values of the same looseness as in medicine/social sciences have been used to claim intercessory prayer actually works on sick people (halfway down the linked page), for example, or here (two-third down the linked page):

Targ's paper is not the only questionable study on the efficacy of prayer that has been published by medical journals. The editors and referees of these journals have done a great disservice to both science and society by allowing such highly flawed papers to be published. I have previously commented about the low statistical significance threshold of these journals (p-value of 0.05) and how it is inappropriate for extraordinary claims (Skeptical Briefs, March 2001). This policy has given a false scientific credibility to the assertion that prayer or other spiritual techniques work miracles, and several best selling books have appeared that exploit that theme. Telling people what they want to hear, these authors have made millions.

Also, per a blogger, I came across a good statement on how many people misunderstand p-values in general:

First, the p value is often misinterpreted to mean the “probability for the result being due to chance”. In reality, the p-value makes no statement that a reported observation is real. “It only makes a statement about the expected frequency that the effect would result from chance when the effect is not real”.

In short, as I’ve tried to explain to people over at Kevin Drum’s blog, p values in medicine are simply too loose.

But, as the study’s authors claim, doesn’t meta-analysis take care of all those p-value problems? No.

Meta-analysis, no matter how much it’s defended, can’t totally cover that up.

I’m not saying that the results of a meta-analysis are no stronger than the weakest study in its umbrella. I am saying that, with p values as loose as they are in health/medicine (and social sciences), is that no massive amount of individual research studies being included under one meta-analysis will make the meta-analysis’ results anything more than a little bit stronger than the best individual study.

In other words, in medicine, and in social sciences, meta-analysis adds a very modest bump, nothing more. The problem is, most people believe it does much more than that when it doesn’t.

Or, to put it another way, meta-analysis is no better than the material it’s analyzing.

So, what’s needed is medical studies to continue with the p of 0.05, because we don’t want to risk screening out potentially life-saving study, but, to re-crunch research studies at the same time. I’m not saying we need to do that with a p of 0.0001, or 1/100 of 1 percent, like the natural sciences, especially physics, normally do. But to re-crunch with a p of 0.01, or 1 percent instead of 5 percent? Absolutely.

Research that made the 5 percent cutoff but not the 1 percent cutoff would be categorized as “worthy of further study but without any immediate conclusions from it being acceptable.”

A sidebar benefit would be that a lot of alt-medicine research would get a less than full imprimatur.