How to get a study to show what you want it to.
In 2018, the journal Metabolism: Clinical and Experimental published a study titled "Keto-adaptation enhances exercise performance and body composition responses to training in endurance athletes." (Bold emphasis added)
Given other studies on this topic have shown low-carb/ketogenic diets do not improve or even worsen performance, I was naturally puzzled as to how this study found they could now "enhance" performance.
The content below was originally paywalled.
As soon as I saw Jeff S. Volek's name on the author list, I had my answer.
Volek has co-authored a number of low-carb books, including "New Atkins For a New You: The Ultimate Diet for Shedding Weight and Feeling Great", “The Art and Science of Low Carbohydrate Living”, and “The Art and Science of Low Carb Performance.” Volek is also an "Atkins Health Professional" and member of the "Atkins Advisory Board." In other words, Volek is on the payroll of one of the world's best-known low-carb food companies and makes money from pimping low-carb diets.
Yet the disclaimer section of the paper simply reads: "Dr. Volek receives royalties from books on nutrition and exercise." It conveniently leaves out the critical detail that those nutrition books happen to be staunchly pro-keto. There was no reason for the authors to leave out this important detail; I can only assume it was done in an attempt to prevent readers realizing the full extent of Volek's vested interest in the topic.
When I see this kind of dubious behaviour, I promptly lose respect for the researchers and immediately become suspicious of their true motives.
Reinterpreting Results to Make Your Pet Diet Look Good
The contributions section at the end of the paper credits design of the study to a group of researchers from Ireland (McSwiney et al), who have published a number of other pro-keto papers. The US-based Volek is listed as contributing to "data interpretation and manuscript preparation."
To qualify for the study, subjects had to be athletes who competed in endurance events, completed at least 7 hours of training per week, and had at least 2 years' training experience.
Forty-seven male endurance trained athletes (18–40 years) were enrolled, but only 20 completed the study. That's a massive dropout rate of 68%, and occurred mostly in the low-carb ketogenic diet (LCKD) group (18 dropouts versus 9 in the high-carb group). When twice as many people are bailing from the low-carb group, that's your first sign that "The Ultimate Diet for Shedding Weight and Feeling Great" may not be all it's cracked up to be.
The reasons for dropping out during the 12-week study were:
injury or illness not related to the intervention (HC n = 7; LCKD n = 9);
intervention too time consuming (HC n = 1; LCKD n = 1);
dietary intervention too difficult to adhere to (LCKD n = 5);
participants unable to complete post-intervention testing (LCKD n = 2);
strength and HIIT training too difficult to incorporate into training week (HC n = 1);
technical difficulty at post-intervention testing (LCKD n = 1).
So we have not only a higher rate of dropouts in the keto group, but 3 participants in that group who could not complete or had "technical difficulty" at the all-important final testing.
Why?
Were they too fatigued and glycogen-depleted to make it through the test?
And of the five who reported the keto intervention was too difficult to adhere to, was this for logistical reasons (e.g. lack of food variety) or because they felt and/or performed worse during training?
All of which raises the question of how the results would have looked if these athletes had been forced to push on and complete the final testing as best they could? How would the results of this study looked had there not been such a high dropout rate?
What makes the dropout rate even more alarming is that this was not a randomized study but a self-selected study. What that means is that the participants were not randomly assigned by the researchers to the dietary groups, but were instead allowed to choose their own dietary allocation. The reason the researchers opted for self-selection, they explain, was to enhance adherence to the study protocol. Yet the study was still marred by an unusually high dropout rate.
So this was not a randomized controlled trial but something more akin to a multiple case study. The obvious problem with this strategy is the removal of random assignment, and the introduction of unknown confounders. When participants are allowed to choose their own intervention, it raises the very real possibility that the resultant groups possess different attributes, be they physical or psychological, that may impact upon the study’s final results.
While I don’t possess ESP and can’t read the minds of these Irish athletes from here in Australia, one thing readily apparent from the baseline data is that the athletes who self-selected the keto diet were, on average, heavier and fatter than the high-carb participants. At the start of the study, the keto group weighed a mean 86.3 kg and had 17.5% body fat, which seems uncharacteristically chubby for a group claiming to have trained at least 7 hours each week for the last 2 years or more. Meanwhile, the high-carb group weighed 76.5 kg and sported 12.8% body fat at baseline.
This makes one wonder if the LCKD group chose the keto diet in the belief it would help them lose weight and fat. This adds a potential extra element of motivation for those subjects, when compared to the high-carb subjects who were notably leaner.
One thing I can’t help but suspect in studies where the researchers have a demonstrated bent towards a particular dietary style is whether they gave extra encouragement or otherwise “customized” the study in a manner that would favour their pet diet.
That suspicion was heightened in this case when I viewed the dietary data. Both groups were reportedly given target macronutrient percentages to shoot for during the study. Given their key roles in growth, repair, and energy utilization, there has been ample research into the protein and carbohydrate requirements of athletes. Research is now pretty clear on the minimum grams per kilogram of body weight requirements of protein and carbohydrate for athletes. Therefore, simply giving macro percentages to shoot for, without explicitly instructing minimum intake of defined protein and carbohydrate amounts in grams, seems a rather inadequate and 1980s way of doing things.
It does appear, however, that the keto group got more specialized advice. For reasons they don't explain, the authors instructed the keto subjects to "adhere to carbohydrate and protein guidelines, and consume dietary fat ad libitum." The high-carb group group, report the authors, got a "nutritional handout (that) included guidelines on how to formulate a HC diet according to their daily energy requirements." The keto group's handout, meanwhile, “included information on how to formulate a LCKD diet, a shopping list and example meal plans."
This imbalanced approach to dietary instruction appears to have resulted in a whopping discrepancy in self-reported protein intake among the two groups. At the start of the study, protein intake in the two groups was similar: 118.9 g and 110.3 g per day in the high-carb and keto groups, respectively.
The dietary self-report from week 12, however, indicates that the researchers’ advice caused the HC group to reduce their protein intake down to a pathetically low 91 grams daily. No adult male athlete should be eating such a low amount of protein, unless they are a jockey. Even using the more generous yardstick of lean body weight instead of overall body weight, that works out to 1.4 grams of protein per kg, a woefully inadequate amount for hard-training athletes.
The LCKD group, in contrast, increased their self-reported protein intake to 131 grams daily, or 1.9 grams per kg of lean body weight - a far more generous protein intake in line with the recommendations of sports scientists.
This is all based on a 3-day dietary self-report obtained in week 12 of the study. If these differences hold true for the entire 12 weeks, it means the high-carb group consumed inadequate protein for the duration of the study.
In addition to dietary advice, subjects in both groups were told to perform HIIT training 2 x a week and to do several sets of squats or leg presses twice weekly.
At week 12, the LCKD group reported eating an extra 179 calories per day than they did at baseline. Yet during the study, they lost 5.9 kilograms and reduced their body fat by 5.3%. No-one in the history of humankind has ever been shown under metabolic ward conditions to lose body weight while increasing their calories - the only place this ever happens is in the wishful minds of flabby low-carb dogmatists.
Which means one of two things: The dietary self-report was wrong (quite possible) or the subjects in the LCKD group increased their training volume and/or intensity when compared to the HC group (also possible). The researchers actually acknowledge the following possibility when they write “Added weight loss within LCKD participants could be due to slightly greater volume of training undertaken each week.”
So What Were the Performance Results?
The subjects’ performance was tested at baseline and at the end of the study. The test was preceded by a 10 minute warm up on a stationary bike. After the warm up, participants completed a six second sprint in which peak and average power output (in watts) were measured. Participants then commenced a 100 km time trial, in which they were instructed to complete the 100 km as fast as possible.
Immediately after the 100 km TT, participants completed a 3-minute power test on the bike, where peak VO2, peak power, and average power were recorded. Participants were instructed to maintain as high a power output as possible while remaining seated for the 3 minutes.
After 12 weeks, VO2max improved similarly in both groups (HC +8.7%, LCKD +6.9%).
Improvements in the 100km time trial were observed in 6 out of 9 LCKD participants, and 7 out of 11 HC participants. The average 100 km time improved by 01:13 min·s (0.7%) in the HC group, and 04:07 min·s (2.5%) in the LCKD group. The difference was not statistically significant, but the researchers repeatedly made a point of mentioning that, in competition, a difference of this magnitude could have a significant impact on competitive placings. They even wrote "the difference between winning and losing the Tour de France may be seconds."
This line of argument might mean something if the results came from a randomized, tightly controlled trial. Considering that they instead came from a non-randomized study plagued by potential confounders, runaway attrition and the presence of at least one vested researcher, I can assure you absolutely zero UCI pro teams will be switching to keto diets on the basis of these results.
And until something resembling quality science can confirm these results, neither should you.
Oh, and the ‘power’ tests?
Getting Jiggy Sneaky With It
The researchers claim there was a significant difference in six second peak power between groups, with a significant increase in the LCKD group. Average power did not change in either group.
Peak power in the final 3 minutes was also reported as significantly different between groups; decreasing in the HC group (−0.7 w/kg), and increasing in the LCKD group (1.4 w/kg). Again, average power during this segment remained unchanged.
So while average power output did not differ between groups, these results seem to indicate that a LCKD creates more powerful athletes, when compared to a HC diet.
Which is nonsense.
Astute readers may have noticed the researchers quoted the figures in watts per kilogram. But the keto subjects, remember, lost weight during the trial. When we look at the before and after absolute wattages produced by each group - i.e. the total wattage the subjects produced, irrespective of how much they weighed - the LCKD group in fact experienced declines in their power output.
In absolute terms, by the end of the study both groups - both following what I consider to be subpar diets - experienced, not improvements, but slight declines in the 6 second sprint peak wattage (1.8% vs 1.4% in the HC and LCKD groups, respectively).
In absolute terms, the average wattage the subjects maintained during the 6 second sprint improved by 1.8% in the HC group, compared to a 3% decrease in the LCKD group.
In absolute terms, the peak wattage during the final 3 minutes of the test decreased by 8.6% in the HC group, while increasing 8.9% in the LCKD group. However, the ultimately more important average wattage during the final 3 minutes increased by 1.2% in the HC group but declined by 4.4% in the LCKD group.
All of a sudden, the results of the LCKD group don’t look so great.
The only reason the LCKD group improved on the very selectively reported peak watts/kg measures was because they lost more weight, not because they experienced some remarkable performance-boosting metabolic shift in fat utilization, as the researchers surmised.
Relative versus Absolute Power
Power:weight ratio is undeniably important in many sports. In fact, in sports like distance cycling or running, relative power is a better predictor of performance than absolute power. Relative power, or the power:weight ratio, refers to the maximum amount of power one generates in relation to their body weight.
Absolute power simply refers to the maximum amount of power one can generate, irrespective of their weight.
Let me give you an example. Alberto is an elite cyclist who weighs 73 kg and has clocked a maximum power output of 1,000 watts while going all out on a stationary Wattbike. His rival Esteban, meanwhile, weighs 65 kg and has recorded a maximum power output of 900 watts.
Alberto clearly has a highest absolute power output than Esteban (1,000 versus 900 watts, respectively).
However, when considered in relative terms, the situation is reversed. Alberto’s relative power output is 13.7 watts/kg, while Esteban can generate 13.84 watts/kg. All other things being equal, Esteban actually has the power advantage.
The reason he has the advantage is that cycling is not a static sport, but one where you have to propel your body through space. In sports like Olympic weightlifting or bodybuilding, your body remains in one place while you generate large amounts of force to move an external object (i.e. a heavily loaded barbell) through space. In sports like cycling, running or rock climbing, you generate force to move your own body through space. To do this as efficiently as possible, you want to be as powerful as possible at the lightest practical body weight possible.
So if Alberto can drop some weight while maintaining his maximum power output, he will establish a relative power advantage over Esteban. Let’s say Alberto drops down to 70 kg while maintaining his 1,000 watt maximum power output. His relative power output has just increased to 14.2 watts/kg, which means he now has a higher power:weight ratio than Esteban.
So what does this all mean for the McSwiney et al study?
The absolute figures suggest that, had the HC subjects in this study also lost weight, they would have shown the most improvement in relative peak power as well as absolute power.
Remember how the researchers presented the before-and-after changes in terms of watts per kilogram of body weight?
And remember how the keto group began with a significantly higher body fat level?
Well, at the end of the study, body fat levels were similar in the high-carb and keto groups (12.1% versus 12.3%, respectively).
A body fat level of 12% might sound like nirvana to many but it’s definitely on the high side for elite male endurance athletes, among whom single digit body fat levels are commonplace. At 12% body fat, the McSwiney subjects still had considerable room to drop some weight and increase their relative power further.
The best way to drop weight, of course, is to shed body fat while retaining as much lean mass (muscle) as possible.
And this is where we discover yet another unsupportive finding in the McSwiney data. When you use the data they’ve supplied in the paper to calculate the before and after watts per lean kg of body weight, we see a similar situation to what occurred with the absolute power changes.
Expressed in terms of watts per kg of lean mass, average six second speed power marginally increased from 14.7 to 14.9 watts/kg in the high-carb group, but fell from 15.7 to 15.2 in the keto group. Average power in the 3-minute critical power test remained unchanged in the high-carb group, but dropped from 5.7 to 4.7 watts/kg in the keto group.
Which begs the question: What would have happened if both groups leaned out further? As they approached their ideal competitive weight, would the keto group suffer a further decline in power output?
That’s difficult to answer based on the results of this study, because it was a non-randomized endeavor featuring subjects with disparate baseline characteristics. Further alienating the study from the crucial “control your variables” principle is that the two groups strongly appear to have utilized disparate protein intakes and training volumes.
What we can say about this study is that it was not the win for ketogenic diets its authors made it out to be. They selectively presented data suggesting a performance-enhancing effect of a ketogenic diet, while ignoring other metrics that indicated a worsening of performance from that diet.
This was not really a study comparing the effect of two different diets on performance, but the effect of weight loss on selectively cited measures of performance. Weight loss is ultimately a function of calories in versus calories out.
Perhaps the only take home finding is that some (clearly not most) of the LCKD group were able to maintain a high volume training schedule for the duration of the study, achieving weight loss and improving their time in a simulated 100 km TT.
But to infer from this poorly controlled, confounder-prone and attrition-plagued study that LCKD diets improve performance to a greater degree than a properly implemented high-carbohydrate diet is pure nonsense.
See also: My 2015 article on Steven Phinney's infamous and hopelessly flawed keto cyclist study
—
The Mandatory “I Ain’t Your Mama, So Think For Yourself and Take Responsibility for Your Own Actions” Disclaimer: All content is provided for information and education purposes only. Individuals wishing to make changes to their dietary, lifestyle, exercise or medication regimens should do so in conjunction with a competent, knowledgeable and empathetic medical professional. Anyone who chooses to apply the information on this substack does so of their own volition and their own risk. The author/s accept no responsibility or liability whatsoever for any harm, real or imagined, from the use or dissemination of information contained on this substack. If these conditions are not agreeable to the reader, he/she is advised to leave this substack immediately.
Leave a Reply