Scoring Key
1. Scientific accuracy
The primary reviewer will identify three key claims that are the most central to the book’s thesis and evaluate them scientifically. This may involve some searching/reading in the scientific literature, depending on the reviewer’s expertise. Reviewers are encouraged not to rely entirely on references provided by the author, but to perform independent literature searches.
To each of these three claims, the reviewer will apply the following three criteria:
Criterion 1.1: How well is the claim supported by current evidence?
| Score | Summary | Definition |
| 0 | Claim is opposed by current evidence | Overall, relevant evidence is intrinsically convincing and predominantly undermines the author’s claim |
| 1 | Evidence relevant to claim is neutral, nonexistent, or based on personal observations | Overall, relevant evidence neither supports nor undermines the author’s claim. Or, the reviewer is unable to locate evidence in the book or elsewhere that meaningfully evaluates the author’s claim. Or, evidence for the claim relies overwhelmingly on personal observations such as anecdotes about the author’s patients/clients. |
| 2 | Claim is weakly supported by current evidence | Overall, relevant evidence is intrinsically weakly convincing but is consistent with the author’s claim. Or, relevant evidence is intrinsically convincing but only weakly supports the author’s claim. |
| 3 | Claim is moderately supported by current evidence | Overall, relevant evidence is intrinsically moderately convincing and is consistent with the author’s claim. Or, relevant evidence is intrinsically convincing and moderately supports the author’s claim |
| 4 | Claim is strongly supported by current evidence | Overall, relevant evidence is both intrinsically convincing and predominantly supports the author’s claim |
Relevant evidence includes peer-reviewed scientific journal articles; college-level textbooks; government publications such as those issued by the CDC, USDA, or FDA; neutral NGOs such as the Cochrane Collaboration; and other similarly credible sources. Relevant evidence can include citations provided in the book as well as citations the reviewer identifies independently. Personal observations are not relevant evidence, including the clinical observations of medical practitioners. Personal observations can be used to illustrate claims that are supported by other evidence, but not as primary support for claims.
Criterion 1.2: Are the references cited in the book to support the claim convincing?
| Score | Summary | Definition |
| 0 | References undermine claims | References are intrinsically convincing and predominantly undermine the author’s claim |
| 1 | References are not convincing | References offer evidence that is intrinsically unconvincing but is consistent with the author’s claim. Or, references offer evidence that is irrelevant. Or, references offer evidence that is intrinsically convincing but does not support the author’s claim. |
| 2 | References are weakly convincing | References offer evidence that is intrinsically weakly convincing but is consistent with the author’s claim. Or, references offer evidence that is intrinsically convincing but only weakly supports the author’s claim. |
| 3 | References are moderately convincing | References offer evidence that is intrinsically moderately convincing and is consistent with the author’s claim. Or, references offer evidence that is intrinsically convincing and moderately supports the author’s claim. |
| 4 | References are very convincing | References offer evidence that is both intrinsically convincing and predominantly supports the author’s claim |
Criterion 1.3: How well does the strength of the claim line up with the strength of the evidence?
| Score | Summary | Definition |
| 0 | Strength of the claim is greatly overstated | The evidence suggests very small and/or highly uncertain effects relative to the author’s claim |
| 1 | Strength of the claim is substantially overstated | The evidence suggests substantially smaller and/or substantially more uncertain effects relative to the author’s claim |
| 2 | Strength of the claim is moderately overstated | The evidence suggests moderately smaller and/or moderately more uncertain effects relative to the author’s claim |
| 3 | Strength of the claim is slightly overstated | The evidence suggests slightly smaller and/or slightly more uncertain effects relative to the author’s claim |
| 4 | Strength of the claim aligns well with strength of the evidence | The evidence suggests that the author’s claim accurately represents the size and uncertainty of the effect |
Each of the three claims receives one 0-4 score per criterion, for a total of nine scores, which are recorded separately. All scores are then averaged together to yield a single score for scientific accuracy.
2. Reference accuracy
The primary reviewer will randomly select ten references from the book, identify the claim associated with each reference, and evaluate how well each reference supports its claim. If applicable, the reviewer will identify the page range of the book that contains the reference section (the portion that is tied to specific statements). To select each reference, he/she will select a page from within that range using a random number generator, then number the references on that page and select one using a random number generator (Random.org is recommended).
If the reference format of the book does not permit this method, the reviewer will adapt the method to the reference format of the book such that references are selected at random. For example, if references are at the end of each chapter, the reviewer will randomly select a chapter, then randomly select a reference from the reference list at the end of that chapter.
Then the reviewer will randomly select ten references in the book and apply the following criterion:
Criterion 2.1: Does the reference support the claim?
| Score | Summary | Definition |
| 0 | Reference undermines the claim | Reference offers evidence that is intrinsically convincing and predominantly undermines the author’s claim |
| 1 | Reference does not convincingly support the claim | Reference offers evidence that is intrinsically unconvincing but is consistent with the author’s claim. Or, reference offers evidence that is irrelevant. Or, reference offers evidence that is intrinsically convincing but neither predominantly supports nor refutes the author’s claim. |
| 2 | Reference offers weak support for the claim | Reference offers evidence that is intrinsically weakly convincing but is consistent with the author’s claim. Or, reference offers evidence that is intrinsically convincing but only weakly supports the author’s claim. |
| 3 | Reference offers moderate support for the claim | Reference offers evidence that is intrinsically moderately convincing and is consistent with the author’s claim. Or, reference offers evidence that is intrinsically convincing and moderately supports the author’s claim. |
| 4 | Reference offers strong support for the claim | Reference offers evidence that is both intrinsically convincing and predominantly supports the author’s claim |
Each of the ten references receives a 0-4 score, and these are recorded separately in the scoring document. All scores are then averaged together to yield a single score for reference accuracy. If the book contains fewer than ten references, the reviewer will take the average score of the references that he/she scored (rather than averaging in 0s for missing references). If the book contains no references, this scoring measure will be left blank.
3. Healthfulness
The primary reviewer will identify the health-related intervention proposed by the author and record a summary of it in his/her scoring document. The primary reviewer will then identify and record the condition targeted by the book, if applicable (for example, type 2 diabetes, excess body fat, or cardiovascular disease). He/she will then identify the audience that appears to be targeted by the book (for example, a general audience, people who want to lose weight, or the elderly).
With the book’s intervention, target condition, and target audience in mind, the reviewer will apply the following three criteria:
Criterion 3.1: Is the intervention likely to improve the target condition in the target audience, relative to typical diet and/or lifestyle patterns, in the medium-to-long-term (6+ months)?
| Score | Summary | Definition |
| 0 | Intervention is likely to worsen the condition | Relevant evidence suggests that the intervention proposed by the author is likely to worsen the target condition in the medium-to-long-term (6+ months) |
| 1 | Intervention is likely neutral for the condition, or effects are unknown | Relevant evidence suggests that the intervention proposed by the author is likely neutral for the target condition in the medium-to-long-term (6+ months). Or, relevant evidence is unavailable or insufficient to meaningfully evaluate the healthfulness of the intervention for the target condition in the medium-to-long-term (6+ months). |
| 2 | Intervention is likely to slightly improve the condition | Relevant evidence suggests that the intervention proposed by the author is likely to slightly improve the target condition in the medium-to-long-term (6+ months) |
| 3 | Intervention is likely to moderately improve the condition | Relevant evidence suggests that the intervention proposed by the author is likely to moderately improve the target condition in the medium-to-long-term (6+ months) |
| 4 | Intervention is likely to greatly improve the condition | Relevant evidence suggests that the intervention proposed by the author is likely to greatly improve the target condition in the medium-to-long-term (6+ months) |
Relevant evidence includes peer-reviewed scientific journal articles; college-level textbooks; government publications such as those issued by the CDC, USDA, or FDA; neutral NGOs such as the Cochrane Collaboration; and other similarly credible sources. Relevant evidence can include citations provided in the book as well as citations the reviewer identifies independently. Personal observations are not relevant evidence, including the clinical observations of medical practitioners. Personal observations can be used to illustrate claims that are supported by other evidence, but not as primary support for claims.
Criterion 3.2: Is the intervention likely to improve general health in the target audience, relative to typical diet and/or lifestyle patterns, in the medium-to-long-term (6+ months)?
| Score | Summary | Definition |
| 0 | Intervention is likely to worsen general health | Relevant evidence suggests that the intervention proposed by the author is likely to worsen general health in the medium-to-long-term (6+ months) |
| 1 | Intervention is likely neutral for general health, or effects are unknown | Relevant evidence suggests that the intervention proposed by the author is likely neutral for general health in the medium-to-long-term (6+ months). Or, relevant evidence is unavailable or insufficient to meaningfully evaluate the impact on general health in the medium-to-long-term (6+ months). |
| 2 | Intervention is likely to slightly improve general health | Relevant evidence suggests that the intervention proposed by the author is likely to slightly improve general health in the medium-to-long-term (6+ months) |
| 3 | Intervention is likely to moderately improve general health | Relevant evidence suggests that the intervention proposed by the author is likely to moderately improve general health in the medium-to-long-term (6+ months) |
| 4 | Intervention is likely to greatly improve general health | Relevant evidence suggests that the intervention proposed by the author is likely to greatly improve general health in the medium-to-long-term (6+ months) |
The reviewer should consider diet as well as lifestyle guidelines such as physical activity, smoking cessation, and stress management. “General health” is defined broadly as including chronic disease risk, body composition, reproductive health, and physical and cognitive performance. Reviewers are strongly discouraged from basing their answers on opinions that are not based on evidence. If relevant evidence is unavailable or uninformative, the reviewer should assign a score of 2.
Criterion 3.3: Does the diet portion of the intervention promote an adequate nutrient intake for general health in the target audience, relative to prevailing recommendations, in the medium-to-long-term (6+ months)?
| Score | Summary | Definition |
| 0 | Diet is likely substantially nutritionally inadequate | Diet will likely provide moderately insufficient levels of three or more relevant nutrients, and/or dangerously insufficient levels of 1-2 relevant nutrients, in the target audience over the medium-to-long-term (6+ months) |
| 1 | Diet is likely somewhat nutritionally inadequate | Diet will likely provide moderately insufficient levels of 1-2 relevant nutrients in the target audience over the medium-to-long-term (6+ months) |
| 2 | Diet is likely nutritionally adequate | Diet will likely provide adequate levels of all relevant nutrients in the target audience, but not a level of nonessential health-promoting nutrients that is superior to typical diets over the medium-to-long-term (6+ months) |
| 3 | Diet is likely more than nutritionally adequate | Diet will likely provide adequate levels of all relevant nutrients in the target audience, and a level of nonessential health-promoting nutrients that is somewhat superior to typical diets over the medium-to-long-term (6+ months) |
| 4 | Diet is likely substantially more than nutritionally adequate | Diet will likely provide adequate levels of all relevant nutrients in the target audience, and a level of nonessential health-promoting nutrients that is substantially superior to typical diets over the medium-to-long-term (6+ months) |
Relevant nutrients include 1) essential micronutrients such as vitamins and minerals, 2) protein quantity and quality, and 3) nonessential but health-promoting nutrients such as fiber and polyphenols. The reviewer will use Institute of Medicine Estimated Average Requirements (EAR) as the benchmark for adequate essential nutrient intakes, except carbohydrate because it is not an essential nutrient. We recognize that some diet interventions will be difficult to evaluate for many essential nutrients. Accordingly, the reviewer may judge the risk of specific nutrient insufficiency based on the general diet pattern rather than a detailed analysis of all nutrients in the proposed diet. Typically, diets that promote a minimally refined, diverse eating pattern should score well on criterion 3.3, but this will not necessarily be the case for all combinations of diet and target population.
4. Difficulty
Difficulty is a supplemental measure that will not receive a numerical rating and will not be averaged into the overall score. Rather than representing difficulty with a star value, it will be described verbally underneath the other ratings using the terms “very difficult”, “fairly difficult”, “fairly easy”, and “very easy.” For example: “Difficulty: Fairly easy.”
| Summary | Definition |
| Very difficult | Intervention proposed by the author would be very difficult for the target audience to implement. |
| Fairly difficult | Intervention proposed by the author would be fairly difficult for the target audience to implement. |
| Fairly easy | Intervention proposed by the author would be fairly easy for the target audience to implement. |
| Very easy | Intervention proposed by the author would be very easy for the target audience to implement. |
The primary reviewer will identify the audience targeted by the author and the intervention proposed by the author, and record a summary of them in his/her scoring document. The reviewer will then determine how difficult it would be for the target audience to follow the author’s advice. Difficulty is based on the following five considerations:
- How much discipline the intervention requires. For example, a diet that requires a person to regularly refuse tempting food in her personal surroundings involves a high level of discipline.
- How much time the intervention requires. For example, a diet that requires a person to shop at a farmer’s market and cook complex recipes from scratch involves a large amount of time.
- How much effort the intervention requires. For example, a diet that is very complex requires a high level of effort.
- How much money the intervention requires. For example, a diet that is heavy in grass-fed meat, exotic ingredients, and/or organic nuts can be expensive.
- How difficult it is to obtain the required foods. For example, all-organic foods or free-range meats may not be readily available in some areas.