The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach

The GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) is a method of assessing the certainty in evidence (also known as quality of evidence or confidence in effect estimates) and the strength of recommendations in health care.^[1] It provides a structured and transparent evaluation of the importance of outcomes of alternative management strategies, acknowledgment of patients and the public values and preferences, and comprehensive criteria for downgrading and upgrading certainty in evidence. It has important implications for those summarizing evidence for systematic reviews, health technology assessments, and clinical practice guidelines as well as other decision makers.^[2]

Background and history

Judgments about evidence and recommendations in healthcare are complex. Healthcare evidence and subsequent recommendations leave decision makers with differing degrees of certainty in that evidence. Sources of evidence range from small laboratory studies or case reports to well-designed large randomized controlled trials that have minimized bias to a great extent. The GRADE began in the year 2000 as a collaboration of methodologists, guideline developers, biostatisticians, clinicians, public health scientists and other interested members. GRADE developed and implemented a common, transparent and sensible approach to grading the quality of evidence (also known as certainty in evidence or confidence in effect estimates) and strength of recommendations in healthcare^[3] The GRADE approach separates recommendations following from an evaluation of the evidence as strong or weak. A recommendations to use, or not use an option (e.g. an intervention), should be based on the trade-offs between desirable consequences of following a recommendation on the one hand, and undesirable consequences on the other (Table 2). If desirable consequences outweigh undesirable consequences, decision makers will recommend an option and vice versa. The uncertainty associated with the trade-off between the desirable and undesirable consequences will determine the strength of recommendations.^[4] The criteria that determine this balance of consequences are listed in Table 2. Furthermore, it provides decision-makers (e.g. clinicians, other health care providers, patients and policy makers) with a guide to using those recommendations in clinical practice, public health and policy. To achieve simplicity, the GRADE approach classifies the quality of evidence in one of four levels—high, moderate, low, and very low (Table 1).^[5]

Table 1. Quality of evidence and definitions

High	We are very confident that the true effect lies close to that of the estimate of the effect
Moderate	We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different
Low	Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect
Very low	We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

(modified from Balshem et. al.)

The GRADE working group has developed a software application that facilitates the use of the approach, allows the development of summary tables and contains the GRADE handbook. The software is free for non-profit organizations and is available at http://www.gradepro.org The GRADE approach to assess the certainty in evidence is widely applicable, including to questions about diagnosis,^[6][7] prognosis,^[8][9] network meta-analysis^[10] and public health.^[11]

Current practice

Over 90 organizations (including the World Health Organization, the UK National Institute for Health and Care Excellence (NICE), and the Canadian Task Force for Preventive Health Care) have endorsed and/or are using GRADE to evaluate the quality of evidence and strength of health care recommendations.

Table 2. Factors and criteria that determine the direction and strength of a recommendation

Factor and criteria*	How the factor influences the direction and strength of a recommendation
Problem This factor can be integrated with the balance of the benefits and harms and burden.	The problem is determined by the importance and frequency of the health care issue that is addressed (burden of disease, prevalence or baseline risk). If the problem is of great importance a strong recommendation is more likely.
Values and preferences	This describes how important health outcomes are to those affected, how variable they are and if there is uncertainty about this. The less variability or uncertainty there is about values and preferences for the critical or important outcomes, the more likely is a strong recommendation.
Quality of the evidence	The confidence in any estimate of the criteria determining the direction and strength of the recommendation will determine if a strong or conditional recommendation is offered. However, the overall quality that is assigned to the recommendation is that of the evidence about effects on population-important outcomes. The higher the quality of evidence the more likely is a strong recommendation.
Benefits and harms and burden	This requires an evaluation of the absolute effects of both the benefits and harms and their importance. The greater the net benefit or net harm the more likely is a strong recommendation for or against the option.
Resource implications	This describes how resource intense an option is, if it is cost-effective and if there is incremental benefit. The more advantageous or clearly disadvantageous these resource implications are the more likely is a strong recommendation.
Equity This factor is often addressed under values preferences, and frequently also includes resource considerations	The greater the likelihood to reduce inequities or increase equity and the more accessible an option is, the more likely is a strong recommendation.
Acceptability This factor can be integrated with the balance of the benefits and harms and burden.	The greater the acceptability of an option to all or most stakeholders, the more likely is a strong recommendation.
Feasibility This factor includes considerations about values and preferences, and resource implications.	The greater the acceptability of an option to all or most stakeholders, the more likely is a strong recommendation.

• Factors for which overlap is described are often not shown separately in a decision table.

References

1. Schünemann, HJ; Best, D; Vist, G; Oxman, AD (2003). "Letters, numbers, symbols, and words: How best to communicate grades of evidence and recommendations?". Canadian Medical Association Journal. 169 (7): 677–80.
2. Guyatt, GH; Oxman, AD; Vist, GE; Kunz, R; Falck-Ytter, Y; Alonso-Coello, P; Schünemann, HJ (2008). "GRADE: an emerging consensus on rating quality of evidence and strength of recommendation". BMJ. 336: 924–26. doi:10.1136/bmj.39489.470347.ad.
3. Guyatt, GH; Oxman, AD; Schünemann, HJ; Tugwell, P; Knotterus, A. "GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology". Journal of Clinical Epidemiology. 64: 380–382. doi:10.1016/j.jclinepi.2010.09.011.
4. Andrews, J; Guyatt, GH; Oxman, AD; Alderson, P; Dahm, P; Falck-Ytter, Y; Nasser, M; Meerpohl, J; Post, PN; Kunz, R; Brozek, J; Vist, G; Rind, D; Akl, EA; Schünemann, HJ. "GRADE guidelines: 15. Going from evidence to recommendations: the significance and presentation of recommendations". Journal of Clinical Epidemiology. 66: 719–725. doi:10.1016/j.jclinepi.2012.03.013.
5. Balshem, H; Helfand, M; Schünemann, HJ; Oxman, AD; Kunz, R; Brozek, J; Vist, GE; Falck-Ytter, Y; Meerpohl, J; Norris, S; Guyatt, GH (April 2011). "GRADE guidelines 3: rating the quality of evidence - introduction". Journal of Clinical Epidemiology. 64: 401–406. doi:10.1016/j.jclinepi.2010.07.015. PMID 21208779.
6. Schünemann, HJ; Oxman, AD; Brozek, J; Glasziou, P; Jaeschke, R; Vist, G; Williams, J; Kunz, R; Craig, J; Montori, V; Bossuyt, P; Guyatt, GH (2008). "GRADEing the quality of evidence and strength of recommendations for diagnostic tests and strategies". BMJ. 336: 1106–1110. doi:10.1136/bmj.39500.677199.ae.
7. Brozek, JL; Akl, EA; Jaeschke, R; Lang, DM; Bossuyt, P; Glasziou, P; Helfand, M; Ueffing, E; Alonso-Coello, P; Meerpohl, J; Phillips, B; Horvath, AR; Bousquet, J; Guyatt, GH; Schünemann, HJ (2009). "Grading quality of evidence and strength of recommendations in clinical practice guidelines: part 2 of 3. The GRADE approach to grading quality of evidence about diagnostic tests and strategies". Allergy. 64 (8): 1109–16. doi:10.1111/j.1398-9995.2009.02083.x.
8. Iorio, A; Spencer, FA; Falavigna, M; Alba, C; Lang, E; Burnand, B; McGinn, T; Hayden, J; Williams, K; Shea, B; Wolff, R; Kujpers, T; Perel, P; Vandvik, PO; Glasziou, P; Schünemann, H; Guyatt, G (2015). "Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients". BMJ. 350: h870. doi:10.1136/bmj.h870.
9. Spencer, FA; Iorio, A; You, J; Murad, MH; Schünemann, HJ; Vandvik, PO; Crowther, MA; Pottie, K; Lang, ES; Meerpohl, JJ; Falck-Ytter, Y; Alonso-Coello, P; Guyatt, GH (2012). "Uncertainties in baseline risk estimates and confidence in treatment effects". BMJ. 14: 345. doi:10.1136/bmj.e7401.
10. Puhan, MA; Schünemann, HJ; Murad, MH; Li, T; Brignardello-Petersen, R; Singh, JA; Kessels, AG; Guyatt, GH (2014). "A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis". BMJ. 24: 349. doi:10.1136/bmj.g5630.
11. Burford, BJ; Rehfuess, E; Schünemann, HJ; Akl, EA; Waters, E; Armstrong, R; Thomson, H; Doyle, J; Pettman, T (2012). "Assessing evidence in public health: the added value of GRADE". J Public Health. 34 (4): 631–5. doi:10.1093/pubmed/fds092.