Do the National Board of Medical Examiners (NBME) subject examinations predict success on the MCCQE Part I? Author: Dr. Carolina Escudero Date: September 24, 2011
Introduction • Evaluation of medical students is an important but challenging process • Essential to identify students who are not meeting the requisite minimum level of knowledge and clinical competence • Many different tools have been used in the evaluation process • several tools shown to be poor at identifying students who are below average (Colleti, 2000; Speer et al., 2000)
2
Introduction • National Board of Medical Examiners (NBME) subject examinations • Discipline-based, standardized multiple choice tests • Provide normally distributed scores and have correlated with overall clerkship performance (Nahum, 2004) • Security of the examination • Widely used in clerkship evaluation - In 2001, 67% of medical schools in Canada and US used the NBME subject examination in pediatrics (Kumar et al., 2004)
3
Introduction • NBME subject examinations are available in the following 6 subjects: • • • • • •
Pediatrics Internal medicine Surgery Obstetrics/gynecology Psychiatry Family Medicine
• Memorial University uses 5 NBME examinations to evaluate students during their clerkship rotations
4
U. of Ottawa NOSM U. of Saskatchewan U. of Calgary
No NBME exams used
≥ 5 NBME exams used
Some NBME exams used
MUN Queen’ s U. U. of Manitoba UBC
Dalhousie U. McGill U. U. of Western Ont. U. of Alberta McMaster U. U. of Toronto
NBME subject examination use by English-speaking Canadian medical schools * From personal correspondence in April, 2011 5
NBME and Licensing Examinations • NBME exams correlate well with scores on the USMLE Step 2 • Correlations ranging from r = 0.53 - 0.74 (Ripkey et al., 1999; Myles et al., 2002) • Suggested that low scores on the NBME could predict students at risk for low scores on the USMLE Step 2
• Canadian licensing examination (MCCQE Part I) • No previously published research on using the NBME to predict MCCQE Part I performance despite wide uptake in Canada
6
Purpose • To compare Memorial University medical students’ scores on the NBME subject examinations to their scores on the MCCQE Part I • To help determine if NBME scores can identify students at risk of MCCQE Part I failure
7
Methods • Obtained NBME scores from 4 academic years • Total MCCQE Part I, MCQ, CDM, and disciplinespecific MCQ scores collected • To be involved in the analysis, NBME and the MCCQE scores were required • Only the results of the first attempt for each examination were included • Pearson’s correlation, chi square, and ROC analysis were performed
8
Confidentiality • Student names and randomly generated numerical identifiers sent to the Medical Council of Canada (MCC) • MCC matched MCCQE Part I scores to name and numerical code • Student names removed and the remainder sent the Canadian Post-MD Education Registry (CAPER) • List of identifying numbers and NBME scores sent to CAPER • CAPER matched the MCCQE Part I scores and the NBME scores • After matching, the numerical codes were removed and the de-identified list was sent to the authors • Student confidentiality was ensured as reverse deidentification could not be performed
9
Results • 224 student scores were analyzed • Mean NBME scores: • • • • •
Pediatrics: Internal Medicine: Surgery: Obstetrics/Gynecology: Psychiatry:
73.8 75.9 70.9 73.6 78.0
(SD (SD (SD (SD (SD
7.0) 6.7) 7.5) 6.6) 7.0)
• Mean MCCQE Part 1 score: 522.8 (SD 63)
10
Results • Correlations between the examinations • NBME exams and total MCCQE Part I score (r=0.5190.613, p=0.000) • NBME exams and the MCCQE part I MCQ score (r=0.5060.605, p=0.000) - Similar to those reported between the NBME exams and the USMLE Part II (0.53-0.74)
• NBME exams and their respective subject-specific MCQ scores (r=0.333-0.489, p=0.000)
11
Results • 6 students failed the MCCQE Part 1 • 3 passed all of the NBME exams • 3 failed 2 or more examinations
• 12 students failed at least one NBME examination (score <60) • 25% of these students failed the MCCQE Part 1 - 1 NBME exam failed: 0% - 2 or more NBME exams failed: 75%
• Failing at least 1 NBME exam gives an odds ratio of failing the MCCQE Part I of 23.2 (95% CI = 4.1-131.5)
Results • ROC Analysis • To determine the sensitivity and specificity of NBME threshold scores in predicting MCCQE Part 1 failure
ROC Curve Comparing Average NBME Score to MCCQE Part 1 Performance Average NBME score
ROC Curve Comparing Average NBME Score to MCCQE Part 1 Performance Average NBME score of ≤70 gives 100% sensitivity and 76% specificity Average NBME score of ≤69 gives 83% sensitivity and 83% specificity Average NBME score of ≤68 gives 67% sensitivity and 90% specificity
NBME Exam
Area under the Curve
P value
Overall Average
0.922
0.000
Pediatrics
0.932
0.000
Internal Med.
0.898
0.001
Surgery
0.886
0.001
Obs/Gyne
0.830
0.006
Psychiatry
0.819
0.008
Results • ROC Analysis for average NBME score • Average NBME score of <70 is a good predictor of students at risk of MCCQE Part 1 failure
• Students with an average NBME score <70 are significantly more likely to fail the MCCQE Part 1 (χ2 = 19.9, p = 0.000)
Overall Findings • NBME subject examinations show moderate correlations with MCCQE Part I scores • Failure of at least one of the NBME subject examinations is associated with an increased risk of MCCQE Part I failure • When looking at the average score on the NBME examinations, a cut-off score of <70 gives a good sensitivity and specificity for identifying students at increased risk of MCCQE Part I failure
Study Limitations • Results from only one medical school • Were not able to correct for the time at which students wrote the examination during their clerkship year • Studies show improvement in scores when NBME written later in the clerkship year (Reteguiz et al, 2002; Manley et al, 2006)
• Failure of either the NBME subject examinations or the MCCQE Part I is an uncommon event • Small number of students who failed the MCCQE Part I, it is difficult to draw firm conclusions from these results
Acknowledgements • Dr. Anne Drover (Memorial University) • Medical Council of Canada (Dr. Tim Wood) • CAPER
References 1. Nahum, G.G. (2004). Evaluating medical student obstetrics and gynecology clerkship performance: Which assessment tools are most reliable? American Journal of Obstetrics and Gynecology, 191, 1762-71. 2. Colletti, L.M. (2000). Difficulty with negative feedback: face-to-face evaluation of junior medical student clinical performance results in grade inflation. Journal of Surgery Research, 90, 82-7. 3. Speer, A.J., Solomon, D.J., and Fincher, R.M.. (2000). Grade inflation in internal medicine clerkships: results of a national survey. Teaching and Learning in Medicine, 12, 112-6. 4. Reteguiz, J.A. and Crosson, J. (2002). Clerkship order and performance on family medicine and internal medicine national board of medical examiners exams. Family Medicine, 34(8), 604-8. 5. Manley, M. and Heiss, G. (2006). Timing bias in the psychiatry subject examination of the national board of medical examiners. Academic Psychiatry, 30, 116–119. 6. Levine, R.E., Carlson, D.L., Rosenthal, R.H., Clegg, K.A., and Crosby, R.D. (2005). Usage of the national board of medical examiners subject test in psychiatry by U.S. and Canadian clerkships. Academic Psychiatry, 29, 52–57. 7. Barzansky, B., and Etzel, S.I. (2001). Education programs in US medical schools, 2000-2001. Journal of the American Medical Association, 286, 1040-1055. 8. Kumar, A., Gera, R., Shah, G., Godambe, S. and Kallen, D.J. (2004). Student Evaluation Practices in Pediatric Clerkships: A Survey of the Medical Schools in the United States and Canada. Clinical Pediatrics, 43, 729-735.
References 9. Ogunyemi, D. and Taylor-Harris, D. (2005). Factors that correlate with the U.S. Medical Licensure Examination Step-2 scores in a diverse medical student population. Journal of the National Medical Association, 97(9), 1258-62. 10. Myles, T.D. and Henderson, R.C. (2002). Medical licensure examination scores: relationship to obstetrics and gynecology examination scores. Obstetrics and Gynecology, 100(5 Pt 1), 955-8. 11. Ripkey, D.R., Case, S.M, and Swanson, D.B. (1999). Identifying students as risk for poor performance on the USMLE Step 2. Academic Medicine, 74 (10), S45-8. 12. Hemmer, P.A. and Pangaro, L.N. (2002). Natural history of knowledge deficiencies following clerkships. Academic Medicine, 77, 350–3.. 13. Minion, D.J., Donnelly, M.B., Quick, R.C., Pulito, A., and Schwartz, R. (2002). Are multiple objective measures of student performance necessary? The American Journal of Surgery, 183, 663–5. 14. Kuo, A.A and Slavin, S.J. (1999). Clerkship curricular revision based on the Ambulatory Pediatric Association and the Council on Medical Student Education in Pediatrics guidelines: does it make a difference? Pediatrics, 103(4 Part 2), 898-901. 15. Rockney, R.M. and Allister, R.G. (2005). Dropping the shelf examination: does it affect student performance on the United States Medical Licensure Examination Step 2? Ambulatory Pediatrics, 5, 240 243. 16. Potts, M.J. and Phelan, K.W. (1997). A drop in pediatric subject examination scores after curriculum changes that emphasize general pediatric topics. Archives of Pediatrics and Adolescent Medicine, 151 (9), 938-42.
Questions?
Cut-off for 83% Sensitivity
Associated Specificity
Overall Average
≤69
83%
Pediatrics
<65
93%
Internal Med.
<71
78%
Surgery
<66
78%
Obs/Gyne
<70
73%
Psychiatry
<78
55%
NBME Exam
