Abstract: Background: Annual and/or semiannual evaluations of pain medicine clinical faculty are mandatory for multiple organizations in the United States. We evaluated the validity and psychometric reliability of a modified version of de Oliveira Filho et al clinical supervision scale for this purpose.
Methods: Six years of weekly evaluations of pain medicine clinical faculty by resident physicians and pain medicine fellows were studied. A 1-4 rating (4 = "Always") was assigned to each of 9 items (eg, "The faculty discussed with me the management of patients before starting a procedure or new therapy and accepted my suggestions, when appropriate").
Results: Cronbach α of the 9 items equaled .975 (95% confidence interval [CI], 0.974-0.976). A G coefficient of 0.90 would be expected with 18 raters; the N = 12 six-month periods had mean 18.8 ± 5.9 (standard deviation [SD]) unique raters in each period (median = 20).Concurrent validity was shown by Kendall τb = 0.45 (P < .0001) pairwise by combination of ratee and rater between the average supervision score and the average score on a 21-item evaluation completed by fellows in pain medicine. Concurrent validity also was shown by τb = 0.36 (P = .0002) pairwise by combination of ratee and rater between the average pain medicine supervision score and the average operating room supervision score completed by anesthesiology residents.Average supervision scores differed markedly among the 113 raters (η = 0.485; CI, 0.447-0.490). Pairings of ratee and rater were nonrandom (Cramér V = 0.349; CI, 0.252-0.446).Mixed effects logistic regression was performed with rater leniency as covariates and the dependent variable being an average score equaling the maximum 4 vs <4. There were 3 of 13 ratees with significantly more averages <4 than the other ratees, based on P < .01 criterion; that is, their supervision was reliably rated as below average. There were 3 of 13 different ratees who provided supervision reliably rated as above average.Raters did not report higher supervision scores when they had the opportunity to perform more interventional pain procedures.
Conclusions: Evaluations of pain medicine clinical faculty are required. As found when used for evaluating operating room anesthesiologists, a supervision scale has excellent internal consistency, achievable reliability using 1-year periods of data, concurrent validity with other ratings, and the ability to differentiate among ratees. However, to be reliable, routinely collected supervision scores must be adjusted for rater leniency.
References: CMS. Centers for Medicare & Medicaid Services (CMS) requirements for hospital medical staff privileging. S&C-05-04. Available at: https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/SurveyCertificationGenInfo/Downloads/SCletter05-04.pdf. Accessed December 9, 2019.
FPPE/OPPE. The Joint Commission. Standards BoosterPak for Focused Professional Practice Evaluation/Ongoing Professional Practice Evaluation (FPPE/OPPE). Available at: http://www.mc.vanderbilt.edu/documents/CAPNAH/files/Forms/Competency%20Evaluation%20Forms/TJC%20Booster%20Pack%20FPPE-OPPE.pdf. Accessed December 9, 2019.
ACGME. Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements (Residency). Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2019.pdf#page=28. Accessed December 9, 2019.
University of Iowa HealthCare. Carver College of Medicine, Office of Faculty Affairs and Development, Annual review of tenured faculty. Available at: https://medicine.uiowa.edu/facultyaffairs/faculty/annual-review-tenured-faculty. Accessed December 9, 2019.
de Oliveira Filho GR, Dal Mago AJ, Garcia JH, Goldschmidt RAn instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg. 2008;107:1316–1322.
Hindman BJ, Dexter F, Kreiter CD, Wachtel REDeterminants, associations, and psychometric properties of resident assessments of anesthesiologist operating room supervision. Anesth Analg. 2013;116:1342–1351.
van der Meulen MW, Smirnova A, Heeneman S, et al.Exploring validity evidence associated with questionnaire-based tools for assessing the professional performance of physicians: a systematic review. Acad Med. 2019;94:1384–1397.
Copeland HL, Hewson MClinical teaching effectiveness instrument: development and psychometric testing. Available at: https://files.eric.ed.gov/fulltext/ED426656.pdf. Accessed December 8, 2019.
Pearson ACS, Dexter FObservational study of the distribution and diversity of interventional pain procedures among hospitals in the state of Iowa. Pain Physician. 2019;22:E157–E170.
Shanthanna H, Bhatia A, Radhakrishna M, et al.Interventional pain management for chronic pain: a survey of physicians in Canada. Can J Anaesth. 2020;67:343–352.
Epstein RH, Dexter F, Pearson ACSPain medicine board certification status among physicians performing interventional pain procedures in the state of Florida between 2010 and 2016. Pain Physician. 2020;23:E7–E18.
van der Hem-Stokroos HH, van der Vleuten CP, Daelmans HE, Haarman HJ, Scherpbier AJReliability of the clinical teaching effectiveness instrument. Med Educ. 2005;39:904–910.
Zuberi RW, Bordage G, Norman GRValidation of the SETOC instrument – student evaluation of teaching in outpatient clinics. Adv Health Sci Educ Theory Pract. 2007;12:55–69.
Zibrowski EM, Myers K, Norman G, Goldszmidt MARelying on others’ reliability: challenges in clinical teaching assessment. Teach Learn Med. 2011;23:21–27.
Pearson ACS, Dexter F, Epstein RHHeterogeneity among hospitals in the percentages of all lumbosacral epidural steroid injections where the patient had received 4 or more in the previous year. Anesth Analg. 2019;129:493–499.
Dexter F, Ledolter J, Smith TC, Griffiths D, Hindman BJInfluence of provider type (nurse anesthetist or resident physician), staff assignments, and other covariates on daily evaluations of anesthesiologists’ quality of supervision. Anesth Analg. 2014;119:670–678.
Inter-Rater or Inter-Observer Reliability. Types of reliability. Available at: http://www.socialresearchmethods.net/kb/reltypes.php. Accessed October 10, 2019.
Feldt LS, Woodruff DJ, Salih FAStatistical inference for coefficient alpha. Appl Psychol Meas. 1987;11:93–103.
PSYCTC. Confidence intervals for a sample Cronbach coefficient alpha value. Available at: https://www.psyctc.org/stats/R/Feldt1.html. Accessed October 10, 2019.
Wikipedia. Cronbach’s alpha, internal consistency. Available at: https://en.wikipedia.org/wiki/Cronbach%27s_alpha#Internal_consistency. Accessed December 8, 2019.
Dexter F, Szeluga D, Masursky D, Hindman BJWritten comments made by anesthesia residents when providing below average scores for the supervision provided by the faculty anesthesiologist. Anesth Analg. 2016;122:1999–2005.
Dexter F, Ledolter J, Hindman BJMeasurement of faculty anesthesiologists’ quality of clinical supervision has greater reliability when controlling for the leniency of the rating anesthesia resident: a retrospective cohort study. Can J Anaesth. 2017;64:643–655.
Cohen JStatistical Power Analysis for the Behavioral Sciences. 1988:2nd ed. Hillsdale, NJ: Erlbaum; 77–80, 223–225, and 281–287.
Wikipedia. Effect size eta-squared (η2). Available at: https://en.wikipedia.org/wiki/Effect_size#Eta-squared_(%CE%B72). Accessed December 8, 2019.
Bayman EO, Dexter F, Ledolter JMixed effects logistic regression modeling of daily evaluations of nurse anesthetists’ work habits adjusting for leniency of the rating anesthesiologists. PCORM. 2017;6:14–19.
Dexter F, Bayman EO, Wong CA, Hindman BJReliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scores. J Clin Anesth. 2020;61:109639.
Wikipedia. Family-wise error rate. Available at: https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini%E2%80%93Hochberg_procedure. Accessed December 7, 2019.
Dexter F, Hindman BJQuality of supervision as an independent contributor to an anesthesiologist’s individual clinical value. Anesth Analg. 2015;121:507–513.
Dexter F, Masursky D, Hindman BJReliability and validity of the anesthesiologist supervision instrument when certified registered nurse anesthetists provide scores. Anesth Analg. 2015;120:214–219.
Dexter F, Szeluga D, Hindman BJContent analysis of resident evaluations of faculty anesthesiologists: supervision encompasses some attributes of the professionalism core competency. Can J Anaesth. 2017;64:506–512.
De Oliveira GS Jr, Dexter F, Bialek JM, McCarthy RJReliability and validity of assessing subspecialty level of faculty anesthesiologists’ supervision of anesthesiology residents. Anesth Analg. 2015;120:209–213.
Dexter F, Ledolter J, Hindman BJBernoulli Cumulative Sum (CUSUM) control charts for monitoring of anesthesiologists’ performance in supervising anesthesia residents and nurse anesthetists. Anesth Analg. 2014;119:679–685.
Processing Request
No Comments.