Grant A Justin,1,2 Mohammed Soleimani,3 Sidra Zafar,4 Kasra Cheraqpour,5 Catherine Green,6 Mohammad Moin,7 N Venkatesh Prajna,8 Karl C Golnik,9 Fasika A Woreta4
1Department of Vitreoretinal Surgery, Duke Eye Center, Durham, NC, USA; 2Department of Surgery, Uniformed Services University of the Health Sciences, Bethesda, MD, USA; 3Department of Ocular Trauma and Emergency, Tehran University of Medical Sciences, Tehran, Iran; 4Wilmer Eye Institute, Johns Hopkins Medical Institute, Baltimore, MD, USA; 5Department of Eye Research, Tehran University of Medical Sciences, Tehran, Iran; 6Glaucoma Unit, Royal Victorian Eye and Ear Hospital, Melbourne, Australia; 7Department of Ophthalmology, Amee ud Din Medical College, Lahore, Pakistan; 8Department of Cornea and Refractive Eye Services, Aravind Eye Hospital, Madurai, India; 9Department of Ophthalmology, University of Cincinnati, The Cincinnati Eye Institute, Cincinnati, OH, USA
Correspondence: Grant A Justin, Department of Vitreoretinal Surgery, Duke Eye Center, Durham, NC, USA, Tel +1 609-468-2468, Email grant.a.ju[email protected]
Background/Aims: To produce an internationally developed rubric to assess surgical competency in open globe management.
Methods: An international expert panel of seven ophthalmologist educators developed a standardized competency-based rubric. The steps to perform an open globe repair were outlined. Based on a modified Dreyfus model, the experts agreed on the steps of surgery and certain global indices. They then assigned descriptors for the competency expected of a novice, beginner, advanced beginner, and competent surgeon. The tool was then vetted by another panel of ten ophthalmologists. The main outcome measure was the final version of the tool as agreed upon by the expert review panel.
Results: The steps of open globe repair and key global indices were established. Descriptive wording for each step and global indices for novice, beginner, advanced beginner, and competent surgery were listed. All the expert comments were integrated to establish face and content validity.
Conclusion: This standardized rubric to evaluate resident competency should be used globally in training programs to assess open globe repair surgical skills. Using a modified Dreyfus model, four different levels of training competency were defined allowing a non-biased, objective, numerical and simple assessment.
In 1999, the Accreditation Council for Graduate Medical Education (ACGME) established 6 core competencies.1 In response to the initial struggles by ophthalmology residency programs in the early 2000s to implement these competencies, the American Board of Ophthalmologists established the Program Directors’ Task Force on the Competencies which identified surgical proficiency as an additional competence that should be achieved during ophthalmology training.2,3 At that time the group identified surgical case logs as already implemented, with low faculty burden and having a moderate ability to assess surgical competency, while surgical skills assessment had a high faculty burden, but the greatest ability to evaluate surgical competency.2 However, since then there has been little advancement towards implementation of codified surgical skills assessment in ophthalmology resident training. In part this was due to the lack of well-established grading rubrics. The International Council of Ophthalmology recognized this gap and in the early 2010s began to develop various Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCARs). The ICO-OSCARs facilitate assessment and acquisition of surgical skills and many rubrics have been created ranging from phacoemulsification, strabismus, trabeculectomy, vitrectomy,4–7 and in-office procedures for panretinal photocoagulation and intravitreal injections.8,9 However, no ICO-OSCAR nor another surgical competency rubric for open globe surgical management has been developed.10 The development of OSCARs is now led by the Ophthalmology Foundation, which supports the continuation of this important initiative.
Ocular trauma, including open globe injuries, is a major cause of monocular blindness worldwide.11 In the United States, open globe injuries occur at a frequency of 4.49 per 100,000, with most injuries occurring among those aged 40 or less.12 Early and competent surgical repair to restore structural globe integrity is central in open globe management and the subsequent visual rehabilitation. Further, open globe injuries can have a significant economic impact.13 Currently, the ACGME Review Committee for Ophthalmology deems that graduating residents should achieve at a minimum four primary surgeon globe trauma cases,14 however, the document explicitly states that achievement does not signify competence. Thus, despite these well-established surgical case minimums, there is a critical need for supplemental tools to assist residency program directors in objectively assess the surgical competency of their residents.15–17 The goal of this project was to develop an internationally valid tool to evaluate the progression of a resident ophthalmologist’s surgical competence in performing open globe repair.
This study was approved by the Institutional Review Board of the Johns Hopkins University School of Medicine. All study activities adhered to the tenets of the Declaration of Helsinki. A group of seven content experts from the United States, Iran, Pakistan, India and Australia worked together online to develop an initial rubric draft for open globe repair. These experts represent multiple different subspecialties in ophthalmology. For this study, a rubric was defined as an explicit set of criteria for assessing procedural skill. Open globe repair was divided into nine steps and there were nine additional global indices. Initially, indications for open globe repair, assessment of open globe injury, and pre-operative and post-operative medical management were included, but were later excluded as the rubric’s goal was to score surgical competency in performing open globe repair.
Using a modified Dreyfus model for skill acquisition (novice, beginner, advanced beginner, and competent), each step and global indices was assigned behavioral narrative anchors for each level of skill.18 As per previous studies,7,8 the expert category was omitted as this level of skill is not achieved in training. Further, each category was assigned a numerical value so that scores can be calculated to track improvement. Finally, descriptors were written to reduce bias in assessment, ensure completion of certain steps and reduce any interpretative errors in the criteria.
The rubric was modified repeatedly until there was full consensus on the rubric’s face and content validity. A second group of content experts from India, United States, Australia, China, Singapore, Hungary, and the United Kingdom were then invited to review the draft for constructive feedback. Aiming for global representation, the reviewers were selected for their expertise, diverse ophthalmic subspecialties and experience in teaching open globe repair. Their suggestions were cataloged, reviewed, and incorporated to produce the final rubric.
The steps of open globe repair were broken down into nine key steps (Figure 1). Further, nine global indices demonstrating the surgical safety, fluidity and critical surgical knowledge of instrumentation were developed. A modified Dreyfus model was then applied to each of the 18 steps and indices. Depending on the competency in each of the 18 components between 2 and 5 points could be scored for a total of 90 possible points.
Figure 1 Continued.
Figure 1 OSCAR:OpenGlobe.
The international panel had general and specific suggestions on the initial draft. All expert comments were considered, and the authors incorporated appropriate suggestions, thus establishing a level of face and content validity. Most comments pertained to the addition of new items. These were mostly related to preoperative evaluation, anesthesia considerations, postoperative complications and/or postoperative care. The content experts considered these comments but believed including them would be out of scope for a surgical skill assessment tool. Additionally, including these other parameters would make the rubric too extensive and burdensome to complete, which could potentially discourage uptake of the tool. Thus, the consensus of the experts was to not add these categories. The final draft was called the Ophthalmology Surgical Competency Assessment Rubric for open globe repair (OSCAR:OpenGlobe).
As demonstrated by the OSCAR: OpenGlobe, open globe injuries frequently require complex repair and a diverse range of ophthalmic skills including iris manipulation, muscle disinsertion, and the management of vitreous and lens fragments. These cases can place a practicing ophthalmologist in an unfamiliar position surgically, due to the unpredictability given the various mechanisms and degrees of presentations.19 Hence, having a rubric that allows for accurate grading of surgical management of open globe injuries is paramount in ensuring that residents develop critical sight-saving surgical skills. In the United States, graduating ophthalmology residents are required to complete only a minimum of 4 globe trauma cases as primary surgeon.13 However, these globe trauma cases may include anterior chamber washout and intraocular foreign body removal as well as open globe repair, which means that residents may meet their minimum numbers without necessarily gaining experience in globe repair. From ACGME surgical case logs, the average graduating resident in United States in 2018 performed 7.9 ± 5 open globe repairs as a primary surgeon.20 Further, 3.6% of residents did not meet the requirement, and 9.6% of residents met only the bare minimum of four ocular trauma procedures. Internationally, even more variation exists as many countries do not require a minimum number of surgeries, let alone define standards for competence. Further, simply logging the number of globe repairs performed is insufficient for assessing competence.21 Surgical logs do not demonstrate the quality of the procedure performed nor do they assure competency. Finally, surgical case logs are frequently miscoded making it difficult to assess what procedures were actually performed.22
A recent systematic review of the literature found that there is currently no universally accepted standard competency assessment for open globe repair.10 This assessment tool therefore fills a critical gap and serves three purposes: first, it will decrease subjectivity of the assessment OSCAR:OpenGlobe by defining for the assessor what behavior must be observed for each level of proficiency; second, the rubric clearly communicates to the learner what is expected to attain competence allowing self-directed learning; third, it provides a numerical system that allow for monitoring for improvement by the residency program director. Further, the grading sheet is simple and allows feedback in a timely fashion. Thus, this standardized rubric is a globally validated and standardized platform to allow non-biased objective teaching and evaluation of surgical skills in open globe repair.
In conclusion, the OSCAR:OpenGlobe is a key step towards standardizing training and evaluation in this domain that can be used globally. Its ease of use and the ability to generate assessment for each resident and from multiple evaluators allows it to be a meaningful and reliable tool. Moreover, by including an international panel of experts, we believe the rubric to have content and face validity.
The view(s) expressed herein are those of the author(s) and do not reflect the official policy or position of the Defense Health Agency, The US Army Medical Department, The US Army Office of the Surgeon General, The Department of the Navy, The Department of the Army, Department of Defense, The Uniformed Services University of the Health Sciences or any other agency of the US Government. Mention of trade names, commercial products, or organizations does not imply endorsement by the US Government. The authors would like to thank Drs. Bernando Soares, Christopher Rapuano, Rekha Khandelwal, Elsie Chan, Jacqueline Beltz, Ferenc Kuhn, Marcus Coyler, Gangadhara Sundar, Rupesh Agrawal, and Yip Chee Chew for their review and critical feedback on the OSCAR:OpenGlobe.
There is no funding to report.
The authors report no conflicts of interest in this work.
1. ACGME. The milestone guidebook; 2020. Available from: https://acgme.org/Portals/0/MilestonesGuidebook.pdf.
2. Mills RP, Mannis MJ; American Board of Ophthalmology Program Directors’ Task Force on Competencies. Report of the American Board of Ophthalmology task force on the competencies. Ophthalmology. 2004;111(7):1267–1268. doi:10.1016/j.ophtha.2004.04.003
3. Lee AG, Volpe N. The impact of the new competencies on resident education in ophthalmology. Ophthalmology. 2004;111(7):1269–1270. doi:10.1016/j.ophtha.2004.04.004
4. Golnik C, Beaver H, Gauba V, et al. Development of a new valid, reliable, and internationally applicable assessment tool of residents’ competence in ophthalmic surgery (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2013;111:24–33.
5. Golnik KC, Law JC, Ramasamy K, et al. The ophthalmology surgical competency assessment rubric for vitrectomy. Retina. 2017;37(9):1797–1804. doi:10.1097/IAE.0000000000001455
6. Green CM, Salim S, Edward DP, et al. The ophthalmology surgical competency assessment rubric for trabeculectomy. J Glaucoma. 2017;26(9):805–809. doi:10.1097/IJG.0000000000000723
7. Golnik KC, Motley WW, Atilla H, et al. The ophthalmology surgical competency assessment rubric for strabismus surgery. J AAPOS. 2012;16(4):318–321. doi:10.1016/j.jaapos.2012.04.005
8. Law JC, Golnik KC, Cherney EF, Arevalo JF, Li X, Ramasamy K. The ophthalmology surgical competency assessment rubric for panretinal photocoagulation. Ophthalmol Retina. 2018;2(2):162–167. doi:10.1016/j.oret.2017.06.002
9. Kiew SY, Yeo IYS, Golnik KC, Muriel-Herrero MA, Fuertes-Barahona V, Grzybowski A. The ophthalmology surgical competency assessment rubric for intravitreal injections (ICO-OSCAR: IVI). J Clin Med. 2021;10(7):1476. doi:10.3390/jcm10071476
10. Wood TC, Maqsood S, Nanavaty MA, Rajak S. Validity of scoring systems for the assessment of technical and non-technical skills in ophthalmic surgery-a systematic review. Eye. 2021;35(7):1833–1849. doi:10.1038/s41433-021-01463-8
11. Négrel AD, Thylefors B. The global impact of eye injuries. Ophthalmic Epidemiol. 1998;5(3):143–169. doi:10.1076/opep.184.108.40.20664
12. Mir TA, Canner JK, Zafar S, et al. Characteristics of open globe injuries in the United States from 2006 to 2014. JAMA Ophthalmol. 2020;138(3):268–275. doi:10.1001/jamaophthalmol.2019.5823
13. Lee BWH, Hunter D, Robaei DS, Samarawickrama C. Open globe injuries: epidemiology, visual and surgical predictive variables, prognostic models, and economic cost analysis. Clin Exp Ophthalmol. 2021;49(4):336–346. doi:10.1111/ceo.13944
14. ACGME Review Committee for Ophthalmology. Case log information: ophthalmology. Available from: https://www.acgme.org/Portals/0/PFAssets/ProgramResources/OPH_CaseLogInfo.pdf?ver=2021-03-15-133325-270.
15. Golnik KC, Lee AG, Wilson MC. A national program director survey of the shift to competency-based education in ophthalmology. Ophthalmology. 2008;115(8):1426–1430. doi:10.1016/j.ophtha.2008.01.017
16. Gedde SJ, Volpe NJ, Feuer WJ, Binenbaum G. Ophthalmology resident surgical competence: a survey of program directors. Ophthalmology. 2020;127(8):1123–1125. doi:10.1016/j.ophtha.2020.02.017
17. Bartley GB. Verifying surgical competence: our fiduciary responsibility. Ophthalmology. 2020;127(8):997–999. doi:10.1016/j.ophtha.2020.03.022
18. Dreyfus SE, Dreyfus HL. A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Berkeley: Operations Research Center, University of California; 1980.
19. Blanch RJ, Bishop J, Javidi H, Murray PI. Effect of time to primary repair on final visual outcome after open globe injury. Br J Ophthalmol. 2019;103(10):1491–1494. doi:10.1136/bjophthalmol-2017-311559
20. Abousy M, Schilling A, Qiu M, et al. Ophthalmology resident ophthalmic trauma case exposure: trends over time and an ACGME case log analysis. Clin Ophthalmol. 2022;16:1365–1373. doi:10.2147/OPTH.S359384
21. Stride HP, George BC, Williams RG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surgery. 2018;163(3):488–494. doi:10.1016/j.surg.2017.10.011
22. Qiu M, Boland MV, Woreta FA, Goshe JM. Deficiencies in ophthalmology residents’ case logging of glaucoma surgery. Ophthalmol Glaucoma. 2020;3(3):218–220.