Rx for Better Orthopaedic Surgeons: Track Errors as Well as Skills

    In a small study to determine the best way to assess the operating skills of would-be orthopaedic surgeons, researchers at Johns Hopkins have found that tracking the trainees’ performance on cadavers using step-by-step checklists and measures of general surgical skills works well but should be coupled with an equally rigorous system for tracking errors.

    “The takeaway message is that checklists of procedural steps are a good way to assess the technical skills of these surgical residents,” said Dawn LaPorte, MD, professor of orthopaedic surgery at the Johns Hopkins University School of Medicine and an author the study published by the Journal of Surgical Education.

    “But they don’t measure quality, highlighting a need to measure and give feedback on errors as part of the training,”

    Measuring Volume, Not Skill

    Under most current training models, Dr. LaPorte said, case numbers often serve as a proxy for a resident’s mastery of a procedure but really only measure volume, not skill level.

    What’s more, learning opportunities have been reduced with recent caps on resident work hours, implemented to prevent mistakes due to exhaustion.

    Under the current system, she said, residents rarely receive formal feedback on their motor skills or errors, allowing mistakes to perpetuate. Evaluating residents as they work on cadavers helps them identify areas in need of improvement and learn from mistakes.

    “They can practice repeatedly in a safe environment before they’re holding the knife to operate on real people,” Dr. LaPorte said.

    Checklist of Technical Skills

    The new research grows out of her team’s bid to address shortcomings in the assessment of motor skills by using a well-established checklist called the Objective Structured Assessment of Technical Skills, or OSATS, that she and her colleagues modified to fit certain surgical procedures of the shoulder.

    The team then tested the checklist in residents practicing 3 different approaches to access a shoulder in need of repair: from the front, back, or side.

    Three orthopaedic specialists from Johns Hopkins then used the grading systems to evaluate the work of 23 Johns Hopkins medical residents ranging from their first to fifth year of residency as they performed each of the procedures on cadavers.

    Under OSATS, the residents received a point for the successful completion of each step in the checklist and a zero for failed or incomplete steps. They also received zeros if they performed the steps out of order. The crucial steps in this case involved making the incision in the right place and avoiding important structures, such as nerves and arteries.

    Besides the OSATS checklists, the faculty surgeons rated residents using the so-called Global Rating Scale and a simple pass/fail system. The Global Rating Scale was developed by Richard Reznick in 2006 and is applicable to all surgical procedures, taking a holistic view of the surgical process and offering residents feedback on such things as whether they used the surgical instruments correctly and whether they displayed a broad understanding of the procedure.

    Under the pass/fail system, residents receive a failing grade if they commit an egregious error — in the case of shoulder surgery, severing a nerve or blood vessel.

    The Johns Hopkins team found that OSATS and the Global Rating Scale provided good, objective ways of measuring resident performance, while the pass/fail system gave residents unambiguous feedback.

    How Did the Residents Do?

    As expected, more advanced residents received higher OSATS and Global Rating Scale scores than those just beginning their residencies. Scores on all 3 procedures ranged from the low single to double digits for first-year residents to the high double digits for fifth-year students.

    Across the 3 procedures, examiners observed 11 incidents in which residents damaged the nerves or veins. First- and second-year residents were responsible for 9 of those mistakes.

    Crucially, however, the Johns Hopkins team found that none of the 3 evaluations adequately captured those mistakes. The pass/fail scheme came closest, in that residents received a failing score for severing a nerve or major blood vessel, but it is not set up to subsequently inform residents of the precise nature of their mistakes.

    In addition, residents could theoretically perform well on the OSATS checklist even if they made an egregious error because points are not deducted, only earned in the grading system.

    To mitigate this, the researchers propose adding safety steps, such as identifying and protecting important structures, to the checklist.

    Future Steps

    Moving forward, Dr. LaPorte and her team hope to implement these training protocols for other common surgical procedures.

    The residents reported that they like an objective evaluation system such as the checklist because it gives them the confidence they need before entering an actual operating room, Dr. LaPorte said.

    “The goal,” she added, “is to make objective evaluations standard for all procedures expected of residents.”


    Bernard JA, Dattilo JR, Srikumaran U, Zikria BA, Jain A, LaPorte DM. Reliability and validity of 3 methods of assessing orthopedic resident skill in shoulder surgery. J Surg Educ. 2016 Jun 3. pii: S1931-7204(16)30036-8. doi: 10.1016/j.jsurg.2016.04.023. [Epub ahead of print]