Simulation Program Benchmarks Surgical Proficiency

OREF grant recipient tests unique tool for assessing surgeons’ skills

Mark Crawford, OREF contributing writer

Several years ago, Matthew D. Karam, MD, clinical associate professor and orthopedic traumatologist at the University of Iowa in Iowa City, helped develop an orthopedic surgical simulation platform. Created in collaboration with Geb W. Thomas, PhD, professor in industrial and systems engineering, Donald D. Anderson, PhD, professor in orthopedics and rehabilitation, and Steven Long, then a doctoral candidate in biomechanics, the simulator replicates the look and feel of navigating a wire through bone utilizing fluoroscopic guidance. Participants maneuver a surgical guide wire through an opening in a foam rubber-exterior (representing soft tissue) into a plastic femur, drilling from the lateral cortex through the neck and into the femoral head.

“Although we initially developed this platform to simulate the navigation of a wire for treating intertrochanteric hip fractures, we have since added more challenging applications that require navigation of wires in the pelvis and distal humerus, broadening the relevant range of anatomy and fostering deeper learner engagement,” said Dr. Karam.

Interested in establishing that their simulator could be used as a training tool to improve residents’ skill at navigating a guide wire, Dr. Karam and Dr. Thomas applied for and received an American Board of Orthopaedic Surgery (ABOS)/Orthopaedic Research and Education Foundation (OREF) Surgical Skills Simulation Training for Residents Research Grant for a research project entitled “Benchmarking Expert Performance to Establish Competency in Wire Navigation.”

The study is designed to test the effectiveness of the simulator as a method for characterizing the performance of fully trained skilled surgeons. This performance data will then be used to define benchmarks for levels of performance for new surgeons, “before they enter the operating room,” said Dr. Karam. “Ultimately this will create a safer environment for patients and learners alike.”

Expert trauma surgeons establish proficiency standards

The three-year research project (September 2018 to August 2021) will measure the distribution of performance measures for approximately 120 expert orthopaedic trauma surgeons, drawn from Orthopaedic Trauma Association (OTA)-sanctioned trauma fellowship programs, in different scenarios, using the simulator. Nearly 300 residents in orthopedic surgery will also be tested on the simulator, with their results compared to those of the OTA fellows. The performance levels of the skilled surgeons will be used to establish benchmarks for training and proficiency levels for certification purposes for orthopaedic residents.

The research has three specific aims:

• To measure the skill distribution of experts assessed with both intertrochanteric hip fracture and iliosacral screw exercises on the wire navigation simulator;
• To define measurement criteria of competent performance; and
• To validate that the defined criteria may be reasonably achieved by orthopaedic residents in their second or third year of training.

Surgical performance is assessed using several different measures, each addressing resident skill and directly relevant to patient safety.

“The first measure, the tip-apex distance [TAD], evaluates the accuracy of guide wire placement within the femoral head,” said Dr. Karam. “The second measure of performance is the duration of the procedure. Shorter durations demand economy of motion and lower the risk of infection. The third measure is the number of fluoroscopic images used in placing the wire. Fewer images require a higher degree of planning and spatial reasoning from the resident and expose the resident, other health care personnel, and the patient to less radiation.”

Evaluation under way

Orthopaedic surgeons can be evaluated according to individual procedures or fundamental skills. “Fundamental skills persist, even as surgical techniques evolve over time,” said Dr. Karam. “A central premise of our research is that fluoroscopic wire navigation is a prime example of a fundamental skill. It requires a combination of physical control, spatial visualization, and planning, with wide applicability to general orthopedic practice. We chose our initial application of placing a wire as part of treating a hip fracture because it is a fundamental skill that is part of a wide range of orthopaedic surgeries involving fluoroscopic wire navigation.”

The goal of the research is to measure expert performance and use this data to set proficiency benchmarks for fluoroscopic wire navigation tasks. As the first phase of the project, Dr. Karam and his team gathered data on 28 expert participants who attended the OTA Trauma Fellows Course in the spring of 2019. With funding from the Agency for Healthcare Research and Quality (AHRQ) they also conducted a complementary study to evaluate 85 residents, most in their first or second year of training, on the simulator during the OTA Residents Comprehensive Fracture Course in April 2019.

“The initial results show, as one would expect, that experts perform wire navigation tasks at a higher level than novice residents,” said Dr. Karam. “The real strength of this study will come as we gather additional data and develop the statistical power to determine the performance levels we expect novice learners to achieve for simulated procedures, before they enter the operating room.”

Dr. Karam and his team are extremely grateful for the support of ABOS/OREF, which he said is critical for moving this important research forward.

“Having the support of ABOS/OREF in developing expert-level benchmarking to determine proficiency gives us much more credibility and allows us to connect to a larger audience,” said Dr. Karam. “It’s a game changer for us.”

Back to Profiles