Testing a complex medical device
When approached to conduct a summative test on a novel and complex medical system, HHF leveraged our expertise in clinical and human factors engineering and a deep understanding of the healthcare system to design the study. Expert planning resulted in a successful study and regulatory approval for our client’s novel device.
FDA Pre-market Human Factors Evaluation, Regulatory Compliance, Usability Testing
Although novel medical devices have the potential to save lives through cutting-edge treatment, there is also potential for harm to patients and users if they are not rigorously tested and given regulatory approval before they are released. Summative usability testing is an important step in the regulatory approval process for medical devices, as it validates the use-safety of a manufacturer’s product. Most published accounts of how to conduct human factors summative tests for medical devices feature relatively simple, single user products such as insulin injectors or infusion pumps. However, manufacturers of more complex products may face unique challenges while planning for summative testing. When Elekta approached us to conduct a summative test for a complex and novel medical system–the Elekta Unity–we needed to leverage our team’s clinical engineering knowledge and our many years of experience in summative usability testing to design and execute a study that would meet such challenges.
Elekta Unity is the world’s first high-field MR-Linac (MRL), enabling real-time viewing of diagnostic quality images during daily delivery of radiation therapy in cancer treatments, essentially merging an MR imaging device with a Linac (linear accelerator). Our summative test needed to adequately assess the use-safety of Unity, which meant that we would need to consider all of the system’s complexities, including its physical size and the many days of training that a highly-specialized clinician is expected to receive before using one. From a regulatory perspective, Elekta Unity consists of a number of products, yet a user might touch all of these systems in a single use scenario. We first needed to decide whether to evaluate Unity as separate products or the full system.
How do we make sure everything impacted by Unity gets included in our validation?
To better understand how the device would be used in the real world, we shadowed radiation therapists at Princess Margaret Cancer Centre. Princess Margaret, one of the top 5 cancer research centres in the world, is located within the same hospital network as us—the University Health Network—and our observations in the busy treatment centre allowed our human factors specialists to develop a deep understanding of the workflow of Unity’s end users.
Developing the Test Plan
From our initial discovery phase, we decided to design a single summative test and create a single report that would be referenced by all of the different regulatory submissions.
When we began, Unity’s clinical workflow (from initial consultation through to radiation treatment therapy) had not been performed end-to-end on the production-ready system. In addition, the novel mix of radiation therapy and MR imaging meant that many users would be combining these tasks for the first time. This resulted in some challenges when developing test scenarios and critical tasks that appropriately assessed the use-safety of the complex system. Formative evaluations were initially only able to focus on subsystems until late in the process when a full Unity system could be integrated. Our strategy was to complete a pre-summative study in order to confirm which users would do which tasks and build plans for the full-scale summative evaluation. We conducted a thorough task analysis to identify the correct critical user groups for the summative study, as the everyday clinical workflows for Unity involve multiple users who have different skills and roles. Once a thorough test plan was developed, we guided Elekta through an FDA pre-submission to confirm the appropriateness of the plan and protocol with the FDA.
Conducting the Full-Scale Study
Unity’s space requirements meant that we needed to find a location to host the study that would provide the correct space for an MRL, which required an area appropriate for a bunker, MRI safety zones, and an area for the simulator. We developed a close partnership with a hospital that had the expertise, custom-built space, staff, and flexibility to support the execution of the summative test. HHF’s large team of clinical engineering experts were able to travel to the location and conduct the study over several months.
The end users for Unity’s critical tasks would require deep and diverse clinical knowledge of MR imaging, radiation therapy, and dosimetry, and they would also need to be available for the time commitment of the study. Before using the device, users would need to undergo extensive training over the course of several months. Translated to the summative test, it was necessary to conduct pre-training on pre-existing software tools 3 to 6 months in advance with participants. In addition, 2-5 days of training on Unity itself was necessary. Recruiting enough of these highly specialized individuals to meet the minimum requirement of 15 participants per user group (45 participants in total) meant that we needed to recruit from the UK, Canada, the USA, and Australia.
- 10HHF team members involved
- 90+People on the project
- 8Nights per participant
HHF’s experience and expertise in conducting human factors studies, particularly with radiation therapy devices and complex systems, helped us to successfully plan and execute this project. The usability plan ran almost exactly as it was originally scheduled and resulted in Elekta Unity receiving CE Mark and FDA 510(k) clearance. The rigorous testing involved in this study allowed a product that was first-of-its-kind to be safely rolled out to patients.