We use cookies on this site to improve our service, perform analytics, and remember website preferences. By using this site, you consent to these cookies. For more information visit our Privacy Policy
Setting a new standard for comparative evaluation
HHF developed a rigorous methodology to comprehensively compare the usability of four different ventilators. The findings highlight the importance of good design in the engineering of MedTech and can help hospitals to make sound procurement decisions around life-saving devices.
Client
Maquet-Getinge Group
In Collaboration With
University of Toronto
Services
Comparative Usabililty Testing
Challenge
While usability testing is an important part of the design and regulatory approval process for medical devices, there have not been many publicly available studies that rigorously compare the safety and usability of different devices on the market. Without this knowledge, it can be challenging for hospitals to understand which device is safest for their patients and best suited to their staff, as well as for manufacturers to understand the importance of usability engineering when designing the next generation of their product.
To fill this knowledge gap, Maquet approached us to independently design and conduct a usability study that would compare their new ventilator against three others from leading manufacturers and then publish the results in a peer-reviewed medical journal. In order to objectively demonstrate that one ventilator was safer and easier to use than another, we needed to design a novel methodology and overcome the logistical challenges of what would be a large and complex study. It was hoped that the results would create a new standard for comparative evaluations and the design of future medical devices.
Process
Developing the Methodology
As there was no data in the existing literature to guide us, we began with a pilot study to estimate the sample size needed for the full-scale study and refine the test metrics and scenarios (based on ISO 80601-2-12). The pilot also helped us develop the model to statistically compare and rank the ventilators based on their use safety and user experience. This model was refined to use three metrics: use safety (UE/CC), usability (PSSUQ), and workload (NASA-TLX).
Conducting the Study
The full-scale study included over 350 hours of observed use and involved 48 respiratory therapists from three hospital networks performing a series of 16 tasks. The order of ventilators was fully counterbalanced, and each participant performed all 16 tasks using each ventilator. In order to ensure objectivity, the principal investigator was fully blinded to which dataset corresponded to which ventilator while he refined the study design and performed the statistical analyses. Participants also did not know which company was sponsoring the study. We synthesized the results, ranked the ventilators based on their performance, and published a paper in Critical Care.
- 350+Hours of observed use in the full-scale study
- 48Respiratory Therapists Participated
- 16Tasks performed, representing 7 clinical scenarios
Outcomes
Our method determined that Maquet’s SERVO-U demonstrated the highest levels of use safety and user experience, ranking above the other ventilators on seven of nine possible comparisons, followed by the Hamilton G5. The results of this study can help manufacturers to understand how human factors and usability engineering methods should influence the design of better, safer products. The published results also allowed Maquet to market their product as objectively safer than their competitors’, and provided data to their own team that would reinforce the value of usability engineering during product design. By developing the methodology needed to successfully conduct this study, HHF set a new standard for rigorous comparative evaluations that can be applied to the design and procurement of a range of medical devices, products, and services.