Enhancing an Expert Review for Summative Testing

HHF was asked to perform a summative evaluation of a patient monitor using an expert review for EU MDR certification. Because existing standards provide limited guidance on conducting expert reviews, HHF developed an Enhanced Expert Review (EER) method that combines the objectiveness of usability evaluation with the efficiency of analytical review. When compared against usability testing on the same user interfaces, EER identified additional use issues, suggesting it may provide a resource-efficient source of additional usability insight.

Services

Enhanced Expert Review

Challenge

Medical device teams often face a difficult decision when evaluating design changes. Simulated-use testing can provide valuable empirical evidence, but conducting another study may not be practical when timelines, budgets, or development stages limit the feasibility of testing. Although IEC 62366-2 identifies expert review as a potential alternative, it provides little guidance on how such reviews should be conducted or how confidence in the findings can be increased.

Despite the lack of guidelines, IEC 62366-2 states in Annex E.7 that expert reviews can be a cost-saving alternative to conducting usability testing in simulated use conditions. While a single usability specialist sitting in a room listing out strengths and weaknesses is less costly than conducting a usability test, this may not be the most effective method for identifying all usability issues and use-related risks.

To address this, HHF developed an Enhanced Expert Review (EER) method designed to combine some of the strengths of usability testing with the efficiency of expert inspection methods.

Process

An EER combines empirical and analytical usability evaluation methods, comprising of a realistic task completion, a Perception-Cognition-Action (PCA) interview, and a direct assessment of design mitigations. Participants include both subject matter experts (SMEs) and usability experts to provide a broad range of perspectives when conducting tasks and assessing mitigations.

1- Realistic Task Walkthrough

The first phase of an EER involves the completion of realistic tasks within representative workflows using the subject prototype/device. A facilitator will provide clinically (or technically) representative prompts and have the expert conduct tasks with the subject device while thinking aloud. As the session proceeds, the facilitator records any observed use issues. If the expert omits a task/sub-task or does not execute the task/sub-task as expected, the facilitator will guide them through the expected workflow. The facilitator will then ask a series of debrief questions to identify the root causes of any identified use issues.

2 – Perception-Cognition-Action Interview

After the completion of each task or scenario, the facilitator will ask specific questions about how the end-user might perceive, understand, and execute each task. This component of the EER allows experts to frame their perspectives as end users and identify use issues that they did not necessarily experience themselves during their task walkthrough. The PCA approach is consistent with the user-device interaction model described in AAMI HE75:2025 Section 5.3, which characterizes user interactions in terms of Information Perception, Cognitive Processing, and Control Actions, and is adapted from IEC 62366-1 Figure A.1.

3 – Direct Assessment of Mitigations

Following the interview, the facilitator will then describe any known use issues associated with each mitigation under evaluation (i.e., existing, new, or modified mitigations). These use issues may have been identified in previous evaluations, or at other stages of the risk management process. The facilitator will ask open-ended questions to directly assess the effectiveness of those mitigations and gather important feedback to determine whether further design changes are necessary.

Illustrated timeline showing three stages of a healthcare evaluation process. From left to right: a hospital building labeled “Emergency” representing in-hospital simulated-use testing; an “IFU 2.0” instruction manual representing design changes to instructional materials; and a patient monitor displaying vital sign waveforms representing enhanced expert review. A horizontal timeline connects the three milestones.

In Practice

HHF applied EERs to evaluate critical care patient monitors (a physiologic patient monitor and a MR-safe patient monitor). In both cases, EER was used after (formative and summative) usability testing to evaluate design changes that were made to the accompanying instructional materials to address new and existing use issues. Because the device interfaces themselves remained unchanged, these projects provided a unique opportunity to compare the issues identified through EER with those previously identified during empirical usability testing and assess the additional value of the method.

Comparison table showing results for two patient monitors evaluated using usability testing and enhanced expert review. For Patient Monitor 1, usability testing involved 30 participants over 12 testing days and identified 15 unique use issues, while enhanced expert review involved 4 reviewers over 3 days and identified 14 unique use issues and 9 recurring use issues. For Patient Monitor 2, usability testing involved 60 participants over 14 testing days and identified 13 unique use issues, while enhanced expert review involved 5 reviewers over 3 days and identified 4 unique use issues and 7 recurring use issues. The table highlights differences in participant numbers, testing duration, and issue identification between the two evaluation methods.

Table 1 

Note: The use issues included here are limited to those that could have occurred in either simulated use testing or EER, as they were in scope of both evaluations and unrelated to the design changes.

Comparison table showing issue identification by individual participants during enhanced expert review of two patient monitors. For Patient Monitor 1, four participants identified between 4 and 6 total issues each out of 14 possible issues, and between 1 and 4 unique issues each; a fifth participant was not applicable. Participant totals were 6, 6, 5, and 4 issues, with 2, 2, 4, and 1 unique issues respectively. For Patient Monitor 2, five participants identified between 0 and 2 total issues each out of 4 possible issues, and between 0 and 1 unique issues each. Participant totals were 2, 2, 2, 1, and 0 issues, with 1, 0, 1, 0, and 0 unique issues respectively. The table illustrates variation in issue detection across expert reviewers for both patient monitors.

Table 2 

Participants 1-5 are not the same participants in each testing format.

Both EERs identified issues that had not been observed in usability testing. However, there were also issues that were only discovered in usability testing and not by the EER.

When used together, these methods can identify a wider range of usability issues than either method standing alone. Thus, EER can be a highly effective and resource-efficient complement to usability testing and can greatly strengthen any usability risk management process to ensure the safety and efficacy of medical devices.

When to Consider Using an EER

A design modification has been made following usability testing
Existing mitigations require evaluation before further development
Early feedback is needed before investing in a larger study
Project timelines make additional testing difficult
Specialized clinical expertise is needed to evaluate workflows
Looking for possible summative evaluation alternatives (when following IEC 62366-2)

When We Would Not Recommend EER

As a replacement for summative human factors evaluation for FDA regulatory purposes
When critical questions require evidence from representative users
When risk acceptability depends on observed user performance

Key Takeaway

Our experience suggests that EER occupies a useful middle ground between traditional expert inspection and empirical usability testing. It cannot replace testing with representative users, but it can provide meaningful evidence, uncover previously unidentified issues, and help teams evaluate design changes efficiently. Most importantly, it appears to identify different types of usability concerns than those captured through usability testing alone, making it a valuable addition to the usability engineering toolbox.

A note on the use of EERs: EER was used in a summative evaluation because the scope was limited to minor changes to the user interface that did not involve any hazard-related use scenarios associated with serious harm. This summative evaluation was not used for validation testing for pre-market submission to the FDA. Although EER does not align with the FDA’s guidance for conducting HF validation / summative testing, it does align with its guidance for conducting preliminary analyses and evaluations.

About HHF’s Evaluation & Regulatory Services

Our team works with healthcare organizations, including the top 10 medical device companies in the world, to ensure products are safe and effective for intended users, uses and use environments. We have extensive experience with FDA, European, and international usability requirements and can tailor our services specifically to your needs.

Authors:
Ashleigh Shier, Human Factors Specialist
Justine Steven, Human Factors Specialist
Ryan Cheng, Human Factors Specialist
Wayne Ho, Managing Director, Healthcare Human Factors

Last Updated September 29, 2026

ANSI/AAMI Guidance Support

How the new ANSI/AAMI HE75 updates impact your Medical Device design development and usability process

FDA Regulatory Guidance Support

Decision Point D and Other Changes That Will Affect Your Next FDA Submission

Medical Device Reprocessing

Integrating FDA reprocessing guidance into human factors validation tests