We use cookies on this site to improve our service, perform analytics, and remember website preferences. By using this site, you consent to these cookies. For more information visit our Privacy Policy
What FDA’s GenAI Medical Device Discussion Paper Signals for Human Factors
This article responds to a recent paper published by the FDA’s Center for Devices and Radiological Health that sought feedback on potential approaches to risk assessment, premarket evaluation, and post market monitoring for generative AI-enabled medical devices. In this article, we present 3 key considerations for developing generative AI features for medical devices.
Topics Covered
FDA guidance, regulatory support, generative AI, human factors validation testing
FDA’s Center for Devices and Radiological Health (CDRH) recently published a discussion paper seeking feedback on potential approaches to risk assessment, premarket evaluation, and post market monitoring for generative AI-enabled medical devices.
For manufacturers developing GenAI-enabled medical devices, the paper raises a broader human factors question:
How do we evaluate safe use when what the AI produces—and how users respond to it—can vary across users, contexts, and interactions?
Here are three areas where manufacturers may want to start thinking differently about human factors.
1. What Happens After the AI Produces an Output?
As Cory Doctorow states in The Reverse Centaur’s Guide to Life After AI, “the most important fact about a technology isn’t what it does, it’s who it does it for, and who it does it to.” This outlook is reflected in the FDA’s two-axis framework. CDRH is considering a risk framework based on both the level of activity performed by the GenAI and the consequences of relying on an incorrect output. The paper specifically considers treating action-directing functions as higher risk than simple, non-directive information.
From a human factors perspective, however, risk does not necessarily end with the output. It extends to what users understand the output to mean and what they do because of it. An informational output could influence a clinical decision if users perceive it as authoritative. Conversely, an action-directing recommendation may be reviewed by a clinician or constrained by workflow safeguards.
The paper acknowledges that the distinction between informational and action-directing functions may not be binary and identifies factors including specificity, personalization, context, user interpretation, and downstream safeguards.
For manufacturers, this suggests an important question for use-related risk analysis:
What might users reasonably infer from this output, and how could those interpretations affect subsequent actions?
That question can help connect GenAI functionality to the use-related hazards and risk controls documented for the device.
2. Can Users Appropriately Judge the AI's Output?
CDRH is considering a competency-based approach to premarket evaluation that includes communication quality and user comprehension. The paper also identifies automation bias as a potential failure mode and considers whether users understand system limitations and when escalation is appropriate.This shifts the evaluation question beyond “Is the AI output correct?” to “Can the intended user appropriately understand, evaluate, and act on the output?”
For manufacturers, this creates practical design and evaluation questions:
- Does the system communicate uncertainty and limitations clearly?
- Does the interaction support appropriate verification?
- Will users recognize when they should question or escalate an output?
- Changes in indications for use or changes in use environment, in comparison to other devices of the same device type
- Does the design encourage appropriate reliance rather than over-reliance?
These questions cannot be answered by evaluating the AI in isolation. They require understanding how the AI is presented, what the user is expected to do with its output, and what safeguards exist before that output can affect patient care.
3. What Will We Learn from Real-World Use?
GenAI-enabled devices introduce another challenge: user interaction and system behavior may vary across real-world contexts. As a result, CDRH raises the possibility of greater reliance on post market monitoring.
For manufacturers, this creates an opportunity to look beyond technical performance and ask how people actually use and respond to the system after deployment.
Naturalistic observation and interviews can reveal patterns that may not emerge during simulated evaluations, including over-reliance, misunderstanding, workarounds, or inappropriate escalation. From an HF perspective, this matters because these behaviors can reveal a gap between how the device was expected to be used during development and how people incorporate it into their work. For GenAI-enabled devices, those gaps may also reveal new patterns of reliance or interaction that were not anticipated during premarket evaluation.
Post-production monitoring can be more than a way to identify complaints. It can provide insight into whether the assumptions made during design and evaluation continue to hold in real-world use.
What This Could Mean for Manufacturers
The discussion paper points toward an expanded role for human factors in GenAI-enabled medical devices because safe performance depends on the interaction between the AI, the user, and the context in which the device is used.
For manufacturers, three questions provide a useful starting point:
Risk
What might users reasonably infer from an AI output, and how could that contribute to use-related risk?
Design and Evaluation
Does the system support appropriate verification, uncertainty recognition, escalation, and decision-making?
Real-World Use
How might human-AI interaction change across users, contexts, and workflows after deployment?
While the final regulatory approach has not yet been established, manufacturers do not need to wait for official requirements to consider these questions.
Thinking about them early can help connect use-related risk analysis, product design, premarket evaluation, and post-production surveillance into a more complete picture of how the GenAI-enabled device will actually be used.
Author
Ashleigh Shier, Sr. Human Factors Specialist, BASc in Industrial Engineering, MHSc in Clinical Biomedical Engineering
Last Updated October 16, 2026
At HHF, we help medical device manufacturers apply human factors throughout that process—from understanding use-related risks and designing safer interactions to evaluating usability and studying real-world use.
Services that may interest you:
Use-Related Risk Analysis Documentation Support
We help medical device teams transform user insights into actionable risk management documentation by linking user needs, use errors, hazards, and mitigations in a clear, auditable framework. The result is a comprehensive Use-Related Risk Analysis that supports IEC 62366-1, ISO 14971, and human factors validation activities as well as design and regulatory readiness.
UX/UI Design for Medical Devices
Our UX/UI team uses co-design methods to ensure that the lived experiences of patients and providers shape the design of the devices they will use. These approaches help to identify issues before they emerge in a validation test and reduce use-related risks by informing design decisions with direct user feedback throughout development.
HFE Post-Market Monitoring & Evaluation
Our post-market monitoring method involves proactively collecting use-related data for medical devices by observing and interviewing clinicians in their natural work environment to understand how users interact with devices and gather insights that help avoid post-market complaints and inform new and improved device versions.
Want to work with us?