GenAI Devices Don't Need New Security Rules — They Need FDA's Existing Ones

FDA's discussion paper on generative AI-enabled medical devices gets a lot right, including calling for testing against prompt injection and adversarial prompting. But it treats those as benchmark competencies, scored alongside clinical accuracy, rather than as a cybersecurity problem. That framing leaves real risk on the table: a benchmark is a snapshot, attackers adapt, and a clean pass rate against a frozen test set is evidence, not proof. FDA already knows this from penetration testing. It should apply the same lesson here.
We submitted comments focused on three areas where our work with manufacturers — threat modeling, SBOMs, third-party component risk, postmarket vulnerability management — is most directly relevant.

Adversarial manipulation is a cybersecurity risk, not a scorecard item

A successful prompt injection doesn't just produce a wrong answer. It can expose patient data in a model's context, trigger actions through connected tools, alter records, or take a device offline — and indirect injection can arrive through documents, EHR fields, retrieved content, or connected systems, which makes it an architecture problem that only threat modeling can find.
FDA doesn't need new tools for this. Section 524B and the premarket cybersecurity guidance already expect threat models, security risk assessments, and security testing — and FDA's own January 2025 draft guidance on AI-enabled device software functions already treats data poisoning, model evasion, model inversion, and data leakage as cybersecurity threats. We recommended the GenAI framework stay consistent with these existing sources: add exposure to untrusted input as a risk dimension, and for agentic devices, expect least-privilege tool permissions, model outputs treated as untrusted before high-consequence actions, human oversight enforced by system design (not model instructions), and tamper-evident logs.

Foundation model changes are a supply chain problem with existing precedent

When a third-party model changes because someone else changed it, that's the same problem manufacturers already navigate with any third-party component — and FDA already addresses it through SBOM and postmarket vulnerability management expectations. We recommended treating the model like any other component: inventory it (provider, version, hosting, system prompt and configuration version, fine-tuning, retrieval sources), get supplier commitments in writing on advance notice of changes and retirements, and build in automated regression testing against a benchmark subset on a schedule and on any detected change.
On PCCPs specifically: they should prespecify the evaluation protocol and acceptance criteria for an upstream model change, not the change itself — that's how secure patching already works, and we've helped manufacturers build PCCPs the same way.

Postmarket monitoring should build on cybersecurity practice, not start from scratch

GenAI monitoring will hit the same problems postmarket security teams already know: alert volume, signal thresholds, and accountability. In our experience, alert volume is often the single biggest obstacle — real issues get missed when teams are buried in findings that don't apply. Security teams solve this with exploitability statements (VEX) that say whether a known vulnerability actually affects a given product. We recommended FDA consider a VEX-style statement for whether a newly published jailbreak or injection technique affects a device as configured, and encourage manufacturers to fold GenAI monitoring into existing postmarket cybersecurity processes rather than standing up a parallel one.

Close

Many of the most serious GenAI failure modes are security failures, and FDA has already built strong tools for those. Our recommendation to CDRH: extend them explicitly to GenAI-enabled devices, rather than relying on performance benchmarking alone.