Who is this for?
If you landed on this page, you're probably interested in applying large language models to medical devices. You may have a medical device or may not know if your application is even considered a medical device at all. Or you may be looking at different regulatory pathways and wondering which one is the right one for your device. As of writing of this article, the regulatory pathway for LLMs are unclear. And there's many gray zones.
I could not find suitable answers from FDA or other sources so I decided to forge a new pathway. I took my previous experiences with AI applied to convolutional neural networks and tried to generalize them to this new age of large language and large multimodal models. I believe large language models should be used responsibly and to find low-risk usages of them first. Incorrect or hallucinated data is worse than no data at all. And therefore, the use case of LLMs should be carefully monitored and made part of the risk assessment of your device. A key risk assessment should look at the possibility of LLM hallucination and mitigate it appropriately either by design or by risk control measures and almost never by labeling because no one really reads those, just being completely honest.
Key Takeaways from the LLM Presubmission
- FDA stated that the topic of incorporating LLM is a novel idea and more internal conversation needs to be had about these types of devices, which might affect future feedback.
- Sponsor asked the Agency about using synthetic data. Sponsor proposed having a board-certified radiologist review it. FDA stated that given their current understanding there would a tremendous amount of information needed on the model. Thus, it might be easier to provide real data.
- FDA stated that depending on prevalence, consecutive collection is usually good method to follow. However, the Agency emphasized that for LLM, FDA does not have clear criteria on what they are looking for, but welcomes Sponsor justifications.
- Sponsor stated that it is now possible to run LLM locally, so have reasonable understanding that it would be safe to use in production. FDA stated that using a “frozen” model and providing a clear understanding of how it is being deployed would be recommended in future submission.
- Sponsor asked the Agency if there are any other general concerns for the use of LLM. FDA stated they haven’t seen a lot of projects that use LLMs. FDA stated that a frozen application would be desirable, but if Sponsor is not able to do that, then reason to not do that then there would be a concern of testing device in a “black box” environment would be much more difficult.
What are the steps to a presub anyway?
Step 2: FDA responds 1 week before the agreed upon meeting date
Step 3: Send your presubmission presentation to the FDA ahead of time at least 48 hours.
Where to from here?
This is unprecedented regulatory territory. If you need a thought leader or team of thought leaders on your side, please don't hesitate to reach out. We are happy to forge a pathway for your large language model, large vision model, large action model, or large whatever model, including radiology foundation models. No one has done it before, and that's why you need to hire thought leaders that can generalize the existing regulations and guidances into brand new territory. Contact us today.
Why stop at a presub?
Do you want access to a fully un-redacted 510(k) submission package for a cleared software as a medical device with a machine learning component? This would have honestly saved me a year on my first FDA application.
COMMENTS