In May 2024, the Stanford Institute for Human-Centered AI (HAI) hosted a closed-door workshop attended by 55 influential figures from various sectors, including policymakers, healthcare providers, AI developers, and patient advocates. The workshop aimed to chart a path forward for artificial intelligence (AI) policy in healthcare, focusing on its transformative potential to improve diagnostic accuracy, enhance administrative operations, and increase patient engagement. Over the period from 2017 to 2021, the healthcare sector garnered more private AI investment than any other, amassing a staggering $28.9 billion globally.
However, this enthusiasm for innovative healthcare technologies is tempered by significant concerns regarding patient safety, inherent biases, and data security. As regulators strive to foster the development of these advanced tools, they face the formidable challenge of adapting existing regulatory frameworks that were designed for traditional healthcare models reliant on physical devices, paper records, and analog data. The rapid integration of AI into healthcare processes underscores the urgent need for a thorough review and overhaul of these outdated frameworks.
Workshop Context and Objectives
Recognizing the gap in existing regulations, the HAI gathered leading experts to discuss the shortcomings of federal healthcare AI policies. Participants explored three critical areas: AI software for clinical decision support, healthcare enterprise AI tools, and patient-facing AI applications. Under the Chatham House Rule, discussions were candid and aimed at identifying key policy gaps and galvanizing support for necessary regulatory changes.
The Regulatory Landscape: A Historical Perspective
The healthcare industry in the United States is one of the most heavily regulated sectors, and existing regulatory frameworks are beginning to extend to AI technologies. For instance, the Food and Drug Administration (FDA) oversees many software systems through its 510(k) device clearance process, which classifies software as a medical device (SaMD). AI applications used in both administrative and clinical settings must comply with rules set forth by the Office of the National Coordinator for Health Information Technology, emphasizing algorithmic transparency. In contrast, direct-to-consumer health AI tools fall under various consumer product regulations, though enforcement has been sparse in this emerging field.
The regulatory structures currently in place, however, are outdated. The FDA’s authority was established in 1976, focusing primarily on hardware devices and not considering the complexities of software reliant on training data that demands ongoing performance monitoring. Similarly, the Health Insurance Portability and Accountability Act (HIPAA), enacted in 1996, set national standards for the privacy and security of health information but did not anticipate the extensive digital health data necessary for training machine learning algorithms.
One workshop participant aptly likened the situation to driving a vintage 1976 Chevy Impala on modern roads, highlighting the struggle of regulators to adapt to the rapid pace of AI development in healthcare. The consensus among workshop attendees was clear: a new or significantly revised regulatory framework is essential for effective governance of healthcare AI.
AI in Software as a Medical Device
The participants identified a significant hurdle for developers of AI-powered medical devices, particularly those with diagnostic functions. The current FDA clearance process requires evidence submission for each diagnostic capability, which is impractical for AI products with numerous functionalities—like an algorithm that detects various abnormalities in a chest X-ray. This process often leads to the market entry of less innovative products, hindering the growth of AI medical device innovation in the U.S.
To streamline market approval for these multifunctional software systems while ensuring clinical safety, workshop participants proposed several policy changes. They emphasized the importance of fostering public-private partnerships to alleviate the evidentiary burden of approval and focus on post-market surveillance to track the performance of AI tools after they enter the market. Improved information sharing during the clearance process was also suggested, enabling healthcare providers to better assess the safety and efficacy of software tools prior to implementation. Despite close to 900 medical devices incorporating AI or machine learning having received FDA clearance, clinical adoption has lagged, partly due to the limited information available to guide purchasing decisions.
Participants called for the establishment of more nuanced risk categories for AI-powered medical devices. Currently, most are classified as Class II devices, indicating moderate risk. However, the clinical risk associated with different AI software can vary widely. For example, an algorithm that measures blood vessel dimensions for human review poses a lower risk compared to an algorithm responsible for triaging mammograms without human intervention.
AI in Enterprise Clinical Operations and Administration
The workshop sparked a debate on the necessity of human oversight when integrating autonomous AI tools in clinical settings. Technologies that autonomously diagnose conditions or automate routine tasks, such as drafting responses to patient emails or documenting progress notes, promise to alleviate the strain on healthcare providers amid severe workforce shortages. However, opinions diverged regarding the level of human involvement required to ensure safety and reliability.
Some participants argued that maintaining human oversight is critical to mitigate risks associated with AI deployment, while others cautioned that such requirements might increase the administrative burden on doctors, potentially leading to diminished accountability for clinical decisions. They cited laboratory testing as a successful hybrid model where AI tools are monitored by physicians, who regularly check for quality control.
As AI technologies continue to proliferate in clinical settings, a fundamental question arises: what level of transparency is necessary for healthcare providers and patients to safely utilize these tools? The responsibility of developers to clearly communicate model design, functionality, and associated risks was emphasized, with suggestions to create model cards akin to “nutrition labels” to inform healthcare providers.
There was discussion on whether patients should be informed when AI is utilized in their treatment. Many patients delegate decisions regarding technology—ranging from surgical instruments to decision-support tools—to their healthcare providers and organizations. However, participants felt that transparency was particularly important in certain scenarios, such as when an email appears to be from a healthcare provider but is generated by AI.
Patient-Facing AI Applications
The emergence of patient-facing applications, such as mental health chatbots powered by large language models (LLMs), raises important questions about access to healthcare and the quality of information provided to patients. While these applications have the potential to democratize healthcare access, there is an urgent need for regulatory guardrails to ensure they do not disseminate misleading or harmful medical information. This concern is amplified when chatbots present information that resembles medical advice, even when disclaimers suggest otherwise.
Workshop participants highlighted the necessity for a clear regulatory framework for patient-facing AI products. There was considerable debate about whether generative AI applications should be regulated like medical devices or treated as medical professionals. The inclusion of patient perspectives in the development, deployment, and regulation of AI applications was emphasized as critical to ensuring the trustworthiness of healthcare AI and the healthcare system as a whole. Many attendees noted the lack of patient involvement in these processes and the need to consider the needs and viewpoints of diverse patient populations to address health disparities exacerbated by AI.
Collaborative Path Forward
The closed-door workshop organized by the Stanford Institute for Human-Centered AI illuminated the complex landscape of AI policy in healthcare. While the potential for AI to enhance healthcare delivery is immense, the current regulatory frameworks are ill-equipped to address the unique challenges posed by these technologies. As the industry navigates this transition, collaboration among policymakers, healthcare providers, technology developers, and patient advocates will be essential in shaping a regulatory environment that supports innovation while safeguarding patient safety.
The discussions highlight a pressing need for updated regulations that reflect the realities of modern healthcare and technology. By embracing a collaborative approach and actively involving stakeholders at all levels, we can ensure that AI serves as a transformative force for good in healthcare—one that not only enhances patient outcomes but also prioritizes safety, equity, and trust in the evolving healthcare landscape.
For a deeper dive into the discussions and insights shared during the workshop, we encourage you to read the full article Standfor University HAI.
