"It is important to recognize the potential value that well‑designed, purpose‑built AI tools can bring to mental healthcare when deployed responsibly," the American Medical Association wrote in letters this week to Congressional leaders.
American Medical Association urges Congress to act
The American Medical Association (AMA), the largest U.S. professional association for physicians and medical students, told congressional co‑chairs that the rapid rise of consumer AI chatbots delivering mental‑health guidance demands statutory guardrails. In several letters sent Wednesday to the co‑chairs of the Congressional Artificial Intelligence Caucus, the Congressional Digital Health Caucus, and the Senate Artificial Intelligence Caucus, the AMA said chatbots are being used by patients to share intimate health and mental‑health details — often before they ever see a clinician — and that current protections are inconsistent.
AMA's specific requests to Congress
- Require "meaningful limits" on the collection and retention of sensitive information by AI developers.
- Establish safeguards against unauthorized access and sharing of that information.
- Mandate clear user consent for use of personal data by AI tools.
- Require transparent disclosures that users are interacting with AI technology rather than a human being.
The AMA framed these requests as measures to prevent a mismatch between public expectations of privacy and the realities of how chatbot data are stored and handled.
FDA oversight and statutory boundaries for diagnosis and treatment
The AMA asked Congress to set statutory boundaries that would prohibit AI chatbots from engaging in diagnosis or treatment of mental‑health conditions, giving the example of offering a diagnosis of anxiety or depression or recommending medications. The organization said any such action by a chatbot should trigger mandatory review by the Food and Drug Administration as a medical device.
In its letters the AMA warned that regulatory gaps could leave patients exposed to unsafe or unvalidated clinical advice absent explicit legal limits and medical‑device review.
Risks the AMA and others flagged: privacy failures, emotional dependency, and self‑harm
The AMA detailed multiple harms beyond data privacy: emotional dependency on chatbots, misinformation, inadequate crisis response, and examples of chatbots encouraging self‑harm. "Many individuals interacting with chatbots for mental health support understandably treat these interactions as private, even when the chatbot is not part of a healthcare system," the AMA wrote. "Chatbot conversations can be retained, logged, reviewed or inadvertently revealed, and the sensitivity of what individuals share is often greater than they would disclose in other online settings."
The group singled out children and teens as a population of special concern, noting young users may share highly sensitive information without understanding how it could be stored, accessed, or disclosed. The AMA also warned that many chatbots are built on complex software and cloud services and that "privacy and security can fail even when an AI developer's own code and policies appear sound. A single weakness in a data center can expose chatbot data and erode confidence."
Those technical and safety concerns echo an earlier warning from the patient safety research organization ECRI Institute, which identified chatbots as the top health‑technology hazard in 2026. ECRI researchers noted that broadly accessible AI tools are not designed or validated for clinical use yet are being used by patients to self‑diagnose and self‑treat.
What this means for patients, AI developers, and Congress
- Patients: The AMA advised caution. Dr. John Whyte, CEO of the AMA, told ISMG that while "AI chatbots can be incredibly helpful to patients trying to better understand a diagnosis, a recommended treatment plan or even lab results," they "cannot exercise actual clinical judgment and should never replace medical decisions made between a patient and their physician." The AMA also stressed that many consumer AI tools are not covered by laws protecting medical privacy such as HIPAA, meaning shared medical information may not be recoverable once disclosed.
- AI developers: The AMA’s demands — limits on data collection and retention, explicit consent, transparent disclosures, and technical safeguards — create a potential legislative pathway that could require product‑level changes, data‑handling practices, and new compliance mechanisms, especially for tools used by minors or in mental‑health contexts.
- Congress and regulators: The AMA delivered its recommendations directly to the co‑chairs of three congressional caucuses and recommended that any chatbot activity that crosses into diagnosis or treatment be subject to mandatory FDA review as a medical device. Parallel legislative activity already exists: Sen. Marsha Blackburn is sponsoring the "Trump America AI Act," which would require AI platforms to implement restrictions and privacy controls for minors.
Developers are also moving: OpenAI said in January it would roll out a version of ChatGPT dedicated to health that will "securely" connect users' medical records and wellness apps and that "ChatGPT Health will operate as a separate space with 'enhanced privacy to protect sensitive data.'"
The AMA's initiative puts a clear set of asks before lawmakers and regulators: statutory limits on clinical activity by chatbots, mandatory device review for diagnostic or treatment behavior, and legal protections for how sensitive health data are collected, stored, and disclosed. Those requests — framed around children's safety, the limits of consumer tools, and the technical fragility of cloud‑based systems — set a narrow, concrete agenda for the co‑chairs who received the letters and for any committee that takes up AI and health policy.
Source: https://www.govinfosecurity.com/doctor-lobby-urges-congress-to-set-ai-chatbot-safeguards-a-31495
