A stolen credit card can be relatively easy to cancel and replace. Leaked medical data cannot. In the United States alone, consumer losses from identity theft linked to just four major breaches involving data brokers have been estimated at $20.9 billion, according to a 2026 report by the U.S. Senate Joint Economic Committee. Yet an increasing number of people are turning to AI chatbots with questions about symptoms, test results and possible treatments.
The trend has also caught the attention of the largest technology providers, which are investing heavily in generative AI solutions for healthcare. Tools such as Copilot Health, ChatGPT Health and Amazon’s Health AI assistant are expected to help users interpret medical documentation and ask questions about symptoms, laboratory results and available treatment options.
Although such services are not yet widely available in Poland, many internet users are already relying on general-purpose AI chatbots to analyse symptoms, interpret test results or search for information about treatment.
This is hardly surprising. At a time when healthcare systems face shortages of specialists and long waiting times for appointments, a chatbot available around the clock can become the first source of information for many people. The problem is that convenience does not always go hand in hand with safety.
A study conducted by the University of Oxford and published in Nature Medicine found that users often did not know what information they needed to provide for an AI model to correctly interpret their situation. In the experiment, participants correctly identified their health condition in only around one-third of cases after interacting with a chatbot. Just 43% made the right decision about what to do next.
Researchers also observed that chatbots could give significantly different answers to questions that differed by only a few words. In addition, the models mixed useful recommendations with potentially harmful advice, while study participants struggled to tell the difference.
However, experts stress that the risks are not limited to inaccurate answers. The privacy of information shared with chatbots is equally important.
“Many people treat a conversation with a health chatbot as if it were a conversation with a doctor. That is a mistaken assumption. In the case of many consumer AI services, the user is not using a tool covered by medical confidentiality, but a commercial application whose rules for processing data may differ significantly from the standards applicable in healthcare,” says Kamil Sadkowski, a cybersecurity analyst at ESET.
In practice, this means that information entered into a publicly available chatbot may be stored, analysed or used to further develop AI models, depending on the policies of the service concerned. In the event of a security incident or data breach, this information may also end up in the wrong hands.
Medical data is especially attractive to cybercriminals. Unlike a payment card number or password, it cannot simply be changed. It can be used for identity theft, medical service fraud, blackmail attempts or highly targeted phishing campaigns.
“Medical history, test results and information about treatment are among the most sensitive types of data that can be shared online. A criminal who knows details about a victim’s health condition can use that knowledge to prepare exceptionally convincing phishing messages or attempts to obtain even more information,” Sadkowski says.
Experts point out that the risk increases with the number of organisations processing a user’s data. The more companies have access to health-related information, the more potential points exist where an error, misuse or security incident may occur.
The situation is further complicated by the fact that many consumer AI tools are not subject to the same strict regulations as healthcare providers.
“That is why, before using an AI chatbot for health-related matters, it is worth taking a moment to check the privacy policy and data-processing rules. Users should pay particular attention to whether the information they provide may be used to train models and whether it is shared with third parties,” Sadkowski adds.
AI chatbots may be useful for finding general information or preparing questions for a medical appointment. They should not, however, be treated as a substitute for professional medical advice, diagnosis or treatment. Just as importantly, users should think carefully before entering identifiable medical records, test results or detailed health histories into a consumer AI service.
