KEY POINTS
- AI chatbots now shape the daily lives of millions worldwide.
- They act as assistants, creative partners, even pseudo-therapists.
- Users often assume chats are confidential, but risks remain.
- Chatbots collect prompts, files, images, audio, and metadata.
- Personal details like names and payment data are stored.
- Only 22 percent of users knew about opt-out settings.
- Sam Altman said chats are not legally protected.
- Technical flaws expose sensitive information through bugs and leaks.
Artificial Intelligence (AI) chatbots—a computer programme that understands and responds to human conversations in a natural, interactive way—like ChatGPT, Copilot, DeepSeek, Perplexity, and Gemini, among others, have become an indispensable part of the routine lives of millions of users worldwide.
These chatbots act as personal assistants, creative partners, and even pseudo-therapists.
However, while these platforms offer unprecedented convenience, their data processing practices raise questions about how user data is collected, stored, and used. Users often assume that their conversations are private. But are they?
Audit reveals policy gaps
AI chatbots collect a wide range of user data, including text prompts, uploaded files, images, audio, and metadata such as IP addresses, device information, and geolocation.
AI chatbots collect a wide range of user data, including text prompts, uploaded files, images, audio, and metadata such as IP addresses, device information, and geolocation.
Users also provide personal details like names, email addresses, and payment information at the time of account registration.
OpenAI’s Privacy Policy states that this data may be used to improve model performance, train future versions, detect abuse, and ensure safety.
Unless users explicitly opt out, the content of their conversations, including sensitive or personal content, can be reviewed by human trainers or used for reinforcement learning.
A 2024 audit by Nightfall AI revealed that 63 per cent of ChatGPT user data contained personally identifiable information (PII), while only 22 per cent of users were aware of opt-out settings.
The lack of granular control over what is collected and how it is used raises serious transparency concerns.
A 2024 audit by Nightfall AI revealed that 63 per cent of ChatGPT user data contained personally identifiable information (PII), while only 22 per cent of users were aware of opt-out settings.
However, for users concerned about this, OpenAI, the company behind ChatGPT, offers some safeguards, such as allowing users to turn off chat history, and enterprise and education customers have the option of entering into zero-data retention agreements.
False sense of privacy
Many users assume that AI chatbots offer the same level of confidentiality as encrypted messaging apps or private consultations with professionals. Many users treat ChatGPT as a private, even therapeutic tool. Some go as far as sharing highly personal information, expecting confidentiality.
However, OpenAI Chief Executive Officer Sam Altman recently clarified that conversations with ChatGPT are not legally protected and should not be considered private.
In a July 2025 podcast, Altman warned that deeply personal chats, often used as therapy or coaching, could be summoned in legal proceedings.
OpenAI Chief Executive Officer Sam Altman recently clarified that conversations with ChatGPT are not legally protected and should not be considered private.
This misconception is dangerous. Unlike Signal or WhatsApp, AI chatbots do not offer end-to-end encryption or legal protections. Users who share financial records, medical details, or proprietary business strategies risk unintended exposure.
ChatGPT logs are stored on centralised servers, where authorised personnel and legal authorities can access them under certain circumstances.
Privacy risks highlighted
In 2025, a US court, in an ongoing copyright lawsuit filed by The New York Times, ordered OpenAI to preserve all user conversations, including those that had been deleted or temporary chats.
According to The Verge (an American news and media website), this order applies across all tiers of users: Free, Plus, Pro, and Team, unless the users are protected under enterprise-grade zero-retention agreements.
In 2025, a US court, in an ongoing copyright lawsuit filed by The New York Times, ordered OpenAI to preserve all user conversations, including those that had been deleted or temporary chats.
The court order overrides OpenAI’s previous policy of deleting chats permanently after 30 days unless flagged for abuse. The indefinite retention of sensitive content poses serious risks.
The company is appealing the ruling, arguing it creates unnecessary and disproportionate risks for user privacy.
Unclear legal landscape
The legal landscape surrounding AI chatbots remains unclear. Under the General Data Protection Regulation (GDPR), users have the right to erase their data, known as the “right to be forgotten.”
However, indefinite data retention and vague anonymisation practices may violate the regulation’s “storage limitation” principle. In 2023, Italy temporarily banned ChatGPT over GDPR concerns, and Poland launched an investigation into its data processing practices.
Ethically, the lack of informed consent is troubling. Users are often unaware that their data may be used for training or shared with third-party vendors. The absence of a unified global framework for AI privacy exacerbates these risks, especially in cross-border data transfers.
Bugs and accidental leaks
Beyond legal risks, technical vulnerabilities have highlighted the dangers of data exposure. In March 2023, a bug in ChatGPT’s infrastructure allowed users to view others’ chat titles and payment information.
Beyond accidental leaks, AI systems are vulnerable to ‘prompt injection’ attacks, where hackers manipulate inputs to extract confidential information.
Samsung employees inadvertently leaked proprietary code and internal documents while using ChatGPT for work tasks.
Beyond accidental leaks, AI systems are vulnerable to “prompt injection” attacks, where hackers manipulate inputs to extract confidential information. The problem is compounded by GPT Actions, which allow third-party apps to integrate with ChatGPT.
A study conducted by arXiv (a free online repository for sharing scientific research papers) found that numerous third-party plugins collect extensive user data, including passwords and behavioural patterns, often without sufficient user awareness.
Users who rely on ChatGPT for image generation should proceed with caution. Wired (a magazine and website about technology, culture, and innovation) reported that generated images often contain metadata, including GPS location or device details, which users may not realise they are sharing. This could expose personal information unintentionally.
Stronger privacy practices
To protect themselves, users should avoid sharing sensitive personal, financial, legal, or medical information with AI tools, including ChatGPT.
Users need to opt for “temporary chats” or disable chat history to prevent input from being stored and used for training.
To protect user privacy, companies should minimise data collection, apply differential privacy to anonymise inputs, use local AI models that avoid cloud transmission, and educate users about sharing risks through clear warnings and opt-out options.
