You can rest assured that your private ChatGPT conversations remain secure, unaffected by the recently reported breach of OpenAI’s systems. The hack itself, although concerning, seems to have been only surface-level. However, it serves as a reminder that AI companies have quickly become prime targets for hackers.
Following a recent podcast where former OpenAI employee Leopold Aschenbrenner made reference to the hack, The New York Times reported it in greater detail. Describing it as a “significant security incident,” the individual referred to the unauthorized access as a breach into an employee discussion forum, according to undisclosed sources within the company. (I contacted OpenAI to verify and obtain their input.)
Every security breach should be taken seriously, and gaining access to internal OpenAI development discussions can be quite valuable. However, the chances of a hacker gaining access to internal systems, ongoing models, confidential roadmaps, and similar sensitive information are quite low. 
However, it is important to be concerned about this matter, not solely due to the possibility of China or other adversaries surpassing us in the AI arms race. It’s undeniable that these AI companies now control access to an immense wealth of valuable data.
Now, let’s delve into three types of data that OpenAI and other AI companies have at their disposal: high-quality training data, bulk user interactions, and customer data, albeit to a lesser extent.
The companies are extremely secretive about their hoards of training data, making it difficult to determine exactly what they have. However, it would be incorrect to assume that they are merely vast collections of scraped web data. Indeed, web scrapers and datasets such as the Pile are utilized for this purpose. However, transforming the raw data into a format suitable for training a model like GPT-4o is an immense undertaking. This task demands a significant investment of human labor, as it can only be partially automated.
Many machine learning engineers have pondered the significance of dataset quality in the development of a large language model or any transformer-based system. It is widely speculated to be the most crucial factor among all the contributing elements. That’s precisely why a model trained on Twitter and Reddit will never match the eloquence of one trained on the vast collection of published works from the last century. (And perhaps the reason why OpenAI allegedly utilized sources of questionable legality, such as copyrighted books, in their training data—a practice they assert to have discontinued.)
The training datasets developed by OpenAI hold immense value for various stakeholders, including competitors, other companies, adversary states, and regulators within the United States. Wouldn’t it be important for the FTC or courts to have a clear understanding of the data being used and whether OpenAI has been transparent about it?
However, what truly sets OpenAI apart is the vast amount of user data it possesses. This data consists of countless conversations with ChatGPT covering a wide range of topics, likely amounting to billions of interactions. Similar to how search data used to be crucial for grasping the overall mindset of the internet, ChatGPT has a keen awareness of a specific population that may not be as vast as Google users but offers much greater insight. (Just so you know, unless you choose not to, your conversations are being utilized for training purposes.)
If you notice an increase in searches for “air conditioners” on Google, it indicates that the market is becoming more active. However, those users do not engage in a comprehensive discussion about their preferences, budget, home specifications, preferred manufacturers, and other relevant details. It’s clear that this information holds great importance, as even Google is actively encouraging users to share this kind of data by replacing traditional searches with AI interactions!
Consider the multitude of conversations individuals have had with ChatGPT and the invaluable insights it provides. This information is not only beneficial to AI developers but also to marketing teams, consultants, and analysts. It is a treasure trove of data.
The final category of data holds immense value in the open market; it pertains to how customers are actively utilizing AI and the data they have personally provided to the models.
Many major companies, as well as numerous smaller ones, rely on tools such as OpenAI and Anthropic’s APIs to accomplish a wide range of tasks. In order for a language model to be truly valuable, it typically needs to be fine-tuned or provided with access to the organization’s internal databases.
These could range from mundane items like outdated budget sheets or personnel records (to enhance their searchability, for example) to highly valuable assets like unreleased software code. How they utilize the capabilities of AI and whether they find them beneficial is up to them, but it’s important to note that the AI provider has exclusive access, similar to any other SaaS product.
These valuable trade secrets are now in the hands of AI companies, who have become central players in this domain. The novelty of this aspect of the industry brings along a unique risk, as AI processes are still not standardized or fully comprehended.
Just like any SaaS provider, AI companies have the ability to offer industry-standard levels of security, privacy, and on-premises options, and overall, they are committed to providing their service responsibly. I am confident that the private databases and API calls of OpenAI’s Fortune 500 customers are highly secure and well-protected. It is crucial for them to have a deep understanding of the potential risks associated with managing sensitive information within the realm of artificial intelligence. It is up to OpenAI to decide whether or not to report this attack, but their decision does not instill confidence in a company that is in dire need of it.
However, it is important to note that the value of what needs to be protected remains unchanged, regardless of the security measures in place. It is also crucial to acknowledge that there are constant threats from malicious individuals and adversaries who are constantly attempting to breach the defenses. Security goes beyond simply selecting the correct settings or ensuring your software is up-to-date, although, of course, these fundamentals are crucial as well. It’s a constant game of hide and seek that, interestingly enough, AI itself is now escalating. Agents and attack automators are thoroughly exploring every corner and crevice of these companies’ vulnerable areas.
There’s absolutely no need to worry. Companies that have access to large amounts of personal or commercially valuable data have been dealing with and successfully handling similar risks for a long time. However, AI companies present a more modern, youthful, and potentially more enticing opportunity compared to typical poorly set-up enterprise servers or untrustworthy data brokers. Even a security breach like the one mentioned above, which hasn’t resulted in any significant data leaks so far, should concern anyone who engages with AI companies. They have made themselves vulnerable to criticism. Expect to face criticism from anyone and everyone.
