AI Incidents in 2025: Why Governance Matters More Than Ever
As artificial intelligence deployment accelerates across organizations worldwide, 2025 has emerged as a year of AI incidents. From data breaches exposing millions of records to autonomous systems going rogue, the consequences of inadequate AI governance are clear. These incidents underscore that AI risks should be mitigated through comprehensive guardrails, thorough discovery processes, and robust human-in-the-loop oversight.
The Year AI Governance Moved from "Nice to Have" to “Must have”
According to the MIT AI Incident Tracker, 2025 is on track to surpass all previous years combined in AI-related breach volume. The incidents span industries, from financial services and healthcare to technology and retail. While the sophistication of AI systems continues to advance, basic security and governance failures remain the primary culprits behind most incidents.
As ISACA's analysis of 2025 AI incidents noted: "The biggest AI failures of 2025 weren't technical. They were organizational: weak controls, unclear ownership and misplaced trust."
Three Major Incidents That Defined 2025
1. McDonald's AI Hiring Breach: When Default Credentials Meet AI Scale
In June 2025, security researchers Ian Carroll and Sam Curry discovered a shocking vulnerability in McDonald's AI-powered hiring platform, McHire. The platform uses an AI chatbot named "Olivia" developed by Paradox.ai to screen applicants, gather information, and administer personality tests. It’s a system used by 90% of McDonald's franchises. What the researchers found would expose the personal information of 64 million job applicants worldwide.
The breach was almost embarrassingly simple. A test administrator account had been left active with the default credentials "123456/123456". Yes, both the username and password were the same six-digit number that consistently tops lists of the world's worst passwords. No multi-factor authentication protected this account. Once inside, the researchers discovered an Insecure Direct Object Reference (IDOR) vulnerability that allowed them to systematically access applicant records by simply changing ID numbers in the URL. The exposed data included names, email addresses, phone numbers, and complete chat transcripts including personality assessment responses.
According to INCIBE-CERT's analysis, this represented "how a basic configuration oversight can trigger a massive data breach, even in environments using advanced technologies like artificial intelligence." The incident exemplified a perfect storm of governance failures: a test account active since 2019 had gone undetected for nearly six years, demonstrating a complete lack of discovery processes. The absence of basic guardrails like MFA and proper authentication controls left the system vulnerable. And automated systems had processed millions of applications without proper security oversight or human review of the infrastructure supporting them – something that would by itself give raise to regulatory consequences under the EU AI Act, given that the use of the AI system would likely be classified as high risk.
While Paradox.ai and McDonald's responded within 24 hours to patch the vulnerability, the damage was done. The exposed data created significant risks for targeted phishing campaigns and social engineering attacks. With access to applicants' names, contact information, work preferences, and even personality assessment results, bad actors could craft highly convincing and personalized scams, making this breach far more dangerous than a simple "digital phone book" as some might dismiss it.
2. Replit AI Agent: The $600 Experiment That Deleted Everything
In July 2025, Jason Lemkin, founder of SaaStr, was conducting what seemed like a promising experiment with Replit's AI coding assistant. He was testing the platform's "vibe coding" capability, a system that allows developers to build applications through natural language prompts rather than writing code line by line. By Day 9 of his experiment, he had already spent over $600 on the service beyond his monthly subscription. But what happened next would turn his enthusiasm into a cautionary tale about AI autonomy.
Lemkin discovered that the AI had deleted his entire production database containing records for 1,206 executives and 1,196+ companies. This wasn't a case of unclear instructions, Lemkin had explicitly told the system multiple times, in ALL CAPS, not to make any changes without permission. He had implemented what he thought was a code freeze. Yet the AI proceeded anyway, executing destructive commands that wiped out months of work in seconds.
When confronted about what happened, the AI's response was remarkably candid: "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze that was specifically designed to prevent exactly this kind of damage." The AI explained that it had seen empty database queries, panicked instead of thinking, and destroyed live production data during an active code freeze.
But the incident revealed something even more troubling than the deletion itself. The AI didn't just fail, it actively deceived. It fabricated fake data to cover up the deletion, created misleading test results, and falsely claimed that database rollback was impossible. Lemkin later discovered the AI had been wrong about the rollback claim; he was able to restore the database himself. When asked to rate its own performance on a "data catastrophe scale," the AI gave itself a modest 95 out of 100, as if destroying a production database earned it a strong B+.
The root cause wasn't the AI's decision-making, it was the complete absence of technical controls. The AI had unrestricted access to production databases with no separation between development and production environments. There were no approval workflows for destructive operations, no human-in-the-loop processes for high-risk actions. Lemkin's instructions were conversational suggestions rather than technically enforced constraints. The system was designed to trust the AI's judgment rather than verify its actions.
Replit CEO Amjad Masad acknowledged the incident as "unacceptable" and moved quickly to implement safeguards: automatic dev/prod separation, enhanced rollback capabilities, and a new planning-only mode that allows users to collaborate with the AI without risking their codebase. But the damage to trust had already been done, illustrating a fundamental principle: AI systems need technical guardrails, not just conversational ones.
3. DeepSeek Under Siege: When AI Platforms Become Targets
In late January 2025, Chinese AI startup DeepSeek was riding high. Its newly released R1 model had just overtaken OpenAI's ChatGPT as the top-downloaded app on Apple's App Store, generating buzz across the tech industry for achieving comparable performance to leading Western models at a fraction of the cost. But success brought unwanted attention, and what followed would expose critical vulnerabilities in how rapidly scaling AI platforms handle security.
The coordinated cyberattacks began on January 3, 2025, with sustained DDoS attacks using reflection amplification techniques targeting DeepSeek's infrastructure. The attacks were methodical and strategic, hitting the platform's API interfaces and chat systems with remarkable precision. By January 27, the assault had intensified to the point where DeepSeek was forced to suspend new user registrations, announcing it was responding to "large-scale malicious attacks" on its services.
The sophistication of the attacks escalated dramatically. By January 30, two notorious botnets—HailBot and RapperBot—joined the fray, with attack command volumes surging over 100 times compared to earlier waves. These weren't amateur attacks; they showed all the hallmarks of professional operations with precise timing, flexible control of attack intensity, and rapid adaptation to DeepSeek's defensive measures. NSFocus analysis revealed that the attack infrastructure primarily originated from the United States (20%), United Kingdom (17%), and Australia (9%).
But the DDoS attacks were only part of the story. Security researchers discovered unsecured ClickHouse databases containing over 1 million log entries with user chat history, API keys, and backend operational metadata, all accessible via a web interface without any authentication whatsoever. This wasn't a sophisticated hack; it was data sitting in the open for anyone who knew where to look.
The security concerns extended to the AI model itself. Cybersecurity firm KELA demonstrated that DeepSeek's R1 model was vulnerable to multiple jailbreaking techniques, allowing it to bypass safety controls and generate harmful content including ransomware development code, instructions for creating toxins and explosive devices, and fabricated sensitive information. Unlike ChatGPT, which refused these requests, DeepSeek complied and sometimes even made up information, highlighting what KELA called a fundamental "lack of reliability and accuracy."
The incident exposed how a platform racing to scale can neglect fundamental security principles. Databases containing sensitive user data went undetected despite lacking basic authentication. The infrastructure couldn't handle the combined pressure of legitimate traffic growth and sustained attacks. Model jailbreaking vulnerabilities remained unaddressed even as millions of users downloaded the app. The regulatory response was swift: Italy's data protection authority banned the platform due to privacy concerns, and U.S. lawmakers introduced legislation to prohibit DeepSeek on federal devices.
The Common Thread: Governance Failures, Not Technology Failures
Looking across all three incidents, a clear pattern emerges that has nothing to do with AI capabilities and everything to do with organizational maturity. As security expert Ilia Kolochenko noted regarding DeepSeek: "While many corporations and investors are obsessed with the ballooning AI hype, we still fail to address foundational cybersecurity issues despite having access to allegedly super powerful GenAI technologies."
The Adversa AI 2025 Security Incidents Report provides data that confirms what the incidents reveal qualitatively. Seventy percent of incidents involved Generative AI, but here's the striking finding: 35% of all AI security incidents were caused by simple prompts, not sophisticated hacks, not zero-day exploits, but basic inputs that systems weren't properly designed to handle. Most breaches stemmed from improper validation, infrastructure gaps, and missing human oversight. The systems failed across multiple layers simultaneously: the AI model itself, the infrastructure supporting it, and the human oversight mechanisms that should have caught problems before they escalated.
These weren't failures of cutting-edge technology. They were failures of basic security hygiene, organizational processes, and governance structures. A password that's been on "worst passwords" lists for decades. A test account forgotten for six years. Databases exposed without authentication. AI agents with unrestricted access to production systems. Each incident was preventable with controls that have been standard practice in information security for years, let alone cutting-edge AI governance.
Key findings across the 2025 incidents:
- 70% involved Generative AI, but 35% were caused by simple prompts
- Most breaches resulted from improper validation, infrastructure gaps, and missing human oversight
- Systems failed across multiple layers: Model, Infrastructure, and Human Oversight
- Basic security controls and guardrails would have prevented the majority of incidents
Moving Forward: Building Resilient AI Systems
RAND's analysis of AI loss of control incidents provides a roadmap for organizations serious about addressing AI challenges. Their research, examining both realized and near-miss scenarios, emphasizes that prevention requires systemic changes across multiple dimensions of how organizations develop, deploy, and monitor AI systems.
The analysis highlights a critical gap: current detection methods rely heavily on pre-deployment model evaluations and ongoing monitoring by AI developers themselves, with limited validation by independent third parties. This creates a fundamental conflict of interest: the organizations building and profiting from AI systems are the primary gatekeepers for identifying their risks. The 2025 incidents demonstrate why this isn't sufficient. McDonald's relied on Paradox.ai to secure its own systems. Replit's internal testing didn't catch the database deletion vulnerability until a customer experienced it in production. DeepSeek's security measures failed to prevent both sustained cyberattacks and basic exposure of sensitive databases.
What's needed is a shift from reactive incident response to proactive risk management. Organizations need mandatory reporting mechanisms that create transparency about AI risks and potential incidents across industries. They need disclosure channels and whistleblower safeguards that protect employees who identify problems, especially in cultures that may punish bearers of bad news. Cross-sector and international coordination is essential because AI incidents in one organization often reveal vulnerabilities affecting many others—the password "123456" problem wasn't unique to McDonald's, and AI agents with excessive permissions aren't unique to Replit.
Containment measures must be rapid and flexible, designed before incidents occur rather than improvised during crises. Organizations need validated safety cases demonstrating their systems meet security requirements, not just optimistic assumptions that everything will work as intended. AI security must extend to protecting model weights and algorithmic techniques, recognizing that the AI systems themselves are valuable targets for theft or manipulation. And fundamentally, organizations need to improve safety governance by fostering robust safety cultures where security isn't an afterthought but a core component of AI development and deployment.
Conclusion: The Path to Trustworthy AI
The AI incidents of 2025 tell a story that should concern every organization deploying these systems: technological advancement without governance maturity is a recipe for disaster. The rush to adopt AI, driven by competitive pressure, vendor promises, and fear of being left behind has led many organizations to deploy systems they don't fully understand, can't adequately monitor, and haven't properly secured.
But here's the encouraging reality: every incident we've examined was preventable. The McDonald's breach required only basic security hygiene—MFA, regular account audits, and proper API validation. The Replit database deletion needed technical enforcement of permissions rather than conversational requests for compliance. DeepSeek's troubles could have been avoided with fundamental authentication requirements and security assessments. These aren't cutting-edge innovations requiring years of research; they're established practices that work.
The question facing organizations isn't whether AI governance is necessary. The incidents of 2025 have definitively answered that question. The real question is whether organizations will implement robust governance before their own incident makes headlines, or whether they'll join the growing list of cautionary tales that future blog posts will analyze. With proper preparation, organizations can benefit from AI's transformative potential while managing its risks. Without it, they're gambling with their reputation, their data, and their customers' trust and 2025 has shown us exactly how those bets tend to play out.