Sullivan and Cromwell, fake citations, and why lawyers still need human review

# Sullivan and Cromwell, fake citations, and why lawyers still need human review

In April 2026, Sullivan & Cromwell, a top US law firm, submitted a bankruptcy filing riddled with AI-generated citations. The incident underscored the critical need for human oversight in AI-assisted legal processes.

Understanding AI Hallucinations

AI hallucinations refer to instances where AI systems produce information that is incorrect or nonsensical. This phenomenon poses significant risks, especially in legal contexts. Such hallucinations can stem from issues within the AI's training data or from algorithmic errors. When AI models are trained on incomplete or biased datasets, they may generate outputs that deviate from reality. Similarly, flaws in the algorithms that process this data can lead to erroneous conclusions.

Defining AI Hallucinations

AI hallucinations occur when an AI system confidently presents false or fabricated information. Unlike human errors, which might be obvious due to context or style, AI-generated errors can appear highly plausible. These errors often arise from the AI's reliance on patterns it identifies in data, rather than an understanding of the content's accuracy. Training data, if not meticulously curated, can lead to such hallucinations. For example, if legal texts fed into an AI system contain inaccuracies, the AI might replicate these errors.

Examples in Legal Settings

In the legal field, AI hallucinations can have severe consequences. A misinterpreted precedent or a fabricated statute could lead to flawed legal advice, impacting case outcomes. The 2026 incident involving Sullivan & Cromwell highlighted this risk when their bankruptcy filing included multiple fake citations generated by AI. Such errors not only undermine the credibility of legal documents but also expose firms to potential liabilities. The legal industry must remain vigilant to these risks, ensuring that AI tools are used as aids, not replacements, for human expertise.

The Sullivan & Cromwell Case Study

Incident Overview

In April 2026, Sullivan & Cromwell, one of the most prestigious law firms in the United States, faced an unexpected crisis. The firm submitted a bankruptcy filing that contained multiple fake citations, all generated by an AI system used for drafting legal documents. This incident highlighted a growing concern within the legal industry: the reliability of AI in producing accurate and trustworthy outputs. The AI, intended to streamline the preparation process, instead introduced errors that could have significant legal repercussions. Such errors are often the result of AI hallucinations, where the system produces content that appears plausible but is factually incorrect.

Immediate Consequences

The immediate fallout from the incident was significant. Upon discovery, Sullivan & Cromwell had to retract the filing and issue a corrected version, an action that not only delayed proceedings but also risked damaging their reputation. Clients and partners questioned the firm's reliance on AI for critical tasks. The incident prompted a review of their AI usage policies and underscored the necessity of integrating robust human oversight into AI-assisted processes. This case serves as a cautionary tale, illustrating the potential consequences of over-reliance on AI without adequate safeguards. The firm's experience is a stark reminder that while AI can offer efficiency, its outputs must be meticulously verified to prevent costly errors.

Human Oversight in AI-assisted Legal Work

AI technologies have become integral to legal work, yet human oversight remains crucial. Lawyers play an indispensable role in reviewing AI-generated content to ensure both accuracy and legal compliance.

The Role of Lawyers

Lawyers are essential in the AI-assisted legal process. Their expertise allows them to identify errors that AI might overlook. While AI can efficiently analyze large volumes of data, it lacks the nuanced understanding required for complex legal interpretations. Human oversight can catch AI errors that technology alone might miss, such as misapplied legal precedents or fabricated citations. For instance, in the Sullivan & Cromwell case, human review could have identified the AI-generated inaccuracies before submission. Legal professionals ensure that AI outputs align with current laws and ethical standards, safeguarding against potential liabilities.

Challenges in Implementation

Despite its importance, implementing effective human oversight presents challenges. Lawyers must balance AI's efficiency with the need for thorough review, often under tight deadlines. Training legal teams to work effectively with AI systems is another hurdle. Many professionals require additional skills to interpret AI outputs critically. Furthermore, integrating human oversight into existing workflows can be resource-intensive, necessitating changes in both process and culture. Organizations must invest in continuous education and develop robust protocols to facilitate seamless collaboration between AI systems and human reviewers. These efforts are vital to maintaining the integrity of AI-assisted legal tasks.

The Role of AI Governance

Robust AI governance frameworks are necessary to minimize risks associated with AI hallucinations. These frameworks establish clear guidelines for AI use, ensuring that AI systems operate within defined boundaries and meet legal and ethical standards.

Key Elements of AI Governance

Effective AI governance involves several key elements. First, comprehensive policies must be established to guide the deployment and use of AI technologies. These policies should outline acceptable use cases, data privacy requirements, and security measures. Organizations like the European Union Agency for Cybersecurity (ENISA) emphasize the importance of clear guidelines to safeguard against AI-related risks.

Second, regular audits and assessments are crucial. These evaluations help identify potential vulnerabilities in AI systems and ensure compliance with established guidelines. The AI Act, proposed by the European Commission, mandates periodic reviews of AI applications in high-risk areas, including legal services, to maintain accountability.

Lastly, transparency in AI operations is vital. Organizations must document AI decision-making processes and provide explanations for AI-generated outputs. This transparency helps build trust and facilitates human oversight, reducing the likelihood of AI hallucinations.

Legal Industry Practices

The legal industry has begun implementing governance practices to address AI-related challenges. Law firms are increasingly adopting AI governance frameworks to regulate the use of AI tools in legal processes. These frameworks often include protocols for human review of AI-generated documents, ensuring that any erroneous outputs are detected and corrected.

Some firms have also established AI ethics committees. These committees monitor AI applications and provide guidance on ethical considerations, aligning AI use with the firm's values and legal obligations. For example, the UK Law Society has issued guidelines encouraging firms to adopt AI governance practices that emphasize human oversight and accountability.

In conclusion, robust AI governance frameworks are essential for minimizing the risks associated with AI hallucinations. By setting clear guidelines and implementing best practices, the legal industry can harness the benefits of AI while maintaining control and compliance.

Training AI for Legal Accuracy

Effective AI training is vital to minimizing hallucinations in legal contexts. This requires attention to both the quality of training data and the refinement of algorithms.

Training Data Considerations

The foundation of accurate AI output lies in well-curated datasets. In legal applications, the datasets must include a wide range of case law, statutes, and legal commentary to ensure comprehensive understanding. Poorly curated datasets can lead to AI systems generating incorrect citations or misinterpreting legal precedents. For example, an AI trained on outdated or biased legal texts may produce misleading results, impacting case outcomes.

Ensuring dataset diversity is also crucial. This means incorporating legal materials from different jurisdictions and time periods. Such diversity helps AI models generalize better and reduces the risk of hallucinations. Quality control measures, including regular updates and expert reviews, further enhance dataset reliability.

Algorithmic Adjustments

Beyond data, algorithmic adjustments play a critical role in legal AI accuracy. Algorithms must be designed to prioritize precision over speed, especially in tasks like drafting legal documents or analyzing case law. Fine-tuning algorithms to understand legal language nuances can prevent common errors.

Regular algorithm audits are essential. These audits identify potential biases or inaccuracies, allowing for timely corrections. Moreover, integrating feedback loops from human legal experts can improve algorithmic performance. Such feedback helps refine the AI's decision-making process, aligning it more closely with legal standards and practices.

In conclusion, reducing AI hallucinations in legal contexts hinges on meticulous training and dataset management. By focusing on data quality and algorithmic precision, legal professionals can harness AI's potential while maintaining accuracy and compliance.

Future of AI in the Legal Field

AI technologies are reshaping the legal landscape, offering potential benefits that could redefine traditional practices. However, these advancements come with ongoing risks that necessitate vigilant oversight.

Potential Benefits

AI has the capacity to streamline routine legal tasks, allowing legal professionals to focus on more complex matters. Document review, due diligence, and contract analysis can be expedited with AI tools, reducing the time and cost associated with these processes. For example, AI systems can quickly sift through thousands of documents in e-discovery, identifying relevant information much faster than human teams. This efficiency not only saves time but also enhances productivity across legal departments.

Moreover, AI can assist in predictive analytics, offering insights into case outcomes based on historical data. This capability allows lawyers to develop more informed strategies, aligning their advice more closely with likely scenarios. Such predictive tools can be particularly valuable in litigation, where anticipating the opposition's moves and understanding potential verdicts can significantly impact case management.

Ongoing Risks

Despite these benefits, the integration of AI in legal services presents significant risks that must be managed. AI systems can generate errors, such as hallucinations, which pose a threat to the accuracy and reliability of legal documents. The Sullivan & Cromwell incident, with its AI-generated fake citations, exemplifies these risks. Without proper oversight, such errors could lead to severe legal and reputational consequences.

Furthermore, AI's reliance on historical data can perpetuate existing biases, potentially influencing legal decisions in unintended ways. Ensuring that AI systems are trained on diverse and representative datasets is crucial to mitigating these biases. Continuous monitoring and adjustment of AI algorithms are essential to maintain fairness and compliance with evolving legal standards.

In conclusion, while AI offers promising advantages for the legal field, it is imperative to balance these with robust oversight mechanisms. By doing so, the legal industry can harness AI's potential while safeguarding against its inherent risks.

Conclusion: Balancing AI Use with Human Expertise

AI technologies continue to evolve, offering promising capabilities for the legal field. However, the integration of AI into legal practice requires a balanced approach, combining AI's efficiency with the critical oversight of human expertise.

Integrating Human-in-the-Loop Systems

Human-in-the-loop systems are crucial for enhancing the reliability of AI outputs. These systems involve human intervention at critical stages of AI processes, ensuring that AI-generated information is accurate and compliant. For example, legal professionals can review AI-generated documents to verify citations and legal interpretations. This approach not only reduces the risk of AI errors but also ensures that the final output aligns with legal standards and practices.

Building Trust in AI

Building trust in AI systems is essential for their successful adoption in legal services. Trust can be fostered through transparency about AI processes and outcomes. Clearly defined governance frameworks help establish guidelines for AI use, ensuring accountability and mitigating risks. By demonstrating AI's reliability and maintaining rigorous oversight, legal professionals can confidently integrate AI tools into their workflows.

As AI continues to play a significant role in legal services, maintaining a balanced approach that combines AI capabilities with human expertise is vital. This synergy not only enhances the efficiency of legal processes but also safeguards against potential pitfalls of AI missteps. Velatir supports companies in navigating this balance, providing the tools needed to ensure compliance and maintain human oversight.