​The Supreme Court of India has stepped into the frontier of legal technology by addressing a dangerous new trend: the use of Artificial Intelligence to manufacture non-existent judicial precedents. During a recent hearing, the bench expressed deep concern over instances where "hallucinated" judgments—created by large language models—were presented as legitimate authorities during legal arguments. To counter this, the Court has called upon the Bar Council of India (BCI) to constitute a high-level expert panel to examine the technological, ethical, and regulatory dimensions of AI in the legal profession.
​The issue of AI "hallucinations"—where an AI tool confidently provides false information—has moved from a technical quirk to a serious threat to the rule of law. In several jurisdictions globally, and now reportedly within India, lawyers have inadvertently (or otherwise) cited cases that do not exist, complete with fabricated citations and convincing judicial reasoning. The Supreme Court noted that such practices not only mislead the bench but also waste judicial time and undermine the "adversarial system," which relies on the accuracy of cited law.
​The Court’s directive to the BCI emphasizes that the legal fraternity needs a clear framework for the use of AI tools. While acknowledging that AI can be a powerful asset for legal research, translation, and administrative efficiency, the bench warned that it cannot replace the "human element" of professional responsibility. The proposed expert panel is expected to include legal scholars, tech experts, and senior advocates who will draft guidelines on the permissible use of generative AI in drafting pleadings and citations.
​One of the primary concerns highlighted by the Court is the "duty of an officer of the court." Every advocate has an ethical obligation to verify the authenticity of the authorities they cite. The Court suggested that reliance on AI-generated summaries without manual verification could potentially constitute professional misconduct. The BCI has been tasked with defining these boundaries and perhaps introducing mandatory "AI literacy" modules as part of Continuing Legal Education for practitioners.
​This judicial intervention aligns with a broader global movement toward "Human-in-the-Loop" legal tech. The Supreme Court’s proactive stance aims to ensure that technology serves as a tool for justice rather than a source of misinformation. The bench remarked that while the legal profession must adapt to the 21st century, the foundational principles of "veracity and accountability" must remain non-negotiable.
​The expert panel will likely explore several key areas, including:
- ​Certification of AI Tools: Establishing a list of verified legal databases and AI platforms that meet judicial standards.
- ​Disclosure Mandates: Whether lawyers should be required to disclose if a particular draft or research was assisted by generative AI.
- ​Liability Frameworks: Who bears the responsibility when an AI tool produces a fake judgment—the lawyer, the law firm, or the software developer?
​As the BCI begins the process of forming this committee, the legal community is being urged to exercise extreme caution. The Supreme Court’s message is clear: the convenience of a "click-of-a-button" summary cannot override the rigorous tradition of legal research. The evolution of AI in law is inevitable, but its integration must be guarded by a robust regulatory shield to protect the sanctity of the courtroom.
​In conclusion, the Supreme Court’s call for a BCI panel marks a historic turning point in Indian legal history. It recognizes that in the age of deepfakes and digital hallucinations, the judiciary must be the ultimate guardian of truth. For legal practitioners, the era of "trust but verify" has officially arrived, where the "verify" part is now a mandatory constitutional and professional duty.