Can technology augment order writing capacity at regulators?

by Natasha Aggarwal, Satyavrat Bondre, Amrutha Desikan, Bhavin Patel and Dipyaman Sanyal.

Indian regulators have extensive quasi-judicial powers that they express through adjudicatory orders. It is critical that these powers are exercised in a proportionate, legitimate, and well-reasoned manner, as they not only impact the persons directly involved, but also the wider ecosystem in which they operate. Arbitrary actions, unsubstantiated by clearly articulated reasoning, can raise serious concerns around the legitimacy of regulatory actions and lead to a loss of confidence in the regulator. Such actions may also be set aside by appellate and review fora. Clearly written, well-researched, and reasoned orders help provide clarity, predictability, and knowability of the law, which are key indicators of a rule of law system (Aggarwal, Patel and Singh, 2025). Our study of the state of Indian regulatory order writing shows there is room for improvement in this regard.

We notice a growing interest in the use of Generative Artificial Intelligence (Gen AI) to resolve procedural inefficiencies at quasi-judicial and judicial authorities in India (Supreme Court Committee on AI, 2025; Kerala High Court, 2025), coupled with concerns around the potential dangers of using such technologies without adequate safeguards. Against this background, in a new working paper titled, 'Can technology augment order writing capacity at regulators?' we critically examine the opportunities and challenges of using technology, in particular Large Language Models (LLMs), to assist regulatory order writing in quasi-judicial settings.

The paper proposes augmenting rather than replacing human decision-makers, aiming to improve regulatory order writing practice through responsible use of LLMs. It identifies the core principles of administrative law that must be upheld in these settings - such as application of mind, reasoned orders, non-arbitrariness, rules against bias, and transparency - and analyses how inherent limitations of LLMs, including their probabilistic reasoning, opacity, potential for bias, confabulation, and lack of metacognition, may undermine these principles.

While the available Indian literature on the topic focuses largely on these limitations, and on critiquing proposals based on an over-reliance on technocratic means to improve state capacity, this paper's contribution lies in its integrative work: we draw upon the design principles articulated in frameworks developed in other jurisdictions and relate them to the applicable principles of Indian administrative law. We use this synthesis to develop a Problem-Solution-Evaluation (PSE) framework that is attentive to international practice, the legal principles underpinning quasi-judicial decision-making in India, and problems and limitations inherent to GenAI and LLMs.

The PSE framework proposed in the paper maps specific technical, design, and systemic solutions to each identified risk, and outlines evaluation strategies - end-to-end, component-wise, human-in-the-loop, and automated - to ensure ongoing alignment with legal standards. An overview of the framework is set out in the table below:

Table 1: Applying the Problem-Solution-Evaluation framework. This table illustrates how the PSE framework can be operationalised to align the design, development and use of LLMs for order writing assistance with the requirements of Applicable Law.

Problem	Applicable law	Solution	Evaluation
Non-application of mind	Non-application of mind; Failure to provide reasons; Arbitrariness	Interface Checkpoints; Confidence Score Display; Dual-Prompt Pipelines; Functionality Limitation; Constraint Enforcement; Workflow Design for; Review Role-Based Access	Edit Rate; Turnaround Time (TAT); Prompt Divergence Rate; Coherence Score
Black-box problem	Failure to provide reasons; Transparency	Chain-of-thought prompting; Input Token Influence Identification Symbolic Reasoning Systems Traceability tools; Visualisation; Simplified model explanations	Clarity rating; Audit Trail Incidence Document Traceability Rate
Potential for bias	Rules against bias; Arbitrariness	Data Preprocessing; Bias penalisation; Domain-specific content filters; Automated Bias Flagging Tools; Establishment of Legal Fairness Criteria; Mandatory Periodic Benchmarking	Bias Flag Rate Override Percentage; Fairness Benchmark Scores
Confabulation problem	Non-application of mind; Failure to provide reasons; Arbitrariness	Retrieval Augmented Generation; Post-Generation Verification; Legal Knowledge Graph Integration; Mandatory reviewer verification; Watermarking for traceability; Communicate technical limitations	Secondary LLM ''Judge'' for Fact-Checking; End-to-End Evaluation Tools Hallucination Rate; Retrieval Precision@k/ MRR NLI Coherence Checks; Self-Consistency Rate
Lack of metacognition	Non-application of mind; Arbitrariness	Prompt engineering; LLM as a judge; Iterative improvement from feedback	Closeness Metric; Human evaluation on overconfidence in output
Training corpus	NA	Adaptive Scraping Frameworks; Sector-specific pre-training; Structured Entity; Extraction and Legal Knowledge Graphs; Isolated Model Containers; Source inclusion; Perplexity tracking; Legal Retrieval Benchmarking; Curate sector-specific legal databases	Crawl coverage; OCR Error Reduction; Validation perplexity; Retrieval lift
Data security and privacy	NA	Stringent access control; Synthetic supervision-based PII detectors; NLP filters for information masking; Isolated Model Containers; On-premise infrastructure	Unauthorised access attempts; Mean Time To Remediation (MTTR); Penetration Test Pass Rate; PII Detection Accuracy

By itself the framework may be insufficient. It must be supplemented with systemic measures taken at the regulatory level. We offer stage-wise recommendations on how LLM-based order review tools can be built for and used in regulatory adjudication.

References

Natasha Aggarwal, Bhavin Patel, and Karan Singh, "A Guide to Writing Good Regulatory Orders" [2025] Trustbridge Rule of Law Foundation Working Papers.

Anurag Bhaskar and others, "White Paper on Artificial Intelligence and Judiciary" Centre for Research and Planning, Supreme Court of India, 2025.

High Court of Kerala, "Policy Regarding the Use of Artificial Intelligence (AI) Tools in District Judiciary" Official Memorandum HCKL/7490/2025-DI-3-HC Kerala, 2025.

Natasha Aggarwal, Amrutha Desikan and Bhavin Patel are researchers at the TrustBridge Rule of Law Foundation. Satyavrat Bondre and Dipyaman Sanyal work on AI and technology at dōnō consulting.