Survey-based measurement of Indian courts

by Pavithra Manivannan, Susan Thomas, and Bhargavi Zaveri-Shah.

Public institutions do not face a market test. Achieving state capacity is about establishing checks and balances. The traditional idea is to instrument the operations, and construct an operational MIS, which is released into the public domain. Through this, deficiencies of the working of the organisation are visible to researchers and the public. The other pathway is to ask the persons who interact with the state institution about what they feel, to elicit their perceptions. This is an important pathway to obtain evidence and thus create feedback loops. For instance, citizen surveys are commonly used to assess the quality and impact of public services such as health and education (UNDP 2021, Clifton et al, 2020, OECD-ADB 2019).

In the legal system, perception surveys of court users can generate useful knowledge about how well courts function in their delivery of justice (National Center for State Courts, 2005). Ongoing surveys of user experience of courts can help measure the performance of a component of the entire legal system, and in assessing the impact of interventions made for reforming the legal system.

Surveys of court users and the public on their perception of the judiciary have been prevalent in developed countries from the 1990s, and are gaining currency in India (eg., Dougherty et al, 2006; Rottman and Tyler, 2014; Staats et al, 2005; Daksh 2016). Such surveys seek to capture the perceptions of court users on qualitative metrics (Manivannan et al, 2022). Such metrics can be used to evaluate the functioning of a single court, or compare alternative courts.

On one hand, perceptions are not reality. On the other hand, the views of end-users of the justice system are particularly important because, ultimately, the justice system exists to serve end-users whose interests and preferences may differ from those of judges and lawyers. We can readily discern certain difficulties in survey-based measurement of perceptions:

1. There are many different users of a court, who differ in their extent of knowledge. Litigants who see a court case as a disruption of their daily lives, may see things differently when compared with lawyers, for whom courts are part of their professional lives.
2. A person who loses a case is likely to be unhappy with his experience of the court and vice versa.
3. Different individuals might be working on non-comparable cases, and their subjective experience of the court is then not comparable.
4. It is not clear what is an objective benchmark of sound performance. A perfect court may be prohibitively expensive. Users of courts may have normalised a variety of difficulties; their `satisfaction' may only flow from learned helplessness.
5. It is important to narrowly measure a court or a group of courts, and make claims about the narrow unit of observation, as opposed to bigger claims about the Indian legal system.

In 2023, we conducted two pilot surveys to evaluate their utility as feedback loops for courts.

One survey was administered to understand the functioning of five alternative forums that can be approached to adjudicate matters of debt disputes: the Bombay benches of the National Company Law Tribunal (NCLT), the Debt Recovery Tribunal (DRT), the Bombay High Court (Bom HC), the Metropolitan Magistrate (MM) courts (which adjudicates criminal proceedings for cheque bouncing cases), and the Alternative Dispute Resolution (ADR) process.

To help improve data quality, the survey was conducted on practitioners who had multiple instances of interacting with the five courts. By selecting practitioners that have had repeated instances of approaching these forums to resolve disputes, the survey results are less vulnerable to the 'loser' effect. To obtain comparability, we presented a hypothetical, canonical problem of debt dispute resolution to each survey respondent. We then asked them to rank the five forums on five dimensions of court performance, namely, efficiency, effectiveness, predictability, independence, cost and convenience, and calculated the average rank for each forum on each of these dimensions.

The second survey was conducted with litigants at the DRT, with the objective of understanding the functioning of this court. For this, we deployed a team of four, who visited the premises of the Bombay bench of the DRT. The team administered a survey questionnaire on individuals, in order to evaluate the performance of the DRT on the above mentioned five dimensions. The participants were asked to rate their experience at the DRT on a five-point scale.

Method

Survey design

We used a combination of qualitative (in-depth expert interviews and open-ended comments) and quantitative surveys (multiple choice and scaled questions). Qualitative surveys with experts provide more contextual insights, enable comprehensive analysis. They helped validate our founding conjecture, the idea that there was a class of disputes which could go to multiple different forums. However, these surveys were time-intensive and it was difficult to obtain the interest and involvement of experts.

Survey mode

We administered the survey in both online and offline formats. Surveying litigants on court premises was challenging in two ways. First, litigants do not always accompany their lawyers to courts, especially in disputes of larger sizes involving firms. Second, one forum may deal with multiple type of disputes (civil v. criminal; mergers v. insolvency). This poses difficulty in identifying a litigant with a desired case-type.

The questionnaire used for the surveys and the responses collected can be found here.

Results: The perceptions of practitioners

The practitioner survey involved eliciting their choice of forum for the following hypothetical, canonical problem:

Q is a large public listed company. It has availed of a working capital loan of Rs. 7 crores from N, a small sized NBFC, repayable within three years with simple interest @16% p.a. Q and N are 100% domestically owned. As collateral for the loan, Q has granted N a floating charge over some of its movable assets, for example, its machinery or its inventory. One year into the loan, Q defaults on its loan to N. The outstanding amount exceeds Rs.1 crore. Post-dated cheques issued by Q towards interest payment bounce due to insufficient funds. The collateral is not sufficient to cover the outstanding amount. You are advising N.

The survey respondents were asked to make two assumptions, namely, that the limitation period is the same across all the courts; and that all courts have jurisdiction.

We collected responses from 18 respondents, of which 16 were lawyers and two were key managerial personnel at an asset reconstruction company and a debt restructuring advisory firm. Six of our respondents had between 20 to 30 years of experience in this area, eight of them had experience of less than 20 years, and two of them had more than 30 years experience in this field. They had significant experience with many of the venues of interest: 14 had experience with the NCLT and the Bom HC, 11 with the DRT and ADR process, and 5 with the MM Courts.

We aggregated the ranks assigned by the respondents to each of these forums on the parameters of independence, efficiency, effectiveness, predictability and access, and averaged them to arrive at an overall rank for each forum. The specific statements on which the respondents ranked the forums and their ranks are presented in Table 1. The forums are arranged in increasing order of the average rankings on each parameter. The NCLT was ranked the highest on the parameter of Efficiency, followed by ADR, the Bom HC, the DRT and the Metropolitan Magistrate. On the other hand, the Bom HC was ranked as the most preferred forum of choice on the parameter of independence.

Table 1: Preference ordering of five debt enforcement forums

Metric	Survey Statement	Ranking
		1	2	3	4	5
Efficiency	Most likely to dispose of my matter in a timely manner	NCLT	ADR	Bom HC	DRT	MM Courts
Effectiveness	Easiest to recover the amount awarded in the judgement decree.	NCLT	Bom HC	DRT, ADR	MM Courts
Predictability	(i) Expected sequence of stages in my matter was clear.	NCLT	ADR	Bom HC	DRT	MM Courts
	(ii) Hearings are most likely to be held as scheduled.	ADR	NCLT	Bom HC	MM Courts	DRT
Independence	Decisions are most likely made based on the merits of the case.	Bom HC	ADR	NCLT	MM Courts	DRT
Access	(i) Can afford to take my case to this forum.	MM Courts	DRT	NCLT	Bom HC	ADR
	(ii) Ease of navigation; staff helpfulness; website; ease of filing process	ADR	Bom HC	NCLT	DRT	MM Courts

Table 1 contains new insights on a specific court on each attribute. For example, while the Bom HC and the ADR process are perceived to be most unbiased, they are perceived as more expensive to access. ADR is perceived to be most predictable, but less effective on actually getting the relief. The NCLT, on the other hand, is perceived to be more efficient and effective, when compared to the other forums, but less likely to also be unbiased. The DRT and the Metropolitan Magistrate courts are perceived unfavourably on all aspects, except affordability.

Results: The perceptions of litigants

The in-person survey conducted at the DRT observed 55 persons, who were presently a party to a dispute at the DRT. Among these, 24 were debtors, 19 were creditors, and 12 belonged to the residual category, such as court/privately appointed receivers and auction awardees. Of these, 30.6% were at early stages (admission), 28.6% were at advanced stages (such as post-admission or pending last hearing), and 22.4% were awaiting a final hearing or pronouncement of judgement.

Litigants at the DRT had more positive perceptions than practitioners. Litigants ranked the DRT the highest on predictability of the hearing: most litigants agreed that when a hearing for their case is scheduled at the DRT, it will be held on the scheduled date. About 67-69% of litigants perceived the DRT to be an affordable and unbiased forum to resolve their dispute. More creditors ranked it higher (85-89%) on these two metrics than debtors (58-62%). However, 52% of litigants did not think that the DRT resolves cases in a timely manner.

Discussion

Good performance by the judicial branch in a country is essential. As with all aspects of public policy, this requires the loop of evidence, identification of difficulties, creative policy proposals, policy reforms, and measurement of the gains. In the legal system, generally, evidence and measurement involves quantitative measures. In this article, we have shown a case study where survey-based evidence was useful. This constitutes a useful additional pathway to measurement of the legal system.

Litigants are the ultimate end-users of courts, so their views matter greatly, but their information set may be limited. Legal practitioners have better information through repeated interactions and potentially observation of multiple venues, but their views may not capture the views of the litigants themselves. In the future, it would be useful to go further, by way of surveying the general public, measuring the view of persons who have not experienced litigation at a given location.

References

Shaun Bowler, Joseph L. Staats, and Jonathan T. Hiskey (2005). Measuring Judicial Performance in Latin America, Latin American Politics and Society.

Judith Cliftona, Marcos Fernandez-Gutierrez and Michael Howlett (2020). Assessing public services from the citizen perspective: What can we learn from surveys?, Journal of Economic Policy Reform.

Daksh (2016). Access to Justice Survey, A DAKSH report.

David B. Rottman and Tom R. Tyler (2014). Thinking about judges and judicial performance: Perspective of the Public and Court users, Onati Socio-legal Series.

Devendra Damle and Tushar Anand (2020). Problems with the e-Courts data, NIPFP Working Paper Series 314.

George W. Dougherty, Stephanie A. Lindquist and Mark D. Bradbury (2006). Evaluating Performance in State Judicial Institutions: Trust and Confidence in the Georgia Judiciary, State and Local Government Review.

Institute of Social Studies and Analysis (2021). Satisfaction with Public Services in Georgia, United Nations Development Programme.

National Center for State Courts (2005). CourTools: Trial Court Performance Measures.

Pavithra Manivannan, Susan Thomas and Bhargavi Zaveri-Shah (2022). Evaluating contract enforcement by courts in India: a litigant's lens, XKDR Working Paper No. 16.

Pavithra Manivannan, Susan Thomas and Bhargavi Zaveri-Shah (2023). Helping litigants make informed choices in resolving debt disputes, The Leap Blog.

OECD-ADB (2019). Government at a Glance Southeast Asia, Serving Citizens: Citizen satisfaction with public services and institutions, OECD Publishing, Paris.

Pavithra Manivannan and Susan Thomas are researchers at XKDR Forum, Mumbai. Bhargavi Zaveri-Shah is a doctoral candidate at the National University of Singapore. We thank Surya Prakash B.S., Renuka Sane, and Anjali Sharma for their suggestions on the design of the surveys. We acknowledge the very diligent assistance by Nell Crasto and Balveer Godara, students at Kirit P. Mehta School of Law, NMIMS Mumbai, on conducting the litigant survey. We are grateful to all the survey respondents for their generous participation, and thank Mahesh Krishnamurthy, K.P. Krishnan, Sachin Malhan, Harish Narsappa, Rashika Narain, Geetika Palta, Siddarth Raman, Ajay Shah, and Arun Thiruvengadam for their comments and suggestions on this work.