Reinventing the Rules
Discover the Latest Innovations and Lessons Learned in Rule of Law and Legal Empowerment Projects
Selecting the Right Indicators to Improve Court Performance
In May, The World Bank released a short paper titled ‘An Introduction to Selecting the Right Indicators to Improve Court Performance.’ The report provides an overview of three different models used to measure court performance and also looks at data collection, the type of data that can be relied on, and how to ensure data collected translates into court reform. View some of the excerpts below!
Credit: Pajhwok
Who Can Influence Court Reform and Select Useful Measures?
Naturally, the constellation of people who have interest and influence will vary in each context and for each reform process. That said, some guiding principles include involving those who oversee and manage courts (Chief Justices, Chief Registrars, court managers and administrators, Attorneys-General and Ministers of Justice) in the selection and use of measures.
This will allow them to understand the trajectory of reform and make managerial decisions on the basis of the data. It can also be useful to think about expanding the circle of those involved to others who may have interest in reform and can provide positive reform pressure, including judges, court personnel, lawyers and court user associations, Ministries of Finance, relevant NGOs, and the media. These stakeholders should contribute to identifying the areas of reform that are most important to them and the measures by which they can confidently track that reform is happening – or not.
Five Ways to Measure Court Performance
Whilst the indicator development process needs to be rooted in local reform priorities, it can be useful to think about measuring performance in at least the following five areas: 1. Efficiency; 2. Quality; 3. Transparency and Accountability; 4. Access; and 5. Independence. These measurement areas, of course, are very much interrelated. For example, lengthy delays in case processing may demonstrate inefficiency, result in inaccessibility and be correlated with poor quality (e.g., lack of fairness).
Developing Court Indicators
The process of developing court indicators can fall roughly into three phases: (1) developing the right measures, (2) ensuring the right delivery of the information, and (3) enabling the right use of the measures.
1. Develop the right measures
Q: Are there generally accepted measures of court performance? A: Yes, more or less.
There are three tried and tested models of performance measures listed in the table below. These models will not be relevant or useful for every context. The process of determining whether to consider, adapt or adopt should be grounded in local priorities, the nature of the reforms being undertaken, and capacity to collect the relevant data.
Q: How many measures should courts develop? A: The vital few instead of the trivial many.
It is better to select, collect and effectively use a small number of indicators that are relatively easy to collect, sustain and maintain, rather than attempt to use many indicators for which the data is not consistently available, difficult to capture or which deluge court managers, staff and stakeholders with information they are unable to digest.
First and foremost, the number of performance measures developed depends on the reform priorities identified. Within these parameters, a few key indicators that are reliably collected and thoroughly interpreted are preferable to many measures that are infrequently collected and so complex as to be ignored. For example, regularly reporting data on just two indicators – user satisfaction and case backlog – will in many jurisdictions provide a good understanding for courts and the general public about how much progress is being made, as well as provide impetus for reform.
Starting with a vital few indicators also creates a strong foundation on which to build in the future. After gradually building understanding and buy-in for a first set of indicators, more can be added as capacity allows. For each measure, it is important to disaggregate or break out the data collected (e.g. respondents to a court user survey could be asked about their ethnicity, gender, income, frequency in the court- house, and “business” in the court, etc.).
Q: What types of data should be relied upon? A: Baskets of data.
Whilst being mindful of limiting the total number of indicators, it can be useful to have more than one source of data for each area of reform (a basket of indicators). This helps to guard against problems in any one data source. It is also useful to have data that draws from different sources such as administrative data (e.g. court files), experience and perception surveys (e.g. of users and the general public), and expert opinions (e.g. of lawyers).
This is particularly important for measuring a performance area that is multi- dimensional or not easily observable, like access to justice. For example, access to justice could be measured by way of administrative data: a case file analysis could indicate whether both men and women are filing cases, the number of unrepresented litigants, and whether legal aid is being offered. It could also be measured using user surveys to determine user characteristics such as income or ethnicity, their understanding and experience of court processes, and experiences in physically accessing the court.
Q: How can we collect the data that feeds the indicators? A: Simply and cheaply.
In general, it is best to have a strong manual system established before trying to institute it electronically. For resource-constrained courts, data collection can be done relatively cheaply. For example, a court user survey or an employee engagement survey can be a one page sheet of paper or a free online survey (such as Survey Monkey) in areas where internet access is more common. A good system of court records can be built on simple register books – as long as they are faithfully filled out and the data regularly submitted for analysis. It is important to ensure that responsibility is clearly assigned for each step in the data chain: collecting, transmitting, analyz- ing and disseminating findings.
Q: What’s wrong with measures such as the number of judges? A: Outcomes not inputs
For one, the number of judges per population doesn’t take into account the number of cases (one measure of a judge’s workload) let alone the length of time each case takes. Secondly, a greater number of judges do not necessarily provide more efficient and effective courts – not least because of the other inputs needed to run the system. Further, the definition of “judge” varies in each country. The same considerations apply to picking measures like percentage of budget allocation to the judiciary.
Instead, it is generally considered preferable to pick measures that focus on changes in the experience of those served by courts, over those that track internal aspects — that is, measures that indicate results, rather than resources, activities, or level of effort. Courts are often keen to measure internal matters they have closer control over (e.g. the number of personnel trained) as these can be more easily improved year-to-year. Whilst it might be useful to keep track of these, ultimately the impact of internal activities needs to be felt by users (e.g. training of registry staff leads to increased reports of satisfaction) for the purpose of the activity to be fulfilled.
2. Ensure the Right Delivery of the Information
Q: How should the data be delivered? A: Make it visual.
It makes no sense developing the right measures and assembling critical data, if the performance information is not delivered to people who have an influence over reform, in a manner they can use. Those running justice institutions may not have a statistical background, therefore it is important to present data in an easily accessible manner such as graphs and tables.
It can be useful to have an agreed small set of measures which are reported over time (the “vital few”). This can take the form of a performance scorecard. In addition to targeting particular actors who have influence over reforms, data should be made generally available to the public in an engaging and easy to interpret format. This will not only increase transparency and accountability but also allow for insights which may be of interest for reformers ad allow the media and NGOs to convey shifts in performance to a broader audience.
Q: When should the data be delivered? A: As regularly as possible.
Whenever possible, data delivery should happen in “real-time” to pro- vide the most accurate picture of court performance at any particular moment.
3. Enable the Right Use of the Measures
Q: How can I ensure that the data collected translates into changes in the court system? A: Integration. To best utilize measures, a court should attempt to integrate them into key management processes and operations, including budgeting and finance, resource and workload allocation, strategic planning, organizational management, and staff development. This creates a consistent feedback loop between performance measurement data and larger reform processes, allowing the data to inform future actions.
Q: What should I look for in the data? A: Look for differences and then ask “what, where, how, and why”.
Looking for variations in performance between, say, courts, types of people served (e.g. men and women) and personnel (e.g. judges) can identify both areas of strong performance (“bright spots”) and those in need of particular attention. Then, explore why the indicators are moving. A bright spot may reveal a particular court that has significantly less delay than the average. By exploring the practices of this court, lessons may be learned that are applicable to courts across the jurisdiction.
Reinventing the Rules 