Reinventing the Rules
Discover the Latest Innovations and Lessons Learned in Rule of Law and Legal Empowerment Projects
Event: Best Practices in Impact Evaluations: Anti-Corruption and Rule of Law
A few weeks ago, the Society for International Development hosted a panel on “Best Practices in Impact Evaluations: Anti-Corruption and Rule-of-Law.” The panelists offered some great insight on the challenges involved with monitoring and evaluation (M&E), best practices, and recommendations going forward. Read the excerpts below!
Credit: EVARISTO SA/AFP/Getty Images
Peggy Ochandarena, Director, Monitoring and Evaluation, Chemonics
CHALLENGES:
Justice Sector Actors Are Uneasy About Being Scrutinized
Collecting data for M&E can be hard because some countries have a culture of secrecy and don’t want to give up information or have their work measured. For example, judges have felt uneasy with efforts to analyze how they manage their case load, the quality of their decision, and how fast decisions are rendered. Judges have commented that an effort to measure their work interferes with their independence. In another example, a police commissioner wanted to define case resolution a certain way to make it appear as though they were highly effective, when that may not have been the case.
Problems Defining Rule of Law & Setting Clear Goals
In practice, the concept of rule of law has been defined both narrowly (with the focus being only on justice sector institutions) and broadly (cultural and social norms in a society, how government governs under the law, respect for human rights, etc.) If the concept and goals of the project aren’t clear, it can become difficult to measure. Terms such an “Independent Judiciary” and “Better Citizen Security” also mean different things to different cultures. Unless goals are clear, it’s hard to conduct good impact evaluation.
Difficult to Isolate & Evaluate Sectors That Are Intertwined With Each Other
Rule of Law work involves the intertwinement of different systems and sectors. If you’re tinkering in one area and not the other, it’s difficult to measure. For example, if you work with prosecutors and the prosecution appears to function well. The prosecution depends on the police to feed them information and the courts to try cases. If you haven’t tinkered with other aspects of the system, it’s hard to see a systemic change. Even where you change one part of the system, the other sections that are dependent on this system will have to adjust. If you do impact evaluation when other parts of the system haven’t adjusted, then the evaluation will turn out poorly, unless you wait until the system starts to shift. It’s also difficult to isolate the effects of a particular intervention when there are multiple donors involved working on different pieces.
Safety Risks in Collecting Data
It’s difficult and even unsafe to collect data in conflict-prone, post-conflict, active conflict zones, or fragile countries. You have to think carefully about the risks involved when there’s tension between groups. How a term is defined, who collects the data, how often it’s collected, how it’s analyzed, interpreted, shared, and disseminated are all factors that can exacerbate tensions.
Rule of Law Is A Long-Term Process
Turning institutions around and changing social norms takes years. Rule of law projects are often short-term and take three to five years, but building rule of law is a long term process. In the US, our own evolution in rule of law programming took decades. It took 150 years to separate the courts from the Department of Justice.
Specific Skill Set Needed for Rule of Law Evaluations
Impact evaluation takes sophisticated skills to set it up, analyze data, and run quantitative analysis. It’s difficult to find evaluators with this skill set that also have a deep understanding and experience in the rule of law field. There’s a scarcity of good rule of law impact evaluations. Some of the top M&E experts believe conducting impact evaluations in development is questionable and inappropriate.
Credit: SID
BEST PRACTICES:
Outcome Mapping and the Most Significant Change (MSC) technique can tell us a lot and it may be worthwhile to focus on these methodologies.
It may take years before the achievements of a program are evident. We should wait some time after a project has closed before collecting data. There has been some work to do this in discreet pieces and this might be more feasible and realistic than a meta analysis on a project or sector as a whole.
We should also do a ‘No Harm Analysis’ about what this measurement is going to mean in a situation where there’s a potential for conflict or exacerbating conflict.
RECOMMENDATIONS:
We need to think about what’s appropriate to measure and we shouldn’t put too many resources into conducting evaluations, if it doesn’t make sense. Impact evaluation might not be appropriate on a particular project, country, or sector and we shouldn’t be afraid to say it’s not going to work.
Bert Spector, Anti-Corruption Specialist, Management Systems International (MSI)
In many countries, anti-corruption is pervasive. It exists across many sectors, it’s deeply woven into the culture, and it’s passed on from generation to generation. Donors may ask if it’s worth doing a project in this context and the answer is yes, with qualifications. There will be some improvement in the short term, but it takes generations in most cases and there needs to be a long-term commitment to fighting corruption.
CHALLENGES:
Time is often the biggest challenge
Congress and donors expect immediate feedback, especially in Afghanistan and Iraq. The bigger impacts that can cause change in the Transparency International corruption perception index or the World Bank Institute Index take years.
Projects that are limited and localized can see some impact in the short-run. Examples include handling corruption complaints, using local legal support offices to resolve corruption, and teaching the public how to stand up for their rights when confronted with corruption. When working with civil society organizations, most believe nothing can be done until they have their first success. No matter how small the success is, this is often a huge motivation for them to continue.
Credit: Mohammad Moniruzzaman
Measuring Change in Corruption Levels
Corruption by its very nature is meant to be secretive and it’s hard to measure change. Experts have argued that rather than measure corruption levels in an institution or locality, one should look at changes in government performance. One of the factors that improve government performance is anti-corruption activity, but there can be many other factors.
The Cross-Sectoral Nature of Corruption
Corruption is part of a system and it can invade citizen-government interactions at a local, regional, and national level. It also cuts across sectors such as law enforcement, elections, health, education, etc. While there are similarities between how corruption impacts sectors, there are also sectoral differences. USAID has a handbook with helpful tools.
Reforming Laws vs. Practicing What You Preach
There’s a big difference between words on paper and reforms that make it into practice. International financial institutions may pressure developing countries to write new anti-corruption laws, establish accountability institutions and anti-corruption commissions. Often times, these countries do a good job of that and they have some of the best laws and institutions out there, but this isn’t followed up by funding, resources, political will, or staffing.
There are also regional anti-corruption venues that send out peer review teams to a country and this reinforces the system because they look at whether the right laws and institutions exist and not whether they’re doing anything.
BEST PRACTICES:
Evaluating the level of corruption through public perception is not a good measure. It’s better to measure behavioral experience with corruption rather than looking at perceptions and attitudes. When there is an increase of anti-corruption programs in a country, this has a tendency to bring corruption issues to the fore and it becomes much more visible in the public agenda. The more corruption programming there is, the perception of corruption seems to be more widespread.
It’s better to find visible indicators of change in corruption behaviors and this is more possible on a local level than a national level. For example, citizen report cards which monitor improvements in public service delivery or citizens who go to legal support offices to file complaints and then see a tangible result following their complaint. People are empowered and this catalyzes more activity when they see results.
Developing an inventory of anti-corruption indicators that are sensitive to different sectors is a growing field. Since there are numerous sectors and corruption can be pervasive, it’s hard to create a comprehensive set, but it seems to be helpful.
When you’re dealing with improving public accountability and transparency of government officials and actions, the more you can measure information accessibility, the more you can evaluate the real impact of anti-corruption programs. Examples include the existence of accountability mechanisms like ethics boards, open hearings, and an ombudsman’s office.
RECOMMENDATIONS (All 3 require more research):
1) You have to continue looking at the impact of projects after they are over. Some special studies can be done to focus on the impact of projects 3-5 yrs after they’ve ended, but there’s a need to control for other factors that might have intervened during that time.
2) Due to reductions in funding for development programs, anti-corruption initiatives are being mainstreamed into sectoral programs such as health and education. It’s important to not just mainstream it or treat it as a cross-sectoral issue, but look at the problem of corruption within these sectors more head on.
3) You should look at good government performance over time and assess what can be done to measure performance that’s relevant to what was done with anti-corruption programming.
Credit: SID
Kenneth Barden, Senior Governance and Anticorruption Advisor, USAID (comments are his own and not reflective of USAID)
CHALLENGES
New Aid Policy: Developing Countries Identify Their Own Priorities
The biggest challenge effecting USAID and other donors are the agreements reached in Paris on aid effectiveness. The new policy allows developing countries to set their own agenda and it will call for more government to government direct assistance. Nevertheless, we will still rely on implementers. Under the new model, the country will be responsible for how it spends its money. We will conduct cursory evaluations of their capabilities to administer that money, but they are sovereign, and there will be limits on accountability.
BEST PRACTICES:
Evidence-Based Programming
When USAID funds a project, it looks for two things: Has it worked in the past under similar circumstances or is the programming particularly tailored to the country in question. USAID is in the process of mapping out programs and drawing best practices from them. There will be more publications on best practices for rule of law, governance, and anti-corruption.
Indicators used in the past have been driven by performance, such as how many judges were trained, how many laws were passed, etc. We haven’t done a study to see whether it’s made a difference 5 years after the project ended. This is something we have to study more.
A professor at Southern Methodist University is developing an experimental technique to test an anti-corruption hypothesis. There are also other academic approaches to evaluating these type of interventions. Francesca, at the World Bank, is doing research to identify tipping points when a developing country has gone from bad to good. By identifying factors that lead to the tipping point, perhaps you can extrapolate back and determine what indicators can be used to see if we’re on the right road for reform.
4 comments on “Event: Best Practices in Impact Evaluations: Anti-Corruption and Rule of Law”
Leave a Reply
Reinventing the Rules 
This is a very interesting topic, given USAID and other donors’ push for improved evaluation of impacts. Unfortunately, most work with institutions does not lend itself well to USAID’s ‘gold standard’ of randomized comparison experimental designs. Rather, as some of the contributors above suggest, more mixed-method and other non-standard approaches (included measuring impacts years after the intervention has finished, which development agencies should probably be doing more of in any case). The panelists also seemed to touch on an issue endemic to work on corruption issues: often interventions focus on the form of institutional change (laws passed, judges trained, prosecutor’s offices equipped, anti-corruption citizen commissions established, etc), rather than the function. In many developing country contexts (and probably developed countries as well), public institutions’ performance is determined more by the informal rules, norms, culture, incentives, etc., than its formal structure and systems. I like Matt Andrews analogy of institutions as icebergs (formal aspects are whats visible above surface, informal below the surface, and much bigger). This points to a larger issue of addressing what are inherently political challenges (corruption) with technical assistance. Thomas Carothers and Diane de Gramont address this issue in their recent book “Development Aid Confronts Politics: The Almost Revolution”.
Hi Brendan! Thank you for reading my post and for your comment! You are the third person to recommend this book written by Thomas Carothers, so I am going to try and get a hold of it as soon as possible. I also thought the suggestion to evaluate programs 3-5 years after it has ended was an interesting one. If you know of anyone who has done this, please let me know. I would be happy to speak to them about what they’ve learned. I am also keeping an eye out for organizations or academics who may have done this. From what I have been told, under the current development framework, it is difficult for donors to fund these types of evaluations years later. But I am interested (and also keeping an eye out) for suggestions on how current donor strategies/funding could be re-modeled to institutionalize M&E 3-5 yrs after a project ends.
Christina,
I saw a recent webinar (http://www.odi.org.uk/events/3250-aid-effectiveness) by the UK’s Overseas Development Institute in which a senior policy adviser from USAID mentioned the agency’s interest in doing some evaluations 3-5 years after projects have ended, to understand the sustainability of impacts. He didn’t mention a timetable or any examples, but its interesting that it is even on the table. Something to keep tabs on. I’m sure there have been some academics who have done this kind of work, but I’m not familiar with any specific examples. Personally, I don’ think its a matter of the development framework being an obstacle, I think its fear that the results won’t be very good. However, I think the demands for data and evidence are beginning to overcome this hesitancy, so it might be an opportune moment for some donors to go back and see what long-term impacts their interventions have had, especially as we begin to think about new approaches in a post-MDGs world.
Thank you Christina for writing this up. Delayed evaluation seems to have an inherent problem. If one cannot tell for several years whether a program is effective, does it make sense to pour resources into the program at all? I would hope that there would be some indicators of success or likelihood of success in some sense at the time the program is undertaken if not earlier.