Measuring Impact on the Lives of Clients
Monique Cohen, Senior Technical Advisor1,
Office of Microenterprise, U.S. Agency for International Development (USAID)
![[Cohen]](http://www.microcreditsummit.org/graphics/cohen.jpg)
USAID’s AIMS project started in 1995 with the purpose, simply stated, to gain an understanding of the process by which microenterprise services strengthen businesses and improve the welfare of microentrepreneurs and their households; and, secondly, to strengthen the ability of USAID and its partners, the NGOs, to assess the impact of microenterprise programs.
The project has two central components. The first one consists of three higher-end studies which I regard as the responsibility of donor agencies. These are longitudinal surveys, covering two time periods, large-scale, methodologically rigorous and use comparison groups of clients and non-clients. Each survey includes about 750 individuals. They are being undertaken in three countries: Zambuko Trust in Zimbabwe, SEWA Bank in India, and Accion Communitaria del Peru. All are well established deliverers of microcredit, have a substantial number of clients, and have stable methodologies. This is important for such large-scale impact assessments since one does not want to be in the middle of a study spanning over two years and find that the methodology has changed. This could have consequences for the impacts being assessed and the usefulness of the information.
The second component of the AIMS project, and one that poses a greater challenge, is the development of lower-cost impact assessment tools. We work with a lot of NGOs who need or want impact assessment. This demand is not just driven by donor agendas, but reflects the NGOs’ own desire to know if they are meeting their objectives. For many NGOs their mission is to serve the poor, to reach poor women, and to make a difference to clients by helping them build their asset base. They want to see if they are on track with their mission. Thus, many NGOs see impact assessment not only as serving an accountability function, but also in part helping them as they seek to improve their programs. In this context an impact assessment is seen as a management tool.
In taking up this challenge AIMS wanted to design a set of tools that were within the resource capacity of NGOs—and by that I mean they had to be low enough in cost to be affordable. We set as a goal not hundreds of thousands of dollars but about US$25,000 net—and even that sounds high—but we decided that we would focus on that as a goal. Then we wouldn’t be tempted to do something too large. Working with one of its subcontractors, the SEEP Network, AIMS has developed and tested impact tools that meet these criteria.
When one talks about impact people often come up with a Christmas tree of impacts they want to measure—they want to know about health, they want to know about education, they want to know about enterprise growth, and so on. A wish list of desired impacts could cover a whole page. The problem has always been how to order this list, how to be selective about what one measures. If one is serious about impact assessment, one can’t do everything. So the question is, what are the impacts you can reasonably expect to occur as a result of access to microenterprise services?
Answering this question has guided much of the AIMS work on impact assessment, both the large studies and the tools. It has meant asking, what happens when money comes into the household? A client gets money, brings it into a household where the household economy isn’t very separate from the enterprise economy. The two are intertwined. A loan is an additional cash resource. When the money comes in many lenders hope that it will be invested in the enterprise, because that way it will generate the cash flow needed to repay the loan, or enterprise development may be one of the NGO’s objectives. However, often it goes to school fees, because school fees are due the day the client gets the loan. At other times there is an emergency or it’s spent on a wedding or it is used for home improvement. So what AIMS wanted to understand is how does money come into the household, how is credit used and repaid by the household, and how does it add to the household’s pool of financial resources.
The challenge in developing the tools was to come up with a set of impact assessment instruments that are lower cost, credible, and useful. We are struggling with this issue within the AIMS project and we are struggling with the same subject within the CGAP Impact Assessment Working Group. We are trying to figure out a way that we can measure impact credibly, using sound methods that are appropriate for what these impact assessments are intended to do: provide information about how microfinance programs are doing, determine whether they’re reaching the poor, whether they’re having an impact, and whether they’re helping the clients improve their ability to cope with crises.
We have made tentative steps in this direction. In the CGAP Working Group we have begun to come up with guidelines for what can be termed ‘middle range’ impact assessments. Among the emergent guidelines is that need for impact assessments to include a time perspective. One needs to compare at least two points in time. One has to compare clients and non-clients so as to be able to tell not only how the clients are doing, but how they are doing relative to people who have not accessed microenterprise services.
The impact assessment has to be tailored to the context, and you have to start with a limited number of hypotheses. This is to get away from the Christmas tree approach I mentioned earlier, where you put everything out there and try to prove everything. Let’s start with a limited number of issues that we really think are plausible, that can be linked to the impact of financial services, and test them.
We need lower-cost impact assessments. It’s a point that I raised earlier when I mentioned that the goal was to be under US$25,000. I should add that we intend to go lower and are doing so. However, this was set at the start as the upper limit.
And lastly, an impact assessment has to be useful. A lot of impact assessments are done to demonstrate that the resources have been well spent. Yet, in focusing on accountability, we have ended up separating out the subject of impact assessment from any discussion of the operations of a microfinance institution. As a result only a few practitioners have seen impact assessment as pertinent to the management of their institution.
It is time to make impact assessment relevant. This is the final challenge I want to mention. How can the information that is being generated from an impact assessment be used to inform the operations of a microcredit institution? To answer this we have to recognize that impact assessments are about clients and client behavior and how clients change. If one is managing a financial institution such information is very valuable. It helps address issues such as whether products and services are responding to the changing needs of clients, how to change the design of the services, why people drop out of a program or how to attract people who do not access existing services.
Many of these questions can be answered by the information generated by impact assessments. How-ever, the integration of impact assessment into the operations of microfinance institutions will only happen if we move toward lower-cost or middle-range, credible impact assessments, which serve both as an accountability and a management tool. For me this is the challenge we need to address in undertaking the impact assessment of microfinance programs.
1The views expressed here are those of the author and do not reflect those of USAID.
For more information, contact Monique Cohen by e-mail at AIMS@msi-inc.com or by fax at 1 202 488 0754.
Miguel Navarro, Deputy Director, Organización el Desarrollo Empresarial Femenino (ODEF)
Organization for Women’s Enterprise Development
![[Navarro]](http://www.microcreditsummit.org/graphics/navarro.jpg)
I work for the Organization for Women’s Enterprise Development (ODEF) in San Pedro Sula, Honduras. This is an organization that began its activities in 1986 and started a credit program in 1989.
I want to talk to you about the experiences of ODEF in the execution of an impact evaluation program. . . . [With the Small Enterprise Education and Promotion Network (SEEP)] we have a portfolio of just over US$2 million. The number of clients that we have is about 9,200. Currently we have an institutional self-sufficiency of 103 percent. We have an annual income of nearly US$800,000 that comes solely from the loans or the interest that clients pay on loans, and our financial self-sufficiency, before adjusting to inflation costs, is about 121 percent.
These financial results from ODEF show that the institution has been growing without a doubt, and that the results have been very good. . . . But it’s one thing to have good results in the institution and it’s another thing that the clients have good results.
It was not until the SEEP project that we included questions about client and ex-client satisfaction, to ask the people how they have really felt with the program.
What were the objectives of the impact evaluation that took place at ODEF? The first was to test tools that were developed for this process to make sure they were simple, that they could be used, that they were believable, and above all that they had a low cost-benefit. The other objective was to make sure that this evaluation was possible and that it could be done by ODEF or by the practitioner organization. Normally in our country this type of impact evaluation is done by foreign consultants that arrive at the organization, carry out all of these surveys, and then six or seven months later send a report.
The field surveys were done under the supervision of SEEP, but were usually done by us, by the practitioner. We used a strategy for collecting data where . . . the surveys were carried out by ODEF staff, and the staff would exchange zones [did the surveys in zones where they would not normally work] . . . in order to obtain objectivity in the collected data. And the analysis of information was done with computer software, and SEEP staff also trained the people at ODEF on how to use this software.
. . . In economic aspects, we found that the businesses of clients grow and are more profitable than those of non-clients. . . . An important aspect of this survey is that you interview clients and non-clients. And the non-clients were people that have all of the conditions to receive loans from the program but who still had not received credit. Many times these non-clients were just about to receive their first loan. . . . Another important point is that the savings of clients has increased more than that of non-clients. . . .
About 85 percent of the clients in ODEF’s programs are women. An important indicator for us was this . . . that clients spend less time on their business and gain more than non-clients. Why is this indicator so important for us, that clients spend less time? It is because when you work with women, they normally have a double activity—the activity that generates income and also the activity of continuing to care for their family, to continue doing all of the daily chores. . . . A social result is that the women clients show higher self-esteem than the non-clients. . . . And I think that the self-esteem of clients has been increasing . . . and this perhaps is the key point, let’s say, that differentiates between a good business woman and a bad business woman.
Our clients were very satisfied with the efficiency of ODEF. They were very happy with methodologies. . . . But also they were unsatisfied with the little supervision of loans. Primarily, those in the first or second level of loans were dissatisfied because there was not a more continuous supervision on the part of the promoters regarding controlling the loans. This was something that we seriously took into account.
And they were dissatisfied with the interest rate. About 67 percent of them mentioned that they were dissatisfied with the interest rate. . . . If you obtain a response that the majority of your clients consider your interest rate to be too high, I think that it should be a concern of ours to see how we can give our clients loans at a lower cost. And this is what we have done in ODEF. . . .
What are the advantages for us with this evaluation method? For us, that one can initiate changes immediately. . . . And above all, that the people who actually do the operative labor in ODEF were learning of the results. The very promoters were doing the major part of the work. Because of this, they did not show resistance to the changes. One of the biggest problems that one has when you do an evaluation by an outside evaluator is that when they send you the report, they make a ton of recommendations of things to change, but the people do not want to change. The people that do the work in the field do not want to make the changes because they are not convinced that [the report] is right. Meanwhile, with this type of evaluation, the advantage that exists is that the very promoter knows that this change is necessary. . . .
This type of evaluation can substitute for more expensive studies—that’s to say that many times it could, especially in our country. We have many studies about the impact on microenterprise, the necessity of credit in microenterprise, and they are very voluminous, very expensive studies, and they take a long time to carry out. With this survey you can know within a few weeks what it is your clients like, what are the needs of the non-clients, how would they like a credit program to be, what things do they not like in a credit program, and this would substitute for any large study that one could do.
. . . What we want is to have not only a stable and consistent program from an institutional point of view, but rather a program that is also stable and consistent from the point of view of our clients.
For more information, contact Miguel Navarro by e-mail at ODEF@sdnhon.org.hn or by fax at 504 5 580 374.
