Nunya, adidoe, asi metunee o. (Akan and Ewe)
La science est le tronc d'un baobab qu'une seule personne ne peut embrasser (French)
Wisdom is like the trunk of a baobab tree[1]; no one individual can embrace it. (English)

--Proverb from Benin, Ghana and Togo

This African proverb speaks well to impact assessment—it may seem as intractable as trying to embrace the trunk of a baobab tree. Yet the combined efforts of many can let us accomplish a goal that seems impossible for one or two.

At the 1999 Microcredit Summit Meeting of Councils in Abidjan, Ivory Coast, we went out on a limb and tried to heed this proverbial wisdom. We urged practitioners, donors, consultants and academics to stay focused on our ultimate goal—substantially reducing or eliminating poverty among our clients. We also issued a call to action, challenging practitioners to take the lead and develop impact assessment and monitoring systems that use internal feedback loops to integrate field knowledge into management decision-making. Using the analogy of a financial audit, we outlined the concept of an impact audit that:

· Provides analysis of trends over time and produces results comparable with previous impact data;

· Integrates data collection with the regular information system of an institution;

· Utilizes internal staff with a limited role for external experts; and

· Costs no more than what it costs to track and audit financial information.

One year later, we are pleased with significant progress in the industry. The Microcredit Summit Campaign continues to emphasize the issue through the updating of our original paper and by giving the topic priority at regional meetings in Africa, Asia, and Latin America. We also celebrate the publishing of the SEEP-AIMS Draft Impact Assessment Manual funded by USAID[2], the full-scale implementation of the tools in the Philippines and Peru, and the recent training sessions offered in the US and Kenya. We applaud the Ford Foundation for joining this effort with a new three-year grant project to assist practitioners in achieving their impact assessment goals. And we laud the renewal of the SEEP Impact Working Group to promote peer-learning exchanges. Together these efforts demonstrate that, just as one individual can’t embrace a baobab tree, practitioners can’t create these systems alone. Donors, practitioners, academics and consultants each have important roles to play.

In this updated paper, we want to further focus on practitioners by asking, “what does impact monitoring look like from a practitioner’s perspective?” We readily recognize that there is no single answer, no “magic bullet” that will meet the needs of everyone. In fact, different combinations of tools and systems are appropriate in different contexts and for different purposes. But we want to examine how managers can use impact assessment and monitoring as essential tools for decision-making and organizational learning.

To that end, we will take a look at one practitioner in the early stages of pioneering an integrated impact monitoring approach that, while still under development, embodies the kind of practitioner-led teamwork we recommend. The Christian Enterprise Trust of Zambia (CETZAM) has worked closely with its primary donor, the British Department for International Development (DFID), to develop a system that is well integrated into its regular operations. Its experience has been specific to a particular set of needs as it embarks on a national expansion plan, with a goal of creating a system that provides good information to support good decision-making. But while the findings are still preliminary and the system is still evolving, CETZAM’s experience may offer some insight into common issues facing many practitioners. Some of our questions will include:

1. How did CETZAM customize and implement its impact assessment and monitoring tools?

2. What are the initial findings and how has management used them to make better client-focused decisions?

3. How cost-effective are these efforts? How do they compare to the guidelines for an impact audit?

4. What lessons has CETZAM learned along the way? What challenges remain?

CETZAM has learned just as much, if not more, from the process of developing and implementing an impact monitoring system as it has from the preliminary findings. Therefore we hope that by sharing CETZAM’s experience, other practitioners can also learn from this process and consider how to use impact assessment and monitoring as an essential management tool in better serving clients, staying competitive and fulfilling their mission to reduce poverty[3].

Why do CETZAM & DFID care about impact?

CETZAM’s focus on impact assessment is driven by two considerations: 1) its mission; and 2) its competitive environment. Headquartered in Kitwe, Zambia, it was founded in 1995 by a group of Zambian business and civic leaders. The board’s mission is to “transform the lives of the poor” by “providing opportunities to create employment and generate income through credit and training services.” The mission to enable clients to become agents of transformation within their communities is one that CETZAM shares with all other members of the Opportunity International Network. The Network has defined “transformation” as “a deeply rooted positive change in beliefs, values, attitudes, actions, relationships and structures manifested in a higher level of existence of an individual and/or community.” Transformation may be economic, social, political or spiritual in scope[4]. Being able to assess and monitor holistic client transformation helps CETZAM determine whether it is fulfilling its mission.

At the same time, while its mission was the starting point, its competitive environment is an increasingly important consideration in focusing on impact. Presently, CETZAM is the largest provider of microfinance services in Zambia. But change is on the horizon as other microfinance institutions enter the landscape. As a result, CETZAM also needs to know whether clients have any negative experiences that might diminish their quality of life or cause them to leave CETZAM and procure services elsewhere. Understanding the dynamics behind client retention is critical to CETZAM’s future growth and expansion, as is obtaining other customer information through market research. In its view, impact monitoring and market research mutually reinforce each other and are closely intertwined.

Like CETZAM, the British Department for International Development (DFID) is driven by its strategy to effectively reduce poverty and improve livelihoods in Zambia[5]. It also wants to ensure that the institutions it has supported in Zambia and elsewhere in Africa are sustainable. Therefore, when DFID awarded CETZAM an initial four-year grant of £2.29 million (US $3.45 million) in 1998, it included £98,000 (US $147,470) to set up an impact assessment and monitoring system. DFID did not want to “reinvent the wheel” in each new country or with each new funding proposal. Instead, it wanted to build on the tools, systems and results that would be generated at CETZAM, using them to develop appropriate efforts elsewhere across Central Africa. In fact, all major microfinance programs supported by DFID in the region are expected to include impact assessment and monitoring systems. So although the amount allotted to CETZAM was higher than what is typically considered for “low-cost” efforts, it represented a long-term investment in the region.

What questions was CETZAM trying to answer?

As it pursues its goals of holistic client transformation and sustainability, CETZAM is focused on a “triple bottom line”—i.e. maximizing outreach, quality and impact. Breadth of outreach has been simple to measure. CETZAM uses a group-based poverty lending methodology called “Trust Banks” designed to reach the poorest of the self-employed poor, especially women[6]. With this methodology, CETZAM served 1,700 active clients within 10 months of start-up. Two years after start-up[7], CETZAM expanded its outreach to 7,500 active clients and had branches in Kitwe, Ndola, Chingola, Mufulira and Kalalushi, exceeding all its grant targets. Within five years, CETZAM hopes to expand nationally to 24 retail outlets across the country.

As with outreach, quality of the portfolio has been easy to track. Presently, arrears (over 30 days) stand at 1.71 percent, and portfolio-at-risk (over 30 days) stands at 1.99 percent. So despite CETZAM’s strong growth trajectory, it has maintained high quality performance.

Unlike outreach and quality, however, impact has been difficult to assess because there are no standard industry measures. Therefore, CETZAM had to decide what questions it wanted to answer and why. The board started the process at the policy level by deciding what kinds of change they were hoping to see. Then based on that guidance, CETZAM’s management team emerged with the following four questions:

· Are we reaching clients in our target market and having positive effects in their lives?

· Are we meeting our goals for generating income and creating employment?

· Are we facilitating positive changes in the economic, social and spiritual lives of our clients?

· How many clients leave our program? When? Why?

Why these four questions? First, although it could easily measure its breadth of outreach, CETZAM wanted to know the depth of outreach too, since its primary lending product was designed to reach the poorest of the self-employed poor.

Second, CETZAM’s mission statement included the goals of generating income and creating employment. To that end, two specific targets for measurable impact were included for the first year of operations: 1) an increase in clients’ business income by an average of 20 percent; and 2) improvements in employment and household welfare among 75 percent of clients[8].

Third, CETZAM had focused on a client-centered approach to product development and program management from the outset. Within three months of start-up, it conducted a comprehensive stakeholder survey, including clients, community leaders, board members and donors. CETZAM used the responses to learn what its key constituencies expected in terms of how loans would improve clients’ lives and the community. This feedback provided additional criteria against which the CETZAM board and management could measure impact, and included positive changes in the economic, social and spiritual domains of clients’ lives.

Finally, CETZAM understood that client satisfaction was critical to its long-term sustainability. High client turnover has a high cost; it also might signal that CETZAM’s products and services are not designed to meet clients’ needs, or that they have resulted in negative impacts on clients’ lives.

How did CETZAM choose its consultant team?

CETZAM, together with DFID, looked for a consultant team that could help design a system to answer these questions. Of primary importance to CETZAM’s management was finding a consultant team that could offer:

· Experience in developing impact assessment tools and monitoring systems, especially within the region;

· Knowledge of the local context and culture; and

· Readily available on-site support and training.

They chose a Zambian consultancy, M&N Associates, with the Centre for Development Studies at the University of Bath, UK. The team had recently completed an impact assessment study of the CARE/PULSE microfinance program in Lusaka, but was keen to go beyond a single-study approach by helping to build ongoing impact monitoring and assessment systems[9].

Their first task was to develop a plan based on the following factors: a) a focus providing useful results on client-level impact and client dropout that would help CETZAM’s management make better decisions; b) a recognition that CETZAM’s program was a young but growing one; and c) the desire to build on other tools and systems that had been field-tested and worked well elsewhere. Clarifying the consultants’ role and the project’s objectives emerged as important keys to a good working relationship with CETZAM staff and management. The consultants’ roles were as follows:

· Design of tools

· Testing of tools

· Training of local Impact Monitoring Officer (IMO)

· Supervision of IMO’s work conducting surveys and coding data

· Analysis and development of quarterly reports.

How did CETZAM customize and implement the tools?

Given these priorities and tasks, three primary impact assessment and monitoring tools were chosen: a ) a means test, b) an impact survey (matching sample survey), and c) an exit survey. CETZAM customized and implemented them in the following ways:

Means Test: The “means test” is a poverty-ranking tool that helps determine whether residents within a given area fit within the target group and are eligible to become potential clients. In the process, it gathers standardized economic and social data that can serve as “baseline” information. CETZAM adapted its means test from those used by other partners in the Opportunity International Network. The adaptations required to fit the Zambian context were few and centered on identification of demographic factors that represented the target market group. In Zambia, those factors included items such as the number of dependents, number of children attending school and the spouse’s likely support/interference of the client’s involvement with CETZAM.

Loan officers conduct the means test in a potential client’s home or business after the client has attended the first informational Trust Bank meeting. Completing the test is a requirement for continuing with the orientation process and receiving a loan.

Impact Survey: The impact survey compares existing one-year-old clients with a “comparison group” of comparable clients receiving initial orientation. The survey was chosen because it provides data that can be quantified and interpreted relatively quickly and easily, while also helping to build up a database that can be analyzed more rigorously within a quasi-experimental analytical frame at a later stage.

Adapted from the survey used with CARE/PULSE in Lusaka, the questions also were matched with those from other business and household surveys used in Zambia. This helps produce data comparable with other surveys. And based on the AIMS guidelines, the survey was refined further after two rounds of pre-testing with CETZAM clients. Some of the resulting questions include:

· What were the gross profits of your primary business for the last full month?

· How many full-time paid employees did you have last month? One year ago?

· How would you describe your role in household decision-making?

· Do you have a paid subscription to a health clinic?

CETZAM was eager to ensure the successful implementation of the survey, but its loan officers were focused on implementation of the means test and the demands of expansion and growth. So it created a new full-time impact monitoring officer (IMO) position to conduct the surveys and code and enter the data. Although officially hired by the local Zambian consultancy, the IMO was seconded to CETZAM and works closely with the loan officer staff[10].

Exit Survey: The exit survey was adapted from a standard one developed and documented by Carter Garber in the AIMS Draft Manual. (The AIMS survey, in turn, was adapted from one originally developed by the Women’s Opportunity Fund and used by other Trust Bank programs within the Opportunity International Network. CETZAM therefore benefited from a healthy process of testing, adaptation and best practice exchange, and now contributes to that process through its own testing.) The exit survey expands upon the client satisfaction questions in the impact survey and helps answer the question, “How many clients exit? When? And Why?” Some of the specific questions include:

· What was the best aspect of joining a Trust Bank? The worst aspect?

· How did you find the weekly meetings?

· What was your main reason for leaving your Trust Bank?

· Was your withdrawal temporary or permanent?

CETZAM’s clients save 10 percent of their loan amount in compulsory savings, and initially loan officers were to carry out the exit surveys with each client prior to the withdrawal of his/her savings. But the pressure of expansion meant that loan officers either completed the surveys improperly or not at all. As a result, the IMO has taken on the responsibility for carrying out exit interviews with a random sample of Trust Bank clients in different loan cycles.

Although an added “transaction cost” to clients, completion of the exit survey ensures that clients have an opportunity to express their satisfaction, or dissatisfaction, with CETZAM’s products and services in a timely and direct way. Also, clients can bring up issues not included in the impact survey, which may help interpret other responses.

What are the initial findings?

CETZAM has completed two rounds of quarterly monitoring. The first was in January 2000, and the second was in May 2000. The first round of impact survey results covered responses from 81 clients total, including 41 existing one-year-old clients and 40 “new” clients in the comparison group who had completed the means test but had not yet received loans. The second round covered 104 clients total, with 60 existing clients and 44 comparison group clients[11].

CETZAM was eager for answers to its impact questions but did not want to fall into the trap of measuring impact over too short a time period. The first two rounds have been useful in refining the tools and identifying questions that require further study. Among the useful preliminary data about the target market were findings on the gender breakdown of clients, their average number of dependents, and their access to household services such as electricity.

Also interesting were preliminary indications of economic, social and spiritual changes in the lives of clients, including:

· Business: 93 percent of existing clients reported that being a Trust Bank member gave them business contacts and new ideas. Also, 55 percent reported that being a Trust Bank member helped keep their business going, 27 percent said it helped expand their business and 10 percent said it helped change their business.

· Health: 62 percent of existing clients reported paying a health clinic subscription, compared to 38 percent of the comparison group[12].

· Decision-making: 33 percent of existing clients reported making the final decisions regarding business activities and household expenditures, compared to 13 percent of the comparison group[13].

· Leadership: 87 percent of existing clients felt that being a Trust Bank member developed their leadership skills.

Beyond this client-level impact information, CETZAM was also interested in the dynamics behind its client dropout figures. Its management information system (MIS) provides numbers on how many clients exit the program, but it does not indicate when or why clients leave. Responses from existing clients to the client satisfaction questions in the impact survey provided some insight, however, including:

· Loan size: 9 percent of clients said the loan sizes were too small (compared to 29 percent in the first round).

· Interest rates: 10 percent felt the interest rates were too high (compared to 32 percent in the first round).

· Repayment frequency: 32 percent said that loans had to be repaid too quickly (compared to 76 percent in the first round).

· Conflict: 22 percent said that their loans had been a source of conflict.

How has CETZAM’s management used the findings in its decision-making?

CETZAM is still in the early stages of impact assessment and monitoring, and management has been cautious in making substantial changes based on the initial findings. In fact, the first round of data gave some impressions that changed upon further information in the second round. This has led CETZAM to incorporate the use of qualitative impact assessment tools in the next round of monitoring to generate more in-depth information. It has also led CETZAM to direct more resources to its new internal market research department. This department will work on key market research questions and outsource very specific, detailed projects to a local firm. Both will be important in better understanding the dynamics behind the results generated through the client satisfaction and exit survey questions.

In the meantime, however, CETZAM’s management has used the preliminary information from the first two rounds of impact monitoring to make better client-focused decisions in the following areas:

1. Determining loan sizes. The first impact monitoring report indicated that new clients coming into the program are on average poorer and less educated than those who joined earlier. CETZAM believes this is resulting from its movement into poorer townships. Management used this information to conduct a sample survey of the average capital that new incoming clients had invested in their business. As expected, they had smaller businesses and thus could absorb smaller initial loan sizes than their counterparts from one year earlier.

Management then worked with loan officers to strengthen the training in determining loan sizes and forming groups. This resulted in an overall drop in the initial loan size and a slower growth rate in loan sizes among new clients. Lowering the initial loan size will have a negative impact on those borrowers in the short run; but CETZAM hopes that this will avoid potentially burdening the more vulnerable incoming clients with too much debt. At the same time, it also hopes that their businesses and loan sizes will grow over time and that in the long term clients will benefit.

Yet despite this poorer group of incoming clients, CETZAM also serves another group of clients who said that initial loan sizes were too small. In fact, 29 percent of existing clients in the first round of monitoring reported this. CETZAM investigated further, discussing the issue with loan officers and gathering more information. In the end, given the currently high inflationary environment in Zambia, it decided to increase the maximum initial loan size by 40 percent. This amounts to an inflation-adjusted increase of about 10 percent; it also makes the loan officers’ role in determining the appropriate initial loan size more important.

2. Changing repayment frequencies. Prior to receiving the impact monitoring reports, management had heard some anecdotal information that clients were finding the weekly payments too burdensome. As a result, it was about to change the repayment frequency from weekly to fortnightly. But the impact monitoring revealed that clients overall were satisfied with weekly repayments. Therefore, in the end management decided not to change the repayment frequency, and instead worked with loan officers to emphasize to clients the importance of weekly repayments during orientation sessions and weekly meetings.

3. Introducing new products. As part of the follow up on the anecdotal information about burdensome weekly repayments, CETZAM discovered that the majority of complaints were coming from clients in their fourth or fifth loan cycle. These clients were making new investments in their businesses and had different cash flow requirements than clients taking out their first or second loan. Equipped with this knowledge, management worked with these clients to develop a new loan product more appropriate to their needs.

The new product is a solidarity group loan. Designed to meet the financing needs of the most successful Trust Bank clients, the product rewards them for previous on-time repayments by offering larger loans with less frequent repayments. Solidarity groups are comprised of five to seven members who have completed four Trust Bank loans and who have written records of their cash flow for the past six months. The loans have a maximum amount of US$800, a maximum term of 18 months and carry fortnightly repayments. To date, clients have maintained 100 percent on-time repayments.

CETZAM also learned that cash flow requirements are not the only burden that clients bear. Funerals for family members also place an intense burden on clients’ lives and their ability to make weekly or fortnightly repayments. To address this, CETZAM began working with a local insurance company to develop an affordable insurance product that will cover funeral costs for clients and six of their dependents. This product will be introduced in September 2000 and will cover all borrowing clients. CETZAM will act as an agent for the product, with a local insurance company underwriting the policies.

4. Understanding client dropout. CETZAM’s MIS reported a dropout rate of 25 percent annually, in line with other programs in the region but higher than its goal of 15 percent. However, the MIS doesn’t indicate when clients leave or why.

Initial responses from the exit surveys seem to indicate that the dropout rate is consistent across 5 loan cycles. If the full analysis confirms this, CETZAM can further probe the reasons why clients leave. They may be internal ones related to CETZAM’s products or procedures (e.g. bad relationship with the loan officer), or external ones related to the economy or other pressures. Anecdotal evidence suggests that most clients exit because they cannot meet their repayments, either due to illness, a funeral in the family or business problems. But some clients simply may want to rest and rejoin later. CETZAM plans to complete the analysis of the exit survey data, implement the qualitative impact monitoring tools and conduct more in-depth market research to better understand these trends.

5. Improving its incentive system. 89 percent of CETZAM’s existing clients were either positive or very positive about their experience. But the dropout rate is much higher than CETZAM would like and there is room for improvement. Consequently, CETZAM has added a client retention rate measure to its loan officer bonus scheme. If a loan officer has higher than a 20 percent dropout rate, then he/she is not eligible for a portion of the monthly bonus. This 20 percent rate will be adjusted over time, along with the weighting of the client retention factor, as CETZAM gathers more in-depth information on the reasons behind client dropout.

How are the results presented?

One of the key factors that make the findings useful is that CETZAM presents the results quickly to clients, staff, management and board members in a combination of forms. These include:

· Quarterly Impact Monitoring Reports

· Quarterly Feedback Meetings

· Annual Impact Monitoring Reports

· Annual Focus Group Discussions

The consultants produce the quarterly Impact Monitoring Reports which consist of three parts: a) an executive summary of findings; b) a tabular summary of data from the means tests, exit interviews and sample surveys; and c) an analysis report that interprets differences in the data for each category of impact indicators. This extensive feedback is another key element in ensuring that the impact monitoring process is practitioner-led and geared to improving management decision-making.

Using these reports as a basis for discussion, CETZAM’s management team holds quarterly meetings for staff to provide feedback on the findings. They discuss the implications for operations, existing products, potential new products, incentive systems, expansion, etc. This is a participatory process that encourages open dialogue and interpretation of the findings. When staff members are involved in this way, it gives them ownership over the process and results.

The CETZAM board also reviews the report in light of what they were hoping to achieve and then gives feedback as part of the quarterly meeting. Specific areas (e.g. holistic transformation) have gone to specific committees dealing with these issues. For example, when the impact monitoring results revealed issues concerning initial loan sizes, the board’s credit committee used the findings to form its new policy decisions.

As the end of the first year of assessment and monitoring approaches, the consultants will produce a summary annual report. It will include both quantitative results and narrative explanations of the major trends and upcoming issues concerning client-level impact and client satisfaction. This report will be presented to CETZAM’s board, management and staff, as well as to DFID.

Based on this report, a local market research firm will conduct an annual focus group discussion to allow clients to provide feedback on the findings. Clients will be chosen at random and the discussion will be videotaped. Management will use the discussion to help interpret the survey results and improve the impact monitoring process.

How do these efforts compare to the principles of an impact audit?

So far, we’ve presented a history of what CETZAM has done, but one outstanding question is: how does the concept of an impact audit hold up in the real world? CETZAM did not set out to implement an impact audit system per se. But it may be instructive to examine how its efforts compare to the principles outlined earlier, i.e. does CETZAM’s approach:

Provide analysis of trends over time and produce results comparable with previous impact data? Yes. CETZAM’s data collection includes responses from existing clients and new clients who serve as a comparison group. Next year, the existing client group will include two-year old clients, as well as one-year-old clients. And with each succeeding year the existing client group will include the oldest client group (e.g. five-year-old clients) and another mid-range group (e.g. two-year-old clients). This aging of the sample population will provide analysis of trends over time. Also, because the “baseline” data is generated from a standard means test and the impact data is generated from random samples of the client population, future results will be comparable to previous findings. Finally, because the impact survey questions were adapted in light of other household and income surveys conducted in Zambia, the results can be compared against national and other external sources of data.

Integrate data collection with the regular information system of the institution? Yes and no. CETZAM tracks its impact monitoring data independently from its loan portfolio information system (which tracks loan disbursals and repayments). However, the two systems (impact and portfolio) both feed into the overall information and decision-making system that provides reports to clients, loan officers, managers, board members and donors. CETZAM views its management information system in a broad sense, i.e. not as a single database, but as a collection of systems which together provide the information needed to make good decisions. Thus the most salient feature of the impact database is its flexibility in “cutting” the data. For example, perhaps traders are more satisfied than manufacturers, or clients in their fourth loan cycle are less satisfied than those in their first. Analysis by loan officer, business sector, location, loan cycle, age of the client, etc. can help answer those types of questions. Together with data on outreach and portfolio quality, this allows CETZAM to make better client-focused decisions.

Utilize internal staff with a limited role for external experts? Yes and No. CETZAM did not have the in-house capacity to set up its own impact assessment and monitoring system. Together with DFID, it had to invest resources of time and money to create measures, tools and systems for doing so, and it needed outside consultants with expertise and experience.

The consultants worked closely with CETZAM’s management in setting up the system and training the impact-monitoring officer. But now that the system is in place, the external consultants’ role has shifted. It has become one of overall responsibility for the technical quality of the data and analysis, as well as development of the qualitative impact assessment and monitoring tools that will complement the existing quantitative ones.

Although the consultant still plays a significant role, it is a clearly defined one that is appropriate in CETZAM’s view. CETZAM’s core business is making loans and developing new financial services, not statistically analyzing impact data or developing new impact assessment and monitoring tools. However, the IMO hired and trained by the local consultant works closely with the staff.

Cost no more than a system to track and audit financial information? It depends. Presently CETZAM spends approximately £2,500 (US $3,762) annually for its financial audit, while it spent £25,000 (US $37,620) on its first year of impact assessment and monitoring. But a straight comparison cannot be made between the two figures, as it would be like comparing apples to oranges. At start up, CETZAM had to invest in hiring a financial director, accounts assistant and bookkeepers in order to track and audit its financial information. In the same way, it has had to invest in hiring an impact monitoring officer and consultants to be able to track and audit its impact information. Only unlike with tracking financial information, the measures, tools and systems did not already exist and had to be created.

Of the £25,000 spent in the first year, £16,000 (US $24,077) went to the local consultancy for development of the survey instruments, set up of the monitoring system, analysis of the data and training of the IMO. Creating this type of infrastructure is not reflected in the comparable costs of a financial audit, since these tools were developed long ago in the financial community. To compare them equally, the costs of maintaining all financial and accounting information, including the costs of the MIS software and salaries for the financial manager, accounts assistant and bookkeepers, would have to be included[14].

Nevertheless, while this is still a substantial sum of money, there are two additional and important factors to keep in mind. First, once the system is functioning well and CETZAM brings it fully in-house, it will be substantially less expensive. Second, it must be remembered that for CETZAM the impact monitoring costs (including consulting costs) are under two percent of its portfolio on an annual basis. If the information provided by the impact monitoring helps to reduce arrears by just one percent and increases retention rates by five percent—it already pays for itself. In CETZAM’s view, the potential benefit is worth much more than the cost. This is increasingly true as it uses the data gathering and analysis processes for the dual purposes of impact assessment and market research.

What lessons has CETZAM learned along the way?

Impact assessment and monitoring systems can be donor-funded without being donor-driven. Both DFID and the consultant team were dedicated to designing a system based upon CETZAM’s decision-making needs. The goal was not just to provide data for donors, but to help CETZAM fulfill its mission, remain competitive and better serve its clients. Staff witnessed this through the close working relationship between the consultant team, DFID and the CETZAM management. And while CETZAM has used grant funding to develop its system, the benefits for decision-making have already convinced CETZAM to use its own operating income for in-depth market research. This will help it answer client satisfaction questions raised through the impact monitoring.

Consultants play an important role in creating an impact assessment and monitoring infrastructure. CETZAM could have pulled any number of assessment and monitoring tools “off the shelf” and implemented them in isolation. But it was interested in creating a system that would feed into management decision-making. It did not have the in-house skills to customize the tools and to do the data analysis that would produce high quality results useful for management and staff. Yet it developed a very clear and well-defined role for external consultants that brought added value to its own skills. Then it chose a consultant that was located close enough to the organization to quickly build solid communication, trust and mutual learning.

The measures, tools and system are not static. Initially, CETZAM used only the means test and impact survey. However, the initial round of data collection only hinted at the dynamics behind client drop out. The management team then asked the consultant to expand the section on client satisfaction in the impact survey and add the exit survey to produce more comprehensive data on client retention. (Interestingly, this evolution means the tools are now more similar to the AIMS tools.)

When initial responses from the exit survey indicated that clients might not be responding accurately, CETZAM’s management team sought feedback from loan officers. They suspected that clients might have felt uncomfortable disclosing their reasons for dropping out to loan officers, for fear it might jeopardize their chances to access future loans. At the same time, loan officers were completing the surveys poorly or not at all because of time and expansion pressures. Thus, CETZAM shifted the responsibility for conducting the exit surveys to the IMO. This has helped improve the quality of survey responses.

Integration into daily operations helps build ownership among the staff. In the early stages of implementation, loan officers were apprehensive that the IMO was there to check up on them. But as he worked with loan officers on integrating the implementation of the tools into daily operations, they built a close working relationship that eased loan officers’ concerns.

Sharing the results and asking for feedback in a timely way also helps build ownership among the staff. More importantly, loan officers soon began seeing the results of the data collection and analysis, were asked for feedback on the initial findings in a timely way, and saw that management was taking their input seriously. This built support for and ownership of the impact monitoring efforts.

The results can increase morale. While loan officers often have an intuitive or anecdotal sense of various types of impacts, quantifying them can demonstrate the influential role they play with clients on a day-to-day basis. Also, when loan officers see positive outcomes recognized through incentive schemes, it may help them feel more appreciated and motivated to better performance.

The process and findings can communicate powerfully to donors and policy-makers. Even though CETZAM’s findings are preliminary and it has a long way to go in improving its system, the board, management and staff have embraced the concept of impact monitoring. Instead of being afraid of the findings, they are using them to try to better serve clients, become more competitive and fulfill the organization’s mission. Also, the data and analysis provided through the impact monitoring are more powerful than anecdotal stories when CETZAM seeks additional funding or advocates for policy decisions on microfinance issues with government officials. This communicates to donors and policy-makers that CETAM is serious about reducing and eliminating poverty.

What challenges remain?

For rapidly growing organizations like CETZAM, staff resources will have to keep pace with growth and expansion pressures. To date there has been only one impact-monitoring officer. Initially this was appropriate because CETZAM’s client base was small and the system was being developed. But CETZAM’s rapid growth and its addition of qualitative impact assessment tools now require the hiring of two more impact monitoring officers. In retrospect, CETZAM could have used an additional person sooner to speed up the collection of the exit survey data and analysis of the means test data. As it moves forward, it will have to continue fine-tuning the balance between its impact assessment needs and its staff resources.

Similarly, organizations will have to seek the optimal balance between carrying out certain impact assessment and monitoring functions in-house and outsourcing other functions to consultants. Bringing everything in-house may seem like an attractive, short-term way to lower costs. But in CETZAM’s view, focusing on its core business (i.e. making loans) and letting the consultant team do the same (i.e. designing impact assessment and monitoring tools) has been an important factor in making the system successful in the short run and cost-effective in the long run. Regardless of the specific arrangements, however, both consultants and in-house staff play critical roles, and working through the many issues surrounding contracting is important for ensuring a smooth process overall.

In addition, establishment of standards against which organizations can measure their performance is a crucial need. Financial audits can clearly compare internal financial practices against established external standards. Microfinance organizations could benefit from a set of common standards for “good practice” that would help them assess their impact performance just as they assess their outreach or quality performance. In the meantime, organizations like CETZAM can compare their performance against internal standards and look at evolving trends.

Also, using both impact assessment and market research to produce better client-level information is an exciting challenge. As already noted, CETZAM’s approach serves the dual purposes of monitoring impact and providing information for market research. From its perspective, these do not represent an “either/or” choice, but two different sides of the same coin. As such, CETZAM is devoting resources to both. It is implementing a new coding system for its impact data so that it can be “cross cut” and analyzed in more ways for market research purposes. To reduce costs, however, data collection and analysis for very specific in-depth market research questions will be outsourced to a local firm. At the same time, CETZAM is introducing qualitative impact assessment and monitoring tools based on participatory approaches. These will help generate richer data about the economic, social and spiritual impacts that more mature clients experience as they continue to participate in CETZAM’s program.

Finally, as CETZAM approaches the end of the first year of its impact monitoring efforts, it will present the findings to clients and ask for their feedback through a combination of reports and focus group discussions. Maintaining a focus on clients’ participation and feedback is critical if the findings are to be of use to them as well as to other decision-makers. Creating opportunities for client input and reflection throughout impact assessment and monitoring process is an important task for everyone.

Conclusion

CETZAM’s goals are not the goals of every microfinance institution, and its approach is not one that can be adopted “as is” by every microfinance institution. Also, CETZAM is still a young organization, and its impact assessment and monitoring system is changing and evolving as it grows. Its efforts represent one approach to designing a system that helps management make better decisions, improve products and services to clients and understand whether it is fulfilling its mission to reduce and eliminate poverty. We hope that other practitioners will learn from and improve upon CETZAM’s experience, “embracing the baobab tree” and sharing their wisdom with the microfinance industry.

For more information

If you would like a copy of the tools developed for CETZAM, you may download them here:

Intake Form
Exit Interview Form
Means Test Form

About the Authors

Susy Cheston is the Executive Director of the Women’s Opportunity Fund, which provides technical assistance and funding for group-based poverty lending through the Trust Bank methodology. Larry Reed is the Managing Director of the Opportunity International Network, a global network of over 50 microfinance institutions. Suzy Salib is a Program Manager with the Women’s Opportunity Fund. Rodger Voorhies is Opportunity’s Assistant Regional Director for Africa, based in Kitwe, Zambia. James Copestake is a member of Bath University’s Centre for Development Studies in the United Kingdom.

The opinions expressed are those of the authors and do not necessarily reflect those of the Microcredit Summit Campaign.

Return to papers

Return home

[1] The baobab tree is native to Africa and has a barrel-like trunk that may reach up to nine meters (30 feet) in diameter and 18 meters in height.

[2] The Small Enterprise Education and Promotion (SEEP) Network is a coalition of North American NGOs working in microenterprise development; the Assessing the Impact of Microenterprise Services (AIMS) Project is sponsored by the United States Agency for International Development (USAID).

[3] CETZAM is still early in the process of doing this impact work and, while the results are statistically reliable, it is still too early to draw broad based conclusions. Also, the results tell us something about clients’ behavior before and after involvement with CETZAM but they do not under any circumstances prove causation. The impact work CETZAM is doing is in the beginning stages and, as CETZAM says, “we have a long way to go”.

[4] Indicators of Economic transformation may include increased household income or greater business growth, as well as economic changes at the community level; Social transformation may include an increased self-confidence or improved social relations; Political transformation may include increased political participation, as well as greater gender equity; and Spiritual transformation may include increased application of moral values or greater freedom and hope.

[5] DFID Country Strategy Paper Zambia, 1999.

[6] Trust Banks typically are comprised of 15-40 people, divided into sub-groups of 5-7. Several Trust Bank programs have achieved high standards of sustainability and financial efficiency while promoting economic, social, and spiritual goals through leadership development, community service, and activities such as training during weekly meetings.

[8] Note that it is very difficult to measure business and household income and changes in employment and household welfare. CETZAM’s measurement instrument involved a long-term interview using a variety of questions.

[9] Copestake, J, 2000. "Impact Monitoring and Assessment of Microfinance Programmes: Scope for Integrating Impact Studies and Market

Research.” Development in Practice, 10(4).

[10] This solution worked well in terms of ensuring high-level technical supervision from the consulting team and protecting the impact efforts from the pressures of expansion, but it also creates a dependency on one person’s technical skills and does not build in-house knowledge as quickly. (This is an issue for any highly specialized work.) This will change when CETZAM brings the IMO in-house as part of the Marketing Department, and with the addition of a second IMO. Also, since the IMO is in the CETZAM office every day, he is already a part of the team.

[11] This means that, in the first two rounds, four different groups were surveyed.

[12] CETZAM is trying to see how the group acted upon (the people who are part of the program) compare to those who have not yet entered the program (the control group). CETZAM uses entering clients as the control group since they are assumed to be similar to the people who have been with CETZAM a year. CETZAM then tests this assumption with general demographic data. CETZAM measures the new people after one year and compares what has changed.

[13]The rest report decisions made by husbands, family, etc. The significance is the difference between the two groups, not the absolute number.

[14] For example, CETZAM has a finance manager and 10 bookeepers to maintain its financial information. If together they earn £2500 per month (reasonable given salaries in Zambia), this equals £30,000 per year, or more than the cost of the impact monitoring system. This is for illustrative purposes only, but demonstrates the need to compare costs equally.