PREPARED BY
Md. Mustafa Kamal

Introduction

Any organization that implements a development program is eager to have answers to questions such as: 
> Are our objectives being fulfilled?
> Are the right clients being reached?
> What are the obstacles to our clients' development?
> Are any changes or modifications to our objectives or our programs needed in light of scientific assessment?

The same was true for The Association for Social Advancement (ASA) whose management felt the need for impact evaluation to ensure better performance. The Managing Director of ASA once said, "We want to strengthen evaluation as a mechanism in order to develop reliable tools and techniques so that greater reliance can be placed on our own capabilities." With this determination, ASA management established a structure and developed a mechanism to assess the impact of activities launched by the institution. This approach ensures combined efforts from the Research and Evaluation Cell, Program Coordination Cell and Management Information System (MIS) Cell. Regular and periodic assessments are conducted from these cells to determine actual achievements, make accurate decisions on the basis of recommendations, and formulate appropriate policies for better performance. 

Great emphasis is placed on assessing the impact of ASA's programs due to the positive correlation between the success of these programs and the target clients' livelihoods. ASA's impact assessment mechanism works smoothly and efficiently and adheres to the development strategy of the institution. After reviewing the impact assessment results, ASA promptly created several innovative changes to attain its present standing.

EVOLUTION AND TRANSFORMATION OF ASA: SUMMARY

ASA started working as a development non-governmental organization (NGO) in Bangladesh in 1978 with a radical action agenda. The program's focus was on consciousness-raising and group organization for the poor. ASA later redesigned its strategies in accordance with its philosophy of poverty alleviation and empowerment of the poor. This brief account of the changes in its approach and programs will give a clear picture of how the organization reached its current mission. Three distinctive phases can be identified in the transformation of ASA: 1) foundation phase (1978-1984); 2) reformative phase (1985-1991); and 3) program specialization phase: savings and credit for income generation (1992 to date). 

Foundation phase (1978-1984):

During this phase, the main activity of ASA was to organize the landless and the poor to establish their rights. The programs at this phase were: awareness-raising for social action, legal aid, training for the landless and training for journalists in the rural areas. All of these programs were designed to make the poor conscious of their rights and enable them to take steps to stop social exploitation. Some examples of the actions taken by the groups mobilized by ASA include: initiatives to gain access to khas land (government owned land); ensuring fair wages; and protecting the legal rights of the poor. During this period, ASA only organized men in the rural areas.

Reformative phase (1985-1991):

In 1985, ASA assessed the impact and limitations of its initial programs. The conclusion was that the impact of the programs was generally positive but could not be sustained. Another realization was that development could not be sustained without the economic emancipation of the poor. ASA introduced an integrated development program comprising: development education for empowerment, credit for income generation - first introduced on an experimental basis in 1987, mini-irrigation programs for the landless and marginal farmers, a program for women's development, a health program (primary health care, nutritional improvement, training of traditional birth attendants), human development training, and a rehabilitation program. Along with individual credit, ASA emphasized loans for collective enterprise that were taken by all members of a group. Unfortunately, the group projects failed miserably.

Program Specialization Phase (current phase of ASA):

During the 1980s, the credit program became very popular and development practitioners became convinced that providing financial resources in the form of credit was an effective way to craete self-employment, providing for income generation and poverty alleviation for the poor. Most NGOs introduced credit programs for accelerated development, but ASA lagged behind. Thousands of group members were asking for loans which ASA was unable to provide for several years. Members started to leave ASA and join other lending organizations. The members' demand for loans prompted ASA to review its approach. Since 1992, ASA has concentrated on a credit program to provide loans to members at a reasonable interest rate allowing clients to improve their standard of living. ASA also changed its strategy by becoming an organization for women instead of men. By the end of 1996, 98.6% of its members were women.

ASA's cost-effective and sustainable development model

ASA has developed a model for implementation of savings and credit programs which provides quality financial services and meets the needs of its members. In 1995, ASA was successful in merging the concept of financial sustainability with empowerment through increasing the number of its disadvantaged women clients with quality financial services. At the same time, ASA was able to provide sustained service delivery by attaining financial viability as an organization. The borrowers' profitability led ASA to design a cost-effective and sustainable development model.

Institutional Mission

The institutional mission of ASA is to support, impel and strengthen the lower status economy through facilitating, dispensing and expanding savings and credit for those segments of the population that find it difficult to gain access to credit.

Broad Objectives

The overall objectives of ASA are to provide financial services which help to alleviate poverty and improve the quality of life of the landless and assetless rural poor.

Development Strategy

The development strategy of ASA is to provide credit to all members, especially women and the disadvantaged, and resource mobilization through voluntary savings, to fight poverty.

Present programs under implementation

ASA has specialized its program in microfinance and gradually phased out all of its other programs. This process of exclusion began in early 1991. Since then, the institution has focused on strengthening and empowering the country's poor women. ASA turned towards financing the small income-generating activities (IGAs), in which poor women may be involved in addition to their household work. The following programs are under implementation:

1. Microcredit program: ASA provides small loans to the landless poor group members with a 15% service charge. The amounts are invested in the IGAs, which are profitable. ASA's microcredit program aims to ensure financial sustainability of the members as well as the organization. 

2. Microsavings program: Small accumulated savings help further lending and provide benefits for increased number of the poor. The savings prevent dependence on outside financial assistance. Both mandatory and voluntary savings programs are being implemented at ASA. 

3. Small Business program: ASA implements this program for the small traders who lack necessary capital to run their businesses smoothly. The traders' livelihoods are fully dependent on these small trades.

4. ASA-NGO Partnership program: The main objective of this program is to assist in giving loans to the local small NGOs who are involved in microfinance activities in the remote areas of Bangladesh. 

Impact evaluation mechanism

In ASA, the impact of programs is assessed through:

1. program and management monitoring at different levels;

2. monthly, semi-annual and annual coordination meetings at different levels;

3. regularly conducted evaluation and impact assessment studies; and

4. analysis of the various field reports.

Program and management monitoring at different levels:
ASA conducts regular monitoring activities to maintain the quality of its programs. Close monitoring is conducted at all levels of management. At the field level, the Branch Manager oversees the activities of the branch. Central to this monitoring is the verification of records in the branch office and that of the members' passbooks. The accounting system acts as the source of all information. The Area and the Divisional Managers are also responsible for monitoring the activities of the branch offices. 

From the central office, program monitoring is basically conducted through the Coordination and Implementation Cell. The cell consists of 4 General Managers (GMs), each with well-defined responsibilities. The GMs report to the Managing Director (MD) with whom they regularly share problems, policies and activities for quick decision-making, program implemention and smooth functioning of the organization. The cell is responsible for selecting activity areas, developing job descriptions for field workers, submitting annual plans and implementing and coordinating the program component according to an operational plan. The General Managers check and approve monthly savings and credit reports, monthly receipts and payments accounts. They also arrange monthly coordination meetings. The members of the central monitoring teams visit the field three times a month and each of them is required to spend at least 15 (fifteen) working days monitoring all the branch activities. Monitoring checklists have been produced for this purpose and are used at different levels. The checklists include the important issues of:

1. Personnel management: Assessing staff quality, orientation and training.

2. Office management: Record keeping and account maintenance.

3. Program operations management: Credit program, savings program, dropout, overlapping and default management.

4. Fund management: Money transfer, bank transactions and bank account maintenance.

5. Implementation of policy matters.

6. Group management.

7. IGA management by the group members.

8. Influence of programs and organizational activities on the group members, their families and on the people of the locality.

The findings along with recommendations are presented to the Managing Director for immediate action. Policy matters are shared in the coordination meetings where decisions are made after elaborate discussion of the issues.

Monthly, half-yearly and yearly coordination meetings at different levels:
Regular coordination meetings are held at all levels of the organization - the central office level, area manager level, and branch manager level. The central office level meeting is held once a month and attended by general managers (GMs), divisional general managers (DGMs) and area general managers (AGMs) only. Divisional managers attend coordination meetings once every three months. The area level meeting is held twice a year and the branch manager level meeting is held once a month in the field. At each of these meetings there is a specific agenda with particular issues on which decisions must be made.

The Managing Director makes trips to the field to discuss new policies and concerns and listen to the complaints of the branch and the area managers. The Managing Director's as well as the General Manager's frequent personal contact and dialogue with area managers during the semi-annual area managers' meeting and other times is indispensable in keeping the central office in tune with what is happening in the field.

Central office coordination meetings last one day. A central office level coordination meeting may have 3-5 minutes discussion on an issue that will have an immediate impact on beneficiaries. Senior staff discuss the pros and cons of certain policies, and if the pros seem to outweigh the cons, implementation begins.

When a decision has been made to introduce a new policy or alter an existing policy, a circular is prepared at the central office and sent to each branch. In the field, the branch manager is held directly responsible by his area manager and divisional manager for any and all information in the circulars. Circulars are taken very seriously and kept in a separate file. Within a month after the circular has been published, any policy change is to be incorporated into each branch's operations with credit officers fully aware of any changes and the implications. Coordination meetings and communication through circulars and frequent field visits help keep ASA dynamic. 

ASA arranges area and divisional-level staff coordination meetings. The meetings are held half-yearly and annually during which reports of the activities are produced, staff performance is evaluated and plans for the future are made.

Conducting evaluation and impact assessment studies regularly:
ASA's Research and Evaluation Cell conducts regular impact evaluation studies. Problems or issues are identified through field visits and findings of the reports from the field offices. After sharing their importance in the coordination meetings, a decision is made to conduct a study on a topic selected at the meeting. To conduct the study, a required methodology is followed and necessary tools are developed. The recommendations of the study are considered by the management and required policies are formulated.

A checklist is used for management monitoring which is a regular function of this cell. The Research and Evaluation Cell also helps pinpoint any problems or irregularities in operations and presents the issues at coordination meetings for policy formulation.

Analysis of field reports of all kinds:
The field level offices send regular reports to the central office. Three cost-effective offices are maintained at the field level: divisional, area and branch offices. It is notable that there are no separate offices for the Area Manager and Divisional Manager. They are part of the branch offices, from which they oversee all the branches within their respective jurisdiction. However, the branch offices, area offices and divisional offices send reports to the MIS and RLF Cell. 

The branch offices send reports to the area office through prescribed forms on a monthly basis. On receiving the reports of all the branches from within the area, the area office asseses and consolidates the reports and sends them to the MIS and RLF Cell in the central office using prescribed forms. Copies of the reports are also sent to the respective divisional office.

The MIS and RLF Cell led by the Deputy General Manager prepares the overall credit plan of the organization based on branch projections and continuously develops and simplifies forms and registers for preserving loan accounts properly. The cell enters all the information received from the field into the computer and produces monthly reports and analysis on a regular basis. The MIS and RLF Cell produces the following reports:

·        ASA - At a Glance

·        ASA Savings at a Glance

·        ASA Current loans at a Glance

·        Credit Disbursed, Realized, Outstanding and Total Fund Information

·        Details Information of Small Credit & Savings

·        Details Information of Small Business Credit & Savings

·        District wise Overdue Loan Information

·        District wise and Category wise Savings Information

·        District wise and Category wise Credit Information

In addition, the area offices are required to send a report to the divisional office focusing on the position and progress of activities of all the branches in the area. The divisional office consolidates the reports from all the areas and prepares an analytical report on the overall situation of activities along with progresses and failures, and sends it to the Coordination and Implementation Cell in the central office. The central 'Coordination and Implementation Cell' assesses all the reports from the divisional offices and reports to the Managing Director with necessary recommendations.

ANNUAL IMPACT EVALUATION: A REVIEW

In addition to the monitoring checklists and questionnaire for impact assessment studies, ASA has developed a separate questionnaire for annual impact assessment. It is worth noting that since ASA has been implementing its microcredit and microsavings programs for a long time, its programs have had a deep-rooted influence in the lives of the target people. In the impact assessment process, a sample size is determined through use of a convenient sampling method. Data collection is made by credit officers in the field. Branch managers and area managers are responsible for supervision and follow-up. The Research and Evaluation Cell of the institution arranges an orientation on data collection techniques for the credit officers. Data is collected from the group members through focus group discussion and observation. Necessary notes are also taken from the information supplied by the respondents at the time of discussion.

The last impact evaluation of ASA activities to determine how the poor are benefiting from their self-employment efforts, was carried out at the end of 1999 according to the discussed procedure. The questionnaires were sent to all of the 92 program areas of the institution, of which 90 responded. Of the total 800 branch offices in those areas, 270 were selected as a sample, covering about 34% of the total branches. A total of 1,080 groups were chosen by randomly selecting 4 groups from each branch. Each group had an average of 20 members. The total number of group members was 26,400, of which 23,288 were interviewed during the data collection process. Interviews of the borrowers were conducted in the groups; that is, data was collected through focus group discussion (FGD). The groups had been in the program for a minimum of 3 years. During the FGD, questionnaires were filled out and necessary notes were taken down. The following table reflects the information received from the respondents regarding their positions in the mentioned sectors:

In response to the raised questions on the issues furnished in the table, the information given by the respondents is as follows:

1. Capital increase:
   A question was posed to the group members regarding whether their business capital had increased. In reply to the question, 90.42% of the group members answered affirmatively and the remaining 9.58% answered negatively. Those who answered affirmatively said that their schemes were running well. The loan from ASA and savings after consumption from earned profits had increased their capital. With this increase in capital more investment became possible, which ensured more investment in the IGA. On the other hand, those who responded negatively indicated that they faced loss due to unexpected accidents and natural calamities and consumption during the lean seasons.

2. Income increase:
   A question was raised regarding whether the borrowers' income had increased. In reply to this question, 90.88% of the respondents answered affirmatively and 9.12% answered negatively. Those who answered affirmatively stated that they were able to invest the loans in profitable schemes and devoted considerable time to the management of the schemes. Those borrowers who responded negatively mentioned that due to lack of experience they did not select profitable schemes and spent the loan on unproductive ventures such as house repairs and family consumption. Some stated that the loan amount had been taken away by their male counterpart.

3. Quality food intake increase:
   A question was raised to the group members regarding whether the quality of their food intake increased. About 89.12% of the respondents replied affirmatively and 10.88% replied negatively. Those respondents who replied affirmatively indicated that with an increase in family income the living standard of the family members improved. In addition, the quality of their food intake had improved. Some mentioned that although the quantity and frequency of food intake remained the same, the quality of their food had improved. Those respondents who replied negatively mentioned that the quality of their food intake did not improve as their family income did not increase or increased only slightly.

4. Literacy rate increase:
   In reply to the question regarding whether literacy rates increased, 86.89% of the total respondents answered affirmatively and 13.11% responded negatively. Those who responded affirmatively indicated that literacy rates in their families rose with increased income. They also indicated that their children were attending school regularly. Those who responded negatively indicated that although an awareness of the need for literacy developed through association with the group, the rate of literacy did not increase due to insufficient family income.

5. Availability of better medical services:
   A question was raised to the group members regarding whether they had better access to medical services. In response to the question, 88.41% of the group members responded affirmatively and 11.59% responded negatively. Those who responded positively indicated that they became more aware about the health of family members. The increase in income made it possible for family members to take advantage of better medical services. 

6. Family assets increase:
   A question was raised to the group members regarding whether their family assets - land, livestock and ornaments such as jewelry made of gold or - had increased. In reply to this question 38.93% said their land had increased, 59.66% said their livestock had increased, and 30.32% said their ownership of ornaments had increased. 61.07% said their land had decreased, 40.34% said their number of livestock had decreased and 69.68% indicated that their ownership of ornaments had decreased. 

7. Savings increase:
   95.47% of the group members answered affirmatively and 4.53% answered negatively. Those who answered affirmatively indicated that due to the motivation of the Credit Officers they became conscious about the necessity of accumulating savings to ensure future protection and also to meet emergency needs. Some said that they were encouraged by ASAs introduction of a voluntary savings program that allowed clients to withdraw savings at any time the funds were needed.

From the analysis of the above findings it may be concluded that the overall impact of the credit and savings programs of ASA is positive. It must be noted that the time allotted for the impact assessment was not sufficient. Because it was done quickly, no measuring scale was used for assessing to what extent the capital, income, quality of food intake, rate of literacy, availability of better medical treatment, family assets and savings of the group members increased. Hence, it could not be ascertained whether the increase was small, medium or great. But, according to the respondents, the majority of them could increase their capital, income, quality of food intake and savings to a great extent. On the other hand, they admitted a medium increase in the rate of literacy and availability of better medical services. The increase of family assets such as land, live stocks and ornaments was small.

DISSEMINATION OF IMPACT EVALUATION FINDINGS

The results of the impact evaluation are published in ASA's annual report. The findings are also published in the quarterly house journal 'New Vision' and are disseminated to the national newspapers.

Copies of the annual report and the house journal are sent to the branch offices of the organization, donor agencies, government and non-government development organization, media offices and other stakeholders.

A cost-effective, Reliable tool

The whole process of ASA's impact evaluation is simple and cost-effective. ASA develops a simple questionnaire as the tool for annual impact evaluation of the programs. There is no additional cost involved except that of printing the questionnaires. As ASA has an organizational set up at the field level, it is advantageous for the management to utilize the field workers for data collection. The credit officers are assigned to collect data from the group members. Because ASA implements its credit and savings programs through a group approach, every credit officer is required to attend all group meetings. After the end of each meeting the credit officer can easily collect data from the members through focus group discussions.

Before collecting data, the credit officers go through an orientation which takes place during their regular coordination meetings. Hence, no additional expenditures are necessary.

Since there is no need to deploy extra staff for data collection and processing, a considerable amount of money is saved each year. Since the questionnaire and the process are very simple, the institution does not have to spend money for consultants for its annual impact evaluation.

It is evident from the above discussions that ASA's impact assessment process is entirely cost-effective and the tool used for this purpose is very simple. 

The impact assessment mechanism may be helpful and cost-effective for other NGOs because the organizational structure of institutions everywhere is similar. There is also similarity in the process of program implementation. For example, NGOs involved in implementation of microcredit programs have a central office equipped with staff members headed by the chief of the organization. They also have field-level staff members who are associated with the program implementation process. The field workers are directly in touch with the group members. Almost all NGOs practice a group approach for implementation of their programs. Hence, the management of the NGOs can easily utilize their field-level staff members during impact assessment and avoid involving outside consultants. 

USE of Impact evaluation RESULTS in management decision making

The management system of ASA is innovative in nature. Highest priority is given to the findings and recommendations of the impact evaluation in decision making and policy formulation. Due to the innovative strategic approach of ASA's management, the institution undergoes rapid changes. ASA has attained its present position through need-based transformations. The transition from a social to a financial institution shows that the management is firmly committed to policy changes based on the findings of the impact evaluation. A few findings of the impact assessment during different phases of ASA's development will be cited here for better understanding of the importance of impact evaluation. 

The overall impact of the foundation phase of ASA's development was found to be positive, but there were some limitations due to development strategies the poor could not sustain. From analysis of the findings of impact assessment, ASA's management realized that sustainability could not be achieved without the economic emancipation of the poor. Decisions were made to bring about reforms with a view to implement an integrated development program including development education, credit, health, human development training and rehabilitation programs. 

After a few years, ASA again assessed the impact of the reform phase and identified that credit was a critical need for the poor, but the integrated development program did not efficiently prepare the members to be creditworthy. Due to this lengthy process, they could not get a loan and ASA could not meet their needs for credit to conduct income generation activities. This situation led to an increased rate of member dropouts.

In response to this assessment, ASA concentrationed on microcredit and microsavings programs to create self-employment of the group members through income generating activities.

Thus, ASA emphasizes the findings of impact evaluation, uses the results of the assessment in management decision-making, and brings about major changes in the policies and programs of the organization.

Some major policy decisions taken by the ASA management in the past are:

1. Provision for low-cost housing loans for group members.
When devastating floods, cyclones, tornados, etc. occur in different areas of the country, low-cost housing programs were adopted for post-disaster rehabilitation of the affected group members.

2. Feeding program for malnourished children.
A lack of awareness about nutritious food was identified through impact assessment. As a consequence, steps were taken to implement a nutrition improvement program. In conjunction with awareness development of the group members, a small-scale feeding program was introduced for children under-five.

3. Credit for tube-wells.
When a lack of awareness regarding waterborne diseases was identified among the group members, loans for tube-wells were sanctioned to create safe drinking water.

4. Credit for sanitary latrines.
Through impact assessment it was found that the use of sanitary latrines by the group members was rare, thus a decision was taken to provide them with loans for purchasing and installing sanitary latrines.

5. Traditional birth attendant (TBA) training program.
It was found that the mortality rates of mothers and infants were increasing in the project areas due to the absence of a scientific delivery system. Hence, decisions were made to train midwives to help assure safe delivery and thereby reduce the mortality rate.

6. Introduction of 'voluntary savings program'.
Based on the findings that while the group members were eager to accumulate more savings, there was no scope created by the organization to accept more than the mandatory savings, decisions were taken to introduce a voluntary savings program.

Recently, based on the findings of impact assessments, ASA management made a number of decisions for better performance and achievements. The decisions are:

• Provision for savings withdrawal
  ASA management provided group members with the opportunity to withdraw their savings at any time in spite of existing loans. ASA is the only NGO in Bangladesh to provide such a service to its members. As a result of this decision:
  a.  The group members can withdraw their savings deposit at any time and thus satisfy their needs accordingly in times of emergency.
  b.  The tendency of the group members to deposit more savings has increased.

• Providing Savings withdrawal facility in the group
  With the introduction of the provision the group members can withdraw savings at their homesteads instead of going to the office. As a result:
  a.  The group members do not have to bear the expense of transport to and from the office.
  b.  Time is saved, as the group members do not have to go to the office for withdrawing savings. They can utilize the saved time in their income generating activities.
  c.  The group members feel encouraged to deposit a larger amount as savings.

• Increase of the rate of interest on group member's savings
  As a result of the increase in the rate of interest on savings:
  a.  The amount of savings deposit of the group members increased considerably.
  b.  The group members became able to increase their capital.
  c.  The organization can extend services to more distressed people with the increased amount of group member's savings.

• Increase of loan ceiling
  As a result of the increased loan ceiling:
  a.  Investment of the group members in the income generating activities increased.
  b.  Income and capital for more investment increased.
  c.  Investment and income of the organization increased.

• Restructuring the field level program area
  The program area at the field level has been restructured and made smaller for effective and efficient management. As a result of making the areas smaller:
  a.  Close monitoring of activities by the area managers was ensured.
  b.  Quick identification and solution of problems became possible.
  c.  Accuracy in checking accounts and proper preservation of accounts was ensured.
  d.  Internal auditors are withdrawn as the area managers themselves can perform the audit-related functions efficiently. The decision of withdrawing internal auditors helped the organization to save money, thus increasing the cost-effectiveness of ASA’s operations.

• Introducing individual approach in the credit program instead of a group approach
  As a result of introducing the individual approach:
  a.  An individual member gets credit and savings services without attending the group meeting. The borrower saves both time and money.
  b.  Visiting the scheme by the credit officer is ensured. Utilization of loan amount and progress of the scheme is ensured due to regular follow-up by the credit officer.
  c.  Risk of failure of the scheme is reduced.
  d.  Isolated people from sparsely populated areas can gain access to the credit program.

CONCLUSION

Impact evaluation plays an important role in assessing the successes and failures attained through implementation of programs. The changes in the livelihood of the people are revealed from the findings. But there is no hard and fast rule that every organization should assess the impact of their programs following the same methodology. Different organizations are bound to follow different procedures. Being an innovative and fast growing institution, ASA uses various techniques in assessing the impact of its programs. A low cost mechanism is always emphasized by the institution. Based on its experiences, ASA made the process of its annual impact assessment cost-effective and time saving. During group formation ASA conducts a baseline survey and selects the group members according to pre-determined criteria. Authenticity of the assessment is vividly reflected from the findings when compared with that of the baseline survey. ASA achieved tremendous success through making appropriate decisions based on the findings of impact evaluation. The whole impact evaluation mechanism plays a positive role in providing reliable information for decision making and policy formulation.