WTP54 WORLD BANK TECHNICAL PAPER NUMBER 54 A Manager's Guide to Monitoring and Evaluating Urban Development Programs: A Handbook for Program Managers and Researchers Michael Bamberger and Eleanor Hewitt SECTORAL LIBRARY INTERNATIO'IAL BA'IK FOR RECO'IISTRllCTIO~ '-'110 D£\'El.OPMENT FEB 11 1988 *** HT 169 . 5 .B24 nn~ *** HT169.5 .B241987 c.3 1987 c.3 A"'"'"' ' ""'" '"iili~Hilfti~jj' '~··· SLC036353 -~ · WORLD BANK TECHNICAL PAPERS No. 1. Increasing Agricultural Productivity No. 2. A Model for the Development of a Self-Help Water Supply Program No. 3. Ventilated Improved Pit Latrines: Recent Developments in Zimbabwe No. 4. The African Trypanosomiases: Methods and Concepts of Control and Eradication in Relation to Development (No. 5.) Structural Changes in World Industry: A Quantitative Analysis of Recent Developments No. 6. Laboratory Evaluation of Hand-Operated Water Pumps for Use in Developing Countries No. 7. Notes on the Design and Operation of Waste Stabilization Ponds in Warm Climates of Developing Countries No. 8. Institution Building for Traffic Management (No. 9.) Meeting the Needs of the Poor for Water Supplv and Waste Disposal No. 10. Appraising Poultry Enterprises for Profitability: A Manual for Investors No. 11. Opportunities for Biological Control of Agricultural Pests in Developing Countries No. 12. Water Supply and Sanitation Project Preparation Handbook: Guidelines No. 13. Water Supply and Sanitation Project Preparation Handbook: Case Studies No. 14. Water Supply and Sanitation Project Preparation Handbook: Case Study (No. lS.)Sheep and Goats in Developing Countries: Their Present and Potential Role (No. 16.)Managing Elephant Depredation in Agricultural and Forestry Projects (No. 17.)Energy Efficiency and Fuel Substitution in the Cement Industry with Emphasis on Developing Countries No. 18. Urban Sanitation Planning Manual Based on the Jakarta Case Study No. 19. Laboratory Testing of Handpumps for Developing Countries: Final Technical Report No. 20. Water Quality in Hydroelectric Projects: Considerations for Planning in Tropical Forest Regions No. 21. Industrial Restructuring: Issues and Experiences in Selected Developed Economies No. 22. Energy Efficiency in the Steel Industry with Emphasis on Developing Countries No. 23. The Twinning of Institutions: Its Use as a Technical Assistance Delivery System No. 24. World Sulphur Survey No. 25. Industrialization in Sub-Saharan Africa: Strategies and Performance (also in French, 25F) No. 26. Small Enterprise Development: Economic Issues from African Experience(also in French, 26F) No. 27. Farming Systems in Africa: The Great Lakes Highlands of Zaire, Rwanda, and Burundi (also in French, 27F) No. 28. Technical Assistance and Aid Agency Staff: Alternative Techniques for Greater Effectiveness No. 29. Handpumps Testing and Development: Progress Report on Field and Laboratory Testing No. 30. Recycling from Municipal Refuse: A State-of-the-Art Review and Annotated Bibliography No. 31. Remanufacturing: The Experience of the United States and Implications for Developing Countries No. 32. World Refinery Industry: Need for Restructuring ( ) Indicates number assigned after publication. (List continues on the inside back cover.) WORLD BANK TECHNICAL PAPER NUMBER 54 A Manager's Guide to Mo~itoring and Evaluating Urban Development Programs: A Handbook for Program Managers and Researchers Michael Bamberger and Eleanor Hewitt The World Bank Washington, D.C., U.S.A. The International Bank for Reconstruction and Developf!1ent/THE WORLD BANK 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. All rights reserved Manufactured in the United States of America First printing February 1987 Technical Papers are not formal publications of the World Bank, and are circulated to encourage discussion and comment and to communicate the results of the Bank's work quickly to the development community; citation and the use of these papers should take account of their provisional character. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. Any maps that accompany the text have been prepared solely for the convenience of readers; the designations and presentation of material in them do not imply the expression of any opinion whatsoever on the part of the World Bank, its affiliates, or its Board or member countries concerning the legal status of any country, territory, city, or area or of the , authorities thereof or concerning the delimitation of its boundaries or its national affiliation. Because of the informality and to present the results of research with the least possible delay, the typescript has not been prepared in accordance with the procedures appropriate to formal printed texts, and the World Bank accepts no responsibility for errors. The publication is supplied at a token charge to defray part of the cost of manufacture and distribution. The most recent World Bank publications are described in the catalog New Publications, a new edition of which is issued in the spring and fall of each year. The complete backlist of publications is shown in the annual Index of Publications, which contains an alphabetical title list and indexes of subjects, authors, and countries and regions; it is of value principally to libraries and institutional purchasers. The continuing research program is described in The World Bank Research Program: Abstracts of Current Studies, which is issued annually. The latest edition of each of these is available free of charge from the Publications Sales Unit, Department F, The World Bank, 1818 H Street, N.W, Washington, D.C. 20433, U.S.A., or from Publications, The World Bank, 66, avenue d'Iena, 75116 Paris, France. Michael Bamberger is a regional coordinator for the Economic Development Institute and Eleanor Hewitt a research assistant in the Water Supply and Urban Development Department, both at the World Bank. Library of Congress Cataloging-in-Publication Data Bamberger, Michael. A manager's guide to MOnitoring and evaluating urban development programs. (World Bank technical paper, ISSN 0253-7494 ; no. 54) 1. City planning--Developing countries--Evaluation-~ Handbooks, manuals, etc. 2. City planning--Evaluation-- Handbooks, manuals, etc. I. Hewitt, Eleanor, 1950- II. Bamberger, Michael. MOnitoring and evaluating urban development programs. III. Title. IV. Series. HT169.5.B35 1987 351.8'18'091724 87-2188 ISBN 0-8213-0883-1 - iii - ABSTRACT This Manager's Guide provides an overview of Monitoring and Evaluatin Urban Develo ent Pro rams. A Handbook for Mana ers and Researchers. The Handboo is comprehensive, but can be easily understood by-urban policy-makers, managers and evaluation practitioners in developing countries. It provides guidance on all stages of the design and implementation of a monitoring and evaluation system and presents the main options with respect to scope, key research issues and organization. Monitoring and evaluation systems that can be applied to both individual projects and to integrated multi-component urban development programs are described. This guide highlights the aspects of most interest to managers and development planners. lt is cross-referenced to facilitate use of the Handbook. Urban development projects vary widely in scope and complexity and in the resources that are available for monitoring and evaluation. The Handbook is designed to help managers and policy-makers decide on the types and complexity of the studies that are most appropriate for their project and to select among the range of available research and analytical procedures. A distinction is made throughout between basic monitoring and evaluation techniques, which are simple and economical to apply in any project, and more complex techniques, which are only appropriate in certain circumstances. All of the methods described in the Handbook have been field tested, most but not all of them as part of World Bank projects. The unique contribution of the Handbook is to show how approaches taken from the fields of sociology, economics, anthropology and accountancy can be combined in an integrated monitoring and evaluation strategy. 1. Bamberger, Michael, and Eleanor Hewitt. 1986. Monitoring and Evaluating Urban Development Programs: A Handbook for Program Managers and Researchers. World Bank Technical Paper 53. Washington, D.C. v TABLE OF CONTENTS ABS TRAer. • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 11 i 1. THE IMPORTANCE OF MONITORING AND EVALUATION AS MANAGEMENT TOOI.S • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 1 2. THE EXPERIENCE ON WHIQI THE PROPOSED SYSTEMS ARE BASED ••••••••• 3 3. A MODEL OF THE MAIN STAGES IN THE DESIGN AND IMPLEMENTATION OF AN URBAN DEVELOPMENT PROJECT •••••••••••••••••••••••••••••••• 4 4. THE FUNCTIONS OF MONITORING •••••••••••••••••••••••••••••••••••• 6 4.1 Performance Monitoring •••••••••••••••••••••••••••••••••••• 6 4.2 Process Monitoring ••.•...•..••..•..••.•••.•••.•••..••..•.• 7 5. THE FUNCTIONS OF EVALUATION •••••••••••••••••••••••••••••••••••• 9 5.1 Impact Evaluation •••••.••••••.•••.••.•••••••..•..•...•.... 9 5.2 Cost-Effectiveness and Cost-Benefit Analysis ••••••••••••• 11 6. INTEGRATING MONITORING AND EVALUATION INTO A MANAGEMENT INFO'RMATION SYSTEM •••• ~ •••••••••••••••••••••••••••••••••••••••• 13 7. PLANNING THE MONITORING AND EVALUATION SYSTEM ••••••••••••••••• 14 7.1 Defining the Issues to Be Studied •••••••••••••••••••••••• 14 7.2 Organizing the Monitoring and Evaluation ••••••••••••••••• 14 7.3 The Scope and Intensity of the Studies ••••••••••••••••••• 14 7.4 Ensuring That the Evaluation Is "User Oriented" and Not "Technique Oriented"•••••••••••••••••••••••••••••••••16 7.5 Defining Resource Requirements ••••••••••••••••••••••••••• 16 8. MANAGING THE MONITORING AND EVALUATION •••••••••••••••••••••••• 17 8.1 The Role of the Project Manager in Monitoring and Evalu tion . ....•........................................ . 17 8.2 Defining the Main,Users of Monitoring and Evaluation ••••• 17 8.3 The Outputs of Monitoring and Evaluation and Their Practical Utility •••••••••••••••••••••••••••••••••••••••• 18 8.4 The Importance of Regular Reviews of Monitoring and Evaluation Outputa ••••••••••••••••••••••••••••••••••••••• 19 9• SUMMARY OF THE HANDBOOK ••••••••••••••••••••••••••••••••••••••• 20 1. THE IMPORTANCE OF MONITORING AND EVALUATION AS MANAGEMENT TOOLS (Introduction and Chapter 1)~ Monitoring and evaluation are practical tools which should form an essential part of good management practice. Monitoring is an internal project activity which assesses (a) whether project resources (money, materials, staff, etc.) are being delivered and used in accordance with the approved budget and timetable, (b) whether the intended outputs (numbers of houses constructed, training courses given, patients treated, etc.) are being produced in a timely and cost-effective manner, and (c) the efficiency with which the project is being implemented. Projects which do not have an effective monitoring system are more likely to suffer some of the following types of problems: 0 delays and cost overruns 0 exclusion or under-representation of certain sectors of the target population 0 problems of quality control 0 long delays in detecting problems or conflicts among the implementing agencies or between these agencies and intended beneficiaries The primary purposes of evaluation are to assess the extent to which the intended impacts (increases in income, reduced incidence of certain infections, improved housing quality, etc.) have been produced and to compare the cost-effectiveness of a project with possible alternatives. When a project does not have an effective evaluation system the danger is increased of the following kinds of problems: 0 continuing a project which is not producing the intended benefits 0 wasting money by not selecting the most cost-effective option 0 greater difficulty in detecting and correcting some of the factors which are reducing project impact. A well designed and implemented monitoring and evaluation system can be a cost-effective way to: 0 Provide constant feedback on the extent to which the projects are achieving their goals 0 Identify potential problems at an early stage and propose possible solutions. 0 Monitor the accessibility of the project to all sectors of the target population. o Monitor the efficiency with which the different components of the project are being implemented and suggest improvements. 1/ All references are to the appropriate sections of Monitoring and Evaluating Urban Development Projects: A Handbook for Managers and Rsearchers (Technical Paper 53). - 2 - o Evaluate the extent to which the project is able to achieve its general objectives. o Provide guidelines for the planning of future projects. Many project managers have been discouraged by the belief that monitoring and evaluation are highly technical fields which should be left to research specialists or which are too expensive and complex to be of practical utility. However, monitoring and evaluation need not be complicated or expensive, and the size and complexity of the studies can be adapted to suit the needs and resources of each project. For example, the monitoring studies described in Section 4 are all simple, short and economical. The role of the manager is to define the topics which need to be studied, to make sure that researchers use the most cost-effective methods, and to arrange for reports to be reviewed, discussed and acted upon. - 3 - 2. THE EXPERIENCE ON WHICH THE PROPOSED SYSTEMS ARE BASED (Introduction) The methods described in this Handbook are based on ten years of World Bank experience and a review of the extensive evaluation literature produced since the early 1970s. The World Bank urban evaluation experience began in 1975 with a cooperative venture with the International Development Research Centre (Ottawa) which supported a five-year evaluation of four of the first World Bank financed urban shelter projects in El Salvador, Zambia, Senegal and the Philippines. Since then the Bank has provided assistance to governments in Asia, Africa and Latin America in the design of their own monitoring and evaluation systems; and new evaluation techniques designed to provide more rapid feedback or a better understanding of the point of view of intended beneficiaries, have been tested. All of the methods described in the Handbook have already been field tested, most but not all of them as part of World Bank projects. - 4 - 3. A MODEL OF THE MAIN STAGES IN THE DESIGN AND IMPLEMENTATION OF AN URBAN HOUSING PROOJECT (Chapter 1 - Section B and Annex A) In order to be able to design and interpret a program of monitoring and evaluation, it is essential to have a model of the process of implementation of the project being studied. This helps managers and evaluators define the key issues to be studied, and provides the framework for interpreting the results of the individual studies. Figure 1 describes a model which can be used to identify and describe the main stages in any urban development project. The figure shows how the model can be used to describe the main stages in a project to renovate urban housing in a large city. The project can be broken down into six sequential stages: Stage 1: Planning and design of the project. During the planning stage, assumptions are made about the needs of certain sectors of the population and about the relative effectiveness of alternative delivery systems. Stage 2: The resources allocated to the project. The resources include money, land, professional staff, machinery and equipment. Stage 3: The delivery systems through which resources are converted into outputs. Stage 4: Defining the outputs which the project is intended to produce. Stage 5: Defining the impacts which it is hoped will occur as a result of the project. (See Chapter 4-Section 5 and Annex A). Stage 6: Using the results of the first project to help in the selection and design of future projects. Many projects form part of an ongoing urban development program and frequently the results of the first project are intended to provide guidelines for the definition and design of future projects. The model also refers to the economic and political context within which the project develops and to the characterisitcs of the target population. Projects do not develop in a vacuum, and success is significantly affected by the characteristics of the target population, and by the political and economic context within which the project is implemented. Fig 1-1 FRAMEWORK FOR MONITORING AND EVALUATION OF THE PROJECT IMPLEMENTATION PROCESS TECHNICAL, FINANCIAL, PROJECT COMPLETION E>< POST PROJECT MONITOR~G AND ECONOMIC ANAL VS IS REPORT EVALUATION . LOAN APPRO\/ AL ~ 7 ~ 7 7 ' t7 PROJECT IMPLEMENT AT ION ' ' ' PLANNING AND DESII::.'i INPUTS OUTPUTS IMPACTS PREP AR AT ION METHODS OF NE'w' PROJECTS ~~~ VI Pl~nn1n9 ~nd ProcurPmtnt ~nd SPlf-~lp hous• lfnp lpm•nt•tion ()p•r~t;on lncrttasPd incom• Cost-.ff•ctiv•nus d•s~n ~ ~m1nis tr ahon ~ construction vs. ~ ~ ~ ~nal~sis, r•d•fmf' of moM~, un of contr~tors; Hous•s bum. M~1ntpnancP. lmprovtod hulth tM'Qt't population •"1 D•f1nihon of mattorials, diff•r•nt mt'thods dt hnr~ s~ sttm compontnts and •quipm•nt, ~nd of ~dministtr;nQ Pahtnts truttd DtvtlopinQ lncrttast'd tarQft population staff. buildin9 mat•dal communit~ ttmp lot~ m•nt crttdit, ttc. Industrial plots 9roups. sold. Cost rpcovpr~ . L. ~ i -~ i i :f I CONTE>