Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) Report Number : ICRR0020302 1. Project Data Project ID Project Name P098132 ET-Tourism Develop. Project SIL Country Practice Area(Lead) Ethiopia Trade & Competitiveness L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46240 30-Dec-2014 39,500,000.00 Bank Approval Date Closing Date (Actual) 30-Jun-2009 30-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 35,000,000.00 0.00 Revised Commitment 34,898,354.17 0.00 Actual 34,790,501.46 0.00 Sector(s) Other industry(45%):Other domestic and international trade(41%):Public administration- Industry and trade(9%):General education sector(5%) Theme(s) Rural non-farm income generation(44%):Other Private Sector Development(31%):Cultural Heritage(15%):Education for the knowledge economy(6%):Infrastructure services for private sector development(4%) Prepared by Reviewed by ICR Review Coordinator Group Antonio M. Ollero George T. K. Pitman Christopher David Nelson IEGFP (Unit 3) 2. Project Objectives and Components a. Objectives According to the Financing Agreement (page 5), the project development objective of the Ethiopia Sustainable Tourism Development Project of the Federal Democratic Republic of Ethiopia is: "to enhance the quality and variety of tourism products and services in targeted destinations so as to increase the volume of tourism, foreign exchange earnings, and jobs". Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components The project comprised four components: Destination Development (US$18.3 million estimated, US$19.47 million actual) consisting of: (a) the rehabilitation and enhancement of basic infrastructure in heritage sites in Addis Ababa, Axum, and Lalibela (US$8.7 million estimate, US$17.4 actual); (b) the enhancement of visitor services in selected destinations focusing on the upgrading of tourist/visitor facilities in Addis Ababa, Axum and Lalibela (US$7.7 million estimate, US$0.96 million actual); and, (c) the development of tourism products to diversify offerings in targeted destinations (US$1.9 million estimate, US$1.11 million actual). Market Development (US$7.85 million estimated, US$3.66 million actual) consisting of: (a) branding, positioning and marketing activities to enable Ethiopia to develop and implement strategies to attract specific target segments from key markets through the use of marketing collateral, communication, and promotion (US$2.7 million estimate, US$0.79 million actual); and, (b) a demand-driven linkages program to enhance the competitiveness of the Ethiopian tourism private sector through a US$3.0 million grant scheme and to foster community involvement in the tourism sector to optimize benefits for the poor through a US$2.15 million community involvement activity (US$5.15 million estimate, US$2.87 million actual). Institutional Development and Capacity Building (US$5.55 million estimated, US$3.1 million actual) consisting of: (a) institutional development, including: the development of policy implementation frameworks, the development of a sector strategic plan, the strengthening of the Ministry of Culture and Tourism (MOCT) and the regional tourism bureaus; the collection of tourism data and the strengthening of tourism statistics; institutional development and support to the Tourism Council; and the establishment of a Tourism Board to foster public-private partnerships in tourism promotion (US$3.95 million estimate, US$2.57 million actual); and, (b) capacity building, including the assessment of tourism education and training strategic needs, the improvement of tourism institutions’ programs, the enhancement of staff skills at the MOCT, and the upgrading of the competencies of tourist guides and the hospitality industry workforce to professional levels (US$1.6 million estimate, US$0.53 million actual). Implementation Support (US$2.85 million estimate, US$3.9 million actual) covering the setup of a Project Coordination Unit (PCU) and Site Management Units (SMUs). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Total project cost was estimated at US$35.0 million at appraisal, and actual cost was US$34.8 million. Many sub- component project costs varied widely between the estimates at appraisal and the actual at closing. Notably, the cost for the rehabilitation and enhancement of basic infrastructure under the Destination Development component more than doubled: although the number of rehabilitation sites was reduced from ten to six, rehabilitation costs were higher than initially estimated; the displacement of residents in Lalibella required the development of a relocation neighborhood in Kurakur; and 21 kilometers of roads were rehabilitated rather than the original target 12 kilometers. The cost for the visitor service centers was reduced by 85 percent as the visitor services enhancement activity was dropped, and so were signage in selected destinations. The cost for the Market Development component was more than halved because the matching grant scheme and the community involvement activity were scaled-down. The cost for the Implementation Support component rose by US$1.0 million because the project took a year longer to implement. Financing: The project was financed by the IDA in the amount of Special Drawing Rights (XDR) 23.6 million (US$35.0 million) at project approval. The actual disbursement was XDR 23.5 million (US$34.7 million). The undisbursed balance was cancelled at project closing. Borrower Contribution: The Federal Democratic Republic of Ethiopia, which was to provide US$1.5 in counterpart funds at appraisal, actually contributed US$2.2 million. Dates: The project was approved in June 30, 2009 and became effective in February 25, 2010. Disbursements for the Destination Development component were suspended from November 16, 2010 to June 29, 2012 due to the involuntary resettlement of displaced Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) households in Lalibela and the resulting non-compliance by the Government with the terms of the Financial Agreement. The project had a Level-II restructuring twice. The first restructuring in March 15, 2013 adjusted the targets to the project outputs and outcomes. The second in June 27, 2014 re-allocated project resources to under-budgeted activities and extended the original closing date by a year to December 30, 2015, at which time the project closed. 3. Relevance of Objectives & Design a. Relevance of Objectives The project remains highly relevant to current economic conditions and development priorities in Ethiopia. In a country with an annual GDP per capita of only US$574 in 2014 (the 13th lowest globally among Bank member countries), the diversification of the economy outside of agriculture remains crucial to generate economic activity, lower unemployment, and reduce poverty. Considering the country’s wealth in natural and cultural tourism assets (the country has eight UNESCO World Heritage sites), the services sector, and in particular the tourism sector, offers untapped opportunities to create employment and to generate foreign exchange earnings. The project helps the Government address the challenges and constraints to a sustainable development of the tourism industry, including the poor preservation and presentation of tourist sites, the shortage of infrastructure and low-level of public investment, the lack of high-quality services, and the limited capacity of both public and private sector tourism institutions. The project is also well aligned with the strategic directions of the Bank’s Country Partnership Strategy (CPS) for Ethiopia for FY13-16, which continues to support the structural transformation of the economy articulated in the Government’s Growth and Transformation Plan I and II (GTP I and II) for FY2010-15 and FY 2016-21 respectively. The project is relevant to the second of the four dimensions of the economic transformation envisioned in the GTP I, namely, the increased productivity and competitiveness of the industrial and service sectors for rapid and sustained job creation. Specifically, the GTP I considers the tourism sector as a source of job creation for small and medium enterprises (SMEs) and for women and the youth (CPS, pages 22 and 26). The GTP II reinforces the relevance of the tourism sector to the country’s overall economic development, with targets to triple foreign tourist arrivals and to increase tourism revenues to over US$3 billion by 2016. The CPS considers the project to be a vital part of the Bank portfolio that is directed at the productivity and competitiveness objective of the GTP I and II and of the CPS. Rating High b. Relevance of Design The overall design of the project is broadly aligned to the project’s objectives. The results framework for the project is logical. The components and activities of the project --- destination development, market development, and capacity building --- comprise an integrated set of interventions that address the key constraints to the sustainable development of the tourism industry in the country. The components of the project are to produce a defined set of outputs in three target tourist destinations --- tourist sites and access roads that are rehabilitated; visitor service centers that are established and staffed; tourism product development plans that are prepared and marketing and promotion strategies that are adopted; beneficiary firms that utilize matching grant financing and new business ventures that are established; tourism strategies, grading regimes, and tourism statistical systems that are established; and participants that are trained in the tourism industry. The project outputs, which enhance the quality and variety of tourism products and services in the target destinations, are, in turn, to lead to three key outcomes --- an increase in tourist volume, an increase in foreign exchange earnings from tourism, and an increase in tourism-related jobs. While essentially logical, the causal link from component activities and outputs to outcomes cannot be considered totally unambiguous. Factors extraneous to the project could reasonably influence the project outcomes. This ambiguity is typical of tourism projects where it remains challenging to attribute the economic effects directly and exclusively to the tourism development activities undertaken since the outcomes --- the increase in the number of international visitors, in particular --- can be influenced by external factors. The results framework of the project was revised during the first restructuring to refine the attribution of the project outcomes to the project activities. Among others, the tourism jobs indicator was confined to the four tourism industry sectors assisted directly by the project --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) accommodations, catering, tourist transportation and tourist guides. Rating Modest 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective To enhance the quality and variety of tourism products and services in targeted destinations so as to increase the volume of tourism, foreign exchange earnings, and jobs. Rationale Outputs The project completed the following outputs in the Destination Development component: • Number of historic sites and monuments rehabilitated in Addis Ababa, Axum and Lalibela: six tourist sites were rehabilitated, according to target - -- the Addis Ababa museum, the Stelae field in Axum, the Archaeological Museum of Axum, the Tsion Church in Axum, the museum in Melka Qunture, and the archaeological museum and ethnographic museum in Tiya. • Kilometers of access and town roads rehabilitated: a total 21.1 kilometers of access and town roads were rehabilitated, exceeding the revised target 4 kilometers. • Number of visitor centers established and staffed in Addis Ababa and its environs, Axum and Lalibella: three visitors centers, one each in the three target destinations, was established and fully staffed, in line with the target. • Number of tourism product developed plans prepared in secondary destinations: three tourism product development plans were prepared, according to target --- one each for the southern, western and eastern tourism routes. The plans were prepared following extensive consultation with stakeholders. Their completion now enables the tourism industry to expand beyond the original three destinations in the northern tourism route covered by the project. The project produced the following outputs in Market Development component: • Number of businesses in the tourism value chain that successfully implement service and product upgrading plans through the 50-50 percent cost-sharing matching grant scheme: a total 79 firms upgraded their service and product plans using the matching grant financing, well in excess of the target 50 firms. The matching grant scheme was well utilized across the tourism value chain in all three destinations, Addis Ababa, Axum and Lalibela. • Number of community-based tourism-related business ventures established: the community involvement activity established 54 businesses --- eight in Addis Ababa and its environs, 26 in Axum, and 20 in Lalibela --- in excess of the target 45 businesses. The activity has enabled communities in the three destinations to benefit directly from tourism activity. The project produced the following outputs in the Institutional Development component: • A national tourism strategy developed, with a detailed 3-5 year action plan: the Government has developed a national tourism strategy to guide Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) the development of the tourism industry in the country. The strategy was prepared in consultation with stakeholders. • A lodging grading regime developed and pilot tested: the industry has developed a lodging grading regime compatible with international standards. Fifty two grading experts have been trained and 337 hotels in seven regions of the country have been graded. • An upgraded national tourism statistics system, capable of capturing basic tourism indicators: the Government has designed and established a national tourism statistics system to better track and monitor the performance of the tourism sectors across all geographic areas in the country. The system has been made operational with the software migrated to the Government’s information technology system. • Number of instructors trained on curricula developed at vocational schools: a total 44 instructors, exceeding the target 40 instructors, have been trained overseas. The trainees were required to share the knowledge gained from the overseas program with colleagues in their training institutions. • Number of people trained in the tourism industry in Addis Ababa, Axum and Lalibela: a total 1,661 participants in the tourism industry were trained, well above the target 200 people. The training enabled an improved understanding of the tourism industry and of the standards necessary to offer a competitive product. Of those trained, 834 were women. Beneficiary counts were added as output indicators following the first restructuring of the project, for which the results were: • Direct project beneficiaries: some 3,837 people were counted as direct beneficiaries of the project, in excess of the target 500 people. In addition, some 26,738 individuals benefitted from an improved access to water and sanitation in Lalibela as a result of the project. Moreover, 8,316 were employed, mostly in construction, during the implementation of the project. • Women beneficiaries: 1,704 women were counted as direct beneficiaries of the project, in excess of the target of 100 women. The women benefitted from grants, training and in-kind support provided by the project. In addition, some women benefitted from temporary jobs, including in construction, during the implementation of the project. Outcomes The efficacy of the objective "to enhance the quality and variety of tourism products and services in targeted destinations so as to increase the volume of tourism, foreign exchange earnings, and jobs" is assessed as Substantial. Increased volume of tourism: the achievement of this outcome is rated as Substantial. • The average increase in the number of international visitors to Axum was 61% over the period 2012-15, exceeding the target 23%. That to Lalibela was 24%, falling short of the target 33%, but the arrivals nonetheless increased overall by over 20,000 (from 41,158 in 2011 to 62,420 in 2015). There were no targets set for Addis Ababa and its environs as data were not available. • It is noteworthy that the number of hotels increased significantly in both destinations during the project period: from nine small hotels in Axum in 2010 to 43 by 2015, and from eight hotels in Lalibela in 2010 to 55 by 2015. • The sharp increase in tourist arrivals and in hotel openings in 2014 coincided with the accelerated disbursement of project funds to 86 percent in that year. In arguing that the increase in tourist arrivals can be attributed to the project, the ICR (page 14) explains that the increase in the volume of tourism in the project sites could not have been achieved without the project, considering that the unavailability of tourism service and the lack of tourism infrastructure had deterred both tour operator-led group visits and independent travel before the project . Increased foreign exchange earnings: the achievement of this outcome is rated as Substantial. • The foreign currency expenditure per international tourist per day averaged US$362 in the three destinations (US$604 in Addis Ababa, US$252 in Axum, and US$230 in Lalibela), exceeding the target of US$350 (the baseline was US$310). • Considering that the number of international tourists increased, and the daily foreign currency spending by the average tourist increased, then the country’s foreign exchange earnings from tourism increased during the project period. • The ICR (page 15) similarly argues that the capital, skills training, and technical assistance provided by the project enabled the growth of tourism businesses, products and services in the target destinations, leading to an increase in expenditures by international tourists in the three project Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) sites. Increased tourism-related jobs: the achievement of this outcome is rated as High. • The number of new jobs created in four tourism-related industries (accommodations, catering, tourist transportation, and tour guides) over the project period 2010-2015 was 5,553 exceeding the target 1,440 (the baseline was 1,090) --- 2,324 in Axum, exceeding the target 580, and 3,229 in Lalibela, exceeding the target 860. • The ICR (page 15) attributes the job increases directly to the project. Firms participating in the matching grant scheme created 457 new permanent jobs following the implementation of their projects. Fifty four new business ventures established under the community involvement activity created 883 new permanent jobs. The implementation of five other donor-funded tourism development projects at the time of the Ethiopia Sustainable Tourism Development Project in 2010-2015 (Project Appraisal Document, pages 43-44) raises the question of whether these projects also contributed to the increase in tourist arrivals, foreign exchange earnings and tourism-related jobs that are attributed to the Ethiopia Sustainable Tourism Development Project. The ICR does not analyze this issue, but it is worth considering. • The USAID-funded project (US$7.0 million), implemented by the Global Sustainable Tourism Alliance in 2008-13, supported community-focused tourism activities in the southern and eastern tourism routes, while the Ethiopia Sustainable Tourism Development Project focused on the northern tourism route (Addis Ababa-Axum-Lalibella), so there is no overlap. • Similarly, there is no overlap with the Netherlands Development Organization-funded technical assistance project, implemented by the U.N. World Trade Organization, which supported community-based tourism activities in the southern region (in Konso and Chencha-Dorze), or with the German Technical Agency-funded project which supported market branding and hotel training activities in Hawassa University in the southern region (in Awassa). • This leaves the Spanish Government-financed cultural diversity project (US$5.0 million) in the Amhara region, including Lalibella, and the Norwegian-Government and World Monuments Fund-financed conservation action plan (US$1.8 million) in Lalibella. The IEG lacks sufficient information about these two projects to venture an attribution. But considering that the amount spent by these two project in Lalibella is a fraction of that which the Ethiopia Sustainable Tourism Development Project did, their share in the tourist arrival, earnings and job outcomes would not detract much from that which the Ethiopia Sustainable Tourism Development Project achieved. Rating Substantial 5. Efficiency The efficiency of the project is rated as Modest. Economic Efficiency The Economic Rate of Return (ERR) for the project was computed at 37% at closing, lower than the 43% at appraisal. The ERR was computed for parts of Component 1 (Destination Development), Component 2 (Market Development, specifically the matching grant scheme), and Component 3 (Institutional Development and Capacity Building) of the project. The lower ERR at closing was principally due to lower net benefits from, and a longer implementation period for, the project. Benefits from the project were lower because actual tourist arrivals during the project period were less than originally assumed at appraisal. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) To a certain extent, the lower tourist arrivals was offset by the slightly higher spending by the average tourist. Operational and Administrative Efficiency The project took a year longer than estimated to be completed, largely due to the suspension of disbursements for Component 1 (Destination Development). The civil works, which accounted for more than half the project cost, was completed in four and a half years, shorter than the five years originally estimated. Some civil works activities were dropped, however, which in part accounts for the shorter completion period. However, the cost of the civil works exceeded by US$3.0 million that estimated following the second restructuring. The operating cost for the project topped US$3.3 million, double that estimated at appraisal, although only slightly higher than that estimated following the second restructuring. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 56.60 Appraisal  43.00 Not Applicable 47.40 ICR Estimate  37.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The outcome for the Project is rated as Moderately Satisfactory. The objective --- to enhance the quality and variety of tourism products and services in targeted destinations so as to increase the volume of tourism, foreign exchange earnings, and jobs --- is highly relevant to the development priorities of Ethiopia, as articulated in the Government’s Growth and Transformation Plan I and II (covering 2010-15 and 2016-21 respectively), and in the Bank’s CPS for Ethiopia for 2013-2016. The relevance of objectives is rated High. The design of the project was improved with the revision of the results framework during the first restructuring. The attribution of outcomes to interventions was refined and the output and outcome targets were set to more realistic levels. The relevance of design is rated Modest. The efficacy of the Project is assessed as Substantial. The project exceeded almost all the targets for the tourist arrivals, tourism foreign exchange earnings (tourist foreign exchange expenditures), and tourism-related jobs set in the project restructuring of March 2013. Moreover, the project is judged to have contributed to enhancing the capacity of the industry to catalyze investment, promote entrepreneurship, drive business linkages, and create employment opportunities over the longer-term. The efficiency of the Project is assessed as Modest. The ERR of the project was lower at closing, 37%, than at appraisal, 43%. The project took a year longer to be completed and was far more expensive to manage than originally estimated. a. Outcome Rating Moderately Satisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) 7. Rationale for Risk to Development Outcome Rating The risk to the sustainability to the development outcome is assessed to be Modest. The outlook for growth of the tourism industry in Ethiopia over the next ten years, 2016-2026, is generally positive according to World Travel and Tourism Council in its publication, "Travel and Tourism Economic Impact, 2016". Arrivals are expected to increase by 35 percent over the period. Travel and tourism-related jobs are estimated to slow down in the short run, but gain over the medium term by 0.2 percent by 2016. Tourism investment is projected to rise by 3.7 percent in 2016 and by an additional 5.4 percent over the next ten years. Overall, the direct contribution of the travel and tourism sector to GDP is forecast to increase by five percent. There are downside risks to the sustainability of the development outcome of the project however. One, as in the case for tourism globally, potential economic weakness in the tourist-sending countries --- in Europe and in the United States --- can decrease the supply of tourists to, and dampen tourism spending in, Ethiopia. Two, adverse political events in Ethiopia or in its neighboring countries, can have negative consequences on security and safety, or on perceptions or security and safety, in the country. Three, changes in the priorities accorded by the Government to tourism can result in decreased public spending on the sector or in public institutional changes that affect the performance of the sector. And four, a lack of follow-through in the operation and maintenance of the civil works undertaken by the project can lead to a deterioration of the tourist sites. These risks are likely to be moderate. They are expected to be mitigated by interventions undertaken by the project itself. Management plans had been developed under the project for the administration of the tourist sites. The provision in the annual operating budgets of local administrations for the management of the tourist sites should help in their maintenance. The institution building initiatives undertaken by the project should help improve the capacity of the private and public sectors to address challenges to the industry. Overall, the continuing commitment of the Government to the expansion of the tourism industry, as articulated in GTP II for 2016-2026, should help moderate these downside risks. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The project benefitted from strong points at entry: The project was grounded on strong technical analysis competently conducted by the Bank at appraisal. The project sought to address the binding constraints to the sustainable development of the tourism industry in Ethiopia by structuring an integrated set of interventions in destination development, market development, and institution building in the industry. The Bank integrated lessons learned from previous interventions into the project. Specifically, the Bank drew on the experience of the Cultural Heritage Project (CHP, completed 2008) to better inform the design of this project. The CHP was a US$5 million learning and innovation loan that pilot-tested activities for fully integrating the conservation and management of cultural heritage into local and national economic development. The CHP aimed to contribute to poverty reduction by increasing income generated from cultural assets and from handicrafts development. The Bank gave adequate consideration to the poverty, social development and gender aspects of tourism development in the country. In Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) particular, the project adopted a strategy of linking tourism development with community development to enhance the project’s impact on poverty reduction. The Project however suffered from shortcomings at entry: The scale of the project --- the number and complexity of the project activities (the project sub-components) --- was ambitious relative to the Government’s capacity for project implementation and activity coordination. To the Bank’s credit though, the project had been down-sized at appraisal from an even more ambitious request by the Government. The Government had previously asked the Bank to support the development of not three but eleven tourist destinations. The Bank did not adequately identify the risks to project implementation. It failed to conduct a more realistic assessment of the capacity of the Government for efficient project implementation. Moreover, the ability of the project’s implementing agency to comply with Bank safeguards and fiduciary policies, practices and standards were not well vetted at appraisal. The original results framework and M&E design for the project was deficient in some aspects. The first outcome indicator --- the increase in the number of international visitors to the three target destinations --- could not be exclusively attributed to the interventions undertaken by the project. An increase in tourist arrivals into Ethiopia can be the result of many factors external to the project. The outcome indicator for foreign exchange earnings was not explicitly defined at appraisal. The numerical targets for some of the project output indicators (intermediate outcome indicators) were overly ambitious that they had to be downshifted to more realistic levels by mid-term. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The supervision of the project was generally adequate. The Bank completed 12 supervision missions and filed 14 supervision reports --- Implementation Status and Results Reports (ISRs) --- over the six-year life of the project. It spent US$1.6 million on the task and fielded a supervision staff with the appropriate mix of skills. The Bank supervision team reacted quickly to the involuntary resettlement of residents in Lalibela and suspended disbursements for the Destination Development component of the project. The team exerted substantive effort to ensure the completion of a Resettlement Action Plan (RAP) for Lailibela to remedy the problem and facilitated the lifting of the disbursement suspension order. The Bank used the Mid-Term Review (MTR) and two project restructurings to engineer a turnaround of the project. The community involvement activity and the matching grant scheme were expanded and implemented faster. The indicators for the project’s outputs and outcomes were reformulated, and the targets were reset to more realistic levels. The Bank provided hands-on support for procurement and for contract management. Experts were recruited to support the monitoring and evaluation of the project. There were shortcomings with the quality of some supervision tasks, however. The Bank neglected to more regularly follow up the implementation of the Environmental and Social Management Framework (ESMF), which was designed to address the environmental and social risks of the project. Had it done so, the Bank would have discovered, earlier than at closing, that site-specific safeguards instruments --- the environmental clearance letters and the environmental M&E reports --- were not prepared according to plan or approved by the competent environmental authorities in the four regional states. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) a. Government Performance The strength of the Government’s ownership of, and commitment to, the project varied over the life of the project. The commitment was evident at the time of project appraisal but wavered during project implementation. At the sub-national level, the degrees of commitment to the project varied by jurisdiction. Members of the Project Steering Committee from Axum and Lalibela were active, but those from Addis Ababa did not consistently participate in committee activities. The Government failed to comply with the Bank’s social safeguards policies. The involuntary resettlement of residents in Lalibela delayed the implementation of the Destination Development component of the project inordinately. Government processes for securing contracts were extremely lengthy. Requirements for the approval of major contracts at the state ministerial level caused delays in contract delivery during the early years of project implementation. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The Ministry of Culture and Tourism (MOCT) acted as implementing agency of the project. A Project Steering Committee chaired by the Minister of Culture and Tourism and comprising of representatives of the Government, the private sector and civil society, including religious institutions, provided oversight of the project. A Project Coordination Unit (PCU) within the MOTC coordinated the overall implementation of the project. And three Site Management Units (SMUs) functioned as local implementation offices of the project in Addis Ababa, Axum, and Lalilbela. The PCU and the SMUs comprised small teams with little expertise in tourism and limited capacity in grants management and in monitoring and evaluation during the early years of project implementation. Capacity at the PCU and the SMUs was enhanced with the recruitment of experts following the restructuring of the project in March 2013. To the PCU’s and the SMUs’ credit, almost all of the project’s planned components and activities (sub-components) were implemented and completed by the project closing date. Numerous setbacks and delays, notably the two-year suspension of disbursements for the Destination Development component following the involuntary resettlement of residents in Lalibela, shortened the time period to complete the project, notwithstanding the one-year extension of the project closing date. The PCU and the SMU at Lalibela were instrumental in resolving the involuntary resettlement problem in Lalibela. They helped conduct 30 rounds of consultations with residents affected by the resettlement, followed up on the development of the resettlement plan, and collaborated with the Bank to lift the disbursement suspension order. Nonetheless, there were shortcomings with the performance of the project implementing units. The PCU failed to recruit an Environmental Specialist. Consequently, the implementation of the ESMF was not closely supervised. The PCU and the SMUs also under-staffed other critical specialist roles, forcing the Bank had to hire the necessary experts itself and embed them in the project implementing units. The transition arrangements for the operation of some of the facilities developed under the project, notably the Tiya and the Menka Qunture, were not clearly specified. Overall, there were gaps in contract management, particularly in the supervision of contract works. An action plan was developed to address the shortcomings in contract management, but the problems persisted through the end of the project. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) a. M&E Design The indicators to track the project outputs and outcomes were specified at appraisal. Targets were set for each year of the project implementation period. The data sources for the indicators and the units responsible for data collections were identified. The targets set at appraisal would turn up to be ambitious during project implementation, leading the Government and the Bank to agree on more realistic goals during the first restructuring of the project after mid-term review. The numerical targets for the indicators were reset. Moreover, the targets were to be reckoned as end-of-project, rather than as annual, goals. b. M&E Implementation The process to contract a qualified firm to conduct the baseline survey for the M&E was lengthy and protracted. Hence, data to implement the M&E was not collected until after the baseline survey was submitted in October 2012 and the MTR was completed in November 2012. This did not adversely affect the implementation of the M&E as many of the project’s activities were implemented later in the project’s timeline. Nonetheless, this would not have been the ideal case. Data collection and M&E reporting generally improved after the project restructuring in March 2013. However, questions about the methods used in data collection persisted. The Bank engaged an M&E consultant in 2013 to help the PCU with appropriate data collection methods. The PCU, assisted by the Bank supervision team and the M&E consultant, carried out several studies to assess the impact of the project’s activities. The Project Development Objective Results Status Survey provided data to track the progress of the PDO indicators in the three target destinations over six years from 2010 to 2015. The Matching Grant and Community Involvement Evaluation assessed the impact of the community involvement activity on the performance of business ventures established under the project. And, the Terminal Evaluation measured the impact of the project. These studies provided data that were used in the supervision reports and the ICR c. M&E Utilization Data gathered from the M&E implementation were used to restructure the project and to improve project performance. The supervision reports had rated the Development Outcome (DO) as Moderately Unsatisfactory for four consecutive reporting periods by the time the MTR commenced in October 2012. Then, US$6.06 million of the US$35.0 million project funds had been disbursed. The rating for Implementation Performance (IP) had swung from Satisfactory to Moderately Unsatisfactory to Moderately Satisfactory in October 2012. The findings from the supervision reports and the mid-term review led to the first restructuring of the project in March 2013. The three outcome indicators of the project were reformulated to enhance the attribution of the outcomes to the project interventions. The numerical targets for many of the output indicators (the intermediate outcome indicators) were reset downwards to reflect less ambitious and more realistic goals. For instance, the number of historic sites to be rehabilitated was decreased from ten to six. The number of kilometers of access and town roads to be rehabilitated was cut from 12 kilometers to four kilometers. Data gathered from the baseline survey were especially useful for setting the more realistic targets. Only 20.9% of the project funds had been disbursed by the time the project was restructured for the second time in June 2014. The supervision report reported a Moderately Satisfactory rating for both Development Outcome and Implementation Performance at that time. The restructuring re-allocated the project resources to under-budgeted activities. It also extended the completion time of the project by a year. Data gathered from the M&E implementation are also deemed useful for the overall development of the tourism industry in the country. The National Tourism Information System (NTIS), from which the project gathered data to report on the achievement of the output and outcome targets, was improved under the project. The NTIS effectively consolidated tourism-related data in the country in a single repository. M&E Quality Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) Modest 11. Other Issues a. Safeguards Environmental Safeguards: The project was rated as Category 'B' under OP/BP 4.01 Environmental Assessment and OP/BP 4.12 Involuntary Resettlement safeguard policy was triggered. The project was to use an Environmental and Social Management Framework (ESMF) to address environmental and social risks. A Bank environmental assessment of the project at closing concluded the following: (a) the Government did not follow ESMF procedures in preparing and implementing the safeguards instruments --- the environmental clearance letters and environmental M&E reports --- for the target sites; (b) these clearance letters and reports were not approved by the competent environmental authorities in the four regional states either; (c) safeguards training provided at the federal level did not cascade to the local stakeholders; and, (d) the implementation of the ESMF was poorly supervised. Bottom-line, although the project was rated an Environmental Category 'B', some of the sub-projects, notably the sanitary landfill in Lalibela, are expected to have adverse impacts when eventually operated such that the project could likely be rated an Environmental Category 'A'. The sanitary landfill would have been rejected had the guidelines and procedures of the ESMF been faithfully followed. Other sites and works of questionable future environmental sustainability include Tiya and Melka Qunture, as well as the walkway from the Sandiago/Senbet Washa to the Church Ticket Office in Lalibela. The poor compliance with environmental safeguards was not helped by the absence of an environmental specialist in the staff of the PCU and inadequate supervision by the Bank, as well, of the implementation of the ESMF. The PCU failed to recruit an environmental specialist for the project as required in the ESMF and in the project implementation manual. Social Safeguards: The displacement by the project of 138 households in Lalibela without a Resettlement Action Plan (RAP) violated the Bank’s social safeguards policy, OP 4.12, and the terms of the Government’s financial agreement with the Bank. The involuntary resettlement merited a suspension by the Bank of disbursements of project funds for the Destination Development component of the project beginning in November 16, 2010. The disbursement suspension order was lifted in June 29, 2012 following the drafting and implementation of the RAP, in which: (a) a resettlement neighborhood, Kurakur, was created and developed with necessary infrastructure; (b) 547 households were moved from the Lalibela core site to Kurakur; (c) 670 persons affected by the project were assisted under a livelihood and income-restoration scheme; and, (d) US$2.2 million was paid in compensation to 307 displaced persons for property and assets lost. An independent Implementation Completion Audit (ICA) of the RAP concluded that the resettlement adhered to Bank policies and RAP guidelines and was implemented effectively and with the active participation of the people affected by the project. b. Fiduciary Compliance Procurement: Compliance with Bank policies and procedures for procurement and contract management proved to be a major problem during project implementation. Procurement was delayed and inefficient because: the Government did not fully appreciate the importance of effective procurement management to efficient project implementation; staffing for procurement functions at the PCU was inadequate; the implementing units had scant knowledge of procurement practices and standards; and, government decision-making on major contracts was generally slow, with little accountability for delayed actions. The Bank exerted effort to improve procurement. High-level Bank experts provided technical support on procurement, including through specialized training, and the Bank recruited a procurement specialist who was assigned to the PCU to expedite the procurement process. While procurement issues were eventually resolved, contract management continued to be a problem through the end of the project. The PCU did not closely supervise contracts. Consequently, the completion of contracts was not only delayed but the quality of some of the civil works was compromised, in particular those in Tiya and Melka Qunture. The Bank similarly exerted extra effort to improve contract management, helping the project with major contracts delivery. Financial Management: The budgeting, accounting, internal controls, and audit arrangements were generally acceptable. Minor issues were rectified in a timely manner. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) c. Unintended impacts (Positive or Negative) The project did not have any unintended impacts. d. Other No other issues were raised by the ICR. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Modest Modest --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The five lessons are drawn from the ICR (page 25), with some adaptation. One, to achieve a sustainable outcome, it is important that a tourism project encourage private participation and crowd-in private investment. The community involvement activity supported the establishment of 54 new business ventures in Addis Ababa, Axum and Lalibela. The matching grant scheme disbursed US$2.1 million to 79 participant firms in the hotel, restaurant, tour operator, tourism promotion and marketing, and handicraft businesses for purposes of product and service improvement, business systems improvement, marketing, and training. Two, integrating community development into tourism development can foster pro-poor growth. The community involvement activity disbursed US$1.1 million for the training of participants in the tourism industry to enhance their roles as employees, suppliers and business operators in the industry. The matching grant scheme also provided training to employees of firms participating in the scheme. Some 1,608 permanent jobs in Addis Ababa, Axum and Lalibela were created from the implementation of the two sub-projects. Approximately 1,490 employees benefitted from training activities. Three, where resettlement is indicated, it is vital that there be broad coordination in planning and implementing the resettlement with stakeholders at the community, regional and national levels. An independent audit found the resettlement of displaced residents of the project site in Lalibela to have been successful because of the active participation of the community members and of the local administrations apart from the national government. The resettlement neighborhood was developed with the necessary infrastructure and the community members were provided with livelihood and job opportunities. Four, the Bank should carefully consider the Government’s capacity for project implementation as early as during the project preparation stage. The scale and scope, as well as the timeline, of a project should be well aligned with the implementing agencies’ capacity for operations Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tourism Develop. Project SIL (P098132) management, procurement, contract management, financial management, environmental and social safeguards compliance, and staff recruitment. Where necessary, the Bank should consider front-loading the training of government staff, embedding experts in project implementation units, and more rigorously supervising the weaker aspects of project implementation. And five, outcome attribution is inherently difficult in a tourism project because the natural choice for a project outcome --- the increase in tourist volume --- can be the result of many factors, other than the project activity. 14. Assessment Recommended? No 15. Comments on Quality of ICR One, the ICR is thorough and comprehensive, and covers more than the basics. It explains in detail the poverty impacts, the social development effects, and the gender aspects of the project. It traces the origins, sketches the resolution, and expounds on the implications, past the project completion date, of the environmental safeguards, social safeguards, and procurement and contract management issues encountered during project implementation. It presents a very informative account of the involuntary resettlement issue in Lilibela, including: the background of the problem, the actions taken to provide a resettlement site for the displaced residents, the findings of an audit of the process, and the views of the people affected by the resettlement. Two, the ICR is very analytical and evidence-based. It details the planned (at appraisal) and the actual (at completion) outputs and outcomes of the project, by component, and by sub-component, in a very lengthy annex. It also details the assumptions, the benefits and cost flows, and the computations of the ERR of the project. It includes a detailed assessment of the factors contributing to the achievement of the project objective and of the challenges faced during project implementation. It analyzes the findings and conclusions of various stakeholder surveys, and summarizes the findings by focus group. It presents the full text of the Government’s ICR and explains the differences between the Government’s and the Bank’s assessment and rating of the project. And three, in assessing the outcome of the project, the ICR goes beyond the enumeration of project outputs and outcomes. It makes the enlightening assessment that that the project has contributed to enhancing the capacity of the tourism industry to catalyze investment, promote entrepreneurship, foster business linkages, and create employment opportunities over the longer-term. It also presents a candid assessment of the challenges to the environmental sustainability of some tourist site civil works and improvements going forward. a. Quality of ICR Rating Substantial