For Official Use Only CLR Review Independent Evaluation Group 1. CAS/CPS Data Country: Armenia CAS/CPS Year: FY14 CAS/CPS Period: FY14 – FY17 CLR Period: FY14 – FY18 Date of this review: March 18, 2019 2. Ratings CLR Rating IEG Rating Development Outcome: Moderately Satisfactory Moderately Satisfactory World Bank Group Performance: Good Good 3. Executive Summary i. Armenia is a lower middle-income country with a GNI per capita of $3,990 in 2017. It is a small and landlocked economy with borders closed with Azerbaijan and Turkey as a result of the unsettled Nagorno-Karabakh conflict 1. It faces significant trading costs while trade accounts for 75.7 percent of GDP (2016). As a result of the 2014/15 Russian crisis and the slump in metal export prices through 2016, Armenia’s annual GDP growth declined from 4.3 percent during 2009- 13 to 3.6 percent during 2014-17, even though this growth reflects a sharp rebound to 7.5 percent in 2017. Slower growth and increased unemployment slowed progress in poverty reduction. Unemployment increased from 16.2 percent in 2013 to 18.3 percent in 2015, where it remained through 2017. After declining from 35.8 percent in 2010 to 30.0 percent in 2014, the headcount poverty ratio changed little through 2016. Income inequality (the Gini coefficient) also changed little, from 31.5 in 2013 to 32.5 in 2016. During the CPS period, broader measures in social conditions improved slightly. Armenia’s Human Development Index improved from 0.729 in 2010 (76th among 169 countries) to 0.755 in 2017 (83th among 189 countries). ii. The World Bank Group’s Country Program Strategy (CPS) had three pillars, or focus areas, including the cross-cutting area on governance. These covered broadly the same areas as the previous CPS (FY09-13): (i) supporting competitiveness and job creation; (ii) improving efficiency and targeting of social services; and (iii) improving governance and decreasing corruption. The CPS was broadly aligned with the Government of Armenia (GoA) Development Strategy 2025 (ADS) adopted in 2014. The ADS sought to boost shared prosperity and reduce poverty through accelerated economic growth and job creation. World Bank Group’s support was also aligned with a number of GoA’s strategies and programs, including in the areas of strengthening competitiveness, enhancing social and environmental sustainability and improving the efficiency and transparency of public administration. The Performance and Learning Review (PLR) confirmed the relevance of the pillars and maintained most CPS objectives. PLR adjustments primarily reflected changes in country circumstances (stalled recovery and fiscal constraints). 1 Only 20 percent of Armenia’s border is open. See https://armenianweekly.com/2014/12/23/land-locked- necessity-open-borders-armenia/. CLR Reviewed by: Panel Reviewed by: CLR Review Manager/Coordinator Claude Leroy-Themeze Mauricio Carrizosa Jeff Chelsky Senior Economist, IEGEC Consultant, IEGEC Manager, IEGEC Takatoshi Kamezawa, Senior Lourdes Pagaran Evaluation Officer, IEGEC CLRR Coordinator, IEGEC For Official Use Only CLR Review 2 Independent Evaluation Group iii. At the beginning of the CPS period, the World Bank’s portfolio was $443.6 million 2 consisting of 19 IPF operations. During the CPS period, IBRD and IDA 3 approved 15 new operations amounting to $648.9 million. In the new lending, energy accounted for 18.8 percent (IPF and the PforR project). Other areas included education, agriculture, export development, jobs/tourism, roads, social protection, and public sector management (PSM). The pre-existing portfolio covered all the areas above, except for export development and jobs. New grant and trust fund commitments ($37.0 million) broadly complemented lending operations. IFC made net commitments of $190.7 million, primarily in the financial and infrastructure sectors. MIGA did not underwrite any new guarantee but had a small outstanding guarantee of $3.7 million that was cancelled in FY16. iv. IEG rates the overall development outcome as Moderately Satisfactory. Of the eight objectives, five were Mostly Achieved, two were Partially Achieved and one was Not Achieved. In Focus Area I (supporting competitiveness and job creation), progress was made in reducing tax compliance costs and the overall regulatory burden, streamlining export institutions, bringing in international investors, improving irrigation, investing in renewable energy, and improving road connectivity. However, progress was limited in expanding tourism activities and tourism related jobs outside Yerevan, improving agricultural productivity, and increasing power reliability. In Focus Area II (improving efficiency and equity in social services), significant progress was made in improving access to health services, but progress in quality of maternal and child health (MCH) and non-communicable diseases (NCD) services was limited and there is no information on improvements in efficiency. On education, there was significant progress in improving student readiness for elementary school among those having access to newly created preschools, as well as in improving the quality of secondary education. On social safety nets, Family Benefit Program (FBP) targeting did not improve and limited progress was made in increasing the coverage of poor households by the FBP. In Focus Area III, (improving governance and anti-corruption measures in public services), significant progress was made in improving the efficiency and coherence of tax administration and some progress was achieved on the institutional anti-corruption framework, but this institutional progress has not reduced perceptions of corruption yet. IEG could not verify progress in strengthening external and internal audit functions in the public sector. However, a more recent Bank assessment shows that little to no progress has been made in internal audit over the CPS period. v. On balance, IEG rates World Bank Group performance as Good. Design. The CPS addressed well-identified development challenges and benefitted from alignment with government programs and consultations with stakeholders. The selected CPS areas were consistent with WBG twin goals of poverty reduction and shared prosperity. The CPS selected a limited number of objectives, with proposed interventions based on Bank Group’s extensive knowledge and experience and the priorities of the GoA’s ADS. Planned selectivity also took into account a division of labor with major development partners. Planned interventions included operations that could reasonably be expected to achieve the objectives in most areas. The results framework could have been stronger by more closely aligning indicators with outcomes and objectives, and Bank Group interventions. The use of World Bank programmatic DPFs, IPF, PforR, and ASA and IFC investments and Advisory Services (AS) projects was appropriate. The knowledge base for the CPS was well aligned with the areas covered by World Bank Group financing. There were possible synergies between DPF and IPF in some areas (e.g., tax reform and social protection). The CPS and PLR identified risks and mitigation measures adequately. vi. Implementation. Implementation was affected by a change in government in 2014 and the deterioration of regional economic conditions after 2014. These reduced Armenia’s fiscal space, with a negative impact on portfolio performance. The World Bank Group responded by intensifying the reform dialogue with government and development partners, helping the government to raise additional budget support and mobilize additional project financing, strengthening portfolio 2 The portfolio consisted of IDA ($179 million) and IBRD ($264.6 million). 3 Armenia graduated to IBRD on July 1st, 2014, one year into the CPS. For Official Use Only CLR Review 3 Independent Evaluation Group management, and intensifying external communication with stakeholders. During implementation, the PLR made some changes to objectives, reduced the ambition of targets in some areas, and provided more time for achieving objectives. However, the changes were not sufficient to address the initial design shortcomings, including indicators that could not be verified from project documents or other public sources, or that did not fully reflect objectives or outcomes. There was little or no evidence of coordination between the Bank and IFC. Closed projects validated by IEG were reported to have satisfactory compliance with safeguards policies. Three Bank projects were investigated by the Inspection Panel (IP). IP was not able to establish material harm that may have resulted from non-compliance with Bank policies and procedures. INT substantiated three cases of attempted fraud in relation to procurement processes. Risk-mitigation measures were taken and sanctions pursued against some of the entities implicated in these cases. CAO investigated formal complaints over health concerns related to IFC’s Amulsar Gold Mine project. The CAO noted some weaknesses in IFC’s pre-investment review that translated into problems during the initial years of supervision but were subsequently addressed by IFC. vii. IEG broadly agrees with CLR lessons. Some of the key ones are summarized as follows. First, early detection of adverse shocks may help mitigate their effect on program implementation. Second, sustained engagement with development partners supported by analytical work and policy dialogue can help leverage resources from those partners. Third, citizen engagement may help prevent delays in project implementation and increase beneficiary satisfaction rates. Fourth, intensive consultations with stakeholders will enhance the relevance of ASA. viii. In addition, IEG provides the following three lessons: • First, using PEFA assessment as outcome indicators without clear plans that it will be undertaken undermines appropriate monitoring of achievements on public financial management (PFM). In the case of Armenia, outcomes from the program’s efforts in this area could not be assessed due to the absence of PEFA assessment at the time of CPS completion. Accordingly, future PFM efforts need upfront planning of PEFA products, appropriately timed to measure expected results. In addition, alternative measures of PFM efforts need to be provided in the event that PEFA assessment does not materialize as planned. • Second, supporting the establishment of improved anticorruption institutions is necessary but not sufficient to reduce actual corruption. While Armenia made progress in developing governance and anticorruption institutions and tools, lagging implementation explains the limited success in reducing corruption. On the way forward, programs in Armenia may consider supporting more downstream actions on implementation of the Anticorruption Strategy that Armenia adopted in 2015. • Third, achieving synergy by World Bank and IFC activities will require closer collaboration between the two institutions. During the CPS period, the Bank and IFC worked largely independently. The IFC’s activities on banking finance and energy investments were relevant for the program’s competitiveness and job creation pillar but had little or no coordination with Bank activities in this area. To strengthen synergies, the Bank and IFC will need to spell out together how their activities can mutually reinforce each other. 4. Strategic Focus Relevance of the WBG Strategy: 1. Congruence with Country Context and Country Program. The CPS areas of focus and objectives were broadly aligned with the GoA’s development objectives under its 2014 Armenia Development Strategy 2025 (ADS). The Government’s overarching objective is to boost shared prosperity and reduce poverty through accelerated economic growth and job creation. The ADS is built around three pillars: jobs, social protection, and State effectiveness. CPS objectives were also aligned with sector government strategies and programs under those areas, including business environment and investment climate, public revenue mobilization, industry and export promotion, For Official Use Only CLR Review 4 Independent Evaluation Group tourism development, agriculture and rural development, labor market and employment, irrigation and others. The CPS targeted significant binding constraints to achieving GoA’s overarching goals. Those constraints include poor labor market outcomes, notably high unemployment and low-paid employment, insufficient coverage and targeting of the social protection systems, and insufficient investment and firm creation that would spur competitiveness and growth. The severe underutilization of labor translates into lack of opportunities, notably for the bottom 40 percent of the population, whose households have fewer people employed and lower earnings. Improved labor outcomes require policies that promote equality of opportunity and create an enabling environment for private sector-led job creation. 2. Relevance of Design. The proposed WBG interventions could reasonably be expected to have had an impact toward CPS objectives in most areas. For example, the World Bank’s interventions (including the DPO series, Tax Administration Modernization project) and IFC Investment Climate AS projects were all geared towards improving the business environment and the investment climate. Similarly, the Local Economy and Infrastructure Development project and the Trade Promotion and Quality Infrastructure project as well as an ASA on a strategy for stimulating the tourism potential of the South Corridor were supporting the CPS objective of expanding tourism activities. However, some interventions (e.g., in tourism) lacked enough traction to achieve significant outcomes during the CPS period. Planned ASAs covered critical knowledge inputs to design policy reforms and sector investment projects in most of the areas targeted by CPS objectives (including the power sector policy note, the export development strategy note, the CEM on growth, ASA on health sector financing). The CPS was appropriately flexible on financing instruments by planning an appropriate mix of programmatic Development Policy Financing (DPF) and Investment Project Financing (IPF) and adjusting lending to changing country circumstances. The CPS also leveraged financing from trust/fund financing to complement IDA/IBRD financing operations. Planned collaboration between the World Bank and IFC sought to build synergies and catalyze more private resources toward development. Planned coordination with IMF was intended to help maintain macro stability and mitigate external shocks. Selectivity 3. The CPS was selective in focusing on three areas including a cross-cutting theme of governance, and eight objectives. Proposed interventions were based on the WBG’s extensive country knowledge and implementation experience, their congruence with the ADS goals, and their potential impact. The CPS covered some of the critical areas (e.g. lifeline roads, agriculture, irrigation, social fund, social protection projects) that would support poverty reduction and/or shared prosperity. Program selection also considered a division of labor through collaboration with other development partners. For example, in public financial management reform, the Bank Group envisaged collaboration with the IMF, the European Union (EU), and the German Corporation for International Cooperation (GIZ, Deutsche Gesellschaft für Internationale Zusammenarbeit). Also, on domestic revenue mobilization, the CPS outlined the division of labor and collaboration with the IMF, EU, USAID, and GIZ. In lifeline roads, the CPS identified collaboration with the Asian Development Bank (ADB), the European Investment Bank (EIB), and the Eurasian Development Bank (EDB). Alignment 4. The CPS objectives were broadly aligned with the 2013 corporate twin goals of poverty reduction and shared prosperity. The CPS and PLR maintained the Bank Group’s focus on key areas to reduce poverty and raise incomes for the poorest 40 percent of the population (e.g. increasing the productivity and sustainability of livestock management and pasture leasing, raising incomes in some of the poorest areas in the country, and increasing the coverage of the Family Benefit Program (FBP)). Other interventions (investment climate reforms, competitiveness, access to energy, infrastructure) were also expected to contribute indirectly to these goals by raising incomes through increased overall economic growth. For Official Use Only CLR Review 5 Independent Evaluation Group 5. Development Outcome Overview of Achievement by Objective: 5. Following the shared approach, this assessment considers the extent to which CPS objectives were achieved, as updated in the PLR. Objectives are then aggregated to arrive at focus area ratings (as an intermediate step) and overall development outcome ratings. This review uses the following structure and terminology: three focus areas, including one cross-cutting area, eight objectives, and 16 outcomes. Focus Area I: Supporting Competitiveness and Job Creation. 6. Focus Area I had four objectives: (i) improve the business environment and investment climate, especially for trade; (ii) increase investment in sectors with job-creating operations in rural and peri-urban areas outside of Yerevan; (iii) improve efficiency and sustainability of irrigation and pasture-land as key inputs to rural economy; and (iv) improve access, quality and sustainability of key infrastructure with nationwide-scope. 7. Objective 1: Improve the business environment and investment climate, especially for trade. This objective was supported through a program of four Development Policy Operations (DPOs, FY14, FY15, FY16, and FY17), the Tax Administration Modernization Project (FY13), the Improving Fiscal Management and Tax Compliance Capacity grant (FY10), the Trade Promotion and Quality Infrastructure (TPQI) project (FY15), the E-Society for Innovation for Competitiveness Project (FY11), a Non-Lending Technical Assistance (NLTA) project (FY15) on implementation of Armenia’s Export Development Strategy, and a Trust Fund Grant in support of Armenia’s Regulatory Guillotine Project. IFC provided support through its AS Investment Climate (IC) Reform Project. This objective had two outcomes and six related indicators: • Armenian businesses face lower “cost and time” barriers to profitable operations. (i) Lower tax compliance cost: [Partially Achieved]. According to the Doing Business 2018 report, the tax compliance cost for Armenia declined from 380 hours in 2013 to 313 hours per year, still above the CPS target of 190 hours. (ii) Fewer legal norms affecting businesses and citizens: [Mostly Achieved]. The target was to reduce legal acts and business processes by at least 15 percent. According to the Regulatory Guillotine Project final report – produced by the National Center for Legislative Regulation Foundation, there was a 9.7 percent reduction in the number of legal acts (251 out of 2,546 legal acts were repealed) and a 16 percent reduction in the number of business processes (101 out of 633 business processes were repealed). (iii) Reduction in total costs associated with compliance: [Achieved]. The target was a minimum drop of 35 percent in both cost and time. The Completion and Results Memorandum for the Guillotine Project indicated that under the World Bank supported reform of legal acts and business processes, the overall compliance cost declined by 50 percent and the duration (days) of the business processes declined by 47.7 percent from 2013 to 2015. IEG rates this outcome as Mostly Achieved. • Institutional framework for export development improved. (i) Streamlined export and investment promotion institutions: [Achieved]. The Armenian Development Agency (ADA), and the National Competitiveness Foundation of Armenia (NCFA) achieved the target of merging into a new agency, the Development Foundation of Armenia (DFA) in 2015. (ii) Strengthened export standards and quality infrastructure: [Not Achieved]. None of the Industrial Laboratories of the National Institute of Metrology (NIM) supported by a World Bank project have achieved the target of international accreditation. (iii) Liberalized air services (through “open skies” policy reform) leads to more diversified regional air connections: [Partially Achieved]. A Common Aviation Area agreement between Armenia and the EU was signed on November 24, 2017. However, this target does not reflect more diversified regional air connections. IEG rates this outcome as Partially Achieved. For Official Use Only CLR Review 6 Independent Evaluation Group 8. Indicators broader than those discussed above suggest some progress on competitiveness. WEF’s Global Competitiveness Index for Armenia improved from 4.0 (rank 85) in 2014 to 4.2 in 2017 (rank 73). Armenia’s Doing Business distance-to-frontier improved from 68.3 (rank 37) in 2013 to 72.5 (rank 50) in 2017. On balance, IEG rates Objective 1 as Mostly Achieved. 9. Objective 2: Increase investment in sectors with job-creating operations in rural and peri-urban areas outside of Yerevan. The World Bank supported this objective through the Local Economy and Infrastructure Development (LEIDP) project (FY16) and the Trade Promotion and Quality Infrastructure project (FY15). IFC support included investments in a mining company and a hydropower generation company. This objective had two outcomes and three related indicators: • Expansion of tourism activities (i) Increase average spending per tourist trip: [Not Achieved]. The ISR (November 2017) for the Local Economy and Infrastructure Development project reports that average spending per tourist trip ($749.50) did not change and thus remained below the target of $862.00 by September 2017. (ii) Increase tourism- related jobs in target regions outside of Yerevan: [Not Achieved]. The ISR (September 2017) for the Local Economy and Infrastructure Development project reports that the number of tourism related jobs created in the target regions remained at 133,500 by September 2017, below the target of 147,070. IEG rates this outcome as Not Achieved. • New large-scale investors in industry, based outside of Yerevan. The CPS defined one target for this outcome: International firms making first-ever Armenia investment in manufacturing or industry, located outside of Yerevan: [Achieved]. The target was met with the launch of the construction of a gold mine and the purchase and the rehabilitation of a hydropower utility. However, the contribution of the WBG to these results is unclear. In particular, for reasons not explained in the CLR, IFC stepped out of the gold mine project. IEG rates this outcome as Partially Achieved. 10. Overall, there was progress in attracting investors outside of Yerevan, but limited or no evidence of increase in private sector investment and job creation in tourism, and limited job creation in mining and power during the CPS period. On balance, IEG rates Objective 2 as Partially Achieved. 11. Objective 3: Improve efficiency and sustainability of irrigation and pasture-land as key inputs to rural economy. The World Bank supported this objective through the Irrigation System Enhancement project (FY13), the Fourth DPO, the Water Resources Management NLTA project (FY15), the Community Agricultural Resource Management and Competitiveness (CARMAC) Project (FY11) and a related Trust Fund Grant (FY13), and the Second CARMAC project (FY14). This objective had two outcomes and four related indicators: • More efficient irrigation network as a result of targeted investments and informed decision making: (i) Increase energy saved annually, from zero in 2013 to 20.6 million kilowatt- hours in 2018: [Mostly Achieved]. The ISR (September 2017) for the Irrigation System Enhancement project reports that the amount of energy saved annually in operating the irrigation system increased from zero in November 2012 to 16.6 million in August 2017. (ii) Increase water saved annually, in cubic-meters from zero in 2013 to 20.6 million cubic- meters in 2018: [Achieved]. The ISR for the Irrigation System Enhancement project (September 2017) reports that the conveyance efficiency in the targeted irrigation schemes improved from zero in November 2012 to 0.71 liters/s/per 100 meters or 24 million cubic- meters of water saved per year. IEG rates this outcome as Mostly Achieved. • Improved productivity, and sustainability of pasture-land use. (i) Increased community revenues from lease of pastures across all 175 target communities by 55 percent (2018): [Achieved]. In the 81 communities supported by the FY11 CARMAC project, the revenue from the lease of pastures increased by 71 percent. According to the April 2018 ISR for the FY14 CARMAC 2 project which supports 100 communities, revenue of beneficiary communities had increased 98 percent by November 2017. (ii) Improved milk productivity of livestock averaged across target herder communities by 20 percent (cattle) and 15 For Official Use Only CLR Review 7 Independent Evaluation Group percent (sheep) across total of 175 targeted communities: [Partially Achieved]. In 81 communities supported by the FY11 CARMAC project, milk production increased by 37.5 percent for cattle, and 28.7 percent for sheep. According to the September 2017 ISR for the CARMAC 2 project (100 communities), milk productivity of cattle had increased by 13.5 percent in July 2017. Productivity for sheep was not monitored under CARMAC 2, and thus IEG could not verify this indicator. IEG rates this outcome as Mostly Achieved. 12. The results achieved applied to targeted irrigation schemes and herder communities. The CLR does not indicate how important these inputs were to the overall rural economy. IEG rates Objective 3 as Mostly Achieved. 13. Objective 4: Improve access, quality and sustainability of key infrastructure with nationwide-scope. The World Bank supported this objective with the Electricity Supply Reliability (ESR) project (FY11) and its additional financing (FY15), the Power Transmission Improvement (PTI) project (FY15), the Power Sector Financial Recovery P4R project (FY16), the Lifeline Roads Improvement project (FY09) and its two additional financing (FY10 and FY11), and the Lifeline Road Network Improvement project (FY13) and its additional financing (FY16). IFC support comprised a $140 million financing package for a hydropower project. There were two outcomes and four indicators for this objective. • Increased adequacy and reliability of transmission network. (i) Plant and equipment failures reduced in five target facilities from 25 percent in 2013 to 10 percent in 2018: [Not Achieved]. The number of plant and equipment failures per year in target substations increased (instead of reduced) from eight in 2014 to 10 in 2017 and remained at 10 in January 2018 (ISR January 2018); (ii) Increase in international private companies investing in upgraded or greenfield power generation facilities, of which a minimum of 50 percent is exploiting renewable resources: [Partially Achieved]. The target was to have at least two international private companies invested in upgraded or greenfield power generation by 2018. One company invested in generation from a renewable resource: the upgrade of a 400-MW hydropower project was completed through its privatization and mobilizing $100 million from commercial lenders for rehabilitation. According to the CLR, 10 potential investors have been prequalified to participate in the bidding for the development of a 55-MW solar photovoltaic plant through the Utility-Scale Solar Power Project. Given this project has not yet been approved, progress cannot be attributed to World Bank’s support. IEG rates this outcome as Partially Achieved. • Selected rural roads improved as part of a wider strategy to improve both domestic and external connectivity. (i) Increase the share of rural population with access to an all- season road from 51 percent in (2013) to 66 percent in 2018: [Achieved]. This indicator increased from 51 percent in 2012 to 67.7 percent in 2017; (ii) Increased safe travel speed from 20km/h to 40km/h on a sizeable section of rehabilitated lifeline roads, from zero km in 2013 to 300 km in 2018: [Mostly Achieved]. A total of 272 km of rural roads had been rehabilitated under the Lifeline Road Network Improvement project by October 2017. The ISR (December 2017) notes that the average speed on lifeline roads in project areas increased from 20km/h in 2012 to 43 km/h in 2017. IEG rates this outcome as Mostly Achieved. 14. Significant progress was made in improving selected rural roads as part of a wider strategy to improve both domestic and external connectivity but little or no progress was achieved in increasing the adequacy and reliability of the electricity supply. While Armenia’s Doing Business indicator on supply reliability (duration and frequency of interruptions) improved, the evidence above indicates that Bank projects did not contribute to this improvement. The CLR does not provide information on the contribution of power and road project results at the nationwide level as articulated under Objective 4. On balance, IEG rates Objective 4 as Mostly Achieved. 15. On balance, IEG rates Focus Area I as Moderately Satisfactory. Of the four objectives, three were Mostly Achieved and one was Partially Achieved. On the business environment and For Official Use Only CLR Review 8 Independent Evaluation Group investment climate, the CPS made progress in lowering cost and time barriers to profitable operations and some progress in consolidating export institutions. On investment and job-creation, the CPS made little progress in tourism spending and tourism jobs, and some progress in bringing investors to industry outside of Yerevan. On irrigation and pasture land, there was good progress in improving efficiency in energy and water use in targeted irrigation schemes but mixed results in increasing revenues and productivity in targeted communities. However, there is no information on the significance of such results beyond the target communities. On key infrastructure, road connectivity improved but there is limited evidence of WBG contribution to improvements in power over the CPS period. Focus Area II: Improving Efficiency and Targeting of Social Services. 16. Focus Area II had two objectives: (i) improve efficiency, equitable access, and quality of health and education services and (ii) improve coverage and targeting of social safety net programs. 17. Objective 5: Improve efficiency, equitable access, and quality of health and education services. Health outcomes received support from the Health Systems Modernization APL (FY07) and its additional financing project (FY11); the Disease Prevention and Control Project (FY13), and the FY15 DPO. Education outcomes in High Schools received support from the Education Improvement project (FY14) and the Second Education Quality and Relevance (FY09). The CPS sought four outcomes tracked with four indicators: • Improved quality of care in maternal and child care (MCH) and non-communicable diseases (NCD). (i) Increase percentage of women aged 30-60 screened for cervical cancer at least once in last 3 years from 10.2 percent in 2012 to 50 percent in 2018: [Partially Achieved]. As reported in the ISR for the FY13 project (January 2018), this indicator increased from 8.5 percent in January 2013 to 26.8 percent in January 2018. The indicator does not capture quality of NCD care. (ii) Reduce rate of natal and post-natal complications from 86 percent in 2012 to 52 percent in 2018: [Partially Achieved]. The CLR reports that the rate of natal and post-natal complications was 49.6 percent in 2016, based on Armenia 2017 Yearbook. This information could not be verified by IEG and no World Bank Group project monitored this indicator. The Region provided additional data from the National Institute of Health Information Analytical Center that indicate a slight drop in two of the most common natal and post-natal complications (genitourinary complications by 9 percent and the rate of anemia from a baseline of 106 cases per 1,000 deliveries in 2012 to 104 in 2016). IEG rates this outcome as Partially Achieved. • Expand utilization of state guaranteed health services by poor population. Increase the rate of utilization of health services by bottom 40 percent of population (poorest two quintiles) from 4.7 percent out-patient and 5.2 percent in-patient services in 2013 to 6.9 percent of out-patient and 5.7 percent in-patient in 2018: [Achieved]. The Region provided the results of further analysis of the Integrated Living Conditions Survey (ILCS) - conducted by National Institute of Health. The analysis 4 reveals that utilization of out-patient and in-patient health services in the bottom 40 percent of the population in 2016 was 6.8 percent and 6.3 percent respectively. IEG rates this outcome as Achieved. • Enhance teaching and learning environment in high schools. Increase the percentage of students obtaining a passing rate in mathematics in the unified entrance exam from 76 percent (both genders) in 2012 to 80 percent in 2018: [Achieved]. According to a report provided by the region from the Assessment and Testing Center, the mathematics passing rate was 85.5 percent (for all students). IEG rates this outcome as Achieved. • Improved school readiness among those having access to newly created preschool education. Higher Early Development Index scores of students enrolled in new preschools (compared to control group) from no difference in 2012 to 50 percent 4 See http://microdata.worldbank.org/index.php/catalog/2966/study-description For Official Use Only CLR Review 9 Independent Evaluation Group differential in 2018: [Achieved]. According to additional evidence provided by the Region, the Early Development Index (EDI) scores improved in treatment group by 18 percentage points between the beginning and end of an academic year as compared to a 4-percentage point improvement in the control group achieving a differential much higher than targeted. IEG rates this outcome as Achieved. 18. There was significant progress in improving access to health (including for the poor) and education services, but no evidence on progress on the efficiency of those services. On balance, IEG rates Objective 5 as Mostly Achieved. 19. Objective 6: Improve coverage and targeting of social safety net programs. This outcome was supported by the Third Social Investment Fund project (FY07) and its three additional financing projects (FY09, FY10, and FY12), the Second Social Protection Administration project (FY14), the Social Investment and Community Infrastructure (FY15), and the first three DPOs. This objective had one outcome and two indicators. • Further improvement of targeting and coverage of social assistance programs. (i) Improved accuracy of targeting formula for the Family Benefit Program (FBP), increasing proportion of beneficiary families that are poor from 73 percent in 2012 to 85 percent in 2018: [Not Achieved]. According to the 2016 Integrated Living Conditions Survey results, 73 percent of the FBP beneficiaries belonged to the bottom two quintiles compared to 74.2 percent in 2015, suggesting no improvement from the baseline. (ii) Improved coverage of poor households by the FBP from 21 percent in 2012 to 27 percent in 2018: [Partially Achieved]. According to data from the National Statistics Service provided by the Region, the coverage of the poor by the FBP had improved slightly from 21 percent in 2012 to 22.3 percent in 2016 (target 27 percent). However, this indicator was not monitored under any World Bank related projects. IEG rates this outcome as Not Achieved. 20. There was minimal progress, if any, in targeting and coverage of social assistance programs. On balance, IEG rates Objective 6 as Not Achieved. 21. On balance, Focus Area II is rated as Moderately Unsatisfactory. Objective 5 was Mostly Achieved, and Objective 6 was Not Achieved. On health services, there was significant progress in improving access to health services but limited progress on quality and no information is available on efficiency. On education services, the World Bank contributed to improved school readiness resulting from increased access to new preschools and to improved education outcome from higher quality of teaching and learning environment in high schools. The targeting of the FBP did not improve and the coverage of poor households by the FBP increased marginally. Focus Area III: Improving Governance and Decreasing Corruption. 22. This Cross-cutting Area had two objectives: (i) improve the framework for fighting corruption in public services and (ii) strengthen the management of public resources. 23. Objective 7: Improve the framework for fighting corruption in public services. This outcome was supported by the Public Sector Modernization projects II (FY10) and III (FY16), the Tax Administration Modernization Project (FY13), the DPO series, an NLTA project (FY17) to the Ministry of Justice and the Anti-Corruption Council, and an NLTA project (FY16) funded from the Extractive Industries Technical Advisory Facility (EI-TAF). This objective had one outcome and two indicators. • Improved institutional framework for fight against corruption and conflict of interest. (i) Increase Global Competitiveness Indicators (GCI) on Favoritism in decisions of government officials (EU27) from 3.0 in 2012 to 3.5 in 2018: [Mostly Achieved]. Armenia’s score for the “Favoritism in decisions of government officials” improved to 3.4 in 2017 (WEF’s latest Global Competitiveness Report (2017-2018)). (ii) Formal Government of Armenia engagement in Extractive Industries Transparency Initiative (EITI) from no commitment in 2013 to active EITI candidate in 2018. [Achieved]. The target was met as For Official Use Only CLR Review 10 Independent Evaluation Group the EITI Board approved Armenia’s EITI candidature application at a Board meeting in March 2017 (EITI website). IEG rates this outcome as Mostly Achieved. 24. However, new anticorruption institutions have not yet translated into reduced perceptions of corruption. Transparency International’s corruption perceptions index deteriorated from 37/100 in 2014 to 35/100 in 2017. Considering progress on EITI and that the objective focused on improving the anti-corruption institutional framework rather than on perceptions of the extent of corruption, IEG rates Objective 7 as Mostly Achieved. 25. Objective 8: Strengthen the management of public resources. This objective was supported by the Strengthening Public Financial Management grant (FY12), the Capacity Building for Public Sector Auditing grant (FY12), the Tax Administration Modernization Project (FY13), the Improving Fiscal Management and Tax Compliance Capacity grant (FY10), the DPO 1-4 series (FY14-17), and TA in support of reforms to the Armenia Tax Code (FY16). The CPS sought the following two outcomes with two indicators: • Significant strengthening of internal control systems in government (i) Enhanced external scrutiny and audit as measured by an improvement of PEFA PI-26 from D+ in 2008 to B in 2018: [Not Verified]. No PEFA assessment has been undertaken since 2013. The CLR reports that sufficient progress has been made with financial and compliance methodology and commensurate staff training for the PI-26 to have reached a B rating in 2018. However, the CLR provides little evidence that the standards for a B rating have been met 5. (ii) Improved effectiveness of internal audit as measured by an improved score for PEFA PI-21 from D+ in 2008 to B in 2018: [Not Verified] According to the CLR, sufficient progress has been made with internal audit functions to warrant a B rating in 2018. GoA’s internal audit meet the B score for PI-21. However, the CLR does not provide sufficient evidence that internal audit in the public sector meets the criteria for a B rating on PEFA PI-21 6. A recent Bank assessment of PFM including procurement concludes that the fragmented framework of implementation of these reforms caused delays in implementation, insufficiently concrete achievements and poor synergy and impact in terms of efficiency and governance. On internal audit, the same report notes that previous TA (2011) had not been sufficiently consolidated and followed-up on by the authorities or donors 7. IEG rates this outcome as Not Achieved. • Increased efficiency and coherence of tax administration. Progress toward the CPS outcome target was as follows: Strategic and transparent framework for taxation outlined in new comprehensive tax code adopted and implemented: [Mostly Achieved]. The Unified Tax Code including measures to enhance revenues, improve the efficiency, transparency and equity of the tax system and strengthen tax administration was approved by the Parliament in October 2016 and came into full effect only in January 2018, after the CPS period, 8. 5 Criteria for a B rating on external audit: (i) Central government entities representing at least 75% of total expenditures are audited annually, at least covering revenue and expenditure. A wide range of financial audits are performed and generally adheres to auditing standards, focusing on significant and systemic issues; (ii) Audit reports are submitted to the legislature within 8 months of the end of the period covered and in the case of financial statements from their receipt by the audit office; and (iii) A formal response is made in a timely manner, but there is little evidence of systematic follow-up. 6 Criteria for a B rating on internal audit: ((i) Internal audit is operational for the majority of central government entities (measured by value of revenue/expenditure), and substantially meet professional standards. It is focused on systemic issues (at least 50% of staff time); (ii) Reports are issued regularly for most audited entities and distributed to the audited entity, the ministry of finance and the SAI; and (iii) Prompt and comprehensive action is taken by many (but not all) managers. 7 Programmatic Technical Assistance on Public Financial Management in Armenia (P165251)- Concept Note. 8 See https://www.ilex.am/en/archive/183-armenian-new-tax-law-brief,-provisions-that-come-into-force- starting-2017.html For Official Use Only CLR Review 11 Independent Evaluation Group Although this paves the way for increased efficiency and coherence of the tax administration, actual outcomes will depend on sustained implementation. On the other hand, the Region reports that, with the support of the ongoing World Bank-financed Tax Administration Modernization Project (TAMP), the efficiency of the tax administration improved between 2012 and 2017 in many areas. [Mostly Achieved]. 26. On balance, IEG rates Objective 8 as Partially Achieved. 27. Focus Area III is rated as Moderately Satisfactory. One objective was Mostly Achieved, and one was Partially Achieved. Significant progress was made in increasing the efficiency and coherence of tax administration and some progress towards improving the institutional framework to fight against corruption and conflict of interest, although perceptions of corruption did not decline. Evidence of achievement of strengthening internal control systems in government could not be verified. Overall Assessment and Rating 28. IEG rates the overall development outcome as Moderately Satisfactory. Of the eight objectives, five were Mostly Achieved, two were Partially Achieved and one was Not Achieved. In Focus Area I (supporting competitiveness and job creation), progress was made in reducing tax compliance costs and the overall regulatory burden, streamlining export institutions, bringing in international investors, improving irrigation, investing in renewable energy, and improving road connectivity. However, progress was limited in expanding tourism activities and tourism related jobs outside Yerevan, improving agricultural productivity, and increasing power reliability. In Focus Area II (improving efficiency and equity in social services), significant progress was made in improving access to health services, but progress in quality of MCH and NCD services was limited and there is no information on improvements in efficiency. On education, significant progress was made in improving student readiness to elementary school among those having access to newly created preschools, as well as in improving the quality of secondary education. On social safety nets, Family Benefit Program (FBP) targeting did not improve and limited progress was made in increasing the coverage of poor households by the FBP. In Focus Area III, (improving governance and anti- corruption measures in public services), significant progress was made in improving the efficiency and coherence of tax administration and some progress was achieved on the institutional anti- corruption framework, but this institutional progress has not reduced perceptions of corruption. As the planned PEFA was not conducted IEG could not verify progress in strengthening external and internal audit functions in the public sector. However, a more recent Bank assessment shows that little to no progress was made in internal audit over the CPS period. Objectives CLR Rating IEG Rating Focus Area I: Supporting Competitiveness and Moderately Satisfactory Job Creation. Objective 1: Improve the business environment and Mostly Achieved Mostly Achieved investment climate, especially for trade. Objective 2: Increase investment in sectors with job- creating operations in rural and peri-urban areas Partially Achieved Partially Achieved outside of Yerevan. Objective 3: Improve efficiency and sustainability of irrigation and pasture-land as key inputs to rural Mostly Achieved Mostly Achieved economy. Objective 4: Improve access, quality and sustainability of key infrastructure with nationwide- Mostly Achieved Mostly Achieved scope. Focus Area II: Improving Efficiency and Equity in Moderately Unsatisfactory Social Services. Objective 5. Improve efficiency, equitable access, Mostly Achieved Mostly Achieved and quality of health and education services. For Official Use Only CLR Review 12 Independent Evaluation Group Objective 6. Improve coverage and targeting of social Partially Achieved Not Achieved safety net programs. Cross-cutting Area: Improving Governance and Moderately Satisfactory Anti-Corruption Measures in Public Services. Objective 7. Improve the framework for fighting Mostly Achieved Mostly Achieved corruption in public services. Objective 8. Strengthen the management of public Mostly Achieved Partially Achieved resources. 6. WBG Performance Lending and Investments 29. At the beginning of the CPS period, the World Bank’s portfolio was $443.6 million 9 consisting of 19 Investment Project Financing (IPF) operations including Additional Financing. During the CPS period, the World Bank 10 approved 15 new operations amounting to $648.9 million. The new lending consisted of 10 IPF including Additional Financing, four DPO series and one PforR. IPF operations accounted for more than half (57.3 percent) of new lending; while the remainder was divided between the DPO program (38.1 percent); and one PforR operation (4.6 percent). The DPOs covered taxes, irrigation, social protection and public sector management. Of total new lending, energy alone accounted for 18.8 percent (IPF and the P4R project). Other areas with IPF projects (education, agriculture, export development, jobs/tourism, roads, social protection, and PSM) accounted each for less than 10 percent. The pre-existing portfolio covered all the areas above, except export development and jobs. In addition to lending operations, the World Bank leveraged its assistance with new grant and trust fund commitments, amounting to $21.8.0 million, broadly complementing its lending operations. 30. Armenia’s portfolio at exit performed better than the average for ECA and the Bank. Outcomes on all eight projects validated by IEG were rated Moderately Satisfactory or better, surpassing the average for ECA (93.3 percent) and for the Bank (85.4 percent). Risk to Development Outcome ratings (50 percent of projects with risk rated moderate or lower) were similar to ECA (51.7 percent) but above the overall Bank (43.5 percent). The share of projects at risk in Armenia’s active portfolio was lower, compared to ECA and the World Bank. The share of the number of projects at risk averaged 7.9 percent, less than for ECA (13.8 percent) and the Bank (21.3 percent). The share of commitments at risk for Armenia (15.3 percent) was about the same as ECA (15.0 percent) and lower than for the Bank (21.9 percent). The share of projects at risk increased in FY16, as in ECA, but declined close to previous levels in FY17. The CLR explains the portfolio management challenges experienced in FY15 and the successful remedial measures taken by the Bank, including a thorough country portfolio review, a quarterly portfolio disbursement analysis, and regular engagement with the government on crosscutting and project-specific concerns. Overall, the CLR indicates that this performance was underpinned by effective portfolio management. In particular, a 2017 portfolio review resulted in several immediate actions (restructuring of three projects and thorough review of four projects that had shown little progress). 31. During the CPS period, IFC made net commitment of $190.7 million. Two main sectors of IFC investment were the financial sector ($72.0 million or 37.8 percent of total FY14-17 net commitment) and the infrastructure sector ($61.4 million of 32. 2 percent). IFC’s annual net commitments varied from only $3.2 million in FY15, due to a total cancelation of $15.0 million, to $86.0 million in FY16. The largest project was IFC’s $47.9 million investment in the hydro power plant in FY17. IFC’s short-term trade finance guarantee was rather small with $10 million or 5.2 percent of total IFC net commitments during the review period. 9 The portfolio consisted of IDA ($179 million) and IBRD ($264.6 million). 10 Armenia graduated to IBRD on July 1st, 2014, one year into the CPS. For Official Use Only CLR Review 13 Independent Evaluation Group 32. During the review period, IEG validated two Expanded Project Supervision Reports (XPSRs) of IFC investment project and assigned development outcome ratings of Mostly Unsuccessful for two projects. The project experienced implementation delays and missed the revenue targets as the market demand was slow to pick up due to the macro environment following the 2008 crisis. 33. During the CPS period, MIGA did not underwrite any new guarantee. MIGA had a small outstanding guarantee of $3.7 million to an international microfinance group approved in FY11, but it was cancelled by the sponsor in FY16. Analytic and Advisory Activities and Services 34. During the CPS period, the World Bank completed 37 ASA tasks, comprised of 11 ESWs and 26 non-lending technical assistance (NLTA) tasks. Completed activities covered most areas where the Bank provided operational support (economic growth drivers, education, public expenditure management, tax code and policy reform, mining, power, agriculture commercialization, health, water resource management, tourism development, export development, trade, and regional development) and other topics not linked to specific CPS objectives (e.g. tasks on gender dynamics vulnerability), although relevant from a broader development perspective. Several of the reports are publicly available in the World Bank’s Open Knowledge Repository. 35. During the CPS period, IFC approved four AS projects amounting to $3.4 million of IFC funds. IFC carried out projects in the financial sector, agricultural sector, and investment climate reforms. The AS project in the financial sector was complementary to IFC’s IS operation as it sought to improve the risk management capacity of IFC client banks and helped expand its lending business to SMEs. IFC’s AS Investment Climate (IC) Reform Project supported the IC reforms in a wide range of areas, including a review of existing investment law, a review of the agriculture sector, capacity building of investment promotional agencies, and support towards Doing Business (DB) reform. 36. IEG validated three Project Completion Reports (PCRs) of AS projects and assigned Mostly Successful ratings to their Development Effectiveness. IEG’s assessment is that the AS IC project contributed towards improving the investment climate in many areas, including streamlining of the tax regime and simplification of customs valuation procedures. However, due to weak M&E, including the absence of baselines and targets, changing methodologies, insufficient evidence and overreliance on DB indicators, IEG was unable to validate the results in terms of reduction in time and costs, compliance cost savings and private sector savings. On doing business reforms, IEG finds that a solid partnership developed between the Government of Armenia and IFC, and that the IC project met most of its outcome indicators. However, the project was too ambitious with its plans for achievements and required two follow-up AS projects to complete all project objectives. Results Framework 37. The CPS objectives were well-aligned with country development goals and addressed areas with critical development constraints (e.g., power, regional development, health, education and social protection). Results chains were broadly convincing. To improve the business environment, for example, the CPS included assistance toward regulatory reform aimed at reducing regulatory costs. Overall, the outcomes had measurable indicators and had baselines and targets. However, there were weaknesses in the quality of some indicators and their alignment with outcomes or objectives, and interventions. First, some outcome indicators did not accurately reflect the associated outcomes. For example, the CPS outcome to enhance the teaching and learning environment in high schools was monitored by the increase in the percentage of students obtaining a passing rate in mathematics in the unified entrance exam. The latter, although a possible result, does not directly measure the vaguely defined outcome (teaching and learning environment). Second, some outcome indicators reflected only a part of the CPS objectives. For example, outcome indicators for the CPS objective to improve efficiency, equitable access, and quality of health and education services did not cover efficiency. Third, some indicators covered only results directly under World Bank’s projects (e.g., on irrigation and roads), with no reference to the country-wide scope of the CPS objectives. Fourth, some indicators and targets could not be verified, in part because some indicators were not tracked by program interventions supporting the outcomes. At the PLR stage, there were some adjustments, For Official Use Only CLR Review 14 Independent Evaluation Group including the reformulation of objective and adjustment in indicators and targets to reflect the changed country context. However, the changes were not sufficient to reverse the original design shortcomings of the results framework and reflect actual implementation shortcomings. Partnerships and Development Partner Coordination 38. The CPs planned for coordination and collaboration with key development partners as well as mobilization of resources including from the IMF, EU institutions, Asian Development Bank, EDB, EBRD, USAID, and GIZ. During implementation, the WBG mobilized significant resources in support of: (i) policy reforms supported by the Asian Development Bank (ADB), the Eurasian Development Bank (EDB), and the IMF; (ii) school rehabilitation with greater focus on seismic resilience supported by the ADB and UNICEF; (iii) PFM reforms supported by GIZ; (iv) regional development to address territorial disparities supported by USAID, European Union, and the Swiss Development Cooperation (SDC); and (v) energy sector and private sector development supported by the European Bank for Reconstruction and Development (EBRD) and ADB. In addition, the Bank Group, crowded in co- financing in other areas (e.g. $220 million in 2015 in irrigation and education). The Bank Group also helped to establish a platform for donor coordination and co-led sectoral donor coordination groups (infrastructure, road, energy) with the government. Finally, the World Bank and development partners supported the government in developing the new Law on Public Procurement. Safeguards and Fiduciary Issues 39. During the CPS period, eight investment operations were closed and validated by IEG, of which seven triggered environmental and social safeguards in the transport, social protection, water, and health sectors. The CLR points to proper management and implementation of safeguard arrangements, with satisfactory ratings throughout the portfolio. The projects’ ICRs and ICRRs report full participation of local communities, disclosure and proper consultation with stakeholders, constant monitoring with good records on environmental and social outcomes, and no damage to the population or the environment. Some shortfalls included implementation delays and weak fiduciary and project capacity (due to high personnel turnover at implementing agencies) on the ground. Safeguards compliance is reported as consistently satisfactory and all issues are said to have been properly addressed by the projects’ close. 40. During the CPS period, three requests for investigation were submitted to the Inspection Panel (IP) concerning two projects in the education (Second Education Quality and Relevance Project-P107772, Education (Improvement Project-P130182) and one in water (Irrigation System Enhancement Project-P127759) sectors, respectively in 2014 and 2016. The Panel concluded that, while the concerns were legitimate, it was not able to establish that the concerns were “instances of material harm that may have resulted from a failure of the Bank to follow its Operational Policies and Procedures”. According to the CLR, the Bank worked closely with the client to address the concerns. 41. Between FY14 and FY17, INT substantiated three cases of attempted fraud in relation to procurement processes. One was related to a water sector project (Municipal Water), one to a health sector project (Second Health System Modernization Project APL2), and one to a transport sector project (Lifeline Road Network Improvement). In each case, companies had submitted fraudulent documents in their bids. Risk-mitigation measures included recommendations to encourage due diligence by client agencies when validating documentation in bids. Sanctions were pursued against some of the entities implicated in these cases. 42. The IFC’s Office of the Compliance Advisor Ombudsman (CAO) investigated formal complaints filed by NGOs and local residents in 2014 over health concerns regarding IFC’s Amulsar Gold Mine project. The CAO concluded that, while IFC’s support to the client in preparing an Environmental and Social Impact Assessment (ESIA) addressed the risks and potential impacts of the mine, IFC’s efforts met the requirements of IFC’s Sustainability Framework and reflected good international industry practice, but that weaknesses in IFC’s pre-investment review and due diligence translated into problems during the initial years of supervision. The CAO found that IFC did not effectively supervise the client’s delivery of agreed environmental and social (E&S) mitigation measures from 2007 to 2013. In particular, the Environmental and Social Management System For Official Use Only CLR Review 15 Independent Evaluation Group (ESMS) for exploration activities was not completed in a timely manner. As a result, IFC did not have assurance that the exploration activities were being carried out in accordance with its E&S requirements or that the client’s ongoing E&S assessment work was being conducted to IFC standards. However, the CAO noted a marked improvement in IFC’s supervision of the project from 2013 onward. The result was significant improvement in client E&S performance. In its response to the CAO, IFC indicated that it had recently formalized existing practice into documented procedural guidance for the E&S appraisal and supervision of phased development projects to avoid similar problems in future projects. Ownership and Flexibility 43. The GoA demonstrated ownership both at the design and implementation stages. Alignment with GoA’s programs and World Bank’s consultations with a broad range of stakeholders indicates ownership at the design stage. Furthermore, the government’s proactive portfolio management and PIU reorganization to mitigate the adverse impact of emerging fiscal gaps suggest sustained commitment during implementation. At the PLR, the World Bank showed flexibility by adjusting the indicators and targets to reflect changed country circumstances (e.g., fiscal constraints). More focus was provided on supporting resilience to shocks and the revisions reflected the reduced space for net job creation. Indicators for results in tourism, regulatory simplification, and transparency were redefined based on the new Government’s programs (2014 change) and to more accurately reflect project design. Delays in project launch, and changes linked to Eurasian Economic Union (EEU) accession, required revision to indicators for export development and tax administration. World Bank Group Internal Cooperation 44. The CPS and the PLR outlined Bank and IFC areas of collaboration (business and investment climate reforms, tax administration modernization, financial intermediation and access to financial services, implementation of government strategy for agriculture and rural development, mining, and energy). Nevertheless, there are no indications of actual cooperation during implementation. Instead, the Bank and IFC worked largely separately on their own areas of engagement. For example, the World Bank helped the government to prepare a new mining code, improve regulations to enhance the mining sector’s social and environmental sustainability, and become a candidate to EITI. IFC supported the development of Armenia’s largest gold mine but eventually stepped-out. Similarly, the World Bank’s support of energy sector reforms aimed at improving sector sustainability while IFC supported private sector investment in the sector. Risk Identification and Mitigation 45. The CPS identified a number of risks that could significantly derail government programs (macroeconomic conditions, regional security, domestic political developments and vested interests, and natural disaster (earthquakes) and climate change risks). The PLR highlighted risks that had materialized at that stage (stalled recovery, regional recession, currency crises) and that resulted in social tensions and fiscal gaps. Political economy risks also materialized and contributed to implementation delays and to polarized debates on key reforms under the WBG program (energy, taxation). The CPS set out several mitigation measures that were strengthened in the PLR and implemented as the risks materialized. For example, to mitigate risks from the economic slowdown after 2014, the World Bank Group increased its dialogue on structural reforms with government stakeholders and development partners, coordinated more closely with the IMF on macroeconomic issues, and helped mobilize additional budget support. Furthermore, in response to the larger fiscal gaps that materialized, the CLR reports that the World Bank supported the government by developing criteria for prioritizing capital expenditures and a framework for PIU consolidation and conducted a portfolio review to develop immediate mitigation measures. Overall Assessment and Rating 46. IEG rates performance as Good. Design. The CPS addressed well identified development challenges and benefitted from alignment with government programs and consultations with stakeholders. The selected CPS areas were consistent with World Bank Group’s twin goals of poverty reduction and shared prosperity. The CPS selected a limited number of objectives, with For Official Use Only CLR Review 16 Independent Evaluation Group proposed interventions based on WBG’s extensive knowledge and experience and the priorities of the GoA’s ADS. Planned selectivity also considered a division of labor with major development partners. Planned interventions included operations that could reasonably be expected to achieve the objectives set in most areas. The results framework could have been stronger by more closely aligning outcomes indicators with outcomes or objectives and interventions. The use of World Bank programmatic DPOs, IPF, PforR, and ASA and IFC investments and AS projects was appropriate. The knowledge base for the CPS was well aligned with the areas covered by WBG financing. There were possible synergies between DPOs and IPF in some areas (e.g., social protection, tax reform, energy). The CPS and PLR identified risks and mitigation measures adequately. Implementation. 47. Implementation was affected by a change in government in 2014 and the deterioration of regional economic conditions after 2014. These resulted in a higher public debt and narrower fiscal space, with a negative impact on portfolio performance. The World Bank Group responded constructively, by intensifying the reform dialogue with government and development partners, helping the government to raise additional budget support and mobilize additional project financing, strengthening portfolio management, and intensifying external communication with stakeholders. During implementation, the PLR dropped two objectives (on access to finance and innovation), added others (e.g., on investment), and reformulated others (e.g., on irrigation). The PLR also reduced the ambition of targets in some areas and provided more time for achieving program objectives. However, the changes were not sufficient to address initial design shortcomings at the CPS stage. Many outcome indicators did not fully reflect the outcome, or indicators only partly reflected the objectives, and others could not be verified. There were no joint World Bank-IFC operations and little or no evidence of coordination between the two institutions. Donor coordination was strong. The closed projects validated by IEG were reported to have satisfactory compliance with safeguards policies. On three requests for investigation submitted to the IP, the Panel concluded that it was not able to establish instances of material harm that may have resulted from non-compliance with Bank operational policies and procedures. INT substantiated three cases of attempted fraud in relation to procurement processes. Risk-mitigation measures were taken, and sanctions were pursued against some of the entities implicated in these cases. CAO investigated formal complaints filed by NGOs and residents over health concerns related to IFC’s Amulsar Gold Mine project. The CAO noted some weaknesses in IFC’s pre-investment review that translated into problems during the initial years of supervision but were subsequently addressed by IFC. Following the complaints, IFC divested from the project. 7. Assessment of CLR Completion Report 48. The CLR provides an informative and comprehensive assessment. The CLR provides evidence on the extent to which outcomes were achieved as well as on the World Bank Group’s contribution to these results, albeit with the caveats on the inadequacy of some outcome indicators. It discusses links of the WBG program to shared prosperity and poverty reduction goals. There was adequate attention to the role played by ASA. The CLR provided adequate discussion of implementation challenges and Bank’s responses to these challenges. The CLR could have been more explicit on IFC’s pull-out of Amulsar gold mine 11, its implications for investment and mine development, and the role of IFC in this project. More generally, the CLR could have provided a more robust and detailed account of how the Bank and IFC collaborated, especially in mining, energy, and the investment climate. 11 Beginning in 2007, IFC supported exploration efforts and subsequent feasibility studies to advance the project to the construction phase, which commenced in 2017. According to the CLR, the developer succeeded in raising financing for construction of the mine from private sources without IFC participation. In 2017, IFC exited from the company. For Official Use Only CLR Review 17 Independent Evaluation Group 8. Findings and Lessons 49. IEG broadly agrees with CLR lessons. Some of the key lessons are summarized as follows. First, early detection of adverse shocks may help mitigate their effect on program implementation. Second, sustained engagement with development partners, supported by analytical work and policy dialogue, can help leverage resources from those partners. Third, citizen engagement may help prevent delays in project implementation and increase beneficiary satisfaction rates. Fourth, intensive consultations with stakeholders will enhance the relevance of ASA services. 50. IEG provides the following three lessons: • First, using PEFA assessment as outcome indicators without clear plans that it will be undertaken undermines appropriate monitoring of achievements on public financial management (PFM). In the case of Armenia, outcomes from the program’s efforts in this area could not be measured due to the absence of PEFA assessment at the time of CPS completion. Accordingly, future PFM efforts need upfront planning of PEFA products, appropriately timed to measure expected results. In addition, alternative measures of PFM efforts need to be provided in the event that PEFA assessment does not materialize as planned. • Second, supporting the establishment of improved anticorruption institutions is necessary but not sufficient to reduce actual corruption. While Armenia made progress in developing governance and anticorruption institutions and tools, lagging implementation explains the limited success in reducing corruption. On the way forward, programs in Armenia may consider supporting more downstream actions on implementation of the Anticorruption Strategy that Armenia adopted in 2015. • Third, achieving synergy by World Bank and IFC activities will require closer collaboration between the two institutions. During the CPS period, the Bank and IFC worked largely independently. The IFC’s activities on banking finance and energy investments were relevant for the CPS’s competitiveness and job creation pillar but had little or no coordination with Bank activities in this area. To strengthen synergies, the Bank and IFC will need to spell out together how their activities can mutually reinforce each other. Annexes CLR Review 19 Independent Evaluation Group Annex Table 1: Summary of Achievements of CPS Objectives – Armenia Annex Table 2: Armenia IBRD/IDA Planned and Actual Lending, FY14-FY17 Annex Table 3: Advisory Services and Analytics Work for Armenia, FY14-FY17 Annex Table 4: Armenia Grants and Trust Funds Active in FY14-17 Annex Table 5: IEG Project Ratings for Armenia, FY14-17 Annex Table 6: IEG Project Ratings for Armenia and Comparators, FY14-17 Annex Table 7: Portfolio Status for Armenia and Comparators, FY14-17 Annex Table 8: Disbursement Ratio for Armenia, FY14-17 Annex Table 9: Net Disbursement and Charges for Armenia, FY14-17 ($, millions) Annex Table 10: Total Net Disbursements of Official Development Assistance Annex Table 11: Economic and Social indicators for Armenia, 2013-2017 Annex Table 12: List of IFC Investments in Armenia Annex Table 13: List of IFC Advisory Services in Armenia Annex Table 14: IFC net commitment activity in Armenia, FY14 - FY17 Annex Table 15: List of Active MIGA Activities in Armenia, 2014-2017 Annexes CLR Review 21 Independent Evaluation Group Annex Table 1: Summary of Achievements of CPS Objectives – Armenia CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation Objective 1: Improve the business environment and investment climate, especially for trade Outcome 1: Armenian This outcome received support from the This objective was businesses face lower ‘cost Tax Administration Modernization Project clarified at PLR stage. and time’ barriers to profitable (P111942:FY13) and from the Improving The original CPS operations: Fiscal Management and Tax Compliance objective was, Capacity grant (see the August 2012 ISR: "Enabling environment Indicator 1: Lower tax MS P120122: FY10). These projects for business operations compliance cost sought to modernize Armenia’s tax improved." administration to (i) increase voluntary tax Baseline (2013): 380 hours compliance, (ii) reduce tax evasion, (iii) Target (2018): 190 hours reduce compliance costs; and (iv) increase administrative efficiency. The outcome was also supported by the First (P127754:FY14), Second (P143040:FY15), Third (P153234:FY16) and Fourth (P160100 : FY17) Development Policy Operations (DPOs) which sought to strengthen the business environment by reducing the burden of inspections on businesses among other objectives such as unifying and rationalizing taxation and reducing time Major and costs of customs clearances. Outcome Measures This outcome received support from the Trade Promotion and Quality Infrastructure project (P146994:FY15) which sought to strengthen the government's capacity to provide export promotion, investment attraction and quality management services to firms. The outcome was also supported through a Non-Lending Technical Assistance (TA) project (P145530:FY15) to assist the Ministry of Economy of Armenia in the implementation of the Export Development Strategy. A trust fund grant for strengthening the capacity of the Regulatory Reform Unit of the Armenian government also supported this outcome (P128769:FY13). IFC support to this outcome included the Armenia Investment Climate Reform advisory Project (588147: FY12) and the Armenia Investment Climate II project (600244: FY15) which sought to improve the investment climate in Armenia by reducing the time and cost of doing Annexes CLR Review 22 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation business and increasing the effectiveness of regulation in four key areas: Business taxation, Trade logistics, Business inspections, and Food safety. A trust fund grant for strengthening the capacity of the Regulatory Reform Unit of the Armenian government also supported this outcome (P128769:FY13). The Implementation Completion and Results Memorandum that reports on the development of Regulatory Impact Assessments (RIA) undertaken for new regulations, as of July 2015. According to the Doing Business 2018 report, the tax compliance cost for Armenia was 313 hours per year. Partially Achieved Outcome 1: Armenian According to the Regulatory Guillotine businesses face lower ‘cost Project final report – produced by the and time’ barriers to profitable National Center for Legislative Regulation operations: Foundation, 251 out of 2,546 legal acts (9.7%), and 101 out of 633 business Indicator 2: Fewer legal norms processes (16%) were repealed. affecting businesses and citizens Mostly Achieved Baseline (2013): 2546 legal acts and 604 business processes Target (2018): Reduction of at least 15% Outcome 1: Armenian According to the Implementation businesses face lower ‘cost Completion and Results Memorandum of and time’ barriers to profitable P128769, the WB supported reform of operations: legal acts and business processes resulted in an overall compliance cost Indicator 3: Lower Total costs reduction of 50% or 23,667 million AMD, associated with compliance compared to a baseline of 47,086 million AMD. Furthermore, the reforms resulted in Baseline (2013): regulatory a 47.7 percent reduction in duration (days) burden = 13,731 days and of the business processes from the initial 204.5m AMD total of 16,367. Target (2018): minimum drop of 35% in both cost and time Achieved Outcome 2: Institutional This outcome received support from the framework for export Trade Promotion and Quality development improved: Infrastructure project (P146994:FY15) which sought to strengthen the Annexes CLR Review 23 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation Indicator 1: Streamlined export government's capacity to provide export and investment promotion promotion, investment attraction and institutions quality management services to firms. The outcome was also supported through Baseline (2013): Fragmented a Non-Lending Technical Assistance (TA) activities on export and project (P145530:FY15) to assist the investment promotion. Ministry of Economy of Armenia in the Target (2018): Export and implementation of the Export investment promotion activities Development Strategy. consolidated under one According to the September 2017 Government Agency. (ISR:MU), for project P146994, the merger of the Industrial Development Fund (IDF), the Armenian Development Agency (ADA), and the National Competitiveness Foundation of Armenia (NCFA) into a new agency, the Development Foundation of Armenia (DFA) was completed by September 2015. The website of the DFA agency indicates that it was established in 2015. Achieved Outcome 2: Institutional According to the May 2018 ISR:MU for the Although both the CLR framework for export Trade Promotion and Quality and the additional development improved: Infrastructure project P146994, none of evidence provide by the Industrial Laboratories of the National the Region suggests Indicator 2: Strengthened Institute of Metrology (NIM) were that one laboratory export standards and quality accredited by an accreditation (FDA Lab) was infrastructure. organization that is a signatory to an upgraded to Multilateral Recognition Agreements internationally Baseline (2013): Zero quality (MLAs) or Mutual Recognition recognized standards infrastructure (QI) laboratories Agreements (MRAs) from the International in 2016, there is no with international certification/ Laboratory Accreditation Cooperation or evidence that this was accreditation the International Accreditation Forum. achieved as a result of Target (2018): A minimum of 2 World Bank Group QI labs upgraded to Evidence provided by the Region, in the support. internationally recognized form of a certificate of accreditation issued standards by an international standards organization (ANSI-ASQ National Accreditation Board (ANAB) - suggests that one quality infrastructure laboratory (FDA Laboratory) was upgraded to internationally recognized standards in 2016. Not Achieved Outcome 2: Institutional The outcome was supported by the First At PLR stage, the framework for export (P127754:FY14), Second specification of this development improved: (P143040:FY15), and Third indicator was modified. (P153234:FY16) DPOs which supported The original CPS efforts to strengthen the legal, regulatory indicator was specified as follows, “Air Annexes CLR Review 24 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation Indicator 3: Liberalized air services liberalized. and institutional oversight of the aviation services (through ‘open skies’ industry. Number of destinations policy reform) leads served from Yerevan to more diversified regional air According to media reports, the Common Baseline: 33 connections Area agreement between Armenia and the Target: 40 or adoption EU was signed on November 24, 2017. of a Common Aviation Baseline (2013): Zero Area (CAA) agreement participation in regional aviation Partially Achieved with the European zones/ agreements Union. Target (2018): Adoption of a Common Aviation Area (CAA) Agreement with the European Union (EU) Objective 2: Increase investment in sectors with job-creating operations in rural and peri-urban areas outside of Yerevan Outcome 1: Expansion of This outcome was supported by the Local Objective 2 was tourism activities: Economy and Infrastructure Development introduced at PLR project (P150327:FY16) and the Trade stage. Indicator 1: Average spending Promotion and Quality Infrastructure per tourist trip. project (P146994:FY15). These projects aimed at improving infrastructure services Baseline (2013): $749.50. and the institutional capacity for increased Target (2018): US $862.00 tourism contribution to the Armenian economy. The outcome was also supported through an analytical report outlining a strategy on how to stimulate the tourism potential of the South Corridor (P158280:FY16) According to the November 2017 ISR:MS for the Local Economy and Infrastructure Development project, the average spending per tourist trip did not change and was still $749.50 by September 2017. Not Achieved Outcome 1: Expansion of According to the November 2017 ISR:MS According to the 2017 tourism activities: for the Local Economy and Infrastructure World Travel and Development project the number of Tourism Council Indicator 2: Tourism-related tourism related jobs created in the target Economic Impact report jobs in target regions outside of regions did not increase and was still for Armenia (entire Yerevan. 133,500 by September 2017. country), the total contribution of Travel & Baseline (2014): 133,700. According to the Armenia World Travel Tourism to employment Target (2018): 147,070 (10% and Tourism Council reports for 2015 and including jobs indirectly increase) 2018, the total contribution of Travel & supported by the Tourism to employment, including jobs industry was 143,500 *jobs data will be sex- indirectly supported by the industry jobs during 2016 disaggregated. increased from 11.3% of total employment compared to about (133,500 jobs) in 2014 to 14.1% of total 133,500 jobs (directly employment (162,000 jobs) in 2017. and indirectly) in 2015 Annexes CLR Review 25 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation Although IEG could not verify the share of (see the 2015 World tourism-related jobs in the target regions Travel and Tourism outside of Yerevan, there is some Council Economic evidence of increase in the tourism related Impact report for jobs. However, this possible improvement Armenia). can’t be attributed to WB support. Not Achieved Outcome 2: New large-scale This outcome was supported by IFC In 2017 IFC pulled out investors in industry, based through investments in a mining company of the Amulsar gold outside of Yerevan: and a power generation company. mine project. Two formal complaints filed Indicator 1: International firms Achieved by NGOs and residents making first-ever Armenia in 2014 over health investment in manufacturing or and environment industry, location outside of concerns were Yerevan. investigated by the IFC’s Office of the Baseline (2013): Zero. Compliance Advisor Target: Minimum of two. Ombudsman’s. Objective.3: Improve efficiency and sustainability of irrigation and pasture-land as key inputs to rural economy Outcome 1: More efficient This outcome received support from the Objective 1.3 was irrigation network as a result Irrigation System Enhancement project introduced at PLR of targeted investments and (P127759:FY13) which sought to reduce stage. informed decision making: the amount of energy used and to improve the irrigation conveyance efficiency in At PLR stage, the Indicator 1: Energy saved targeted irrigation schemes. specification of this annually, in kilowatt-hours. The outcome was also supported by the outcome was modified. Fourth Development Policy Operation The original CPS Baseline (2013): 0. (P160100:FY17) which supported reforms outcome was specified Target (2018): 20.6 million to improve the financial sustainability of as follows, “Reduced the Armenian Irrigation Sector. energy costs and water The outcome also benefitted from a Water losses as a result of resources management NLTA project (P targeted investments 152862: FY15) that conducted a and informed decision diagnostic of water resources making.” management, reviewed previous reforms The target value for and made recommendations. this indicator was also According to the September 2017 (ISR:S) revised downwards of the Irrigation System Enhancement from 38.9 to 20.6 project the amount of energy saved million KWT/hour. annually in operating the irrigation system increased from zero in November 2012 to 16.59 million in August 2017. Mostly Achieved Outcome 1: More efficient According to the September 2017 ISR:S The target value for irrigation network as a result of project P127759, the conveyance this indicator was efficiency in the targeted irrigation revised downwards Annexes CLR Review 26 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation of targeted investments and schemes improved from zero in November from 20.7 to 20 million informed decision making: 2012 to 0.71 liters/s/per 100 meters. This cubic meters. is the equivalent of 24 million cubic meters Indicator 2: Water saved of water saved per year. annually, in cubic-meters. Achieved Baseline (2013): 0 Target (2018): 20 million Outcome 2: Improved This outcome was supported by the At PLR stage, the productivity, and Community Agricultural Resource specification of this sustainability of land and Management and Competitiveness outcome was modified. pasture use: (CARMAC) Project (P120028:FY11) and a The original CPS related Trust Fund Grant (P144283:FY13). outcome was specified Indicator 1: Increased It was also supported by the CARMAC 2 as follows, “Improved community revenues from lease project (P133705:FY14). efficiency and of pastures across all 175 target The projects sought to improve sustainability of land communities. productivity and sustainability of and pasture use.” pasture/livestock livelihood systems in Baseline (2013): 0 selected communities of Armenia and to Target (2018): 55% increase the marketed production from selected livestock and high value agri-food value chains. According to IEG:S for the CARMAC project which supported 81 communities, the revenue from the lease of pastures increased from AMD 328,893 in 2012 to AMD 561,395 in 2016 (71% increase). According to the April 2018 ISR:S for the CARMAC 2 project which supports 100 communities, the value of this indicator increased from an 80 percent increase by July 2017 to 98 percent increase by November 2017 which is a marked improvement, exceeding the target of a 55 percent increase. Achieved Outcome 2: Improved According to IEG:S for the CARMAC productivity, and project milk production increased from sustainability of land and 1,428 kg/year in 2011 to 1,964 kg/year in pasture use: 2016 for cattle (a 37.5% increase) while for sheep, the increase was from 66 kg to Indicator 1: Improved milk 85 kg/year (a 28.7% increase). productivity of livestock The April 2018 ISR: S for the P133705 averaged across target herder reports that annual milk production per communities. dairy cattle increased from 1,420 liters to 1,612 liters (a 13.5% increase) between Baseline (2013): 0 October 2013 and November 2017. Target (2018): 20% (cattle) and CARMAC 2 project does not monitor milk 15% (sheep) across total of 175 productivity for sheep. communities Annexes CLR Review 27 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation The ISR indicates that the milk production indicator is an average for 94 communities engaged in the project. Overall, the project initially planned for 100 communities is currently engaged with 111 communities. Partially Met (cattle); Not Verified (sheep under CARMAC 2). Objective 4: Improve access, quality and sustainability of key infrastructure with nationwide- scope Outcome 1: Increased This outcome received support from the At PLR stage, the adequacy and reliability of Electricity Supply Reliability (ESR) project specification of this transmission network: (P116748:FY11) and a related additional objective was modified. financing project (P148102:FY15). It was The original CPS Indicator 1: Plant and also supported by the Power objective was specified equipment failures reduced in Transmission Improvement project as follows, “Improved five target* facilities. (P146199:FY15) and a Power Sector access, quality and Financial Recovery P4R project sustainability of key Baseline (2013): 25%. (P157571:FY16). The projects aimed to infrastructures.” Target (2018): 10% improve the reliability and capacity of the The specification of the * four substations reach 22% of power transmission network and support outcome was also the nation’s end-users, and one Armenia’s efforts in ensuring adequate modified at the PLR has nation-wide reach electricity supply. The Power stage. The original Transmission Improvement Project specification of the supported the rehabilitation of two power outcome was as substations while the ESR project follows: “Increased supported the rehabilitation of 3. reliability of transmission network.” According to the November 2017 ISR:MS for the ESR project, the number of plant The outcome was also and equipment failures per year in target supported by substations increased from 8 in March Economic & Sector 2014 to 10 in October 2017. Work that included a The January 2018 ISR:S for Power Power Sector Policy Transmission project P146199 indicated Note (P133834:FY15) that the number of plant and equipment that identified failures per year in target substations was challenges and 10 in December 2014 and remained investment needs in unchanged in the period up to January the power sector and a 2018. Power Sector Financial Recovery Plan that Not Achieved included an analysis of the financial standing of the power sector in Armenia in 2016. Outcome 1: Increased This outcome was supported by an IFC adequacy and reliability of $140 million financing package for a transmission network: hydropower utility, one of Armenia’s major power generating facilities. Indicator 2: Increase in international private companies Annexes CLR Review 28 Independent Evaluation Group CAS FY14-FY17: Focus Area I: Supporting Competitiveness Actual Results IEG Comments and Job Creation investing in upgraded or According to the CLR, 10 potential greenfield power generation investors have been prequalified to facilities, of which a minimum of participate in the bidding for the 50% are exploiting renewable development of a 55-MW solar resources. photovoltaic plant through the Utility- Scale Solar Power Project. Given this Baseline (2013): 0 project has not yet been approved, this Target (2018): 2 progress can’t be attributed to Bank’s support. Partially Achieved Outcome 2: Selected rural This outcome received support from the At PLR stage, the roads improved as part of a Lifeline roads improvement project target value for this wider strategy to improve (P115486:FY09) and two related indicator was both domestic and external additional financing projects in FY10 increased from 62% to connectivity: (P116760) and FY11 (P121287). 66%. The outcome was also supported by the Indicator 1: Share of rural Lifeline Road Network improvement population with access to an all- project (P126782:FY13) and a related season road additional financing project (P150505:FY16). Baseline (2013): 51%. The projects sought to improve access of Target (2018): 66% rural communities to markets and services by upgrading selected sections of the lifeline road network. According to the December 2017 ISR:MS for the Lifeline Road Network improvement project, the value of this indicator increased from 51% in November 2012 to 67.68% in October 2017. Achieved Outcome 2: Selected rural According to the December 2017 ISR:MS roads improved as part of a the Lifeline Road Network improvement wider strategy to improve project, 272 km of rural roads had been both domestic and external rehabilitated under the project by October connectivity: 2017. The report notes that the average speed on lifeline roads in project areas Indicator 2: Safe travel speed increased from 20km/h in November 2012 increased from 20km/h to to 42.99 km/h in October 2017. 40km/h on sizeable section of rehabilitated lifeline roads. Mostly Achieved Baseline (2013): 0. Target (2018): 300km Annexes CLR Review 29 Independent Evaluation Group CAS FY14-FY17: Focus Area II: Improving efficiency, equitable Actual Results IEG Comments access, and quality of health and education services Objective 5: Improve efficiency, equitable access, and quality of health and education services Outcome 1: Improved quality This outcome received support from the At PLR stage, the of care in MCH and NCD: Health Systems Modernization APL specification of this (P104467: FY07) and the related objective was clarified. Indicator 1: Percentage of additional financing project The original CPS women aged 30-60 screened for (P121728:FY11). It was also supported by objective was specified cervical cancer at least once in the Disease Prevention and Control as follows, “Improved last 3 years. Project (P128442:FY13) which had two efficiency and equity of related Trust Fund grants. health services.” Baseline (2012): 10.2% The Disease Prevention and Control Target (2018): 50% Project sought to improve (i) Maternal and The baseline value Child Health (MCH) services and the specified for this prevention, early detection, and indicator was higher management of selected Non- (10.2% in 2012) than Communicable Diseases (NCD) at the the baseline value for Primary Health Care (PHC) level. the project (8.5% in According to the January 2018 ISR: MS of 2013). A possible the Disease Prevention and Control explanation for this Project (P128442, FY13), the value of this could be that the indicator increased from 8.5% in January project level indicator 2013 to 26.8% in January 2018. was defined more Major strictly and required Outcome Partially Achieved that the beneficiary of Measures the screening to have received the results of the screening exercise. Outcome 1: Improved quality The CLR reports that the rate of natal and None of the World of care in MCH and NCD: post-natal complications was 49.6 percent Bank projects included in 2016, based on Armenia 2017 this as an indicator. Indicator 2: Rate of natal and Yearbook. This information could not be post-natal complications. verified by IEG and no project monitored this indicator. Baseline (2012): 86% Target (2018): 52% Maternal and neonatal mortality rates (28 per 100,000 live births and 8.6 per 1,000 live births respectively in 2013) declined to 25 (2015) and 7.4 (2016), respectively. 12 Additional information provided by the Region suggests a 9% drop in the rate of genitourinary complications and a 2% decline in the rates of anemia from a baseline of 106 cases in 2012 to 104 per 1,000 deliveries in 2016. Partially Achieved Outcome 2: Utilization of state This outcome received support from the The specification of guaranteed health services by Health Systems Modernization APL this indicator was poor population expanded: (P104467: FY07) and the related modified at PLR stage. 12 World Development Indicators. Annexes CLR Review 30 Independent Evaluation Group CAS FY14-FY17: Focus Area II: Improving efficiency, equitable Actual Results IEG Comments access, and quality of health and education services additional financing project The original indicator Indicator 1: Rate of utilization of (P121728:FY11). The objective of the was specified as health services by bottom 40% project was to improve the organization of follows: Rate of of population (poorest two the health care system in order to provide utilization of in-patient quintiles). more accessible, quality and sustainable and out-patient health health care services to the population, in services by poorest Baseline (2011): 4.7% out- particular to the most vulnerable groups, quintile of population. patient, 5.2% in-patient services. and to better manage public health Target (2018): 6.9 % out-patient, threats. None of the World 5.7% in-patient The outcome was also supported by the Bank supported * utilization will be sex- DPO series which supported the projects included this disaggregated extension of the coverage of the health as an indicator. benefits (waiver of co-payments) to the same group of households covered by the Family Benefits Program through the Law on 2014 State Budget. The Region provided the results of further analysis of the Integrated Living Conditions Survey (ILCS) - conducted by National Institute of Health. The analysis13 shows that utilization of out- patient and in-patient health services in the bottom 40 percent of the population in 2016 was 6.8 percent (target 6.9 percent) and 6.3 percent (target of 5.7 percent) respectively. Achieved Outcome 3: Teaching and This outcome received support from two None of the projects learning environment in high projects: the Education Improvement that supported this schools is enhanced: Project (P130182: FY14) and the Second outcome monitor this Education Quality and Relevance indicator. Indicator 1: Percentage of (P107772: FY09). students obtaining a passing The projects sought to improve the school rate in mathematics in the readiness of children entering primary unified entrance exam education, improve physical conditions and the availability of educational Baseline (2012): 76 % (both resources in upper secondary schools, genders) and support improved quality and Target (2018): 80 % relevance in higher education institutions in Armenia. The Region provided a report from National Center for Education Technology on the results of the unified exams conducted by the Assessment and Testing 13 See http://microdata.worldbank.org/index.php/catalog/2966/study-description Annexes CLR Review 31 Independent Evaluation Group CAS FY14-FY17: Focus Area II: Improving efficiency, equitable Actual Results IEG Comments access, and quality of health and education services Center, the mathematics passing rate was 85.5% (for all students). Achieved Outcome 4: Improved school According to the June 2016 IEG:MS of Considering that this readiness among those the Second Education Quality and outcome referred to having access to newly Relevance project, the Early preschool education, created preschool education: Development Index (EDI) score was 19% the indicator should higher in children reached by the project have been the Early Indicator 1: Higher Educational compared with a control group. Development Index Development Index scores of and not the Higher students enrolled in new Additional evidence provided by the Educational preschools (compared to control Region indicates that on average, EDI Development Index. group) scores improved in treatment group by 18 percentage points between the beginning Baseline (2012): No difference and end of an academic year as Target (2018): 50% differential* compared to a 4-percentage point *scores will be sex- improvement in the control group. Taking disaggregated the control group as a reference, the average score improvement in the treatment group and control groups was 350 percent higher than in the control group, showing that the target was achieved. Achieved Objective 6 Improve coverage and targeting of social safety net programs Outcome 1: Further This outcome received support from the At PLR stage, the improvement of targeting and Social Investment Fund 3 project specification of this coverage of social assistance (P094225:FY07) and three related objective was clarified. programs: additional financing projects The original CPS (P115743:FY09), (P118158:FY10), and objective was specified Indicator 1: Improved accuracy (P128740:FY12). as follows, “Improved of targeting formula for Family The outcome was also supported by the coverage and targeting Benefit Social Protection Administration II project of FBP and efficiency Program (FBP), increasing (P146318:FY14) which sought to improve of social services proportion of beneficiary families social protection service delivery and to delivery. that are poor: strengthen analytical and monitoring and evaluation functions of the agencies The specification of Baseline (2012): 73% delivering social protection benefits and this indicator was Target (2018): 85% services. modified at the PLR The outcome was also supported by the stage. The original First (P127754:FY14), Second indicator was specified (P143040:FY15), Third (P153234:FY16) as follows: and Fourth (P160100:FY17) DPOs which “Improvement of supported the passing of a Government means tested formula Decree to improve the targeting accuracy for enhanced targeting and coverage of the Family Benefits accuracy of FBP (i.e. proportion of Annexes CLR Review 32 Independent Evaluation Group CAS FY14-FY17: Focus Area II: Improving efficiency, equitable Actual Results IEG Comments access, and quality of health and education services Program through improvements in the beneficiary families targeting formula. that are poor) According to the 2016 Integrated Living Conditions Survey results, 73 percent of the FBP beneficiaries belonged to the bottom two quintiles compared to 74.2 percent in 2015. Not Achieved Outcome 1: Further According to the 2016 Integrated Living improvement of targeting and Conditions Survey (ILCS) results, the coverage of social assistance coverage of the poorest quintile was programs: lower in 2016 (35.3 percent compared to 38.9 in 2015). The ILCS report also noted Indicator 2: Improved coverage that only one out of 4 poor (defined as of poor households by the FBP: those below the upper poverty line) are covered by the FBP. Baseline (2012): 21% Target (2018): 27% According to data from the National * disaggregated to women- Statistics Service 14, the coverage of the headed households (increases poor by the FBP had improved from 21 from 27% to 30%) and men- percent in 2012 to 22.3 percent in 2016 headed (from 18% to 22%) (target 27 percent). Partially Achieved CAS FY14-FY17: Cross-cutting area: Improving Governance Actual Results IEG Comments and Anti-corruption measures in Public Services Objective 7: Improve the framework for fighting corruption in public services This outcome was supported by the Public Sector Modernization Projects II Outcome 1: Institutional (P117384:FY10) and III (P149913:FY16). framework for fight against The projects sought to enhance the corruption and conflict of performance of public sector management interest improved: for better service delivery by: (i) strengthening institutional capacity in Indicator 1: Global policy formulation; (ii) maximizing the Competitiveness Indicators efficiency of human resources; and (iii) (GCI) on Favoritism in decisions developing information systems for of government officials (EU27): internal work flow and external communication. As reported in its PAD, Baseline (2012): 2.99 the PSMP II initiated efforts to enhance Target (2018): 3.5 the integrity of public servants by establishing a system to track conflict of interest in public decision-making and the 14 See http://www.armstat.am/en/?nid=82&id=1988 chapter 9, section 5; and http://www.armstat.am/file/article/poverty_2012e_4.pdf, chapter 10 – table 10.6 reports the coverage of the non-poor, extreme poor and the poor. Annexes CLR Review 33 Independent Evaluation Group CAS FY14-FY17: Cross-cutting area: Improving Governance Actual Results IEG Comments and Anti-corruption measures in Public Services two projects are fully aligned with the Governments’ Public Administration Reform (PAR) Strategy pillar related to conflict of interest and ethics (see PAD for the second project). The Bank also provided NLTA support (P157003:FY17) to the Ministry of Justice and the Anti-Corruption Council to help with the implementation of Armenia's anti- corruption strategy (see the Summary of Activities which reports on the preparation of workshops for anti-corruption focal points and on the preparation of a comparative note and options paper on Ethics Regulations). According to the latest Global Competitiveness Report (2017-2018), Armenia’s score for the “Favoritism in decisions of government officials” indicator was 3.4. Mostly Achieved Outcome 1: Institutional The outcome was supported by a NLTA This indicator was framework for fight against project (P155900:FY16) funded from the introduced at the PLR corruption and conflict of Extractive Industries Technical Advisory stage. interest improved: Facility (EI-TAF). A report produced under this NLTA project was meant to be Indicator 2: Formal Government used as a key document for the of Armenia engagement in development of a national EITI work plan, Extractive as Armenia was preparing to submit their Industries Transparency Initiative candidacy application for the EITI Process. Baseline (2013): No According to the EITI website, the EITI commitment Board approved Armenia’s EITI Target (2018): Active EITI candidature application at a Board candidate meeting in March 2017. Achieved Objective 8: Strengthen the management of public resources Outcome 1: Significant This outcome was supported by the The indicators for this strengthening of internal Strengthening Public Financial outcome were based control systems in Management in Armenia grant on a PEFA score government: (P131687:FY12, see the latest available however, no PEFA ISR:S (March 2015) which did not report assessment has been Indicator 1: External Scrutiny progress at this time in relation to the undertaken since 2013. and Audit (PEFA PI-26): improved transparency and accountability The CPS baseline is of public expenditure measured by the incorrect and Baseline (2013): D+ improved quality of annual financial misleading. Armenia Annexes CLR Review 34 Independent Evaluation Group CAS FY14-FY17: Cross-cutting area: Improving Governance Actual Results IEG Comments and Anti-corruption measures in Public Services Target (2018): B reports”” indicator that was not available PEFA PI-26 was D+ in and to be known when the next PEFA 2008 (not in 2013) and would be conducted ) which aimed to had improved to C+ by improve the management of public 2013. PEFA 2013 expenditures in Armenia. results may not have been available when The outcome was also supported by a the CPS was written, Capacity Building for Public Sector and the 2016 PLR did Auditing grant (P125933:FY12) whose not reflect them. objective was to support the adoption of a capacity building strategy and action plan by the Armenian public sector external audit and the development and use of standardized audit methods. The last available ISR: S (December 2014) reported that the methodology for financial and compliance audit in line with ISSAI has been developed and dedicated staff selected and trained on its application. According to the CLR, sufficient progress has been made with financial and compliance methodology and commensurate staff training for the PI-26 to have reached a B rating in 2018. However, the CLR provides little evidence that the standards for a B rating have been met. In addition, the Region reports that the central government entities representing at least 75 percent of total expenditures are audited annually, at least covering revenue and expenditure; and that reports issued by the Chamber of Control/Chamber of Audit generally adhere to ISSAI methodology. IEG could not verify this assessment. Besides the uncertainty on actual progress made by the country on strengthening external scrutiny and audit, there is no clear indication of the Bank contribution to these possible results. Not Verified Outcome 1: Significant According to the CLR, GoA’s internal audit The baseline is strengthening of internal met the B score for PI-21 because a incorrect. Armenia control systems in functional Internal audit system in the PEFA PI-21was D+ in government: public sector; the establishment of a clear 2008 (not in 2013) and legal and regulatory base, and the had improved to C by development of a manual based on 2013. According to Annexes CLR Review 35 Independent Evaluation Group CAS FY14-FY17: Cross-cutting area: Improving Governance Actual Results IEG Comments and Anti-corruption measures in Public Services Indicator 2: Effectiveness of international standards and Public PEFA 2013, Internal internal audit (PEFA PI-21): Expenditure Management Peer Assisted audit performance had Learning (PEMPAL) network. Despite improved since 2008 Baseline (2013): D+ these achievements, the CLR provides due to the Target (2018): B little evidence that the standards for a B establishment of an rating have been met. operationally independent modern The Region reports that several systems-oriented IA milestones for a B+ rating have been function in 2012 and an achieved (internal audit is operating for the improvement in majority of central government entities, follow-up by and meet professional standards; as of management. January 1, 2018, the IA has been PEFA 2013 results may implemented and operational in 54 state not have been government bodies and 179 local self- available when the governing bodies, covering the largest CPS was written but public sector organizations in Armenia this could have been including all the ministries; reports are corrected in the 2016 issued regularly for most audited entities PLR. and distributed to the audited entity, the ministry of finance and the SAI; and prompt and comprehensive action is taken by many managers, as shown by the fact that the IA provided 6,721 recommendations during audits in 2017, out of which 5,593 have been already implemented). IEG could not verify this assessment. Besides the uncertainty on actual progress made by the country in improving the effectiveness of internal audit, there is no clear indication of the Bank contribution to these possible results over the CPS period. Not Verified Outcome 2: Increased This outcome received support from the At PLR stage, the efficiency and coherence of Tax Administration Modernization Project specification of this tax administration: (P111942:FY13) and from the Improving outcome was modified. Fiscal Management and Tax Compliance The original outcome Indicator 1: Strategic and Capacity grant (P120122: FY10). These was specified as transparent framework for projects sought to modernize Armenia’s follows, “Increased taxation outlined in new tax administration to (i) increase voluntary efficiency of tax comprehensive Tax Code tax compliance, (ii) reduce tax evasion, (iii) administration. reduce compliance costs; and (iv) increase Baseline (2012): None administrative efficiency. Target (2018): Tax Code The outcome was also supported by the adopted and implemented Second DPO 1-4 series (FY14-17) which sought to: (i) increasing tax revenue through reducing tax exemptions to reduce distortions and create more fiscal space Annexes CLR Review 36 Independent Evaluation Group CAS FY14-FY17: Cross-cutting area: Improving Governance Actual Results IEG Comments and Anti-corruption measures in Public Services for growth enhancing expenditures and (ii) to strengthen the business environment by reducing the burden of inspections on businesses. The Bank provided TA in support of reforms to the Armenia Tax Code (P156871:FY16). According to the Program document of the DPO4, the Unified Tax Code including measures to enhance revenues, improve the efficiency, transparency and equity of the tax system and strengthen tax administration was approved by the Parliament in October 2016 and was expected to come into full effect from January 2018. The Region reports that the new Tax Code of Armenia is fully effective from 1 January 2018 and that some provisions entered in effect earlier than this date. While this additional evidence indicates that the target has been met, paving the way for increased efficiency and coherence of the tax administration, actual outcomes will depend on sustained implementation. On the other hand, the Region reports that with the support of the ongoing Bank- financed Tax Administration Modernization Project (TAMP), the efficiency of the tax administration has already improved between 2012 and 2017 in many areas: (i) taxation as a percentage of GDP grew from 17.6 percent to 20.8 percent; (ii) corporate income tax productivity has increased from 0.01 in 2011 to 0.13 in 2016; (iii) the ratio of collected revenue to SRC’s operation cost increased from 74 in 2011 to 101.8 in 2017; (iv) the share of e- returns increased from 20 percent in 2011 to the current 96.61 percent, a level which is on par with many OECD countries; and (v) the time spent preparing and paying taxes decreased according to Doing Business from 500 hours in 2011 to 313 in 2017. Mostly Achieved Annexes CLR Review 37 Independent Evaluation Group Annex Table 2: Armenia IBRD/IDA Planned and Actual Lending, FY14-FY17 Approved Project Proposed Approval Closing Proposed Proposed Outcome Project name IBRD/IDA ID FY FY FY Amount Amount Rating Amount Project Planned Under CPS/PLR 2012-2016 CPS PLR Development Policy Operation 1 P127754 2014 2014 2014 72.0 72.0 72.0 New Series P130182 Education Improvement Project 2014 2014 2020 30.0 30.0 30.0 LIR:S Second Community Agriculture P133705 Resource Management and 2014 2014 2020 30.0 30.0 32.7 LIR:S Competitiveness Project P146318 Social Protection Administration II 2014 2014 2019 21.5 21.5 21.2 LIR:MS Second Development Policy P143040 2015 2015 2015 50.0 75.0 75.0 Operation Power Transmission P146199 2015 2015 2020 50.0 52.0 52.0 LIR:S Improvement Project Trade Promotion and Quality P146994 2015 2015 2020 40.0 40.0 50.0 LIR:MU Infrastructure Electricity Supply Reliability P148102 2015 2015 2019 30.0 50.0 40.0 (Additional Financing) Social Investment and Local P148836 2015 2015 2020 30.0 30.0 30.0 LIR:MS Development Public Sector Modernization P149913 2016 2016 2021 20.0 20.0 21.0 LIR:MS Project III Local Economy and P150327 2016 2016 2021 55.0 55.0 55.0 LIR:MS Infrastructure Develop Lifeline Road Network P150505 Improvement (Additional 2016 2016 2020 40.0 40.0 40.0 Financing) Third Development Policy P153234 2016 2016 2015 50.0 50.0 50.0 Operation Power Sector Financial Recovery P157571 2016 2016 2022 32.0 30.0 LIR:S Program Armenia Fourth Development P160100 2017 2017 2018 60.0 50.0 50.0 Policy Financing P131812 Long Term Export Finance Total Planned 578.5 647.5 648.9 Unplanned Projects during the CPS and PLR Period Approved On-going Projects during the CPS and Proposed Approval Closing Proposed Proposed Outcome IDA PLR Period FY FY FY Amount Amount Rating Amount P094225 Social Investment Fund III 2007 2015 25.0 IEG:S Health System Modernization P104467 Project (Adaptable Program 2007 2016 22.0 IEG:S Loan II) Second Education Quality and P107772 Relevance (Adaptable Program 2009 2016 25.0 IEG:MS Loan II) Lifeline Roads Improvement P115486 2009 2014 25.0 IEG:S Project Social Investment Fund III P115743 2009 2015 8.0 (Additional Financing I) Lifeline Roads Improvement P116760 2010 2014 36.6 Project (Additional Financing) Annexes CLR Review 38 Independent Evaluation Group Public Sector Modernization P117384 2010 2017 9.0 LIR:MS Project II Social Investment Fund III P118158 2010 2015 7.0 (Additional Financing II) P115647 E-Society & Innovation 2011 2016 24.0 IEG:S P116748 Electricity Supply Reliability 2011 2019 39.0 LIR:MS Community Agricultural P120028 Resource Management and 2011 2017 16.0 IEG:S Competitiveness Project Lifeline Roads Improvement P121287 2011 2014 40.0 Project (Additional Financing II) Health System Modernization P121728 2011 2016 19.0 Project (Additional Financing) P126722 Municipal Water Project 2012 2015 15.0 IEG:MS Social Investment Fund III P128740 2012 2015 11.0 (Additional Financing III) Tax Administration P111942 2013 2018 12.0 LIR:S Modernization Project Lifeline Road Network P126782 2013 2020 45.0 LIR:MS Improvement Project Irrigation System Enhancement P127759 2013 2019 30.0 LIR:S Project Disease Prevention & Control P128442 2013 2020 35.0 LIR:MS Project Total On-going 443.6 Source: Armenia CPS and PLR, WB Business Intelligence Table 2b.1, 2a.4 and 2a.7 as of 01/23/18 *LIR: Latest internal rating. MU: Moderately Unsatisfactory. MS: Moderately Satisfactory. S: Satisfactory. HS: Highly Satisfactory. Annex Table 3: Advisory Services and Analytics Work for Armenia, FY14-FY17 Fiscal Proj ID Economic and Sector Work Output Type Global Practice year Macroeconomics, Trade and P129140 Armenia MTDS Baseline FY14 Sector or Thematic Study/Note Investment Programmatic Fiscal Work Public Expenditure Review Macroeconomics, Trade and P133400 FY14 FY13 (PER) Investment Armenia: PEFA Assessment Public Expenditure Financial P143277 FY14 Governance (Sub-National) Accountability Macroeconomics, Trade and P131912 Armenia Policy Notes FY15 Sector or Thematic Study/Note Investment Armenia Power Sector Policy P133834 FY15 Sector or Thematic Study/Note Energy & Extractives Note Country Economic Macroeconomics, Trade and P147949 Drivers of Dynamism FY15 Memorandum (CEM) Investment Armenia Public Expenditure Public Expenditure Review Macroeconomics, Trade and P151948 FY16 Review (PER) Investment Power Sector Financial P154276 FY16 Sector or Thematic Study/Note Energy & Extractives Recovery Plan Agriculture Commercialization P157524 FY16 Sector or Thematic Study/Note Agriculture Study Armenia Corridor Tourism Social, Urban, Rural and P158280 FY16 Sector or Thematic Study/Note Development Resilience Global Practice Gender Dynamics in STEM P157930 FY17 Sector or Thematic Study/Note Education Fields Annexes CLR Review 39 Independent Evaluation Group Fiscal Proj ID Technical Assistance Output Type Global Practice year Armenia #10189 Strength. Finance, Competitiveness and P130211 FY14 Technical Assistance Consumer Protec Innovation Update of Water Resource P132705 FY14 Technical Assistance Water Management Note Armenia Environment Sector Environment & Natural P132911 FY14 Technical Assistance Note Resources Armenia#10313 Supporting Finance, Competitiveness and P144563 FY14 Technical Assistance Capital Market Innovation Social Accountability Rural Social, Urban, Rural and P145602 FY14 Technical Assistance Services Resilience Global Practice Corp Insolvency and P146995 FY14 Technical Assistance Trade & Competitiveness Restructuring TA Armenia#10232Enhancing P144751 FY15 Technical Assistance Governance Corp. Fin. Report Armenia #10316Regulatory Finance, Competitiveness and P144817 FY15 Technical Assistance and Supervisory Innovation Export Development Strategy P145530 FY15 Technical Assistance Trade & Competitiveness Implement Financial Strengthening of P146342 FY15 Technical Assistance Water Operating Wat ECU Challenges and Macroeconomics, Trade and P150676 FY15 Technical Assistance Opportunities Investment Armenia Mining Support P152162 FY15 Technical Assistance Energy & Extractives Development Water Resources P152862 FY15 Technical Assistance Water Management P151048 SILK ROAD TA Armenia FY16 Technical Assistance Social Protection & Labor P152237 Armenia Pensions FY16 Technical Assistance Social Protection & Labor Armenia EI-TAF Mining P155900 FY16 Technical Assistance Energy & Extractives Sector Support Armenia Tax Code and Policy Macroeconomics, Trade and P156871 FY16 Technical Assistance Reform TA Investment Armenia Trade Comp. and P157037 FY16 Technical Assistance Trade & Competitiveness EEU Integration Armenia Regional Social, Urban, Rural and P158284 FY16 Technical Assistance Development Analysis Resilience Global Practice Social, Urban, Rural and P145315 Vulnerability Assessment FY17 Technical Assistance Resilience Global Practice Disaster Resilient Urban Social, Urban, Rural and P153192 FY17 Technical Assistance Planning Resilience Global Practice Armenia Financial Advisory Finance, Competitiveness and P156026 FY17 Technical Assistance TA Innovation Implementing Anti-Corruption P157003 FY17 Technical Assistance Governance Strategy Armenia Second Generation P158679 FY17 Technical Assistance Water Water PPPs Environmental Governance in Environment & Natural P158945 FY17 Technical Assistance Mining Resources Armenia Universal Health P159613 FY17 Technical Assistance Health, Nutrition & Population Coverage Source: WB Business Intelligence 01/23/18 Annexes CLR Review 40 Independent Evaluation Group Annex Table 4: Armenia Grants and Trust Funds Active in FY14-17 Project Approval Closing Approved Outcome Project name TF ID ID FY FY Amount Rating P116680 Energy Efficiency Project TF 12163 2012 2016 1.8 IEG: HS Improving Fiscal Management and Tax P120122 Compliance Capacity in the Government TF 96224 2010 2014 0.5 LIR: MS of Armenia P120210 Implementation of IPSAS Strategy 2 TF 96227 2010 2014 0.5 LIR: S Strengthening National Assembly's P120238 Capacity for Policy Formulation and TF 96229 2011 2014 0.5 LIR: MU Public Dialogue JSDF Strengthening the Livelihoods and P125791 Voice of Vulnerable and Disabled TF 13894 2013 2017 2.7 LIR: MS Persons in Armenia Capacity Building for Public Sector P125933 TF 11901 2013 2016 0.5 LIR: S Auditing P125966 Ministry of Finance Capacity Building TF 11907 2013 2016 0.3 LIR: S P128442 Disease Prevention and Control Project TF 13103 2013 2016 0.4 LIR: MS P128442 Disease Prevention and Control Project TF 14138 2014 2019 1.8 LIR: MS Strengthening Capacity of Regulatory P128769 Reform Unit ot the Armenian TF 11972 2013 2016 0.5 LIR: S Government P129168 Food Safety Capacity Building IDF TF 12224 2013 2016 0.5 LIR: MS P130182 Education Improvement Project TF 14539 2013 2015 0.4 LIR: S Pension Public Awareness and Literacy P131003 TF 13654 2013 2016 0.4 LIR: S IDF Grant Project Electronic Processing and One-Stop P131036 TF 12782 2013 2016 0.5 LIR: MS Shop for Construction Permits Strengthening Training Capacity of the P131126 TF 13656 2013 2016 0.5 LIR: S School of Advocates Reports to Conventions - UNCCD - P131631 TF 13195 2013 2015 0.2 Armenia Strengthening Public Financial P131687 TF 12529 2013 2016 0.9 LIR: S Management in Armenia Armenia Pilot Agricultural Census P132948 TF 13918 2013 2015 0.2 implementation support Environmental Governance, Inclusion P133439 & Transparency in Mining Sector TF 13658 2013 2016 0.4 IDF Grant Project Second Community Agriculture Resource P133705 Management and Competitiveness TF 15171 2014 2015 0.5 LIR: S Project P133831 Preparation of SREP Investment Plan TF 14245 2013 2014 0.3 LIR: S Revision of the NBSAP preparation of the P143212 TF 13529 2013 2016 0.2 5th National Report to the CBD Community Agricultural Resource P144283 Management and Competitiveness TF 13724 2013 2017 0.9 Project Electricity Transmission Network P146199 TF 16604 2014 2015 0.7 LIR: S Improvement Project Annexes CLR Review 41 Independent Evaluation Group Project Approval Closing Approved Outcome Project name TF ID ID FY FY Amount Rating P147919 EDB Irrigation pr preparation TF 17003 2014 2015 0.5 PROMOTING YOUTH INCLUSION IN P149463 TF 16980 2015 2016 1.0 LIR: MU ARMENIA Local Economy and Infrastructure P150327 TF A1063 2016 2017 0.7 LIR: MS Development Project P152039 Geothermal Exploratory Drilling Project TF 18000 2015 2015 0.3 LIR: MS P152039 Geothermal Exploratory Drilling Project TF A0544 2015 2019 8.6 LIR: MS Implementation of the National Strategy P155630 Program for Strengthening of the TF A4543 2017 2021 3.0 LIR: MS National Statistical System P155662 Utility Scale Solar Power Project TF A0418 2015 2018 2.0 Social Investment and Local P156378 TF A3230 2017 2019 2.4 LIR: S Development Trust Fund Eurasian Development Bank Mastara P158899 TF A2706 2017 2018 0.7 Reservoir Preparation Strengthening Armenia's P159054 TF A3852 2017 2019 0.2 LIR: S Integrated Living Conditions Survey P158359 MOA Ag Policy Capacity Building TF A4449 2017 2019 1.8 LIR: S Total 37.0 Source: Client Connection as of 01/23/18 ** IEG Validates RETF that are 5M and above Annex Table 5: IEG Project Ratings for Armenia, FY14-17 Total Exit IEG Risk to Proj ID Project name Evaluated IEG Outcome FY DO ($M) LIFELINE ROADS IMPROVEMENT 2014 P115486 101.5 SATISFACTORY SIGNIFICANT PROJECT 2015 P094225 SIF 3 51.9 SATISFACTORY MODERATE MODERATELY 2015 P126722 MUNICIPAL WATER 14.9 HIGH SATISFACTORY 2016 P104467 HLTH SYS MOD (APL2) 41.9 SATISFACTORY LOW MODERATELY 2016 P107772 EDUC QUAL & REL (APL#2) 24.8 MODERATE SATISFACTORY 2016 P115647 E-SOCIETY & INNOVATION 23.3 SATISFACTORY LOW 2016 P116680 ENERGY EFFICIENCY 0.0 HIGHLY SATISFACTORY MODERATE Community Agri. Res. Managem. and 2017 P120028 15.3 SATISFACTORY MODERATE Comp. Total 273.6 Source: Business Intelligence Key IEG Ratings as of 01/23/18 Annexes CLR Review 42 Independent Evaluation Group Annex Table 6: IEG Project Ratings for Armenia and Comparators, FY14-17 Total Total RDO % RDO % Outcome Outcome Region Evaluated Evaluated Moderate or Lower Moderate or Lower % Sat ($) % Sat (No) ($M) (No) Sat ($) Sat (No) Armenia 273.6 8.0 100.0 100.0 33.6 50.0 ECA 13,280.0 117.0 93.3 79.5 64.1 51.7 World 73,143.8 792.0 85.4 74.2 52.7 43.5 Source: WB Business Intelligence as of 01/24/18 Annex Table 7: Portfolio Status for Armenia and Comparators, FY14-17 Fiscal year 2014 2015 2016 2017 Ave FY14-17 Armenia # Proj 25.0 24.0 20.0 22.0 22.8 # Proj At Risk 1.0 3.0 1.0 1.7 % Proj At Risk 4.2 15.0 4.5 7.9 Net Comm Amt 435.1 548.4 649.6 629.3 565.6 Comm At Risk 79.0 152.2 50.0 93.7 % Commit at Risk 14.4 23.4 7.9 15.3 ECA # Proj 280.0 290.0 279.0 292.0 285.3 # Proj At Risk 37.0 36.0 47.0 37.0 39.3 % Proj At Risk 13.2 12.4 16.8 12.7 13.8 Net Comm Amt 26,927.9 26,544.5 27,637.3 25,808.5 26,729.5 Comm At Risk 2,635.4 3,533.8 4,350.5 5,466.2 3,996.5 % Commit at Risk 9.8 13.3 15.7 21.2 15.0 World # Proj 2,048.0 2,022.0 1,975.0 2,072.0 2,029.3 # Proj At Risk 412.0 444.0 422.0 449.0 431.8 % Proj At Risk 20.1 22.0 21.4 21.7 21.3 Net Comm Amt 192,610.1 201,045.2 220,331.5 224,458.9 209,611.4 Comm At Risk 40,933.5 45,987.7 44,244.9 52,549.1 45,928.8 % Commit at Risk 21.3 22.9 20.1 23.4 21.9 Source: WB BI as of 01/24/18 Annexes CLR Review 43 Independent Evaluation Group Annex Table 8: Disbursement Ratio for Armenia, FY14-17 Fiscal Year 2014 2015 2016 2017 Overall Result Armenia Disbursement Ratio (%) 26.0 20.0 19.5 15.2 19.4 Inv Disb in FY 59.1 48.7 71.4 61.9 241.1 Inv Tot Undisb Begin FY 227.5 243.0 366.1 407.8 1,244.4 ECA Disbursement Ratio (%) 22.7 23.6 17.3 20.6 20.9 Inv Disb in FY 2,585.5 2,633.7 2,218.3 2,781.0 10,218.4 Inv Tot Undisb Begin FY 11,375.8 11,173.5 12,787.9 13,506.4 48,843.6 World Disbursement Ratio (%) 20.2 21.2 18.8 19.8 20.0 Inv Disb in FY 19,414.2 20,317.9 19,401.1 20,572.0 79,705.2 Inv Tot Undisb Begin FY 96,254.9 95,816.0 103,447.2 103,732.6 399,250.6 * Calculated as IBRD/IDA Disbursements in FY / Opening Undisbursed Amount at FY. Restricted to Lending Instrument Type = Investment. Business Intelligence disbursement ratio table as of 01/24/18 Annex Table 9: Net Disbursement and Charges for Armenia, FY14-17 ($ millions) Period Disb. Amt. Repay Amt. Net Amt. Charges Fees Net Transfer FY14 131.2 19.7 111.4 2.9 9.7 98.8 FY15 124.0 22.9 101.1 10.1 9.8 81.2 FY16 121.7 27.7 94.0 19.2 9.6 65.2 FY17 137.0 34.1 102.9 24.6 9.6 68.8 Report Total 513.8 104.5 409.4 56.8 38.6 314.0 World Bank Client Connection 01/24/18 Annexes CLR Review 44 Independent Evaluation Group Annex Table 10: Total Net Disbursements of Official Development Assistance Development Partner 2013 2014 2015 2016 Australia .. .. .. 0.03 Austria 2.86 5.22 3.77 1.7 Belgium 0.21 0.24 0.23 0.21 Canada 0.09 .. 0.04 0.1 Czech Republic 1.13 0.39 0.29 0.41 Denmark 2.86 0.59 0.76 1.82 Finland 0.11 0.29 0.24 0.22 France 11.11 17.19 20.5 81.86 Germany 28.47 31.84 8.98 27.59 Greece 0.35 0.36 0.3 0.05 Hungary 0.01 0.01 0.07 0.02 Italy 0.11 0.38 0.31 0.59 Japan 1.33 1.69 -5.31 -5.67 Korea .. 0.03 0.16 0.15 Netherlands 0.18 0.21 0.07 0.22 Norway 2.95 1.56 0.56 0.27 Poland 0.57 0.33 0.35 0.17 Portugal .. .. .. 0 Slovak Republic 0.07 0.02 0.05 0.01 Spain 0.07 0.03 0 0.52 Sweden 0.08 0.97 1.21 0.6 Switzerland 4.16 5.44 6.48 7.28 United Kingdom 1.29 1.6 1.81 0.59 United States 40.48 30.48 24.98 25.08 DAC Countries, Total 98.49 98.87 65.85 143.82 Estonia 0.12 0.18 0.04 .. Israel 0.06 0.04 0.01 0.17 Kuwait .. .. .. 0.97 Latvia 0.01 0.02 0.01 .. Lithuania 0.01 0.01 0.01 0.04 Romania 0.12 0.11 0.21 0.15 Russia 5.26 6.86 37.37 40.33 Turkey 0.26 0.75 .. 0.02 United Arab Emirates 0.89 0.61 0.67 0.23 Non-DAC Countries, Total 6.73 8.58 38.32 41.91 EU Institutions 69.67 79.23 148.96 111.31 International Monetary Fund, Total 40.43 IMF (Concessional Trust Funds) 40.43 .. .. .. Annexes CLR Review 45 Independent Evaluation Group Development Partner 2013 2014 2015 2016 Regional Development Banks, Total 21.67 79.62 88.72 24.04 Asian Development Bank, Total 21.67 79.62 88.72 24.04 AsDB Special Funds 21.67 79.62 88.72 24.04 United Nations, Total 3.22 2.78 5.64 8.94 Food and Agriculture Organisation [FAO] 0.71 .. .. .. International Atomic Energy Agency [IAEA] .. .. 0.14 0.13 IFAD .. .. 2.86 6.26 International Labour Organisation [ILO] 0.10 0.21 0.18 0.11 UNAIDS 0.06 .. .. .. UNDP 0.94 0.72 0.48 0.61 UNFPA 0.60 0.49 0.47 0.50 UNICEF 0.69 1.16 1.14 0.98 World Health Organisation [WHO] 0.13 0.20 0.37 0.35 World Bank Group, Total 56.15 17.10 15.28 17.88 World Bank, Total 56.15 17.10 15.28 17.88 International Bank for Reconstruction and .. .. .. .. Development [IBRD] International Development Association [IDA] 56.15 17.10 15.28 17.88 Other Multilateral, Total 19.08 15.41 17.82 31.33 Global Alliance for Vaccines and Immunization [GAVI] 0.80 0.68 0.37 1.11 Global Environment Facility [GEF] 2.68 4.15 3.84 4.20 Global Fund 7.65 5.11 2.82 4.82 Nordic Development Fund [NDF] .. .. .. .. OPEC Fund for International Development [OFID] 4.66 2.10 7.88 18.22 OSCE 3.29 3.37 2.91 2.97 Multilateral Agencies, Total 210.2 194.2 276.4 193.5 Development Partners, Total 315.4 301.6 380.6 379.2 Source: OECD Stat database as of 02/14/2018 2017 data not yet available. Annexes CLR Review 46 Independent Evaluation Group Annex Table 11: Economic and Social indicators for Armenia, 2013-2016 Armenia ECA World Series Name 2014 2015 2016 2017 Average 2014-2017 Growth and Inflation GDP growth (annual 3.6 2.3 2.8 %) 3.6 3.2 0.2 7.5 GDP per capita growth 3.3 1.8 1.6 (annual %) 3.1 2.8 (0.1) 7.3 GNI per capita, PPP 9,232.5 19,819.5 16,012.3 (current international $) 8,790.0 9,090.0 9,020.0 10,030.0 GNI per capita, Atlas 3,985.0 8,898.6 10,563.1 method (current $) 4,150.0 4,030.0 3,770.0 3,990.0 Inflation, consumer 1.6 2.4 1.8 prices (annual %) 3.0 3.7 (1.4) 1.0 Composition of GDP (%) Agriculture, value 16.7 5.0 3.6 added (% of GDP) 18.1 17.2 16.4 14.9 Industry, value added 25.5 29.0 25.9 (% of GDP) 25.3 25.7 25.6 25.3 Services, value added 49.2 54.7 64.5 (% of GDP) 47.4 48.2 49.9 51.3 Gross fixed capital 18.8 22.7 23.4 formation (% of GDP) 20.0 20.6 17.4 17.3 External Accounts Exports of goods and 32.2 33.6 31.2 services (% of GDP) 28.6 29.7 33.1 37.3 Imports of goods and 45.3 31.9 30.2 services (% of GDP) 47.0 41.9 42.8 49.5 Current account -3.7 . . balance (% of GDP) (7.6) (2.6) (2.3) (2.4) External debt stocks 82.5 . . (% of GNI) 70.4 81.2 92.4 85.9 Total debt service (% 13.0 8.2 . of GNI) 11.8 14.2 13.8 12.1 Total reserves in 4.2 8.5 13.0 months of imports 2.9 4.4 5.1 4.4 Fiscal Accounts /1 General government 21.5 revenue (% of GDP) 22.0 21.5 21.4 21.2 General government total expenditure (% of 25.8 24.0 26.3 27.0 26.0 GDP) General government net lending/borrowing -4.3 (1.9) (4.8) (5.6) (4.8) (% of GDP) General government 47.2 gross debt (% of GDP) 39.3 44.1 51.9 53.5 Health Life expectancy at .. 74.4 73.0 71.9 birth, total (years) 74.3 74.4 74.6 Immunization, DPT (% of children ages 12-23 93.8 91.6 85.1 93.0 94.0 94.0 94.0 months) Annexes CLR Review 47 Independent Evaluation Group Armenia ECA World Series Name 2014 2015 2016 2017 Average 2014-2017 People using safely managed sanitation .. .. .. .. . 38.9 services (% of pop) People using at least basic drinking water .. .. 98.7 96.1 88.3 98.5 98.9 services (% of pop) Mortality rate, infant 12.2 12.0 30.8 (per 1,000 live births) 13.1 12.5 11.9 11.3 Education School enrollment, 51.3 59.1 48.1 preprimary (% gross) 49.5 51.9 51.5 52.0 School enrollment, 95.0 100.7 103.5 primary (% gross) 96.7 94.8 94.3 94.2 School enrollment, .. .. 86.0 99.0 76.5 secondary (% gross) 85.9 86.0 School enrollment, 49.2 65.9 36.9 tertiary (% gross) 47.1 46.5 51.1 52.2 Pop population, total 2,906,220.0 2,916,950.0 2,924,816.0 2,930,450.0 2,919,609.0 454,127,420.5 7,400,849,944.0 population growth 0.3 0.5 1.2 (annual %) 0.4 0.4 0.3 0.2 Urban population (% of 63.1 65.9 54.1 total) 63.1 63.1 63.1 63.1 Rural population (% of 36.9 34.1 45.9 total pop) 36.9 36.9 36.9 36.9 Poverty Poverty headcount ratio at $1.90 a day .. 2.0 . 10.0 2.3 1.9 1.8 (2011 PPP) (% of pop) Poverty headcount ratio at national poverty .. 29.7 . . 30.0 29.8 29.4 lines (% of pop) Rural poverty headcount ratio at .. .. .. 29.9 . . national poverty lines 29.9 (% of rural pop) Urban poverty headcount ratio at .. .. .. 30.0 . . national poverty lines 30.0 (% of urban pop) GINI index (World .. 32.1 . . Bank estimate) 31.5 32.4 32.5 Source: WB World Development Indicators DataBank 1/30/19 *International Monetary Fund, World Economic Outlook Database, October 2018 . Annexes CLR Review 48 Independent Evaluation Group Annex Table 12: List of IFC Investments in Armenia Investments Committed in FY14-FY17 Project Cmt Project Project Original Original Original Loan Equity Net Net Primary Sector Name Net Comm ID FY Status Size Loan Equity CMT Cancel Cancel Loan Equity 37652 2017 Active Electric Power 122,900 47,900 - 47,900 - - 47,900 - 47,900 38349 2017 Active Finance & Insurance 300 300 - 300 - - 300 - 300 33450 2016 Active Electric Power 200,700 9,009 4,504 13,513 - - 13,513 4,504 13,513 37196 2016 Active Finance & Insurance 20,000 20,000 - 20,000 - - 20,000 - 20,000 37259 2016 Active Finance & Insurance 60,000 50,000 - 50,000 - - 50,000 - 50,000 37540 2016 Active Finance & Insurance 300 300 - 300 - - 300 - 300 37697 2016 Active Finance & Insurance 200 200 - 200 - - 200 - 200 33520 2015 Closed Finance & Insurance 5,000 5,000 - 5,000 - - 5,000 - 5,000 35418 2015 Active Finance & Insurance 3,000 3,000 - 3,000 - - 3,000 - 3,000 35794 2015 Active Finance & Insurance 300 300 - 300 - - 300 - 300 36555 2015 Closed Oil, Gas and Mining 1,288 - 1,288 1,288 - 207 1,288 1,081 1,081 Construction and Real 32970 2014 Active 38,200 9,000 - 9,000 - - 9,000 - 9,000 Estate 33031 2014 Active Finance & Insurance 5,000 5,000 - 5,000 - - 5,000 - 5,000 33518 2014 Closed Finance & Insurance 15,000 15,000 - 15,000 15,000 - - - - 33521 2014 Closed Finance & Insurance 5,000 5,000 - 5,000 - - 5,000 - 5,000 33625 2014 Closed Finance & Insurance 4,000 4,000 - 4,000 - - 4,000 - 4,000 34881 2014 Closed Oil, Gas and Mining 1,628 - 1,628 1,628 - - 1,628 1,628 1,628 35024 2014 Closed Finance & Insurance 2,500 2,500 - 2,500 2,500 - - - - Sub-Total 485,316 176,509 7,421 183,929 17,500 207 166,429 7,214 166,223 Annexes CLR Review 49 Independent Evaluation Group Investments Committed pre-FY14 but active during FY14-17 Project CMT Project Project Original Original Original Loan Equity Net Net Net Primary Sector Name ID FY Status Size Loan Equity CMT Cancel Cancel Loan Equity Comm 32799 2013 Active Food & Beverages 5,500 2,500 - 2,500 - - 2,500 - 2,500 33720 2013 Active Finance & Insurance 1,000 1,000 - 1,000 - - 1,000 - 1,000 31610 2012 Active Finance & Insurance 10,000 10,000 - 10,000 5,000 - 5,000 - 5,000 27497 2011 Active Finance & Insurance - 3,006 - 3,006 - - 3,006 - 3,006 30028 2011 Active Finance & Insurance 2,000 2,000 - 2,000 913 - 1,087 - 1,087 30857 2011 Active Finance & Insurance 2,000 2,000 - 2,000 - - 2,000 - 2,000 28241 2010 Active Finance & Insurance 25,400 111,159 - 111,159 - - 111,159 - 111,159 Construction and Real 28826 2010 Active 5,400 5,400 - 5,400 - - 5,400 - 5,400 Estate 24572 2006 Active Finance & Insurance 4,400 3,000 1,300 4,300 - - 4,300 1,300 4,300 Sub-Total 55,700 140,066 1,300 141,366 5,913 - 135,453 1,300 135,453 TOTAL 541,016 316,574 8,721 325,295 23,413 207 301,882 8,514 301,675 Source: IFC-MIS Extract as of 8/30/2017 Annexes CLR Review 50 Independent Evaluation Group Annex Table 13: List of IFC Advisory Services in Armenia Advisory Services Approved in FY14-17 Primary Project Impl Impl Project Total Project Name Business ID Start FY End FY Status Funds, $ Line 601359 ECA Gender Project 2018 2021 ACTIVE TAC 520,000 601945 Armenia IPP/Agribusiness Project 2018 2021 ACTIVE TAC 1,350,000 596467 Armenia Oncology 2015 2015 TERMINATED CAS 55,775 600244 Armenia Investment Climate II 2015 2018 ACTIVE TAC 1,508,451 Sub-Total 3,434,226 Advisory Services Approved pre-FY14 but active during FY14-17 Primary Project Impl Impl Project Total Project Name Business ID Start FY End FY Status Funds, $ Line 585987 Armenia Food Safety Project 2012 2015 CLOSED MAS 427,794 Armenia Investment Climate 588147 2012 2015 CLOSED TAC 1,606,702 Reform Project Armenia Sustainable Energy 568727 2010 2016 CLOSED FIG 1,052,342 Finance Sub-Total 3,086,838 TOTAL 6,521,064 Source: IFC AS Data as of 10-17-17 Annex Table 14: IFC net commitment activity in Armenia, FY14 - FY17 2014 2015 2016 2017 Total Commercial Financial Markets 5,000,000 - 67,009,027 - 72,009,027 Banking Housing Finance - (5,000,000) - - (5,000,000) Microfinance 6,500,000 3,300,000 (2,200,000) 300,000 7,900,000 TCF (Trade and 20,000,000 (10,000,000) - - 10,000,000 Commodity) Trade Finance (TF) GTFP 6,904,986 13,636,247 7,700,000 4,332,464 32,573,696 Tourism, Retail, Commercial Construction & Real 9,000,000 - - - 9,000,000 Property Estates (TRP) Oil, Gas & Mining Mining 1,543,175 1,300,248 (9,157) (781) 2,833,485 Infrastructure Electric Power - - 13,513,290 47,900,000 61,413,290 Total 48,948,161 3,236,494 86,013,160 52,531,683 190,729,498 Source: IFC MIS as of 2/1/18 Annex Table 15: List of Active MIGA Activities in Armenia, 2014-2017 Max Gross ID Contract Enterprise FY Project Status Sector Investor Issuance 13661 Vorotan Hydro Cascade 2017 Proposed Power To be determined 90.0 ProCredit Group Central Bank 9983 2011 Active Banking Germany 3.7 Mandatory Reserves Coverage Total 93.7 Source: MIGA 1-31/18