FOR OFFICIAL USE ONLY Report No: ICR00004836 INTERNATIONAL DEVELOPMENT ASSOCIATION IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-53140-PK ON A CREDIT IN THE AMOUNT OF SDR32.6 MILLION (US$50 MILLION EQUIVALENT) TO THE ISLAMIC REPUBLIC OF PAKISTAN FOR A PUNJAB PUBLIC MANAGEMENT REFORM PROGRAM PROGRAM-FOR-RESULTS July 17, 2019 Equitable Growth, Finance and Institutions (EFI) Governance Global Practice South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 30, 2018) Currency Unit = Pakistan Rupee (PKR) PKR139.90 = US$1 1.39US$ = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Hartwig Schafer Country Director: Patchamuthu Illangovan Global Practice Director: Edward Olowo-Okere Regional Practice Director: Zoubida Kherous Allaoua Practice Manager: Ismaila Ceesay Task Team Leader(s): Clelia Kalliopi Helena Rontoyanni, Irum Touqeer ICR Main Contributor: Immanuel Frank Steinhilper ABBREVIATIONS AND ACRONYMS ACG Anti-Corruption Guidelines PFM Public Financial Management AGP Auditor General of Pakistan PDO Program Development Objective Public Expenditure and Financial BCA Benefit/Cost Analysis PEFA Accountability CCC Citizen Contact Center PforR Program for Results CPS Country Partnership Strategy PITB Punjab Information Technology Board UK Department of International Project Management and Implementation DFID PMIU Development Unit DLIs Disbursement Linked Indicators PPMR Punjab Public Management Reform Punjab Public Management Reform DPF Development Policy Financing PPMRP Program E&TD Excise and Taxation Department PPP Public Private Partnership Excise Taxation and Narcotics Control ETNCD PPRA Public Procurement Regulatory Authority Department FY Financial Year PAP Program Action Plan GoPb Government of Punjab PRMP Punjab Resource Management Program GDP Gross Domestic Product RTI Right to Information Implementation Completion ICR SDR Special Drawing Rights Report ICT Information and Communication SMS Short Messaging Service Technology IDA International Development Association SNGP Sub-National Governance Program IPF Investment Project Financing TPV Third Party Validation/Verification ISR Implementation Status and Results TTL Task Team Leader IT Information Technology UIPT Urban Immoveable Property Tax KPIs Key Performance Indicators WBG World Bank Group M&E Monitoring and Evaluation MIS Management Information System NPV Net present value OSR Own Source Revenue P&D Planning and Development PAD Project Appraisal Document PAP Program Action Plan TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES.................................................... 6 A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE .................................................................6 B. CHANGES DURING IMPLEMENTATION ................................................................................... 11 II. OUTCOME .................................................................................................................... 12 A. RELEVANCE .......................................................................................................................... 12 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 13 C. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 16 D. OTHER OUTCOMES AND IMPACTS ........................................................................................ 17 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17 A. KEY FACTORS DURING PREPARATION ................................................................................... 17 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 18 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 20 A. QUALITY OF MONITORING AND EVALUATION ....................................................................... 20 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 21 C. BANK PERFORMANCE ........................................................................................................... 22 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 24 V. LESSONS AND RECOMMENDATIONS ............................................................................. 24 ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN ...................................................................................................................... 27 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROGRAM EXPENDITURE SUMMARY ................................................................... 46 ANNEX 4. BORROWER’S COMMENTS ................................................................................... 47 ANNEX 5. ECONOMIC AND FINANCIAL ANALYSIS ................................................................. 49 ANNEX 5. SUPPORTING DOCUMENTS .................................................................................. 54 ANNEX 6. RESULTS CHAIN/THEORY OF CHANGE ................................................................... 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) DATA SHEET BASIC INFORMATION Product Information Program ID Program Name Financing Instrument Pakistan: Punjab Public Management P132234 Program-for-Results Financing Reform Program Country IPF Component Pakistan No Organizations Borrower Implementing Agency Punjab Resource Management Program, Excise & Planning and Development Department Taxation Department, Punjab Information Technology Board, Punjab Public Procurement Regulatory Authority Program Development Objective (PDO) Original PDO To improve transparency and resource management of targeted departments of the Province of Punjab. Page 1 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) FINANCING FINANCE_T BL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Administered Financing 50,000,000 50,000,000 46,291,178 IDA-53140 Total 50,000,000 50,000,000 46,291,178 Non-World Bank Administered Financing Borrower/Recipient 0 61,700,000 61,700,000 Total 0 61,700,000 61,700,000 Total Program Cost 50,000,000 111,700,000 107,991,178 KEY DATES Program Approval Effectiveness MTR Review Original Closing Actual Closing P132234 14-Nov-2013 10-Jan-2014 11-May-2016 31-Dec-2018 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 12-Jul-2017 38.35 Change in Results Framework Reallocation between and/or Change in DLI Other Change(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROGRAM PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 26-Feb-2014 Satisfactory Satisfactory 0 Page 2 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 02 21-Sep-2014 Satisfactory Moderately Satisfactory 5.02 03 10-Apr-2015 Moderately Satisfactory Moderately Satisfactory 5.02 04 16-Oct-2015 Moderately Satisfactory Moderately Satisfactory 20.92 05 12-Apr-2016 Moderately Satisfactory Moderately Satisfactory 20.92 06 25-Oct-2016 Satisfactory Satisfactory 30.03 07 10-Apr-2017 Satisfactory Satisfactory 30.03 08 22-Jan-2018 Satisfactory Satisfactory 38.35 09 07-Sep-2018 Satisfactory Satisfactory 46.29 10 15-Jan-2019 Highly Satisfactory Satisfactory 46.29 SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 100 Other Public Administration 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 100 Public Finance Management 20 Public Expenditure Management 20 Public Administration 80 Transparency, Accountability and Good 80 Governance ADM STAFF Role At Approval At ICR Regional Vice President: Annette Dixon Hartwig Schafer Country Director: Patchamuthu Illangovan Patchamuthu Illangovan Senior Global Practice Director: Deborah L. Wetzel Edward Olowo-Okere Page 3 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Practice Manager: Alexandre Arrobbio Ismaila B. Ceesay Clelia Kalliopi Helena Task Team Leader(s): Zubair Khurshid Bhatti Rontoyanni, Irum Touqeer ICR Contributing Author: Immanuel Frank Steinhilper Page 4 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ABSTRACT The Punjab Public Management Reform Program (PPMRP) aimed to improve transparency and resource management of targeted departments of the Province of Punjab in Pakistan. The Program-for-Results (PforR) operation was completed in 5 years (November 2013 to December 2018) at a cost of $50 million. The PforR was prepared under uncertainty in a complex environment. It was the first Governance PforR in Pakistan and worldwide, prepared in times of increased volatility of fiscal, macro-economic and security conditions, following a period of strong economic growth in the early 2000s. Punjab faced significant service delivery challenges in a context of limited fiscal space and capacities. Critical constraints to service delivery were (i) limited accountability due to limited transparency and access to information for citizens and decision-makers about available services and service delivery performance; and (ii) limited resource management related to low provincial own-source revenues and challenges in public procurement. The operation was conceived deliberately as part of a broader Governance Reform Program of the Government of Punjab (GOPb) which aimed to improve governance and delivery of public services. The PforR focused on strengthening the underpinning public management systems critical for enhancing service delivery and creating fiscal space. Given the complexities of reforms and capacity constraints, the Program was realistic in terms of its expected results. Rather than promising quantitative or qualitative improvements in services for which attribution would have been difficult, the PforR targeted three main key development outcomes: (i) improved transparency and access to information about targeted services, (ii) enhanced performance monitoring systems, and (iii) strengthened management of resources. The PforR was implemented successfully and timely, surpassing all its PDO and intermediate targets: • Regarding improved transparency and access to information, 84 websites provided key information about the respective departments and available services. In addition, Punjab’s Citizen Contact Center (CCC) reported a significant increase from 50,000 citizens’ enquires/comments about public services in 2013 (baseline) to 2.9 million in 2018. During the same year, 153 targeted agencies were using the Punjab Procurement Regulatory Authority’s (PPRA) Management Information System (MIS) for registering and processing their contracts. • Regarding performance monitoring, the Program supported the innovative use of smart management tools to monitor district services throughout the Province in health, agriculture, education, and other areas. For instance, health field workers used smart phones with geo-tagging to report on vaccination activities, which led to a significant expansion of coverage from 28% to 97%. In education, a performance management dashboard informed by smart phone data systematically measured indicators such as student and teacher presence and quality of facilities and allowed rankings among districts. In agriculture, the system monitored the provision of several agriculture extension services. • Regarding resource management, Punjab’s Excise & Taxation Department completed the digitization of Urban Immovable Property Tax (UIPT) records in all 36 provincial districts, adding over 1 million new properties to the registry. Total revenue collection of this provincial tax increased significantly during the Program, reaching Rs. 10 billion in 2018 and exceeding the Program’s target of Rs. 6.9 billion. Given the operation’s high relevance and successful implementation, combined with only minor shortcomings mainly related to some indicators, the overall outcome rating of the PforR is assessed as “Satisfactory”. Page 5 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE Context 1. At the time of appraisal, Pakistan encountered increased volatility of fiscal, macroeconomic, political and security conditions. Following a period of relatively strong economic growth in the early 2000s, Pakistan’s near-term outlook was challenging during PforR preparation, with forecasts of limited growth and risks arising from political, governance and security factors; debt and external financing; and limited governance and public sector systems. The overall environment and outlook during preparation were characterized by several uncertainties (Figure 1). Figure 1: Pakistan GDP Growth Rate (%) Strong growth period Lower growth rates & uncertain outlook 10 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Source: Data from World Bank World Development Indicators. 2. The 18th Amendment of the Constitution of Pakistan (2010) significantly strengthened the role of Provinces in service delivery. The Amendment expanded the competences of Provincial governments, giving them exclusive responsibility over public services (except for the power sector), as well as their finances. Under the new rules, Provinces were assigned 57.5 percent of the pool of divisible revenues as well as new tax raising powers, notably the General Sales Tax (GST) on services. Another important development was the return to competitive elections in 2009, which strengthened political competition at the provincial level and gave provincial politicians a strong incentive to improve public service delivery. 3. Punjab, while being Pakistan’s largest and wealthiest province, encountered significant poverty and inequality challenges. In 2010, Punjab’s 95 million inhabitants accounted for half of Pakistan’s population of 187 million and 59 percent of the country’s GDP. Meanwhile, Punjab faced a relatively high poverty incidence. According to the PAD, in 2011/12, about 20 million citizens (21 percent of the population) lived below the poverty line, up from an estimated 18.3 percent in 2007/08, the second- highest severity of all four provinces. Inequalities among different income groups and between urban and rural areas were on the rise, with wide variations across districts and gender lines (Figure 2). Page 6 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Figure 2: Poverty and income inequality in Punjab Pakistan: Poverty trends at province level 4.0 Income inequalities in Punjab (Ratios) 3.4 3.4 3.7 2.0 1.3 1.3 1.5 0.0 2005/06 2007/08 2010/11 Urban/Rural 5th/1st Quintile Source: World Bank staff estimates. See 2019 Poverty & Equity Brief for updated figures. 4. Public service delivery in Punjab faced critical limitations and outcomes, particularly in human development sectors. Available data on health services indicated low use of government facilities relative to private dispensaries or hospitals, a high rate of absenteeism of key frontline personnel and overall low satisfaction with public services1. These weaknesses were also reflected in the Province’s limited social outcomes. The child immunization rate stood at 64% and challenges of absenteeism were noted in the field workers of health, education and agriculture extension. A largely unregulated and uncoordinated private sector provided a wide range of services. However, this delivery mode exacerbated inequalities, with wealthier households using higher-quality private providers and the poorer ones relying on lower- quality public providers. 5. The absence of reliable performance information for decision-makers and limited information for citizens about available services were two key constraints to improve coverage and quality of public services in Punjab. The absence of reliable and up-to-date administrative data made it difficult for decision-makers to assess the performance of government programs, hold agencies accountable and reward or sanction frontline staff. Information available to the public about institutional performance and service delivery was equally limited and often inaccurate or not user-friendly (in manual form, maintained at different levels in districts), making it difficult to access services and hold government accountable. The limited use of Information and Communication Technology (ICT) also increased transaction costs for citizens in getting information about and/or getting services and created space for intermediaries to take advantage of information asymmetries. 6. Punjab generated limited own resources to meet increasing needs for investment in service delivery. While, 85 percent of Punjab’s overall revenues depended on revenue transfers from the federal government, Punjab’s own source revenues (OSR) were low, and displayed stagnating trends (Figure 3). Especially the potential in Punjab’s urban immovable property tax (UIPT) remained unrealized, due to a narrow tax base and extensive exemptions and presented risks of uneven implementation. 1 Low citizens’ satisfaction of public services was noted in Pakistan Social and Living Standards Measurement Survey of 2011/12. Page 7 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Figure 3: Punjab Provincial Revenues 6.0 Percent of Provincial 5.0 4.0 GDP 3.0 2.0 1.0 0.0 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12 Revenue Transfers OSR Source: Finance Department, Government of Punjab. 7. Punjab’s public procurement lacked systems to reliably monitor implementation of rules and procurement performance, training for officials and a strategy to move towards e-procurement. The nascent Punjab Public Procurement Regulatory Agency (PPRA), created in 2009, was understaffed with limited technical capacity and could not rely on a system for public procurement outlays. While PPRA had drafted a reform road map, it did not have a plan to move towards e-procurement. These limitations were corroborated by a 2012 Punjab PEFA assessment which found that procurement practices remained limited and hindered cost-effective service delivery.2 8. Cognizant of these challenges, the Government of Punjab developed the Governance Reform Roadmap (“Government Program”) to improve public service delivery. Consolidating several reform initiatives, the Government Program aimed at increasing fiscal space, e.g., through the urban properties digitization pilot, and at enhancing transparency, e.g., in procurement and by using ICT for citizen engagement. The scope of the Government Program is outlined further in paragraph 12 and Figure 4. 9. The priorities of the Bank’s 2010-2013 Country Partnership Strategy (CPS) for Pakistan3 also emphasized the importance of public sector governance improvements. The CPS recognized that “the achievement of Pakistan’s development objectives depends on the improved governance of the public sector—greater transparency and accountability, strengthened legal and regulatory frameworks…, improved responsiveness, and better interface with citizens.” PforR activities also directly contributed to the CPS’s economic governance pillar which aimed to “strengthen tax policy and administration, strengthen public expenditure, financial and procurement management, and enhance capacity and accountability in public sector management.” Rationale for PforR Support and Program Scope and Boundaries 10. At approval, a Program-for-Results (PforR) was considered the most appropriate instrument to support Punjab’s development objectives in Governance for several reasons. First, the PforR could rely on a clearly articulated Government Program with a results and expenditure framework and systems assessed as satisfactory. Second, the PforR did not include any “Category A” activity, thus presenting low safeguard risks. Third, compared to a DPF, the PforR allowed for more targeted support in the area of governance. Fourth, compared to a traditional Investment Project Financing (IPF), the PforR was expected to allow for a more impactful amount of expenditures focusing on addressing critical cross-cutting 2 2012 Pakistan Punjab Province Public Financial Management and Accountability Assessment. 3 World Bank Pakistan Country Partnership Strategy FY2010-2013. Page 8 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) constraints to service delivery and incentivizing results through DLIs.4 Fifth, there were potential broader benefits in testing the new PforR modality in the area of governance and in Punjab. Finally, the PforR provided leverage to address the above-mentioned CPS objectives. Government Program 11. The Government Program to improve public service delivery included nine themes covering cross-cutting governance areas: meritocracy, open responsive government, innovation and knowledge management, results-based management, e-governance, decentralization, resource mobilization and value for money, PPPs and leadership. The total cost of the Government Program was estimated at PKRs 40 billion (approximately US$407 million). PforR Program Scope 12. Given the breadth and ambition of the overall Government Program, the Punjab Public Management Reform Program supported only a subset (Figure 4). Specifically, the PforR aimed to support GoPb in collecting information about service delivery and performance, making it available to citizens and decision-makers, thus strengthening feedback loops for improved accountability. A particular emphasis was placed on actually facilitating citizens’ access to service, including through improved transparency and accountability, and creating fiscal space to sustain further service delivery investments. Figure 4: Government Program and Bank Program for Results Program Performance Meritocracy Transparency Monitoring Main areas of support under PforR Resource Innovations E-Services Management Decentralization Public Private Leadership Partnerships for Results Theory of Change (Results Chain) 13. The Operation had a clearly articulated results chain5 centered on the public service delivery chain (Annex 6). The main rationale and critical assumptions for the PforR were that increased 4 In retrospect, while the result-based aspect of the PforR was highly relevant, a result-based IPF with DLIs and a complementary TA component could probably have been an equally viable instrument and could be considered for similar operations. TA and capacity building activities may also further incentivize successful implementation of program activities by departments. 5 As one of the early PforRs, it was prepared prior to the requirement to formally elaborate on a “theory of change”. Page 9 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) transparency and resource management, facilitated by a strong use of ICT and smart monitoring tools, would lay the foundation for better public service delivery: • Increased transparency and smart management systems in service delivery would allow decision-makers to better manage resources and performance (supply side) while facilitating citizen’s access to services (demand side). • Increased coverage in property tax administration would help increase Punjab’s own resources and fiscal space, which would be critical for future investments in service delivery improvements. • Enhanced practices in public procurement would contribute to strengthened transparency and resource management and lay the foundation for more effectively translating available resources into service delivery outcomes. A common feature of all DLIs and virtually all program interventions was the use of ICT and smart management tools to improve public sector management outcomes. Moreover, DLIs were strategically selected and clearly linked to PDOs, thus serving not just as mere financial incentives, but also as main tool for triggering critical actions on the ground towards achieving Program objectives. Meanwhile, the PforR was realistic regarding expectations of what results could be achieved within the 5-year Program period. Given the capacity constraints and challenges of implementing some of the reforms, the PforR aimed at improved transparency and resource management of targeted departments in Punjab rather than measurable improvements in public service delivery, which would have been difficult to achieve during the Program period solely on account of management interventions and resource management. Program Development Objectives (PDOs) 14. The Program Development Objective was “to improve transparency and resource management of targeted departments of the Province of Punjab”. Key Expected Outcomes and Outcome Indicators 15. The PforR targeted three key outcomes measured by PDO indicators: Table 1: Key Outcomes and PDO Indicators Key Outcome PDO Indicator Improved citizens’ access to information Number of enquiries/comments received by the Citizen Contact about key public services Centers to seek information about or to provide feedback on key services.6 Improved management of key services Decision-makers using reliable service delivery information for improved management. Improved urban property tax collection Urban immoveable property tax collection. 6 Revised indicator. Previously, it captured only “telephone calls received”. Page 10 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Program Results Areas and DLIs 16. The PforR comprised five DLIs that contributed to three results areas: Table 2: PforR DLIs & Results Areas Disbursement Linked Indicators (DLI) $Mn Results Areas (RA) DLI1: Targeted organizations publishing updated institutional information 5 on their websites for information of the public RA1: Transparency and DLI2: Number of enquiries/comments received by the Citizen Contact access to services 10 Centers to seek information about, or to provide feedback on key services DLI3: District services being monitored by smart management tools RA2: Performance 15 monitoring DLI4: Properties added to the urban property registry 15 RA3: Resource DLI5: Targeted organizations using PPRA MIS for targeted contracts 5 management TOTAL 50 B. CHANGES DURING IMPLEMENTATION Revised Definition of DLIs 17. Although the PforR was restructured once, it did not experience any significant design changes and maintained its PDO, results areas, DLIs and theory of change throughout implementation. The Program underwent a minor level 2 restructuring in July 2017. The main objective was to refine the definition of PDO indicator 1 (DLI2) by measuring not only telephone calls but different communication modes through which citizens engaged with the Citizen Contact Center and provided feedback on key public services. While it could be argued that the restructuring made the achievement of the original PDO1 easier at the time, it demonstrated proactivity and accommodated citizens’ preferences to communicate with the CCC not only through telephone calls but also through SMS and electronic messages. In the bigger picture, the restructuring did not change the theory of change or substance of Program objectives. In addition, at the end of the Program, the original target for telephone calls was also achieved. As the continuity of Program activities dominated, no additional analysis pre and post restructuring appeared warranted for the ICR. Other Changes 18. During implementation, other changes, including staff turnover at program implementing agencies, and elections did not have a significant impact on program design or implementation. Several staff implementing agencies management changes occurred in PRMP, PPRA and ETNCD during the Program. There was also a change in Task Team Leaders (TTLs) during the Program. Meanwhile, there was continuity at management positions in P&D Board (borrower/recipient) and PITB (implementing agency) and strong engagement with core technical staff in each program-implementing department, which were less likely to change and served as the necessary institutional memory for reforms. Also, while there was a change in TTLs, the presence of a strong Bank team in the field office and partial overlap of TTLs provided necessary support for uninterrupted implementation. There was also a change in government following the elections in 2018 towards the end of the Program. However, this too did not affect the Program design Page 11 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) or implementation, as the Program had already achieved its main targets at the time and objectives enjoyed broad consensus. II. OUTCOME A. RELEVANCE Relevance of PDO 19. Relevance of PDO is assessed as “High”. At closing, as during preparation, the operation’s development objectives remained fully aligned with the Bank’s CPS. The CPS FY2015-20 placed a “focus on governance as an integral part of the WBG program” by “systematically identifying ways to enhance accountability and transparency in all public and private institutions and systems”. Specifically, the PDO contributed directly to CPS Results Area 4 Service Delivery, particularly areas 4.1: Improved Public Resources Management (improved tax collection in targeted provinces) and 4.4: Adoption of Performance and Transparency Mechanisms in Selected Institutions (departments/services with citizen feedback in place). In line with the new institutional environment created by the 18th Amendment of the Constitution, the CPS also had a cross-cutting theme of deepening engagement at the provincial level to which the PforR contributed directly.7 It also paved the way for several additional sector and PforR operations in Punjab and other provinces. Overall, considering both the CPS at preparation and at closing, there was a high level of continuity in the relevance of objectives of the operation.8 Relevance of DLIs 20. Relevance of DLIs is rated as “High”, reflecting their clear link with the PDO. DLIs were clearly defined and measurable and provided strong incentives to gradually progress towards the achievement of PDOs while providing a timely and regular flow of funds. Five indicators were selected from the results framework as DLIs to incentivize government agencies to use the public management systems strengthened by the program and address critical bottlenecks that might have hampered the Program’s results. The first DLI aimed at providing incentives to departments to publish information, while the remaining four aimed at providing incentives to public agencies to use the systems developed or strengthened through the Program —namely, providing information through the Citizen Contact Center, using smart management systems, the UIPT digitized registry and the procurement management information system. DLIs were annually verified by a third-party verification (TPV) firm, and according to reported results, they all achieved — or exceeded— Program performance targets.9 The DLIs were carefully targeted at reform actions that would have been difficult to implement in the absence of strong incentives. Given the clear alignment between DLIs and PDOs and their careful selection to incentivize difficult reforms, the relevance of DLIs is rated as high. 7 Based on the successful experience with PPMRP, GOPb subsequently also started the Punjab Jobs and Competitiveness and the Strengthening Markets for Agriculture and Rural Transformation (SMART) Program-for-Results operations, while operations in other provinces emulated some performance management aspects of PPMRP. 8 For additional details, see World Bank Group Pakistan CPS FY2015-20, particularly pp. 22-26. 9 While in retrospect and based on the results some targets could have been set higher, the targets were realistic and ambitious at the time of preparation considering that this was the first PfoR as well as prevailing capacity and contextual factors. The overachievement is therefore also a reflection of the overall success of the Program. Page 12 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Rating of Overall Relevance 21. As per ICR Guidelines, overall relevance is rated as “High”. The absence of shortcomings referred to the relevance of both PDO and DLIs justifies the assigned rate. The overall relevance of PDO and associated DLIs is also reflected by their permanence throughout Program implementation, despite of the presence of ex-ante high “country risk” — due to Pakistan’s vulnerable fiscal situation, uncertain security environment, and limited governance. In fact, no policy reversal took place over the life of the operation, there was stability of operational design —despite macroeconomic and political challenges— and the Program contributed to the deepening of sector reforms with no decline in demand for such interventions both from policy makers and citizens. The Program also faced high “technical risk” —given the use of innovative but untested ICT tools and the complexity of Program design in terms of involving multiple departments. However, the involvement of multiple departments was directly related to the relevance of PDOs, and their smooth functioning, complemented by a strong leadership of the implementing agencies, contributed to the achievement of PDOs and their associated DLIs. Another way to assess the Program’s overall relevance is taking into account, on the one side, its clear alignment to the Bank’s priority to strengthen public sector governance in Pakistan, and on the other, the fact that it was conceived as part of a wider Government Program in the province of Punjab. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective or Outcome 22. The assessment of achievement of PDOs is organized around the three key development outcomes of the PforR outlined in the PAD. While the PDO statement included two explicit objectives, “to improve transparency and resource management of targeted departments of the Province of Punjab”, the PAD also defined three related key outcomes (i) Improved citizens’ access to information about key public services; (ii) Improved management of key services; and (iii) Improved urban property tax collection. Assessment of these three key outcomes was considered to provide a comprehensive and granular picture of the efficacy of the Program, while also sufficiently covering the PDO objectives. First development outcome: “improved citizens’ access to information about key public services” 23. Furthering the PDO’s transparency objective, the Program supported the implementation of the Punjab Transparency and Right to Information Act (2013). On the one hand, it significantly strengthened transparency of targeted organizations by providing regularly updated information about key public services, departmental budget, change in policies/regulations, and contact details on 84 websites, compared to the Program’s target of 75. In parallel, the Citizen Contact Center and Citizen Facilitation Center actively facilitated citizens’ access to public services. 24. The evolution of PDO indicator 1: “number of enquiries/comments received by the citizen contact centers to seek info about, or to provide feedback on key services” indicates some of the effects of increased transparency. Citizens’ comments/enquiries registered a very significant increase from 50,000 in 2013 (baseline) to 2.9 million in 2018. These results, based on data provided by Punjab’s Citizen Contact Center (CCC) and verified by the third-party verification firm, clearly exceeded the Program’s target of 750,000 enquires/comments. They reflect the advantages of GovTech with rapid technological changes (e.g., the increasing use of SMS and other similar quick access internet-based tools) over recent Page 13 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) years, as well as the successful implementation of a well-planned awareness-raising campaign to enhance transparency about government services. 25. Progress was also made to various degrees in strengthening transparency in public procurement and public investment management. In the area of procurement, 153 contracting agencies entered procurement contracts in the MIS at the end of the program compared to a target of 100. This contributed to enhanced transparency, in particular in terms of publishing tenders and contract awards on the PPRA website. In turn, information is currently still limited in terms of procurement methods, procurement expenditure, timelines and products or services purchased. A multiplicity of data is available in the database, but there could be room for strengthening the analysis of the data, moving towards e- procurement systems, and using the data for policy decisions (e.g., using consolidated procurement for large procurement categories). Under the program, 77 percent of public investment projects of targeted departments were geo-tagged and published online compared to a target of 75 percent, although user- friendly access and coverage across districts could be strengthened. Second development outcome: “improved management of key services” 26. As a result of the Program’s ICT-based interventions, the effectiveness of targeted departments’ internal processes appears to have increased. This supported both the transparency and the resource management objective of the PDO. By deploying “smart management tools” in the form of smartphone and tablet apps for performance data collection in the field as well as management dashboards for their aggregation and analysis, real-time monitoring of six public services —related to health, school education, and agriculture sectors—was implemented in all 36 districts of the province. This modern, up-to-date and flexible type of monitoring has helped decision makers to use field performance information for effective and timely decision making, although it might have benefitted from being embedded in an overall performance strategy and related key performance indicators (KPIs). By the end of Program implementation, the performance of government field staff was being regularly and transparently monitored by supervisors, which provided an evidence-based foundation for improved managerial actions for better service delivery. 27. The evolution of PDO indicator 2: “decision-makers using reliable service delivery information for improved management” showed positive results. Measured by “the percentage of periodic administrative reports submitted by supervisors/managers to a dashboard and triggering managerial actions”, all submitted reports in 2018 (100 percent) complied with this characteristic, which surpassed the Program’s target of 65 percent. The chosen indicator reinforced a result-based spirit by focusing on actual actions taken rather than mere collection of data. However, in retrospect, it was probably sub- optimal due to the ambiguity of what constitutes “managerial action” and the inherent difficulty to independently verify actions taken. 28. Other evidence beyond the results indicators points to significant positive impacts of the performance management systems supported by the program. For instance, in the health sector, in a context of the new possibility of quasi-real time tracking, the attendance of immunization workers increased very significantly from 36 percent in 2014 to over 95 percent by the end of the program. Similarly, geographical coverage, evidenced through geo-tagged pictures of vaccinated children, also expanded from 25 percent to 88 percent, expanding beyond the previous concentration on easily accessible areas along main transportation corridors. During the ICR missions, several stakeholders also Page 14 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) reported the benefit of supervisors being able to take informed decisions for effective human resources deployment, for instance, through effective workload management and schedule planning. In education, other evidence also corroborates that the introduction of the new performance monitoring tools had tangible effects. For instance, overall, student and teacher attendance, and school facilities meeting minimum requirements (electricity, sanitation, safety, running water) showed visible improvements during the Program period.10 A few randomized control trials on agricultural extension services also appear to support the positive impact of the use of smart phones for service delivery.11 The PforR also had a positive impact beyond Punjab as other provinces approached the Government of Punjab regarding the introduction of similar reforms.12 Third development outcome: “improved urban property tax collection” 29. Related primarily to the PDOs’ resource management objective, activities in this area also showed significant results. According to information shared by the Excise and Taxation Department and confirmed by TPV reports, the digitization of Urban Immovable Property Tax (UIPT) records was completed in all 36 districts. In total, over 1 million new properties were added to the property registry, surpassing the whole-of-Program target of 662,000 and respective annual targets, as verified by the TPV reports. The resulting updating and expansion of the tax base translated into increased voluntary compliance and reduced corruption risks. Overall, “revenue collection of UIPT” (PDO indicator 3) increased significantly to PKR 10 billion in 2018, more than doubling the baseline of PKR 4.6 billion in 2013, and exceeding the Program’s end target of PKR 6.9 billion. The increase could be achieved despite a significant reduction in the property tax rate, from 20-25 percent to 5 percent, as property valuations were also increased, although they remain significantly below actual market values.13 Overall, Program activities contributed to institutional strengthening in tax administration and improvements in billing and collection systems. Rating of Overall Efficacy 30. Overall efficacy is rated “Substantial”. As described above, the PforR achieved in substance all of its outcomes and met or exceeded the targets of all of its results indicators and disbursed all its funds14, with only minor shortcomings related to indicator quality and available data that do not always allow full attribution to the PforR alone. Additional indicator-by-indicator information is provided in Annex 1A and can also be found in the annual Third-Party Verification Reports. 31. The existence of well-structured and annually monitored DLIs contributed to the success of the overall Program. Disbursement-linked indicators (DLIs) were carefully selected to provide incentives on critical roadblocks to improving the Program’s overall effectiveness. Moreover, in some cases they also have the potential to boost government management of day-to-day service delivery. For example, the 10 For detailed data, refer to the Government’s “Open Punjab” web portal. 11 Arman Rezaeee, Ali Hasanain, Yasir Khan. Crowdsourcing government accountability. Experimental evidence from Pakistan. November 12, 2015. 12 For instance, the governments of Khyber Pakhtunkhwa, Balochistan and Gilgit Baltistan drew on Punjab’s experience to implement the eVaccs application to monitor and manage field staff to increase their attendance and vaccine coverage. 13 At 20-30 percent of market values. 14 At the end of the Program, the PforR disbursed $46 million, which represented 100 percent of its fund. The difference to the $50 million estimated in the PAD is explained by exchange rate fluctuations. Page 15 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) increase registered in the number of government services monitored by smart management tools (DLI3) allowed for periodic monitoring in all 36 provincial districts of attendance, availability of supplies, control of facilities, etc. in key sensitive areas such as school education and health. 32. As reflected on the ex-post economic analysis,15 benefits accrued from the Program were far beyond those forecasted by the PAD. The ex-post economic analysis was based only on inputs for which reliable evidence and data was available. Therefore, although actual benefits have most probably accrued from all Program’s main components, as estimated in the PAD, only incremental tax revenues, i.e., benefits accruing from the Program’s third development outcome, were considered for such an exercise. Despite of the use of this conservative approach, the Program’s Net Present Value reported by the ex - post economic analysis under different scenarios yielded US$305 million, US$206 million and US$ 162 million, using 0 percent, 6 percent and 10 percen discount rates, respectively, against the PAD baseline that was set at US$47.5 million in 2013. C. JUSTIFICATION OF OVERALL OUTCOME RATING 33. In light of the ratings for Relevance of Objectives (High) and Efficacy (Substantial), with only minor shortcomings, the Overall outcome rating for the Program is assessed as “Satisfactory”. On the one hand, the Program was very relevant to the needs of Punjab Government for improving transparency and resource management, in consonance with its governance road map of 2013 and closely linked to CPS objectives. On the other hand, sufficient evidence-based information was obtained from Government executing agencies and a TPV firm indicating progress towards all three development outcomes. The PforR achieved or even exceeded all of its indicators. In substance as well, the results of the Program can be deemed significant (e.g., the improved tax collection, active use of the Citizen Contact Center, and visibly expanded performance management of health and agriculture extension services, among other). The rating also reflects the capability, understanding and capacity of the Government of Punjab to successfully implement Program-for-Results operations. 34. The ICR considers that there were only minor shortcomings in the operation’s achievements of its objectives, which underpins the Satisfactory rating. A few considerations related to program design and attribution of results prevented an even higher overall outcome rating. Overall outcome was rated as “Satisfactory” due to the following. First, one of the three key development outcomes, “improved management of key services” was not as clearly captured in the Program’s PDO “improving transparency and resource management” as the other two. Also, PDO indicator #2, defined by the results framework as “decision-makers using reliable service delivery information for improved management as measured by percentage of reports triggering managerial action” had limitations in terms of its clarity and measurability. Finally, although all Program’s outcomes were fully achieved, even surpassing pre- established targets, overall results should be assessed considering also the compounded effect of external factors operating in the same direction, such as the fast evolution and massive adoption of new technological tools during the period of Program implementation, which the Program anticipated, and the co-existence of a broader Government Program and other sectoral reform initiatives. Taking these factors into account and avoiding an overly optimistic perspective of a “perfect” Program, a “Satisfactory” rating is deemed most appropriate. 15 See Annex 5. Page 16 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) D. OTHER OUTCOMES AND IMPACTS Gender 35. The project did not target any specific gender impact. Public services were to be provided to all citizens of Punjab irrespective of gender, although some services under the Government Program were particularly relevant for women (e.g., the expansion of lady health workers and supervisors). For instance, female customers randomly interviewed by the ICR team at the Citizen Facilitation Center in Lahore reported their satisfaction with the cleanliness and safety of the center, which made themselves and their families comfortable to request a service there. In retrospect, from a gender perspective, there could probably have been an opportunity to gather more gender-disaggregated data. For instance, under DLI 2 (Citizen Contact Center), disaggregation of data by gender might have provided some insight into the usefulness of the helpline to women. Likewise, under DLI3, the program could have foreseen gender- disaggregated data of service beneficiaries, which would have been particularly relevant in measuring the access of women farmers to agricultural and livestock extension and veterinary services, potentially together with a target for female beneficiaries. The share of female CCC staff and number of female field works could also have been measured, providing employment opportunities for women. While not specifically measured during the program, the mechanisms in place would allow to capture gender- disaggregated data in the future. Poverty Reduction and Shared Prosperity 36. The PforR did not specifically target or measure poverty reduction and shared prosperity, the latter of which was not yet an explicit WBG goal at the time of PforR preparation. Since the program did not directly target improved service delivery, the impact on poverty could not be directly measured. However, the expansion in the coverage of critical services, notably immunization and school education, are likely to have a positive impact on poverty in the medium term and – in terms of multi-dimensional poverty that includes access to education and health – the short term as well. In addition, expanded delivery of agricultural extension and livestock services to farmers should have a direct impact on their incomes. A dedicated evaluation considering data by level of income and origin for people requesting and receiving services would be required to more comprehensively assess the impact of the program. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 37. Program’s objectives/outcomes. The Program was deliberately and realistically conceived as a way to support the Government of Punjab on specific areas of its broader governance reform strategy. As such, the Program’s development objective focused only on two areas for governance improvement — transparency and resource management— for targeted departments of the Province, which were linked to the achievement of three well-defined key development outcomes. 38. Results Framework & DLIs. The results framework was overall well-designed with only minor shortcomings. Indicators were relevant, aligned to development objectives, and set appropriate baselines and feasible targets. Five indicators were selected from the results framework as DLIs to incentivize government agencies to use the public management systems strengthened by the project and address Page 17 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) the critical bottlenecks that might have hampered the Program’s results. DLIs included a clear protocol and methodology for monitoring and verification (annual third-party verification). As outlined in paragraph 23, PDO indicator 2 on “managerial action” would have benefitted from clearer definition and measurability. Also, while measuring the number of departmental websites that published relevant information on their activities and services provided a strong incentive for implementation, from a citizen- perspective, it might have been easier to have a single website hub providing information and links to all relevant services. Finally, defining cumulative rather than annual targets for adding properties to the registry could have facilitated implementation. 39. Stakeholders Engagement. The departments and services targeted for this program reflected the focus on service delivery. The departments covered by the Program —school education, livestock, irrigation, agriculture, health, and higher education— constituted the bulk of state-citizen interaction and were chosen to make service delivery improvements more visible as well as to spawn political ownership. 40. Risks and Mitigation Measures. Overall risk was adequately assessed as “high” during preparation, with a special concern on technical risk, considering the use of innovative ICT tools and performance management concepts relatively limited capacity of implementing agencies, and the fact that Program activities involved different agencies and layers of government, requiring strong coordination between central agencies administering core public management systems and line departments using them. Appropriate mitigation actions were identified to minimize these risks, including, in particular, the addressing of key technical gaps through dedicated DLIs and PAP actions, the implementation support plan, and the set-up of a Steering Committee, chaired by the Chairman of the Planning and Development Board, to monitor progress and facilitate coordination among participant institutions and different levels of government. The program would have benefitted from a dedicated TA component supporting participating departments in the implementation of program and change management activities, in particular in the field. 41. Legislation and regulations. In 2013, the Government passed Punjab’s Transparency and Right to Information Act, a key legislation aimed to promote higher transparency and good governance as well as performance improvements in public service delivery. The PforR played a key role on the implementation of this Act, by focusing on transparency of process-oriented information about key public services, departmental budget, policies and regulations, and up-to-date contact details. One area where the Program could have benefitted from even stronger support were provisions for the institutionalization of program interventions to support their sustainability. For instance, regulations or guidelines for departments to operationalize the RTI Act or dedicated financing for its implementation could have supported this. Similarly, rules and standard operating procedures to emphasize that electronic/digital records enjoyed the same validity as paper records. In retrospect, the performance management activities would have benefitted from greater institutionalization, e.g., by being embedded in an overall strategy with government-wide performance targets and key performance indicators. B. KEY FACTORS DURING IMPLEMENTATION 42. Commitment & Leadership. Program implementation benefited from experienced political leadership and continuity—the same political leadership had already completed five years at the Punjab’s Government. Moreover, the Program’s three key development areas were among the nine results areas contemplated by the Punjab’s Governance Reform Roadmap. Leadership and reform commitment in the Page 18 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) implementing agencies and participating departments was in place but also encouraged by a strong performance orientation of top political leadership. 43. Co-ordination and Engagement. The Program had four implementing agencies—PRMP, PITB, PPRA and E&TD—, working along line departments to implement Program interventions in their respective result areas as per their mandate, agreed timelines and sequencing. PRMP played a crucial role, effectively worked as the Program’s executing agency. It served as a secretariat to the Steering Committee regarding Program implementation, supporting day-to-day implementation of Program activities and facilitating coordination across different government departments. Overall coordination and engagement of stakeholders remained strong throughout the Program. 44. Monitoring & Evaluation. An adequate monitoring plan was in place, and realistic and tangible indicators made it easy to collect information to provide evidence supporting the achievement of the outcomes. PRMP, as executing agency, had overall responsibility for coordinating, monitoring, and reporting on the Program‘s result indicators. Each of the four implementing entities was responsible for monitoring the intermediate indicators and specific PDO indicators relevant to their area of intervention. The verification protocols were clearly defined and agreed on to ensure that PRMP could collect, analyze and report on DLI indicators, which were annually scrutinized by a Third Party Validation firm. 45. Program coordination and human resources. The contribution of the PRMP to the successful implementation of the program was critical, as were the roles played by PITB and the Urban Unit. Institutional capacity building was an inherent part of the program. The PforR addressed initial technical gaps through targeted DLIs and PAP actions, as well as the implementation support plan. In particular, a capacity building plan was developed, and PAP included important milestones like the institutional review of PITB’s capacity to implement e-services and Performance management. Overall, adequate human resources and organizational capacity was in place to effectively implement the Program. 46. PforR Funds Flow. As per the standard practice for PforRs, funds from the PforR were directed to the Account 1 maintained by the Finance Department, whereas implementation of program activities was carried out by P&D and other departments. The absence of regular release of funds to implementing departments and the fact that many activities were funded from the non-development Budget, made implementation more difficult. This factor was probably more pronounced in the present PforR in the absence of a complementary technical assistance component. 47. Fiduciary. PFM/procurement, as well as institutional capacity building measures, were an inherent part of the Program. Fiduciary mechanisms were effective overall, albeit not always perfect. Especially during the early stages of implementation, supervision missions identified issues related to understaffing (delay in hiring of accounts staff in Implementing Agencies to support effective implementation of the program), capacity, record keeping, and minor internal control weaknesses detected in implementing agencies. These issues were effectively addressed over time. 48. Governance and Politics. Overall, there was a favorable political environment and broad consensus, which highly benefited program implementation. Political support for Program results remained strong. Page 19 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 49. Environmental and social. The program complied with environmental and social regulations. Proposed activities were environmentally benign and did not have any negative environmental effects. The main social risk was that Program activities might reinforce inequalities in terms of access to services and information, due to its focus on modern ICT-based interventions. However, the program had in place a number of measures for mitigating these risks —such as the use of a phone helpline for people in remote areas without access to the internet; and the hiring of female agents to respond to women’s queries. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION M&E Design 50. The PDO, key outcomes to be achieved and associated indicators included in the results framework were found to be consistent, well designed and effective for tracking progress. The PforR operation was designed to support specific interventions, constrained to a few (four) implementing agencies, instrumental in fostering improvements in terms of effectiveness and efficiency in the Government of Punjab’s overall Reform Program. The Results Framework was prepared as per the Bank’s guidelines. Disbursement-linked indicators (DLIs) were robustly designed and carefully selected to provide incentives on critical roadblocks to improving the Program’s effectiveness, efficiency, and sustainability. Additionally, specific institutional strengthening activities set for the implementing agencies were specified in the Program’s Action Plan and Capacity Building Plan, and appropriate targets and results were reflected in the results framework. The DLI measuring the addition of properties to the tax registry would have benefitted from defining cumulative rather than annual targets to facilitate measurement. 51. Adequate institutional arrangements were contemplated. The creation of a Steering Committee—headed by the chairperson of the Planning and Development Board—was planned to oversee implementation, provide strategic direction and guidance, and facilitate coordination between implementing agencies. Punjab’s Resource Management Program (PRMP), the Program’s executing agency, acted as a secretariat of the Committee and was designated responsible for overall management and coordination, including monitoring results and generating performance and financial reports on implementation. While PRMP was given overall responsibility for coordinating, monitoring, and reporting on the Program’s result indicators, each of the four implementing agencies (including PRMP) were responsible for monitoring intermediate indicators and specific PDO indicators relevant to their respective areas of intervention. The design of the Program also included the hiring of a Third-Party Validation firm to assess eventual progress on DLIs and results. 52. M&E design would have benefitted from cumulative targets for DLIs. The annual targets of most DLIs (DLIs 2, 3 and 4 related to CCC enquiries, monitoring of district services, and properties added to the tax registry) were set up to be non-cumulative, which at times complicated monitoring of progress toward the end-of-program targets and data reported in ISRs being prepared in the middle of the fiscal year may appear inconsistent (due to lower current than previous data). Complex disbursement formulae that varied over time for the same DLI were also complicated to calculate. Cumulative targets and more straightforward DLI pricing could have mitigated this. Page 20 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) M&E Implementation 53. PDO-related targets were duly and timely reported on the basis of information provided by the implementing agencies, and, in the case of DLIs, by annual Third-Party Validation reports. According to TPV reports and information provided by the four implementing agencies, all end-of-Program targets had been achieved (or exceeded) by June 2018, which led to timely and full disbursement of Bank proceeds according to DLIs’ results. As a result, the Program closing letter was sent by the Government of Punjab, confirming the formal closing of the Program by the initially planned closing date of December 31st, 2018. 54. DLI verification protocols were rigorous and detailed. EY Ford Rhodes, a firm member of Ernst & Young Global Limited, was hired for TPV services by assessing progress referred to the Program’s five DLIs. TPV annual reports were timely available and consistently used as the main basis for authorizing Bank’s disbursements upon the successful achievement of DLI-associated targets. Verification reports were also found to be consistent with implementation agencies information and the Bank internal reporting system. M&E Utilization 55. All PDO-level indicators, including DLIs, were consistently and regularly tracked on the basis of good quality data. Data provided by implementation agencies was verified by TPV annual reports and Bank field missions. The Bank internal reporting system (ISR) was regularly and timely updated on the basis of this information and provided results for all PDO-level indicators, including DLIs. When it became clear that the original indicator capturing only “telephone calls received by the CC” was not sufficient to take into account citizen-government interactions, this was highlighted and addressed in a minor restructuring of the operation. Justification of Overall Rating of M&E Quality 56. The overall quality of the M&E system is rated Substantial. A suitable design, implementation, and utilization of the M&E system allowed for reliable follow-up of progress across all program stages. Technical capacity-building activities executed during the first year of implementation were crucial for ensuring this result. PDOs, and their associated indicators and targets have been unchanged through the Program lifespan and deemed as pieces of a well-articulated monitoring system. It is also important to remark that the Government of Punjab has taken steps to ensure long term sustainability of Program results and associated monitoring activities. One clear case was given by the digitalization of Urban Immovable Property Tax (UIPT) records, which has been completed in all 36 provincial districts. Another case was the use of the Punjab Procurement Regulatory Authority’s (PPRA) Management Information System (MIS) for contractual registration and processing by 153 targeted organizations, which led the Government of Punjab to take steps for incorporating interventions supported by the Program in the regular structures and budgets of corresponding implementing entities. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social 57. The program fully complied with environmental and social regulations. The Program did not include any physical interventions such as construction, rehabilitation, or renovation works. As a result, Page 21 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) its activities did not trigger any environmental safeguards and were not considered to have any negative environmental effect. The above-mentioned social risk that Program activities might entail inequalities in access to services and information was mitigated through outreach, the use of helplines, and facilitation centers in rural areas. 58. The program had in place adequate procurement, financing, and financial management mechanisms. The Program mainly supported the provincial government expenditure through its normal budgeting. Disbursements were made upon achievement of the DLIs, which was verified by the TPV firm, rather than through an IPF component. The program prepared regular financial reports and benefitted from close implementation support by the task team and Bank Financial Management Specialist. The Bank received the Program’s audited Annual Financial Statements within six months of the close of each Financial Year (FY), except for FY2013-14 for factors beyond the Borrower’s control.16 The Program’s external audit was conducted by the Office of Auditor General of Pakistan (AGP), the supreme audit institution of Pakistan, which is acceptable to the Bank as external auditor. The external audit was conducted in accordance with International Standards of Supreme Audit Institutions, and an unqualified opinion was expressed by AGP in the audit reports of each FY. As outlined above, financial management issues identified in the early phase of implementation were identified and addressed. Overall, the fiduciary aspects were well supervised, and the fiduciary mechanisms were effective, as reflected in the “Satisfactory” rating for fiduciary at the end of the Program. C. BANK PERFORMANCE Quality at Entry 59. From its design stage, the Program was part of and intended to support a wider-scope Government Reform Program. At the time of Program design, the Government of Punjab had undertaken several initiatives to improve the efficiency of service delivery through ICT interventions, with important roles for PITB (ICT-related issues) and the Urban Unit (geo-spatial mapping) in support of government departments, and created the Punjab Revenue Authority. The PforR took these initiatives into account and was deliberately thought as a way to strengthen implementing agencies deemed as necessary to support and sustain outcomes under the government’s “Governance Reform Roadmap”. PforR-specific capacity building activities were identified and appropriate result indicators and DLIs included in the Program’s results framework. The Government played a central role in the development and design of the Program, which was critical for strengthening ownership during implementation. In retrospect, the Program could have benefitted from being even more institutionalized from the start, e.g., by defining government-wide KPIs or sector targets and engaging the key agencies more intensively. 60. The Program was conceived through a highly consultative process assuring the alignment with the agenda of the Government, the donor community, and other relevant stakeholders. The PforR operation was fully compatible with Pakistan’s CPS (2010–2013) priorities (Report No. 53553-PK), which remarked that “the achievement of Pakistan’s development objectives depends on the improved governance of the public sector—greater transparency and accountability, strengthened legal and regulatory frameworks, improved responsiveness, and better interface with citizens.” On the other side, 16Since this was the first PfoR in Pakistan, AGP was not yet familiar with the audit requirements for PforRs, which delayed the audit during that year. Page 22 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) the Program was developed in close coordination with the Sub-National Governance Program (SNGP) supported by DFID, whose main focus was incubating service delivery innovations at the district level. In fact, this Program and SNGP shared similar objectives and were designed to support the same government reform program. Both programs shared implementation structures to facilitate coordination and exploit the synergies between the activities to be supported at the provincial and district levels. Finally, as the PforR aimed to enhance demand-side governance for better service delivery, the role of information sharing and transparency became all the more important for engaging stakeholders at various levels, and the Government had the right incentives to create further awareness about the Program’s development both among citizens and government officials. 61. Overall, the Program was developed upon a strong evidence base and underwent a rigorous quality review process. The Bank was well placed to provide support to the government’s program because of: (i) its global experience in the area of governance reform; (ii) its strong basis for engagement in Punjab due to sectoral engagement in health, education, urban development, and other service delivery sectors; (iii) given its prior engagements using ICT innovations for accountability; and (d) given the analytical basis created through engagement on the Public Expenditure and Financial Accountability (PEFA) assessment. Also, as reported in the PAD, the Punjab Government had a solid track record of governance reforms, and was familiarized with the monitoring of district-level performance indicators. As the first Governance PforR worldwide, the operation underwent a particularly rigorous quality review mechanism. The Bank rating for Quality at Entry is assessed as “Satisfactory”. Quality of Supervision 62. The Bank’s performance in ensuring quality of supervision is rated as ‘Satisfactory’. Proposed Bank inputs and processes were adequate, included regular implementation support missions and several reporting mechanisms using a team with the full range of relevant expertise and with a strong country- based presence, allowing for frequent interactions. The task team prepared aide-memoires after each implementation support mission, making implementing agencies timely aware of implementation issues, and suggesting possible remedies, in conformity with Bank procedures. Periodic ISRs candidly and realistically rated the performance of the program, both in terms of achievement of development objectives and project implementation. The Bank’s task team played a crucial coordinating and proactive role in suggesting and promoting solutions to implementation challenges, sometimes in partnership with other donors (UK-DFID). For instance, as potential bottlenecks were identified with the DLI measuring telephone calls from citizens during the first implementation support mission, this indicator was suggested to be revised. Missions also highlighted the need to improve the pace of progress in different DLIs to avoid any future performance issues with possible implications for overall results’ achievement. Justification of Overall Rating of Bank Performance 63. Based on the assessments above, on “quality at entry” and “quality of supervision”, respectively, overall Bank Performance is rated as “Satisfactory”. There were only minor shortcomings in quality of entry (e.g., the mentioned imprecise definition of an indicator) and virtually no shortcomings during supervision, despite a change in TTLs. Overall, the Bank rating is therefore assessed as “Satisfactory”. Page 23 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) D. RISK TO DEVELOPMENT OUTCOME 64. The risk to development outcome is assessed as low. Risks that achieved development results will not be maintained in the future are considered low overall. The likelihood of a political reversal is low, given that program objectives are central to Punjab’s growth and development efforts and enjoy broad political consensus. Similarly, technological advancement in public administration is a broader general trend unlikely to reverse. The digitization of property records for tax collection or of departmental records was mainly a one-time effort that will continue to be beneficial unless the new records will not be used anymore, which is considered unlikely. In fact, there are indications that the successful experience in property tax may spur similar efforts for other taxes. At the time of writing of this ICR, the comparatively most significant potential risks in terms of impact would be a funding stop for sustaining program activities or waning ownership or insufficient technical capacity by departments to maintain high levels of transparency and using performance management tools. The likelihood of these risks materializing is also considered low. 65. Government of Punjab feedback on the ICR indicates that implementing agencies and departments own PPMRP interventions and have taken measures for budget allocation. Implementing Agencies (PPRA, ET&NC, and PITB) and target departments (School Education, Higher Education, L&DD, Agriculture, Primary & Secondary Health Care, Specialized Healthcare & Medical Education, Irrigation, and LG&CD) have taken measures to ensure sustainability of the intervention outcome through government budgetary appropriations under the ‘Schedule of New Expenditures’. The Citizen Contact Center already received funding and continued operations beyond program closing, with additional departments using it for new campaigns. Similarly, PPRA has received a budget allocation to continue its work on strengthening public procurement. Finally, as far as the ICR could assess, the use of websites and smart performance monitoring tools has been engrained in daily routines with supervisors appreciating the added efficiency, making a complete reversal unlikely. Further institutionalizing the reforms through integrating the performance monitoring systems in an overall government performance framework, ideally linked to the province’s growth strategy and operationalized through KPIs and targets for each sector/department, enhancing the regular publication of procurement statistics, and retaining capacity developed in PITB, Urban Unit and PRMP may be critical steps ahead. Future operations could also consider designing interventions in a way that their sustainability is assured, through government’s progressive on-budget financing of continuing activities, by the penultimate year of the program. V. LESSONS AND RECOMMENDATIONS 66. Swift and successful implementation of this type of Program benefits from adequate governance-sector reform momentum and strong political ownership. A key factor for success of this operation stemmed from the fact that it was born as a means to deliberately support a wider-scale governance reform program in Punjab, the so called “Governance Reform Roadmap”. Then, ownership was facilitated and Program’s main components were aligned to this broader initiative. For instance, the Punjab’s Transparency and Right to Information Act was passed in 2013, for which the transparency component of the Program provided some implementation support. Also, Government’s focus on provincial revenue mobilization, as highlighted on its Growth Strategy (2015-2018), was supported by the Program through tax administration reforms in UIPT, Punjab’s second most important tax revenue source. Page 24 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 67. Focus on service delivery was another factor of success of the PforR. The Program maintained a focus on service delivery through the strategic selection of targeted sectors. While program interventions were directly related with upstream provincial institutions and systems, it engaged with seven specific sector departments and services, —school education, livestock, irrigation, agriculture, health, local government, and higher education—which constitute the vast bulk of Punjab’s state-citizen interaction. Therefore, service delivery improvements were highly visible and promoted stakeholder’s ownership. 68. The choice of a results-based operation provided right financial and non-financial incentives to deliver intended results. First, the PforR aimed at improved transparency and resource management of targeted departments rather than measurable qualitative or quantitative improvements in Punjab’s overall public service delivery, which might have been difficult to attribute to the Program. Second, PDO indicators were associated to realistic targets, and Disbursement-linked indicators (DLIs) were carefully selected, thus providing right incentives contributing to the successful and timely achievement of development outcomes, in a context characterized by implementation challenges and capacity constraints. Third, as presented on the Action Plan and Capacity Building Plan, the Program focused on strengthening the institutional capacity of a few key agencies, which helped on ensuring that crosscutting reforms were effectively implemented. In hindsight, however, the implementation of the program would have benefitted from the inclusion of a TA component to support departments with complex interventions such as the implementation of ICT-based monitoring tools. 69. Engaging broadly with counterparts at both political and technical levels is critical to sustain reform ownership and momentum. Regular turnover of counterparts, particularly at the higher levels of the administration, is a common occurrence in Pakistan and in Punjab. These frequent changes can pose a risk to implementation continuity and sustainability. For similar operations, it is therefore highly recommended to broadly engage with counterpart institutions rather than only current project counterparts and decision makers. This includes continuous dialogue with officials at the technical level that are less prone to rotation and will be critical to drive changes in procedures and actual behavior. The broader engagement can strengthen institutional knowledge and memory and is critical for reform sustainability. Future operations could also more closely engage implementing and target departments in the design of program thematic and result areas and development and result indicators. 70. Field-based Task Team Leaders are critical for the success of such Programs. The fact that the two TTLs and most of the team members were field-based was critical for the quality and continuity of policy dialogue and Program supervision. It allowed the team to provide just-in-time support throughout the operation and quickly identify and resolve any implementation issues in partnership with the Government. 71. Selectivity was also important for the success of the Program. Rather than attempting to comprehensively address challenges across all sectors and services at once, the PforR was focused on few priorities based on government commitment and the status of Punjab’s public management systems. This allowed the operation to put in place the foundations for improved service delivery (supporting basic accountability and monitoring systems, revenue mobilization systems, and engaging Line Ministries) with realistic targets. A potential follow-up operation could further build on these achievements and set higher targets in terms of performance management, coordination and service quality. Page 25 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 72. The PforR demonstrated that “GovTech” solutions can be transformational for public sector performance. The innovative use of technology in the Program, e.g., the digitization of property records, the move toward a public procurement management information system and the use of smart phones and related dashboards for performance monitoring of public services and public investments, could develop tangible improvements for public sector performance. Such tools can provide a strong basis for evidence-based and timely decision-making while at the same time facilitating citizens’ access to public services by reducing information and transparency gaps. They also depend on strong political leadership and the ability to attract and retain high-quality ICT professionals in the public sector. . Page 26 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN Annex 1A. RESULTS FRAMEWORK (i) PDO Indicators Objective/Outcome: Improve transparency of targeted departments of the province of Punjab Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Number 50000.00 750000.00 750000.00 2916885.00 enquiries/comments received by Citizen Contact Center to 15-Nov-2013 30-Dec-2018 12-Jul-2017 30-Dec-2018 seek information about, or to provide feedback on key services Comments (achievements against targets): The indicator was fully achieved (exceeded). As per the Third-Party Validation (TPV) reports, enquiries and comments received by the Citizen Contact Center (CCC) reached over 2.9 million, including 967,909 inbound calls and 1,948,976 two-way SMS, which significantly exceeded the original target. The initial target in the Project Appraisal Document included only inbound calls. However, during implementation, it became clear that citizens preferred multiple channels of communication with the CCC, not least through text messages. This also meant that in the early years of the project, it was harder to achieve the targeted number of calls. Citizens' communication preferences were taken into account during the restructuring, which changed the target from "number of telephone calls received" to "number of comments/enquiries received" by the CCC. While it could be argued that not changing the initial target of 750,000 facilitated reaching the target, at the end of the Program, not only the combined enquiries and comments but also the inbound calls alone exceeded the initial target. In terms of substance, citizens approached the Citizen Contact Center mostly to seek information on application forms, process (eligibility criteria, documentation requirements, intermediary approvals required, turn-around time, delivery mechanism), applicable fees/taxes/levies, payment mechanisms, Page 27 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) office locations, and grievance redressal mechanisms. Indicators in future projects could continue to take into account different modes of communication. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Decision-makers using Percentage 0.00 60.00 100.00 reliable service delivery information for improved 30-Jun-2013 30-Dec-2018 30-Dec-2018 management measured as the percentage of reports that trigger managerial action Comments (achievements against targets): The indicator was fully achieved (exceeded). Department-wise service delivery information was generated on dashboards for supervisors (at district, division and provincial levels), which were populated through smart monitoring tools that were used for monitoring selected public services being provided in all districts of Punjab. These services relate to health, school education, and agriculture. The service delivery information has helped supervisors/managers take informed decisions for effective human resources management (e.g., schedule planning, staff encouragement, penalty or other corrective action). While the indicator was fully achieved according to the regular project reports, it was in retrospect not an ideal indicator for several reasons: "managerial action" and the basis of cases on which the percentage was calculated could have been more precisely defined. In addition, the nature of the indicator made the independent verification of its achievement difficult. Finally, the link between this indicator and the two objectives of the PDO (improved transparency and improved resource management) could have been stronger, making it more explicit whether the indicator would measure contribution to improved transparency, to improved resource management, or to both. For future similar projects, an alternative indicator could be considered. Page 28 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Objective/Outcome: Improve resource management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Urban immoveable property Number 4657.00 6977.00 10021.60 tax collection (in PKRs Million) 30-Jun-2013 30-Dec-2018 30-Dec-2018 Comments (achievements against targets): The indicator was fully achieved (exceeded). Urban immovable property tax (UIPT) collections in FY18 reached over 10 billion Pakistani rupees (PKR), more than double the baseline amount, and significantly surpassing the end-of program target of 6.9 billion PKR (Punjab Civil Accounts). The improved tax collection could be achieved despite a significant reduction in the property tax rate. The overall increase of 115% of collected amounts between the baseline and the end-of-program value is also significantly higher after taking into account inflation (inflation would have cumulatively increased the baseline amount by approximately 28% over the program period). (ii) Intermediate Results Indicators Results Area: Transparency and Access to Services Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Targeted organizations Number 0.00 75.00 84.00 publishing updated institutional information on 30-Jun-2013 30-Dec-2018 30-Dec-2018 their websites for Page 29 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) information of the public. Comments (achievements against targets): The indicator was fully achieved (exceeded). Targeted organizations, including provincial and division/district offices of the government departments and attached bodies, have published updated information about their policies, services, budget, staff, vacancies and contacts, etc. The achievement of 84 information on 84 websites surpassed the end-of-program target of 75. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Public investment projects of Percentage 0.00 75.00 77.00 targeted departments geo- mapped and published 30-Jun-2013 30-Dec-2018 30-Dec-2018 online. Comments (achievements against targets): The indicator was achieved. Of the total 320 approved public investment projects, 246 are being e-published by the targeted departments, including the School Education, Local Government and Community Development, Primary & Secondary HealthCare, and Specialized HealthCare & Medical Education departments. This represents 77%, slightly above the original target of 75%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Targeted departments with Number 0.00 7.00 8.00 automated records system for defined disclosable 30-Jun-2013 30-Dec-2018 30-Dec-2018 Page 30 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) information. Comments (achievements against targets): The indicator was fully achieved. For all disclosable information, departmental records are automated in 8 departments compared to an original target of 7. The 8 departments with an automated records system include the School Education Department, Local Government & Community Development Department, Primary & Secondary Health Department, Specialized HealthCare & Medical Education Department, Agriculture Department, Livestock & Dairy Development Department, Irrigation Department, and Higher Education Department. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Information about key Number 2.00 30.00 68.00 services available to the public through the Contact 30-Jun-2013 30-Dec-2018 30-Dec-2018 Center. Comments (achievements against targets): The indicator was fully achieved (exceeded). The Citizens Contact Center is providing information on 68 public services, more than double the original target of 30. In terms of substance, citizens mostly requested information about farming inputs, agriculture subsidy, immunization, hospital emergency services, vehicle registration, token tax, and professional services. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Services accepting online Number 1.00 13.00 14.00 Page 31 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) applications 30-Jun-2013 30-Dec-2018 30-Dec-2018 Services using online Number 3.00 7.00 15.00 processing. Services accepting mobile Number 0.00 10.00 10.00 applications and/or payments Comments (achievements against targets): The indicator was fully achieved. Province-wide, online application services are available for the following 14 services, compared to an original target of 13: College admission; Board of Intermediate and Secondary Education; E-Stamping; Government jobs and freelancing training; Punjab Residency Program for doctors; Nurses training admission; Teachers recruitment; Registration of visually impaired persons; Dealers Vehicle Registration; Registration of partnership firm with Industries, Commerce & Investment Department; Registration of shops or establishment with Labour & Human Resource Department; Registration of businesses with Punjab Employees; and Social Security Institution. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Functional facilitation Number 0.00 36.00 161.00 centers 30-Jun-2013 30-Dec-2018 30-Dec-2018 Functional facilitation Number 0.00 300.00 393.00 centers - using PPP model Page 32 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Comments (achievements against targets): The indicator was fully achieved (exceeded). There are total 161 functional facilitation centers in Punjab, including province-wide 152 Arazi centers at district/tehsil level (for issuance of property titles, land mutation) and 9 citizen facilitation centers at divisional level (also called e- Khidmat centers; providing domicile, birth/marriage/divorce certificate, driving license, e-stamp paper, construction permit, token tax payment, vehicle registration & ownership transfer, property title document, National Database and Registration Authority services, bank services). This largely exceeded the original target of 36 functional centers. Results Area: Performance Monitoring Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion District services being Number 18.00 180.00 216.00 monitored by smart management tools 30-Jun-2013 30-Dec-2018 30-Dec-2018 Comments (achievements against targets): The indicator was fully achieved (exceeded). The indicator measures the number of district services being monitored by smart management tools. Field workers and officials regularly report data about their activities related to service delivery through smart phones. This information is automatically populated on dashboards of field supervisors, enabling them to monitor and analyze the performance of personnel delivering field services. Other policy decisions are also informed by this system (e.g., targeted and timely measures on detection of dengue larva). For this indicator, 6 district services related to health, school education, agriculture, and livestock are being smart-monitored. Any public service that uses smart management tools in all 36 districts is counted as 36 (=1x36). At the end of the program a total of 216 services were monitored, which significantly exceeded the original target of 180. Page 33 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion District services for which Number 0.00 100.00 144.00 proactive citizen feedback is collected 30-Jun-2013 30-Dec-2018 30-Dec-2018 Comments (achievements against targets): The indicator was fully achieved (exceeded). Citizen feedback is proactively collected in all districts on agriculture services (Agri-smart, 2 services per district), on health services (eVaccs and Health Watch, 2 services per district), and on performance of field staff, reaching a total of 144 services and significantly exceeding the original target of 100 services. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion District services for which Number 0.00 180.00 180.00 performance information and citizen feedback is 30-Jun-2013 30-Dec-2018 30-Dec-2018 published online Comments (achievements against targets): The indicator was fully achieved. Performance information of 180 district services is published online in line with the original target. These services are related to agriculture (Agri Smart and compliance; 2 Services in each of the 36 districts), and health (E-Vaccs, HealthWatch and compliance; 3 services per district). http://www.monitoring.punjab.gov.pk/healthwatch/citizenFeedback Page 34 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) https://open.punjab.gov.pk/evaccs/ http://www.monitoring.punjab.gov.pk/evaccs/feedback/stats http://www.agrismart.pitb.gov.pk/open_public/ppmrp http://www.agrismart.pitb.gov.pk/open_public/farmers https://open.punjab.gov.pk/schools/ Results Area: Resource Management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Districts with functional Number 0.00 34.00 36.00 digital property registry 30-Jun-2013 30-Dec-2018 30-Dec-2018 Comments (achievements against targets): The indicator was fully achieved. All 36 districts of Punjab have the digital property registry, compared to the original target of 34. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Properties added to the Number 3227899.00 150000.00 155594.00 Page 35 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) property registry 30-Jun-2013 30-Dec-2018 30-Dec-2018 Comments (achievements against targets): The indicator was fully achieved. During 2018, 155,594 properties were added to the property registry, which surpassed the end-of-Program annual target of 150,000 by 4%. During the five years of the Program, a total of 1.08 million properties were added to the registry. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Targeted contracts processed Percentage 0.00 75.00 100.00 through basic e-procurement modules in targeted 30-Jun-2013 30-Dec-2018 30-Dec-2018 organizations. Comments (achievements against targets): The indicator was fully achieved (exceeded). 100% of targeted contracts are being processed through basic e-procurement modules in targeted organizations, significantly surpassing the original target of 75%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Targeted organizations using Number 0.00 100.00 153.00 PPRA MIS for targeted contracts 30-Jun-2013 30-Dec-2018 30-Dec-2018 Page 36 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Comments (achievements against targets): The indicator was fully achieved (exceeded). The Punjab Procurement Regulatory Authority's (PPRA) Management Information System (MIS) supports organizations in the better management of procurement performance, for increased comparability and transparency. The indicator measures whether the PPRA-MIS is implemented, and the targeted organizations are effectively using the system to carry out their procurements. As per available information, 153 targeted organizations are using the PPRA MIS for processing their targeted contracts, substantially exceeding the original target of 100 organizations. The number of organizations added to use PPRA MIS on annual basis are as follows: FY 2014-15: 32 organizations FY 2015-16: 68 organizations FY 2016-17: 34 organizations FY 2017-18: 19 organizations Total 2014-2018 (FY15-18): 153 organizations ANNEX 1B. DISBURSEMENT LINKED INDICATORS DLI I N0068 7433 TA BLE DLI 1: Targeted organizations publishing updated institutional information on their websites for information of the public. (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 Page 37 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Actual values 25.00 50.00 75.00 75.00 75.00 Allocated amount ($) 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 5,000,000.00 Disbursed amount ($) 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 5,000,000.00 Comments (achievements against targets): DLI1 was fully achieved and 100% disbursed. The targets refers to websites created by government departments as well as internal divisions and subordinated agencies to disclose institutional information to the public. Websites of divisions and agencies under the same department are available through the department’s website to ensure easy access for users to the services that interest them. In order to meet the requirements under the Program, organizations are required to publish updated institutional, organizational and budgetary information. The indicator also measures sustaining the maintenance and comprehensive disclosure of websites created during the previous year. Based on the Third Party Validation (TPV) report, 84 targeted organizations have published updated information about their policies, services, budget, staff, vacancies, and contacts, which surpasses the end-of-Program target of 75. DLI I N0068 7434 TA BLE DLI 2: Number of enquiries/comments received by the Citizen Contact Center to seek information about, or to provide feedback on key services (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Total Original values 50,000.00 0.00 0.00 0.00 0.00 0.00 Actual values 65,000.00 150,000.00 250,000.00 400,000.00 750,000.00 Allocated amount ($) 2,000,000.00 3,000,000.00 2,000,000.00 2,000,000.00 1,000,000.00 10,000,000.00 Disbursed amount ($) 2,000,000.00 3,000,000.00 2,000,000.00 2,000,000.00 1,000,000.00 10,000,000.00 Comments (achievements against targets): DLI2 was fully achieved and 100% disbursed. The DLI originally targeted 750,000 telephone calls received by the Citizen Contact Center (CCC) at the end of the Program. Reflecting the preference of citizens to engage with the CCC through various communication channels, not only Page 38 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) phone calls, on the request of the Government of Punjab, the definition of this indicator was changed during a Level 2 Project Restructuring. The indicator was revised from the ‘number of telephone calls received by the Citizen Contact Center to seek information about key services’ to the 'number of enquiries/comments received by the Citizen Contact Center to seek information about, or to provide feedback on key services', with retroactive effect from July 1, 2016. The adjustment was needed to take into account the preference of a large share of the public for obtaining information about public services through SMS and other forms of electronic communications, as well as the high number of calls and messages received by Citizen Contact Center (CCC), providing feedback on public service delivery. The change of the indicator was clearly explained and justified. While the change facilitated achieving the annual DLI targets at the time of the restructuring, at the end of the Program, the CCC had received over 960,000 telephone calls, which by itself exceeded the original target. In addition, over 1.9 million two-way SMS were exchanged. Based on the TPV report, the target for FY18 has been achieved with 2.9 million enquiries and feedback calls/SMS received (967,909 inbound calls and 1,948,976 two-way SMS) during FY18. The reported increase appears to have been a result of an intensified awareness campaign as well as reported citizen satisfaction with the handling of complaints through the CCC (about 78% of a sample of surveyed callers reported satisfaction on accuracy of information received and agent’s ability of understanding and responding to their queries). Based on this trend, the previous (400,000), current and final (750,000 calls and SMS) targets of the Program have been exceeded. The Citizen Contact Center was mostly approached to seek information on application forms, process (eligibility criteria, documentation requirements, intermediary approvals required, turn-around time, delivery mechanism), fee/taxes/levies, payment mechanisms, office locations, and grievance redressal mechanism. DLI I N0068 7435 TA BLE DLI 3: Services being monitored by smart management tools. (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Total Page 39 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Original values 18.00 0.00 0.00 0.00 0.00 0.00 Actual values 50.00 90.00 130.00 180.00 180.00 Allocated amount ($) 3,000,000.00 5,000,000.00 3,000,000.00 3,000,000.00 1,000,000.00 15,000,000.00 Disbursed amount ($) 3,000,000.00 5,000,000.00 3,000,000.00 3,000,000.00 1,000,000.00 15,000,000.00 Comments (achievements against targets): DLI3 was fully achieved and 100% disbursed. The indicator measures the number of district services being monitored by smart management tools. Field workers and officials regularly report data about their activities related to service delivery through smart phones. This information is fed into dashboards for field supervisors, enabling them to monitor the performance of personnel delivering field services. Other policy decisions are also informed by this system (e.g., targeted and timely measures on detection of dengue larva). The Third Party Validation (TPV) report and data of the Punjab Information Technology Board (PITB) confirm that the following six services were monitored in all 36 districts of Punjab: 1. PMIU (Program Monitoring Implementation Unit) for school education: regularly monitors 52,695 primary and secondary schools, including teacher presence, student attendance, and availability of facilities such as electricity, water and sanitation. 2. Regular inspection visits of primary and secondary healthcare facilities (presence of medical and paramedical staff, availability of medicines and supplies, condition of facilities). 3. Health Monitoring and Evaluation Assistant (regularly monitors a wide range of indicators for 124 district and city headquarter hospitals, including occupancy rates, functionality of equipment, availability of medicines and supplies, medical procedures conducted, disposal of medical waste, etc.). 4. E-vaccs (monitors daily field activities of vaccinators to ensure comprehensive coverage of infant immunization program). 5. Agri-smart (monitors daily field activities of agricultural extension staff). 6. Vet Inspect system monitors services delivered by veterinary staff to livestock farmers). DLI I N0068 7436 TA BLE DLI 4: Properties added to the urban property registry (Number) Page 40 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Total Original values 3,227,699.00 0.00 0.00 0.00 0.00 0.00 Actual values 97,000.00 133,000.00 138,000.00 144,000.00 150,000.00 Allocated amount ($) 3,000,000.00 5,000,000.00 3,000,000.00 3,000,000.00 1,000,000.00 15,000,000.00 Disbursed amount ($) 3,000,000.00 5,000,000.00 3,000,000.00 3,000,000.00 1,000,000.00 15,000,000.00 Comments (achievements against targets): DLI4 was fully achieved and 100% disbursed. As per the information shared by the Excise and Taxation (E&T) Department, the digitization of Urban Immovable Property Tax (UIPT) records has been completed in all 36 districts as of June 30, 2018. The Third-Party Verification (TPV) report confirms that the number of properties added to the registry during July 2017- June 2018 totals 155,594, exceeding the end-of- Program target of 150,000 properties. Data on the achievement of previous DLI targets, as confirmed by previous TPV reports, was as follows: Year 1: 108,667 properties added (target: 97,000); Year 2: 151,511 properties added (target: 133,000); Year 3: 138,714 properties added (target: 138,000); Year 4: 152,556 properties added (target: 144,000); Year 5: 155,594 properties (target 150,000). A total 1.08 million properties were added to the registry during the five years of the Program. Note: The targets for this DLI are annual. DLI I N0068 7437 TA BLE DLI 5: Targeted organizations using PPRA MIS for targeted contracts. (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 25.00 50.00 75.00 100.00 Allocated amount ($) 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 5,000,000.00 Disbursed amount ($) 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 1,000,000.00 5,000,000.00 Page 41 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Comments (achievements against targets): DLI5 was fully achieved and 100% disbursed. As per the Third-Party Verification (TPV) report and data shared by the Punjab Procurement Regulatory Authority (PPRA),w the DLI target was exceeded for the year. A total of 153 contracting organizations entered procurement contracts in the MIS compared to a target of 100. The DLI targets for the first four years have also been met. The number of organizations added to use PPRA Management Information System (MIS) on an annual basis were as follows: FY 2014-15: 32 organizations FY 2015-16: 68 organizations FY 2016-17: 34 organizations FY 2017-18: 19 organizations Total (FY15-18): 153 organizations The PPRA MIS supports organizations to better manage procurement performance for increased comparability and transparency. The indicator measures whether the targeted organizations are using the system effectively to carry out their procurements. ANNEX 1C. PROGRAM ACTION PLAN PAP_T BL Achieved Action Timing Completion Measurement (Yes/No) Planning and Development Department Due Date 30-Mar-2018 Yes Recommendations approved and implemented. will undertake an institutional review to determine how PITB can better implement Page 42 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) e-services and performance monitoring. The PITB Board and GoPb will take necessary steps to implement the recommendations. PITB financial rules, as authorized under Due Date 10-Oct-2014 Yes PITB financial rules, as authorized under the PITB the PITB Ordinance, are approved by the Ordinance, are approved by the PITB Board. PITB Board. PITB, PPRA, PRMP and E&TD shall appoint Due Date 10-Jul-2014 Yes Existing vacant positions as well as identified new and train key program fiduciary positions filled. management and procurement staff as identified in the staffing plan, with qualifications and on terms and conditions satisfactory to the Association. Implementation of the PforR Anti- Recurrent Quarterly Yes MIS system established. Corruption Guidelines (ACG) to aggregate, distill, process, report and disclose material and credible allegations related to fraud and corruption and including application of ACG guidelines on procurement. PPMR Program Steering Committee holds Recurrent Quarterly Yes Regular reports communicated. quarterly meetings on Program implementation issues. Page 43 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Clelia Kalliopi Helena Rontoyanni, Irum Touqeer Task Team Leader(s) Uzma Sadaf Procurement Specialist(s) Akram Abd El-Aziz Hussein El-Shorbagi Financial Management Specialist Zulfiqar Ali Raza Team Member Immanuel Frank Steinhilper Team Member Rahat Jabeen Environmental Specialist Babar Naseem Khan Social Specialist 1. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY13 104.101 301,108.48 FY14 24.624 -494,632.84 FY15 3.963 -139,432.39 Total 132.69 -332,956.75 Supervision/ICR FY14 44.674 740,476.86 FY15 36.570 329,488.02 Page 44 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) FY16 18.502 99,752.00 FY17 7.800 33,967.66 FY18 10.150 99,450.44 FY19 29.541 167,977.33 Total 147.24 1,471,112.31 Page 45 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 3. PROGRAM EXPENDITURE SUMMARY Estimates at Actual Expenditures (Disbursement) Source of Program Type of Co- Appraisal (in Percentage of Percentage Financing (US$) Financing Actual USD million) Appraisal of Actual World Bank IDA Credit 50.0 46.2* 71% 43% Government Borrower 20.22 61.7** 29% 57% Funds Other Partners N/A N/A N/A N/A N/A Total 70.22 107.9 100% 100% * The IDA credit was denominated in Special Drawing Rights (SDR). 100% of the credit of SDR 32.6 million was disbursed. The difference between USD46.2 million and USD50 million is due to exchange rate fluctuations between SDR and USD. ** The amount includes expenditure up to June 30, 2018, as reported in the Program’s Annual Audited Financial Statements received by the Bank for FY2017-18. Page 46 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 4. BORROWER’S COMMENTS Punjab Public Management Reform Program (PPMRP) Comments on World Bank’s Implementation Completion and Results Report (ICR) July 1, 201917 Punjab Public Management Reform (PPMR) Program was a five-year World Bank funded program implemented from 2013 through 2018 by PMU Punjab Resource Management Program, Planning and Development Board, Government of the Punjab. PPMRP focused on addressing critical constraints to service delivery. In particular, the Program aimed to: (i) Improve transparency and access to information about specific services. (ii) Strengthen performance monitoring systems and feedback-loops. (iii) Strengthen resource management systems. Following is P&D Board, Government of the Punjab’s, response to the draft Implementation Completion and Results Report (ICR) for the Punjab Public Management Reform Program: • Capability to Implement Programs for Results (PforR): The overall Satisfactory rating of the program reflects capability, understanding, and capacity of the Government of the Punjab to implement the Program-for-Results (PforR) instrument. PPMRP was the first implemented in Pakistan and South Asia and successfully completed through the PforR instrument. Given the success of PPMRP in the early years of its implementation, the World Bank gained confidence in government capabilities and started Jobs & Competitiveness (J&C) and Strengthening Markets for Agriculture and Rural Transformation (SMART) programs financed through the PforR instrument. • However, certain issues have been noted with the PforR mode. It is quite difficult to keep track of and link the development activities/DLIs with the amount received in Account I, as the Account is maintained by the Finance Department whereas the development activities come in the ambit of the Planning & Development Board. Secondly, many activities are funded from the non-development Budget, which is difficult to monitor. A mechanism has to be devised to streamline the monitoring of projects using a DLI mode. • Replication of Intervention in other Provinces: The program interventions were aimed to improve access to information, strengthening performance and resource management systems in targeted departments in Punjab. Inspired by improved services delivery and performance through the use of technology enabled interventions, other provinces approached the Government of the Punjab to get assistance regarding introduction of similar reforms. The Governments of Khyber Pakhtunkhwa, Balochistan and Gilgit Baltistan were assisted in the implementation of eVaccs, an application to monitor and manage field staff to increase the attendance and coverage of various vaccines. • Future Operations: The Government of the Punjab would like to take leverage from interventions introduced under PPMRP during World Bank funded future operations. Nevertheless, Government departments (including implementing agencies and target departments) may be thoroughly engaged while planning future operations to better define and design program thematic/result areas and development and result indicators that will help implement the program interventions in the best 17For formatting consistency, the Government of Punjab comments were slightly edited with respect to formatting and language, without any changes to the substance of the comments. Page 47 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) possible manner, leading to achievement of intended results impacting governance and service delivery. • Sustainability of Interventions: Feedback from Implementing Agencies (PPRA, ET&NC, and PITB) and Target Departments (School Education, Higher Education, L&DD, Agriculture, Primary & Secondary Health Care, Specialized Healthcare & Medical Education, Irrigation, and LG&CD) was obtained regarding continuity, sustainability and accountability of interventions under PPMRP. The agencies/departments reported that interventions are owned by the target departments and timely measures have been taken for budget allocation and converting these program interventions into SNE mode. In future operations, programs may be designed in a way that sustainability of the program interventions must be ensured and must be put in place in the target departments by the end of the penultimate year. • Future operations may also entail latest innovative technology. Technical Assistance (TA) and capacity building of the implementing and User departments may be addressed. Furthermore, incentivizing implementing departments may also be considered to encourage them for their efforts and commitment to successfully implement program interventions. Page 48 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 5. ECONOMIC AND FINANCIAL ANALYSIS Ex-Ante and Ex Post Economic Analyses 1. For the ex-ante economic analysis in the project PAD (2013), a benefit/cost analysis (BCA) was carried out where the benefits accrued from 3 components: Performance Monitoring; Provision of Information; and Digitizing Urban Immoveable Property Tax (UIPT). The BCA considered the following benefits accruing for the 3 Components (Table A5.1): • For Component A, Performance Monitoring benefits were calculated on the basis of the reduction of absenteeism among public servants multiplied by their various wage scales minus the replacement cost (assumed to be 30%) of hiring private temps. The hypothesis to calculate the benefits considered real public service wages. The Scenario analysis considered 20%, 25% and 30% reduction in costs over the duration of the project for the following subsets: o Health: General health supervision; Nutrition supervision; Vaccination; and Maternal and child health supervision. o Agriculture: Agriculture extension. o Livestock: General Livestock Supervision; Vaccination; and Artificial insemination. o Education: Education supervision; Data collection of administrative data of teacher and student attendance from schools. o Irrigation: Canal level data collection to gauge the flow of water down the canals to check water theft. • For Component B, the Provision of Information benefits were based on better access to information leading to the reduction of the cost incurred by citizens due to lack of information (forgone cost to obtaining not readily available information to citizens) and monetized by the citizens’ forgone extra cost. However, gained time, which was not explicitly considered in the benefits, could also have been considered and would have ultimately led to increased economic activities and leisure time. However, the cost saving hypothesis and methodology were not made explicit in detail in the PAD. The Scenario analysis considered 20%, 25% and 30% reduction in costs over the duration of the project for the following subsets: o Domicile and property registration cost. o Birth registrations and death certificates cost. • For Component C, UIPT register and tax collections considered the benefits in terms of increased tax collection which would eventually be partially allocated toward public investments. The hypothesis considered 2012 real property tax collection figures. The Scenario analysis considered 5%, 7% and 10% increase in total property tax revenues over the first 5 years and constant tax generation starting Year 6 over the next 5 years. Page 49 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 2. The results of the original and recalculated ex ante economic analyses18 with a set of 2 discount rates are illustrated in Table A5.1. Hence, the same dataset was used to calculate the operation’s viability using 0%, 6% and 10% discount rates for the 3 scenarios. Regardless of the discount rate used, the pessimistic scenario was not viable, while both the base and optimistic scenarios were viable. The recalculated ex-ante economic analysis resulted in a net present value (NPV) of US$4.6 million, instead of US$16.24 million as per the PAD.19 When using a 0% discount rate, the NPV reached US$14.6 million, with an Economic Rate of Return (ERR) of 20% and a Present Value Benefit over Cost (PV B/C) ratio of 1.3. Table A5.1: Ex Ante Economic Analyses Base Pessimistic Optimistic Component Case Scenario Scenario Scenario Original Ex Ante BCA Results Cost (US$ million) 50.0 50.0 50.0 Benefit (US$ million) 49.39 66.99 81.58 A-Performance Monitoring 10.8 13.6 16.3 B-Provision of Information 23.6 29.5 35.3 C-Resource Management 15.0 24.0 30.0 (UIPT with fiscal collection) NPV (US$ million) -0.61 16.24 31.58 Recalculated Ex Ante BCA at 0% Discount Rate NPV (US$ million) -2.1 14.6 28.6 ERR (±%) -3% 20% 41% PV B/C 0.96 1.29 1.57 Recalculated Ex Ante BCA at 6% Discount Rate used by the Bank starting 2016 NPV (US$ million) -5.1 7.7 18.6 ERR (±%) -3% 20% 41% PV B/C 0.88 1.18 1.43 Recalculated Ex Ante BCA at 10% Discount Rate used by the Bank prior to 2016 NPV (US$ million) -6.3 4.6 13.9 ERR (±%) -3% 20% 41% PV B/C 0.84 1.12 1.36 Note: Benefits for BCA with positive discount rates accrue between Year 2 and Year 5 for Components A and B and between Year 2 and Year 10 for Component C. Ex-Post Economic Analysis Benefits 3. The project achieved all its targets as per the Third-Party Verification (TPV) reports and data reviewed. Most of the benefits derived for the BCA are based on the results reported by the World Bank External Evaluation report of 2018.20 18Due to a computer crash, the exact underlying data of the original ex ante analysis in the PAD were lost and no longer available at the time of ICR preparation. In addition, the original analysis had not used any discount rates. The ex-ante analysis was therefore recalculated using different discount rates to allow for better comparison with the ex post analysis. 19 The NPV listed in the PAD Annex is US$ 16.24 million instead of US$ 16.99 million. 20 World Bank. 2018. Interim External Evaluation of the Punjab Public Management Reform Project Final Report prepared by DevTech Systems, Inc. Washington, D.C. Page 50 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) 4. The ex-post economic analysis represents a conservative estimate of project benefits, based only on inputs for which reliable evidence and data was available. Two sets of ex-post economic analysis were carried out: (i) BCA1 with the benefits accruing from Component C, i.e., incremental tax revenues as they were the only benefits readily available considered in the ex-ante BCA; and (ii) BCA2 which did not consider the same subsets accruing from the ex-ante analysis, because the specific indicators were not systematically captured and reported during the project. Although actual benefits have most probably accrued from all Project components, as estimated in the PAD, only the benefits reported that could be valued at the time of the ICR preparation were considered for the ex-post economic analysis. Ex post BCA 1 5. The benefits considered under the first ex post BCA1 are only those accruing under Component C. Discount rates of 6% and 10% are also considered: • For Component C, 1.08 million properties were added to the UIPT register and tax collections increased from Rs. 4.6 billion in 2013 to Rs. 10 billion in 2018 and significantly exceeding the target of Rs. 6.9 billion. The incremental benefits were considered for 2014-2023 in 2014 constant prices (5 first year actual tax collection and Year 6 to Year 10 and the tax collection of Year 5 from Year 6 to Year 10). A given US$ to Rs exchange rate was also considered for each end of year. 6. Component C benefits made the project highly profitable and significantly exceeded the results of the ex-ante economic analysis. The BCA1 results are illustrated in Table A5.2 with an NPV discounted at 0%, 6% and 10% reaching US$305 million, US$206 million and US$162 million, respectively. The baseline used for the ex-ante BCA was US$42 million in 2012, whereas the PAD monitoring indicator baseline was set at US$47.5 million in 2013 and was used for the ex post BCA calculation of the incremental tax revenues. Table A5.2: First Ex Post Economic Analysis Using only Component C in Terms of Benefits Component BCA1 at 0% Discount BCA1 at 6% Discount BCA1 at 10% Discount Rate Rate Rate Ex Post BCA1 Cost (US$ million) 50.0 50.0 50.0 Benefit (US$ million) 354.5 354.5 354.5 A-Performance Monitoring 0 0 0 B-Provision of Information 0 0 0 C-Resource Management (UIPT with fiscal 354.5 354.5 354.5 collection over 10 years) NPV (US$ million) 304.5 205.9 162.1 ERR (±%) >100% >100% >100% PV B/C 7.1 5.1 5.3 Note: Benefits for BCA accrue between Year 1 and Year 10 for Component C. Ex post BCA 2 7. The benefits considered under the second ex post BCA2 are those accruing under Component C as well as under Component A and derived from the Interim External Evaluation of the Punjab Public Management Reform Project (2018). These are only a sample from numerous benefits that could have Page 51 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) been calculated during and beyond the project duration. Discount rates of 6% and 10% are also considered. • For Component A, the benefits accruing from the improvement of waste collection are considered as an example. The waste collection effectiveness increased from 75% in October 2016 to 86% in January 2018 to reach 300,248.2 tons. Only waste collection benefits were considered in the BCA under Component A, as it is difficult to assess the treatment and landfilling of additional waste collected by the project, although a Waste-to-Energy (WtE) plant started operations in August 2018, which is too early to determine whether some of the additional collection was diverted to the WtE plant. Waste generation per capita per day ranges between 0.5 and 0.65 kg21 in Punjab (a mid-point between these two values was considered for the BCA). Collection benefits are based on the World Bank benchmark of 1 to 1.5 percent of yearly household disposable income (household income remaining after deduction of taxes and other mandatory charges) is commonly used, and considered affordable by households given a certain level of service.22 Pakistan’s Disposable Income amounted to US$ 650 per capita in 2016 and was maintained constant over 2017 and 2018 when waste collection improvement was registered.23 The net effectiveness rate of 11% was maintained in 2018 as recent figures have not been published. The improved effectiveness amounted to 38,404 tons per year. The benefit per year amounted to US$0.24 million. • For Component C, the same methods and figures are used as above, but benefits are accrued over 5 years only (2014-18), and they are expressed in 2014 constant prices. 8. The BCA2 results are illustrated in Table A5.3 with an NPV discounted at 0%, 6% and 10% reaching US$86 million, US$68 million and US$59 million respectively. Again, Component C alone makes the project highly profitable and significantly exceeds the results of the ex-ante economic analysis. Table A5.3: Second Ex Post Economic Analysis Using Additional Benefits Component BCA1 at 0% Discount BCA1 at 6% Discount BCA1 at 10% Discount Rate Rate Rate Ex Post BCA1 Cost (US$ million) 50.0 50.0 50.0 Benefit (US$ million) 135.6 135.6 135.6 A-Performance Monitoring 0.48 0.48 0.48 B-Provision of Information 0 0 0 C-Resource Management (UIPT with fiscal 135.1 135.1 135.1 collection over 10 years) NPV (US$ million) 85.6 68.2 59.1 ERR (±%) >100% >100% >100% PV B/C 2.7 2.63 2.57 Note: Benefits for BCA accrue between Year 1 and Year 5 for Component C. Masood, Maryam, Claire Y. Barlow and David C Wilson. 2014. “An assessment of the current municipal solid waste management system in Lahore, Pakistan.” Waste Management & Research 2014, Vol. 32(9) 834–847. Website: 22 Raich, Uri. 2009. The State of Solid Waste Management in Maputo, Mozambique: Presentation. World Bank. Washington, D.C. 23 CEIC website: . Page 52 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) Improving Project Efficiency 9. The project efficiency could be improved by building synergies between the conceptualization of the ex-ante BCA in terms of foreseen quantifiable benefits, project indicators and monitoring. Indeed, several other benefits could well have been calculated should a better integration of benefits accruing from the project and monitored indicators was judiciously considered at the onset to better gauge the project impact and outcome. For instance, the vaccination campaign could tremendously reduce the morbidity and mortality associated with infectious diseases whereas proper waste management could have numerous benefits (clean environment, opportunity for recycling and composting, reduction of the landfill areas, encouraging all forms of tourism, reducing communicable diseases, etc.). • For Component A, the Transparency and Access to Services was mainly implemented by PITB: indicators were considered for the number of enquiries and the means used. However, to derive benefits in terms of transaction cost reduced, the time needed to provide the service should have been accounted for to be compared with the ex-ante transaction cost. The case of vaccination is analyzed as an example: o Vaccination. Evacc vaccination coverage increased from 22% to 98% of the population. Vaccines provided by the health sector are monitored in real time24 and include: Bacillus Calmette–Guérin vaccine, primarily used against tuberculosis; Polio vaccines ; 5-in-1 vaccine, a combination of five individual vaccines intended to protect people from multiple diseases including Haemophilus influenzae type B, whooping cough, tetanus, hepatitis B and diphtheria; Pneumococcal vaccines are vaccines against the bacteria Streptococcus pneumoniae; Measles vaccine; and the rotavirus vaccine . However, the valuation of the vaccination effectiveness is difficult to calculate due to the lack of statistics. An attempt at quantifying the effectiveness of these vaccination campaigns could bring on significant benefits that could tremendously increase the project’s efficiency. • For Component B, the Provision of Information was mainly implemented by PITB and indicators were monitored in terms of the number of citizens' calls/feedback SMS to which has substantially increased from a baseline of 50,000 in 2013 to 2.9 million in 2018. However, a survey should be considered in the future to determine the time saved by citizens using these platforms improving the flow of information. Usually, transaction cost savings could be calculated in terms of time saved with/without these new platforms —Disposable Income per capita per hour saved multiplied by the number of users. 24 Punjab Government website: < https://open.punjab.gov.pk/evaccs/home/heat_map/antigens?month=02&year=2019>. Page 53 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 5. SUPPORTING DOCUMENTS • Project Appraisal Document • Project ISRs, Aide-Memoire’s and Management Letters • Program Mid-Term Evaluation • Third-Party Validation Reports • 2011/12 Pakistan Social and Living Standards Measurement Survey • 2012 Punjab Province PEFA Report • Power Point Presentations from Project Workshop • Project audited financial statements and external audit • Government of Punjab websites • Open Punjab web portal: https://open.punjab.gov.pk/ • Academic papers covering aspects of the program, e.g.: • Arman Rezaee, Ali Hasanain, Yasir Khan. Crowdsourcing government accountability: Experimental evidence from Pakistan. November 12, 2015. • Michael Callen, Saad Gulzar, Ali Hasanain, Yasir Khan. The Political Economy of Public Employee Absence. Experimental Evidence from Pakistan. June 24, 2013. • 2019 World Bank Group Poverty & Equity Brief: https://databank.worldbank.org/data/download/poverty/33EF03BB-9722-4AE2-ABC7- AA2972D68AFE/Global_POVEQ_PAK.pdf • Facebook Videos on PPMRP results: https://www.facebook.com/WorldBank Pakistan/videos/ • Beschel, Robert P.; Cameron, Blair James; Kunicova, Jana; Myers, C. Bernard. 2018. Improving Public Sector Performance: Through Innovation and Inter-Agency Coordination (English). Global Report Public Sector Performance. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/833041539871513644/Improving-Public-Sector- Performance-Through-Innovation-and-Inter-Agency-Coordination. • World Bank Group Country Partnership Strategy 2010-13. • World Bank Group, Pakistan Country Partnership Strategy, 2015-2020 (Report no. 84645-PK) approved by the Board on April 4, 2014, extended by the Performance and Learning Review (Report no. 113574) approved by the Board on June 15, 2017. Page 54 of 55 The World Bank Pakistan: Punjab Public Management Reform Program (P132234) ANNEX 6. RESULTS CHAIN/THEORY OF CHANGE Long-term Activities Outputs/Intermediate Outcomes Outcomes Outcomes (beyond (PDO) PforR) Result Area 1: Transparency & Access to Services Proactive disclosure of defined Targeted organizations publishing disclosable information, updated defined disclosable and establishing automated record information through websites management systems within key departments (for backend automation). Improved citizen access to key public services (virtually and in Geo-mapping and publishing online person) development schemes of selected departments Improved transparency of Providing process-oriented information targeted about key public services, using ICT departments of interfaces (helplines and SMS), managed the Province of by citizen contact center Punjab Efficient & Transparent Using ICT to electronically provide Public Service selected services (electronic application, processing and payment) Delivery in Punjab; Establishing facilitation centers to Improved Bank Funding: US$ 50 million provide selected public services under one roof and closer to citizens fiscal space, and effective Result Area 2: Performance Monitoring project Implementing a field-level performance management data augmentation system, using in Punjab; smartphones and dashboards, for key services in health, education and Improved agriculture Reliable performance and service Economic delivery information available to decision-makers and citizens and Social Using field-level performance reports and citizens’ feedback for decision making, Improved Indicators in and publicly disclosing performance resource Punjab information management of targeted Result Area 3: Resource Management departments of Developing a digital database of updated Digitized property registries the Province of (survey based) property records in Punjab functional at district level, and Punjab increase in urban property tax Developing and implementing a procurement Management Information Procurement MIS system used by System, digitizing procurement cost centers for targeted contracts documents for efficient monitoring and (efficient monitoring an evaluation evaluation; and piloting basic e- of procurements) procurement module Critical Assumptions: • Increased availability of reliable information, combined with IT-based performance monitoring systems, will (i) facilitate citizens’ access to public services and (ii) enable decision-makers to take timely and evidence-based decisions to improve service delivery performance. • Improved digitization of property records will lead to an increase in tax base and revenues. • The use of a procurement MIS system will increase transparency and compliance in public procurement while laying the foundation for increased cost efficiency in the future. Page 55 of 55