Document of The World Bank Report No: ICR2064 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-76950 IDA-37680) ON A CREDIT IN THE AMOUNT OF SDR 14.7 MILLION (US$ 20 MILLION EQUIVALENT) AND ON A LOAN IN THE AMOUNT OF EURO 10.5 MILLION (US$13.5 MILLION EQUIVALENT) TO SERBIA AND MONTENEGRO FOR A SERBIA HEALTH PROJECT AND ADDITIONAL FINANCING September 27, 2012 Human Development Sector Unit (ECSHD) South East Europe Country Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective September 2012) Currency Unit = Serbian Dinar 1.00 Dinar = US$ 0.011 US$ 1.00 = 89 Dinar FISCAL YEAR January 1 –December 31 ABBREVIATIONS AND ACRONYMS AF Additional Financing IT Information Technology AIDS Acquired Immunodeficiency Syndrome LIS Laboratory Information System ALOS Average Length of Stay M&E Monitoring and Evaluation CHIS Chambers of Health Institutions of Serbia MOH Ministry of Health CIDA Canadian International Development Agency MTR Mid-term Review CIS Central Information System NHA National Health Accounts CME Continuous Medical Education NHS National Health Survey CP Clinical Pathway PAD Project Appraisal Document CPG Clinical Pathways Guidelines PCU Project Coordination Unit CPS Country Partnership Strategy PDO Project Development Objective DILS Delivery of Improved Local Services PEDPL Public Expenditure Development Policy Loan DRG Disease Related Group PHC Primary Health Care DZ Dom Zdravlja PHRD Policy and Human Resources Development Fund EAR European Agency for Reconstruction PPEDPL Programmatic Public Expenditure Development Policy Loan EU European Union QER Quality Enhancement Review EC European Commission QSA Quality of Supervision GOS Government of Serbia SHP Serbia Health Project HIF Health Insurance Fund SOSAC Sector Adjustment Credit HIS Health Information System SPEAG Social Protection Economic Assistance Grant HIV Human Immunodeficiency Virus TB Tuberculosis HTA Health Technology Assessment TOR Terms of Reference IHIS Integrated Health Information System TSS Transitional Support Strategy IPH Institute of Public Health TTL Task Team Leader ICR Implementation Completion and Results Report WAN Wide Area Network ISR Implementation Status Report WHO World Health Organization Vice President: Philippe H. Le Houerou Country Director: Jane Armitage Sector Manager: Daniel Dulitzky Project Team Leader: Ana Holt ICR Team Leader: Anne Bakilana COUNTRY Project Name CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 8 3. Assessment of Outcomes .......................................................................................... 13 4. Assessment of Risk to Development Outcome......................................................... 21 5. Assessment of Bank and Borrower Performance ..................................................... 21 6. Lessons Learned ....................................................................................................... 24 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 26 Annex 1. Project Costs and Financing .......................................................................... 27 Annex 2. Outputs by Component ................................................................................. 28 Annex 3. Economic and Financial Analysis ................................................................. 31 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 31 Annex 5. Beneficiary Survey Results ........................................................................... 32 Annex 6. Stakeholder Workshop Report and Results................................................... 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 33 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 51 Annex 9. List of Supporting Documents ...................................................................... 51 MAP IBRD 34847R A. Basic Information Country: Serbia Project Name: Health Project (Serbia) IBRD-76950, Project ID: P077675 L/C/TF Number(s): IDA-37680 ICR Date: 09/27/2012 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: SIL Borrower: SERBIA AND MONTENEGRO Original Total USD 20.00M Disbursed Amount: USD 35.32 M Commitment: Revised Amount: USD 33.50M Environmental Category: B Implementing Agencies: Ministry of Health Co financiers and Other External Partners: not applicable B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/12/2002 Effectiveness: 10/10/2003 10/10/2003 Appraisal: 03/05/2003 Restructuring(s): - 06/30/2005 Approval: 05/22/2003 Mid-term Review: 09/30/2005 09/15/2006 Closing: 02/28/2008 03/31/2012 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Modest Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Moderately Quality at Entry: Government: Unsatisfactory Satisfactory Moderately Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Moderately Overall Borrower Moderately Performance: Satisfactory Performance: Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Health 90 90 Non-compulsory health finance 10 10 Theme Code (as % of total Bank financing) Health system performance 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Philippe H. Le Houerou Johannes F. Linn Country Director: Jane Armitage Orsalia Kalantzopoulos Sector Manager: Daniel Dulitzky Armin Fidler Project Team Leader: Ana Holt Loraine Hawkins ICR Team Leader: Anne Bakilana ICR Primary Author: Anne Bakilana F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) To build capacity to develop a sustainable, performance oriented health care system where providers are rewarded for quality and efficiency and where health insurance coverage ensures access to affordable and effective care. Revised Project Development Objectives (as approved by original approving authority) The PDO was not revised. (a) PDO Indicator(s) Original Target Formally Actual Value Baseline Values (from Revised Achieved at Indicator Value approval Target Completion or Target documents) Values Years Indicator #1: Hospital master plan and planning standards and guidelines adopted. Decree on The Gov has prepared a Health Care Decree on Health Care Institutions Plan and planning institutions networks network standards adopted. N/A Plan (Official Gazette plan - 42/ 2006 defining a new Official organizational model Gazette for all hospitals. 13/97 Date achieved 05/22/2003 2006 This indicator lacked specificity as to which planning standards or guidelines were to be adopted. However, based on the experience of the 4 hospitals that the project supported, a new organizational model for Comments hospitals has been prepared and decreed. A number of other guidelines have been adopted including the IT Guidelines on Investments; Clinical Pathways Guidelines; Human Resources Plan, these are discussed under other indicators below. Indicator #2: In the four general hospitals participating in the project. i) Reduction in financial arrears RSD 15.5 billion in Value - quantitative/ qualitative No baseline No arrears N/A arrears Date achieved 05/22/2003 2010 The target of no arrears was not achieved. This was an indicator outside the scope of the project. Instead, the project tracked the total amount of arrears in health care facilities not just the 4 project supported hospitals. Comments Trend data show the following: 2005: RSD 11 Billion paid to cover debt over previous 15 years; 2006: RSD 5.6 Billion surplus; 2007: RSD 4.5 Billion surplus (as of August 2007); 2008: RSD 0.811 Billion surplus; 2010: RSD 15.5 Billion in arrears. ii) Reduction in bed numbers Project Project supported supported hospitals: hospitals: 2, 906 Reduction, no 2, 312 Value - quantitative/ qualitative N/A numerical target. All hospitals: All hospitals: 15, 601 15, 253 Date achieved 12/31/2003 12/28/2009 07/15/2011 For the 4 project supported hospitals, there has been a reduction in the number of beds as shown in these data: 2004: 2, 658; 2005: 2, 142; 2006: 2, 305; 2007: 2, 316; 2008: 2, 344; 2010: 2, 312. Comments For all hospitals target was met, there has been a reduction on the numbers of beds as shown in these data: 2005: 14, 997; 2006: 14, 575; 2007: 15, 514; 2009: 15, 401; 2010: 15, 248. iii) Reduction in average lengths of stay (ALOS). For the 4 project supported hospitals ALOS declined to 7.52 No numerical target in 2010. Value - quantitative/ qualitative No baseline. N/A values. For all hospitals, ALOS declined to 7.21 in 2010. Date achieved 05/22/2003 2010 Target met. For the 4 project supported hospitals ALOS declined: 2004: 7.86; 2005: 7.65; 2006: 7.3; 2007: 7.43; 2008: 7.21; 2010: 7.52. Comments Target met. For all hospitals, ALOS declined from 8.57 in 12/31/2003 to: 2005: 8.15; 2006: 7.57; 2007: 7.33; 2009: 7.33; 2010: 7.21 iv) Increase in bed occupancy. For the 4 project supported hospitals: No numerical target 71.7% in 2010 Value - quantitative/ qualitative No baseline. N/A values. For all hospitals in Serbia: 69% in 2010 Date achieved 05/22/2003 2010 For the 4 project supported hospitals: 2004: 64%; 2005: 69%; 2006: 69%; 2007: 64%; 2008: 63.84%; 2010: 71.7%; Comments For all hospitals in Serbia: 2005: 69%; 2006: 69%; 2007: 64.7%; 2009: 67.4%; 2010: 69.0% v) Reduction in flows of area patients to tertiary care. Value - quantitative/ qualitative No baseline. No target values. N/A See comment below. Date achieved 05/22/2003 2010 This indicator as defined was difficult to track. However, the ICR team held discussions with teams at hospitals supported by the project and the Comments overwhelming view was that there has been a reduction in flows of area patients to tertiary care given the emphasis and improvements at PHC level. vi) Increase in use of primary care and output patient services. Value - quantitative/ qualitative No baseline. No target values. N/A N/A Date achieved 05/22/2003 This indicator was not tracked. Indicator was ill-defined, it had no Comments baseline or target and there was no mechanism in place to measure it. Indicator #3: HIF and MOH increase the number of staff trained in health policy, finance, and management, and use their skills in review of the basic benefits package, public private mix, resource allocation and provider payment systems; a critical mass remains in post. No specified No numerical target 6,223 people received Value - quantitative/ qualitative numerical N/A value. training. baseline. Date achieved 12/31/2003 03/31/2012 Originally no specified target. The project team later set a target of 5, Comments 000. This indicator was not very well defined to include a baseline of the number of staff in various MOH departments. The project team has monitored the number of people who have received various types of training. A total of 6223 people received training; 2004: 9; 2005: 1304; 2006: 2835; 2007: 1075; 2008: 500; 2009: 500. The ICR team met with various MOH, PCU, HIF, IPH, and Agency for Accreditation, hospital management, and they all acknowledged the usefulness of training received which allowed them to accomplish all the work supported by the project. Indicator #4: Contracts and payment methods for hospitals participating in the project provide improved incentives for efficiency and quality of care; are consistent with restructuring plans; and avoid perverse incentives. Payment methods for Input based hospitals have been Value - quantitative/ qualitative Qualitative target. N/A system. redesigned and will be implemented in 2012. Date achieved 05/22/2003 03/31/2012 At start of implementation the hospital budget was input based. By project end capitation formula for PHC has been developed through EU project, registration is ongoing, with plans to start using capitation Comments 07/2012, Law on Amended salaries was approved 12/2011. Preparations are ongoing for change in reporting according to DRG in 2012. Details are found in main text para. 46, 47, 61. Indicator #5: MOH/ HIF monitor and analyze at least annually the revenue, expenditure and arrears of the HIF and public healthcare institutions, and out of pocket expenditure on health by the population. No target value, qualitative target to HIF monitors and demonstrate analyses revenue and Value - quantitative/ qualitative No baseline N/A increased capacity expenditures of health for monitoring and institutions. analysis. Date achieved 12/31/2003 12/28/2009 Target has been met. The HIF has been producing analyses and reports Comments since 2008. These reports are produced regularly. Indicator #6: Policy and plan adopted for financing, resource allocation and provider payment methods for public health services, with the aim of increasing the effectiveness of disease prevention and health promotion. Discussion Adoption of policy Policy and plans for regarding Value - quantitative/ qualitative and plans for N/A financing reforms are in financing financing reforms. place. reforms. Date achieved 05/22/2003 03/31/2012 See indicator # 4 above. Plans for reforming financing, resource allocation and provider payment methods for public health services, with the aim of increasing the effectiveness of disease prevention and health Comments promotion have been designed and tested. Use of the new financing methods will begin in 2012. Details are found in main text paras. 46, 47, 61. Indicator #7: Licensing and recertification program for doctors in place. Value - quantitative/ qualitative No licensing Medical N/A All medical system is in professionals are professionals are place. registered in registered in Medical Chambers. corresponding Medical Chambers. Date achieved 12/31/2003 12/28/2009 Target has been met. All medical professionals are registered in Comments corresponding Medical Chambers. Indicator #8: New central HIS information system for Serbia in use and meta data bases in use to allow coherent approach to HIS development; regional integrated HIS used by management in all participating health institutions. Database of Database of insures Database of insures has insures is is complete and been completed and fragmented allows for better does allows for better and HIF Value - quantitative/ qualitative collection of payroll N/A collection of payroll branches are contributions and contributions and better not better access to access to health connected to health insurance insurance HQ Date achieved 12/31/2003 12/28/2009 7/15/2011 This target has been met. The central HIS system allows the HIF to host a database of insures, allows for better collection of payroll contributions Comments and better access to data that allows for analyses of revenue and expenditures of health institutions. See indicator # 5 above. Indicator #9: MOH, HIF, IPH/ISM conduct/ publish regular analytical reports on performance of health system and health sector institutions, using the WHO WHR 2000 categories, and local measures agreed with IDA. Key players publish regular analytical reports on performance Value - quantitative/ qualitative No baseline. No target values. N/A of health system and health sector institutions Date achieved 12/31/2003 7/15/2011 Target was met. NHA Unit established in IPH. National Health Survey carried out and Final report of NHS prepared and disseminated; NHA Comments published for 2003, 2004, 2005, and 2007. NHA data posted on IPH Website within Statistics and Analytical Report under Centre for Informatics and Biostatistics. (b) Intermediate Outcome Indicator(s) Actual Value Original Target Formally Baseline Achieved at Indicator Values (from Revised Target Value Completion or Target approval documents) Values Years Component 1: Health Services Restructuring Indicator #1: Hospitals master planning standards and implementation strategy completed. Decree on Plan and planning The Gov has prepared Value - quantitative/ qualitative N/A Health standards adopted a Decree on Health Care Care institutions Institutions networks Plan network (Official Gazette 42/ plan – 2006 defining a new Official organizational model Gazette for all hospitals. 13/97 Date achieved 12/31/2003 See PDO indicator #1 above. Based on the experience of the 4 hospitals Comments that the project supported, a new organizational model for hospitals has been prepared and decreed, and other guidelines have been adopted. Indicator #2: Number of health services and public health managers trained. Originally no No No numerical target specified target. The Value - quantitative/ qualitative N/A baseline. value. team set a target of 5,000 Date achieved 12/31/2003 03/31/2012 See PDO indicator #3. The number of people who have received various types of training was 6, 223. Meetings for ICR preparation Comments found acknowledgement of the usefulness of training received allowing officials to accomplish he work supported by the project. Indicator #3: Hospital restructuring and rehabilitation implemented in four regional hospitals (Kraljevo, Valjevo, Vranje and Zrenjanin) Restructuring and Restructuring and rehabilitation rehabilitation of 4 Value - quantitative/ qualitative 0 N/A completed for 4 hospitals has been hospitals. completed. Date achieved 12/31/2003 12/28/2009 7/12/2011 This target was met, for details of rehabilitation see paras. 48, 50 in Comments main text. Component 2: Health Services Restructuring Indicator #1: HIF/ MOH establish and maintains a unit located in the HIF of at least 4 staff ; responsible for reviewing the benefits package, public private mix, resource allocation and the provider payment system; staff in this unit are trained in health policy, economics, management, epidemiology or other relevant disciplines. HIF and MOH have Review and analyses No trained, and are of components of baseline adequately staffed Value - quantitative/ qualitative the health care N/A was with skills in health system are defined. policy, finance, and completed. management. Date achieved 12/31/2003 03/31/2012 HIF and MOH staff has been trained, and are adequately staffed with skills in health policy, finance, and management, and use their skills in review of the basic benefits package, public private mix, and resource Comments allocation and provider payment systems. MOH/ HIF staff has overseen the National Health Survey and report of NHS prepared and disseminated; HIF monitors and analyses revenue for 2008 and expenditures of health institutions. Arrears are measured. Indicator #2: Licensing body is established, staffed and is operating a licensing and re- licensing system for health professionals with the aim of ensuring quality and up-to-date clinical skills. All medical professionals are registered in corresponding No Medical Medical Chambers. Licensing professionals are MOH gave Value - quantitative/ qualitative N/A system in registered in responsibility of place Medical Chambers licensing and registration of doctors to Chambers and MOH can request private sector. Date achieved 12/31/2003 7/15/2011 Comments Target has been met. Indicator #3: Accreditation body is established and staffed, and plan adopted for developing Accreditation processes with the aim of ensuring safety and minimum quality standards in all healthcare providers, and encouraging quality improvement. Accreditation system Quality of Accreditation is developed, health care system is developed, operational and Value - quantitative/ qualitative is not N/A operational and institutionalized. 17 being institutionalized hospitals undergone measured accreditation. Date achieved 12/31/2003 7/15/2011 Comments Target has been met. Indicator #4: New HIS national standards and data bases in place; pilot of integrated HIS carried out in one area, and rolled out to further three areas. Database of insurees Database of insurees is is completed and fragmented New information allows better and there system contribute to Value - quantitative/ qualitative collection of payroll N/A is no improve collection contributions and network and coverage better access to among HIF health insurance branches with HQ Date achieved 12/31/2003 7/15/2011 Target has been met. A new Central Information System for Serbia Comments allows for coherent approach to health insurance development. Indicator #5: MOH implements communications strategy to build constituencies for planned reforms; and inform decision-makers about public, patient and health sector staff opinion of reform and health system performance. No Two information Second information communic campaigns inform campaign Value - quantitative/ qualitative ation patients and doctors N/A implemented, third strategy in of health sector information campaign place reform under preparation Date achieved 12/31/2003 7/15/2011 There is increased capacity in communication: MOH implements Comments communication strategy to build constituencies for planned reforms Component 3: Project Management, Monitoring and Evaluation Indicator #1: Project outputs are produced on time and on budget; PCU shows proactivity in solving problems and seeking to achieve outcomes. Project outputs were produced on time, No project Timely production PCU showed Value - quantitative/ qualitative monitoring of project outputs, N/A proactivity in solving in place. within budget. problems and seeking to achieve outcomes. Date achieved 12/31/2003 03/31/2012 This target was met. As detailed in section 5.2 b of main text, after initial delay in implementation, the PCU managed the project in a Comments satisfactory manner, outputs were timely and the PCU was very proactive in seeking to achieve outcomes. Indicator #2: Project monitoring system is in place and reporting six monthly (quarterly financial reports). Financial No project Satisfactory project management for the Value - quantitative/ qualitative monitoring financial N/A project was system. management. Satisfactory. Date achieved 12/31/2003 03/31/2012 Though financial management was for a short period less than Comments satisfactory at MTR, implementation progress in general and project management in particular were by and large rated satisfactory. Indicator #3: Effective communication and working relationships developed with key representatives in the four areas participating in health services restructuring. Effective working relationships were Value - quantitative/ qualitative N/A N/A N/A developed and nurtured. Date achieved 12/31/2003 03/31/2012 This is an indicator that is difficult to measure and track. ICR mission did confirm that there was effective communication between the Comments project’s management and the management of the hospitals and other health care management participating in the project. See indicator #4 below. Indicator #4: Evaluation of Kraljevo initiatives carried out and lessons disseminated for other restructuring initiatives and development of national master plan. Evaluation and Value - quantitative/ qualitative N/A lessons See comment below. disseminated. Date achieved 12/31/2003 03/31/2012 Anecdotal evidence from ICR Mission shows that the Kraljevo site was Comments by all measure a Hospital that provided a learning ground for other hospitals supported by the reform process. Other hospital managers visited the hospital to learn how Kraljevo was implementing reforms under an administrator who was a real champion for improvements in delivery of care at the hospital. A review of capitation pilot conducted in cooperation with the HIF was used as the basis for revisions of the capitation formula. Indicator #5: End of project evaluation of all components, with input from stakeholder workshops. Value - quantitative/ qualitative N/A - N/A Date achieved End of project review was undertaken by the MOH, and is attached under annex 7: a project closing workshop was held at Zrenjanin hospital during the Dec 2-9, 2011 implementation support mission. The Comments ICR preparation mission met with various stakeholders and others who benefitted from the project. The views garnered show that this was a project that has made positive contribution to the sector. Indicator #6: Annual and final project audit. Project audits have Value - quantitative/ qualitative None. - N/A been completed in a timely manner. Date achieved 12/31/2003 03/31/2012 Comments Target achieved. Additional Financing Indicators (a) PDO Indicator(s) Original Target Formally Actual Value Achieved Baseline Indicator Values (from Revised at Completion or Target Value approval documents) Target Values Years Indicator #1: Standards and guidelines are used to make decisions on investments in information technology. No Investments in IT Guidelines and guidelines are made in standards have been Value - quantitative/ qualitative exist for accordance with IT N/A developed and all Investments standards and investments follow standards guidelines these guidelines. Date achieved 02/03/2009 12/28/2009 7/13/2011 Comments This target has been met. Indicator #2: Ministry of Health safeguards quality of services by introducing internationally certified Clinical Practice Guidelines for clinical pathways. Guidelines for preparation 6 CPGs revised or 7 CPGs have been Value - quantitative/ qualitative of CPGs are developed according N/A revised and Guidelines ready but to Guidelines. completed. yet to be certified Date achieved 02/03/2009 12/31/2011 9/30/2011 Comments This target has been met. Indicator #3: Ministry of Health publishes regular review of performance of hospitals. Review of hospital No review Hospitals infections performance on hospital survey has been Value - quantitative/ qualitative published for N/A performance completed for 60 hospitals supported is published hospitals. by project. Date achieved 6/25/2010 12/31/2011 Comments This target has been met. (b) Intermediate Outcome Indicator(s) Actual Value Original Target Formally Baseline Achieved at Indicator Values (from Revised Target Value Completion or Target approval documents) Values Years Component 1: Information Technology: The cost-effectiveness of investments in information technology in the health sector is demonstrated and mechanisms are in place which facilitate and promote such investments Indicator 1: Modern, standardized, nationwide framework for medical informatics established. 3 standards were adopted in 11/2009(Standards No At least 3 standards for PHC and Hospital Value - quantitative/ qualitative standards N/A adopted by MOH Information System, adopted National Coding systems, Minimal data sets Date achieved 02/03/2009 12/31/2011 11/2009 Comments This target has been met. Indicator 2: Number of hospitals with fully upgraded health information system. HIS software All hospitals implemented in 18 Value - quantitative/ qualitative 0 N/A supported by project hospitals and 2 institutes Date achieved 02/03/2009 12/31/2011 3/15/2012 Comments This target has been met. Indicator 3: Proportion of health institutions connected to the IT Network. Connected to IT network i.e. connected to the main Value - quantitative/ qualitative 30% 80% N/A network node which will be housed in ICT unit of MOH. Date achieved 02/03/2009 12/31/2011 7/13/2011 Comments The target has been met. Component 2: Hospital Management and Quality: System is introduced to improve quality of services and measure performance of health care providers Indicator #1: Number of common clinical pathways prepared by at least nine pilot hospitals. 6 CPs in 9 hospitals, 4 CPs in 1 hospital, Value - quantitative/ qualitative 0 3 N/A 2 CPs in 2 hospitals, 1 CP in 4 hospitals Date achieved 2/3/2009 12/31/2011 03/15/2012 Comments The target was met. Indicator #2: Proportion of general hospitals reporting according to DRG classification. DRG by law has been adopted, software license purchased and Value - quantitative/ qualitative 0% 50% N/A tested, anticipation that hospitals will begin using DRG system in 2012. Date achieved 2/3/2009 12/31/2011 03/31/2012 This target is likely to be met. The preparation needed to make hospital Comments DRG reporting possible have been accomplished. Component 3 -Monitoring and Evaluation and Communication in MOH and HIF: Capacity strengthened to monitor and evaluate performance of health care providers Indicator #1: Evaluation prepared by the Ministry of Health and the Health Insurance Fund of hospitals with new reporting system. Evaluation is No evaluation Value - quantitative/ qualitative 0 prepared by project N/A performed. end. Date achieved 2/3/2009 12/31/2011 03/31/2012 Capacity in this area is still being built in order to support the new Comments reporting system once it has been introduced. G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 06/18/2003 Satisfactory Satisfactory 0.00 2 12/30/2003 Unsatisfactory Unsatisfactory 0.00 3 06/24/2004 Satisfactory Satisfactory 0.44 4 12/16/2004 Unsatisfactory Unsatisfactory 0.65 5 12/22/2004 Unsatisfactory Unsatisfactory 0.65 6 06/04/2005 Unsatisfactory Unsatisfactory 0.90 7 12/16/2005 Unsatisfactory Satisfactory 1.77 8 06/27/2006 Satisfactory Satisfactory 6.26 9 11/13/2006 Satisfactory Satisfactory 10.46 10 06/20/2007 Satisfactory Satisfactory 15.18 11 12/20/2007 Satisfactory Satisfactory 19.12 12 03/27/2008 Satisfactory Satisfactory 19.96 13 07/14/2008 Satisfactory Satisfactory 20.87 14 06/02/2009 Satisfactory Satisfactory 21.95 15 12/28/2009 Satisfactory Satisfactory 22.11 16 06/29/2010 Satisfactory Satisfactory 23.59 17 01/26/2011 Satisfactory Moderately Satisfactory 23.94 18 12/10/2011 Satisfactory Satisfactory 27.98 19 03/31/2012 Satisfactory Satisfactory 31.86 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Project complexity and lack of implementation progress. PDO was not 06/30/2005 U S 1.01 Restructuring to improve changed project design and more closely link activities to the PDO. I. Disbursement Profile Disbursement Summary(in US s Millions) - - Original - - - - Formally Rev ised - - A ctual 40 ....-------- 30 - " .!! 20 .. = :E ~ ::> 10 0 .,. .... .,. .,. .,. .... .,. .... .,. .... .,. ... .,. .... .. .. .. .... .... Cl .. .. .. .. .. .. .. .. - .. .. .. .. - .. - .. "' N N .,. .,. .., - .. .. - .. .. - .. Cl Cl Cl Cl Cl Cl .., Cl Cl Cl Cl Cl Cl Cl Cl Cl Cl Cl Cl .... .. .... .... .... "' .... N "' "' N N .... .... N N N N e. e. .... .... N - .... N .... N - N N 1. Project Context, Development Objectives and Design 1. The Serbia Health Project was approved on May 22, 2003. The credit was signed on June 13, 2003 and became effective on October 10, 2003. 1.1 Context at Appraisal 2. At the time of project appraisal, the Republic of Serbia was still part of the two member state union of Serbia and Montenegro that came into being in 2003 (and later peacefully dissolved in 2006) 1 2. In 2002, the Gross Domestic Product (GDP) per capita was estimated at $2,200, and poverty affected about 14.6 percent of the population. The country has had years of political stability and, until the recent economic crisis (which led to a drop in real GDP of 3.5% in 2009), economic stability that had led to economic growth and reduction in poverty through strong increases in incomes. In 2010 Serbia had a GDP per capita of approximately US$5,150. According to the latest Country Partnership Strategy (CPS), the government is committed to fiscal discipline, further economic reforms and to European Union (EU) accession 3. 3. Despite the difficult conditions that existed in the 1990s, vital health indicators did not show adverse regression in health status during this time. A study conducted by UNICEF showed that childhood mortality rates continued to decline and infant mortality stood at 11.23 deaths per 1000 live births in 2000. In 2003, life expectancy at birth stood at 69.8 years for men and 74.5 years for women. A significant proportion of morbidity and mortality was due to chronic illness such as heart disease, strokes and cancer, with smoking being responsible for 30% of mortality in the country. Access to basic services such as improved drinking water and sanitary disposal of human waste was universal. 4. On the other hand, the Serbian health care system showed characteristics of high inefficiencies that jeopardized macroeconomic stability. The system, like many others in the region, was budgeted using an input based system creating overcapacities and large inefficiencies. The Serbian health care system was, and is still, financed through a combination of public (through the Serbia Health Insurance Fund (HIF) financed by compulsory social insurance) and private out of pocket financing. At appraisal, public spending on health stood at 6% of GDP and was among the highest in the region, and total health expenditure stood at 11% of GDP, a figure that was regionally comparatively higher and close to that in high income countries. On the other hand, Serbia had an advantage over other countries in the region by having a public health financing system that was funded by mandatory contributions to the HIF and not from the 1 World Bank. 2003. Project Appraisal Document (PAD) on a Proposed Credit in the amount of SDR 14.7 Million to Serbia and Montenegro for a Serbia Health Project. 2 The proposed project covered Serbia only and did not include Montenegro or Kosovo. 3 World Bank and International Finance Corporation. 2011. Country Partnership Strategy for The Republic of Serbia for the period FY12-FY15. 1 budget and hence having some experience in contracting providers for health care services. However, this system of financing was facing serious challenges including: administrative complexity and inefficiency from separate contribution laws and revenue collection; existence of non independent branches of the HIF; and poor financial performance, which threatened the fiscal sustainability of the HIF. HIF’s expenditures could have been better managed by providing incentives to providers to increase efficiency and pay closer attention to pharmaceuticals prescriptions patterns. The sector also suffered from other inefficiencies including over-capacity in the hospital sector compared to utilization and also a monopoly and poorly controlled pharmaceuticals sector. The other area that was at that time considered out of sync with a well performing health sector was the need to look at the level of entitlements in the package of benefits which, for example, included treatment abroad and needed to be brought in line with available resources. 5. In 2002, the Government prepared a Health Care Statement which indicated directions to be pursued in reforming the health sector, and were the basis for the World Bank support through the SHP. The Statement included the following guiding principles: (i) organization of the health care delivery system into functional levels that ensure affordable and effective service; (ii) equal availability and access to basic health care services for all citizens and financial coverage for these services from the HIF; (iii) basic health care services to be selected on the basis of cost-effectiveness; (iv) high priority to be placed on preventive and primary health care services; (v) increased involvement of the private profit and nonprofit sector in the delivery of HIF-financed health care; (vi) the HIF to continue to be the main resource base for the financing of health care, with an expansion of the role of supplementary health insurance and private insurance schemes; (vii) development of a master plan that will categorize health care institutions as a step towards decentralized planning, management and delivery of health services; (viii) separation of the role of users, payers and providers; and (ix) promotion of quality and monitoring of services and facilities through a quality assurance and licensing system. 6. The proposed project had strong links with Country Partnership Strategy. At appraisal the Bank’s strategic program with Serbia and Montenegro was outlined in the 2001 (updated 2002) Transitional Support Strategy (TSS) which had four development objectives: (i) restoring macroeconomic stability and external balance; (ii) stimulating near-term growth and creating the basis for a sustainable supply response; (iii) improving the social well-being of the most vulnerable and building human capacity; (iv) improving governance and building effective institutions. The health investment project aimed to influence objectives (iii) and (iv). The project aimed to contribute to improvements in social well being of the most vulnerable, building human capacity and effective institutions, and improving governance. The objectives of project fitted well with the overall strategic program to assist the GOS in its economic development program. The design of the project in terms of the areas it aimed to tackle carefully reflected areas in need of investments and capacity building in order to reduce inefficiencies in the system. 2 1.2 Original Project Development Objectives (PDO) and Key Indicators 7. The original project development objective according to the main text of the Project Appraisal Document (PAD) was to assist the Borrower to build capacity to develop a sustainable, performance oriented health care system where providers were rewarded for quality and efficiency and where health insurance coverage ensured access to affordable and effective care. And according to the Development Credit Agreement: “the objective of the Project was to build capacity to develop a sustainable, performance- oriented health care system by restructuring health care services and improving financing, management and decision making in the Serbian health care systemâ€? 4. 8. The key performance indicators were: (i) reduction in arrears, reduction in bed numbers and average length of stay; increase in hospital bed occupancy; reduction in flows of patients from the area to tertiary care; increase in use of primary care and outpatient services in the four general hospitals participating in the project; (ii) HIF and Ministry of Health (MOH) increase the number of staff trained in health policy, health finance, or health management, and use their skills in review of the basic benefits package, the public/private mix in health services, resource allocation and provider payment systems; and a critical mass of these staff remain in post; (iii) MOH, with input from other relevant Serbian health sector agencies, monitors and analyzes at least annually the revenue, expenditure and arrears of the HIF and public health care institutions; the distribution of HIF expenditure per capita in different areas and among vulnerable and other population groups (taking account of risk factors); and out-of-pocket expenditure on health by the population; and (iv) MOH, with input from other relevant Serbian health sector agencies, publishes regular analytical reports on the performance of the health system and health sector institutions, using data from enhanced health information systems, and using the WHO World Health Report 2000 categories and local measures. 9. Performance indicators that were negotiated and included in the Supplemental Letter #2 included the following indicators: (i) Hospital master plan and planning standards and guidelines adopted; (ii) Contracts and payment methods for hospitals participating in the project provide improved incentives for efficiency and quality of care, are consistent with restructuring plans and avoid perverse incentives, (iii) policy and plan adopted for financing, resource allocation and provider payment methods for public health services, with the aim of increasing the effectiveness of disease prevention and health promotion; (iv) New Central HIS information system for Serbia in use and metadata bases in use to allow coherent approach to HIS development; local integrated HIS used by management in all participating health institutions in four areas of Serbia. 4 World Bank. 2003. Development Credit Agreement (Serbia Health Project) between Serbia and Montenegro and the International Development Association. For the purpose of this ICR, in accordance with ICR Guidelines updated 10/05/2011, the discussion is centered on the PDO as discussed in the PAD. 3 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 10. The PDO was not revised during implementation. 1.4 Main Beneficiaries 11. The main beneficiaries identified in the PAD were: (i) the general public, the MOH, health providers, and opinion leaders; (ii) people currently eligible for publicly financed healthcare services; health care providers; (iii) the poor, who were most likely to not seek care because of inability to pay; (iv) health care providers, plus patients who may be less likely to be pressed for informal payment; and (v) the public, and the GOS. 12. The main benefits expected from the project were: (i) availability of reliable data on a variety of health system indicators such as performance measurement, health expenditures and public health which increases the level of confidence in the quality of care provided and the viability of the publicly financed system; (ii) improvements in provider payment methods, to encourage providers to provide higher quality health services, leading to improved health outcomes; (iii) increased revenue for the HIF targeted for vulnerable groups relative to out-of-pocket payments; (iv) improved financial position of hospitals due to rationalized services; and (v) more informed and engaged public feeling empowered to contribute to health reform rather than to merely be affected by it. 1.5 Original Components 13. Component 1: Health Services Restructuring (US$13.0 million of total projects costs). The objective of this component was to build capacity for implementation of restructuring of health services delivery so as to improve efficiency while maintaining quality: to support initial stages of implementation of short-term restructuring initiatives focused on general hospitals and related health services in four areas. This component had sub-components as follows: (i) master-planning and integration of the public health system into restructuring in the four regional restructuring initiatives; (ii) restructuring in the Kraljevo demonstration site; and (iii) restructuring of health services in Valjevo, Vranje and Zrenjanin. The restructuring part of the support included hospital building renovations and purchase of equipment. 14. Component 2: Health Finance, Policy and Management (US$8.9 million of total project costs). The objective of this component was to support policy development and build capacity for analysis, decision-making, health financing, health system regulation and reform implementation in the MOH, HIF, the Institute of Public Health (IPH), and Serbian health sector regulatory agencies, and to support this with information systems development. This component had five (5) sub-components as follows: (i) basic benefits package and provider payment systems; (ii) financing of public health services; 4 (iii) licensing and accreditation; (iv) health information systems; and (v) MOH capacity- building and communication. 15. Component 3: Project Management, Monitoring and Evaluation (US$1.5 million of total project costs). The objective of this component was to ensure that project outputs were delivered on time and within budget and oriented towards achieving project outcomes effectively. Risks were to be identified and risk management strategies put in place and that project performance was subject to monitoring, audit and evaluation so as to ensure that objectives were achieved and lessons learnt for future health reforms. This component was to support: (a) the establishment of a Project Coordination Unit (PCU) to prepare, implement, and coordinate activities of the health investment project; and (b) to finance monitoring, evaluation and audits. The PCU was expected to take responsibility for production of routine six-monthly project reports and quarterly financial management reports. The Credit provided resources for annual and final project audits, and also for commissioning expert advice on monitoring and evaluation of project activities. 1.6 Revised Components 16. The project was restructured in 2005. Main changes to the project were: All project components were revised when the project was restructured in June 2005 (see paragraphs 17-19). The scope and number of activities were streamlined and rationalized without changing the development objectives, the general architecture of the project and the total budget (please refer to the Implementation Report of June 27, 2005). The main revisions to project components also included changes in estimated costs for some activities; and the introduction of some changes on procurement methods, (Schedule 3 of the Credit Agreement, i.e. introduction of National Competitive Bidding procedures for civil works and elimination of the two-stage bidding procedures for information technology equipment). The three components were restructured as follows: 17. Component 1: Health Services restructuring (US$10.5 million of total project costs). The component was restructured to improve efficiency and quality in service delivery and build up management capacity in four general hospitals (Kraljevo, Valjevo, Zrenjanin and Vranje). The component still had three sub-components which were better defined, with rehabilitation limited to 3 out of the 4 hospitals as originally planned, (excluding Vranje due to larger than anticipated scope of civil works). The revised component had 3 sub-components: (i) site reconfiguration, (ii) provision of medical equipment, and (iii) support to the definition, planning and gradual implementation of key measures to increase performance, control operating costs and improve quality of services provided, and waste management. In addition, previously unallocated amounts were directed towards The Institute of Virusology, Vaccines and Serums (Torlak) to support the reconstruction of its flu vaccine production line. 5 18. Component 2: Health Finance, Policy and Management and Quality Improvement (US$8.2 million of total project costs). The component was restructured by streamlining some activities, changing estimated costs for activities under the component as well as changing procurement methods for some items. The component aimed to support policy development and build capacity for analysis, decision-making, health financing, health system regulation and reform implementation in the MOH, HIF and IPH. First, it aimed to assist in building the capacity of MOH and the HIF to develop and implement measures to improve health financing. Second, it aimed to help the early stages of establishing licensing and accreditation bodies, support the development of health technology assessment-HTA and of Continued Medical Education (CME). Third, in coordination with other partners (notably the European Agency for Reconstruction (EAR), it aimed to help develop ICT services to enable health sector stakeholders to use and share a common set of medical and health related data that meet relevant European and International standards. Fourth, the component aimed to help build the capacity of MOH in the areas of policy development and planning and public relations and communications. 19. Component 3: Project Management, Monitoring and Evaluation (US$ 1.3 million of total project costs). The restructuring for this component involved changes to the total cost of activities under this component which was revised upwards reflecting costs of technical assistance and training. The component had two sub-components: (i) project coordination and fiduciary tasks of procurement and financial management, and (ii) monitoring, evaluation and audits. 1.7 Other significant changes 20. An Additional Financing (AF) of Euro 10.5 million (US$13.5 Million equivalent) to the project was approved on March 19, 2009 with a closing date of December 31, 2011. The AF was in accordance with Guidelines for Additional Financing (OB/BP 13.20). With the AF, the PDO remained the same. The additional financing was to scale up financing for activities that would contribute to the effectiveness and financial sustainability of the health system, and specifically to assist the Government in implementation of a case based payment system. The AF was to fund the provision of: (i) management information technologies (hardware and software) for hospitals and the Health Insurance Fund (HIF); (ii) technical assistance to hospitals, the HIF, and MOH to support institutional and management reforms both for the sector and hospital level so that hospitals can react to the new financial incentives set by case-based payment systems such as Diagnosis Related Groups (DRGs) and eventually improve hospital productivity and efficiency, and (iii) technical assistance to support monitoring and evaluation of the payment reform within hospitals and the HIF in order to evaluate and adjust the impact of the payment change and to prevent adverse effects and cost escalations. 21. The AF impacted mostly activities under Component 2 (Health Finance, Policy and Management and Quality Improvement). In particular sub-component 2.1 (Health Finance), sub-component 2.2 (Quality Improvement), sub-component 2.3 (Health 6 Information System) and sub-component 2.4 (Capacity Building and Communication of the MOH). 22. The AF brought in additional KPIs. Progress towards achievement of development objectives supported by the AF was to be monitored using the following key performance indicators: (i) Standards and guidelines are used to make decisions on investments in information technology, (ii) Ministry of Health safeguards quality of services by introducing internationally certified Clinical Practice Guidelines for Clinical Pathways, and (iii) Ministry of Health publishes regular review of performance of hospitals. These are presented in the Results Framework Analysis. 23. The Results Framework evolved over time. The PDO never changed through restructuring or additional financing. The results framework was not formally changed in the 2005 restructuring. In 2008/2009, the results framework was informally refined to correct some deficiencies in the original framework; these changes were only communicated through an Aide Memoire and not through a credit agreement amendment. The applicable indicators are therefore the original key performance indicators as defined in the Supplemental Letter No. 2 of the Minutes of Negotiations together with those added with the approval of the AF. The 2008/2009 ‘refined’ indicators were the ones used for monitoring progress during implementation. The difference between the official indicators and the refined indicators are individually discussed in the results framework analyses. In some cases the differences are minor, in some cases the original indicators were not tracked at all. 24. Credit closing dates were revised. The closing date of the Credit was extended four times for a total of 22 months: (i) from February 28, 2008 to August 30, 2008 (six months); (ii) to February 28, 2009 (six months), (iii) to August 28, 2009 (six months), and (iv) December 28, 2009 (four months). 25. The first extension was granted to allow time for: (i) renovation of the vaccine production unit of Torlak Institute; (ii) review and finalization of the Health Sector Configuration Strategy; and (iii) completion of the analysis of options for hospital payment reform. The second extension was approved to allow time for: (i) technical discussions for the Additional Financing Loan to be upgraded to Negotiations; and (ii) the Additional Financing Loan to be approved by the Bank Board (March 19, 2009). The third and fourth extensions were approved to allow for the financing agreement of the Additional Financing to become affective (October 15, 2009) and a short overlap with the start-up of activities to avoid the loss of momentum between the two operations. 26. The additional financing was expected to close in December 2011, but the credit closing date was extended to March 30, 2012 to allow for completion of IT activities and most importantly to allow for quality assurance and due diligence in final delivery of IT networks, hardware and software. At the time of writing this ICR disbursements amounted to 97% of AF proceeds. 7 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry The ICR team rates design and quality at entry as Moderately Unsatisfactory on the basis of analysis of the following features: 27. Soundness of background analyses: This was the first Bank supported standalone health project in Serbia, its preparation was supported through a US$500,000 PHRD grant (TF051137). The preparation of the project benefitted from the experience of earlier operations that supported activities in the sector, especially the Social Protection Economic Assistance Grant (SPEAG) co financed with the Government of Canada which had supported activities across Ministry of Social Affairs; Labor and Employment and Health. The outcome of the project in the area of health was rated satisfactory. According to the ICR for SPEAG, while activities in the health component of the project did not focus on policy reforms, the technical assistance was important in shaping reforms of the health system and supported the preparation of the subsequent health project as well as other operations that had health sector activities such as the Sector Adjustment Credit (SOSAC). Preparation also benefitted from the International Committee of the Red Cross (ICRC) basic health services pilot project in Kraljevo (2000-2004) that provided information on information system development and tobacco use. So while this was the first health sector only operation the project team did have experiences from other operations to draw on. 28. Assessment of project design: Various alternatives for investments in the sector were properly considered before the team settled on the design of the project. The team took into account the resource envelope and the large needs of the sector, and decided to settle on a project that would have a large capacity building component that was needed in order for long term effects in the sector to really have an effect. The team settled on health financing challenges facing the sector based on the importance of financial sustainability and the large inefficiencies that had to be addressed. The team also considered addressing specific diseases such as HIV/AIDS and TB; challenges in the pharmaceutical sector as well as medical education and emergency medicine, and decided not to include these since these were covered by other donors. 29. Soundness of the assessment as to what the needs of the sector were was sound, however, the design of components and the whole organization of the project were lacking. The component activities were quite complex and the degree of realism was low, and the expectations were quite high for that time in the history of the country. Project activities required a high degree of innovation in a sector that was still growing its capacity within the implementing agency. 30. The degree of Government commitment during preparation and appraisal was quite high, though the political reality, including a volatile political environment, was not factored into the design of the project. Counterparts showed a level of commitment that was demonstrated in the many meetings that the project team and 8 management had with the Government. It is clear, however, that the political environment and processes at the time of preparation and appraisal were not adequately factored in to inform the design of the project and to inform the risks to the pace of implementation of the project once the project was approved. In some aspects, project implementation plans were secondary to the primary goal of securing funding for the project. No mitigation measures were put in place to address issues such as change in Ministerial leadership. 31. The project preparation team requested a Quality Enhancement Review whose meeting was held in January 2003. The Report made a number of recommendations which were addressed by the Bank team as described in an email dated March 2003. For discussion see Section 5.1. 2.2 Implementation 32. The ICR team rates overall implementation as Satisfactory. The project was successful in delivering most of the outputs under the four components and achieving its expected outcomes (see data sheet, section 3.2 and annex 2). The initial phase of project implementation, the years prior to 2006, was largely rated unsatisfactory both for PDO and IP; though there was marked improvement after the June 2005 restructuring (see Data Sheet section G). Implementation was successful in many ways, for example, actions taken to restructure the project and respond to some aspects of procurement challenges; firm commitment of the Government towards the PDO was instrumental in moving implementation forward and the quality of MOH and PCU staff that were responsive to challenges encountered. Implementation was however less successful in a number of areas as follows: 33. Three months after project effectiveness in October 2003, the project team rated progress towards achievement of the PDO as Unsatisfactory. The following were the reasons: (i) In August 2003, approximately two months after the project was signed, at a crucial phase for the project, The Minister of Health as well as most assistants to the Minister resigned, seriously affecting implementation due to lack MOH leadership. This was also a period of uncertain political environment whereby most counterparts in the HIF and directors of hospitals in four pilot regions supported by the project were expected to change; (ii) Uncertainty created by the likelihood of a change in policy direction after the appointment of a new Minister of Health; (iii) Delays in the appointment of project working groups and no activities by the Project steering committee, basically a PCU without authority. In this environment, implementation preceded slowly, disbursement reached only 5 percent by June 2005 versus an expected 20 percent. The U rating for IP lasted until June 2004 when it was upgraded to Satisfactory, only to be downgraded to Unsatisfactory in December 2004 due to very little progress in implementation, resignation of the PCU Director (and 9 months without PCU leadership) and weak project management. All aspects of project management (apart from financial management), i.e. meeting procurement schedule; financial performance, monitoring and evaluation, etc, were unsatisfactory at this time. 9 34. It was at the end 2004, about 14 months after effectiveness, that the lack of implementation progress triggered the decision for restructuring. In addition to the above, the project experienced significant delays caused by structural/design factors that were noted as reasons for the restructuring in June 2005. One, the project as approved was ambitious. Not only was the project not clear on definitions of activities to be implemented, it also lacked an implementation strategy and lacked consistency between and among components. Two, at credit effectiveness, the overall strategy for the IT investments was not in place and the Terms of Reference (TOR) for key activities were not ready. Three, delays in hiring key personnel for the Project Coordination Unit (PCU) further stalled initiation of activities. For example, the PCU coordinator was not in place until months after, this in turn delayed the appointment of staff needed to support health financing and quality assurance activities. It was also noted that PCU-MOH coordination was poor and that planning, reporting and monitoring of activities was still weak at this early stage of implementation. 35. The project restructuring improved project implementation significantly. Together with other efforts by the MOH to improve management of the project, a year after restructuring, and all major aspects of implementation were satisfactory apart from monitoring and evaluation which up to this point had received a limited amount of attention. 36. The mid-term review (MTR) was originally planned for September 2005, but due to initial delays in implementation, took place a year later in September 2006 5. The MTR noted that following project restructuring, project implementation had significantly accelerated and therefore disbursements had increased considerably. Disbursement and commitments had speeded up, and by October 2006, 46.5 percent of the total IDA credit had been disbursed and US$5.4 million had been contracted with a projection of 90 percent of the credit proceeds committed by end of 2006. An area that was still unsatisfactory was monitoring and evaluation which the project team and management acknowledged as being a challenging area given the paucity of data needed to provide a meaningful baseline and to monitor achievements 6 . This area was later addressed by the refinement of indicators 7. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 37. The ICR team found the design of the M&E as weak. This ICR review finds that key performance indicators as described in the PAD was not a good reflection of the PDO, with a number of indicators aiming to measure achievements that were outside the 5 World Bank. Implementation Status Report #6 (June 4, 2005) and #10 (June 20, 2007) 6 World Bank. Implementation Status Report #9 (November 13, 2006). 7 World Bank. Implementation Status Report #10 ( June 20, 2007) 10 scope of the project. Moreover, baselines were not identified, baseline data was not readily available and there was little indication of how much of the gaps could be obtained further down the implementation calendar. Project documents show that key performance indicators needed to be more closely connected to project components, and again, to have indicators that were realistic in the expectation of project outcomes. The original M&E design for the project did not include baseline or target values for outcome or output indicators, and in most cases these had to be inferred. 38. Implementation and utilization of M&E framework was very limited. For the larger portion of project implementation, M&E indicators were not used because there was no baseline information and given all the structural deficiencies in project design, there were no initiation of activities to address this deficiency until much later in implementation. While the 2005 project restructuring did not change the PDO, outcome indicators were reduced in numbers but no baseline or target values were identified and no strategy was put in place to amend this deficiency soon after. In 2008/2009, the team addressed this lack of baseline and target values and refined output indicators, which the team continued to use for supervision and tracking implementation progress. With the approval of additional financing to the project in 2009, another set of outcome and output indicators were introduced to the project, these had baseline and target values and were used for monitoring of the project going forward. 2.4 Safeguard and Fiduciary Compliance 39. The environmental category of the project at the time of appraisal was B; the project preparation team envisaged that the project will have a positive environmental impact through improvements in infection control and safety within hospitals. All facilities supported by the project were expected to adopt new national standards and guidelines consistent with EC requirements including fire, medical radiation, hazardous chemicals and biomedical waste. The midterm review and supervision reports noted that safeguards compliance was satisfactory. A mission undertaken for the purpose of ICR preparation visited two hospitals that received support from the project and a review with hospital management shows that standards and guidelines were complied with. In addition, the new standards for medical waste collection and disposal have now been supported for wider introduction around the country by two EU projects that support similar activities at all levels of health care delivery. 40. Procurement: After the deficiencies in fiduciary management noted earlier in implementation, by and large, compliance with procurement, financial management and disbursement and auditing for the project has been satisfactory. Project records show that procurement was unsatisfactory from mid 2004 to mid 2005 when it was rated moderately satisfactory. Procurement was satisfactorily managed by the PCU which was staffed by a full time Procurement Officer with 6 years of procurement experience that complies with Bank procurement rules. The PCU regularly updated procurement plans and management of procurement processes, even for complex items, was overall satisfactory. 11 41. Financial Management: Similarly the PCU was also staffed with qualified staff in this area; and financial management has been satisfactory with proper management of records, internal controls to enable timely preparation of financial reports and annual financial statements. Project files show that FM was unsatisfactory during the same period of unsatisfactory procurement performance, showing that this was a temporary period of overall unsatisfactory project management. 2.5 Post-completion Operation/Next Phase 42. Ongoing Bank supported activities in the sector and transitional arrangements. Currently the Bank continues to support the sector (investments and dialogue) through the multi-sector DILS Project (which is scheduled to close in December 31, 2012). The DILS PDO is to increase the capacity of institutional actors and beneficiaries in order to improve access to, and the efficiency, equity and quality of local delivery of health, education and social protection services, in a decentralizing environment. For the health sector the project supports decentralization of responsibility for primary health service delivery which in 2007 was transferred from the state level to provincial and municipal governments. Implementation of the DILS Project is coordinated by the same PCU and MOH team that were involved in the implementation of the SHP. The project ensures the continuation of dialogue in the sector; it provides an opportunity for the implementation team to remain engaged and also provides for some continuation of a select group of activities from SHP. Progress towards the achievement of the DILS PDO is satisfactory. 43. The Bank-Government of Serbia Partnership Strategy aims to continue supporting the GOS in its efforts to strengthen the health sector. The 2011 CPS aims to assist the GOS to improve efficiency and outcomes of social spending. The Bank will continue to work with other development partners in supporting the GOS to strengthen its capacity to monitor and evaluate social spending. The Bank will continue to work with the GOS through a new health project planned for FY13 and which is likely to use the new Program-for-Results (P4R) lending instrument. There is also a Regional Health Finance AAA planned for FY14 as well as a third operation in the three series Public Expenditure Development Policy Loans (PEDPLs) that will further strengthen Government’s efforts to restrain growth in public sector wages and pension obligations, while improving public expenditure management. The aim of the PEDPL 3 will be to reduce the size of Serbia’s large public sector and support reforms that will improve the efficiency of public spending while mitigating the persistent social impact of the economic crisis. It is designed to: (i) increase the efficiency of expenditure in the health and education spending; (ii) strengthen public expenditure and debt management; and (iii) strengthen social assistance to cushion the impact of the economic crisis and expand coverage of social assistance programs. 12 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 44. The ICR team rates the relevance of objectives, design, and implementation arrangements as high. The project’s overall objective remains relevant now as it was at the time of appraisal. While the country’s health system has seen some improvements in the last decade during which the project was implemented, several systemic challenges remain in the health system underscoring that at the time of writing of this ICR, the objectives of the project remain relevant. The relevance of objectives of the project is also underscored in the Bank’s own 2007 Health Nutrition and Population Strategy which prioritizes strengthening health systems 8 . The original project and the additional financing remain consistent with the FY 2011 CPS for Serbia, which aims to improve delivery of social services at all levels of Government. The relevance of the AF to the project is clear in areas of focus of project activities, i.e. on reforms of provider payment mechanisms at secondary and tertiary hospitals, and complement investments made through the DILS Project, which supports capitation payment reforms in Primary Health Care (PHC) level. These are areas that have also been identified by the Canadian International Development Agency (CIDA), the EC and the ICRC, as areas in need of support. The areas tackled by the project, though ameliorated by the achievements of this project, remain major challenges for the sector. At the time of the ICR mission, the PDO continues to be highly relevant because the health care system: (i) requires continued reforms to ensure affordable and equal availability and access to basic health care services for all citizens; (ii) requires more cost-effective services provided in a manner that ensures quality of health care services; and (iii) needs a financing of health care model that will ensure fiscal sustainability. 3.2 Achievement of Project Development Objectives 45. The project’s achievement of its objectives is rated Substantial based on the project achieving almost all of its outcome indicators and because through the project’s support, capacity to develop a sustainable performance oriented health care system has been built. The ICR evaluation was based on analyses of outcome and intermediate indicators and data linked to the development objectives of the project as well as an evaluation of how project activities were linked to the achievement of the PDO. As will be demonstrated below, the support of the project has built a solid foundation necessary for a performance oriented health care system where providers are rewarded for quality and efficiency and where health insurance coverage ensures access to affordable and effective care. Achievement of the PDO, that is to assist the Borrower to build capacity to develop a sustainable, performance oriented health care system where providers are rewarded for quality and efficiency and where health insurance coverage ensures access to affordable and effective care, is addressed below. 8 World Bank. 2007. Healthy Development. The World Bank Strategy for Health Nutrition and Population. 13 46. The project supported investments in analysis, testing and training that led to the design of a health system that will be based on output models of care financing, consistent with the Project objectives. This includes capitation for PHC (supported by DILS) and DRGs for hospitals that was supported by the SHP. The DRG system is a system that will allow providers of services in hospitals to be paid according to the types of services that are offered to patients and not on the basis of inputs (e.g. staff, hospital beds, etc) available in their facilities. In addition to supporting the introduction of a more efficient system, both the capitation formula for PHC and the DRGs for hospital care have been designed in a way that a proportion of salaries of health care providers is directly linked to preventive services provision (quality) and to number of registered patients and services provided (efficiency). 47. In support of the development of the new financing system, the Parliament has adopted MOH Strategy 2010-2015 which endorses these financing reforms. The project also supported the necessary investments in information technology in the form of software that will be used in the reporting of services rendered according to DRGs. The legislation that enables performance based financing in health care is now in place, the Amended Law on Salaries of Public Servants was adopted by Parliament in Dec 2011. Together with other accompanying by-laws adopted by the Government in January 2012, which more closely defines care financing in primary and secondary health care, this provides the needed conditions for implementation of a performance oriented health care system. In addition, the project has assisted the government in investing in information technology, modernizing the information system of health care institutions; and for the HIF, the project enabled the connection of all its local branches, an important step in increasing transparency and efficiency of the whole system, both on the provider payment but also for access to users. 48. The achievement of the PDO is demonstrated by data on the following indicators (for more details see the results framework analyses, Section F of the datasheet). Outcome indicator #1: Hospital master plan and planning standards and guidelines adopted. The project supported the preparation of a new organizational model for hospitals which was based on the experience of the 4 hospitals that the project supported. The project also supported development of a number of guidelines including the IT Guidelines on Investments; development and introduction of Integrated Plans for Quality Improvement; Guidelines for reduction of hospital acquired infections; Clinical Pathways Guidelines; Human Resources Plan which are discussed in the relevant sections below. Indicators that were added with the approval of the AF, also show that the project did contribute to the achievement of the PDO. For example, the project supported the development of standards and guidelines that are used to make decisions on investments in information technology (AF PDO indicator #1); and the MOH introduces internationally certified CPGs for clinical pathways (AF PDO indicator #2); and the project also supported the regular review of performance of hospitals (AF PDO indicator #3). 14 Outcome indicator #2: Reduction in arrears; reduction in bed numbers and average length of stay; increase in hospital bed occupancy; reduction in flows of patients from the area to tertiary care; and increase in use of primary care and outpatient services in the four general hospitals participating in the project: 49. The project made satisfactory contribution towards the achievement of a number of the above KPIs that were intended to measure increasing efficiency of the health care system. Data show that the number of beds has decreased between 2003 and 2005 and though it increased again in 2010, the overall 2003-2010 trend is one of decrease. The average length of stay has shown a consistent downward trend from 2003 to 2010, and the mean occupancy rate has increased within the life of the project. Data also show that the number of hospital discharges has also increased consistently between 2003 and 2010. Health expenditure as a proportion of GDP has remained steady and seen no significant increase between 2003 and 2010 from 5.3 percent to 5.5 percent. The one KPI that did not show improvement over the life of the project was the reduction in arrears held by health care facilities. While the trend was in the right direction between 2003 and 2005 when arrears accumulated over 15 years were paid off, and between 2006 and 2008 which were years of surpluses, this trend reversed in 2010. The lack of achievement of this KPI is not an indication of unsatisfactory achievement of the PDO and is more a reflection of an indicator that is out of control of the project and is subject to economic well-being of the economy as a whole, in reality it is a reflection of a deficient M&E design or even an overambitious project design that overstated the objectives of the project. 50. Other outputs that have contributed to the satisfactory achievement of PDOs include reconstruction and reconfiguration of large, disconnected and inefficient hospitals that were supported by the project. For example, the project supported reconfiguration of buildings housing diagnostic services and reconstruction of a new Pediatric Department at Kraljevo hospital. The total space of the outpatient polyclinic has been reduced by about 3000 m2 and three principal buildings of the hospital are now connected. Similarly in Vranje Hospital, the project supported the connection of ten wards that were dislocated in ten different and separate buildings; in addition the hospital had no central surgical block or a central intensive care unit. Valjevo hospital had four separate hospital sites and in order to decrease costs (for maintenance, utilities, etc) and increase efficiency (reduce the time required for transport, make services more accessible), the project supported the reconfiguration of the hospital as well as purchase of more updated hospital equipment. Work undertaken in Zrenjanin hospital also involved connecting small and old buildings with an incomplete 10-storey new hospital that was largely unused. The project’s support led to the completion of the admission room, pre-delivery room, 4 delivery rooms, post-delivery room, operating theater for C- sections and patient rooms with 30 beds as well as a neonatology unit for newborns as well a designated incubators area for prematurely born babies. These and other changes at 15 Kraljevo, have improved patient flows as well as diagnostic support for emergency services and for PHC 9. 51. The achievement of the PDO is further demonstrated by data on outputs related to the following key performance indicators: i) Outcome indicator #3: HIF and MOH increase the number of staff trained in health policy, health finance, or health management, and use their skills in review of the basic benefits package, the public/private mix in health services, resource allocation and provider payment systems; and a critical mass of these staff remain in post: 52. Review of project investments and outputs achieved demonstrate that the project was successful in creating a HIF and MOH workforce trained in health policy, health finance and health management that is capable of reviewing the Basic Benefit Package (BBP), manage resource allocation and designing a new provider payment system. The project supported the training of over 6, 000 staff between 2004 and 2010; and supported by a Human Resource (HR) policy, the labor force in the health sector has been rationalized; data show that the number of health sector employees on Government payroll has continuously been reduced over the life of the project, from 129,506 in 2004 to 114, 432 in 2010. During the life of the project, the HIF has increased the percent of the insured with complete information in HIF database to 100 percent. The MOH and HIF have increased their capacity in analyzing the BBP, which is supported by a legal framework. The project supported intensive management training for management at the 4 regional hospitals in order to improve the level of awareness of modern health management practices in public hospitals. Through project supported activities, institutional capacity of both mid-level and senior managers received training in service management including in areas of performance, operating costs control, quality of services, safe and efficient medical waste management. The new standards medical waste collection and disposal have now been supported for wider introduction around the country by two EU projects that support similar activities at all levels of health care delivery. Through the Chamber of Health Institutions of the Republic of Serbia (CHIS) the project also supported the training of health care institutions managers in areas of managerial responsibilities, efficiency and cost savings. 53. The achievement of the PDO is further demonstrated by data related to the following key performance indicators: ii) Outcome indicator #5: MOH, with input from other relevant Serbian health sector agencies, monitors and analyzes at least annually the revenue, expenditure and arrears of the HIF and public health care institutions; the distribution of HIF expenditure 9 In addition to reconfiguration and improving patient flow and support for diagnostic services, the project supported the purchase of equipment including: ECGs holter, EEG, EMG, defibrillator with monitor, non-invasive blood pressure monitor, ophthalmoscopes, arthroscope and US unit for urology. biochemistry analyzer, blood cell count apparatus, glucose analyzer, flame photometer, capillary electrophoresis device, centrifuge, microscopes, operating-ENT microscope, endo-video gastroscope, general surgery operating lamps and tables, cardiac ultrasound imaging system, hot air sterilizers). 16 per capita in different areas and among vulnerable and other population groups (taking account of risk factors); and out-of-pocket expenditure on health by the population; and outcome indicator #9: MOH, with input from other relevant Serbian health sector agencies, publishes regular analytical reports on the performance of the health system and health sector institutions, using data from enhanced health information systems, and using the WHO World Health Report 2000 categories and local measures: 54. A review of project outputs shows that the MOH and HIF have built their capacity to monitor and analyze revenue and expenditures of health institutions. The project supported the establishment of the National Health Accounts (NHA) team within the Institute of Public Health (IPH). The first collection of NHAs was completed in 2006, conducted in collaboration with international consultants. Follow up rounds of data collection efforts were undertaken in 2007, 2009 and 2010. Sustainability of the NHA System is ensured by the fact that since 2008, NHAs have been financed from the Government’s budget under a line-item budget in MOH. NHAs data collection and management are wholly under the responsibility of IPH, who are annually commissioned to produce accounts by the MOH. NHA data is available on the IPH Website under the Centre for Informatics and Biostatistics of the IPH. In addition, the MOH has increased its capacity to collect and analyze data on other areas of interest to the sector. For example, the HIF produces reports that analyze revenue, expenditure and arrears of the HIF and other public health care institutions, and analyses the distribution of HIF expenditure per capita in different areas and among vulnerable and other population groups as well as out-of-pocket expenditure on health by the population. 55. In addition to the above, the contribution of AF towards the achievement of the PDO can be traced through the following indicators that were added: (i) Standards and guidelines are used to make decisions on investments in information technology: 56. The MOH has adapted three sets of Standards for IT investments in PHC and Hospital Information System, National Coding systems and Minimal data sets. All investments supported by the project were in accordance with IT standards and guidelines as described in the MOH IT Rule Book. In addition, the project supported the development of a national body (ProRec Serbia founded in 2008) that oversees PHC systems quality labeling and certification, the body is made up of key personnel of the MOH, HIF, health institutions, software solution providers and Belgrade University. The project also supported the introduction of a Certified Information Security Management System in accordance with ISO 27000 in General Hospital Zrenjanin. In addition, a roadmap and templates that allow users to implement ISMS has been developed. 57. In addition to support for installation of IT hardware, HIS software, there was also support for Wide Area Networks (WANs) and infrastructure for data exchange. This includes scheduling of specialist examinations, access to electronic health records in hospitals, and the integration of PHC software with Laboratory Information System (LIS) in referral hospitals. The LIS is connected with analyzers and receives 17 results from analyzers in 20 hospitals 10 and from digital technology that processes radiological images. Such a web based system allows for data access from different locations within a medical facility, as well as from outside the institution providing a basis for telemedicine. The WAN connects over 360 institutions of the health sector including 18 hospitals and 2 institutes. Investments in IT supported by the Project have benefitted the development of a database of those insured which provides key data to the HIF as well as better collection of payroll contributions and allows for improved access to health insurance. The project also supported the development of a nationwide framework for medical information system providing the regulation and standards for PHC and hospital information systems development. The national health care information and communication infrastructure has been implemented connecting health care institutions into a single communication structure, thus allowing future electronic data exchange and better integration. ii) MOH safeguards quality of services by introducing internationally certified Clinical Practice Guidelines for clinical pathways; and Ministry of Health publishes regular review of performance of hospitals: 58. The project supported the MOH in establishing the Quality Unit which helped to spearhead activities in Continuous Quality Improvement, Accreditation, Health Technology Assessment (HTA), Licensing and CME. The project supported the establishment of the Agency for Accreditation, which is operational and accreditation process is going on for 44 health care facilities (dom zdravlja –DZs) -supported by DILS project and 17 hospitals supported by AF. In addition to setting up the accreditation office (office space, equipment later to be paid for by accreditation fees), the project supported the development and implementation of an accreditation program supported by technical assistance on international best practice in accreditation. The project also supported training surveyors and quality coordinators; development of standards and survey procedures; piloting and the roll-out of accreditation standards. So far 15 PHCs have been accredited and 2 of the 4 hospitals that went through the accreditation process were accredited. In addition from 2009, 13 secondary and tertiary health care institutions have been accredited. 59. In order to further improve on quality and efficiency of health care services, the project supported capacity building activities. This included training and study tours for staff in health care centers, MOH, HIF, PHI and other stakeholders in the sector in development of methodology on Clinical Pathways (CPs) as well as their use in treatment and management of care. During the life of the project CPs in child asthma, primary prevention of cardiovascular diseases, secondary prevention of cardiovascular diseases, iron deficiency anemia, depression, diabetes, screening of colon cancer, low 10 Hospitals supported by the project: Clinical Center of Serbia - Obstetrics and Gynecology Clinic, Clinical Center Kragujevac - Surgery (the whole building), Institute for Mother and Child, Clinical Hospital Center “Dr DragiÅ¡a MiÅ¡ović - Dedinjeâ€?, general hospitals in Vrbas, Sremska Mitrovica, Å abac, Požarevac, Jagodina, Smederevska Palanka, Aleksinac, ZajeÄ?ar, Užice, Leskovac, Petrovac na Mlavi, Surdulica, Gornji Milanovac, Novi Pazar, the Narodni front Obstetrics and Gynecology Clinic, Institute of Mental Health in Palmotićeva St. in Belgrade. 18 back pain, myocardial infarction, colon carcinoma, child asthma, normal delivery, stroke, and cholecystectomy were introduced and evaluated. 60. The project also supported the development of Continuous Medical Education. The legal framework, such as the Licensing and CME by-laws, has been set up and the Medical Chambers have been established. Investments in CME have enabled the Ministry to launch an examination of HAIs in all 22 institutes of public health and designed procedures for handling the most frequent HAIs including the use of antibiotics and procedures for monitoring and preventing HAIs. 61. It is important to note that since the implementation of the developed and tested DRG system will only begin later in 2012, the KPI on publication of regular review of performance of hospitals was partially met. In 2011, a hospital infections survey and review was conducted for 60 hospitals and the review on hospital performance published for hospitals participating in the project. This KPI shows partial achievement because a review based on DRG reporting was not completed due to delays in adopting DRG by-law. There is however no reason to believe that this will not be achievable once the DRG system is fully implemented and more data from the system are collected and analyzed. 3.3 Efficiency 62. An economic analysis to measure whether the costs of involved in achieving project objectives were reasonable in comparison with the benefits and recognized norms was not carried out for the ICR. The PAD included a modest cost benefit analysis of the project and acknowledged the challenges of taking into account the full benefits of health system interventions such as those supported by the SHP. It is important to note that the project restructuring did not include a revised economic analysis for the restructured project and data that would allow for meaningful economic analyses was lacking. Description of activities that would lead to efficiency gains, such as those under paras. 49 and 50 and other data presented in the ICR shows positive trends towards a more efficient health sector including reduction in bed numbers and lengths of stay. It is also important to note that the support provided by the project was but a component of the overall reform agenda financed by the Borrower hence separating the contribution of the project alone is difficult. This inability to specifically quantify gains in efficiency does somehow detract from the outcomes performance rating by the ICR team, which is weighed project relevance, its achievement and efficiency. 3.4 Justification of Overall Outcome Rating 63. Rating: Satisfactory. The Project’s overall outcome is considered Satisfactory based on the Project’s significant achievements, its continued relevance, its contribution towards strengthening Serbia’s health sector program, and making it more accountable and efficient. Although the project as designed was weak, restructuring undertaken in June 2005 considerably improved the design and more closely linked activities to the PDO. In addition while the M&E framework was somewhat weak and remained deficient, 19 the team did make efforts to refine indicators that have been used to monitor the project. Some indicators were introduced later in the project 11 and with the approval of the AF, the M&E does provide some way of linking project outputs to the desired achievements. Data show that most of the key performance indicators were fully achieved, and most exceeded their targets. Despite the deficiencies mentioned, this is a project that supported the development of the foundation of a system of care that is more efficient, it improved the quality of care that patients in the participating hospitals received, from renovated buildings that are safer for patients and where data is more readily available, and most importantly it also introduced institutional changes that are highly likely to be long lasting and difficult to reverse. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 64. The SHP and AF to the project mainly tackled systemic issues facing the sector. While the project identifies some vulnerable groups (see section 1.4) as beneficiary of improvements in the functioning of the system, it is not easy to attribute quantitative or qualitative impact on poverty or gender aspects which have been tackled through specific indicators because none were included in the results framework. The Bank does however support another project in the sector (DILS Project) that addresses gaps in access to social services, including health services, faced by the Roma who face much worse socio-economic conditions than the rest of the population. (b) Institutional Change/Strengthening 65. This project had an extensive focus on strengthening institutions. The above section shows that the project has been successful in building long term capacity and institutional development. The project supported training for more than 6, 000 persons, who are involved in: (i) the monitoring and analyses of revenue, expenditures and arrears of the HIF; (ii) financial reforms including provider payment reforms and resource allocation: (iii) preparation and analyses of data from the NHA established at the IPH; (iv) licensing and accreditation of providers and care facilities; and (v) development, installation and maintenance of HIS. The project supported a large number of training activities that are instrumental in the development of a performance oriented health care system; and has put in place the structure needed for a performance oriented health care system based on DRGs. Other institutional changes are in the way that IT has introduced efficiencies in the system, such as those now found at the HIF or at the hospitals supported by the project. Other institutional changes pertain to quality assurance such as measures geared at Clinical Practice Guidelines for clinical pathways. The project supported the introduction of laws and by-laws that are critical to the introduction of reforms of the health care system. In reality, few of the achievements of the project would have been possible without significant improvement in the capacity of the sector staff. 11 World Bank. Implementation Status Report #13 (July 14, 2008) 20 (c) Other Unintended Outcomes and Impacts (positive or negative) 66. There has been very close coordination and synergies with the DILS project and activities financed by the EU (e.g. in hospital waste management and IT), all of which have resulted in better outcomes for the sector. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A 4. Assessment of Risk to Development Outcome 67. Rating: Modest. Achievements in the area of health sector reform supported by this project have been reached under a climate of strong overall commitment to financing reforms not only in the health sector, but in other social sectors that constitute the larger part of the Government’s budget. At the time of this ICR’s preparation, the country has just had elections scheduled in the spring of 2012. This is also a time when the country and the region are still facing the economic crisis. While there is a risk that there could be weaker commitment to reforms in the health sector, the country’s strong commitment to EU accession is likely to modulate this tendency. As far as risks to an erosion or reversal of specific PDO achievements, for example in IT infrastructure, the Government has put in place a mechanism that will allow for continued funding of IT systems through the DRG system. In addition, additional support in IT investments is being made by the EU who in a coordinated manner with the SHP and MOH is supporting an additional 19 hospitals with IT investments. Other investments supported by the project have been institutionalized and are part of standard health care delivery in the country. In addition, most of the reforms are now enshrined in laws and bylaws and hence less likely to be reversed. Sustainability and likelihood of continuation of Bank supported activities can be found in the Plan of Development of the Health Care System 2010-2015 adopted by the National Assembly, and the Strategy for the Development of the Informatics Society in the Republic of Serbia Until 2020, e-Health2015 IT Strategy. Caution regarding the sustaining of reforms is needed given the new leadership of the Ministry; it is not yet known exactly what direction the sector will be steered in. This caution in the political economy of the sector and the country is important given the changes that have been supported by the project, uncertainties that led to resignation of the PCU Coordinator after successfully steering the project for years. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry 68. Rating: Moderately Unsatisfactory. The Bank helped to prepare the project by drawing on lessons learnt from previous engagement in the sector, the country and the ECA region and was successful in identifying areas of the system that needed reform. For the initial credit, however, the Bank was not successful in helping to establish the right 21 conditions for implementation, specifically, assisting to put in place principal elements for project implementation (e.g., administrative structures, capabilities for procurement, accounting, etc.). 69. The project was prepared very quickly in order not to lose an opportunity for IDA eligibility, and therefore missed on opportunities to put in place strategies for implementation. It also missed an opportunity to put in place a meaningful M&E system for monitoring implementation and measuring results, for example, there were shortcomings in the lack of initial baseline data information for indicators and the unavailability of analysis of data for certain key indicators. While these were later ameliorated, they do contribute to a less than ideal appreciation of the project's progress and achievements, of which there are many. Deficiencies identified in the requested QER were later amended in the restructuring, but this did cost the Borrower and the Bank time and effort to correct avoidable design deficiencies. 70. The Quality Enhancement Review raised reservations regarding data and analyses of the Serbian health system that was provided in the PAD. It raised doubts as to the achievability of the PDOs should the project proceed as was described in the PAD at that stage of preparation. The QER panel appreciated reasons for the urgency in getting the project approved (upcoming expiration of IDA eligibility), but recommended streamlining the project and collecting more data on the inefficiencies of the health care system that had not been well articulated at that stage of project preparation. 71. The panel raised a number of concerns regarding project readiness. These were that: (i) the PAD described a very ambitious project development objective (to achieve a sustainable, performance oriented health care system where providers are rewarded for quality and efficiency and where health insurance coverage ensures access to affordable and effective care); (ii) the development objectives as stated at that stage of project preparation were not consistent with specific actions proposed under the project and would in any case be highly ambitious and unlikely to be achieved in the context of the proposed investment operation; (iii) the project could not meet the expectation of achieving a sustainable, performance oriented health care system especially since the HIF was at the center of challenges to sustainability of health care financing. 72. Suggestions were made, and some were taken into account, that the operation should be designed as a stepping stone to better understanding on policy options for reforming the system. The panel also suggested that rather than aiming to improve its sustainability, the development objectives should also aim to articulate more clearly the expectations for improvements in the service delivery area. (b) Quality of Supervision 73. Rating: Moderately Satisfactory. This is a project that required intensive supervision especially at the initial stages of implementation; after all, this was a country coming out of conflict and a tumultuous decade. Documentation shows that soon after the project was approved, the then Minister of Health resigned, and in the environment of 22 uncertainty, a decision was made to concentrate on very few activities during this period of transition in the Government. While this was probably a prudent approach, this did introduce significant delays in implementation. 74. The most important and positive aspect of Bank supervision included the focus on the PDO. All interviews with stakeholders show that the Borrower greatly valued the quality of technical dialogue that the Bank brought to the project. The Bank consistently and diligently carried out procurement and financial management supervision. In addition, following restructuring the Bank team was very proactive in key aspects of the project including the background work needed to introduce the DRG system, the procurement of the right software and licensing rights for this purpose, and in the introduction of the Accreditation Agency. The Bank team should be commended for the close collaboration between the teams managing the DILS project that has a focus on primary health care and the SHP, ensuring complementary of activities between the two levels of care. 75. Less positive aspects of supervision included periods where lack of missions or issuance of Aide memoires. This protracts somewhat from a more comprehensive account of project implementation, supervision, achievements and lessons. (c) Justification of Rating for Overall Bank Performance 76. Rating: Moderately Satisfactory. As discussed in other sections, the quality at entry for this project was less than satisfactory, the Bank team must however be commended for corrections introduced through restructuring and the close supervision that enabled satisfactory implementation and steadfast focus on the development objectives. The quality of technical expertise dialogue was highly satisfactory especially once the project was restructured implementation sped up. Due to some shortcomings in initial design, poor M&E design and in supervision in the early stages of project implementation, Bank performance justifies a Moderately Satisfactory rating. 5.2 Borrower Performance (a) Government Performance 77. Rating: Moderately Satisfactory. This was a project that for the most part was implemented by a Government that was committed to reforms in the health sector, and one that for the larger portion of implementation demonstrated a long term strategic vision in the sector. Unlike the Bank team, the Government leadership in the sector changed a few times and the same Minister of Health oversaw the implementation for most of the 9 years of implementation. However, not all leadership showed the same degree of commitment, and some delays experienced towards the end of project implementation, suggest less than full appreciation of the significance of the reforms that have been supported by the project. The project suffered significant delays through delayed approval of contracts which in turn delayed some completion of IT activities. The role of the MOF in supporting the linkages between the SHP, AF, DILS and PEDPL3 was satisfactory. The Borrower’s performance in ensuring political and 23 institutional commitment was satisfactory; and while the initial period of implementation was very slow, it is important to remember that this was a period of political transition for the Government and it is hard in such an environment to have key Ministerial personnel in place. This is a process that usually takes time to become settled as is the case in most post-conflict realities. (b) Implementing Agency or Agencies Performance 78. Rating: Satisfactory. The satisfactory assessment is justified by the achievement of planned outputs; timely accomplishment of major project milestones; adaptation to unforeseen circumstances such as the delays in procurement and contracting of DRG licensing rights and quick response to requests for information. The performance of the MOH as the implementing agency was satisfactory. After a period of political instability, the MOH sustained a continued period of strong leadership and commitment to reforms, and ensured necessary support to the PCU in the running of the project. In addition, the PCU performance was also satisfactory. This was an action-oriented PCU and together with the World Bank team worked effectively to address implementation challenges. Though financial management was for a short period less than satisfactory at MTR, implementation progress in general and project management in particular were consistently rated satisfactory. As with most project of this era, the PCU was established outside the MOH, and it did allow the PCU to get the project up and running and to have specialized staff working on various aspects of implementation. This is however an institutional arrangement that made the project to be seen as operating outside the MOH and limited as close collaboration between the project and the MOH. This was raised during the ICR mission as a situation that at times created an atmosphere of distrust between MOH and PCU especially with the change in MOH leadership in the final months of implementation. It should be noted that project management was somewhat strained toward the end of the project with the departure of the long-time executive director, the need to disburse remaining project funds. (c) Justification of Rating for Overall Borrower Performance 78. Rating: Moderately satisfactory. The moderately satisfactory performance of the Government and the satisfactory performance of the Implementing Agency yield an overall rating of moderately satisfactory according to ICR Guidelines. 6. Lessons Learned 79. The need for speed in project preparation and a project’s quality at entry must be balanced. This is a project that was prepared, from identification to Board presentation, in about 11 months. The project was prepared for a country that was just exiting a tumultuous period in its history, and was identified when the MOH had no Minister for Health and approved about a month after the assassination of the Serbian Prime Minister. While one can appreciate the fact that losing IDA financing was an issue at stake, the haste with which the project was prepared did contribute to very slow implementation in the initial years which were a period of political uncertainty. The 24 country was essentially a post conflict country at the identification stage of the project, more attention to the political economy should have informed the design of the project, maybe to a much simpler project, and one with more implementation support in place. 80. The Bank needs to increase its knowledge on matters around licensing of Information Technology software and licenses in the health sector. This project lost about 7 months of implementation of DRG activities while issues surrounding the licensing of DRG classification system were being resolved. While the task team worked diligently to resolve fiduciary considerations, the whole episode showed the Bank to be less knowledgeable and more of a hindrance in the process. Since the Bank supports a large number of clients with HMIS needs, upping expertise in this area is important and would be beneficial to our clients. 81. An improperly designed M&E can raise serious questions about overall quality of project identification and of project design. This is in addition to making project monitoring a real challenge and making it difficult to demonstrate the achievements/ failure of a project. The lack of baseline indicators data for KPIs, the misalignment of KPIs to project activities and the PDO, can bring one to question the relevance of the project and how specific activities were identified to be the focus of the project since one needs an analytical base on which to anchor a project. The M&E framework for this project was not only misaligned but it was also promising to measure achievements that were not under the influence of the project. 82. The next Bank supported operation must seriously explore implementation arrangements that integrates the functions of a PCU within the MOH. The establishment of a PCU outside the MOH was probably an appropriate decision for the project at the time when capacity within the MOH was still in development. However, it is clear that with the increased capacity of the MOH, the Bank and the Borrower must consider having implementation arrangements that fosters closer collaboration, ensure an environment conducive to mutual learning and exchange of knowledge, and implementation experiences between the project coordination team and the MOH. An arrangement that allows for closer team work and exchange of ideas will facilitate building of technical expertise more broadly in the sector. 83. When financial sustainability is a desired goal then closer MOH-MOF-Public Administration dialogue through the Programmatic Public Expenditure Development Policy Loan could have further strengthened desired outcomes. It was overly ambitious for the project to list as one the KPIs, a reduction in arrears of health care centers without linking this KPI to activities directly supported by the project. It was also very important to ensure that the team engaged on the PPEDPL more aggressively and more closely scheduled the PPEDPL actions and prior actions with the implementation of the SHP. 84. Long-term engagement of the Bank in Serbia must recognize that health sector reforms are complex and often take a number of years. This is particularly the case when they are comprehensive as it is the case in Serbia, where they were addressing 25 financing reforms at all levels of health care, efficiency, quality and resulting in number of positive outcomes. However, in order to keep the momentum and provide enabling environment to successful completion of the reforms, the Bank should consider a longer- term approach in its support through a follow up project. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies: see Annex 7. (b) Co-financiers: not applicable. (c) Other partners and stakeholders: not applicable. 26 Annex 1. Project Costs and Financing 12 13 (a) Project Cost by Component (in USD Million equivalent) Components Appraisa Restructure Actual/ Percentage Appraisa Actual/ Percentag l d Estimate Latest of l Latest e of AF Estimate IDA Estimate Restructure Estimate Estimate Appraisal 14 Original (USD d Estimate AF AF 14 IDA millions) (USD (USD (USD (USD millions) millions) millions millions) ) 1.Health 11.60 10.50 11.60 100% 0.00 0.00 n/a Services Restructurin g 2.Health 7.40 7.50 6.20 83.8% 13.10 12.50 95.4% Finance, Policy and Management and Quality Improvement 3.Project 1.00 1.20 2.20 220% 0.40 0.65 162.5% Management and M&E Unallocated 0.70 Estimated 20.00 20.00 22.17 110.9% 13.50 13.15 97.4% Total Project Cost (b) Financing Appraisal Estimate Actual/Latest Estimate Percentage of Source of Funds (USD millions) (USD millions) Appraisal Borrower 3.48 3.48 100% International Development Association 20.00 22.17 111% (IDA) IBRD 13.50 13.15 97% Total Project Cost 36.98 38.80 105% 12 Government’s contribution of US$3.48 million financed activities under Component 1 (US$1.42 million); Component 2 (US$0.49 million) and Component 3 (US$0.57 million). 13 Increases in actual cost for project management and M&E against the restructured estimate under the IDA Credit is due to an additional 22-month extension of the closing date to cover for the preparation period of the AF and the awaiting period for Parliament ratification of the AF Loan, and the intensity of implementation following the restructuring of the original Project. As for the discrepancy between the AF appraisal estimate and the actual cost for project management and M&E, this also reflects the need to accelerate the pace of implementation during the last year until closing. In addition, critical technical assistance for monitoring and evaluation was required, including field work, during implementation to collect data, among others, on providers’ performance. 14 Increases in total actual project costs reflect exchange rate fluctuation in the SDR (IDA) and EURO (AF Loan) to dollar exchange rate over project life. 27 28 Annex 2. Outputs by Component Component (Actual costs) Output Component 1: Health Services Restructuring Civil works in Kraljevo completed; Medical (US$11.6 million) equipment for Kraljevo purchased, installed and operational; Civil works in Valjevo completed, Civil works in Valjevo completed; Medical equipment for Valjevo purchased; Civil works in Zrenjanin completed, Medical equipment for Zrenjanin purchased; Civil works in Vranje completed, Medical equipment for Vranje purchased; Civil works in Institute Torlak completed, Medical equipment for Torlak Institute purchased. Component 2: Health Finance, Policy and Audit of 15 Health Care Institutions completed; Management (US$6.2 million) Capacity related to processing of medical waste built, changes implemented in pilot hospitals; Strategy for Quality Improvement and Patient Safety adopted by Ministry of Health; Indicators on Quality Improvement established, and annual quality assessment of HCIs performed, First annual NHAs produced, DRG system chosen and piloted; Initial analyses related to introduction of capitation produced, Health Insurance Fund Database of Insurees built, Health Insurance Fund WAN built, Hospital Information System in Kraljevo built, Hospital Information System in Valjevo built, Hospital Information System in Zrenjanin built, Hospital Information System in Vranje built; Capacity for Accreditation built, first standards produced, first HCIs accredited, Agency for Accreditation of HCIs introduced, Accreditation of HCIs referred to in the Health Care Law, HTA referred to in the Health Care Law, Feasibility study for HTA institutionalization produced, HTA capacity built, first HTA reports produced; Requirements for licensing and relicensing produced, National Health Survey completed and results published; Three public awareness raising campaigns conducted. 29 Component (Actual costs) Output Component 2: AF (US$12.5 million) Sub-component A.1 - Initial Steps on Implementation of IT strategy in Health and Introduction of Supervisory Body for Health IT with MOH: Zrenjanin hospital information system is ISO27001 certified related to data safety (ISMS); Sub-component A.1 - Initial Steps on Implementation of IT strategy in Health and Introduction of Supervisory Body for Health IT with MOH: Update of IT Rulebook prepared; Sub-component A.2 - Establishing IT Network among Healthcare Institutions in Serbia: WAN network installed and built connecting more +90% of HCIs in Serbia; Sub-component A.3 - Improvement of Hospital Software and Equipment Purchase for Hospitals: Updated Hospital Information System installed in 20 hospitals; Sub-component A.3 - Improvement of Hospital Software and Equipment Purchase for Hospitals: Laboratory Information System implemented in 19 hospitals; Sub-component A.3 - Improvement of Hospital Software and Equipment Purchase for Hospitals: Radiology Information System implemented in 6 hospitals; Sub-component A.4 - Improvement of System Software: Database of Insurees (DoI) and Central Information System (CIS). Sub-component B.1 - Institutionalization of Health Management: Database of available courses and programmes in Health Management built; Sub-component B.1 - Institutionalization of Health Management: Chamber of Healthcare Institutions of Serbia became a member of HOPE (European Hospital and Healthcare Federation); Sub-component B.1 - Institutionalization of Health Management: Promotion of Energy Efficiency among HCI managers through meetings and fair; Sub-component B.2 - Quality Improvement and Control: Eight (8) new Clinical Practice Guidelines developed; Sub-component B.2 - Quality Improvement and Control: Thirteen (13) hospitals participating in the project passed through 30 Component (Actual costs) Output process of accreditation; Sub-component B.2 - Quality Improvement and Control: Seven (7) model clinical pathways (CP) have been developed. Six (6) CPs have been implemented in nine (9) project hospitals, five (5) CPs implemented in two (2) hospitals, four (4) CPs implemented in one (1) hospital, two (2) CPs implemented in two (2) hospitals and one (1) CP is implemented in four (4) hospitals; Sub-component B.3 - Implementation of the New Reporting System: License for diagnosis related groups system (AR-DRG) purchased; Sub-component B.3 - Implementation of the New Reporting System: Bylaw on new nomenclature of health interventions on secondary and tertiary level of health care adopted. Component 3: Project Management, Sub-component C.1 - Institutionalization of Monitoring and Evaluation (with AF: US$2.85 Reporting According to DRG System: Support million) to IPH Planning Department related to introduction of new reporting system for hospitals; Sub-component C.2 - Providers Performance Analysis: Capacity on introduction of new reporting system for hospitals and further introduction of diagnosis related groups raised in HIF; Sub-component C.2 - Providers Performance Analysis: Support to HIF related to introduction of reporting system for hospitals; Sub-component C.3 - Analysis and Health Policy Setting: National Conference on Continuous Quality Improvement and Patient Safety held; Sub-component C.3 - Analysis and Health Policy Setting: Action plan for Strategy for Quality Improvement 2010-2015 and Bylaw on quality indicators in healthcare prepared; Project management and fiduciary services. 31 Annex 3. Economic and Financial Analysis N/A Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Laszlo Balkany Consultant, Health Information System - Jan Bultman Lead Health Specialist ECSHD Dorothee Eckertz Junior Professional Associate ECSHD Michael Gascoyne Senior Financial Sector Specialist ECA Loraine Hawkins Senior Health Specialist ECSHD Virginia Jackson Senior Operations Officer ECSHD Vesna Kostic Communications Advisor ECCYU Tom Novotny Consultant Public Health - Marina Petrovic HD Operations Officer ECSHD Bjarne Lading Rasmussen Consultant, Health System Planning - Laura Rose Senior Health Economist ECSHD Lindsay Sales Hospital Management Consultant - Yingwei Wu Senior Procurement Officer LCSPT Supervision/ICR Ivana Aleksic Human Development Specialist ECSH2 Jan Bultman Lead Health Specialist ECSHD Sarbani Chakraborty Senior Health Specialist EASHH Olav Rex Christensen Senior Public Finance Speciali HDNED Aleksandar Crnomarkovic Financial Management Specialis ECSO3 Francois Decaillet Lead Public Health Specialist ECSH1 Armin H. Fidler Advisor, Policy and Strategy HDNHE Michael Gascoyne Senior Resource Management Off WBICA Dominic S. Haazen Lead Health Policy Specialist AFTHE Ana Holt Health Specialist ECSH1 Nikola Kerleta Procurement Analyst ECSO2 Johanne Angers Sr. Operations Officer ECSH1 Sreypov Tep Program Assistant ECSHD Hermina Vukovic Tasic Program Assistant ECCYU Marina Petrovic Operations Officer ECSHD Gennady Pilch Senior Counsel LEGOP Pia Helene Schneider Lead Evaluation Officer IEGPS Andreas Seiter Sr. Health Spec. HDNHE Ethan Yeh Economist ECSH1 32 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY02 4 42.38 FY03 47 363.62 Total: 51 406.00 Supervision/ICR FY03 2 0.67 FY04 21 104.97 FY05 51 228.21 FY06 49 195.14 FY07 38 127.20 FY08 37 93.65 FY09 18 0.00 Total: 216 749.84 Annex 5. Beneficiary Survey Results N/A Annex 6. Stakeholder Workshop Report and Results N/A 33 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR A. Summary of Borrower’s ICR prepared by the Ministry of Health of Serbia Serbia Health Project (2003 – 2009) and Additional Financing (2010 – 2011) Project Implementation and Operational Experience Serbia Health Project was approved by the WB’s Board on May 22, 2003 and the loan became effective on October 10, 2003. Serbia Health Project – SHP and the Serbia Health Project Additional Financing - SHPAF were focusing on comprehensive reform process. Thus, the number of activities was large, with constant danger of under financing and neglecting of some focus areas. MOH bridged this through the coordination of international aid. Projects implemented and financed by the EU, ICRC and Canadian government were building strong operational experience which integrated nicely into the overall reform framework. This framework was mostly the task of the WB financed projects implemented by the MOH. Both projects were crucial in driving reform of the Serbian health system. Several factors allowed these projects to achieve good performance: reliance on local capacities, constant programmatic support from the WB (including both design and implementation phases), combining different lending mechanisms to strengthen implementation, good collaboration with other donor activities, strong administrative foundation. During the preparation phase and in early stages of implementation, the project relied heavily on international technical assistance. However, shift was made to try to engage local staff in combination with top individual foreign experts and advisors. Apart of better utilization of loan funds, the result was creation of a broad pool of trained local experts that was capable of taking most of the burden during the rollout phases. Administrative WB procedures introduced through the loan were instrumental in the quick implementation of project activities. Even when procedure was deemed lengthy and complicated, long-term benefit was evident since all tenders were completed without complaints and delays that usually accompany local Serbian procurement. Financial management was fully implemented according to the WB procedures, but provision for some local bookkeeping requirements should be made when staffing the finance unit. Achieving Project Objectives Most of the project objectives were achieved. Some achievements were not part of the results framework (activities described in sections 15 to 41), but were an important factor in preparing Serbia for upcoming health reforms. Three major achievements of the WB and MOH that were facilitated through the project could be clearly stated. Quality of health care and health care management became a clear priority for the health system in Serbia. The first operational Agency for accreditation in the region has been established and almost half of the general hospitals went through the process of 34 accreditation. Health financing reform was pushed through the project and foundation was laid, although this long overdue reform would not have been possible without technical assistance provided through the project and constant effort exercised through DPL. Information and Communication Technology (ICT) strategy for health, prepared by the project team, was one of first IT sector strategies in Serbia. Standards and accompanying regulation established for software used in health institutions is unprecedented in wider region, and it is now used to set an example. Implementation of the hospital information system (HIS) is done with minimal cost and (in combination with EU funding) 58% of bed capacities will be covered by IT by mid 2013. Government and World Bank Performance SHP was one of the first projects that the WB financed in Serbia. At the beginning, both partners were ambitious to use the project in remedying problems that were built over the previous 10-year period. Such an ambition quickly proved to be too optimistic, since political environment in Serbia didn’t allow health to become a priority and capacities for policy-making, change and implementation needed time to grow. Also, the project, as designed, was too ambitious, especially considering the fact that this was the first WB- financed project in the health sector in Serbia. It lacked clarity in the definition of activities to be implemented as well as strategy for how to implement activities under each sub-component. The WB response to these problems was inadequate since 4 TTLs were changed in just over two years (including preparation, grants supervision and project design). From the first quarter of 2005, the WB and MOH strengthened their monitoring activity and, during the following 12 months, the project was fully up and running and in the position to regain satisfactory status. During the same period the project was restructured. While the development objectives, the general architecture of the project and the total budget remained unchanged, the scope as well as the number of activities was reduced. The restructuring proved that WB was able to act as a partner and politically and technically support MOH in finding realistic strategy for improvement of the health system performance. Agreement reached on restructuring permitted MOH to focus its efforts on a more manageable number of activities and to enable the project to complete them in a more effective manner. In that sense, the MOH was able to build project findings and recommendations in the new Health Care and Health financing Laws and various by-laws, showing commitment towards achieving the project goals. In addition to that, the Government increased its share in the project by 4 million dollars maximizing benefits for the health sector. Furthermore, the WB and MOH were able to influence conditions for disbursement of PDPL, thus ensuring that implementation of reforms started by the project would take place. Beside that, WB directly contributed to enhancing analytical capacity by conducting of a Baseline Survey of Cost and Efficiency in Primary Health Care Centers before Provider Payment Reforms that took place in 2008; with intention of repeating the survey after provider payment changes actually take place. 35 Sustainability Most of the processes activated by the SHP and SHPAF become standard in delivery of health care in Serbia. Input from the WB-financed projects is evident in the Plan of Development of the Health Care System 2010-2015 adopted by the Parliament of the Republic of Serbia, Poverty Reduction Strategy, Strategy for the Development of the Informatics Society in the Republic of Serbia Until 2020, e-Health2015 IT Strategy. However, part of the activities just started or started in a limited number of health institutions. Some reform breakthroughs are only in the very initial phase after a long period of training and survival of the early phases of change management. Taking into account that global economic crisis and further reduction of the health budget as a percentage of gross domestic product (GDP) are limiting investment capacity of the MOH, it is necessary to ensure additional support for the reform process. Logically, the focus of this support should be the health financing reform. 1. Health Services Restructuring 1.1. Hospital Renovation and Equipping a) Unnecessarily large space (app 4500m2), dislocated diagnostic services and lack of some medical equipment were the main issue in the Kraljevo Hospital. The main activity in Kraljevo Hospital, in this component, included reconstruction of the new Pediatric department and complete vacation of the old Pediatric building. The total space of the outpatient polyclinic in the hospital was reduced by app. 3000 m2. The three principal buildings of the Kraljevo Hospital were disconnected, so the construction of an underground passage was financed from the SHP. Top level of internist building was reconstructed to accommodate the pulmonary ward and ground floor of the surgical building was reconstructed to accommodate the diagnostic centre and centre for day surgery. The newly-established diagnostic centre is now fully functional. Centralized diagnostic centre increased the efficiency of patient flows and improved diagnostic support for emergency services and for PHC. Medical equipment procured for Kraljevo Hospital improved services and ensured faster diagnostics (US devices, EEG, ECG devices, fiber-optic laryngoscope, ECG halters, tissue processor and automatic strainer for pathology lab, microscopes) and treatment in life-threatening conditions (surgical instruments, anesthesiology apparatus, operating table, video-endoscopy sets). Vranje Hospital had two main issues: ten wards were dislocated in ten different buildings without physical connection and it had not had a central surgical block and central intensive care unit. Immediately after the start of the SHP, minor works included moving the neurology ward from the old, inadequate building to a larger adequate space with better working conditions. The result of the moving was evident from the economic point of view because less money is now used for fuel, electricity and water supply and the maintenance expenses have been reduced. Major hospital reconstruction in Vranje, supported by SHP, started with hospital plan for 4 phases needed to reach the intended 36 flow of services, detailed design for the first phase of the construction works, geotechnical and seismic assessments as well as cadastral survey. It was obvious that civil works will take more time and resources than envisaged by the SHP and after restructuring of the project MOH committed its own budget to start with construction of the new building. This new building, the central surgical block, will be attached to main complex, with the idea that it will provide better multidisciplinary health care service (especially for the cases of the most difficult polytrauma patients) as well as better efficiency and effectiveness in the use of medical equipment. The equipment procurement for Vranje Hospital included ECGs holter, EEG, EMG, defibrillator with monitor, non-invasive blood pressure monitor, ophthalmoscopes, arthroscope and US unit for urology. And second delivery that was mainly for laboratory and surgical ward included: biochemistry analyzer, blood cell count apparatus, glucose analyzer, flame photometer, capillary electrophoresis device, centrifuge, microscopes, operating-ENT microscope, endo-video gastroscope, general surgery operating lamps and tables, cardiac ultrasound imaging system, hot air sterilizers). They were aimed at the improvement of the general quality of services (reduced number of patients going out to other health institutions, improved quality of services delivered and faster proceeding of diagnostic services). For the needs of urology, gynecology, pediatrics and surgery ward, the ultrasound probes that had been out of function were replaced. That enabled Vranje hospital to perform diagnostic procedures to the patients of the aforementioned wards who had been obliged to go to other institutions (other neighboring health centers, private clinics, etc.) Valjevo hospital has four separate hospital sites. Connection of the buildings and consolidation of these services was proposed as a way to decrease costs (for maintenance, utilities, etc) and increase efficiency (reduce the time required for transport, make services more accessible). Hospital reconstruction in Valjevo, supported by the local fund-raising and SHP, provided space sufficient for consolidation and reorganization of services in Valjevo hospital. Construction of the new building (pulmology department with 55 beds) connected to the existing buildings allowed sufficient space for reorganization of the pathology, cardiology and coronary unit. Rehabilitation of the hospital included also some minor works: interior modifications of the hospital surgery block entrance hall and external illumination works. These activities resulted in safer conditions for staff and patients, improving access and organization of services. Valjevo hospital had been equipped with old and out of date equipment that required frequent maintenance. The project supported the purchase of X-ray tubes that had been overused with an excessively high total number of expositions, thus creating unsafe environment for the patients. Major medical equipment was procured in two phases. The first phase of equipment procurement aimed at improvement of general quality of services, providing ECGs, defibrillators, monitors, for all departments, and ophthalmoscopes, inhalators and fetal monitors for some departments. The second phase provided equipment mostly for the new (pulmology) building, improving services of departments placed there. The second tender also brought some items that are in 37 accordance with the idea of improvement of the service quality for the entire hospital (syringe pumps, electrosurgical knives, laboratory equipment, etc.). Zrenjanin hospital was a campus of number of disconnected small old buildings (some older than 60 years) and unfinished huge 10-storey new hospital that was an empty shell and in 2003 utilized less than 10%. The initial plan to use SHP to furnish a huge central sterilization ward in the new hospital building for Zrenjanin hospital was abandoned in early 2005 since it was impossible to secure the next phase of the investment. Therefore restructuring of Zrenjanin hospital included rehabilitation of part of the new hospital for the needs of moving the delivery ward, neonatology and obstetrics from the old buildings (total surface of 1500 m2). Finalization of this area included admission room, sanitary pass, pre-delivery room, 4 delivery rooms, post-delivery room, operating theater for Cesarean Section, nurse stand, patient rooms with 30 beds, neonatology area with 25 beds for newborn babies, area to accommodate the incubators for prematurely born babies as well as sanitary and auxiliary premises and communications. Construction works included purchasing and installation of 2 elevators and mounting of distribution channels for the 04 floor, where the surgery inpatient ward was planned. The positive results of moving departments from the old buildings to the new hospital building is evident from the economic point of view (less money is used for fuel, electricity and water supply and the maintenance expenses have been reduced). Hospital in Zrenjanin had two rounds of medical equipment supply, mostly intended for the delivery ward and the neonatology department. Part of the equipment was intended for diagnostic imaging, coronary and intensive care unit. Medical equipment provided improvement of the service quality in the mentioned departments. 1.2. Improving Hospital Management In the four general hospitals participating in the project, the number of beds and average length of stay show negative trend that is faster than for other hospitals in Serbia: number of beds (from 2906 in 2003 to 2312 in 2010) and average length of stay (from 8.56 in 2003 to 7.52 in 2010). Concurrently, the mean occupancy rate and number of discharges show a positive trend that is also faster than for other hospitals in Serbia; bed occupancy rate (increase from 62% in 2003 to 71.7% in 2010) and the number of discharges from hospitals (from 77109 in 2003 to 80427 in 2010). Through the work of the so-called “Club 4â€?, made up of top management of 4 regional hospitals, the project was working thoroughly on improvement of the general level of awareness of modern health management practices in public hospitals as well as on implementation, monitoring and evaluation of standards for the secondary level hospitals. Moreover, that was the way of strengthening the institutional capacity around the country enabling the training for both mid-level and senior managers (more than 30 of them have passed a 5-module management course organized by the School of Public Health in Belgrade). This led to the improvement of functional ability of senior clinical/administrative staff assigned to 4 designated hospitals to manage their services more effectively and efficiently e.g. better implementation of key measures to increase performance, control operating costs and improve quality of services provided at the 38 hospital level. At the same time, working with the same group of managers, the project has established milestones in the area of building up safe and efficient medical waste management. The project carried out a needs-assessment/situation analysis and developed a plan of action by setting up medical waste management teams and organizing a three- month activity of collecting data on the type and the quantity of medical waste in all 4 pilot-hospitals. Recommendations together with new standards for collecting and disposing medical waste according to type and origin (the color-coding system) served as a starting point for the two EU projects that followed later on and worked on upscaling the achievements all around the country and on all three levels of care. The project supported the preparation and implementation of an audit of 18 health care institutions (15 largest hospitals and 3 largest primary health care centers) with a view to help define key measures to better control and reduce operating costs. It helped streamline the audit and results showed great improvement in the area of performance and quality indicators, but real financial audit was not easy to perform due to the bookkeeping procedure. The project also supported the Chamber of the Health Institutions of the Republic of Serbia (CHIS) in the light of improvement of performance of managers of health institutions. Regarding this fact, the draft criteria for assessment of performance of hospital managers have been developed with the database of available courses and trainings in health management aiming to map the main area of performance, to highlight the main responsibilities of managers and to award and motivate the most successful ones. The improvement of health management was deemed very important in the light of introduction of new financing models-using the diagnosis related groups system and capitation payment, by promotion of the model of efficiency and raising awareness of the importance of making significant savings among health leaders. The energy efficiency model was also promoted by organizing the meeting and fair which gathered the managers of all health institutions in the Republic of Serbia. The membership in HOPE (European Hospital and Healthcare Federation) of the CHIS, achieved also through the project, is the significant step in exchanging the knowledge and experience with EU countries. Other activities that have been done: trainings in creation of project proposals for the EU IPA fund, introduction of trainings in EU health policy for health managers; supporting the Chamber in obtaining participation in EU FP7 Project, etc. 1.3. Flu Vaccine Production at Torlak Institute for Virology, Vaccines and Sera Based on strategic documents prepared by European Agency for Reconstruction - EAR Consultant (Strategy for Developing Torlak Institute for Virology, Vaccines and Sera and Investment Strategy for Torlak Institute) MOH and WB embarked on investment into influenza vaccine production at the Torlak Institute, starting from November 2006. Although the plan indicated an investment of approximately US$ 1 million, the whole investment divided in three phases was worth US$ 6.5 million (SHP contributed 1 million, MOH from the budget – US$ 3.6 million, Torlak Institute – US$ 0.8 million and World Health Organization – US$ 1.1 million). During 2006, SHP directly financed technical assistance needed to prepare and implement design required to establish production of influenza vaccine according to good manufacturing practice and purchase of a major piece of equipment, the centrifuge. Towards the end of 2008, MOH provided the 39 investment needed to start bulk production in 2009. Final stage of investment provided by Torlak and WHO was used in 2009, when the test production started. The first test was unsuccessful and, after a change of leadership, Torlak never tried to launch production again. 2 Health Finance, Policy and Management and Quality Improvement 2.1. Health Finance 2.1.1. Provider Payment Reform Before the start of the SHP, the Government of Serbia has already expressed its intention to change payment system for provision of primary health care from input-based system to system based on capitation. To support that intention and in accordance with PDO, MOH carried out the initial work to support introduction of capitation through the SHP project. Comprehensive analyses of different aspects of service provision on primary level have been completed (legal aspect, payments for quality, unavoidable costs, contracting, access, copayments, etc.), and a basic structure of capitation formula has been developed and proposed. Further work on introduction of capitation was continued by EU funded project with main focus on support to implementation of capitation in Serbia. One part of the PDO refers to accessibility of affordable and effective healthcare, where providers are rewarded for quality and efficiency. Furthermore, by the time the SHP project started, the Government expressed interest in changing input-based provider payment system for the secondary and tertiary levels of healthcare provision. In accordance with all that, a group of SHP project activities was directed towards support of provider payment reform for hospital care in Serbia. An analysis of the current provider payment system for hospital care was completed with assessment of different provider payment options for introduction. That analysis proposed that system of diagnosis related groups (DRG) should be introduced in provider payment system for hospitals in Serbia. Afterwards, an analysis of different DRG systems that are in use worldwide was completed to assess which DRG system would suit Serbia best for implementation. That assessment was assisted by medical professionals in Serbia, and Australian DRG system (AR-DRG) was assessed as most suitable. A trial license for the use of AR-DRG was obtained with all other preparatory work done, and piloting of that system took place for three months in 2008 in six hospitals in Serbia. AR-DRG system was piloted as a reporting system only. The results of that pilot were very encouraging, having all hospitals reporting successfully according to the new system with very high percent of correct reporting. Furthermore, medical professionals from hospitals showed high level of satisfaction with the new system, judging that support for reporting that had been provided to them during piloting was of great help. The key conclusions were that AR-DRG system was implementable and applicable in Serbia, that training and continuous support was very important. 40 MOH took activities related to introduction of DRG in Serbia to another level through SHPAF. After a period of negotiations with Australian authorities and the WB, through SHPAF, in early 2011, Serbia obtained a license for use of Australian DRG system. After that, SHPAF supported introduction of components of AR-DRG system into legislation, primarily through work with experts in different medical fields to adopt, accept and support necessary legislative changes. These changes are necessary to make AR-DRG system possible for use first as analytical, and later as a reporting system and finally as payment system. Furthermore, through cooperation with other central institutions of the health system in Serbia (RHIF, IPH) MOH through SHPAF project supported institutional capacity building related to DRG system and its implementation and usage. The usage of AR-DRG system as an analytical tool would be possible as early as 2012. 2.1.1. National Health Accounts National Health Accounts (NHA) in Serbia began in 2004 with the support of the SHP. In 2006, the first NHA was conducted with the guidance of two international consultants and sets of data for 2003 and 2004 were produced. Subsequent NHA rounds followed in 2007, 2009 and 2010. Since 2008, NHA has been entirely government-financed, supported by a routine line-item budget from the MOH. Currently, NHA falls under the purview of the Republican Institute of Public Health (IPH), commissioned by the MOH to produce NHA annually. International consultants do not provide technical expertise and the work was no longer financed since its transfer to IPH. The project support for this activity was limited to occasional provision of training for IPH staff and support in preparing the new Law on Medical Records, which will be instrumental for further improvement of NHA. Presenting capability to work on NHA and keep step with more experienced countries, Serbia has been included in the joint WHO, EUROSTAT and OECD work on “A System of Health Accounts (SHA)â€? Version 1.0 and Version2.0 2.2. Quality Improvement 2.2.1. Continuous Quality Improvement a) Continuous Quality Improvement. The MOH highlighted that health care quality improvement was one of the key priorities in developing health policy in Serbia. MOH established the Quality Unit under the project, comprising four components: Continuous Quality Improvement, Accreditation, Health Technology Assessment, Licensing and CME. The main goals of the continuous quality improvement component within the scope of the SHP were: support to the MOH in setting up a continuous quality improvement system and setting up of effective mechanisms for implementation of quality improvement policy on the national and local level among all stakeholders. Project activities at the national and local levels were related to developing and implementing national quality improvement policy and building capacity of the staff in four pilot centers. The project used its five-year experience and results in the pilot centers to lay down the national Strategy, (“Continuous Improvement of Health Care Quality and Patient Safetyâ€? Draft Strategy). The project established the Annual National Conference on Continuous Quality Improvement. 41 b) Project has supported the National Quality Assurance Committee (NQAC) to shift its regulatory, oversight and quality assurance role by helping the Committee to carry out a list of activities such as: i) producing the National Action Plan on Quality Improvement, ii) Book of Rules on Quality Indicators, iii) public promotion (two National Conferences on Quality and Patient Safety, iv) development of Integrated Quality Improvement Plan and conducting 14 rounds of workshops for quality teams from all types of institutions (all in all, the project has hosted around 500 workshop participants). Integrated Quality Improvement Plan will serve as a practical management tool to increase hospital productivity, efficiency and quality of service delivery; v) moreover the project has supported National Committee for Hospital Infections Surveillance to work on education and motivation health professionals from IPH and hospitals to work together on prevention and suppression of HAI. Through the process of Continuous Medical Education, that covered a network of all 22 institutes of public health and 60 corresponding acute hospitals of different levels of care, the project succeeded in examination of basic epidemiological characteristics of HAI in health care facilities, established the most frequent localization of HAI, elaborated the use of antibiotics and encouraged health professionals to visualize, monitor and carry out prevention and control of HAI. Results based on the sample of more than 13.300 participants will serve for improvement of the national strategy regarding collecting data on the prevalence of HAI. Results have been presented to the top management of all 347 health care institutions at the national conference on quality improvement and patient safety and, moreover, the results were presented in the form of a printed publication; vi) in that sense, the project enabled the campaign entitled Save Your Lives, Wash Your Hands that covered all hospitals (the project has provided promotional material), also, over 30 hospitals have been actively involved in the celebration of May 5th – WHO Clean Hands Day. 2.2.2 Accreditation of Health Institutions a) Accreditation sub-component was designed to support the early phase of establishment of an accreditation body for health care providers, which over the medium to long term will continually improve the quality and safety of their services. For the newly established accreditation body, staff members, office space and office equipment have been provided and initially financed from the loan on a gradually decreasing scale as financing of the organization was planned to be gradually taken up from accreditation fees paid by the providers. Activities supported by the project included development and implementation of an accreditation program for Serbia and plan for development of the accreditation body; technical assistance on international best practice in accreditation; training of surveyors and quality coordinators; development of standards and survey procedures; piloting and roll-out of accreditation standards; study tours and training courses. Project objectives were fully accomplished and 16 primary health care centers (15 received accreditation award) and 4 hospitals (2 received accreditation award) went trough the accreditation process. From 2008, the Accreditation Agency was also supported through an IPA-funded project which was complemented by SHP and DILS activities. 42 b) Support continued through SHP AF from 2009 to 2011 by providing resources for accreditation of 13 secondary and tertiary health care institutions. 2.2.3. Health Technology Assessment a) The project was mainly designed to introduce Health Technology Assessment (HTA) in the process of decision-making and has managed to build up capacity on how to use health technology assessment as a knowledge base for improving the quality of health care. The whole process enabled Serbia to proceed to the institutionalization of HTA. Through the process of “problem-based learningâ€?, the Project has carried out several types of trainings and capacity building on health strategies for policy and decision makers as well as capacity building for Clinical Practice Guidelines and HTA practitioners. The project has resulted in three HTA reports and a guideline on how to develop guidelines. Also, during the extension of activity within the same project, a local group of experts developed the Feasibility Study on Institutionalization of HTA and the first edition of Rules of Development and Position of Basic Benefit Package in Health Care System in Serbia jointly with international consultants. During the same project, Serbia was broadly been recognized and jointly engaged in the preparation of the Handbook on HTA Capacity Building through the Work Package 8 of the EUnetHTA Project (2006-2008). b) The project has supported evidence based principles such as: a) development of new and update of existing clinical practice guidelines. Based on the guideline on how to develop guidelines, the newly-established Committee on CPG developed 8 CPGs: Guideline to Diagnosis and treatment of Lipid Disorders, Guideline for Ischemic Heart Disease, Guideline to Arterial Hypertension, Guideline Inguinal Hernia Treatment of Adults, Guideline for the Disorder of the Thyroid Glandule, Guideline for Depression, Guideline for Lung Cancer, Guideline for Ischemic Stroke. The project supported implementation of these guideline by organizing 12 rounds of workshops in 4 regional centers. All workshops were attended by more than 880 participants –various types of health care professionals of all levels of health care; b) based on policy and decision making process in the area of medical equipment e.g. the obsolete medical devices nomenclature has been harmonized with Global Medical Devices Nomenclature (formal adoption is waiting for the new Law on medical records). Medical equipment inventory has been initiated: an awareness raising meeting with top management of all 347 health care institutions resulted in more than 98% of response rate during the next two levels of trainings among technical staff of all institutions. Namely, 75 trainings with approximately 25 participants per unit was organized at the county level, the first level aimed at general audience was organized in the network of county institutes of public health and advanced level of trainings covering all type of institutions were organized in all primary health care centers and hospitals in the county capital. The task of inventory will be given to the National Institute of Public Health for future work. 43 2.2.4. Licensing of Health Professionals and Continuous Medical Education Using support of the national and international experts engaged through the SHP, in 2005, the MOH drafted the new Health Care Law which, among other reform innovations, enabled the introduction of licensing for health professionals and Continuous Medical Education (CME). In the process of establishing five new chambers of health professionals the SHP supported consultation with more than a hundred esteemed health professionals and process and preparation of by-laws needed for regulation of the new bodies and positioning of the CME in the overall health delivery system. Based on these inputs, MOH published the by-laws in late 2007. Chamber of Doctors, Chamber of Nurses and Medical Technicians and Serbian Chamber of Biochemists had their articles of association prepared in 2006, but full implementation awaited until March 2009, when Health Council was set up and started to accredit CME curricula. 2.2.5. Clinical Pathways The key goal is to support capacity building in the area of Clinical Pathways (CP) development and implementation, as well as the development of methodology of clinical pathways defined according to EPA guidelines. The project has organized large-scale seminars with top management of hospitals and primary health care centers, CP coordinators from the Institutions, MOH, decision makers, the Health Insurance Fund and the Public Health Institute (2 seminars before the start of work with the pilot institutions - 350 participants were educated). The Project has carried out a two-day seminar with the health inspection of the MOH, decision makers, the Health Insurance Fund and the Public Health Institute (130 participants were educated). The project organized study tours to Slovenia and Hungary with directors and CP coordinators, MOH and the RHIF. The main aim of the pilot project was to train physicians and nurses in hospitals all working groups in 18 hospitals on how to prepare, implement and effectively use care pathways in patient treatment and clinical management (around 1000 participants trained); During the project institutions developed 3 to 6 CPs: Child Asthma, Primary Prevention of cardiovascular diseases (CVD), Secondary Prevention of CVD, Iron Deficiency Anemia, Depression, Diabetes, Screening of Colon Cancer, Low Back Pain, Myocardial Infarction, Colon Carcinoma, Child Asthma, Normall Delivery, Stroke, and Cholecystectomy they implemented and evaluated the impact of the regular use of care pathways. They improved quality and efficiency of health care services provided. 2.3. Health Information System 2.3.1. Support to RHIF Republic Health Insurance Fund recognized the need to develop a data model and a functional model for the Database of Insurees (DoI) and a secured private Wide Area Network (WAN) that would provide infrastructure for the Database of Insurees. Some of the RHIF branches did not have local databases of insurees, while most of them had, but they were not homogenous. By 2008, SHP provided establishing of central Database of 44 Insures (DoI) through the process of data migration from databases of 7 local HIFs and installation of WAN network, which connected 170 HIF locations. 2.3.2. Central Information Service The purpose of the Central Information Service (CIS) is to provide information to all stakeholders in the health system. Presently, CIS is a database of health resources, nomenclatures, classifications, registers etc. It was implemented as web portal and web application during the period from 2006 to 2008. Since 2011, through the web application, health institutions have updated their resource data, including those on human resources. A software module for medical equipment was developed in the first phase of the project (2006 – 2008), but it has not become operational yet. After defining/renewing the methodology of keeping records of medical equipment, it is necessary to upgrade the module for medical equipment in line with the new methodology. The CIS portal contains nomenclatures and classifications of interest for the health information system (HeIS), like JKL, ATC, ICD 10, IPCP 2 etc. The national IPH hosts this database and takes responsibility for storing data, updating, training and support. Further development of the Central Information Service will be implemented in phases, where each phase will “go liveâ€? as it is ready, so that the CIS functionality will grow in time. It is expected that further improvement of the CIS will be carried out under the DILS Project in 2012, as a support for the Dr Milan Jovanović Batut Institute of Public Health of Serbia. 2.3.3. Primary Health Care Software SHP provided technical and functional upgrade of existing software in Primary Health Care Center in Kraljevo. The MOH has the proprietary rights over that application. That means that, once developed for the first pilot region (Kraljevo), that application will be offered free of charge to any health institutions which have interest and resources to utilize them. But, in order to ensure good quality (as hospital information systems), MOH started to establish rules-standards through appropriate legislation. 2.3.4. Hospital Information System MOH purchased the LAN network, hardware and the services of development and implementation of the hospital information systems (HIS) in 4 hospitals under the SHP. In order to ensure good quality, two hospital softwares were developed, under the same technical requirements, and implemented - HIS1 in Zrenjanin and HIS2 in Kraljevo, Valjevo and Vranje. Also, in order to ensure good quality of the purchased items/outputs and their compliance with the specified requirements (patient medical data, support of business processes, mandatory reporting) an additional process of quality assessment was performed as a part of the final acceptance in June 2008. All of the main business requirements (admission/discharge, inpatient departments, outpatient departments, diagnostic departments, laboratory and central hospital pharmacy) and the detailed technical requirements were met, and the overall results of the international quality assessment for both HISs were satisfactory. Final acceptances for both systems were performed in June 2008. The MOH holds the proprietary rights over those applications. Hospital information systems (HIS1 an HIS2) are the results of the SHP (see paragraph above). SHPAF provided upgrading of both systems according to The Rulebook on more 45 Detailed Contents of Technological and Functional Requirements for the Establishing the Integrated Health Information System, issued in November 2009 (see paragraph below) and implementation in all 20 hospitals. During the project, with the support of ProRec Serbia, HIS2 (called Heliant) was certified by the EuroRec Institute. Also, the project provided procurement and implementation of the Laboratory Information System (LIS), which is connected with analyzers and automatically receives results from analyzers in 20 hospitals 15 and IT equipment for RIS-PACS software for 6 hospitals 16 . Radiology Information System (RIS-PACS) is state-of-the-art digital technology for processing radiological images. Basing such a system on web technology allows data to be accessed from different locations within a medical facility, as well as from remote locations outside the institution, which is the basis of telemedicine. In the course of 2012, during the warranty period, all systems (HIS, LIS and RIS) will be integrated. 2.3.5. IT Strategy, Regulation and Standardization SHP provided creation and adoption of the Regulation named Program of Work, Development and Organization of the Integrated Health Information System – e-Health 2015 (Official Gazette of the Republic of Serbia, Issue No. 55/2009) and Rulebook on More Detailed Contents of Technological and Functional Requirements for the Establishing the Integrated Health Information System - IT Rulebook (Official Gazette of the Republic of Serbia, Issue No. 95/2009). In accordance with Action Plan of the e- Health2015 program, the first iteration of the entire health care informatization will be completed by 2015. The compliance of software with the IT Rulebook is mandatory for existing software solutions, as well as for new ones. Implementation of software in primary health care, within the DILS Project, is being carried out by each primary health care centre by purchasing one of the 8 certified software solutions (the solution from Kraljevo is one of these 8). With the support of SHP projects, ProRec Serbia, association for standardization of EHR (Electronic Health Record) products, was founded in 2008 and became a full member of a EuroRec Institute (the European Institute for certification and promotion of quality EHR systems). So, the European standards and requirements were included in the IT Rulebook. SHPAF provided implementation of the Information Security management System (ISMS) in accordance with ISO 27000 in some departments of KBC DragiÅ¡a MiÅ¡ović and General Hospital in Zrenjanin. The international certification body will perform assessment of compliance of the Zrenjanin system with ISO 27000. Additional value of this activity is knowledge and experience regarding ISMS that will be implemented in the 15 Hospitals included in the project: Clinical Center of Serbia - Obstetrics and Gynecology Clinic, Clinical Center Kragujevac - Surgery (the whole building), Institute for Mother and Child, Clinical Hospital Center “Dr DragiÅ¡a MiÅ¡ović - Dedinjeâ€?, general hospitals in Vrbas, Sremska Mitrovica, Å abac, Požarevac, Jagodina, Smederevska Palanka, Aleksinac, ZajeÄ?ar, Užice, Leskovac, Petrovac na Mlavi, Surdulica, Gornji Milanovac, Novi Pazar, the Narodni front Obstetrics and Gynecology Clinic, Institute of Mental Health in Palmotićeva St. in Belgrade. The last 4 hospitals are co-financed from SHPAF and IPA projects. 16 Clinical Center of Serbia - Obstetrics and Gynecology Clinic, Clinical Center Kragujevac - Surgery (the whole building), general hospitals in Kraljevo, Valjevo, Zrenjanin and Leskovac 46 new version of the IT Rulebook (DILS project, 2012). So, the requirements related to the ISMS (ISO 27000) will become mandatory for all IT and healthcare providers. 2.3.6. Wide Area Network for Health Institutions SHPAF includes the installation of WAN, the infrastructure for data exchange (scheduling specialist examinations at the hospital level by selected doctors, access to electronic health record by a hospital specialist, integration of the primary health care software with Laboratory Information System which belongs to referral hospital. WAN connects over 360 institutions of the health sector. Through a joint effort of the MOH (SHPAF) and the Ministry of Defense, WAN will be also be connected to the Ministry of Defense Health System that is already on the way of integration with the main health network operated by MOH. 2.4. Capacity Building and Communication of the Ministry of Health 2.4.1. PR Activities and Patient Information Campaigns Public relations (communication) in the MOH and overall public health system was constantly improving thought SHP project. With engagement of local and international technical support, SHP developed the MOH communication strategy and was instrumental in supporting implementation of the strategy through following activities: supporting the PR-Communication Unit (3 persons) that was established in MOH. The position of the PR officer has been introduced in all 400 health institutions in Serbia, including IPH and RHIF. In cooperation with American International Republican Institute (IRI), more than 400 persons passed elementary media and communication training; Two large-scale public information campaigns were conducted, which pushed the issue of a patient’s position within the health system towards the top of the agenda: increasing knowledge about patient’s rights, and improving communication between patients and health professionals. Other PR campaigns of the MOH (anti-smoking campaigns, HIV/AIDS, avian flu, the World Health Day, waiting lists, restriction of patient’s rights in dental services) were all supported by the SHP through provision of technical assistance; PR staff was also part of the SHP implementation team in order to provide constant communication activities, increasing the visibility of the project in public (general and professional). Following increasing media coverage of the health issues, the public became an important quality control instrument. Better information of patients helped in raising awareness needed for decreasing out-of-pocket payment and increasing utilization of “freeâ€? health services and increased overall patient satisfaction with the health care system (measured and controlled by the project PR team). 2.4.2. Health Survey of the Republic of Serbia The 2006 Health Survey of the Republic of Serbia (the 2006 NHS) was a follow-up survey of the NHS, 2000, which had been carried out before the democratic changes in October of the same year. The 2006 NHS pursued practically the same methodology, so that the results of that study provide a solid foundation to evaluate the effects of the MOH’s work on the reform, development of the health care system and health protection and promotion among the Serbian population. The 2006 NHS used a representative sample of the Serbian population over 7 years of age, at the national level and at the level 47 of six geographical regions: Vojvodina, Belgrade, Western Serbia, Central Serbia, Eastern Serbia and Southeastern Serbia, as well as at the level of urban and non-urban settlements. The following persons were interviewed: 6,156 households, 14,522 adults aged 20+ years and 2,721 children and adolescents aged 7–19 years. The health survey provided precious data on social and economic health determinants, health status based on self-assessment of individuals, lifestyles, functional abilities, use of health care services and expenditures associated with health care. 3. Project Management, Monitoring, Evaluation a) Initial Project Coordination Unit (PCU) was staffed with Director, Administrative Assistant, Procurement Specialist and Financial Specialist. The only programmatic full- time consultants during the first nine months of operation were four field coordinators and a PR officer. The first year of operation was marked with the fact that Director resigned after less than six months, leaving PCU without leadership for more than 6 months. This period was also marked with low commitment and disbursement of funds, establishing unsatisfactory status for the project. From, the beginning of the second year, PCU was fully staffed and Health Financing and Quality units were set up. Project picked up pace, especially after restructuring in 2005 and more than 100 contracts were signed. In terms of implementation, the monitoring unit was reporting on a weekly basis during the period of intensive implementation while there was an imminent need to secure the satisfactory status. After regaining the status, the reporting both to the Ministry and the WB was delivered on a needs basis, without insisting on regularity. b) The experience and solid work of the unit was recognized by both the GoS and the Bank and the implementation arrangements were changed in 2009 through establishment of the Fiduciary Services Unit (FSU) composed of the project finance and procurement staff. This FSU was additionally staffed and appointed by the GoS to support multi-sector Delivery of Improved Local Services (DILS) Project 17. Thus, the FSU was successfully supporting both DILS and SHPAF in terms of procurement, financial management and overall coordination. Since each ministry was allowed to form its own consultants implementation team MOH used the same people to implement SHPAF and DILS. All component coordinators (IT, Quality and Health Financing) were coordinators for both projects ensuring complementarities and coordination of activities. Monitoring of both projects was regular in a sense that responsible WB staff was situated in Belgrade and Sarajevo, allowing intensive communication and consultation. Combined DILS and SHPAF Aide Memoires were used as reporting documentation. MOH was using Ministerial Collegiums and standard government reporting as management tools. From mid-2011, the steering board was established by the new Minister of Health and reporting was additionally channeled through the conclusions of the steering board meetings. 17 Ministry of Health, Ministry of Education and Science, Ministry of Labor and Social Policy 48 B. Comments on Draft ICR from the Ministry of Finance and Economy Republic of Serbia MINISTRY OF FINANCE AND ECONOMY No: 401-129l/2012-00l Belgrade, September 19,2012 THE WORLD BANK -Mr. Loup Brefort, Country Manager for the Republic of Serbia Bulevar kralja Aleksandra 86 Belgrade. Republic of Serbia Subject: Serbia Health Pro;ect (Loan No. JBRD-76950. IDA-37680) Drafi Implementation Completion and Results Report Dear Mr. Brefort. Please allow me to express my sincere gratitude for the support that the World Bank is providing to Serbia's health sector reforms and improvement. Regarding the draft Implementation Completion and Results Report, we would like to inform you that we reviewed the Rcpon and in accordance with the competency of the Ministry of Finance and Economy we have no comments. Sincerely. Iv1 a KOJIC •• State Secretary 49 C. Comments on the Draft ICR from the Ministry of Health MINISTRY OF HEALTH OF THE REPUBLIC OF SERBIA COMMENTS ON THE DRAFT FINAL REPORT OF THE WORLD BANK REGARDING «SERBIA HEALTH PROJECT» AND «SERBIA HEALTH PROJECT – ADDITIONAL FINANCING» Paragraph 4 – Serious challenge that the system of financing was facing at the beginning of the project, beside already mentioned, was also a lack of the IT system in HIF that would enable evidence-based management and functioning of the system. For instance, there was no Database of Insurees, whose creation and operationalization was strongly supported by the Serbia Health Project activities. Subtitle 1.7 Other significant changes – As it was mentioned in the text in paragraphs 16-19, project has been restructured in June 2005. However, during the restructuring certain ammount of funds have been unallocated. Therefore after restructuring, at the end of 2005, it has been decided (with World Bank collaboration) that those funds should be used for the reconstruction of the flu vaccine production line on the Institute for virusology, vaccines and serums «Torlak». That represented significant change. Paragraph 33 – It is correct that one of the reasons for Unsatisfactory rating for project implementation in December 2004 was resignation of the PCU Director. However, additional problem was that, after that resignation, PCU was without Director for the following 9 months. That significantly slowed project implementation and made it more difficult and, in combination with other factors, led to project restructuring. Paragraph 42 – It is correct, as it is already mentioned in this Paragraph, that responsibilities for provision of health services are indeed tranferred from State to local level of government as part of the decentralization process. Nevertheless, that is correct only for primary health care, and not for all levels of health care, as text in this Paragraph could be understood. This difference should be clearly underlined in the text. Paragraph 44 – International Committee of Red Cross should also be mentioned in this Paragraph next to Canadian International Development Agency and European Commission, as an organization that identified introduction of capitation as an area that Serbia needs support in. ICRC project in Kraljevo (2000-2004) through its activities started work on introduction of capitation through pilot project in Kraljevo Primary Health Care Center. Paragraph 48, Outcome indicator #1 – Next to already mentioned guidelines, through its activities project supported development and introduction of Integrated Plans of Quality Improvement (gives guidelines for improvement of quality of healthcare and patient safety in an institution); Guideline for Clinical Practice Guidelines (enables that CPGs are made in accordance with international practice, what strengthens evidence- 50 based health care provision); Guidelines for improvement of control and reduction of hospital acquired infections; and Standards for Accreditation. Paragraph 65 – Next to already mentioned projects, close coordination and synergies have been achieved with more EU funded projects, like «Support to the Agency for Accreditation of Health Care Institutions of Serbia», «Health Management Training» and «Support to Introduction of the National Program for early Detection of Cancer in Serbia». Paragraph 66 – It is mentioned in this Paragraph that additional support to IT investments comes from the EU, that in coordination with MOH, supports certain number of hospitals through IT investments. That number of hospitals is 19, and not 40 as it is mentioned in the text. Paragraph 74 – It is necessary to correct numeration of the Paragraphs, after Paragraph 74 Chapter 6. Lessons Learned – In this chapter there should be added another Paragraph with comment that European Commission is withdrawing from health projects, and that it makes continuous and consistent support of the World Bank in health sector in Serbia ever more important. Paragraph 73, 72 and 75 – Procurement process that caused the delays mentioned in Paragraph 75 did not concern the DRG Software, but it concerned a licence (right) to use appropriate diagnostic related groups classification system (DRG Classification System). That procurement process took additional 7 months due to solving of the certain issues that were less about procurement process, but more about format and provisions of the contract. This same omission (mentioning of the DRG Software instead of Licence for DRG Classification System) is found in Paragraphs 73 and 72. 51 IBRD 34847R 19°E To 20°E 21°E Kiskoros HUNGARY To Szeged SERBIA SELECTED CITIES AND TOWNS Subotica ˇ Kanjiza To 46°N Arad Senta OKRUG (DISTRICT) CAPITALS Bajmok POKRAJINE (PROVINCE) CAPITALS Backa ˇ Kikinda To Topola Timisoara Ada NATIONAL CAPITAL Sombor Apatin Ve RIVERS Sivac liki Kula Becej ˇ RO MA NIA MAIN ROADS VOJVODINA RAILROADS a Elemir Tis Temerin To OKRUG (DISTRICT) BOUNDARIES CROATIA ˇ Backa Zrenjanin Timisoara POKRAJNE (PROVINCE) BOUNDARIES To Palanka Vinkovci INTERNATIONAL BOUNDARIES Novi Sad To ˇ ˇ Vrsac Vinkovci Sid Fruska Gora Alibunar This map was produced by the Map Design Unit of The World Bank. Ruma Indija The boundaries, colors, denominations and any other information 45°N shown on this map do not imply, on the part of The World Bank Group, any judgment on the legal status of any territory, or any Pancevo ˇ endorsement or acceptance of such boundaries. Bela Crkva Sremska To ˇ Mitrovica Carbunari Sabac Kovin To To Bijeljina Sa va ˇ Zeleznik BELGRADE Gura Vail Zminjak Danube Prnjavor ˇ Pozarevac Obrenovac Smederevo Kladovo Vla Golubac sic Ho De Loznica Pla nin mo Jov li a Mladenovac ljs an ke Majdanpek Petrovac Pla Velika Dr ni ina Velika- na Negotin M Plana Ljubovija PValjevo Morava To ov al SERBIA ˇ Zagubica Sarajevo l Bel To je jani Bor en BOSNIA ca Vidin n Gornji 44°N Milanovac Kragujevac 44°N Rogacica AND Cacak ˇ Ravni Gaj ´ Cuprija To Vidin ˇ Uzice Jagodina HERZEGOVINA ˇ Pozega (Svetozarevo) ´ Paracin Rtan ˇ Zajecar hor Zag To Zl J u Gorazde ati Za Soko bo padn Banja lav r Kraljevo a Mo rava ˇ Knjazevac ak Ivanjica Aleksinac Ce Krusevac ˇ ˇ Usce me Midzor Ve l i k i Ba (2168 m) Ja gin trebac lk rn Zel Jas an vo Prijepolje Nis ˇ o r Bela Sjenica ˇ Raska Prokuplje PlaSu v Palanka M Golija Beloljin n i na ts a Pirot . Ju zn Novi Pazar Rad na a To an Mo oz PriÅ¡tina 43°N ra 43°N g Prepolak Leskovac Ro va To Vlasotince To Podgorica Sofia To To Sofia MONTENEGRO PriÅ¡tina Priboj Surdulica BUL GAR I A KO SO VO Vranje Bosilegrad To Bujanovac Pernik Presevo ˇ Lake SERBIA Scutari To Kumanovo 0 25 50 75 Kilometers FYR ALB AN I A MA CEDO NIA 0 25 50 Miles 19°E 20°E 21°E 22°E JULY 2009 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders N/A Annex 9. List of Supporting Documents Supporting documents are footnoted in the main part of the ICR. 52