ICRR 13908 Report Number : ICRR13908 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/18/2012 Country : Croatia Project ID : P080258 Appraisal Actual Project Name : Science & US$M ): Project Costs (US$M): 47.4 46.4 Technology Project L/C Number : L7320 Loan/ US$M ): Loan /Credit (US$M): 40.0 41.7 Sector Board : US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 07/07/2005 Closing Date : 11/30/2009 05/31/2011 Sector (s): General industry and trade sector (85%); Central government administration (15%) Theme (s): Education for the knowledge economy (33% - P); Infrastructure services for private sector development (33% - P); Technology diffusion (17% - S); Micro; Small and Medium Enterprise support (17% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Mariluz Cortes-Gorman Kris Hallberg Ismail Arslan IEGPS2 2. Project Objectives and Components: a. Objectives: Original Objectives : As stated in the Loan Agreement (LA-page 16), the original project objectives were : (a) to strengthen and restructure research and development institutions (RDIs) and to promote commercialization of the research outputs; and (b) to upgrade technological capabilities of enterprises . The original project objectives in the Project Appraisal Document (PAD-page 4) are to: (i) strengthen and restructure selected research and development institutions to promote applied research, while maintaining their scientific excellence; and (ii) increase the ability of enterprises to develop, use, adapt and commercialize technology. For purposes of this review, the original objectives in the LA are selected because they include the objective of promoting commercialization of the RDIs' research output, which is maintained in the revised objectives . Revised Objectives : As stated in the July 2009 Project Paper (page 4), the revised project objectives are: (a) to enable RDIs to commercialize research outputs; and (b) to increase the ability of enterprises, particularly small and medium sized companies (SMEs), to invest in research and development (R&D) activities. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 07/09/2009 c. Components: Original Components : The Science and Technology Project (STP) consisted originally of a four components : (A) restructuring RDIs and supporting knowledge commercialization by public research institutions; (B) strengthening private R&D and innovation; (C) engaging Croatian scientific Diaspora in joint projects and other initiatives to transfer their knowledge and skills for the benefit of Croatian economy; and (D) capacity building of the National Innovation System and project implementation support. Revised Components Component A (revised ): Encouraging commercialization of publicly -funded research (in place of implementing RDI restructuring). Activities within this component included : (i) strengthening of Brodarsky institute (BI); commercialization of the Rudjer Innovation Institute (RI); and (iii) supporting commercialization efforts of the University of Zagreb (UoZ) and University of Rijeka (UoR). Cost in EUR million Total Bank loan Estimated at appraisal 11.5 10.5 Actual 9.8 8.2 Component B (revised ): R&D financing program for enterprises. This component was revised to increase the ability of SMEs to invest in R&D activities, by implementing publicly supported innovation -financing programs. Activities included: (i) R&D financing program for enterprises of the Business Innovation Centre of Croatia (BICRO); (ii) development of knowledge-based SMEs through the RAZUM program; (iii) encouragement of SMEs investment in R&D through a risk-sharing scheme: Sponsored Research and Development Program (SPREAD Ir-Cro); and (iv) upgrading or development of technology Centers, Science and Incubators and R&D centers through the TEHCRO program. Cost in EUR million Total Bank loan Estimated at appraisal 17.1 14.0 Actual 17.8 15.8 Component C (original ): Engaging Croatian scientific Diaspora in joint projects and other initiatives, to transfer their knowledge and skills for the benefit of the Croatian economy . The main activity is the provision of grants for joint scientific projects through the The Unity Through Knowledge Fund (UKF). Cost in EUR million Total Bank loan Estimated at appraisal 5.0 3.7 Actual 5.7 4.5 Component D (original ): Capacity building of the National Innovation System /project implementation support. The main activities are: (i) financing the related costs of operating the Project Management Unit (PMU) established in the Ministry of Science, Education and Sport (MSES), including staff, financial management system, audits, equipment, training, technical assistance and studies; and (ii) finance costs associated with the operations of the Croatian Innovation System Group organized by MSES and the Croatian Accreditation Agency. Cost in EUR million Total Bank loan Estimated at appraisal 1.8 1.64 Actual 2.0 1.4 Unallocated funds : Cost in EUR million Total Bank loan Estimated at appraisal 1.3 1.0 Actual 0.1 0.1 d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The estimated project cost at appraisal was EUR 36.7 million of which IBRD would contribute a loan of EUR 31.0 million (US$40.0 million equivalent) and the Borrower would contribute EUR 5.7 million. The financing plan was revised as part of the July 2009 project restructuring which included cancellation of EUR 1 million from the sub-component to finance severance payments as part of the restructuring of one RDI. At project closing, the Bank had disbursed EUR 30 million and the Borrower had contributed EUR 5.4 million to the project. Loan disbursement in dollar terms was higher at closing than at appraisal despite the cancellation due to the devaluation of the dollar with respect to the Euro. The Project was approved in July 2005 but became effective almost a year later, in May 2006, due to delays caused by the adoption of a new Science and Technology Policy and the stakeholders consultations, which took longer than expected. During its first two years, the project performance was hindered by the lack of readiness of some local counterpart agencies at the start of the project . A mid-term review in June 2007 led to a major project restructuring in July 2009. As part of project restructuring, the project closing date was extended from November 30, 2009 to May 31, 2011 to allow full implementation of the envisaged project activities. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of Original Objectives . The relevance of the original project objectives is rated as high. high . The objectives of: (a) to strengthen and restructure research and development institutions (RDIs) and to promote commercialization of the research outputs; and (b) to upgrade technological capabilities of enterprises, remain highly relevant to the goal of helping Croatia to establish and develop the institutions and programs inherent to modern R&D and innovation systems, which are important for Croatia’s successful accession to the European Union (EU), expected on July 1, 2013. These objectives were aligned with the FY05-09 Country Assistance Strategy (CAS) which aimed to support the Government’s growth and reform strategy for successful EU accession, and remain aligned with the current Country Partnership Strategy (CPS) for FY09-FY12; particularly with its second pillar of strengthened private sector led growth and accelerated EU convergence . These objectives have been further calibrated by the CPS Progress Report (of August 2011) which is oriented towards EU 2020 strategic objectives and accelerated absorption of EU funds . The original project objectives were revised in mid -2009 to reflect more accurately what could be accomplished with the time frame and resources available . The objective of restructuring RDIs could not be pursued under this operation given the prevailing institutional conditions in the country, namely, limited appetite for RDI restructuring . However, this remains an important objective for the development of modern R&D and innovation institutions. The objective of upgrading the technological capability of enterprises was also revised to focus on the goal of increasing the R&D capabilities of SMEs, which remains very much relevant to increase the competitiveness of Croatian enterprises . Relevance of Revised Objectives . The relevance of the July 2009 revised project objectives is also high . The objectives of enabling RDIs to commercialize research outputs and to increase the rated as high. ability of enterprises, particularly SMEs, to invest in R&D activities can be seen as steps towards achieving the original project objectives . These objectives remain aligned with EU ’s development priorities as articulated in the Europe 2020 Strategy and its “smart growth� pillar aimed at boosting EU’s performance in R&D and innovation. In this context, the Government has requested the Bank to prepare a follow-up project with the primary objective of ensuring a high absorption rate of EU funds, by strengthening the policy and institutional frameworks and further consolidating the institutional project management capacity. At the same time, to address the funding gap, the project would support firm-level R&D for the period ahead of EU accession and prepare a pipeline of project proposals for the use of EU funds. b. Relevance of Design: Relevance of Original Design . The relevance of the original project design is rated as modest . The STP was conceived as the backbone of Croatia’s S&T Policy, involving a full range of interventions across the entire spectrum of the innovation chain – including policy and institutional reforms, capacity building, and management of innovation-financing programs. Although the interventions envisaged in the STP were appropriate for the many needs of the S&T sector in Croatia, the original project design was too ambitious for one single operation and some of the expected results were somewhat unrealistic. The project had to be restructured after two years of implementation. Relevance of Revised Design . The relevance of the revised project design to achieve the July 2009 revised project objectives is rated as substantial . The revised components focused more on what could be achievable with the existing resources and the country's institutional setup . One activity that was eliminated, besides the restructuring of RDIs, was the development of a local venture fund, which could not be created due to the drop in venture capital financing worldwide as a result of the global financial crisis. However, even after restructuring, there were too many sub-components, which resulted in relatively low levels of financing for some of them . This shortcoming was partly compensated by the introduction of a level of competition in the allocation of project funds during implementation, which allowed the reassignment of funds from the less successful towards the most successful components. A programmatic approach to develop Croatia’s R&D sector would have been more effective, combining different Bank instruments (development policy loans; technical assistance and program for results loans) in order to sequentially address the different challenges in designing, managing and evaluating innovation policies . 4. Achievement of Objectives (Efficacy): Achievement of Original Objectives : RDIs) First Original Objective : to strengthen and restructure research and development institutions (RDIs ) and to promote commercialization of the research outputs . The achievement of this objective is rated as modest . The initial objective of restructuring RDIs was partially achieved for the Brodarsky Institute (BI), which reported a self sufficiency ratio of 96 percent by end-project against a 100 percent target. The sustainability of this achievement is still volatile and jeopardized by high costs for the labor force and for maintenance, Central to restructuring efforts was the governance structure that impeded its transformation into an efficient market-oriented organization that could survive without public funding. The second part of the objective, to promote commercialization of the research outputs, was substantially achieved as discussed below under achievement of revised objectives . Second Original Objective : to upgrade technological capabilities of enterprises . The achievement of this objective is rated as substantial . The R&D financing programs for enterprises supported by the STP achieved the objective of increasing the ability of SMEs to invest in R&D activities (as detailed below under achievement of revised objectives ), Although this was a more limited objective than the original objective of upgrading the technological capabilities of enterprises, this achievement was an important step towards the original objective . Achievement of Revised Objectives : First Revised Objective : to enable RDIs to commercialize research outputs . The achievement of this objective is rated as substantial . This rating is based on the fact that the causal chain between inputs, outputs, outcomes and impacts operated as expected enabling RDIs to commercialize research outputs, meeting or surpassing all the outcome indicators . The activities, outputs and outcomes that contributed to the achievement of this objective are discussed below : Strengthening of Brodarsky Institute (BI). The STP supported BI through assistance in corporate governance, business processes, human resource development, upgrading physical infrastructure and commercialization. The project financed equipment for research projects, consulting services and training, and on a selective basis, incremental operating costs and severance payments for staff downsizing in certain areas. BI was successful in broadening the collaboration with industry and increasing its share of revenues from private contracts . It concluded 41 research contracts with industry worth over EUR 9 million and doubled its share of revenues from private companies from 22 percent in 2006 to 45 percent by end 2010. Support commercialization of the Rudjer Boskovic Institute /Rudjer Innovations (RBI/RI). With project assistance, RI was successful in establishing spillover companies . Five companies were established with project contribution of about EUR 1.5 million. Two of the companies are already sustainable, a third is in incubation period and two made initial steps towards sustainability . RI has concluded 12 licensing agreements amounting to EUR 750,000, including with prestigious international institutions . In addition, five patents were granted and 37 patent applications were filed. Support commercialization efforts of the University of Zagreb (UoZ) and University of Rijeka (UoR). The project supported the research commercialization efforts of the two universities, following the establishment of Technology Transfer Offices (TTOs) in both of them. The UoZ TTO became an important resource of supporting researchers to prepare applications for the Proof of Concept program, managed by BICRO. As result of the services provided to researchers in preparing the project applications, 50 projects were submitted for approval and 20 were approved by BICRO, amounting to about EUR 400,000. The UoR TTO was successful in the cooperation with the City Council of Rijeka, which approved the Sustainable Energy Action Plan, opening further opportunities for collaboration between the university and SMEs in the field of renewable energy . UoR was also successful in attracting young researchers in cross faculty knowledge sharing and commercialization of the research results. The following revised outcome indicators concerning this objective were achieved : (i) Increased number and value of research contracts with the industry : BI's contracts with private companies, UKF's contracts co-financed by business sector and research contracts signed by TTOs and not directly financed by STP increased from 2 contracts worth EUR 0.5 in 2006 to 59 contracts worth EUR 10.9 million at the end of the project, surpassing the revised target of 55 contracts worth EUR 9.8 million. (ii) Increased commercialization of R&D results : (a) the number and value of spill-overs (including joint ventures) signed by RI and other TTOs increased from none in 2006 to 5, below the target of 8, but the value of the contracts was worth EUR 3.1 million, higher than the EUR 2.86 million target; (b) the number and value of licensing agreements signed with the business sector increased from none in 2006 to 12 contracts worth EUR 0.75 million in 2011. The number of licensing agreements exceeded the target of 12 agreements, but the value was below the target of EUR 1 million. Second Revised Objective : to increase the ability of enterprises, particularly SMEs to invest in R&D activities . The achievement of this revised objective is rated as substantial . This rating is based on the fact that the causal chain between inputs, outputs, outcomes and impacts operated as expected, increasing the capacity of enterprises to invest in R&D, meeting or surpassing all the outcome indicators. A survey of beneficiaries carried out by the Zagreb Institute of Economics (ZIE) found that the level of collaboration between enterprises and RDIs, enterprises' R&D spending and quality of scientific output would have been less likely in the absence of the STP supported programs . BICRO R&D financing program for enterprises. The objective of this component was to increase the abilities of SMEs to invest in R&D, development, use and adaptation of new technologies, promote growth of technology-rich companies and improve product quality . Financial support was provided through a mix of financial instruments, including matching grants and subsidized loans . STP supported the development of one previously-existing BICRO program, RAZUM, and of two additional programs: TEHCRO, and SPREAD. A third new program was initially envisaged aiming to promote technology commercialization by stimulating the development of venture capital in Croatia (VenCRO). This program was dropped as part of the July 2009 project restructuring. STP also supported BICRO’ s institutional development through direct co -financing of the different industry support instruments and by helping it upgrade its institutional capabilities . RAZUM Program for Development of Knowledge-based SMEs. The RAZUM program supports the development of knowledge-based SMEs with a potential to have a positive impact on economic development, with subsidized loans covering up to 70 percent of new product development costs, with maximum support of EUR 1.5 million per project and duration of up to three years . BICRO takes the award decisions based on candidates’ business plans and on the viability of projects . At project closing, financing contracts had been concluded with 22 firms selected out of 40 firms that submitted full applications. SPREAD – Ir-Cro. This is a matching-grant program to encourage SMEs to invest in R&D, facilitate ties with RDIs and universities and help the latter expand their industry network and outreach . The program funds up to 50 percent of the cost of joint projects through non-reimbursable matching grants with a maximum contribution of EUR 80,000. At project closing the program had selected 22 firms to implement R&D projects with a total value of EUR 4.72 million, of which the project financed EUR 2 million. TECHNO Program. This program support projects aimed at the establishment, upgrading or development of: (i) Technology Business Centers , supporting establishment and growth of technology-oriented, innovative and knowledge-based companies at the regional level; (ii) Science & Incubation Centers, supporting the incubation of innovative academic spin -offs and start-ups; and (iii) Research & Development Centers : supporting R&D within specific thematic topics . The program provides grant funding for a maximum of 65 percent of the project's operational costs (or 50 percent of total costs) selected by public tenders or on BICRO’s initiative. At project closing, 13 centers had signed contracts to participate in the program with BICRO ’s investment amounting to EUR 10 million. UKF Unity Through Knowledge Fund . This fund was established to engage Croatian scientific Diaspora in joint projects and other initiatives, through grants for joint scientific projects . This program implemented 91 projects and established collaboration arrangements with 544 researchers and 260 institutions, including 166 foreign researchers, of which 41 are scientists from Diaspora and 133 reputable foreign basic scientific research institutions . The following outcome indicators concerning this objective were achieved : (i) SMEs that have new or significantly improved products new to the market maintained (as share of total enterprises) as shown by a Community Innovation Survey. The share in 2011 was 10.8 percent, higher than the share of 7.22 percent in 2006 and the revised target of 7 percent. This accomplishment has to be viewed in the context of the negative effects of the global financial crisis on the Croatian economy; (ii) The volume of private R&D mobilized through programs supported by STP (RAZUM, SPREAD and UKF projects with industry) increased from EUR 6.0 million in 2006 to EUR 13.7 million at project closing, surpassing the revised target of EUR 9.2 million. 5. Efficiency: The efficiency of the project in achieving the original objectives is rated as substantial . Since there is no information in the ICR to compare the cost of the project with the cost of other projects that achieved similar outcomes, this assessment is based on the fact that although there were initial delays in project implementation, these were not significant . The efficiency of the project in achieving the revised projects ’ objectives is rated as substantial . This assessment is based on the fact that after the 2009 restructuring, there were no significant delays or cost overruns in project implementation . The ICR assesses the efficiency of the STP through estimates of (i) social opportunity costs and (ii) cost effectiveness. To estimate social opportunity cost of R&D investments, the ICR compares the rate of return on R&D with other possible public investments in Croatia and concludes that the estimated rate of return on R&D (73 percent) is at least twice the value for infrastructure and seven times higher than for education. This exercise addresses the efficiency of the "sector" (R&D investments), rather than the efficiency of the project . To assess the project’s cost effectiveness (i.e. whether it represents the least cost solution to attain identified results ), the ICR presents the results of the survey of beneficiaries carried out by ZIE, which concluded that the level of collaboration between enterprises and RDIs, enterprises' R&D spending and quality of scientific output would have been less likely in the absence of the STP supported programs . However, this exercise addresses attribution not cost effectiveness. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project restructuring that included the revision of the original objectives was done with 37 percent of the loan funds disbursed, The weighted average outcome rating of the project is satisfactory based on a moderately satisfactory outcome rating before the 2009 restructuring and a satisfactory outcome rating after restructuring. Assessed against the original development objectives the project outcome is rated as moderately satisfactory , based on the following considerations. The relevance of the original objectives is rated as high because they remain important to achieving the goal of Croatia's successful accession to the EU. However, the relevance of the original design is rated as modest because it was too ambitious for one single operation and some of the expected results were somewhat unrealistic . The efficacy in meeting the original objective of restructuring RDIs and to promote commercialization of research outputs is rated as modest. Only one RDI was partially restructured and this part of the objective was dropped when the project was restructured. The commercialization of research outputs remained as an objective after project restructuring and the project's efficacy in meeting this objective is rated as substantial. The efficacy of the project in meeting the objective of upgrading the technological capabilities of enterprises is rated as substantial because the R&D financing programs for enterprises supported by the STP increased the ability of SMEs to invest in R&D activities, which is an important step towards the original objective. Finally, the efficiency in meeting the original project objectives is rated as substantial. Assessed against the restructured objectives the project outcome is rated as satisfactory . In the 2009 restructuring the project objectives were narrowed down and problematic aspects of the project design were addressed. This restructuring was done at an early stage leaving ample time for achieving the revised objectives . The relevance of the revised objectives and design are respectively rated as high and substantial to the current country context. The efficacy of the project in achieving each revised objective is rated as substantial . The project enabled RDIs to commercialize more research outputs and research capacity, not only through the funding of collaboration programs, but also by promoting significant cultural and capacity changes in these institutions . The project's R&D financing program for enterprises helped to increase SMEs' investment in R&D, and the project's cooperation programs improved scientific and technological cooperation . A survey of beneficiaries found that these achievements would have been less likely without the project . The efficiency in meeting the revised project objectives is also rated as substantial . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project faces two main risks to the development outcomes . By the end of the project, funds for continuation of the newly created programs had not been secured and future resources availability is uncertain in the context of currently severe fiscal constraints . Also, the legal status of BICRO and UKF is currently being reviewed by the new administration . TTOs still have to fully integrate with their parent organizations and are currently struggling to finance operating costs . As a mitigating factor to the lack of funding, the Government plans to use EU funds (available as of mid- 2013) to expand and stabilize the funding gap. The Science and Innovation Investment Fund has been established to partially finance TTO activities . A follow up operation to STP (STP II), focused on preparing the R&D sector to efficiently absorb EU funds, has been requested from the Bank and is under preparation . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The Bank correctly identified the strategic importance of helping Croatia to establish and develop the institutions and programs inherent to modern R&D and innovation systems as sources of competitiveness and to fully benefit from economic integration with the EU . Project background analysis was sound in most aspects and was based on lessons from international best practices . The project was prepared with extensive consultations and stakeholders discussions and with significant funding support from Croatia's Second Technical Assistance Loan (TAL 2). However, quality at entry was compromised by shortcomings in project preparation and appraisal . During the first years of implementation, incomplete project readiness and weak design caused delays and affected timely achievement of the project objectives . Despite the high risks and uncertainties identified at project appraisal and negotiations, project objectives were not narrowed down, nor were the problematic aspects of the project design addressed . Project design issues which were not entirely resolved at preparation had to be addressed through a major project restructuring . Another factor negatively affecting quality at entry was the underestimation of the institutional and capacity building requirements of beneficiary agencies – BICRO, UKF and RI. Meeting these requirements also led to delays in project implementation . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank provided consistent and timely implementation support with regular missions and close communication with project agencies. The implementation Status Reports (ISRs) were realistic and informative and addressed promptly the project's problems . The appointment in 2007 of a new TTL and a local project officer in Zagreb gave new impetus to project implementation and reinforced the focus on the project’s development impact. A comprehensive mid-term review (MTR) mission in 2008 led to a major project restructuring which constituted the cornerstone for the satisfactory turnaround of project implementation . Reallocation of funds approved in 2010 and 2011 directed project funds towards better performing activities . The Joint Portfolio Reviews of the Bank and the Government were also instrumental in proactively signaling the project issues, particularly during the first years of project implementation . The Bank team provided technical assistance to national policy issues in S&T, among others, through inputs to the S&T Policy and the draft Science Act. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Government/MSES initial commitment to S&T goals was reflected by the adoption of strategic research policy documents, primarily the “Science and Technology Policy 2006-2010,� the “Action Plan 2007-2010� and the “Action Plan for increasing the level of investment in research and development�. However, public consultations on these documents delayed project effectiveness by twelve months. Other delays in the initial phases were due to fluctuations in Government attention to project issues caused by reorganizations and bureaucratic red tape . Moreover, the risk of insufficient counterpart funding required careful management of the Government ’s contribution in order to avoid major delays . At later project stages, MSES took the necessary actions to solve project implementation issues, including : (i) adequate PMU staffing; (ii) adoption of streamlined procedures to delegate project implementation responsibilities to project agencies; (iii) timely revision/adoption of project documents; (iv) engagement in project coordination at state secretary level; (v) readiness to foster commercialization by allocating funds to the TTOs; and (vi) active participation in Joint Portfolio Reviews with the Ministry of Finance . MSES commitment to the revised project objectives led to timely execution of the revised project schedule and disbursement of 60 percent of loan funds within 18 months. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The performance of BICRO and BI was satisfactory. Both agencies were ready for implementation as they benefitted from TAL 2 financing for training, equipment, and expert assistance . Once MSES had addressed the necessary legal arrangements, the agencies engaged with full intensity and remained highly committed throughout the project. BICRO’s proactive approach to fostering private R&D activities was demonstrated by its development of the Proof of Concept program (within TechCro), which was a missing link in the innovation financing chain . After a set of interventions was agreed with the Bank, BI increased its innovative capacity as indicated by awards of UKF grants to BI research. There was a high level of coordination between the agencies and mutual benefit from each other’s programs. UKF's performance was highly satisfactory throughout the project. It’s good performance resulted in international recognition of its practices by the ILO. Although the RBI/RI subcomponent was initially delayed due to the need to adjust project financing and set up a full team at RI, activities picked up and RI had its first spill-over companies established in line with annual project targets . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Institutional arrangements for M&E were established during project preparation . The change in project objectives introduced with the 2009 project restructuring, was reflected in a revised results framework and changes in key indicators. Of six original outcome indicators, two were dropped because the original objectives could not be achieved, namely restructuring RDIs and promoting a venture capital industry; two were revised to better define the indicators; two became component indicators; and two new indicators were introduced to reflect the revised project objectives, namely the percentage of SMEs that introduced new and improved products to the market and the volume of private R&D mobilized through the programs supported by the project . b. M&E Implementation: Institutional arrangements for M&E were appropriately carried out during implementation . MSES/PMU regularly provided reports on project implementation, including updated information on outcome and output indicators. c. M&E Utilization: Regular monitoring based on the M&E framework and risk assessment allowed the Government and the Bank to take informed decisions during implementation, including on project restructuring and reallocation of funds to better performing activities . The Government also used the M&E framework for internal accountability purposes, such as reporting on progress made in implementing the S&T Policy as part of EU accession talks, and having better insight of R&D programs and policy making . Most of the performance indicators were formally introduced by the Government in the “Action Plan for increasing the level of investment in R&D �. M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project did not trigger safeguards review. b. Fiduciary Compliance: Financial Management (FM) arrangements were rated Satisfactory during the project’s life, with supervision missions routinely monitoring fiduciary compliance and with annual audits performed by external auditors. On one occasion, the rating was downgraded to Moderately Satisfactory due to delays in submitting 2006 audit reports and poor result of several implementing agencies ’ financial statements. The corrective measures recommended by the Bank were implemented in a timely fashion. c. Unintended Impacts (positive or negative): The project components were aligned with the Europe 2020 Innovation Union initiative and contributed to the dialogue related to Croatia’s EU integration. Through the UKF program, the project supported participation in the EU Framework Programs (FPs) and European Research Area (ERA) objectives. Structural and Cohesion Funds for regional convergence and competitiveness FP7 funds awarded to UKF projects represent 34 percent of all FP7 funds attracted by Croatian scientists in 2007-2010, demonstrating high competitiveness of UKF supported research groups . In addition, UKF and MSES were invited in December 2011 to negotiate a project within FP7 for fellowship, training and career development which would help integrate Croatian scientists in ERA . As part of the UKF “Across the Borders Program� the project resulted in publication of 54 academic papers in journals and 73 papers in conference proceedings. Beneficiaries also submitted 30 applications to the FP7 European research program and 19 applications to other international research programs . d. Other: The project had a substantial impact on institutional development across the science and innovation system in Croatia, as indicated by such achievements as : (i) UKF selected as “Best Practice� by the European Regional Economic Forum (EREF) on the topic �Developing human capital and managing migration for more competitive European regions;� (ii) UKF selected as “Good Practice� by the International Labor Organization (ILO) on the topic “Promoting linkages between migration and development;� (iii) BI and RI becoming members of EARTO and participating in FP8 Task Force meetings; (iv) BI strengthening cooperation with the innovation network PERA/iNET for preparation of projects to be financed from FP7; and (v) BICRO being awarded “Development of Innovative Business Parks to Foster Innovation and Entrepreneurship in the SEE Area� and becoming member of the European Enterprise Network and the European Business and Innovation Network. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The main lesson of the STP is that major changes across the entire spectrum of the innovation chain – including policy and institutional reforms, capacity building, and management of innovation-financing programs, cannot be fully accomplished with one project . A programmatic approach to develop Croatia’s R&D sector would have been more effective, combining different Bank instruments (development policy loans; technical assistance and program for results loans ) in order to sequentially address the different challenges in designing, managing and evaluating innovation policies. Another lesson of the STP is that at the time of preparing an ICR it is very difficult to assess the cost effectiveness of complex projects if there is no information on the cost of other projects that have achieved similar outcomes. During project preparation the task teams should obtain information on similar projects and establish some cost effectiveness benchmarks against which to compare the project results. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR follows the guidelines and is clearly written . It provides ample evidence of project implementation and achievement of objectives, Its discussion of Bank and Borrower performance is candid and perceptive, the lessons learned are insightful, and the ratings are consistent with the analysis. One shortcoming of the ICR is that the efficiency analysis is inadequate . However, despite this shortcoming, the ICR is so strong in other areas that it still merits a satisfactory rating . a.Quality of ICR Rating : Satisfactory