ICRR 13027 Report Number : ICRR13027 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/24/2009 PROJ ID : P072654 Appraisal Actual Project Name : Second Small US$M ): Project Costs (US$M): 7.50 7.42 Enterprises Project Country : Timor-Leste Loan /Credit (US$M): Loan/ US$M ): 7.50 7.42 Sector Board : FPD Cofinancing (US$M ): US$M): 0.52 0 Sector (s): Micro- and SME finance (67%) General industry and trade sector (15%) Other domestic and international trade (13%) Central government administration (5%) Theme (s): Other financial and private sector development (29% - P) Small and medium enterprise support (29% - P) Infrastructure services for private sector development (28% - P) Regulation and competition policy (14% - S) L/C Number : Board Approval Date : 09/28/2001 Partners involved : Closing Date : 07/01/2004 12/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Marcelo J. Bueno Jorge Garcia-Garcia James Sackey IEGCR 2. Project Objectives and Components: a. Objectives: The main Project Development Objectives (PDO) of the Second Small Enterprise Project (SEP II) were to: (i) generate employment; (ii) accelerate economic growth; and (iii) improve small and medium enterprise (SME) competitiveness, by providing an SME line of credit, capacity building focused on private sector development (PSD), and rehabilitation of marketplace infrastructure . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components (or Key Conditions in the case of DPLs, as appropriate): SEP II's original components included the following: Component 1 - SME Line of Credit (SLOC) (planned, US$4.0 million, actual, 0-Cancelled). The SLOC was for small and medium enterprise (SME) loans on commercial terms in the range of US$ 1,000-US$50,000 of which 50% was earmarked for agribusiness. In late 2003, almost 1.5 years after the Project's approval, the SLOC was cancelled with no funds having been disbursed, owing to the cross -condition of high levels of non -performing loans under the First SEP (SEP I). The PDO in the legal document was not formally changed despite the cancellation of the SLOC; nonet heless, the objectives in SEP II were expanded to encompass a wider set of activities in Component 2, Private Sector Development (PSD) and Capacity Building, Component 3, Marketplace Rehabilitation, and Component 4, Project Implementation Unit (PIU). In addition, two other activities were added to include the establishment of an investment and export promotion agency, and the development of an economic opportunity zone (see Additional Activities below). The US$4.0 million from the cancelled SLOC was re-allocated to fund the expanded and new activities and key output/outcome indicators were revised for these activities . Component 2 - Private Sector Development Capacity Building (planned, US$2.24 million, actual, US$3.315 million). It had three sub-components: (i) Business Development Services (BDS) which were to be provided from four business development centers (BDC) targeted to both existing and potential SEP I and SEP II borrowers, generally in the districts around Dili; (ii) Civil Servant Training (CST) in PSD theory and practice for 15-30 civil servants; and (iii) development of an enabling Business /Regulatory Environment (BRE). From the cancelled SLOC, funding for Component 2 was increased by US$1.08 million to cover the additional activities subsequently built into the Program including: (i) a newly added sub-component on Government-Business Dialogue Promotion (GBDP) to foster better and more regular consultation between the government and business sectors; (ii) one additional Business Development Center in Maubisse; (iii) improvements in the legal and regulatory environment for businesses through consulting services to the Department of Economic Affairs; (iv) the provision of services to private microfinance providers and borrowers under other credit projects (e.g. credit unions under the Asian Development Bank project ); and (v) training to business associations and market management groups . Component 3 - Marketplace Rehabilitation (planned, US$1.0 million, actual US$1.69 million). US$1.0 million was for grants to the 13 districts of Timor-Leste to: (i) rehabilitate marketplaces based on proposals from community -based market management groups; and (ii) training for capacity building for market cooperatives under the BDS sub-component in Component 2. The additional US$0.69 million was for the construction of new markets and market infrastructure based on the demand proposals from the 13 districts. Component 4 - Project Implementation Unit Technical Assistance (TA) (planned US$0.26 million, actual US$1.2 million). The TA was for consultant services, training, audit services, and incremental operating costs . The additional funds of US$0.94 million was for expanded activities in each of these areas including travel costs, salary increases, and miscellaneous items for the PIU office . Additional Activities - With the restructuring of the Project in mid -2004, two activities were added to the Program, the Investment and Export Promotion Agency (IEPA) and the preparation of an Economic Opportunity Zone (EOZ) (planned US$ 0, actual US$0.98 million). US$0.80 million was for the creation of the IEPA, later re -named TradeInvest Timor-Leste (TITL), while US$0.18 million was for TA on the feasibility work of creating an economic opportunity zone. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Planned co-financing of US$0.08 million and US$0.44 million at appraisal from the Borrower and from unidentified foreign commercial resources, respectively, did not materialize at Project's end . With the re-allocation of the SLOC for new activities after the mid-2004 SEP II restructuring, the Project's main focus shifted from private sector investment to institutional development, mainly to capacity building and systems development, while continuing public investments in marketplace development . The Project's closing date was extended twice to 12/31/2007 for a total of 42 months, in the second instance, due to delays in the GBDP sub -component (new) in Component 2. 3. Relevance of Objectives & Design: Relevance of Objective The PDOs of generating employment, accelerating economic growth, and improving SME competitiveness were highly relevant to Timor-Leste's economic development priorities and consistent with the country's National Development Plan (NDP) and the Bank's 2006 country assistance strategy (CAS). To achieve these, certain key issues needed to be addressed for which the PDOs were designed . They included: (i) improving Timor-Leste's economic performance; (ii) controlling the high underemployment and unemployment levels; (iii) accessing the many unavailable essential services which had a limiting effect in restoring private sector activities; (iv) preparing Timor-Leste for advancing toward an industrial /service economy; (v) improving key enabling frameworks to enhance private sector development (e.g. rule of law, preservation of peace and security, sound government administration, etc.); (vi) addressing issues related to the business environment which needed government's attention; (vii) managing the recurrent budget /fiscal issues; and (viii) preparing for future negative demand shocks resulting from the withdrawal of large international presence in Dili where expatriate spending is mostly concentrated . The expanded set of activities in capacity building and marketplace rehabilitation under Components 2 and 3, respectively, were also considered highly relevant development objectives consistent with the NDP and CAS . Relevance of Design The relevance of the main design feature which was the SLOC for SMEs (Component 1) is moot. In the absence of a detailed assessment on the availability of funds for SMEs from other financial sources, the SLOC's relevance could not be assessed as the outcome that would have been attributable to it did not materialize . Additionally, the relevance of the Project's design was highly questionable owing to major design flaws inter alia : (i) not adequately taking into account lessons learned from SEP I (which led in part to the cancellation of Component 1); (ii) insufficient knowledge of the local financial market and the projected demand and supply of credit; (iii) omission of a needs assessment to establish rehabilitation priorities and cost; (iv) weakness in the PIU design in not strengthening the PIU's limited implementation capacity; and (v) issues attendant to sub-loan disbursement delays. 4. Achievement of Objectives (Efficacy): Overall, the achievement of the PDOs is rated Modest . The objectives attributable to the SLOC (which accounted for 53.3% of the IDA Grant) did not materialize while the outcomes in civil service training and PIU technical assistance (which jointly accounted for 18% of the IDA Grant) had unsatisfactory outcomes . The lack of a logical explanation of the linkage between the Project components and the development objectives, and between the achievement indicators and (non-existent) baseline values casts strong doubts on the efficacy of the Project and of the individual PDO achievements. Thus there was no strong basis for IEG to compare what the Project achieved with the resulting outcome, as no plausible evidence was advanced in the ICR to substantiate or verify the PDO achievements. Despite these limitations, the ICR made note of several results which help prepare the groundwork for private sector development in a post -conflict country with low implementation capacity and non -existent private sector. Employment Generation . Achievement in employment generation was modest . Under the employment sub-projects, new jobs were generated which helped stimulate the economy and improved social structures . The results against employment targets were reasonable given the cancellation of the SLOC which was earmarked for SME expansion and job generation through financial intermediation and new investments . Due to weaknesses in the M&E structure, however, (see Section 10), the conversion into outcomes of some of the employment indicators was difficult . Employment targets, nevertheless, were realized through : (i) local employment created by the BDC -supported businesses where the employment target of 400 was marginally surpassed at 428; (ii) local employment generated from new foreign direct investments /FDI estimated by the TradeInvest Timor -Leste to be around 15,300 jobs over three years against a target of 400 jobs during the same period; and (iii) marketplace construction where 1,090 short-term jobs were created which was equivalent to about 3 times the target of 120 full-time jobs. Acceleration of Economic Growth . Contribution to accelerating economic growth was modest . Achievement in this area was set more at the sectoral level than at the project level, and this created attribution problems with respect to achievements and impacts. Notwithstanding its high-level and broad objective, acceleration of economic growth, as an objective, was deemed appropriate for an SLOC where achievements could have been measured by the actual economic and financial performance of enterprises which the SLOC would have supported. With the cancellation of the SLOC, measuring contributions to economic growth was considered impractical in the absence of a direct impact on SME expansion. Although not verifiably linked to achievement indicators, the types of investments subsequently made in tourism, infrastructure, fishery, and agribusiness from the expansion of activities in the Program, pointed to contribution to economic growth in terms of employment generation, foreign direct investments, private sector development, and marketplace investments and rehabilitation . Improved SME Competitiveness . Achievement in SME competitiveness was modest . This was realized through four activities under Component 2: (i) capacity building in the business community through SME training and training of trainers. Output targets were mostly met (i.e almost 4,000 people trained, about 4 times the target, and 55 trainers trained versus the target of 33), including achievements in the social objectives of training women (45% vs the target of 40%) and the youth (48%); (ii) capacity strengthening in the Government through SME training of 65 civil servants, which while exceeded the target of 15-30, was likely to have had only a marginal impact due to the low quality of the training; (iii) increased foreign direct investments estimated by the TradeInvest Timor -Leste to be US$128.2 million for inward investments at end-November, 2006 and US$202.2 million by mid-2007, as compared to the modest US$2 million target estimate; and (iv) improvement in the business environment through business regulatory initiatives, business-government dialogue, and improved market infrastructure . Additionally, marketplace infrastructure was improved with 69 markets rehabilitated and constructed accounting for about 50% of the total number of markets rehabilitated and built in Timor-Leste. With the establishment of market management committees for these markets, the achievements in terms of market usage was generally positive with female participation rate of 45%, close to the 50% target. 5. Efficiency (not applicable to DPLs): There was no actual rate of return, economic or financial, as a measure of efficiency, that can be attributed to the Project at appraisal. At completion, data was insufficient to carry out such an analysis as well . This is attributable to the weak and unsatisfactory performance of the Bank and the M&E design, monitoring, and implementation of the Project (see Sections 8 and 9 below). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Overall, the Outcome of SEP II is rated as Moderately Unsatisfactory . This rating is based primarily on: (i) unrealized outcomes attributed to the cancellation of the SLOC . The support for SLOC accounted for 53.3% of the IDA Grant, and the expected outcomes assoicated with economic growth and competition were not necessarily realizable through the reallocation of funds; and (ii) the modest and unsatisfactory outcome ratings of the Civil Servant Training sub-component (Component 2) and PIU Technical Assistance (Component 4), respectively, which jointly accounted for 18.0% of the IDA Grant but were largely associated with the objective of improving SME competitiveness through the services the sector provided . While project outcomes were clearly unsatisfactory before component restructuring, moderately satisfactory outcomes were achieved thereafter through : (i) modest contribution to employment; (ii) increased private sector competitiveness through improved private sector capacity development, contributions to business -regulatory reform, better business -government dialogue, and facilitation of foreign direct investments; and (iii) improved trade and marketplace infrastructure . Other Outcomes and Impacts . SEP II exceeded, albeit marginally, the employment target despite the cancellation of the SLOC and thus contributed in part to alleviating poverty . Over the long run, the institutional strengthening and capacity building achievements would possibly lead to increased job creation . While the Project was also designed to enhance the role of women in private sector development, participation in decision -making by women was lower than expected. The acceptance of resettlement, when development so required, was an unexpected positive feature of the Project, especially in a country where land ownership is controversial owing to weaknesses in land titling and registry. On the basis of the above, the overall project outcome is rated moderately unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome of SEP II is rated Significant given the low capacity of the government and the limited financial resources of the Ministry of Economy and Development to fund ongoing development programs and capital development, and to strengthen existing institutions and staff capacity with business sector and direct fiscal support from Government. Beyond SEP II, direct fiscal support from Government budgetary allocations are doubtful which could result in the loss of momentum and impacting on the sustainability of the Project's achievements . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Overall, the Bank's performance is rated Moderately Unsatisfactory for reasons of design flaws (see Section 3, Relevance of Design) but with some effort made at rectifying the weaknesses during supervision . At Quality of Entry , IEG agrees with the ICR rating of Unsatisfactory given the failure of the Bank Team to : (i) identify implementation capacity issues and to provide the PIU with the necessary tools and information to overcome implementation weaknesses; (ii) strengthen the ineffective management and reporting structures both within the Bank team and in-country counterparts which were evident as early as SEP I; (iii) carry-out and provide for a needs assessment; (iv) design the SLOC with a thorough study of the financial market in Timor Leste and the lessons learned from the severity of SEP I's credit problems to which the activation of the SLOC was linked, subsequently resulting in its cancellation; (v) provide for baseline surveys and strengthen a poorly -designed M&E system; and (vi) provide the needed Bank supervision resources to support weak local capacity . Quality of Supervision . IEG rates the Quality of Supervision as Moderately Satisfactory . The Bank made continuous efforts to bridge differences with Government counterparts, reinforce Government's commitment to project activities, intervene in helping to strengthen PIU capacity, organize better private /public dialogues for actionable proposals, and regularly interact with other donors on PSD . There were however some shortcomings : management problems (high staff turnover, a constrained Bank budget, etc .), weaknesses in upgrading performance indicators, inflated progress ratings, slow recognition of credit needs of BDC trained entrepreneurs, and delayed awareness of environmental safeguard issues . But overall, the project was rescued by effort at improved supervision under trying environment . at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: The overall performance of the Borrower is rated Moderately Unsatisfactory . Government Performance . Government's performance was beset with: (i) constant changes in emphasis across components from four governments notwithstanding the Borrower's commitment to the Project activities to ensure progress, and (ii) differing political considerations and pressures in project priorities and staffing decisions . Implementing Agency Performance . The PIU's weak capacity to implement and supervise project activities could not be completely overcome by TA or by supervision assistance and inputs . Severe financial management and procurement issues during the earlier stages of the Project could not be effectively resolved by the Borrower (and the Bank) causing major implementation delays . The PIU lacked the range of skills to match the demands of project activities, and was unable to overcome resource shortages for some sub -components, including the lack of day-to-day management direction and oversight which hampered training at the needed level . In addition, PIU staff were not trained on M&E and despite Bank urging, and the PIU did not conduct baseline surveys before rehabilitating or building large marketplaces . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Overall, M&E was weak and made little contribution to improving project results . The on-going social unrest in Timor-Leste made it more difficult to collect the appropriate data and information further exacerbating the already weak M&E structure. On Design . The M&E: : (i) miscalculated SEP I's credit operation to meet the agreed target (80% performing loans) for activating a second line of credit under SEP II (Component 1) which later led to the latter's cancellation, and (ii) wrongly concluded the availability of financial resources for SMEs from alternative sources which affected the implementation timeframe by the lengthy efforts on the part Government and the Banco Nacional Ultramarino to divert the needed financial support early in the Project to the lagging SEP II componen ts. On Implementation . Weaknesses in the management and operations of the PIU hampered its capacity to monitor all the dispersed and widespread activities of the Project . In addition, the lack of discussion and training on the use of achievement indicators exacerbated the above weaknesses including the already weak M&E skills and training amongst the consultant and staff of the PIU . The absence of baseline surveys prior to the rehabilitation and the building of large marketplaces constrained the tracking of progress and the assessment of results, as well, making it difficult to assess the impact of works on market efficiency and to estimate the benefits accruing from business expansion and job creation . On Utilization, IDA staff assisted in strengthening the PIU's M&E capacity and supported the staff in M&E design and utilization, delivering some results in the process . Such improvements included better M&E practices for the marketplace rehabilitation component (Component 3), and some strengthening measures to the PIU's capacity to conduct the Rapid Market Usage Assessment which was designed by the Bank's Task Team . These assessments were conducted jointly by the PIU and the Task Team during 3 supervision missions. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards and fiduciary compliance requirements under the Project were strong. With regard to resettlement, marketplace rehabilitation could not be undertaken with involuntary resettlement issues or disputes with land ownership. While no provisions were made for voluntary and temporary resettlement during the rehabilitation work of kiosks and vendors, nevertheless, provisions were written by the PIU for such eventualities into the market and construction rehabilitation manual . On issues associated with the environment, projects classified as category B, i .e. projects that could have a potentially adverse effect on the environment, a negative list was established for the SLOC for which an independent review was required prior to funding . The environmental issue, however, later became irrelevant as the SLOC was eventually cancelled . At Project restructuring, environmental management plans and standards were required for the new markets . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Project outcome was clearly Satisfactory Unsatisfactory unsatisfactory prior to restructuring but became moderately satisfactory thereafter, resulting in an overall moderately unsatisfactory rating . Risk to Development Moderate Significant The low capacity of Government and Outcome : the limited financial resources of the Ministry of Economy and Development to fund on-going development programs and capital development casts strong doubt on direct fiscal support from Government and budgetary allocations beyond SEP II . Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Lessons learned include the following : In Project preparation : (i) where a line of credit is the major source of funding, there needs to be built -in sufficient knowledge of the local financial intermediation market, and the projected demand and supply of credit; (ii) there needs to be sufficient preparation of the business environment for financial intermediation; and (iii) a needs assessment is required to establish priorities, sequencing, and costs . In Design : (i) Project design must include adequate inputs from associated and similar operations to overcome weak implementation and supervision capacity; (ii) design and M&E systems must be flexible to adjust to implementation needs; and (iii) well functioning markets are essential for peace and recovery in fragile post conflict states. In Project Management : (i) where there are low or weak capacity constraints, Bank in -country staff needs to be assigned in managing the project while being recognized in supervision budgets; and (ii) an independent monitoring system should be in place where Bank supervision and oversight are impractical . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the ICR is Unsatisfactor y. While the ICR was candid in its assessment of the different aspects and the performance of the operation in terms of achievements of targets, outcomes and shortcomings, and presents considerable information on all important actions taken associated with the expansion of the Project's components . The ICR ignored, nevertheless, the evidence that 73.3% of the Project produced unsatisfactory achievements and outcomes associated with: (i) the cancellation of the SLOC (Component 1) which accounted for 53.3% of the Grant and the achievements and outcomes associated with it did not materialize; and (ii) the Civil Servant Training sub-component and PIU Technical Assistance component which jointly accounted for 18.0% of the IDA Grant produced modest and unsatisfactory rating outcomes, respectively . In addition, the ICR failed to link the Project components with the development objectives, and the key issues that needed to be addressed (see Section 3) to Project outcomes, thus creating attribution issues on the Project, overall . The ICR could have also been improved with more elaboration on the M&E structure, Efficiency, and a more in -depth discussion of PDO outcomes and impacts on poverty issues and social development resulting from the expansion of activities in the Program . a.Quality of ICR Rating : Unsatisfactory