ICRR 13794 Report Number : ICRR13794 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/27/2013 Country : Samoa Project ID : P075739 Appraisal Actual Project Name : Samoa - US$M ): Project Costs (US$M): 4.48 4.54 Telecommunications And Postal Sector Reform Project L/C Number : C3724 Loan/ US$M ): Loan /Credit (US$M): 4.48 4.54 Sector Board : Global US$M): Cofinancing (US$M ): 0.00 0.00 Information/Communi cations Technology Cofinanciers : Board Approval Date : 12/17/2002 Closing Date : 04/15/2008 02/28/2011 Sector (s): Telecommunications (50%); Postal services (40%); Law and justice (10%) Theme (s): Regulation and competition policy (50% - P); Rural services and infrastructure (25% - S); Other financial and private sector development (25% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Melvin P. Vaz Robert Mark Lacey Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The objectives of the project as stated in the Project Appraisal Document (PAD, page 2) and Development Credit Agreement (page 15) are: to assist Samoa in improving the performance of its telecommunications and postal sectors by (a) increasing competition and private sector participation in the telecommunications and postal sectors; (b) strengthening its institutional and regulatory capacities; and (c) enhancing the provision of these services in selected areas . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1. Competition and Sector Reform (US$0.73 million at appraisal, US$0.99 million at closure) The activities under this component included : Selection and award of a digital mobile license through an open, transparent and competitive bidding process; Revisions to the telecommunications and postal legislation to allow for more competition including changes in existing regulations and preparation of new sector regulation, and enabling compliance with requirements on accession to the World Trade Organization (WTO) on telecommunications services; Developing sample model licenses for new operators and amendments to existing licenses of Telecommunications Samoa Cellular (TSC) and Samoa Tel; and Provision of technical assistance to elaborate on an implementable postal reform agenda . The component was to finance technical assistance to support changes to sector legislation in order to increase competition, in the accession to the World Trade Organization (WTO); in the design of a postal policy; engagement of advisors for the award of a digital cellular license; and professional development (training, study visits). 2. Regulatory Framework (US$1.52 million at appraisal, US$2.33 million at closure) The activities under this component included : Recommendations and drafting of new regulations in several areas of service interconnection, licensing, numbering, and others; Provision of new information systems within Ministry of Posts and Telecommunications (MOPT); Installation of spectrum management equipment; Recommendations on sector regulation, including tariff oversight, and preparing a national telecommunications frequency plan and its respective radio regulation; Training and offering of professional development activities to strengthen MOPT regulatory capacity; Monitoring and evaluating the effects of the telecommunications and postal sector reform progress in poverty alleviation; Preparation of a study on rural access to communications, with options to extend access to communications beyond the level offered by commercial operators; Design of a tender for Output Based Aid (OBA) for providers of rural communications, if an OBA piloting scheme were to be the preferred option to extend communications to rural areas; and Award of the OBA subsidy to providers of rural communications, through a least -cost, market-based mechanism. The component was to finance technical assistance to establish a regulatory framework to ensure effective competition; procurement of equipment and software to put in place information systems within MOPT; technical assistance in the area of spectrum planning and frequency plan; procurement of spectrum management equipment; technical assistance to carry out monitoring and evaluation of poverty impact; professional development; and piloting OBA for rural communications . 3. Corporate Reform of Samoa Tel (US$0.79 million at appraisal, US$1.41 million at closure) This component provided assistance to explore options to privatize Samoa Tel, and to carry out its privatization. The project would finance the technical assistance and the studies to be carried out, including advisors for privatization, as needed . 4. Postal Sector Reform (US$3.19 million at appraisal, US$1.13 million at closure) This component involved the following two Phases : (a) Phase I: technical assistance to the government to elaborate a postal sector strategy, including introduction of competition, definition and implementation of an adequate regulatory framework, extension of access, and review of the postal tariff regime; Technical assistance to Samoa Tel to prepare for competition in postal and telecommunications services through the definition and implementation of a corporate strategy; Design internal financial and management systems to provide Samoa Tel management with adequate management tools, including accounting and cost accounting; Separate financial audits for postal and telecommunications activities; and A comprehensive training program to provide staff with a new approach to business through exposure to private sector business management practices. (b) Phase II included the investment to upgrade SamoaTel's postal capacity, both on the production side (sorting center at the Main Post Office and mail boxes throughout the network) and the retail side (upgrading postal outlets and counter organizations). This investment includes network information systems (hardware and software). 5. Project Management (US$0.10 million at appraisal, US$0.14 at closure) This component supported the project steering committee (PSC) and the three implementing agencies with the implementation of the project. In July, 2007, the postal component (i.e., component 4 of the project) was reduced to $1.13 million (ICR, Annex1, page 21) and the key performance indicators of the project were reduced from seven to five . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost . The total estimated cost of the project at appraisal was US$ 6.33 million. The actual cost was US$6.0 million according to Annex 1 of the ICR and confirmed by the project team. Financing . The approved IDA Credit was SDR 3.4 million (equivalent at the time of appraisal to US$4.48 million. Of this, SDR 0.744 million was undisbursed at closure and canceled, mainly due to the non-accomplishment of planned activities in the postal sector. Due to exchange rate fluctuations, the remaining SDR 2.656 million, which was disbursed, was equivalent to US$4.54. There were no other sources of external financing. contribution: The Government contributed US$1.46 million, compared to an appraisal estimate Borrower contribution: of US$1.85 million. Dates. Dates . The original closing date (April 15, 2008) was extended 3 times for a total just short of three years: first, by 18 months (October, 2009) to allow for the privatization of Samoa Tel, the output-based aid Universal Access Program, and the purchase of Samoa Post fixtures; second, by 12 months (October, 2010) to amend the Development Credit Agreement to further reallocate funds to finance activities requested by the Ministry of Finance and Samoa Post; and third, by 4.5 months (February, 2011) to allow time to complete the technical assistance contract by the privatization advisors for the privatization of Samoa Tel. The project closed on February 28, 2011. 3. Relevance of Objectives & Design: a. Relevance of Objectives: : The major issues in the ICT sector that were conditions: The objectives were highly relevant to country conditions identified in the PAD included limited coverage and access to the telecommunications network, poor quality of service, high communication costs, lack of adequate postal services, lack of competition which hindered growth, and lack of effective sector regulation . strategy: The objectives were substantially relevant to Borrower strategy : Under the key strategic outcome "Private Sector Development and Employment Creation" defined in the Strategy for the Development of Samoa (SDS) for the period 2002–2004 (January, 2000) --- "... each household should be able to access a telephone. Additionally a national strategy for the development of ICT in Samoa will be formulated and implemented. Such a strategy will address amongst other things, how to take advantage of the potential benefits information and technology presents.�. Also, in the recent SDS for the period 2012–2016 (July, 2012), "Universal Access to Reliable and Affordable ICT Services� is listed as one of the key outcomes and a priority area under "Infrastructure Sector" (page 2, 16 and 17). However, development of the postal sector for Samoa was not mentioned in either version of the SDS. Samoa: On March 5, 2012, the The objectives were substantially relevant to World Bank strategy for Samoa: first individual Country Program Strategy (CPS) was prepared for Samoa covering the period Fiscal Year 2012-2016. Under the theme "Generate opportunities from global and regional integration" in the CPS (page i), it was mentioned that "... better transport and communications links can mitigate the cost of distance and encourage integration; and a key focus will be to build on the success of telecoms reforms and encourage new private investment to improve broadband connectivity". However, development of the postal sector for Samoa was not mentioned. The Pacific Island Strategy 2000 (Table 6, page 18) identified telecommunications as one of the priority sectors for Samoa. The Strategy also mentioned that "...measures to address the remoteness and isolation faced by the Pacific including in the transport infrastructure and telecommunications sectors and a focus on strengthening the the private sector's capacity to play a greater role in these areas" (page iii, para. xi). Again, however, there was no mention of the postal sector. The overall relevance of objectives is rated Substantial . b. Relevance of Design: The statement of the objectives was not fully clear as it did not specify the selected areas under the sub-objective (enhancing the provision of these services in selected areas) where the services (such as, basic communications, postal and internet) would be provided. The causal chain leading to the achievement of the objective "improving the performance of Samoa's telecommunications sector" is clear and convincing in the results framework (PAD, Annex 1, pages 30-34). For instance, there is a clear linkage between the activity financed (engagement of advisors for the award of a digital cellular license), the output (selection and award of a digital mobile license through an open, transparent and competitive bidding process), the intermediate outcome (increased competition and private sector participation in the telecommunications sector), and the intended achievement of the objective (improving the performance of Samoa's telecommunications sector ). The causal chain leading to the achievement of the objective "improving the performance of Samoa's postal sector" is also clear and convincing. For instance, the financing of the activities: corporate strategy (focusing on internal management and financial systems); legal (determining the cost effective way to reach out in rural and remote areas by defining franchising options and an adequate outsourcing policy); training; and procurement of postal equipment for upgrading the postal infrastructure, might reasonably be expected to lead to increased competition and private sector participation in the provision of postal services which in turn would lead to the achievement of the objective of improving the performance of Samoa's postal sector. Relevance of Design is rated Substantial . 4. Achievement of Objectives (Efficacy): 1: Improvement in the performance of Samoa's telecommunications sector by (a) increasing competition and private sector participation; (b) strengthening institutional and regulatory capacities; and (c) enhancing the provision of these services in selected areas : Substantial . Outputs : Technical assistance was provided for the revisions to the telecommunications and postal legislation in order to allow for more competition as well as facilitate the process of Samoa’s accession to the World Trade Organization, which was completed on May 10, 2012. No information is provided in the ICR on professional development. However, according to the project team, "41 Training courses were completed. Staff of the Ministry of Information Communication Technologies and the Office of the Regulator participated in courses to increase their capacity in the following technical areas: telecommunications operation (including broadband wireless, universal access), spectrum and frequency management and monitoring, broadcasting (public), e-government and Internet, telecommunications regulation, competition policy, telecommunications management, a legal framework for ICT, economics and accounting, and procurement. Attendance at key international conferences was financed". Technical assistance was provided to the Office of the Regulator . Technical assistance was also provided for the preparation of a statement of telecommunications policy, a proposed law on the postal sector and (updated in 2010); broadcasting legislation (2010); and a broader national information and communications policy statement, which was still pending at the time of completion of the ICR. The actual dates for these outputs were significantly later than the original target date of November 03, 2003 stated in the PAD (page 31). Information systems were supplied, installed and a maintenance program elaborated to assist the Ministry of Information Communication Technologies in its business operations . An Interconnection cost study was conducted to assist the Government in establishing fair and non-discriminatory cost-based interconnection rates for mobile and fixed line services. The Office of the Regulator received technical assistance in the area of interconnection dispute resolution (ICR, page 16). A new interconnection regime was in place by the end of October, 2009 (ICR Data Sheet, page vi). An integrated spectrum management system was procured to provide the Ministry of Information with the appropriate communication technology tools to facilitate the efficient use of the radio spectrum (ICR page 22). A spectrum assessment and national frequency allocation plan was put in place in October, 2004 significantly after the original target date of September 15, 2004 stated in the PAD (page 31). It involved technical assistance to the Ministry of Posts and Telecommunications which was later renamed the Ministry of Information Communication Technologies to: (i) prepare a national frequency allocation plan and relevant radio regulations; (ii) establish processes and procedures on spectrum management and monitoring; (iii) establish an appropriate pricing policy; and (iv) assess the system requirements for radio spectrum management and monitoring and prepare the tender documents for procurement. A planned output-based aid scheme to increase access to ICT services in rural areas was not implemented because the activities supporting the roll out of services to rural areas was only initiated in late 2010 when they were handed over from the Ministry of Information Communication Technologies to the Office of the Regulator, which left insufficient time to implement the services . According to the project team, ten training courses were conducted for staff of the Ministry of Finance and the Office of the Attorney General to increase their capacity in telecommunications regulation . A study was undertaken to advise and assist the Government to identify options for the reform of Samoa Tel (ICR, page 23). Advisors on privatization funded by the project provided technical assistance to the Government to valuate Samoa Tel and to prepare the privatization transaction. The privatization of Samoa Tel was completed on March 31, 2011, three years behind schedule. A tariff rebalancing plan (rates rebalancing for the main classes of services) was implemented in the spring of 2006. Outcomes : Competition in the telecommunications sector was increased through the issuance of a second mobile telephone license to Digicel Samoa Ltd. on March 8th, 2006 about 2 years after the target date of December 15th, 2003 (ICR, Annex 9, page 38). Prior to issuing this license, the Government first had to tackle the exclusive rights granted to Telecommunications Samoa Cellular (TSC) in its then existing 10-year license to provide analogue mobile services. Following an analysis of the Government’s options to deal with TSC’s exclusive rights, it was decided to renegotiate the terms of TSC’s license. Pursuant to a mutually acceptable Deed of Settlement, TSC agreed to relinquish its exclusive rights in exchange for being granted a GSM (digital mobile) license. The Government then issued a tender for the license, and awarded it through an internationally competitive process. This second digital cellular license effectively introduced competition in the mobile market . The Office of the Regulator, which was intended to be an independent regulatory body for telecommunications, was established in 2006. The Office is funded by revenues from the telecommunications sector, which the Ministry of Finance collects. The Ministry provides the Office with its budget. According to the ICR (page 16), “the institutional success of the [Office of the Regulator] (design and implementation) is…witnessed by the fact that the [Government] is considering appointing the [Office] to become the regulator for other infrastructure sectors in Samoa.� Increased competition and the operations of the Regulator have contributed to a fall in telecommunication tariffs. According to information provided by the project team, the cost per minute of a prepaid or postpaid call from Samoa to the US fell from 4.60 tala in 2004 to 1.79 tala in 2010, and a call to Australia from 3.07 tala in 2003 to 0,85 tala in 2010. The mobile Internet tariff charged by Digicel fell from 3 tala per megabyte in 2008 to 1 tala in 2010. The retail price of a 5 gigabyte per month Bears System was 1,000 tala in 2010, compared to 1,750 tala in 2008. Increased provision of service, together with greater affordability, has led to a considerable expansion in the size of the sector. In 2010, there were 168,000 customers of fixed and mobile telecommunication services combined (compared to 12,500 in 2002), nearly four times the target of 46,000 set in the PAD and well in excess of the revised target of 115,000 (the PAD target was for the original closing date of 2008; the actual number of customers in 2008 was 152,000). There has also been a large expansion in the provision of ICT services. In 2010 there were 12,000 Internet subscribers, four times as many as in 2002, and in excess of the appraisal target of 10,000 in 2008 (the actual number of subscribers in 2008 was 9,000). Access to telecommunications in selected areas has also increased. In particular, mobile penetration in rural areas increased dramatically, despite the fact that the output-based aid scheme did not materialize. There were nearly 58,000 mobile telephone customers in rural areas in 2010, compared with an estimated 1,200 in 2002, an appraisal target of 6,000 in 2008 and a revised target of 7,000. The improvement is the result of the rapid penetration of mobile services seen in rural areas in a wide range of countries. As the ICR (page 13) points out, some of the huge jump “reflects an underestimation of the expected targets; and weakness in gathering [particularly baselines] data.� Nevertheless, mobile penetration for the whole of Samoa is estimated at 95% (ICR, page 16). The ICR does not provide information of other, commonly used indicators of telecommunication sector performance, especially quality of service, but also operating efficiency and financial sustainability. DO2 DO 2: Improvement in the performance of Samoa's postal sector : Modest Outputs : The original project design called for a postal reform component of US$3.19 million, about 50% of total estimated project cost. This was to enhance the management systems and headquarters office arrangement to make them more commercial and efficient as well as improving the sorting center and retail outlets. In the event, only about one third of the allocation was spent, mostly on technical assistance. According to the project team, the Chief Executive Officer of Samoa Tel “decided that the procurement processes were delaying the rollout of other facilities enhancements that he deemed urgent, and opted to fund part of the plans directly through his own budget.� The savings in project resources were reallocated to the telecommunications activities, including splitting it from the postal business in preparation for privatization . Technical assistance was provided for the preparation of a Postal Bill and Postal Policy. A detailed market analysis and a Strengths Weaknesses Opportunities Threats (SWOT) analysis of Samoa Tel's postal services was conducted to structure strategic recommendations and an action plan (ICR, page 23). Technical assistance was provided on the legal dimension of the postal business regarding partnership/service level agreement contracting. The technical assistance was aimed at strengthening the ability of the postal service to negotiate access of postal retail access points by partners and to seek an optimal outsourcing policy with a view to focus on its core business (ICR, page 23). A study provided Samoa Tel with a detailed methodology for asset valuation, cost accounting, and pricing policy to facilitate splitting telecommunications from postal services . The ICR does not provide any information on staff training which, according to the PAD (page 32), was scheduled for completion by December 31st, 2004. However, according to the project team, "27 training courses were completed, together with study tours and attendance at regional workshops, to increase the staff’s managerial, logistical and retailing capacity .� Outcomes : The ICR does not provide information on the status of the investment to upgrade Samoa Tel’s postal capacity, neither on the production side (sorting center at the Main Post Office and mail boxes throughout the network) nor the retail side (upgrading postal outlets and counter organizations). This was meant to be partially financed by Samoa Tel’s own resources and partly by the project. According to the project team, upgrading of infrastructure was confined to setting up an internet café in the Head Office and putting in place a cost accounting system . According to the ICR (page 13), the Postal Bill and Postal Policy were endorsed by Parliament in 2010. However, there is little information on the contents of the Postal Bill or on postal sector policy. The project team stated that “best practice elements include[ed] sector liberalization, level playing field competition and incentives for Samoa Post to perform better .� The ICR (page 23) reports that the technical assistance on the legal dimension led to only partial implementation of reforms by the postal services. This assistance aimed to strengthen the ability of the postal service to negotiate access of the postal retail access points by partners, and seek an optimal outsourcing policy with a view to focus on its core business. Little information is provided on which activities were outsourced. According to the project team, technical assistance in the area of partnerships for Samoa Post contributed to [the] signing [of] a number of partnership agreements with the utilities for bill payments (Samoa Water Authority, Electric Power Corporation, SamoaTel) as well as with Western Union, National Bank of Samoa, Land Transport Authority, etc.� The total number of postal office mail boxes per 100 inhabitants was 2.8 in 2010, below the target of 3.0 and significantly below the original target of 10.59 for 2008 and even the 2004 baseline of 5.58. The total number of letters per inhabitant in 2010 was at the same level as the target of 3.0, but significantly below the original target of 12.2 for 2008 (PAD, page 2) and the baseline of 4.5 (2004). Since the output-based aid scheme to award subsidy to providers to increase access to communication services in the rural areas was not implemented, the aim of extending postal services in those areas was not achieved. 5. Efficiency: The cost benefit analysis at appraisal (PAD, pages 40 to 43) computed the Net Present Value (NPV) of: (i) the cost of the project; (ii) overall fiscal revenues from the postal, cellular and telecommunications sectors; and (iii) cumulative fiscal gains -- the difference between (i) and (ii). The cumulative fiscal gains were estimated to increase to about US$ 3.5 million by 2008. Although the assumptions were stated in Annex 4 of the PAD, it is unclear how the revenues were estimated during the period 2003-08 for the postal, cellular and telecommunications sectors . In the ICR’s ex post analysis (Annex 3), the economic benefits from the project were derived by estimating the increase in consumer welfare resulting from the introduction of the second mobile operator and from the privatization of Samoa Tel. With regard to the former, the analysis suggests that the welfare benefits were between US$20.3 million and US$56.56 million, depending on "pessimistic" or "optimistic" assumptions concerning price elasticity. With regard to the privatization, the value of operational net cash income during the period 2010-2020 was used as a proxy for the benefits from its privatization . No attempt was made in either the PAD or the ICR to carry out a cost-benefit analysis for the Postal Sector. There were several indications of inefficiency in the use of project resources. The project was extended three times for two years and ten months representing about 53 percent of the original implementation period of five years and four months. Long delays were caused by protracted negotiations between stakeholders, particularly once the decisions to separate the postal and telecommunications business units and to privatize Samoa Tel had been taken. Some planned activities, such as the output-based aid scheme for increased rural access to telecommunications and postal services, were not implemented since attention was focused on privatization of Samoa Tel . Efficiency is rated Modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of objectives and design is rated substantial . Efficacy of the first sub-objective -- improvement in the performance of the telecommunications sector -- is rated substantial given the evidence of increased access, lower prices, increased private sector participation, greater competition, and an effective regulatory regime, although information on service quality is not provided . Efficacy of the the second objective -- improvement in the performance of the postal sector -- is rated modest due to lack of evidence. Efficiency is rated modest reflecting the lack of efficiency analysis for the postal sector and the considerable operational inefficiencies . Overall outcome is assessed as moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated moderate. moderate The political and social risks of a reversal of the liberalization process for the telecommunications sector are low to negligible. The process has resulted in greatly increased access to mobile telephone services for the population and apparently also greater affordability. The privatized Samoa Tel is, according to the ICR, making progress in providing improved telecommunication services to its client (though no details are provided). Moreover, the Government is extending its liberalization policy to other sectors. The institutional risk is moderate. For a small country such as Samoa, the financial burden of a Regulator for the telecommunications and postal sectors alone is high and likely unsustainable (the Office of the Regulator’s budget is covered entirely by levies on sector output). Accordingly, the Government is considering how to extend its responsibilities into the electricity and water sectors. However, the ICR contains no information on the financial, technical and human resource implications of such a broadening of responsibilities . The project achieved few of its objectives in the postal sector. However, there is at least a moderate risk that those that were achieved would be undermined by the incomplete reform process . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project benefited from the Bank’s considerable international experience in supporting telecommunications reform in a wide range of situations, including the particular challenges of small countries (where a minimum critical mass of demand is necessary to provide sustainable services at a profit). Some key lessons from the Bank’s sector experience were taken into consideration in preparing the project. These included (i) a competitive framework and rapid transition to a multi -operator environment increase service coverage, quality, and affordability of service faster than a state monopoly operator, or a private monopoly; (ii) privatization of the incumbent operator is a key step for effective reform; (iii) capacity to regulate the sector is necessary to enable fair competition, to address emerging issues such as interconnection, pricing of noncompetitive services, spectrum (a scarce resource) management, and other disputes that may arise between service providers; (iv) speed is of the essence in winning public support for the reform process through rapid opening of the mobile market; and (v) despite global decline in the importance of postal services, the postal sector can still benefit from competition and a sound policy framework . However, international experience also demonstrates that privatization of telecommunications needs to be preceded by splitting telecommunication and postal services into two separate entities. This is also necessary to facilitate reforms in the postal sector . This split was not built-in at the design stage and had to be implemented later by the Government, causing delays in implementation. Although, the design of the components related to the telecommunication reform was based on the Bank’s considerable international experience, the postal reform was not fully conceptualized at appraisal. As a consequence, the amount of technical assistance required by the postal sector was over-estimated. Despite the reallocation of some of these funds to the telecommunications activities, SDR 0.744 million had to be canceled at project closure . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: The project team reported that the Bank carried out 13 supervision missions to Samoa for the project, an average of one mission every seven to eight months . The missions were supplemented by regular conference calls and exchanges of documents and information by email . This was cost effective considering the remoteness of the country. A total of 15 Implementation Status Reports were filed. A special emphasis on support to the procurement activities was organized, with multiple email exchanges and a procurement officer part of supervision missions on a regular basis . “The task team remained flexible and responsive to the [Government] and the implementing agencies to manage key tasks so that they could be completed within the project, and to add additional support to enhance the capacity of the Ministry of Information and Communications Technology (for instance, the development of a regime for alternative dispute resolution, with a focus on mediation; support to resolve interconnection issues, etc .)� (ICR, page 17). The supervision teams included a postal specialist as well as telecommunications expertise . While outputs or intermediate outcomes related to the postal sector were not monitored in the Implementation Status Reports (ISRs) until close to two years before extended project closure, this was attributed by the project team to the fact that, prior to 2009, the ISR format did not permit the inputting of all output and outcome indicators. Despite slow disbursement rates, implementation progress ratings never dropped below moderately satisfactory throughout implementation . According to the project team, the disbursement lags “were due mostly to project implementation issues rather than changes in orientation on the part of the Government or any unwillingness to execute the project as appraised. The Bank team was proactive [in mitigating] the procurement delays with substantial support provided by a seasoned procurement officer, and the low disbursement was addressed through the three extensions to give more time to the counterparts to implement their part of the project activities .� The project team also stated that “there is room for debate about whether the ISR ratings could have realistically dropped below [moderately satisfactory] at any time, but this is a rather minor matter in comparison with the efforts made, in the context of a capacity-limited client and a complex sector reform, to persevere in obtaining positive results from the project.� Nevertheless, more consistency in flagging procurement issues, especially in the postal sector, would have been appropriate . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The strength of government commitment to reform of the telecommunications sector was demonstrated by the creation of the Office of the Regulator, the splitting of telecommunication and postal services into two separate entities, the opening of the telecommunications market to competition through the granting of the second mobile license, and the privatization of the fixed line service utility, Samoa Tel. However, reforms in the postal sector were not pursued with the same level of energy . One of the contributing factors to implementation delays was slowness in payment of advisers and contractors. The ICR refers to a “Project Steering Committee� but there is no discussion of the constitution of this Committee, its membership, to whom it reported, or the frequency of its meetings . The project team subsequently informed IEG that the Steering Committee “did not play a visible role, as its members (Director of the Ministry of Information and Communications Technologies and representatives of Ministry of Finance, Samoa Tel and Attorney General ) were de facto directly involved in the project implementation, making the line of command between the Project Implementation Unit and the Steering Committee quasi overlapping .� Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: There were three implementing agencies: the Ministry of Finance, the Ministry of Information Communication Technologies and Samoa Tel (and later Samoa Post). “In order to build capacity, an external consulting firm experienced in supporting donor -funded projects was hired as the Project Management Unit to support the Project Steering Committee� (ICR, page 8). The Unit “provided quarterly project management reports containing updates on disbursement, procurement, and financial management� (ICR, page 18). No further information is provided on the Unit’s performance. Initial delays in implementation were caused by tardy submission of procurement and other project related reports mainly due to the Ministry of Information Communication Technologies ’ lack of experience in implementing Bank projects . The internet penetration indicator was not updated regularly and communicated by the Ministry of Information Communication Technologies or Office of the Regulator to the task team . The Ministry of Finance took an effective lead in implementing Samoa Tel's restructuring and privatization but lacked momentum on the rest of the sector reform agenda. This slowed project implementation. The Ministry’s limited capacity to implement the reforms was one of the factors resulting in three extensions to the closing date of the project (ICR, page 19). The Ministry of Information and Communication Technologies and the Office of the Regulator were slow to act on the agenda for rural access. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Baselines and targets were provided for each of the seven key performance indicators in the PAD (page 2). Neither the PAD nor the ICR provided information on the methodology used in the establishment of the baselines. The M&E framework focused on output or intermediate outcome indicators, and there were no indicators (beyond penetration and postal service quality) of improved sector performance for either telecommunications or postal services . There is no evidence in the PAD or in the ICR on the extent to which the M&E design had sufficient stakeholder ownership . According to the PAD, the Project Management Unit (PMU) was to be responsible for monitoring the implementation progress . b. M&E Implementation: Although the PMU had responsibility for M&E management, the ICR reports that the Bank task team itself collected the data for the key indicators from the following agencies during supervision missions : Ministry of Information Communication Technologies, Samoa Tel (and later Samoa Post), Office of the Regulator, Digicel and Computer Service Limited . In July, 2007, as part of the reduction of the postal sector reform component, two indicators (number of inhabitants per full postal service outlet and postal quality of service ) related to the postal sector were dropped. Until November 20, 2009, none of the seven key performance indicators mentioned in the PAD (page 2) were reported by the task teams in the Implementation Status Reports. Significant changes were made to the targets of all 5 intermediate indicators towards the end of the project. For instance, the indicator "Increase in the number of letter item/inhabitant" originally had a target of 12.2 for 2008, but the new target for 2010 was reduced significantly to 3.0; the reasons for this are not explained in the ICR. c. M&E Utilization: The M&E data were used in the reduction of the postal service activities . No further indication is provided in the ICR concerning utilization of the M&E system . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was given a Category “C� for Environmental Assessment purposes, and no safeguards policies were triggered. b. Fiduciary Compliance: Financial Management : The ICR does not discuss fiduciary compliance, and the small amount of information provided on financial management issues is scattered throughout the document. There is no information on: timeliness of the project's external audits and the external auditors' opinions. According to the task team, all external auditor's opinions were unqualified. Procurement : The Ministry of Information and Communication Technologies lacked implementation capacity resulting in delays in the submission of procurement related documents . There were no reported cases of misprocurement. c. Unintended Impacts (positive or negative): None. d. Other: None. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Negligible to Low Moderate The institutional risks – in particular Outcome : the financial sustainability of the Office of the Regulator – are moderate as are those stemming from the incomplete postal sector reform Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately There were moderate shortcomings Satisfactory in Government performance. There were significant shortcomings in the Implementing Agencies’ performance reflecting weak capacity to implement reforms and to manage procurement. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following are key lessons emerging from both ICR and ICR Review : Leadership in formulating and applying the appropriate policy framework in a timely manner is reform. The Ministry of Finance took an effective lead in implementing Samoa Tel's key to sector reform. restructuring and privatization but lacked momentum on the rest of the sector reform agenda. This slowed project implementation. The Ministry’s limited capacity to implement the reforms translated into delays in implementation progress which resulted in three extensions to the closing date of the project (ICR, page 19). Full consideration of the lessons from Bank experience can avoid or minimize implementation difficulties . In this case, a number of important lessons were taken into account. However, the need for splitting telecommunication and postal services into two separate entities prior to sector reform in general, and privatization in particular, was not built -in at the design stage. . A frequent problem in inefficiencies. Readiness for implementation must be assured to avoid later inefficiencies the Information and Communication Technologies sector is that projects are not ready for implementation due to lack of capacity in the implementing agencies. In this case, the Ministry of Information and Communication Technologies lacked procurement capacity and the Ministry of Finance lacked capacity on implementation of reforms . Proactive actions and changes to the project's components should be anticipated by the task team early on during implementation and not delayed until the mid mid- review. In this case there -term review. was an unnecessarily long ten month delay in reallocating resources following the Government’s decision to split Samoa Tel into two separate entities . 14. Assessment Recommended? Yes No Why? To revise the ratings and assess the performance of the telecommunications sector based on affordability, quality and efficiency of service . 15. Comments on Quality of ICR: The quality of the ICR is rated Unsatisfactory : The ICR does not evaluate the project development objectives as written, but rather focuses mostly on outputs and intermediate outcomes. Little information is provided on measures of sector performance such as quality of service, efficiency, sustainability and affordability . The ICR provides insufficient evidence on the extent to which the postal service upgrading investments were carried out, and on which activities, such as outsourcing of services, were completed. The ICR provides very little information on fiduciary compliance, and there is no discussion of external audits. An explanation of why the targets for the M&E indicators were revised and why the baseline values for some indicators are greater than their target values would have been helpful. There is no information on the utilization of the M&E system. There is no discussion of the performance of the Project Management Unit, nor any indication of the constitution or effectiveness of the Project Steering Committee . There is no attempt at an efficiency analysis for the postal sector . Little information is provided on the training provided to postal service personnel, and none on its effectiveness. There were also some issues of internal consistency: total IDA financing in Annex 1b is different from that reported on pages 4 and 5; the date of the second digital license award differs between the Data Sheet and the text of Annex 9; the indicator “number of letters or items per inhabitant" was included in Table 2, although it is reported on page 4 as having been dropped; and the environmental category of the project of the project is shown as "C" in the Data Sheet, but was incorrectly recorded as "B" in the text (page 9). a.Quality of ICR Rating : Unsatisfactory