Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004850 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF 017431-6R ON A SMALL GRANT IN THE AMOUNT OF USD1.5 MILLION TO THE University of West Indies Consortium FOR Mobile Innovation Project under EPIC (P132570) {Date} Finance, Competitiveness And Innovation Global Practice Latin America And Caribbean Region Note to Task Teams: 1. For date, enter the date the ICR is submitted to SECPO. Please delete this note when finalizing the document. Regional Vice President: Axel van Trotsenburg Country Director: Tahseen Sayed Khan Senior Global Practice Director: Alfonso Garcia Mora Practice Manager: Zafer Mustafaoglu Task Team Leader(s): Toni Kristian Eliasz ICR Main Contributor: Toni Kristian Eliasz ABBREVIATIONS AND ACRONYMS AM Aide Memoire CARIRI Caribbean Industrial Research Institute CARICOM Caribbean Community CMIP Caribbean Mobile Innovation Project DEE Digital Entrepreneurship Ecosystem EPE End of Program Evaluation EPIC Entrepreneurship Program for Innovation in the Caribbean ICT Information and Communications Technology ISR Implementation Status and Results Report IT Information Technology ITES IT Enabled Services mHub Mobile application hub that supports mobile technology start-ups in the Caribbean MTR Mid-Term Review M&E Monitoring and Evaluation PIU Project Implementation Unit RETF Recipient Executed Trust Fund SC Steering Committee USD United States Dollar UWI University of West Indies TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 13 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 20 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 21 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 23 ANNEX 2. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 30 ANNEX 3. SUPPORTING DOCUMENTS .................................................................................. 31 The World Bank Mobile Innovation Project under EPIC (P132570) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P132570 Mobile Innovation Project under EPIC Country Financing Instrument Caribbean Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency University of West Indies Consortium Lumin Consulting Project Development Objective (PDO) Original PDO The Caribbean Mobile Innovation Project (CMIP) aims (1) to strengthen the Caribbean mobile innovation ecosystem and (2) to enable sustainable and competitive mobile enterprises to grow. FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-17431 1,500,000 1,479,317 1,479,317 Total 1,500,000 1,479,317 1,479,317 Total Project Cost 1,500,000 1,479,317 1,479,317 Page 1 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) KEY DATES Approval Effectiveness Original Closing Actual Closing 10-Jul-2014 17-Jul-2014 30-Nov-2017 30-Nov-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 30-Nov-2017 1.42 Change in Results Framework Change in Loan Closing Date(s) Change in Procurement Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) Moderately 01 12-Jul-2017 Moderately Unsatisfactory 1.16 Unsatisfactory 02 05-Apr-2018 Moderately Satisfactory Moderately Satisfactory 1.42 ADM STAFF Role At Approval At ICR Regional Vice President: Hasan A. Tuluy Axel van Trotsenburg Country Director: Sophie Sirtaine Tahseen Sayed Khan Director: Robert R. Taliercio Practice Manager: Valerie D'Costa Zafer Mustafaoglu Task Team Leader(s): Toni Kristian Eliasz Toni Kristian Eliasz Page 2 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) ICR Contributing Author: Toni Kristian Eliasz Page 3 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. Context In 2012, many countries around the world were experiencing growth in mobile innovation and entrepreneurship. The World Bank, through infoDev1, had been involved in seeding and supporting regional digital entrepreneurship ecosystems (DEE) in East Africa, South Africa, Central and Eastern Europe, and South-East Asia. With its experience supporting DEEs globally, the World Bank sought to utilize its capacity-building knowledge to catalyze the Caribbean’s DEE, and specifically the mobile entrepreneurship ecosystem. This was expected to address some of the challenges in the region as CARICOM governments were facing serious fiscal difficulties. Unemployment was increasing while private sector investments and external capital flows were decreasing. The notion of an entrepreneurial ecosystem builds on previous understandings of economic agglomerations, such as clusters, national and regional innovation systems, and industrial districts.2 However, it differs from these in its explicit focus on growth entrepreneurship rather than any business activity. The key resources in entrepreneurial ecosystems are thus entrepreneurial resources, defined as “resources speci�c to the entrepreneurship process… rather than other types of industrial bene�ts found in clusters that accrue to �rms of all sizes and ages.�3 Isenberg and others use component models to describe the key elements of ecosystems4, usually highlighting that markets, human resources, entrepreneurial culture, finance, infrastructure, and policy all play a role for ecosystem quality. Yet, recent academic work shows that ecosystems evolve in a complex dynamic process where change can come from unforeseen directions, depending on local conditions.5 This implies that it is difficult to generalize which ecosystem component matters or does not matter in a given context, and how different components block or enable each other. In the end, ecosystem quality is a function of both the system’s endowment of entrepreneurial resources and how well-connected those resources are.6 This internal connectedness refers to both networks between actors (organizations and individuals) as well as the interplay between non-actor ecosystem components (e.g., policy, resources, culture, etc.). DEEs are a particular type of entrepreneurial ecosystems, namely those where digital enterprises rather than any growth- oriented enterprise are at the core of the ecosystem. The Caribbean Mobile Innovation Project (CMIP, P132577) was created under infoDev’s EPIC7 initiative and made possible through a US$1.5 million investment from Global Affairs Canada. The intent was to assist in addressing 1 infoDev is a World Bank Group multi-donor program to promote entrepreneurship and innovation in developing economies 2 Spigel, B. (2017). The Relational Organization of Entrepreneurial Ecosystems. Entrepreneurship Theory and Practice, 41(1), 49 – 72. Stam, E., & Spigel, B. (2018). Entrepreneurial Ecosystems. In R. A. Blackburn, D. De Clercq, J. Heinonen, & Z. Wang (Eds.), Handbook for Entrepreneurship and Small Business. SAGE Publications. 3 Spigel, 2017, p. 52 4 Isenberg, D. J. (2014). What an Entrepreneurship Ecosystem Actually Is. Harvard Business Review. Retrieved October 13, 2015, from https://hbr.org/2014/05/what-an-entrepreneurial-ecosystem-actually-is 5 Mack, E., & Mayer, H. (2016). The evolutionary dynamics of entrepreneurial ecosystems. Urban Studies, 53(10), 2118 –2133. Motoyama, Y., & Knowlton, K. (2016). From resource munificence to ecosystem integration: the case of government sponsorship in St. Louis. Entrepreneurship & Regional Development, 28(5–6), 448–470. 6 Spigel & Harrison 7 The Entrepreneurship Program for Innovation in the Caribbean (EPIC), was a seven-year $20 million Trust Fund program funded by Global Affairs Canada (GAC) and managed by infoDev, which aimed to build an enabling ecosystem for growth- Page 4 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) gaps in the Caribbean DEE and support the creation of growth-oriented businesses that could compete in global markets. The goal was in line with CARICOM Secretariat’s goal to leverage ICT as the new frontier for regional integration and an effective tool to advance regional socio-economic development. The scope was also well aligned with the priority areas defined in the 2014-2018 Strategic Plan for the Caribbean Community that included youth and ICT. In addition, fostering innovation entrepreneurship and mechanisms such as incubators and training were included in the regional partnership strategy for the OECS and in the Country Partnership Strategy for Jamaica, and the EPIC program in general was aligned with World Bank’s regional priorities and themes which included private sector-led growth, competitiveness and youth empowerment. This implementation completion and results report (ICR) is prepared for CMIP. The initial grant was to be spread over a 5-year project with multiple opportunities to scale and expand the services with further investment. CMIP had a regional focus on the 14 English-speaking CARICOM islands (Antigua and Barbuda, The Bahamas, Barbados, Belize, Dominica, Grenada, Guyana, Jamaica, Montserrat, Saint Lucia, Saint Kitts and Nevis, Saint Vincent and the Grenadines, Suriname, and Trinidad and Tobago), which was eventually implemented by UWI Consortium consisting of Lumin Consulting Inc. (formerly UWI Consulting Inc.), Mona School of Business and Management and the Mona Business Support Services, representing different parts of the University of the West Indies. While the initial research showed that there was an opportunity in the region for a strong and functioning mobile technology focused DEE, there were specific challenges unique to the region which created gaps in the ecosystem. The islands are both geographically and economically fragmented. Individual islands did not have the market size to support technology startups which rely on large economies of scale. The mobile application developers and entrepreneurs in the region were also fragmented, and there was little collaboration among application developers on different islands. Another challenge was the difficulty monetizing mobile apps in the region. This included entrepreneurs’ inability to create demand for purchased apps (instead of making them free), consumers’ difficulty in purchasing apps (revenue collection was challenging with fee-based electronic payment methods), and the high price of mobile broadband. Also, while it is easy to start a business in the region8, typical seed financing vehicles for startups were limited. Angel investing and venture capital were almost non-existent, with only a few instances where technology startups received outside funding. Finally, national governments have various ICT and innovation policies though with various levels of implementation and enforcement. While there was a strong awareness for the economic potential of ICT and application development, some policies were more forward thinking and aggressive than others. These challenges led to inconsistency in the maturity of the mobile application entrepreneurship ecosystem on each island, and in the region as a whole. To ensure local stakeholders were included in the project conceptualization for CMIP, feedback and insights were gathered through an extensive scoping effort that included over 300 stakeholders in the Caribbean DEE, and both the demand and supply side of the ecosystem were analyzed. On the demand side, there was a promising oriented and sustainable enterprises across 14 Caribbean Community (CARICOM) countries. 8 Economisti Associati was contracted to prepare a Caribbean Incubation needs assessment and feasibility study in 2012 and the team analyzed 2010 Doing Business Indicators, including the overall “Ease of Doing Business� rank as well as the “Starting a Business� rank for the countries in the region. The report noted that "with the exception of the Dominican Republic, Belize and Suriname (and Guyana for the overall indicator), Caribbean countries have good to average rankings in both indicators, i.e. they are all among the top 100 countries globally." Page 5 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) pool of early-stage entrepreneurs and application developers in the region that evolved out of “grassroots- driven� technology and entrepreneurship communities. While these communities had received incubation-type services prior to 2012, there was a desire at that time to access premium services through a regional approach instead of focusing solely on individual islands. The members of the entrepreneurship communities indicated that they were willing to pay for these formalized services through equity, and multiple stakeholders (governments, universities, and the private sector) expressed interest in supporting them. In addition, many stakeholders expressed that any service-offering also needed to help create a culture shift, as the region is risk- averse. Entrepreneurial culture is not commonplace in the Caribbean, and the openness to innovation was not embraced throughout the region. Training, mentoring, and coaching (both for incubator managers and entrepreneurs) were seen necessary to create this culture shift along with providing inexperienced entrepreneurs with knowledge about the business and technical aspects of startups. From the supply side, the Caribbean region had a relatively new history of business incubators and similar entrepreneurship support organizations. As of 2012, there were a number of physical incubators that were created in the 10 years prior, with varying levels of success. But, for the most part, there had not been many positive results. Recurring lessons from former and current incubators included non-sustainable grants and a lack of detailed partnership strategies for donor-funded incubators, a limited number of graduates, a lack of a clear strategy, and under-funded services by government-funded incubators. Despite the limited success with existing incubators, they continued to be established in the region. In the period of 2010-2012, five new incubators had been created, with more being planned. From consultations with several ecosystem stakeholders, it was clear to the EPIC team that the lessons from the failed incubators should be used to develop the most appropriate entrepreneurship services in order to help achieve success as well as sustainability. This included merit-based support, region-wide community-building and networking initiatives, hands-on mentoring and coaching, a focus on internationalization, and the integration of an equity-financing model via complementary EPIC initiatives9. It also became very clear that CMIP should take an approach of combining the islands into one market. This would help source sufficient deal flow for entrepreneurship support services within the specific sector. It also became clear that instead of creating new organizations to help fill gaps in entrepreneur support services, it would be more efficient to utilize institutions already set up in the region. Also, the services would need to be complementary in nature as not to compete with organizations already established in the region. The extensive effort that was put into stakeholder engagement, ecosystem mapping, and understanding the supply and demand for entrepreneurship support services, fed into a concept model for CMIP that was co- designed with local partners and end beneficiaries. The intent was not only to design CMIP so that it was effective in meeting the needs of the ecosystem, but also to create strong sense of local ownership and the longer-term sustainability of the initiative. Further, this engagement effort resulted in the following guiding principles for CMIP: Focus on mobile applications, implement a regional approach, focus on creating pipeline of growth-oriented entrepreneurs, establish a mentoring and coaching approach, utilize equity-financing and emphasize internationalization. 9 Within the EPIC portfolio, the Access to Finance (A2F) initiative offered ongoing support to existing angel investor networks and supported the development of an innovative investment facility, the Caribbean Investment Facilitation Project (CIFP). Page 6 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Complementarities and synergies with regional programs that had focus on innovation and/or entrepreneurship, such as the Caribbean Communications Infrastructure Program (CARCIP) and the Caribbean Growth Forum (CGF) were also considered, and later explored with the JM Youth Employment in Digital and Animation Industries Project (P148013) as it had an emphasis on youth employment in digital and animation industries, B. Project Development Objectives The Project aimed to (1) to strengthen the Caribbean mobile innovation ecosystem and (2) to enable sustainable and competitive mobile enterprises to grow. C. Key Expected Outcomes and Outcome Indicators PDO level results indicators were specified to measure the following: PDO Indicator 1: IT/ITES Revenue (Revenue generated by the startups) PDO Indicator 2: Investment raised PDO Indicator 3: IT/ITES employment (Direct jobs created by the startups, including breakdown for women) PDO Indicator 4: Sustainable startups created (surviving at least one year) D. CMIP Activities and Hub-and-Spoke Model CMIP was designed to focus on the following types of activities: • Local and regional entrepreneurship support and training activities to target growth-oriented mobile applications entrepreneurs • An annual startup competition to identify and train the top developer and entrepreneurial talent in the region • Pre-incubation services for the winners of the challenge to guide them to market and investment readiness • Continued incubation for graduated companies through local organizations, as well as access to finance, including a co-investment fund implemented under EPIC and additional capital from angels and venture capitalists These activities were to be implemented both virtually and in-person through a series of partnerships with existing entrepreneurship support organizations that formed a Hub-and-Spoke model of project implementation. The intended benefits of involving existing organizations were to increase the capacity of local actors (through a coordinated provision of high-quality support services to entrepreneurs) and strengthen the regional network of entrepreneurship support organizations (these organizations form an important part of entrepreneurship ecosystems). In addition, local organizations already had an existing network of entrepreneurs and relevant stakeholders, making it easier and more efficient to reach potential CMIP beneficiaries. To structure and implement the Hub-and-Spoke model, there would be one regional coordinating organization (the Hub) and several organizations acting as mHubs on specific islands (spokes). mHubs were not a new concept, as in 2012 there were eight infoDev supported mHubs operating in Azerbaijan, Georgia, Kenya, Moldova, Nepal, Page 7 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Tanzania, Uganda, and Vietnam. mHubs were conceptualized as a means to build mobile innovation communities worldwide by convening a variety of local stakeholder groups and mHubs organized informal gatherings and events that involved entrepreneurs, developers, mobile operators, device manufacturers, and local businesses. Depending on local needs, they also implement a choice of “lightweight� activities to strengthen the mobile innovation ecosystem, like peer-learning sessions or conferences. The World Bank Task Team used lessons learned from these existing mHubs10 to guide the design of CMIP. Besides fiduciary responsibility, the CMIP regional coordinating organization would also have multiple roles such as organizing the regional startup pitch competition (PitchIT Caribbean), organizing entrepreneur training sessions related to building businesses, and developing a program for investment readiness. It would also be responsible for partnership management, supporting local mHubs, and business development and fundraising for the project. The World Bank acted as trustee and implementation partner along with assisting in fundraising and partnership development. It was assumed that a high degree of collaboration would be required between the World Bank and the regional coordinating organization, and also between the mHubs and the regional coordinating organization. CMIP would also have a steering committee in order to provide support and guidance to the regional coordinator. These activities came to life through establishing a pipeline of activities that addressed the needs of entrepreneurs at key inflection points on their entrepreneurial journey. The Figure 1 illustrates these key inflection points as three phases: From Ideation to Pitch (i), From Pitch to Market (ii), From Market to Maturity (iii). 10World Bank. 2014. The Business Models of mLabs and mHubs : An Evaluation of infoDev's Mobile Innovation Support Pilots. https://openknowledge.worldbank.org/handle/10986/20079 Page 8 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Figure 1: CMIP Phases and Activities Phase I: From Ideation to Pitch: CMIP activities during this phase were to focus on a mobile startup training consisting of business and technical content streams, and also organizing and running prototyping events and hackathons. The phase also involved holding PitchIT Caribbean, a competition for developers and entrepreneurs to submit their mobile application business idea and prototype in order to win prizes, including further entrepreneurship support. PitchIT Caribbean activities involved an annual pitching event in which 25 finalists pitched their business to a panel of judges. Five finalists were to be selected to receive follow-on support in the form of financing, mentorship, and training. The role of the CMIP regional organization (as the Hub) was to coordinate all the activities, including branding and content, while the mHubs (as spokes) ensured local deal- flow was identified, awareness was raised, local ecosystem stakeholders were involved and local events such as training and outreach events were well organized. Phase II: From Pitch to Market: Activities during this phase involved supporting the five winning teams from each PitchIT Caribbean pitching event. The teams were to be matched with regional and international mentors, and mentors from the Caribbean diaspora. Initiatives also involved arranging networking events (both face-to-face and virtual) between the teams as well as other members of the Caribbean and international mobile innovation ecosystems, and developing and implementing an Investment Readiness Bootcamp for the teams. Planning and Page 9 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) organizing a DEMO Trek in which the winning teams travelled to an international event in order to showcase their product and pitch their companies was also a key initiative of this phase. While the overall coordination of the support was again coordinated by the CMIP regional organization, local mHubs had an important role as local mentors and organizations that provided office facilities where possible. They also assisted to collect data for M&E purposes. Phase 3: From Market to Maturity: During this phase the winning teams of the PitchIT Caribbean challenge were to be provided with lightweight support to help them further develop their startups and their entrepreneurial capacity. This included assisting the teams in applying for financing via EPIC’s Access to Finance facility and matching the entrepreneurs with other potential investors. It also involved maintaining and developing the network of alumni to enable additional opportunities for networking and peer-to-peer learning amongst the teams. This phase was mostly facilitated by CMIP regional organization, although mHubs were expected to maintain the relationship with the winning teams and also assist with the alumni network where applicable. To establish CMIP and fund its operations for the early years, an initial seed investment of $1.5 million USD was to be provided through EPIC (CMIP activities were included in one project component with US$1,500,000 cost at approval, and the actual amount at project closing was US$1,479,317). In the first two years, funding was earmarked to cover 100% of the CMIP operational budget both for the regional coordinating organization and the mHubs. In the subsequent years, the grant portion of the operational budget was to be decreased by 20% yearly. The funding was structured in this manner to ensure the sustainability of the initiative was actively taken into consideration by the regional coordinating organization. Revenue generation through training activities (training, hackathons, etc.), sponsorships, and exits from equity investments were expected to help cover the funding gaps in years 3-5, and set CMIP on a solid path towards sustainability. Theory of Change DEE projects can influence economic indicators for a region, and also the ecosystem itself. Economic impacts focus on the success of a particular startup or set of startups based on metrics such as headcount growth, revenue growth, and profitability. Ecosystem impacts refer to project activities that affect the overall chance of success for growth-oriented startups in an ecosystem. In the case of CMIP, a traditional M&E approach was used, as illustrated in Figure 2. This approach assumes predictable, controlled, impact chains, starting from the activities that were designed as the solution to the challenges in the region outlined earlier. For example, an incubated startup is expected to attain more revenue, which itself directly contributes to local economic development (e.g., measured as GDP). Page 10 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Figure 2: Theory of Change E. Significant Changes Made As a response to the request from the Recipient, the project underwent Level 2 restructuring in November 2017, close to the original closing date. The following changes were discussed with the World Bank Task Team and made at the time of restructuring, reflecting emerging lessons from project implementation: Closing Date: The closing date was extended by 12 months from November 30, 2017 to November 30, 2018. The extension was to allow sufficient time for the Recipient to monitor and collect results data from target beneficiaries to properly capture the growth in revenues, investments and employment attributable to the project, and complete the implementation milestones (to compensate delays in project set-up, as discussed in the summary of implementation chapter of Section III). In addition, the extension was to ensure the budget was fully disbursed. Results Framework: There were few prior examples and benchmarks of projects with similar characteristics and the original targets set for PDO indicators were based on an unscripted model specifically prepared for CMIP. Results data collected during implementation demonstrated that some of the key assumptions used for the original targets were too optimistic and the indicators ‘IT/ITES Revenue’, ‘Investment raised’ and ‘CMIP Reaching Financial Sustainability’ required adjustment to more realistic levels. It was also discussed with the Page 11 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Recipient that after two of potential mHubs withdrew their interest in joining the project (discussed in detail in the summary of implementation chapter of Section III), rather than adding two new mHubs, the Recipient was to invest more resources to ensure project sustainability. Changes in the cumulative target values included the following reductions: i. PDO Indicators: 1. IT/ITES Revenue: from USD 2,509,099 to USD 750,000; 2. Investment raised: from USD 1,626,917 to USD 500,000; ii. Intermediate Results: 1. CMIP Reaching Financial Sustainability Targets: from 49% to 25%; and 2. mHubs Established by CMIP: from 6 to 4. Institutional Arrangements: The clause requiring a project advisory board was removed from the grant agreement. The advisory board, with representation also from the industry, was intended to provide guidance and strategic direction for CMIP. The removal of the clause was based on the recipient’s interest to integrate services and activities more tightly into the University of the West Indies structure as opposed to setting up a separate legal entity (where a more structured advisory function would have had a more significant steering function). Instead, the recipient would invite strategic advisors to assist with the integration. Page 12 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) II. OUTCOME Assessment of Achievement of Each Objective/Outcome The PDO has two parts: (1) to strengthen the Caribbean mobile innovation ecosystem and (2) to enable sustainable and competitive mobile enterprises to grow. While these parts overlap to some extent they aim to capture the achievement for both types of impact, as introduced in the Theory of Chance figure in Section I. The evaluation of PDO achievement, is therefore also divided into these two parts by using both the original PDO indicators and comparing results to the targets as measures of success as well as additional evidence relevant to the PDO. See below table for a breakdown of the PDO and Intermediary Level Results indicators and Annex 1 for further details on the achievement of project indicators, including intermediary ones. Table 1: Achievement of Results Indicators Project Development Objectives (PDO): The project aims (1) to strengthen the Caribbean mobile innovation ecosystem and (2) to enable sustainable and competitive mobile enterprises to grow. Expected PDO Level Results Actual Achievements Unit of Measure Achievements Indicators Original Revised IT/ITES Revenue USD ($) 2,509,099 750,000 954,322 # of jobs 416 665 IT/ITES employment (direct/indirect), (119/297) (190/445) (of which women %) % gender 20% 39% Investment raised USD ($) 1,626,917 500,000 793,877 Sustainable startups created (surviving at # of startups 24 24 least one year) Intermediary Level Results Indicators Startup teams # of startups, 100 592 receiving services (of % gender (20%) (20%) which women %) Level of satisfaction by % rate good or startups of CMIP 75% 100% (respondents to the final survey) very good services Applications received through mobile # of applications 400 745 competitions CMIP reaching Generated 15%, and select CMIP assets and services financial sustainability revenue / 49% 25% integrated into the broader innovation strategy targets expenses and structure of UWI mHubs established by # of mHubs 6 4 4 CMIP To strengthen the Caribbean mobile innovation ecosystem Measuring the development of a DEE is challenging and there are many elements in innovation ecosystems that contribute to growth and quality. In fact, after the Theory of Chance was developed for CMIP, a number of Page 13 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) different ways to assess entrepreneurship ecosystems have been introduced, and it has become clear that ecosystem quality (i.e., the chances of success for growth-oriented startups) depends on the interplay of different ecosystem components, where strength in some ecosystem components cannot necessarily compensate for weakness in others. Even ambitious projects that support various ecosystem components such as CMIP are unlikely to effectively address ecosystems in their entirety and if particularly severe ecosystem bottlenecks are beyond the purview of a given project, startup growth on a wide scale would require complementary efforts. Many recent World Bank projects have prepared entrepreneurship ecosystem diagnostics which have improved the CMIP Task Team’s knowledge related to the topic. The evaluation of whether the project was able to strengthen the ecosystem needs to necessarily go beyond the PDO indicators and be based on the project results. Compared to an ecosystem framework developed by Babson College, CMIP addressed directly several components of the ecosystem framework such as human capital (by increasing technical and entrepreneurial knowledge of startups), entrepreneurial culture (CMIP achieved over 50 media appearances growing general awareness while several entrepreneurs reported mindset change as a direct outcome of coaching and other services), and strengthened support structures (mHubs as local entrepreneurship support organizations and PitchIT Challenge alumni network), but it left mostly untouched policy, market access, financing, and infrastructure components. When considering the PDO indicators, the one that tracked the number of ‘Sustainable startups created (surviving at least one year)’ can be linked to the strength of the ecosystem, as the results strengthen the broader entrepreneurship community and increase the number of experienced mentors and coaches. The project achieved the target of 24 startups created being sustainable and surviving at least one year. Teams that formalized their operations and that survived for at least one year after entering CMIP/mHub were counted (for example unregistered or ideation stage companies that become registered/commercialized after entering CMIP/mHub). The other three PDO indicators explained in the following chapter could also be considered at least partially relevant to this PDO following the logic that when startups achieve positive growth this could be the results of stronger entrepreneurship ecosystem and support structures achieved during the project. Four of the intermediary indicators are linked to ecosystem aspects, namely ‘mHubs established by CMIP’, ‘CMIP reaching financial sustainability targets’, ‘Applications received through mobile competitions’ and ‘Level of satisfaction by startups of CMIP services’. CMIP managed to establish four mHubs and meet the revised target, and importantly CMIP built the capacity of the mHubs so that they could provide training and development support services to mobile innovators and technology enabled entrepreneurs far beyond the project scope. This was backed by the findings of the End-of-Program Evaluation of the Entrepreneurship Program for Innovation in the Caribbean (EPE), prepared by Deloitte. The evaluation indicated that the capacity of local actors was increased including entrepreneurship support organizations in the mobile sector to better understand the needs of entrepreneurs and in some cases, to tailor offerings accordingly. The evaluation report highlights two CMIP mHubs as success stories, as in Barbados the mHub was able to facilitate accelerator services, which led to the creation of a robust framework to train entrepreneurs in mobile technology and other business topics. CMIP also increased the capacity of the mHub manager in St. Kitts and Nevis, enabling the mHub lead to re-assess her business model and client offerings. The indicator ‘Applications received through mobile competitions’ reflected the level of interest to take part of a region-wide activity, and by receiving 745 applications, CMIP demonstrated a healthier rate than expected by exceeding the target amount of 450 by 86%. The indicator ‘Level of satisfaction by startups of CMIP services’ Page 14 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) reflected the level of quality of CMIP activities. In an earlier survey during project implementation 85% of respondents had rated the satisfaction either good or very good and in the final survey those entrepreneurs who responded reported 100% satisfaction rating against the 75% target. Although there were areas to improve, the results indicate that many entrepreneurs were satisfied with the services provided throughout the project by local support organizations. The Recipient also collected additional data relevant to ecosystem development. In particular, the Recipient reported that they formed 74 formal partnerships between CMIP and local providers of services to entrepreneurs and organized 23 knowledge sharing events, further strengthening the linkages within the ecosystem and visibility of the topic. The recipient did not achieve the target of the final intermediary indicator ‘CMIP reaching financial sustainability targets’. With 15% the Recipient reported a lower achieved sustainability percentage compared to the revised target of 25% (the indicator measures the ratio between generated revenues to the cost of implementing activities). This is an important indicator as entrepreneurship support organizations need to be financially sustainable to provide services to their beneficiaries. Being partially dependent on grant financing or other type of a subsidy is very common for most of the incubators and accelerators around the world, but self-generated revenues from activities such as training or events, sponsorships, or returns on investments made into startups create a stronger financial foundation. Reasons such as delayed start-up and condensed timeframes had far reaching effects on attaining financial sustainability milestones. With activities commencing late, CMIP had barely two years to deliver and evolve its services portfolio, develop revenue streams, establish strategic partnerships with private and public sector organizations, and refine the business model. The shortened project execution period also meant that there was inadequate time to systematically test and validate key assumptions on revenue generation and the business model. There were attempts to generate revenue from public who were invited to follow PitchIT Caribbean Challenge events and discussion were held with potential donors and sponsors, but at the end the Recipient failed to prioritize revenue generation, which meant that revenue targets were not met during the project, even after they had been reduced from 49% to 25%. Eventually, the Recipient developed a plan to sustain select key activities by integrating them into a broader innovation umbrella of the University of the West Indies (UWIC innovation ventures initiative) after the initial grant period was over and partial sustainability was reached within the grant period11. At the time of the ICR preparation, UWI is establishing an Innovation Hub in Bridgetown, Barbados, which is proposed as the physical site for a hybrid incubator / accelerator to provide services and programing for potential high-growth startups. Additionally, three courses with the length of two to three months have been developed and are being offered to students aspiring to become entrepreneurs as well as entrepreneurs more widely in collaboration with UWI Cave Hill (through the Centre for Professional Development and Lifelong Learning). As the courses are not part of a degree but are being offered to the public in general, this is expected to provide consistent revenue for operations and serve as an identification and recruitment mechanism for acceleration programs once the physical location is completed. Lumin Consulting is also working alongside the Faculties of Science, Technology and Engineering across the campuses to activate a partnership with University of Waterloo’s Velocity programme. Intended as a knowledge exchange initiative with the university located in Canada’s Ontario, faculty and student entrepreneurs will be immersed in Velocity’s innovation programming to develop concepts and startups with high growth potential. Finally, the creation of the PitchIT alumni network will continue providing opportunities for networking, peer mentorship and learning. With these actions in place, part of CMIP’s activities will continue to be provided to a 11 UWI serves the needs of 17 English speaking countries in the Caribbean and has four campuses with 36,000 students. Page 15 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) new generation of entrepreneurs in the future. To enable sustainable and competitive mobile enterprises to grow Based on the Theory of Change, CMIP activities that targeted entrepreneurs were expected to produce entrepreneurs with higher capacity, having better access to mentors and coaches, and early stage funders. These entrepreneurs, in turn, would be able to establish companies, bring products and services to the market, and eventually create jobs, revenues and raise investments. After the first project manager was replaced, there was a good project turnaround, with efficient and successful completion of most activities and the final results the Recipient reported reflect that CMIP succeeded to achieve this objective. Overall, CMIP managed to support a higher than expected number of entrepreneurs, as the intermediary indicator ‘Startup teams receiving services (of which women %)’ recorded 592 startups across the Caribbean (against the target of 100). This pool of entrepreneurs made it possible for CMIP to attract high quality teams into the PitchIT Caribbean challenge, which included one of the most valued support through training, seed funding, incubation, coaching, mentorship and networking opportunities. The 75 finalist startups that participated in PitchIT Challenge represented 12 different countries (Barbados, Jamaica and Trinidad & Tobago accounted 72% of all the finalists). As further evidence the Recipient reported that 63 products were commercialized by the beneficiaries and 13 inventions had been registered with Intellectual Property office. As highlighted in the previous chapter that discussed the first objective, the 24 sustainable startups created (surviving at least one year) by the project is a PDO indicator that can be taken into account for this second objective, reflecting the results in the intermediary outcome level. The three other PDO indicators on the other hand reflect the ultimate outcome level, including revenues, investments and jobs, and Recipient reported the following results, all which achieved the original or revised targets: - The IT/ITES Revenue (Revenue generated by the startups) reached $954,322, which exceeded target of $750,000 by 27%. - The investment raised by startups in the project was $793,877 against the target of $500,000, exceeding the project target by $243,877 or 48.7%. The breakdown of the investments included four startups accounting $259,000 as early stage finance, and four other startups $153,154 as growth stage finance. Also, 14 startups were successful in raising grant funding totaling $381,743. - 190 direct jobs being created, and with an estimated 445 indirect jobs the overall IT/ITES employment result was 665, exceeding the target by 60%. Also, 39% of the created jobs were reported for women, which was substantially higher than the set target of 20%. Further discussed in section III, throughout the project, data collection from CMIP enterprises proved to be especially challenging and the growth numbers reported by the project are likely to be understated. Other Achievements Relevance: CMIP was rated highly relevant at the end of the program by final EPIC program evaluation (EPE) given the mobile sector was considered a growth opportunity and CMIP was the first to enable entrepreneurs with this specific sector scope. EPE also noted that with additional resources the relevance can be improved even further by tailoring programming and customizing content to target clusters of countries that have similar levels of entrepreneurial ecosystem maturity and social and economic development challenges. From institutional Page 16 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) point of view, the performance and learning reviews of the OECS regional partnership strategy in May 1, 2018 and for the Country Partnership Strategy for Jamaica in May 25, 2017 found the broader topic of fostering innovation and entrepreneurship also relevant. Overall Outcome Rating Even though some of the achieved indicator results were lower than what was anticipated at approval, based on the successful completion of planned activities and performance against project objectives, the overall outcome is rated Satisfactory. This rating considers the project’s ability to meet or exceed the revised targets of all four PDO level indicators and four out of five of the project’s intermediary results (which were adjusted to be better aligned with the realities of implementation). The overall outcome rating also takes into account the shortcoming with the intermediate level target for financial sustainability as well as the initially slow project execution resulting in a 12-month project extension. Other Outcomes and Impacts Successful Exit: The Recipient reported a sale of one CMIP supported startups for US$2.7m and suggested adding the final value to IT/ITES Revenue indicator (as this is revenue gained by the owners of the startup). The figure was eventually not added to the results, as proceeds from a sale of a firm would ideally be reported under a different indicator measuring this particular aspect. Gender: CMIP was able to attract and support a high proportion of female entrepreneurs from across the region, which according to the Recipient was the result of having less focus mobile developer talent and rather trying to attract more general entrepreneurs after the first PitchIT Challenge event. Furthermore, women entrepreneurs accounted for 100% of winners of the second PitchIT Challenge event, and 80% of winners of the third event. Knowledge exchange and international linkages: CMIP was selected as one of the 20 technology hubs around the globe for Slush Global Impact Accelerator in 2015, which increased peer-learning and knowledge exchange with other startup support organizations. In addition, the ability to attract international mentors from Canada as lead PitchIT Caribbean bootcamp coaches provided an opportunity for the Caribbean staff to learn coaching techniques based on global best practices. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME Summary of Preparation The design and pilot of the CMIP model was initiated at the end of 2012, including an extensive co-design process at the beginning of the project. Over 300 stakeholders were consulted across 7 islands. Based on the findings, an ecosystem map and a business plan were drafted by the World Bank Task Team. After the design, UWI Consortium consisting of Lumin Consulting Inc. (formerly UWI Consulting Inc.), Mona School of Business and Management and the Mona Business Support Services was selected as the Recipient and the CMIP regional coordinator through an open and competitive selection process. The UWI Consortium appeared as a strong candidate, as i) UWI had a regional footprint and was a neutral and respected university across the Caribbean, ii) it had experience implementing events and initiatives with the software technology community in the region iii) it had an efficient consortium consisting of established organizations with infrastructure throughout the Page 17 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) region, iv) UWI had prior experience implementing World Bank projects and v) their fiduciary capacity was assessed to be adequate and met the requirements of the World Bank. Many project stakeholders and beneficiaries confirmed that the project activities foreseen in the plan were relevant and appropriate for the Caribbean entrepreneurship ecosystem. However, several of the factors that affected implementation and outcome had been identified as risks, but the planned mitigation efforts were not enough to correct the challenges and reduce the impact. In addition, a risk that was not properly identified nor properly mitigated was the M&E challenge to collect the growth data relevant to three of the four PDO targets. These are discussed in detail below. Summary of Implementation Governance: Governance was identified during preparation period as one of the risks with the Recipient, as the consortium consisted of three entities. To mitigate the risk factor, the members of the UWI Consortium were required to prepare a legal agreement prior to the signing of the grant agreement which included both the roles and responsibilities as well as a specific governance structure. Unfortunately, even with this arrangement in place, the governance framework of CMIP was deemed complex and ineffective. The project manager had to align feedback and actions from many different stakeholders including the CMIP steering committee, the CEO of Lumin Consulting, as well as World Bank Task Team. This created a complex balancing act that was difficult for the project manager to handle efficiently. It was also noted that the CMIP steering committee lacked oversight from senior leadership at UWI. This might have helped ensure the necessary checks and balances were in place so that the steering committee was making key decisions expeditiously, and not negatively impacting the project manager’s ability to meet project milestones. Project Start-up: Although UWI Consortium had appointed contacts from each organization and had an acting steering committee, with an interim project manager, the recruitment of an experienced project manager took more than six months, with additional months to recruit the rest of the project implementation team, which was significantly longer than anticipated. In addition, as a part of a project set-up the UWI Consortium entered into agreements with organizations in four countries (Barbados, Dominica, Saint Kitts and Trinidad and Tobago) to act as mHubs12. As the selection was done competitively, even though the advertisement of the REOI for mHubs was done in December 2014, identifying best candidates and eventually negotiating multiple mHubs contracts at the same time took until the second part of 2015, further delaying the full-scale implementation of the project. The original plan was for six mHubs, not four, however negotiations with prospective organizations in Suriname and Jamaica concluded without the establishment of mHubs in either country. The negotiations with both entities had reached an advanced stage, when a restructuring process was launched by the parent company of one prospective organizations, followed by a change in the leadership in the second organization, which meant a shift in strategic priorities for both. These delays left a shorter than intended period for project implementation, as the initial project end date was end of November 2017. The initial plan was to run four cycles of a three-phase program (consisting of activities listed in Section I), but when feasibility of the original plan was evaluated together with World Bank Task Team it was agreed that due to the lengthy project set-up, the level of complexity 12In sum, CMIP consisted of the UWI Consortium as the regional coordinator and Barbados Coalition of Service Industries (Barbados), National Telecommunications Regulatory Commission of Dominica (Dominica), CARIRI (Trinidad & Tobago), and Quintessence Consulting Inc. (St. Kitts and Nevis) as the mHubs. Page 18 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) involved in facilitating a regional program and financial limitations only three cycles were to be implemented. Hub-and-Spoke model: Once the mHubs were selected, key CMIP stakeholders were dispersed and leaning towards a loose alliance of diverse and independent organizations with various roles and capacities. Although this sought to leverage diverse organizational capacities, it came in the way of clear and fast decision making. Also, the geographical scope in the project was dispersed across 14 countries and the project experienced challenges balancing focus vs dispersion in its approach. New and unproven project design – in particular connecting the local actors as mHubs under a regional hub - had been identified as a risk, and various meetings and a locally based World Bank consultant were introduced to improve coordination and efficiency, but in hindsight, the CMIP Hub-and-Spoke model proved to be overly ambitious for the region. Fundraising and revenue generation: Sustainability was identified as the greatest risk as a number of assumptions were used to build an initial sustainability model in the business plan that were expected to encourage revenue generation and avoid donor dependency. Unfortunately, even with focused efforts by the World Bank Task Team to support CMIP’s fundraising and revenue generation activities (which included training program for CMIP manager, identification of deal flow for grant funding, and direct technical assistance), the CMIP team did not manage to mobilize substantial revenues but in addition there was lack of attention and urgency by both the UWI Consortium and the mHubs to commence the execution and iteration of the sustainability strategy. It was clear to the UWI Consortium and mHubs that they would need to adapt the business plan to what would work best for the evolving ecosystem and their local markets. Ownership of the business plan and program design was intended to be transferred to the UWI Consortium and the mHubs, yet no significant revisions were made until the last year of the implementation. Project Management Consistency: CMIP was impacted by significant changes in project management and at times ineffective project management. Through the inception of the grant agreement in mid-2014 and up until 2018, four different project managers were leading the CMIP project. Although only two were external hires for the position (the other two were sourced internally from Lumin Consulting to ensure there was no gap in leadership), this undoubtedly had an impact on project costs, timeline, and the ability to meet milestones. Additionally, securing relationships with potential partners and funders, and developing deeper relationships with the mHubs, were also hindered by the high turnover of project managers. Project management challenges meant that CMIP lacked a dedicated ecosystem partnership and coordination strategy. The project intensified efforts to increase partnerships during the final year, but there was however inadequate time for the benefits of the partnerships to fully materialize during project implementation. Although the World Bank Task Team included highly experienced members who were able to provide technical assistance to the operational team members, the final EPIC program evaluation (EPE) noted that the project would have benefited if the project managers had direct entrepreneurship experience themselves from the industry (to have a better fit, which might have helped to avoid the high turnover in leadership). Monitoring and Evaluation: As noted earlier, throughout the project, data collection proved to be especially challenging. The CMIP team struggled to collect any M&E effects beyond what could be captured in CMIP participants’ immediate responses to surveys. The UWI Consortium, with significant support from the World Bank, attempted to rigorously follow up with participants, but it was very cumbersome to guarantee complete and meaningful data from across a broad set of project activities over several years. This is a particularly common challenge that most entrepreneurship support organizations face, but essentially the lack of data and an alternative theoretical model to take the data capture challenge into account meant that indicator results are Page 19 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) likely reflecting lower numbers than what the project actually achieved. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME Bank Performance: Due to the innovative design and complex regional model, the World Bank Task Team had agreed to an enhanced supervision effort from the start, with having locally recruited consultants in Jamaica to support both project implementation and communication efforts. In addition, in mid-2015 an experienced technology hub manager was recruited from Kenya to work with CMIP team on a short-term basis to address the delays in project launch. An assessment of the implementation progress and operations of the project was also done at the end of 2015, and actions were taken to improve the rate of implementation. This resulted in the change of the project manager (May 2016) and increased supervision assistance to strengthen the technical, financial management, procurement and other fiduciary capacity, in order to facilitate smoother project administration and implementation. This included additional efforts to help CMIP to revise the sustainability model driven by the decision to integrate CMIP assets to the broader UWI strategy. As a small Recipient Executed Trust Fund (RETF), the project information was migrated into an internal World Bank project information management portal several months after the start of the project, resulting that two ISRs were prepared after the migration. However, there were regular missions and frequent teleconferences with the Recipient throughout the project. Overall, the efforts above helped the project to complete its actions and achieve the results outlined earlier, but the Bank performance is rated Moderately Satisfactory, given there was a need to extend the project end date due to factors that might have been avoided with more careful planning and execution of mitigation efforts around key risks. Fiduciary Compliance: Overall, overall the project was executed in compliance with the World Bank’s policies and procedures but there was room for improvement, requiring Financial Management reviews and more significant than usual FM engagement to improve the timing and quality of financial reporting. Financial management. In general, there were established financial management arrangements which the Bank deemed to be functioning. For the duration of the Grant, the financial management performance was deemed to be Moderately Satisfactory, as the submission of financial reports were at times delayed, or inaccurately prepared and therefore required to be revised. Other aspects that needed to be strengthened included having a systematic budgeting system in place, and having a proper project chart of accounts designed in the accounting system. With that said, all audit reports were deemed acceptable to the Bank in which unmodified (clean) opinions were issued audited financial statements. Modified opinions were however issued in reference to the compliance with applicable laws, regulations and financial clauses, due mainly to the absence of an operations manual and the late submission of financial statements. Procurement. Based on the procurement post review carried out by the Bank, the overall procurement performance under the project was rated as Satisfactory, although there has been room for improvements. Indeed, some improvements were recommended by the Bank regarding procurement record keeping, evaluation methodology for individual consultant selection, expenditures classification as per relevant disbursement category in the legal agreement, and award criteria for contracts related to “training and workshops�, “operating costs�, “goods�, “works�, “services� and “consulting services�, but they were not strong enough to justify any downgrade in the above mentioned procurement performance rating. Monitoring and Evaluation. EPIC included a performance management framework for the program in order to Page 20 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) consolidate results data from various activities, including CMIP. For this purpose, the World Bank Task Team helped CMIP to establish a data collection framework for the project, including online survey tools, and provided targeted M&E training and support throughout the project implementation. However, as discussed in section IV, even with substantial technical assistance from the World Bank Task Team, CMIP’s operational team had significant challenges to collect data from entrepreneurs. V. LESSONS LEARNED AND RECOMMENDATIONS Outlined below is a summary of key lessons drawn from a separate lessons learned document that was prepared for the project after the grant completion. The lessons are organized according to three key themes identified from the analysis. Lessons related to project management, governance, and sustainability - Attracting and retaining competent and entrepreneurial-minded project management and ensuring they can act with alacrity are necessary conditions for the success of a complex, multi-stakeholder project such as the CMIP. - Harmonizing governance levels and decision-making responsibilities will reduce complexity and accelerate decision making. The success of a complex regional DEE project depends on the alignment of key stakeholders, and a governance framework that empowers quick decision making. - Sustainability planning and continued iteration and refinement of the business model needs to be prioritized by project management from the onset. This will ensure assumptions can be validated and adequate time is available to adjust and refine the business plan, while keeping in mind the fact that many entrepreneurship support organizations still depend on donor and government contributions for extended periods of time13 (regions such as the Caribbean are particularly challenging where startup success rates, valuation potential and profitability are often lower than in more advanced ecosystems). Systemic impacts and the measurement challenge - Founders should be exposed to learning opportunities that go beyond technical and business knowledge and extend to entrepreneurial knowledge and mindset change. Additionally, it can be beneficial for an ecosystem’s overall stock of entrepreneurial knowledge if experienced entrepreneurs have frequent opportunities to share knowledge with more novice entrepreneurs. - The impacts of systemic interventions on startup growth may take time to materialize, and they can be stalled by ecosystem bottlenecks beyond reach of a single initiative. - Find ways to continuously generate project knowledge and act on it. This effort goes beyond traditional monitoring and evaluation, and should be set up independently of reporting for accountability purposes. Understanding and designing systemic DEE interventions - Partnerships are essential to achieve ecosystem integration and division of labor. Project design must emphasize the development and adaptation of a partnership strategy, including the allocation of resources for dedicated personnel. - Leadership has to retain the decision-making power to significantly adjust the project design as needed, 13World Bank. 2014. The Business Models of mLabs and mHubs: An Evaluation of infoDev's Mobile Innovation Support Pilots. https://openknowledge.worldbank.org/handle/10986/20079 Page 21 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) in response to feedback from the project constituents. - Ensuring the project is relevant to the evolving needs of the ecosystem will require the right balance of project parameters, scope, and complexity. In addition, there are some additional practical operational lessons to World Bank Task Teams considering similar interventions in the future: - Although the use of the Hub-and-Spoke model faced challenges in the Caribbean, it has worked well in other regional digital entrepreneurship projects supported by the World Bank, for example in East Africa. Given roles and responsibilities have been defined well and expectations are managed from the start, the Hub-and-Spoke model can be an efficient model in strengthening both the capacity and the network of multiple entrepreneurship support organizations. - As the region did not have many local role models and there was a lack of experienced mentors, diaspora networks were initially identified as a channel to source high quality mentors. However, attracting diaspora required more effort and resources than the team could invest, and eventually diaspora support did not play the strong role it could have in supporting growth of the region’s early stage entrepreneurs. - Task Teams should consider the benefits and risks of both the RETF and direct contracting of the implementing organization. RETF was chosen for CMIP to address the need of capacity building via enhanced supervision by the Task Team and the ability to implement a longer-term program under the same contract (direct contracts have limits in terms of their length). Nevertheless, direct contracting can shorten project start-up time significantly if qualified local entrepreneurship support organizations are able to act as World Bank vendors, as the contracting and related administration can be done more efficiently compared to a grant. - After CMIP was approved, several other World Bank projects have been prepared with the objective to strengthen entrepreneurship ecosystems and growth of digital entrepreneurs. Thus, there is a growing pool of approved operations that can guide target setting for similar future projects. However, due to its qualitative nature, ecosystem level impact will continue to be challenging to prove, so indicator selection is likely to emphasize economic and quantifiable results. - To avoid the challenges CMIP faced with data collection, Task Teams could consider including incentive mechanisms to entrepreneurship support organizations (i.e. using results-based contracts), ensuring the project implementor has necessary resources for both an online data collection system as well as a dedicated resource to coordinate M&E collection and analysis, and requiring entrepreneurship support organizations to include clear data reporting requirements into contracts with beneficiary entrepreneurs. Also, traditional M&E has the key function to ensure project accountability and results measurement, but it is often difficult to adjust targets or investigate elements outside of the pre-specified project timeframe. For projects that target systemic interventions such as the CMIP, regular ecosystem diagnostics would be informative to help assess whether the project continues to meet the needs of the ever-evolving ecosystem. There are existing tools for task teams to conduct a diagnostic (which include the Digital Entrepreneurship Ecosystem Diagnostic tool and the Digital Economy for Africa toolkit with a section on digital entrepreneurship), and there is an ongoing project to further create, update, and test metrics and tools to better evaluate entrepreneurship ecosystems (Entrepreneurship – new metrics & pilots, P167399). . Page 22 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Enable sustainable and competitive mobile enterprises to grow Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IT/ITES Revenue (US$) Amount(USD) 0.00 2509099.00 750000.00 954322.00 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): Source of data: Recipient. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IT/ITES Employment (number Number 0.00 416.00 416.00 665.00 of people) 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Page 23 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Comments (achievements against targets): The recipient reported 190 direct jobs, with an estimated 445 indirect jobs, which exceeds the target. Also, the gender target for employment was 20%, and the end result was substantially higher with 39%. Source of data: Recipient. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Investment Raised Amount(USD) 0.00 1626917.00 500000.00 793877.00 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): The target was met. Source of data: Recipient. Objective/Outcome: Strengthen the Caribbean mobile innovation ecosystem Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Sustainable startups created Number 0.00 24.00 24.00 24.00 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): Page 24 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) The recipient corrected the final number to be 24, as the number that was reported earlier (62) was incorrect as the team had misunderstood the definition at the time. Eventual result meets the target. Source of data: Recipient. A.2 Intermediate Results Indicators Component: CMIP activities Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Startup teams receiving Number 0.00 100.00 100.00 592.00 services 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): The recipient reported significantly higher results than what was targeted. Also 20% gender result was reported, which meets the target for the indicator. Source of data: Recipient. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Level of satisfaction by startups Percentage 0.00 75.00 75.00 100.00 of CMIP services 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): Page 25 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) The recipient reported that all the responses to the final satisfaction survey were positive. However, the response rate was low, with only about 20 responses. Overall, with the earlier positive results about satisfaction, this intermediary target can be regarded met. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Applications received through Number 0.00 400.00 400.00 745.00 mobile competitions 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): The project achieved significantly higher number of applicants than targeted. Source of data: Recipient. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion CMIP Reaching financial Percentage 0.00 25.00 25.00 15.00 sustainability targets 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): The revenue targets were not met during the project as the grantee had changed their sustainability strategy, However, as the end result, the recipient successfully integrated select CMIP assets and into the UWI’s broad innovation program, thus reaching sustainability for thos e services. Source of data: Recipient. Page 26 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion mHubs established by CMIP Number 0.00 6.00 4.00 4.00 17-Jul-2014 17-Jul-2014 30-Nov-2018 30-Nov-2018 Comments (achievements against targets): Target was met. Source of data: Recipient Page 27 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) A. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1: Strengthen the Caribbean mobile innovation ecosystem. Outcome Indicators 1. Sustainable startups created (surviving at least one year) 1. mHubs established by CMIP 2. Applications received through mobile competitions Intermediate Results Indicators 3. Level of satisfaction by startups of CMIP services 4. CMIP reaching financial sustainability targets 1. Identified and established 4 hubs 2.Training courses were developed and delivered to regional developers and entrepreneurs through online and in person sessions. 3. Prototyping events and competitions organised for all 3 cycles by MHubs and CMIP Project Office. 4. Identified and mobilized 70 developers and entrepreneurs across the region for the PitchIT Caribbean Challenge events (three cycles of activities). 5. Three PitchIT Challenge events hosted. Two were held in 2016 and the other in 2017. Key Outputs by Component 6. Networking events arranged by MHubs and CMIP project office. (linked to the achievement of the Objective/Outcome 1) These included mingles, meet-ups and seminars. Additionally, peer led networking events were supported by CMIP project office in 2018 in Antigua, Trinidad and Tobago, Grenada, Montserrat, Jamaica. 7. The alumni network was launched in November 2018. Primary channels of engagement include the whatsapp groups and social media pages. 8. The integration of CMIP assets within the wider UWI innovation network formed the basis of the sustainability approach. The project extension period allowed for the identification, testing and confirmation of an appropriate business model. Page 28 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) Objective/Outcome 2: Enable sustainable and competitive mobile enterprises to grow. 1. IT/ITES Revenue Outcome Indicators 2. Investment raised 3. IT/ITES employment (of which women %) Intermediate Results Indicators 1. Startup teams receiving services (of which women %) 1. Supported PitchIT Carribbean Challenge winners to launch products, strengthen market share and investor readiness 2. Provided and managed $75,000 seed funding to 15 PitchIT Caribbean Challenge winners 3. Match some of the winning teams with mentors 4. Provided winning teams with facilities 5. Investment readiness training was delivered for all winners. Key Outputs by Component 6. Demo treks facilitated for all PitchIT Challenge winners. Events (linked to the achievement of the Objective/Outcome 2) included: Tech Crunch, London Book Fair, Design NDaba, Tech Beach Jamaica. 7. A project monitoring system was developed and implemented, utilizing twice yearly surveys. Client satisfaction surveys were also deployed to the entrepreneurs to gauge satisfaction with the project’s services. Knowledge sharing sessions were undertaken to ascertain progress, highlight achievements and assess impact. Page 29 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) . ANNEX 2. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The Project Manager indicated that “The ICR adequately captures and reflects the events leading to the implementation of the grant. The project played an important role in building the capacity of the Caribbean mobile entrepreneurship ecosystem to identify and support the development of innovative mobile start-ups. The support provided by the Bank to the grantee and its partners was beneficial in improving their ability to undertake, monitor and manage complex projects.� Page 30 of 31 The World Bank Mobile Innovation Project under EPIC (P132570) ANNEX 3. SUPPORTING DOCUMENTS Digital Entrepreneurship Ecosystems in the Global South - Lessons from the Caribbean Mobile Innovation Project Final Evaluation Report. Deloitte. 2019. "World Bank Group – End-of-Program Evaluation of the Entrepreneurship Program for Innovation in the Caribbean" Page 31 of 31