WPS8541 Policy Research Working Paper 8541 Can Government Intervention Make Firms More Investment-Ready? A Randomized Experiment in the Western Balkans Ana Paula Cusolito Ernest Dautovic David McKenzie Development Economics Development Research Group August 2018 Policy Research Working Paper 8541 Abstract Many innovative start-ups and small and medium-size ideas to independent judges. The investment readiness enterprises have good ideas, but do not have these ideas program resulted in a 0.3 standard deviation increase in fine-tuned to the stage where they can attract outside fund- the investment readiness score, with this increase occur- ing. Investment readiness programs attempt to help firms ring throughout the distribution. Two follow-up surveys to become ready to attract and accept outside equity fund- show that the judges’ scores predicted investment readiness ing through a combination of training, mentoring, master and investment outcomes over the subsequent two years. classes, and networking. This study conducted a five-coun- Treated firms attained significantly more media attention try randomized experiment in the Western Balkans that and were 5 percentage points more likely to have made a worked with 346 firms and delivered an investment deal with an outside investor, although this increase is not readiness program to half of the firms, with the control statistically significant (95 confidence interval of −4.7 to group receiving an inexpensive online program instead. A +14.7 percentage points) . pitch event was then held for these firms to pitch their This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at dmckenzie@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Can Government Intervention Make Firms More Investment-Ready? A Randomized Experiment in the Western Balkans# Ana Paula Cusolito, World Bank Ernest Dautovic, European Central Bank and University of Lausanne David McKenzie, World Bank Keywords: Investment readiness; start-ups; innovation; equity investment; entrepreneurship; randomized controlled trial. JEL codes: L26, M2, M13, O1  # Funding for this project was received from the European Commission. Funding to support the impact evaluation from the World Bank i2i Trust Fund is gratefully acknowledged. We gratefully acknowledge comments from seminars at Duke, the IGL/Nesta conference, the IPA SME conference, Lausanne, Toronto and from Marius Starke, Janko Milunovic and Peter Trapp. This experiment and a pre-analysis plan were registered in the AEA RCT registry on October 2, 2015: https://www.socialscienceregistry.org/trials/895. 1. Introduction Innovative start-ups and SMEs in developing and transition countries often have good ideas, but may not have these ideas fine-tuned to the stage where they can attract outside funding. This is the case in the Western Balkans, where there is a perceived lack of investment readiness of innovative start-ups to be in a position where they can compete for, and take on, outside equity (Karajkov, 2009). The most common reasons for a lack of investment readiness include a reluctance of entrepreneurs to surrender partial ownership and control of their business, lack of knowledge about the availability of external sources of finance, low investability of business development propositions, a lack of understanding about the key factors investors look for in making investment decisions, and presentational failings such as deficiencies in business pitches (Mason and Kwok, 2010). Investment Readiness Programs which provide individualized training, mentoring and coaching are designed to overcome these constraints, but the programs can be expensive to provide, and to date there is no rigorous evidence as to their effectiveness. We conduct a five-country randomized experiment in the Croatia, Kosovo, Macedonia, Montenegro and Serbia to test the effectiveness of such a program. A sample of 346 innovative SMEs were randomly divided into two groups: a treatment group that received a high-cost and intensive program that involved help developing their financial plans, product pitch, market strategy, and willingness to take equity financing, along with master classes, mentoring, and other assistance; and a control group which received access to an inexpensive online-only basic investment readiness course. After this program, both groups of firms competed in a pitch event, where they were scored by independent judges (blinded to treatment status) on their investment readiness, with the top 50 firms then going onto a finals stage where they pitched to investors. The independent judges scored the pitches on six aspects of investment readiness: team, technology, traction, market, progress, and presentation, with each firm scored by five judges. We find that firms that went through the investment readiness program receive an average of 0.3 standard deviations higher investment readiness scores at this event, and are more likely to get selected to proceed to pitch in front of investors. We then track firm outcomes over the next two years via a six-month and two-year follow-up survey, and through measuring their subsequent media mentions and social media attention. We find that treated firms do get significantly more 2 media mentions and social media buzz over the next two years (our measure excludes any mention related to the competition itself). The judges’ scores are found to be statistically significant predictors of investment readiness and investment outcomes over the next two years in the control group, suggesting that improvements in these scores should result in improved firm investment outcomes. Our point estimates show positive, but statistically insignificant, impacts on firm survival, three categories of investment readiness, and on steps towards receiving external financing, with treated firms being 5 percentage points more likely to receive external financing (95% confidence interval of -4.7p.p., +14.7p.p.). We reconcile the significant impact on judges scores and significant predictive effect of judge scores on firm outcomes with these results through discussion of a funnel of attribution, through which large changes in investment readiness are predicted to result in smaller changes in investment outcomes. The results highlight the difficulty in designing experiments to measure the impacts of such programs. The remainder of this paper is structured as follows: Section 2 discusses what investment readiness programs are, their use around the world, and contrasts them to other types of programs studied in the literature; Section 3 outlines the experimental design and provides details of the intervention; Section 4 provides the impacts on investment readiness; Section 5 examines how investment readiness translates into firm performance; and Section 6 concludes. 2. What Are Investment Readiness Programs and What Is the Evidence on Their Effectiveness? While much policy attention around the world has been given to efforts to expand the supply of equity finance for innovative start-ups and SMEs (through seed and venture capital co-investment funds and other activities to attract capital), the effectiveness of these programs can be hampered by a lack of readiness of these firms to receive equity investment. Mason and Kwok (2010) highlight three main aspects of this lack of readiness: first, many entrepreneurs are believed to be equity-averse, unwilling to surrender any ownership stake in or even partial control of their firms; second, many businesses that seek external finance are not considered “investible” by external 3   investors due to deficiencies in their team structure, marketing strategy, financial accounts, intellectual property protection, and other business areas; thirdly, even if entrepreneurs are willing to consider equity and have investible projects, presentational failings mean that many firms are unable to pitch their ideas successfully to investors. 2.1 What Are Investment Readiness Programs? Investment readiness programs are intended to increase the effective demand for equity financing by helping firms overcome the factors that result in a lack of investment readiness, thereby enlarging the size and quality of the pipeline of potential funding opportunities for investors and increasing the likelihood of new equity investments being made. These programs are a relatively new form of intervention, but there are now a number of examples in the U.S. and Western Europe. Appendix 1 provides details on a number of these programs, and we summarize some of these examples here. In the United States, the Larta Institute uses a combination of personalized mentoring, webinars and learning modules, and market connections to help National Science Foundation grantees in the Small Business Innovation Research program to develop Commercialization Plans. Several universities offer online investment readiness platforms, including the program we offer to our control group. In addition, there are a number of accelerators and incubators that offer investment readiness training as part of their broader array of specialized services. Examples in Europe include investment readiness support services provided by Enterprise Ireland, the Invest Academy Programme of the European Business Angel Network, the European program InvestHorizon and several demonstration programs provided by the UK Government’s Small Business Service. Such programs are rarer in less developed countries, but pilot programs have been introduced in a number of recently developed or higher middle-income countries. For example, the Romanian Innovation Commercialization Assistance Program (RICAP) worked with 30 firms to help technology innovators address commercialization needs,1 and the Malaysia Bioeconomy Accelerator Programme provides mentoring services to assist the commercialization of innovations developed in Malaysia.2 The Getting Ready for Capital (GReaC) project funded by                                                              1 http://portal.larta.org/ricap#what 2 https://portal.larta.org/malaysia   4   the EU aimed to help entrepreneurs in Bulgaria, Poland (and Belgium) understand the private equity market and effectively present their business propositions to investors.3 The World Bank is preparing an investment readiness component to a program in Morocco. While there is substantial heterogeneity in the content of these programs, the most comprehensive programs usually cover four dimensions, based on the core reasons that many investment deals do not materialize (Mason and Harrison, 2001; Mason and Kwok, 2010). The first dimension aims at reducing equity aversion, by explaining to entrepreneurs the potential advantages that equity can bring to the firm, both as a source of funding, and also because of the knowledge outside investors can bring to the firm. The second dimension addresses the investability of the business by helping to train the entrepreneur to demonstrate that they have a viable revenue model, can measure market traction, have dealt appropriately with property right issues, have a competitive strategy, etc. The third dimension works on the presentational skills, teaching the entrepreneur how to effectively pitch their business ideas and provide the key information investors are looking for. Finally, some programs also offer a networking dimension, aiming to facilitate the matching process between entrepreneurs and investors through events such as venture forums. These programs are offered in two modalities: “hard” and “soft” programs. Hard programs usually involve a package of support that combines online tools and training, customized and face-to-face mentoring, group training through masterclasses, and investor demonstration days or pitch events. Soft programs are self-learning online tools structured in modules that entrepreneurs can work through at their own pace. Both types of programs tend to be subsidized by governments, even in developed economies like the U.S. and U.K. There are several possible reasons to justify subsidies. The first is that the targeted firms are frequently liquidity constrained, and therefore unable to pay. Some incubator programs like Y-Combinator overcome this constraint by investing seed capital in the firms in exchange for an equity stake in the business. But since equity-aversion is one of the key constraints investment readiness programs are trying to overcome, investment readiness programs have typically not required equity stakes in exchange for participation. Secondly, since many of these programs are new in nature, potential entrepreneurs may find it hard to assess in advance the                                                              3 http://greac.eu/   5   overall quality of the program, and their payoffs from participation are highly uncertain, making them unwilling to pay the costs of participating. Finally, governments may justify the subsidies in terms of the public benefits (more innovation, higher tax revenues, greater employment) that can come from successful ventures. 2.2 Existing Evidence about Their Effectiveness Currently there is no causal evidence as to the effectiveness of these investment readiness programs. The existing literature consists of several case studies and descriptive evidence. Several studies attempt to argue that a lack of investment readiness hampers equity investment, focusing in particular on presentation skills. Mason and Harrison (2003) use a case study to argue that poor presentational issues dominate the reactions of potential investors to a business proposal and constrain the likelihood of a deal taking place. Clark (2008) uses questionnaires submitted to business investors after an investor forum, and shows that presentation skills are significantly correlated with investment decisions. Mason and Kwok (2010) provide a descriptive evaluation of the U.K. Government’s Small Business Service’s Investment Readiness program. Consultants judged the program to have had success in awareness raising, business development, and funding, but acknowledge that they do not have a counterfactual, and that it was difficult for businesses to reflect on what their behavior would have been without the program. They also report that tracking participants in the Finance and Business program of the North East Regional Development authority in England found businesses reported increases in funding, sales, and jobs two years after the program, but do not have a control group against which to compare the before-after comparisons. More rigorous non-experimental evidence comes from work on related programs.4 One set of related programs are business accelerators and incubators. These differ from investment readiness programs in typically being more intensive and expensive, often offer some seed capital and workspace in addition to training and mentoring, and work with a much smaller number of firms at any one time. For example, accelerators like Y-combinator take an entry cohort of 10 to 20 firms, who then receive seed capital, move to Silicon Valley for 3 months, and culminate with a                                                              4 Somewhat, but less closely, related is regression-discontinuity work that shows the impact of financing to innovative firms on subsequent firm outcomes (Kerr et al, 2014; Howell, 2017); and experimental work on business training and consulting services to firms (reviewed in McKenzie and Woodruff, 2014). 6   demo day in which they present their ideas to selected investors. Several studies have used matching approaches to compare firms going through accelerators to similar firms which did not. Hallen et al. (2014) compare accelerator-backed new ventures to a matched set of non-accelerator ventures and find the former are faster at raising venture capital and gaining customer traction. Smith and Hannigan (2015) match firms going through Y-Combinator and Tech-Stars to start-ups that instead received financial assistance from angel groups, and find participation in a top accelerator program increases the speed of receiving follow-on funding and the speed of exit. In contrast, Yu (2016) matches accelerator and non-accelerator companies and find the former raise less funding, and close down earlier. Gonzalez-Uribe and Leatherbee (2018) use a regression- discontinuity approach to compare start-ups enrolled in the Start-up Chile accelerator program to those just below the qualification threshold, finding a positive relationship between the mentorship and the scale of the start-up and access to seed and venture financing. Finally, two recent experiments examine effects of short, cheap interventions to potential business ventures.5 Wagner (2016) conducts an experiment with 88 Start-up Chile grantees, and finds giving written feedback on their business plans makes firms more likely to survive, as measured by web presence, but is not able to measure other outcomes. Clingingsmith and Shane (2017) provide 30-minute pitch training to undergraduate students in Ohio, who then deliver 90-second pitches to judges. They find training actually lowers the judges scores on average, by helping judges better distinguish bad from good ideas, and having a more negative impact on low quality ideas than the positive impact on better quality ideas.6 Our work differs from this work on accelerators in at least four key aspects. The first is in the type of program being analyzed: investment readiness programs of the type we study here are designed to be able to be scaled up and operate with sizeable numbers of firms at a time, compared to the more intensive focus on a small number of firms at a time in incubator and accelerator programs. Second, ours uses a randomized experiment, overcoming concerns about how well non- experimental methods are able to overcome biases induced by the explicit selection mechanism that aims to choose better firms for the program than the non-participants. Thirdly, the existing                                                              5 Another example of an experiment with early-stage ventures is Bernstein et al. (2017), who randomize the information potential investors on AngelList receive about start-ups, and find that information about the founding team matters for their decision to find out more about the company. 6 Using a difference-in-differences approach, Howell (2018) also finds that negative feedback leads to more abandonment of business ventures.  7   literature has largely relied on a small number of outcome measures that can be collected without the use of firm surveys – survival, whether or not they received venture funding, and web traffic measures. We have much more detailed data, including the use of judge scores and intermediate outcome indicators that allow us to focus on investment readiness, and not just investment outcomes. Finally, the majority of studies focus on the U.S. which has a well-developed venture capital market, whereas we focus on an area of the world where firms are only just starting to engage with outside investors, making investment readiness programs potentially more important. 2.3 Why an investment readiness program in the Balkans? Increasing innovation is a key regional priority in the Balkans region as a means to boost firm productivity and sustain economic growth. While it is generally accepted that debt finance is not the optimal source of funding for early-stage SMEs and start-ups, equity finance is only marginally used in the region. A regional report noted that there is a debate as to how much this lack of use of risk capital reflects a lack of supply of equity finance, versus a lack of readiness of entrepreneurs to attract and accept this financing (Karajkov, 2009). Based on the viewpoint that action was needed on both the supply and demand sides, the Enterprise Development and Innovation Facility (EDIF) initiative financed by the European Commission includes efforts to increase the supply of private equity to the region, improve the legislative frameworks to better encourage venture capital activity, and undertake efforts to increase investment readiness. This paper provides an evaluation of the investment readiness component of this initiative. 3. Experimental Design To implement this intervention, we ran a competitive procurement process where companies specializing in investment readiness programs provided bids. We shortlisted five companies, and together with evaluations of these proposals from Josh Lerner from Harvard University and his team at Bella Research Group, and with advice from experts in the national innovation agencies of the participating countries, chose as the winning firm the company Pioneers JFDI GmbH (Pioneers henceforth). Founded in 2009 and based out of Vienna, they are one of Europe’s leading platforms for entrepreneurship, organizing an annual “Pioneers Festival” (with 3,000 attendees), as well as providing mentoring, pitch training, and opportunities for presentation and networking with European and international founders and investors. They launched a specific investment readiness program called Pioneers of the Balkans for this project. 8   3.1 Generating the Sample Eligibility criteria for the program were developed by the World Bank and Pioneers team, conditional on the rules of the European Commission. To participate in the program, a firm had to be legally registered in at least one of the five countries: Croatia, Kosovo, Macedonia, Montenegro or Serbia. The firm had to be a micro, small, or medium-enterprise, defined as having fewer than 250 employees, and an annual turnover below 50 million euros. It had to be innovative, meaning that “it will in the foreseeable future develop products, services, or processes which are new or substantially improved compared to the state of the art in its industry, and which carry a risk of technological or industrial failure”, and could not be on a sanctions list or operating in a set of negative activities (e.g. gambling or alcohol production). To launch the program, the brand Pioneers of the Balkans was created, and a dedicated website set up.7 The program was marketed as a competitive program designed especially for innovative entrepreneurs seeking or considering venture financing. The main communications therefore promoted a major pan-regional start-up competition due to take place in two stages, with a Semi- finals in Belgrade and subsequent Finals event in Zagreb. It included a preliminary list of investors who had already confirmed their attendance at the Finals and noted that selected firms would receive a training and preparation package. We had set a target of 300 to 350 participating firms. In designing the program, both providers of investment readiness services and experts in the innovation agencies agreed that there was a limit on how many firms potential investors would be willing to listen to pitches from. They also noted a concern that randomly choosing a firm to pitch in front of investors that was not of high quality could have reputational risks to the region, with potential investors observing such firms as a signal more generally that firms in the region are not of high enough quality to merit investments. A two- stage process was designed to overcome these issues: the Semi-finals would be the main phase of our study, with all firms in the study having a chance to present their ideas in the semi-finals and get scored by independent judges on their investment readiness. Then only the top-50 would progress to the finals, with these firms selected on merit.                                                              7 http://www.pioneersofthebalkans.io [accessed May 5, 2018] 9   Pioneers aimed to create broad awareness of the program among entrepreneurial firms in the region, launching the program at the start of August 2015 (see timeline in Appendix 2) and marketing the program rapidly. It used five major instruments to achieve this goal: public sources of information for applicants, direct electronic and physical mailings, social media marketing, a roadshow spanning all five target countries, incentives for early applications (a raffle for a dinner with two leading entrepreneurs from the region), and media relations. A list of more than 1,200 potential contacts was directly emailed using firm names provided by the local innovation funds and government counterparts, and other contacts in the region. LinkedIn and Facebook advertising was used, and “multipliers” were asked to spread the word to their contacts in the region. Applicants had to apply online, with the data from this application form providing the baseline data for this study. More than 1,200 applications were started online, and a total of 584 full applications were received. These were screened for eligibility, resulting in 346 firms being selected as eligible for the program. This process succeeded in generating a sample of young firms involved in a wide range of innovative activities. At the time of application, firms had an average of 6 employees, with a 10- 90 percentile range of (1, 12). They had been in business for 2.5 years on average, and are involved in high-tech innovative industries such as cloud computing and big data, app development for a wide range of business and personal services, pharmaceutical products, etc. Half of the founders have post-graduate education, and 60 percent have a global rather than regional focus as their key market. To make clear the types of firms involved, it is worth giving some more specific examples of the types of innovation these firms are doing. Some examples are as follows:  A firm that is developing virtual reality software that can be used in outdoor interactive missions, with the aim of deploying this in military training exercises and theme park adventures (e.g. a team-based maze/obstacle course where dragons and other objects are flying around)  A firm developing an app that geo-locates users on ski fields in Europe, and provides a way for them to see where all their family members are at any point in time, and to direct them to common meeting places.  A bio-tech firm that has developed a new coating for common medicines that allows the body to better regulate the dose-intensity, to reduce under- and over-dosages of medicines 10    An architecture firm that has developed an innovative luxury “boatel” that runs on an electric motor and can be used on lakes  A firm that has developed solar-powered benches for public spaces that can charge phones and also monitor air and noise quality. A number of the firms were developing apps for the Balkan and global markets, covering a wide range of activities such as making it easier to use public transport, a local version of Uber, an app to connect consumers with producers of organic products, online sports coaching, and an app to manage freight logistics. But there are also firms involved in physical manufacturing of products, such high-end electrical bicycles, smart vending machines, indoor pet houses, and a USB charger that charges while bicycling. 3.2 Random Assignment Applications closed on September 6, 2015 and were then screened to ensure they met the eligibility requirements. All applicants which met the formal eligibility criteria were accepted into the study. Eligible applications were then scored on four criteria to measure their initial level of investment readiness: market attractiveness, product technology, traction, and team. Appendix 3 describes the scoring methodology. The top 10 proposals overall in terms of score were then randomly assigned to 5 in treatment and 5 in control, in order to ensure that some of the very top proposals were in both groups. Then the remainder of firms were divided into strata based on country (Serbia, Croatia, or the rest), and on whether or not they already have a private investor. Within these six stratum firms were ranked into groups of four on the basis of their investment readiness score. Within these quartets two firms were randomly allocated by computer to treatment and two to control. This was done for an initial batch of 333 firms, allocating 167 to treatment and 166 to control. An additional batch took longer to verify their eligibility requirements and were received after this assignment, these were then also randomly allocated and form a separate strata. This resulted in 346 firms, with 174 treatment and 172 control. A pre-analysis plan was registered with the AEA trial registry on October 2, 2015 to pre-specify the initial outcomes of interest.8 This process resulted in treatment and control groups that are evenly balanced and comparable in terms of their initial characteristics. This is seen in Table 1. Figure 1a shows that the two groups                                                              8 https://www.socialscienceregistry.org/trials/895 11   are also similar across the entire distribution in terms of initial investment readiness. As a result, any difference in investment readiness at the conclusion of the program can be reliability assessed as the impact of the program and not due to any pre-existing differences across groups. 3.3 Details of the Treatment and Control Offerings The treatment and control groups were blinded to treatment status, and both were offered a form of investment readiness training – the difference being in the intensity, cost, and medium of the offerings. We summarize both treatment and control programs here. A key issue with understanding the impact of different training programs is that much of the literature does not provide sufficient detail on what was offered, leaving the program as a black box for others seeking to learn or compare. Therefore, in Appendices 4 and 5 we provide much more detailed information on each program. The treatment group received an investment readiness program provided by Pioneers, but branded under the name Startup Live Mini-Accelerator. This was an intensive two-month program that aims to prepare companies to be in a position where they are ready to talk with potential investors. The first phase (“qualification”) was structured around an online training platform called WhatAVenture. Using this tool, individuals are asked to outline and self-critically assess their businesses by describing the problem or need addressed by their product or service, the commercialization concept and expected revenue streams, conduct a market sizing exercise, and describe their competitive positioning. Each business was assigned a lead mentor who supports them through this process and provides feedback and help. After completing this first phase, firms were then brought into an “acceleration phase”. In this phase they had individualized mentoring from both their lead mentor, and from a pool of more than 100 specialized mentors who could help out on specific concrete and sector-specific needs. Mentoring took place both on-site and via video calls. During this phase, there were four masterclass weekends, which took place every week in October from Friday evening through Sunday afternoon. These masterclasses rotated around the different countries, and were recorded so that those who could not attend in person could access the contents online. Each workshop followed a similar format, but with the topics varying. On Friday evenings the attending entrepreneurs would have a chance to introduce themselves and their businesses in just 90 seconds with no presentation materials, and also see examples of the same from the mentors, followed by 12   informal discussions. Saturdays would involve five to eight lectures and/or workshops, with themes such as sales and marketing, team building and human resources, and investment and finance. On Sundays, all participants and mentors focused on presentational skills as well as pitch deck structure and design. The final phase was a “pitch preparation phase” and took place in the last two weeks, in the run-up to the semi-finals. This included working on their pitch decks with their mentors, delivering practice pitches, and then on-site training in Belgrade the day before the semi-finals performance as a final practice run. The total cost of the treatment is estimated to be $614,000, or approximately $4,000 per active participant.9 The main component of the cost is the individual mentoring, which averaged $3,072 per beneficiary, with the masterclasses costing $793 per beneficiary and pitch training $230. The control group companies were offered an e-learning course developed and distributed by the Global Commercialization Group (GCG) of the University of Texas at Austin. This course is distributed under the label Innovation Readiness SeriesTM and was launched in 2011. It is targeted to a broad audience of entrepreneurs, scientists, engineers, and students, with the goal in helping transform their innovative and technology-based concepts into a viable commercialization plan and a convincing pitch. The content is delivered online through 10 modules of 45-60 minutes each, with a multiple-choice quiz at the end of each module. Appendix 5 provides descriptions of the content of each module. They cover key issues such as how to articulate the benefits of an innovation to customers and investors, intellectual property protection, market validation, comparing to competition, and how to pitch and present. The cost of the course was a one-time $5,000 set-up charge to customize to our program, and then $153 per firm. There were several reasons for offering the control group an online investment readiness program rather than not providing any service at all. The first was that, from a public policy point of view, a key question was whether an expensive and intensive program was needed, or whether identical results could be obtained by cheap and accessible online alternatives. This was considered the more interesting policy counterfactual than offering nothing at all. Second, from an evaluation                                                              9 The exact cost per firm differs in terms of services contracted vs services actually delivered, since not all firms used all the mentoring hours they were allocated. Pioneers retrospectively estimates that the actual services delivered to the firms were approximately $3,000 based on actual hours mentoring used. Note further that this calculation does not include the costs of advertising the program through roadshows, or of putting on the semi-final and final events, which were important in attracting firms to the program. These overhead costs are estimated at approximately $1,500 per firm (in both the treatment and control groups). 13   standpoint, offering both groups an investment readiness program lowers the risk of Hawthorne and John Henry effects, since both groups were told they were being provided with an investment readiness program. Finally, we also believed that offering the control group something would minimize the risk of differential attrition compared to the treatment group. 3.4 Take-Up Of the 174 firms randomized into treatment, 157 (90.1%) completed the WhatAVenture online training platform, and 79.3% received individual mentoring. Conditional on receiving individual mentoring, entrepreneurs received a median of 8 and mean of 11 hours of individual mentoring from the lead mentor and pool of specialist mentors.10 76 out of the 174 (43.7%) attended at least one masterclass in person (videos of the masterclasses were also available online, with typically 10-20 firms watching each). There were approximately 1,150 mentoring hours provided during the masterclasses, of which around 390 hours were individual mentoring, and 760 hours were in the form of lectures and presentations. This represents an average of 15 hours per attendee. In addition, before the semi-finals, 76 firms (43.7%) attended a 3-hour final pitch presentation training. Table 2 examines the correlates of take-up of the WhatAVenture tool and of masterclass participation amongst those in the treatment group. We run a probit of take-up on all the baseline characteristics in Table 1, and then run a stepwise procedure to progressively drop the largest coefficient with a p-value above 0.2 to end up with the sparser specifications in columns 2 and 4. For the initial stage of using the WhatAVenture tool, the only variable that is consistent in the sparse model is the initial investment readiness score: firms with higher initial readiness are more likely to complete this first phase. However, we then see in columns 3 and 4 that higher initial investment readiness is associated with a lower probability of attending a masterclass. This might reflect that firms who already are more ready feel they have less need to learn from such workshops. We see attendance is lower for firms from Croatia. This perhaps reflects the masterclass weekend in that country being held in Split, rather than the larger city of Zagreb: a one-day workshop was held additionally in Zagreb, and including this reduces the gap slightly. Attendance is higher for firms whose owners have post-graduate education, and for those who                                                              10 Note firms were eligible to receive up to 30 hours of individual mentoring time, so the majority of teams used considerably fewer hours than allocated to them. 14   have participated in a mentoring or acceleration program before, potentially reflecting a taste or revealed preference for training, or complementarities with existing skills. Finally, companies which use cloud technology were more likely to attend. When asked in our follow-up survey why they did not attend, the most important reasons given were that they did not want to take the time away from their businesses (and in some cases second jobs as employees), and that the locations were too far away. Out of the 172 participants assigned to the control group, 120 (70%) accessed at least once the online Innovation Readiness SeriesTM platform. However, even conditional on accessing the platform, overall usage was relatively low. Conditional on accessing the online platform, 118 participants viewed at least once the modules’ section and 55 viewed it at least 10 times; the mean number of views of the modules section was 21 and the median 9. Each module last approximately half an hour, so we can approximate that the mean time spent on the modules was 10 hours while the median 4.5 hours. Only 63 (37% of the control group) participated in one of the seven quizzes at the end of a module. A total of 51 control group entrepreneurs passed at least 4 quizzes with 45 attaining the threshold of 70% correct answers in all quizzes, necessary to receive a certificate of completion from the IC2 Institute at the University of Texas at Austin. The main two correlates of taking and passing the quizzes among the control group are having postgraduate education (positively correlated), and having previously participated in a mentoring or accelerator (negatively correlated). The online courses are thus done by those who have more schooling and have not previously had exposure to such content. 4. Impacts on Investment Readiness as Scored by Judges 4.1 The Semi-finals and Judging Procedure The semi-finals were held in parallel to, and in cooperation with, the Belgrade Venture Forum, an annual venture capital conference that took place from November 12 to 14, 2015. Participants were invited to present in a pitch event that follows the standard format of such events, with firms giving a 5-minute pitch of their business case, followed by 5 minutes of questions from a jury of judges. Participation required the founder of the firm or a representative to be physically present in Belgrade. To encourage participation, firms received multiple reminders and calls, were sent an invitation letter with a ticket voucher that allowed them one day of free access to the adjoining 15   Belgrade Venture Forum, and were provided with a transport subsidy that was sufficient to cover the cost of bus travel to the event. The travel time was approximately 4 hours from Croatia, 5 hours from Macedonia, and 6 to 7 hours from Kosovo and Montenegro. In total 211 of the 346 invited firms (61%) attended the semi-finals: 110 firms from the treatment group (63.8%) and 101 firms from the control group (58.1). The attendance rate was similar for Serbia (64%) and Croatia (67%), and lower for the other three countries (51%). Attendance rates were higher amongst those who had participated more in the intervention. Amongst the treatment group, 81.6% of those who had attended at least one masterclass attended the semi-finals, versus 49.0% of those who had not. Among the control group, 88.9% of those who had taken any of the quizzes attended, versus 41.3% of those who had not. We discuss robustness to this attrition in the next section. A group of 66 independent judges was used to do the scoring. Panels of five judges were assigned to judge a session of six firms at a time, with judges then being rotated so that they are on panels with different judges for their next sessions. Each batch of six firms consisted of three treatment and three control firms, selected to have a similar range of initial investment readiness scores, and grouped according to industry and country of operation. Judges were assigned to batches based on their availability (some were giving talks at the venture forum), industry, and technology used. Appendix 6 provides details of characteristics of these judges. They were a mix of investors, successful business owners, and experts in mentoring and coaching start-ups. Thirty-seven percent lived in one of the five countries taking part in the competition, while two-thirds were based in other countries. Eighty percent of them regularly mentor start-ups, 64 percent were part of companies that make venture investments, and three-quarters had founded their own companies. They were therefore experienced in what outside investors are looking for in terms of investment readiness. Judges were blinded to treatment status, and were not provided with any information about the company in advance of scoring. They were briefed and asked to score each firm on six factors: 1) Team: the skills and capabilities of the entrepreneur and his or her team 2) Technology: the degree of innovativeness and technological advancement 3) Traction: indications of measureable market success 4) Market: the commercial market attractiveness and size of the potential market 16   5) Recent business progress: the amount of progress firms had made during the last three months (the time since initial application) 6) Presentation performance An aggregate investment readiness score was then formed using the following weights: (team) 28%, (technology) 21%, (traction) 14%, (market) 7%, and (progress) 30%. These weights were not revealed to the judges, but were based on what seed- and early-stage investors would commonly focus on (Kaplan and Strömberg, 2004). They tend to emphasize the quality of the team and their technology (Gompers et al, 2016), and the extent to which the business is continually improving. The presentation score was added to allow judges to independently assess how well the firm presented its ideas, and as “hygiene” factor that could be used if necessary to avoid placing someone unable to present in front of investors at the final. The correlation between this weighted score and an equally-weighted score is 0.995, and we show in Appendix 6 that our results are robust to this choice of weighting. There were two ways for firms to be selected for the finals. The main path was through an overall ranking based on the aggregate investment readiness score. Secondly, judges scored each firm after watching its pitch, and then at the end of the batch of six presentations, discussed the set of six. They then were asked to collectively rank the three best they had seen out of the six, and could choose to directly nominate the top-ranked firm to directly be sent to the finals. They were asked to use this direct nomination selectively, reserving it only for firms they believed should certainly be granted the opportunity to present in the finals. The idea behind direct nomination was to allow for the possibility that through collective discussion, the strength of a firm may be more apparent. Sixteen firms were directly nominated to the finals, of which only four were not in the top-50 overall based on the individual ranks.11 Then firms ranked in the top 46 based on the overall score were also chosen to give a total of 50 finalists. We then examined how sensitive these rankings were to allowing for differences in scoring amongst judges, and re-ranked firms on their residual scores after subtracting judge fixed effects. Four additional firms were chosen as finalists based on having judge-fixed-effect-adjusted scores in the top-50 even though their raw scores were not in the top 50. This gave a set of 54 firms that were invited to the finals.                                                              11 They ranked between 58 and 74. 17   4.2 Estimating the Impact on Investment Readiness as Scored by Judges To estimate the impact of the program on investment readiness as scored by the judges, we use the following (pre-specified) base specification for firm i in stratum s: ∑ 1 (1) Where 1 are strata dummy variables. Note that stratification implicitly controls for baseline investment readiness, country, and whether or not the firm has an outside private investor at baseline. Robust (Eicker-White) standard errors are used. As a robustness check, we also re- estimate equation (1) after controlling for judge fixed effects. The parameter β is then the intention-to-treat effect (ITT). This measures the impact of being assigned to the treatment group, and being offered the expensive and intensive investment readiness program rather than the online course offered to the control group. We could also attempt to measure the local average treatment effect (LATE) of actually receiving treatment. Recall that 90.1% of the treatment group completed the WhatAVenture tool. However, all but one of the treatment group firms that attended the semi-finals (99.1%) had completed this tool, so the non- compliers to treatment status are firms for which we do not have investment readiness scores. As such, the ITT and LATE are almost identical for the firms attending the semi-finals. We therefore just report the ITT results. The first column of Table 3 presents the impact of treatment in our overall measure of investment readiness, as scored by the judges. This is our main outcome in this table, and so our main approach to multiple hypothesis testing for this set of outcomes is to rely on this aggregate. The control group has a mean investment readiness score of 2.9 (s.d. 0.9). We find that treatment increases this score by 0.284, which is significant at the 5 percent level. The magnitude is thus equivalent to 0.31 standard deviations. The second row of estimates show that this impact continues to hold after controlling for judge fixed effects, with a larger magnitude of 0.41. Figure 2 compares the distributions of investment readiness scores for the treatment and control groups, and shows there is a rightward shift in the distribution, so that these gains appear to be occurring everywhere except at the very top. The next five rows of Table 3 examine which components of the overall score have improved with treatment. We find positive impacts on all five components (team, technology, traction, market, 18   and progress), with the impacts statistically significant for three out of five measures, and significant for all five measures after controlling for judge fixed effects. The seventh row then examines the impact on the team’s presentation score. Recall this is not included as part of the overall score, but was scored separately. We find that treatment resulted in a 0.37 unit (0.32 s.d.) increase in the team’s presentational score, which is statistically significant at the 5 percent level. Treated firms are therefore more investment ready in terms of both being able to present their idea, and in terms of the quality of the idea presented. We had hypothesized that the treatment might also reduce the variability among judges in their assessment of how investment-ready firms are. To examine this, in column 8 we consider as an outcome the standard deviation of the individual judge scores for a firm, with a higher standard deviation indicating more divergence amongst judges in their assessment of the firm. However, we see a small and not statistically significant impact of treatment on this measure. Finally, the last column examines whether treated firms were more likely to be selected as one of the top 54 firms that were invited to pitch to investors in the finals. Only 12 percent of the control group firms were selected for the finals, and treatment has an 11.5 percentage point increase in this likelihood. This is a large effect, doubling the likelihood of making the finals, but it is only significant at the 10 percent level. The investment readiness scores are only available for firms which participated in the semi-finals. This raises the concern of bias arising from differential participation patterns among treatment and control firms. The last columns of Table 1 examines balance on baseline characteristics by treatment status for the firms which participated in the semi-finals. We see that, overall, the sample still looks balanced on most observable characteristics, although the overall joint orthogonality test has a p-value of 0.086. Most importantly, the mean of the baseline overall investment readiness differs only by 0.02 between the two groups, and Appendix Figure 6.1 compares the full distribution of the baseline investment readiness score by treatment group and participation status, and shows the distributions also look similar. Our pre-analysis plan specified two approaches to examining the robustness of our results to this attrition: imputing scores for those who did not attend, and using Lee (2009) bounds. Appendix 6 shows the results are robust to both approaches, and are also robust to using alternative weighting schemes to aggregate the different components 19   of the overall score. The program therefore succeeded in making firms more investment-ready, as judged by independent experts. 4.3 Heterogeneity in Impact by Initial Investment Readiness In our pre-analysis plan, we hypothesized that the impact of the program is likely to be greater for firms that were less investment-ready to begin with, since firms that already had very high scores on all components would have had little room to improve. Conversely, this impact could be negative, if training causes less investment-ready firms to present their ideas more clearly to judges, allowing judges to more easily recognize them as low quality as in Clingingsmith and Shane (2017). To test this hypothesis, we interact treatment with an indicator of whether or not the firm had a baseline investment readiness score below the median of 3 (45.1 percent of firms), and include this interaction, along with the level effect of having a below median readiness score in an expanded version of equation (1). Table 4 reports the results of examining this heterogeneity for the same outcomes as were tested in Table 3. The point estimate in column 1 is consistent with this hypothesis, with the estimated effect of treatment on investment readiness being twice as large for below median firms as above median firms. However, our power to detect this heterogeneity is low, and we cannot reject the null hypothesis that there is no difference in treatment effects by initial readiness. The next six columns show positive point estimates on four out of five of the interaction effect for the different subcomponents of the overall score, along with a positive point estimate on the interaction with the presentation score. However, the only significant effect is when looking at the technology sub- component as an outcome. Moreover, after correcting for multiple hypothesis testing using Holm’s (1979) method this impact is no longer significant. Finally, in the last column we see that the interaction is negative for being selected for the finals. This is consistent with the idea that firms that were far from investment-ready to begin with would not be able to improve enough to get into the top group, although this interaction is not statistically significant and so we cannot reject that the treatment effect on progression to the finals is the same regardless of initial score. 5. Longer-Term Impacts on Investment Readiness and Firm Performance The immediate impacts on investment readiness are seen in the performance in the semi-finals. We then track the firms over time in a variety of ways to see whether this short-term improvement 20   in investment readiness translates into longer-term investment readiness, into the chance of receiving investments, and into firm performance. 5.1 Performance in the Finals Event The Finals event was held in cooperation with the Balkan Venture Forum on December 3 and 4, 2015 in Zagreb. This was the largest venture capital conference in the five target countries to date, with more than 400 attendees. The pitching slots were spread over two days and grouped into batches based on industry segments (business and productivity, lifestyle and entertainment, life science and energy, environment, and mobility and transportation). Jury members consisting of partners at venture capital firms and managers of accelerators/incubators choose a category winner for each batch. Out of eight category winners, 6 came from the treatment group and 2 from the control. These category winners were publicly awarded with a large-format printed award and a bottle of sparkling wine following the slogan “honor, fame, and champagne”. The three lead investors of the conference had each publicly committed to choose at least one firm each to give an “invitation to negotiate” for investment by the end of the conference. They extended these invitations to four finalists in total, of which 3 were from the treatment group and 1 from the control. The treatment group therefore did better, but because the absolute number of firms winning is so low, these impacts are still small in absolute magnitude (1 to 2 percentage points), and are not statistically significant (the smallest p-value is 0.157 for being a category winner). Following the finals, a short survey was sent to investors who had attended the finals. Responses were received from 32 investors. Out of these investors, 66 percent said they had talked about a potential investment with at least one firm, 28 percent planned to negotiate with a firm, and 50 percent said they might invest and had added new firms to their watchlists. Only 40 percent had previously invested in the region, and when asked what the main barrier to investing in the region was, the modal answer was in generating deal flow and identifying good investment prospects. Investors were asked whether they planned to increase their investment in the region as a result of attending. Twenty-five percent said they would, 31 percent said they would reallocate their investment from other investments they might make towards firms in the event, and the rest would not change their investment strategies. This provides suggestive evidence that the project may have increased the overall amount of investment towards these types of firms in the region. 5.2 Impact on Media Buzz 21   We examine whether the firm is gaining attention and traction through several measures of media attention and social media buzz. One advantage of these measures is that they are available for the full sample, with no attrition. The media intelligence specialist firm Meltwater was contracted to collect online media mentions of the firms in our sample over the six-month period March 1 to August 31, 2015 (pre-intervention), and then one year and two years later (March 1 to August 31, 2016; and March 1 to August 31, 2017). Note that these time periods exclude the period of the intervention, semi-finals, and finals, so are independent of any media coverage of the program or pitch events, and correspond to an average of 6 and 18 months post-intervention. Meltwater tracks more than 250,000 global news sources in 190 countries in 25 languages (including Serbo-Croatian and Albanian). Thirteen percent of the firms in our sample had at least one media article about the firm during the six months prior to application, with a median of three articles conditional on having any media. Column 1 of Table 5 show that 9.9 percent of the control sample was mentioned at least once in the media during the six-month intervals in 2016 and 2017, and treatment results in a 4.7 percentage point (2016) and 3.9 percentage point (2017) increase. These increases are large relative to the sample mean, but not statistically significant. Column 2 of Table 5 shows that we do see a statistically significant increase in the total number of media mentions in 2017, which are three times as high for the treatment group as the control. Two-thirds of the firms had some form of social media presence at baseline, with Facebook (which 58% use) and twitter (42% use) being the most common. Column 3 shows a small and statistically insignificant impact of treatment on the number of Facebook followers a firm has, and Column 4 shows that treated firms have 20% more twitter followers after two years, but this is also not statistically significant. We pre-specified an overall index of media buzz by taking standardized z- scores of these first four columns.12 The last column of Table 5 shows that treated firms have more media buzz, with this significant at the 5 percent level in 2017. 5.3 Tracking Firm Performance through Follow-up Surveys                                                              12 Our pre-analysis plan also noted we would look at the impact on web-traffic, and being included on AngelList, a web platform for fundraising, but that these would not be included in our overall index of media buzz. We find no significant impact on these other outcomes (Appendix 7). 22   We conducted two rounds of follow-up surveys of these firms. The first, intended to measure short- term effects, was taken between April and August 2016, corresponding to a period of approximately six months after the end of the investment readiness program and judging, and enables us to measure short-term effects. The overall survey response rate was 79.2 percent, and does not differ significantly between treatment (79.9%) and control (78.5%). In addition, we collected information on operating status, number of employees, and whether negotiations for an outside investment had occurred for a further 12 percent of firms, resulting in basic data being available for 92.2 percent of firms. The second follow-up survey took place between August 2017 and March 2018, corresponding to an average of two years since the intervention. Catalini et al. (2017) show that 75 percent of firms that receive venture capital financing in the U.S. receive their first financing within the first two years after incorporation, so this timing covers a window where we should expect many firms to receive external financing if they will ever do so. The overall survey response rate for this second follow-up was 85.0 percent, and again does not differ significantly between treatment (86.2%) and control (83.7%), with data on firm operating status and receipt of equity available for 94.5% of firms. Appendix 8 shows no significant difference in response rates by treatment status, and that treatment and control firms remain balanced on baseline observable data for those responding to the survey. The follow-up surveys focused on measuring changes in the firm in three domains. The first is whether or not the firm is still operating (regardless of whether or not it has been sold to another owner). The second is investment readiness, where we focus on three aspects identified by Mason and Kwok (2009): (1) willingness and interest in taking on equity investment; (2) general investability, as measured whether there is a viable business of interest to investors in terms of employment, sales, and profits; and (3) whether the firm has put in place specific measures investors want to see before making investments, such as separation of outcomes, revenue projections, knowledge of customer acquisition costs, tracking key metrics of traction, and covering intellectual property. The third and final domain looks at steps towards receiving external funding and then external financing received. Steps towards financing include contacting outside investors, making pitches, working with mentors or experts to help obtain financing, and entering into negotiations. Receipt of external financing considers new debt and equity investments, as well as receipt of incubator and accelerator grants. 23   We ask several questions under each domain and sub-domain. Our pre-analysis plan then specifies aggregating these measures to form standardized indices. This reduces concerns about multiple hypothesis testing by focusing on one aggregate outcome in each family of questions. Appendix 3 provides the exact questions used in forming each question, and Appendix 9 provides treatment impacts on each specific question used in these aggregate measures. 5.4 Do Higher Investment Readiness Scores Predict Better Firm Investment Readiness and Investment Outcomes? The investment readiness program resulted in higher investment readiness scores from judges. To investigate whether these judges’ scores are informative about future outcomes for the firm, we use the control group sample to run the regression: ′ (2) We carry out this estimation first with no additional controls, and then with controls X for country (dummies for Serbia and for Croatia), whether or not the firm had received funding from an outside investor at baseline, and the business sector (dummies for business and productivity, and lifestyle and entertainment sectors). We estimate this separately by survey, to examine results at different time horizons. Table 6 presents the results. Column 1 shows that 10% of control firms had died by the first follow- up, and 25% by the second follow-up, two years post-intervention. These high death rates are higher than the average rates in developing countries, and likely reflect the firms being young and in relatively developed countries (McKenzie and Paffhausen, 2018). We then see no significant association between higher investment readiness scores and subsequent survival. Columns 2 through 6 then examine the associations between higher investment readiness scores and our different measures of subsequent investment readiness actions and investment steps and outcomes. We see that the judges’ scores of investment readiness are statistically significant predictors of the subsequent willingness and interest of the owner in taking on equity investment, whether the firm is meeting specific needs of investors before investment can take place, whether the firm has taken steps towards external financing, and whether they have received external financing. This is true both in the short-run (six-month) survey, and in the two-year survey. The magnitudes range from 0.14 to 0.33, suggesting that a one-unit change in the judge scores (which had a mean of 2.9 and 24   standard deviation of 0.9) would predict a 0.14 to 0.33 increase in these indices (corresponding to 0.2 to 0.5 standard deviations). The only measure where we do not find a significant association is in general investability, which is an index of measures of firm employment, profits, and sales. Finally, in the last column we examine whether the firm had made at least one deal with an outside investor since the start of the program (August 2015). 24.4 percent of the control group have made such a deal after two years. A one-mark higher investment readiness score from the judges significantly predicts a 17 to 18 percentage point increase in the likelihood of making such a deal, which is large relative to this baseline rate. We therefore have that treatment has a causal impact ( 0.28) on the investment readiness score received from judges, and that this investment readiness score in turn is a significant predictor (with coefficient ) of firm outcomes in the control group sample. Combining these two estimates allows us to obtain an estimate of the predicted treatment effect . This predicted effect is shown for each outcome in Table 6. It assumes that the only impact of the investment readiness program on firm outcomes is captured through the investment readiness score, that the association between score and outcomes observed in the control group is causal, and that the sequential ignorability assumption of Imai et al. (2011) holds.13 Although these assumptions can be questioned, we believe such an exercise is useful in providing a sense of the magnitudes we might expect to see for treatment effects, given how much our program affected investment readiness scores, and how much a change in scores in turn predicts future outcomes. We see that the predicted treatment effects are small in absolute terms: each of our index measures is predicted to increase by only 0.04 to 0.09 over two years, and the predicted increase in the likelihood of receiving outside funding is 4.6 percentage points. We compare our estimated treatment effects to these benchmarks in the next section. 5.5 Treatment Effects on Investment Readiness and Investment Outcomes Table 7 presents the treatment effects of the investment readiness program on these survey outcomes after estimating equation (1). Panel A shows the short-run impacts six months after the intervention, and panel B the impacts two years post-intervention. Treatment results in a 7.2                                                              13 The sequential ignorability assumption requires that if there are heterogeneous treatment effects, it is not the case that the firms for which treatment increases investment readiness scores are different from the firms for which an increase in investment readiness scores would increase future outcomes. 25   percentage point increase in firm survival over two years, but this is not statistically significant, with a 95 percent confidence interval of (-1.7p.p., +16.1p.p.). We see a reduction in external investment in the very short-run, which comes through less debt financing, but no significant impact on any of our investment readiness or investment outcomes over two years. After two years, the treatment group is 5 percentage points more likely to have made a deal with an outside investor, with a 95 percent confidence interval of (-4.7p.p., +14.7p.p.). The estimated point estimates on all our index measures at two years are all positive, but small, ranging from 0.003 for our external investment index, to 0.089 for general investability. These magnitudes are similar to those of our predicted treatment effects, and in all cases the predicted treatment effect lies within the 95 percent confidence interval for our estimated treatment effects. Appendix 9 shows impacts on the individual measures that make up these aggregate indices. The intervention has a large and significant (p=0.013) impact on employment after two years of 4.5 workers, which almost doubles the employment level in the control mean. Employment is often a key policy outcome by itself, and so this program would compare favorably to a number of other programs when judged on employment alone. However, if we correct for testing 25 different outcomes that make up the aggregate indices, this impact is no longer statistically significant (p=0.425). 5.6 How Should We Interpret the Lack of Treatment Effect? Our results show that the investment readiness program increased investment readiness scores from the judges, that these scores are predictive of future investment readiness and investment outcomes, but that we do not find any significant impacts of the program on these future outcomes. To reconcile these findings, we note that our estimated treatment effects are in line with the predicted treatment effects – while we increased investment readiness scores, we did not increase them by enough to register large enough changes in investment outcomes to be detectable. Our confidence intervals enable us to rule out the program having large absolute impacts on these outcomes, but are wide enough to allow for the program to have moderate sized impacts that are not possible to detect with the sample size we have. One difficulty in detecting end outcomes comes from the issue of statistical power weakening the more steps one takes along a “funnel of attribution” (McKenzie, 2018). Our experiment starts with 26   a group of firms who apply to a program. To get from this stage to receiving equity investment requires satisfying a number of steps – they must be interested in receiving investment and become investment ready, take steps towards receiving investment, and then actually receive investment. The effective number of firms drops as we pass from one step to the next, making it harder to detect impacts for end outcomes than initial outcomes. This program is the first randomized experiment of its kind, but like a number of other experiments involving larger firms, the sample size is set by external constraints in terms of the number of firms that the program attracts and caters to, rather than being a choice parameter. Given the sample size, our funding proposal calculated that we would have 80% power to detect a 0.23 increase in the investment readiness score, based on the mean and standard deviation of the baseline score measure and not accounting for the power gains from stratification. Our estimated treatment effect of 0.28 exceeds this level. In contrast, our funding proposal assumed that it would be very rare for control group firms to receive outside funding, assuming a mean of 3 percent, and then estimated a minimum detectable effect size of 8 percentage points at 80% power, not accounting for the power gains from stratified randomization (since we did not know how strongly our strata would be correlated with the end outcome). Ex post, our randomization strata have an R2 of 0.29 in a regression of making a deal with an outside investor on strata dummies. Given this, our anticipated power would have been 91 percent to detect an 8 percentage point increase. In practice, our estimated impact on receiving outside funding is 5 percentage points (similar in magnitude to the predicted impact 0.046 , which is less than this minimal detectable effect. But the larger reduction in power comes from the control mean being much higher than anticipated. While we expected very few control firms to receive external financing, in practice 24.4 percent of control firms had made a deal within two years. It is much harder to detect an 8 percentage point increase from a control mean of 24.4% than from a control mean of 3%: under our baseline assumptions, power would drop to 33.3% at this mean level. So a key reason for not being able to detect a treatment effect on external investment is that control firms found it easier to get investment than we had anticipated. 27   We explore the types of investment received in Table 8.14 Firms in the two-year long follow-up survey were asked about whether they had made deals with different types of outside investors, and if so, what type of deal. We see that the most common deals occurred with other business owners (17%), angel investors (10%), and venture capital funds (10%).15 Most of these deals were for a share of equity in the firm, with royalty deals, convertible notes, and licensing deals not very common. Both the type of investor and type of deal are similar across the treatment and control groups. We have only partial data on the amount of these deals, but know that 16.8% of firms (69% of those receiving an outside investment) received an amount of at least 25,000 euros, with this again not differing significantly by treatment status (appendix Table 9.5).16 Finally, firms in the long survey were asked what is the main challenge their business faces in their ability to grow over the next two years. Getting financing is seen as the main challenge by 41% of the control firms and 24% of the treatment firms, with this difference statistically significant (p=0.002). In addition to the positive point estimate on getting access to external financing, it is possible that the investment readiness program led some treatment firms to realize that they need to improve other areas in their business first. 6. Conclusions Investment readiness programs have been offered in a range of developing and emerging markets, based on the idea of a gap between the quality of ideas entrepreneurs have, and their readiness to attract and receive outside investment in those ideas. Despite their growing use, there has not been any rigorous study of their effectiveness. Our five-country randomized trial enables measurement of the effect of such a program. We do find that investment readiness increases, as measured by scores in a pitch competition, and that these scores are in turn predictive of future investment readiness and outcomes among firms. Nevertheless, despite finding positive point estimates, our estimates of the treatment effects of the investment readiness program on these firm investment outcomes over the next two years are not statistically significant. Our analysis suggests that this in                                                              14 Note that this table was not pre-specified, and is intended to explore the higher than anticipated rate of outside funding received by firms. 15 Our measure of receiving external financing excludes financing received from family and friends. 10.8% of firms in the long survey received financing from family and friends, but in less than half of the cases this was for an equity stake- terms tend to be less formal in such cases.  16 Using the long follow-up survey only, 43% (42% treatment, 44% control) of those receiving outside funding received at least 100,000 euros in investment. This information is not available for firms doing the short survey. 28   part reflects that the change in investment readiness score is not large enough to generate sizeable impacts on subsequent firm outcomes, and also that more of these firms are able to obtain financing without the program than was originally anticipated. We believe these results offer lessons for governments deciding whether and how to use such policies. Starting with firms that express interest in outside funding, but that require many steps to take place before being in a position to receive funding may end up including many firms for which investment readiness is not the main constraint to receiving outside funding and to firm growth. As a result, investment readiness programs that start from the demand side of outside financing may have stronger impacts on getting firms to take steps towards investment readiness, than on investment outcomes, where other constraints also play a role. A possible alternative to test would be to start from the supply side, starting with investors and asking them for a list of firms that are at the margin of being investible, but for which assistance on specific aspects of investment readiness is needed. References Bernstein, Shai, Arthur Korteweg and Kevin Laws (2017) “Attracting Early-Stage Investors: Evidence from a Randomized Field Experiment”, Journal of Finance 72(2): 509-538. Catalini, Cristian, Jorge Guzman and Scott Stern (2017) “Hidden in plain sight: venture growth with or without venture capital”, Mimeo. MIT. Clark, Colin (2008) “The impact of entrepreneurs' oral ‘pitch’ presentation skills on business angels' initial screening investment decisions”, Venture Capital, 10:3, 257-279 Clingingsmith, David and Scott Shane (2017) “Training Aspiring Entrepreneurs to Pitch Experienced Investors: Evidence from a Field Experiment”, Management Science, advance online access. Holm, Sture (1979) “A simple sequentially rejective multiple test procedure”, Scandinavian Journal of Statistics 6: 65-70. Gompers, Paul, Will Gornall, Steven Kaplan and Ilya Strebulaev (2016) “How do Venture Capitalists Make Decisions?”, Mimeo. Harvard Business School. 29   Gonzalez-Uribe, Juanita, and Michael Leatherbee (2018) " The Effects of Business Accelerators on Venture Performance: Evidence from Start-Up Chile", Review of Financial Studies 31(4): 1566-1603. Hallen, Benjamin, Christopher Bingham, and Susan Cohen (2014) “Do Accelerators Accelerate? A Study of Venture Accelerators as a Path to Success?”, Academy of Management Proceedings 12955. Howell, Sabrina (2017) “Financing Innovation: Evidence from R&D Grants”, American Economic Review 107(4): 1136-64. Howell, Sabrina (2018) “Learning from Feedback: Evidence from New Ventures”, Mimeo. NYU. Imai, Kosuke, Luke Keele, Dustin Tingley, and Teppei Yamamoto (2011) “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review 105 (4): 765–89. Kaplan, Steven and Per Strömberg (2004) “Characteristics, Contracts, and Actions: Evidence from Venture Capitalist Analyses”, Journal of Finance 59(5): 2177-2210. Karajkov, Risto (2009) Bridging the Gap: Access to Risk Finance for Early-Stage SMEs in Southeast Europe. http://www.incubat.ro/uploads/Publicatii_Investment_SMS_report.pdf Kerr, William R., Josh Lerner, and Antoinette Schoar (2014) "The consequences of entrepreneurial finance: Evidence from angel financings." Review of Financial Studies 27.1 (2014): 20-55. Lee, David (2009) “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects", Review of Economic Studies 76(3): 1071-1102. Mason, Colin and Richard Harrison (2001) “Investment Readiness': A Critique of Government Proposals to Increase the Demand for Venture Capital”, Regional Studies, 35(7): 663-668 Mason, Colin and Richard Harrison (2003) “Auditioning for money: what do technology investors look for at the initial screening stage?” Journal of Private Equity 6(2): 29-42. Mason, Colin and Jennifer Kwok (2010) “Investment Readiness Programmes and Access to Finance: A Critical Review of Design Issues”, Local Economy 25(4): 269-92. 30   McKenzie, David (2018) “Statistical Power and the Funnel of Attribution”, Development Impact blog, January 8 https://blogs.worldbank.org/impactevaluations/statistical-power-and-funnel- attribution. McKenzie, David and Anna Luisa Paffhausen (2018) “Small Firm Death in Developing Countries”, World Bank Policy Research Working Paper no. 8236. McKenzie, David and Christopher Woodruff (2014) “What are we learning from business training evaluations around the developing world?”, World Bank Research Observer, 29(1): 48-82 Smith, Sheryl Winston, and TJ John Hannigan (2015) “Swinging for the fences: How do top accelerators impact the trajectories of new ventures?”, http://druid8.sit.aau.dk/druid/acc_papers/5ntuo6s1r5dvrpf032x24x5on5lq.pdf Wagner, Rodrigo (2016) “Does Feedback to Business-Plans Impact New Ventures? Evidence from a Field Experiment”, Available at SSRN: http://ssrn.com/abstract=2766566 Yu, Sandy (2016) “How do Accelerators Impact High-Technology Ventures?”, Mimeo. UC Berkeley 31   Figure 1a: Baseline Distributions of Investment Readiness for Treatment and Control Groups  Baseline Score Distributions by Treatment Status .5 .4 Proportion of Firms .2 .3 .1 0 1 2 3 4 5 Baseline investment readiness score Treatment Control   Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.988  Figure 1b: Baseline Distributions of Investment Readiness for Those Attending Semi‐Finals  Baseline Score Distributions for Those Attending Semi-finals .6 Proportion of Firms .2 0 .4 1 2 3 4 5 Baseline investment readiness score Treatment Control   Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.959  32   Figure 2: Distribution of Investment Readiness Scores after Program as Scored by Judges  Post-treatment Investment Readiness Distributions by Treatment Status .4 .3 .2 .1 0 1 2 3 4 5 Semifinals investment readiness score Treatment Control   Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.017 33   Table 1: Balance Test on Application Data Full Sample Semi‐Final Participants Treatment Control P‐value Treatment Control P‐value Variables stratified on Incorporated/Registered in Croatia 0.25 0.24 0.612 0.25 0.30 0.920 Incorporated/Registered in Serbia 0.46 0.46 0.626 0.48 0.48 0.513 Baseline Readiness Score 2.95 2.92 0.150 2.99 2.97 0.476 Has an outside private investor 0.10 0.09 0.178 0.14 0.06 0.170 Other variables Market attractiveness score 3.08 3.05 0.851 3.13 3.18 0.579 Product technology score 2.47 2.43 0.835 2.56 2.71 0.085 Traction score 3.34 3.27 0.507 3.28 3.06 0.382 Team score 3.04 3.05 0.878 3.08 3.02 0.207 Sector is business and productivity 0.48 0.39 0.107 0.45 0.36 0.436 Sector is lifestyle and entertainment 0.18 0.23 0.295 0.20 0.27 0.215 Uses Cloud Technology 0.20 0.26 0.231 0.20 0.21 0.984 Uses Big Data 0.18 0.21 0.642 0.17 0.20 0.915 Place in value chain is developer 0.61 0.55 0.171 0.60 0.57 0.677 Place in value chain is service provider 0.59 0.54 0.372 0.60 0.54 0.108 Age of firm (years) 2.61 2.66 0.887 2.24 2.29 0.346 Early stage firm 0.30 0.33 0.475 0.35 0.37 0.554 Revenues in 2014 178073 184760 0.959 37642 144012 0.303 Number of employees 6.47 5.88 0.539 4.65 5.32 0.800 Age of main founder 38.22 36.81 0.204 38.02 36.67 0.362 Main founder has post‐graduate education 0.49 0.48 0.816 0.54 0.55 0.740 At least one founder is female 0.16 0.22 0.128 0.16 0.30 0.071 Company has a global focus 0.60 0.58 0.576 0.59 0.63 0.569 Have accepted outside financing 0.34 0.37 0.656 0.42 0.40 0.836 Previouslyin mentoring/accelerator program  0.15 0.16 0.704 0.18 0.22 0.202 Sample Size 174 172 110 101 Joint test of orthogonality of treatment p‐value 0.621 0.086   Notes: Full sample denotes the full experimental sample. Semi‐final participants are the sample that were  scored  by  judges  during  the  semi‐final  pitch  event.  Variables  stratified  on  were  the  variables  used  in  randomized assignment.     34   Table 2: Correlates of the Take‐up Decision among the Treatment Group Take‐up WhatAventure Attend Masterclasses Full Model Stepwise Model Full Model Stepwise Model Baseline Investment Readiness Score 0.494** 0.573*** ‐0.217 ‐0.238* (0.194) (0.169) (0.144) (0.132) Incorporated/Registered in Croatia 0.424 ‐0.757** ‐0.771*** (0.397) (0.296) (0.286) Incorporated/Registered in Serbia 0.409 ‐0.331 ‐0.393 (0.354) (0.261) (0.245) Sector is business and productivity 0.128 0.040 (0.254) (0.262) Sector is lifestyle and entertainment 0.191 0.010 (0.366) (0.327) Uses Cloud Technology 0.137 0.577** 0.610** (0.372) (0.275) (0.259) Uses Big Data 0.740 0.278 (0.521) (0.266) Place in value chain is developer ‐0.181 ‐0.360* ‐0.309 (0.292) (0.215) (0.207) Place in value chain is service provider ‐0.211 ‐0.108 (0.294) (0.235) Age of firm (years) ‐0.005 ‐0.026 (0.056) (0.044) Early stage firm ‐0.147 0.024 (0.335) (0.238) Number of employees 0.008 ‐0.005 (0.013) (0.010) Age of main founder ‐0.020 ‐0.018 ‐0.008 (0.014) (0.013) (0.012) Main founder has post‐graduate education ‐0.141 0.487** 0.485** (0.278) (0.221) (0.212) At least one founder is female ‐0.674** ‐0.455 ‐0.001 (0.302) (0.337) (0.270) Company has a global focus 0.280 0.073 (0.305) (0.225) Have accepted outside financing 0.211 0.210 (0.324) (0.252) Has an outside private investor ‐0.123 ‐0.121 (0.581) (0.373) Have participated in mentoring/accelerator program before 0.558 0.399 0.488* (0.579) (0.308) (0.276) Sample Size 174 174 174 174 Notes: Robust standard errors in parentheses. Coefficients are marginal effects from probit estimation. *, **, *** indicate significance at the 10, 5, and 1 percent levels respectively 90.1% of firms took up WhatAVenture, and 43.7% attended at least one masterclass.     35   Table 3: Impact of Program on Investment Readiness as Scored by Judges Overall Std Dev Selected Readiness Team Technology Traction Market Progress Presentation of Judge to go to Score Score Score Score Score Score Score Scores Finals Base Specification Assigned to Treatment 0.284** 0.167 0.372** 0.206 0.268* 0.373*** 0.372** 0.006 0.115* (0.126) (0.150) (0.152) (0.130) (0.137) (0.137) (0.164) (0.049) (0.068) Including Judge Fixed Effects Assigned to Treatment 0.409*** 0.369** 0.476*** 0.295** 0.463*** 0.440*** 0.514*** ‐0.017 0.090 (0.135) (0.158) (0.174) (0.142) (0.139) (0.143) (0.191) (0.051) (0.076) Sample Size 211 211 211 211 211 211 211 211 211 Control Mean 2.908 3.042 2.970 2.541 3.406 2.794 3.042 0.723 0.122 Control Std. Dev. 0.903 1.068 1.031 0.947 0.940 0.937 1.145 0.317 0.328 Notes:  Robust standard errors in parentheses. Regressions control for randomization strata. *, **, *** indicate significance at the 10, 5, and 1 percent levels respectively. Judge fixed effects controls for which five of the sixty‐five judges judged a particular firm.     36   Table 4: Heterogeneity in Impacts on Investment Readiness Overall Std Dev Selected Readiness Team Technology Traction Market Progress Presentation of Judge to go to Score Score Score Score Score Score Score Scores Finals Assigned to Treatment 0.203 0.014 0.405** 0.138 0.009 0.314 0.249 0.020 0.177* (0.178) (0.208) (0.193) (0.192) (0.180) (0.197) (0.230) (0.062) (0.101) Assigned to Treatment*Baseline Readiness below Median 0.210 0.378 ‐0.083 0.183 0.646** 0.169 0.310 ‐0.019 ‐0.179 (0.254) (0.305) (0.317) (0.251) (0.275) (0.270) (0.335) (0.105) (0.127) Sample Size 211 211 211 211 211 211 211 211 211 Control Mean 2.908 3.042 2.970 2.541 3.406 2.794 3.042 0.723 0.122 Control Std. Dev 0.903 1.068 1.031 0.947 0.940 0.937 1.145 0.317 0.328 Notes:  Robust standard errors in parentheses. Regressions control for randomization strata. *, **, *** indicate significance at the 10, 5, and 1 percent levels respectively. Regressions also control for level effect of having a baseline investment readiness score below the median of 3.   37   Table 5: Impacts on Media Mentions and Social Media Buzz Any media Number of  # Facebook # Twitter Media Buzz   mention Media mentions likes Followers Index Panel A: Impact at Six Months Assigned to Treatment 0.047 0.786 ‐38.0 15.110 0.085 (0.031) (0.483) (145) (18.495) (0.053) Sample Size 346 346 346 346 346 Control Mean 0.099 0.663 1119 112.471 ‐0.060 stddev 0.299 3.410 2388 260.201 0.546 Panel B: Impact at Eighteen months Assigned to Treatment 0.039 0.736** 0.889 22.106 0.112** (0.030) (0.291) (218) (18.974) (0.047) Sample Size 346 346 346 346 346 Control Mean 0.099 0.320 1430 106.866 ‐0.073 Control S.D. 0.299 1.566 3106 249.504 0.528 Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels. All regressions control for randomization strata fixed effects and for baseline values of outcome of interest. Any media mention denotes firm was mentioned in news media in 6 month window, number of media mentions  is the number of times the firm was mentioned, winsorized at the 99th percentile. # Facebook likes and # Twitter  Followers are the number of Facebook likes for the firm's Facebook page, and number of Twitter followers for the  firm, both winsorized at the 95th percentile. Media Buzz Index is an index of standardized z‐scores of these first  four columns.   38   Table 6: Judges Scores Predict Firm Outcomes 6 months and 2 years after program Firm Interested General Specific needs Investment External Made a deal survival in equity Investability of investors Steps investment with investor Panel A: Association at Six Months without controls 0.024 0.201** 0.076 0.336*** 0.222*** 0.213** 0.093** Score assessed by Judges (0.037) (0.076) (0.072) (0.065) (0.082) (0.098) (0.038) with controls for country, prior funding, and sector Score assessed by Judges 0.017 0.168* 0.052 0.300*** 0.179** 0.187* 0.085** (0.037) (0.087) (0.074) (0.069) (0.087) (0.110) (0.039) Sample Size 92 83 83 81 73 82 82 Control Mean 0.898 ‐0.015 ‐0.039 ‐0.059 0.008 0.084 0.083 Control S.D. 0.303 0.764 0.634 0.682 0.720 0.741 0.276 Predicted Treatment Effect 0.007 0.056 0.021 0.094 0.062 0.060 0.026 Panel B: Association at Two Years without controls 0.061 0.153* 0.040 0.136* 0.322*** 0.322*** 0.166*** Score assessed by Judges (0.041) (0.088) (0.073) (0.082) (0.100) (0.072) (0.048) with controls for country, prior funding, and sector Score assessed by Judges 0.053 0.128 0.027 0.140* 0.324*** 0.339*** 0.175*** (0.044) (0.094) (0.078) (0.077) (0.099) (0.077) (0.049) Sample Size 100 92 86 88 80 99 99 Control Mean 0.753 ‐0.005 ‐0.058 ‐0.059 ‐0.032 0.018 0.244 Control S.D. 0.433 0.783 0.650 0.692 0.760 0.698 0.431 Predicted Treatment Effect 0.017 0.044 0.011 0.038 0.090 0.090 0.046 Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively. Firm survival is a binary variable that takes value one if the firm is operating, and zero otherwise. Interested in equity is a standardized index of whether the firm is interested in equity financing, the maximum equity share they are willing to have owned by outside investors, whether they have specific deal terms for investors, and whether they would consider a royalty‐ based investment. General investability is a standardized index of number of employees, whether the founders work full‐ time in the business, whether the firm had positive sales in the first quarter of the year, whether total sales exceed 10,000  euros in that quarter, whether the firm made a positive profit in the past year, and whether the firm made sales to Western  Europe or the United States. Specific needs of investors is a standardized index of whether business and personal accounts are  separated, whether the firm has made a revenue projection for the next year, whether it knows customer acquisition costs,  the number of key metrics tracked, whether it has found out if the product or service can be covered by intellectual property  protection, and whether it has at least one form of intellectual property protection received or pending. Investment steps is a  standardized index of having contacted at outside investor, made a pitch to an outside investor, have a mentor or external  expert supporting them to obtain financing, and entered into negotiations with an outside investor. External investment is a  standardized index of having taken on new debt, having made a deal with an outside investor, have received at least 25,000  euros in outside financing, and have received an incubator or accelerator grant (all since August 2015). Made a deal with an  investor indicates having made a deal with an outside investor since August 2015 (program start).           39   Table 7: Impacts on Survey Outcomes 6 months and 2 years after program Firm Interested General Specific needs Investment External Made a deal survival in equity Investability of investors Steps investment with investor Panel A: Impact at Six Months Assigned to Treatment 0.049 0.051 0.026 0.082 ‐0.017 ‐0.152* ‐0.024 (0.030) (0.094) (0.085) (0.080) (0.098) (0.087) (0.033) Sample Size 319 278 277 269 240 279 279 Control Mean 0.898 ‐0.015 ‐0.039 ‐0.059 0.008 0.084 0.083 Control S.D. 0.303 0.764 0.634 0.682 0.720 0.741 0.276 Predicted Treatment effect 0.007 0.056 0.021 0.094 0.062 0.060 0.026 Panel B: Impact at Two Years Assigned to Treatment 0.072 0.032 0.089 0.084 0.044 0.003 0.050 (0.045) (0.084) (0.082) (0.079) (0.092) (0.080) (0.049) Sample Size 340 309 291 298 282 330 330 Control Mean 0.753 ‐0.005 ‐0.058 ‐0.059 ‐0.032 0.018 0.244 Control S.D. 0.433 0.783 0.650 0.692 0.760 0.698 0.431 Predicted Treatment effect 0.017 0.044 0.011 0.038 0.090 0.090 0.046 Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively. All regressions control for randomization strata fixed effects. Firm survival is a binary variable that takes value one if the firm is operating, and zero otherwise. Interested in equity is a standardized index of whether the firm is interested in equity financing, the maximum equity share they are willing to have owned by outside investors, whether they have specific deal terms for investors, and whether they would consider a royalty‐ based investment. General investability is a standardized index of number of employees, whether the founders work full‐ time in the business, whether the firm had positive sales in the first quarter of the year, whether total sales exceed 10,000  euros in that quarter, whether the firm made a positive profit in the past year, and whether the firm made sales to Western  Europe or the United States. Specific needs of investors is a standardized index of whether business and personal accounts are  separated, whether the firm has made a revenue projection for the next year, whether it knows customer acquisition costs,  the number of key metrics tracked, whether it has found out if the product or service can be covered by intellectual property  protection, and whether it has at least one form of intellectual property protection received or pending. Investment steps is a  standardized index of having contacted at outside investor, made a pitch to an outside investor, have a mentor or external  expert supporting them to obtain financing, and entered into negotiations with an outside investor. External investment is a  standardized index of having taken on new debt, having made a deal with an outside investor, have received at least 25,000  euros in outside financing, and have received an incubator or accelerator grant (all since August 2015). Made a deal with an  investor indicates having made a deal with an outside investor since August 2015 (program start). Predicted Treatment effect  is the treatment effect predicted from association in the control group between the judges score  and the outcome, multiplied by the treatment effect of the program on the judges score.       40   Table 8: Details on Types of External Financing Deals Made Sample Treatment Control Size Group Group P‐value Who was deal made with? Sold Firm 221 0.017 0.038 0.547 Deal with Other Business Owner 221 0.165 0.179 0.385 Deal with Angel Investor 221 0.096 0.104 0.804 Deal with Crowdfunding 221 0.043 0.019 0.164 Deal with Accelerator 221 0.087 0.038 0.096 Deal with VC Fund 221 0.096 0.113 0.967 Deal with Government Fund 221 0.070 0.075 0.706 What type of deal was made? Equity‐Share 221 0.209 0.236 0.765 Licensing Deal 221 0.043 0.009 0.113 Royalty Deal 221 0.035 0.057 0.737 Convertible Note Deal 221 0.026 0.019 0.868 Other Deal 221 0.043 0.057 0.826 Says financing is main challenge 204 0.239 0.411 0.002 Notes: Data are for firms that answered the full survey in the second follow‐up, approximately two years post‐intervention. Some firms made multiple deals, and so numbers given are proportion of firms which made at least one of this deal type. Says financing is main challenge is an indicator of whether financing is viewed as the main challenge the business faces in its ability to grow.     41   ONLINE APPENDICES   Appendix 1: Examples of Investment Readiness Programs Around the World Appendix 2: Timeline Appendix 3: Scoring Methodology and Variable Definitions Appendix 4: Additional Details on Treatment Intervention Appendix 5: Additional Details on Control Intervention Appendix 6: Additional Details on the Semi-Finals and Finals Appendix 7: Impact on web traffic and being included on AngelList Appendix 8: Follow-up survey completion rates and balance Appendix 9: Treatment effects on individual survey outcomes 42   Appendix 1: Examples of Investment Readiness Programs around the World  The  text  provides  some  examples  of  investment  readiness  programs  offered  in  other  countries.  We  provide more discussion of these examples here, with Mason and Harrison (2001) and Mason and Kwok  (2009) also providing reviews of some programs.  Australia:  The  “Impact  Investment  Readiness  Fund”  offers  grants  of  up  to  $100,000  for  enterprises  to  purchase  specialized  capacity  building  support  from  providers  such  as  advisory,  financial,  intermediary  or  legal  services. The program aims to bridge the gap in the Australian market that exists between mission‐driven  organizations in need of funding and investors actively seeking impact investment opportunities.17  United Kingdom  In the UK, there are different types of investment readiness programs. Some of them discriminate in favor  of entrepreneurs that have a social mission, while others focus on all types of firms. Within the first group,  the “Investment and Contract Readiness Fund”18, supported by the Office for Civil Society, assists social  ventures to build their capacity to be able to raise capital. The “Impact HUB Westminster” also offers an  investment readiness program, the “Impact Investment Readiness”, which aims to accelerate investment  into social and environmental businesses based in London. It helps entrepreneurs to learn which type of  investment is right for them, discover how to write investable business plans, and articulate their business  mission  as  an  attractive  impact  investment.  They  usually  offer  two  days  of  free  in‐depth  content  on  relevant topics, led by experts and supported by peer‐to‐peer learning.19   Within the second group, the “Growth Accelerator”20 provides investment readiness services that help  the entrepreneur understand which type of finance is right for her/him, build strong business plan and  investment pitch, ensure the financial information provided to potential investors is credible and robust,  pitch to the right type of investor for the entrepreneur’s business, connect with a wide range of finance  institutions  and  investors  across  the  country  and  secure  finance.  Another  example  in  the  UK  is  the  investment readiness program provided by the “Angel Capital Group”, which focuses specifically around  three key dimensions: (i) positioning in the market, (ii) developing attractiveness to the investor, and (iii)  pitching  the  message  and  opportunity  correctly.  From  its  headquarters  in  central  London,  the  Angel  Capital  Group  works  both  nationally  and  internationally,  providing  access  to  leading‐edge  services  designed  to  improve  investment  readiness,  facilitate  access  to  early  stage  investment,  and  create  opportunities for the development of new early stage co‐investment funds, with a key focus on the angel  investment market.21 The “Greater London Enterprise”22 is also another provider of investment readiness  services, usually through a combination of an e‐learning model and legal and financial advisors, who are  also investors.   Mason  and  Kwok  (2010)  also  provide  details  on  several  other  programs  in  the  U.K.  These  include  the  different variants of investment readiness programs tried by the U.K. Small Business Service’s Investment  Readiness  Demonstration  Project,  the  University  of  Warwick’s  Science  Park’s  Investment  Readiness                                                               17  http://impactinvestingaustralia.com/iirf/.  18  http://www.sibgroup.org.uk/investment‐readiness/.  19  https://westminster.impacthub.net/impact‐investment‐ready/  20  http://www.ga.businessgrowthservice.greatbusiness.gov.uk/what‐we‐offer/access‐to‐finance/  21  http://www.angelcapital.co.uk/.  22  https://www.gle.co.uk/gle‐business‐support.html  43   program; and the Finance and Business program delivered in the North East of England by the North East  Regional Development Agency.  Europe  In  Ireland,  “Enterprise  Ireland”  offers  investment  readiness  support  to  entrepreneurs  by  giving  them  access to the network “Enterprise Ireland Advisers” and allowing them to get specialist support in a range  of  key  strategic  business  development  functions,  including  equity  raising,  technology  development,  market research, and export sales. The “Invest Academy Programme”, is an investment readiness program  sponsored  by  the  European  Business  Angel  Network  (EBAN),  Sun&Sup,  and  Eurada  geared  to  train  entrepreneurs  to  understand  sources  of  financing  for  their  company  by  building  their  knowledge  of  financial sources, and helping them to refine their business propositions and business plans to make them  attractive  to  potential  investors  and/or  lenders.  “InvestHorizon”  is  a  program  designed  to  increase  investments made in Innovative European SMEs through Investment Readiness development and investor  sensitization.    The  program  assists  companies  getting  started,  raising  awareness  amongst  SMEs  about  investment sources, options and requirements, providing coaching services to get funded, and matching  entrepreneurs with specialized and active investors through investment forum events.  A  European  Union  financed  project  led  to  the  Ready  for  Equity23  program  which  now  offers  training  programs for fund‐seeking entrepreneurs throughout Europe, with an 8‐module course that includes an  introduction  to  equity,  discussion  of  the  investment  process,  team  building,  how  to  do  the  perfect  presentation, and how to value the enterprise and manage exit.  United States  In the U.S, there are also several initiatives to foster investment readiness. For example, the “Lean  Startup” methodology developed by Steve Blank24  offers entrepreneurs a framework to focus on what’s  important  to  be  ready.   Teams  use  the  Lean  Startup  toolkit:  the  Business  Model  Canvas  +  Customer  Development process + Agile Engineering to prepare themselves and be ready to present their business  propositions to potential investors. These three tools allow start‐ups to focus on the parts of an early‐ stage  venture  that  matter  the  most  for  investors:  the  product,  market  fit,  customer  acquisition/base,  revenue and cost models, channels and partners. The “Larta Institute” also offers investment readiness  services.25 By working side‐by‐side with entrepreneurs to identify and address their unique challenges and  opportunities, they help them to be ready to raise equity finance. The Larta Institute gives entrepreneurs  access to top‐notch specialists in financial planning and mentors that support the entrepreneur in building  a  credible  and  attractive  business  plan.  They  have  also  worked  with  NSF  grantees  to  help  them  commercialize their ideas.                                                                         23  http://www.readyforequity.eu/article/2010/start_page/  24  http://steveblank.com/about/.  25  http://www.larta.org/services/entrepreneurs  44   Appendix 2: Timeline   Aug 14, 2015: Applications launched   August 2015: Roadshows, advertising  Sept 6, 2015: Applications closed  Sept 10, 2015: Random assignment done by computer   Oct 2, 2015: Registration in AEA RCT registry  Sept 10‐Nov 13, 2015: Investment Readiness program implemented, master classes, mentoring, etc.  November 12‐14, 2015: Semi‐finals and pitch event in Belgrade   December 2‐4, 2015: Finals with the top 54 firms from the semi‐finals pitching in front of the investors VC  fund managers and Business Angels.  April‐August 2016: First follow‐up survey (approximately 6 months post‐program)  August 2017‐March 2018: Second follow‐up survey (approximately 2 years post‐program).    Appendix 3: Scoring and Data Appendix  The key variables are measured and defined as follows:  Baseline Investment Readiness  The applications were scored by a team from Pioneers Ventures, a seed‐stage venture capital investment  unit.  Two  professional  investment  managers  reviewed  each  eligible  application  independently  and  assigned a score, based on for sub‐scores using an agreed scoring metric as detailed below in Appendix  Table 3.1. Where the independent scores differed by more than one unit, they discussed the cases to  arrive at a consensus score, otherwise the scores were averaged. Each business was scored on four sub‐ components as follows:  Appendix Table 3.1: Description of the Investment Readiness Scoring Scale  Category  Weight  Points  Threshold description  Market attractiveness    10%  1  Market does not exist/ no market need      2  Small market well served by competitors or equally  good substitutes      3  Large market well served by competitors or equally  good substitutes      4  Attractive niche in small market with unique solution/  positioning      5  Attractive niche in large market with unique solution/  positioning      6  Very large and mostly untapped/ underserved market  with right offering          45   Co‐founder(s) and team  20%  1  Single founder, no team      2  Team of 2+ people      3  Complimentary team with little experience      4  Complimentary team with significant experience      5  Serial entrepreneur(s)      6  Serial entrepreneur(s) with exit            Product/ technology  30%  1  No/ low innovation ‐ Imitation of existing products or  services      2  Low innovation ‐ Localization of proven business  models from abroad      3  Some innovation ‐ Incremental improvements of  existing products or services      4  Innovative new solutions or business models that  address customer needs      5  Competitive technological innovation/ advantage      6  Patented/ patent‐pending technological innovation   or otherwise protected IP          Traction  40%  1  No traction      2  Soft traction (press coverage, facebook likes etc.)      3  Test users/ prototype testing      4  Non‐financial KPIs (e.g. downloads, pre‐orders)      5  Generating revenues      6  Sustainable business (generated revenues in 2014 >  GPD/capita for each founder)  The  baseline investment readiness score was then calculated as a weighted average of these four sub‐ components, using the weights detailed above.  Semi‐Finals Scores Provided by Judges  Judges scored each of the following on a six‐point scale, with the score being the simple average of the  scores of each of the five judges scoring the pitch:  1. Team: a score for the skills and capabilities of the entrepreneur and team  2. Technology: a score for the degree of innovativeness and technological advancement  3. Traction: a score for indications of measureable market success  4. Market: a score for commercial market attractiveness  5. Progress: a score for recent business development progress (in the last 3 months)  6. Presentation: a score for the presentation performance.  The following two variables were then calculated:  Overall readiness score:  this is calculated as a weighted average of the team (28% weight), technology  (21% weight), traction (14% weight), market (7% weight), and progress (30% weight) scores.  46   Std dev of judge scores:  the overall readiness score is calculated for each judge. We then calculate the  standard  deviation  of  the  five  judge  scores  for  a  firm  to  get  this  measure  of  how  much  disagreement  amongst judges there was in the scoring.  Finally, we also construct a dummy variable  Selected to go to Finals  to denote whether or not the firm  was selected by virtue of having a top overall score or by direct nomination to go through to the Finals  event.  Media Mentions and Social Media Buzz  Any media mention is a dummy variable that takes value one if the firm is mentioned in any of the over  250,000 global news sources covered by Meltwater during the six‐month period March 1 to August 31.  This is measured for 2016 in panel A of Table 5, and for 2017 in panel B.  Number of media mentions: the number of times the firm is mentioned in any of the global news sources  covered by Meltwater during the six‐month period March 1 to August 31. This is winsorized at the 99th  percentile to reduce the influence of outliers.  # Facebook likes:  the number of likes for the firm’s Facebook page, measured approximately 6 months  and 18 months post intervention. This is recorded as zero for firms without Facebook pages (including  firms that have closed down), and is winsorized at the 99th percentile.  # Twitter followers:  the number of followers the firm’s twitter account has, measured approximately 6  months  and  18  months  post  intervention.  This  is  recorded  as  zero  for  firms  without  twitter  profiles  (including firms that have closed down), and is winsorized at the 99th percentile.  Media buzz index: Standardized z‐scores of each of the above four variables are obtained by subtracting  their mean and dividing by their standard deviation (separately by time period). The media buzz index is  then  the  mean  of  the  standardized  z‐scores  for  any  media  mention,  number  of  media  mentions,  #  facebook likes, and # twitter followers.  Survey oOtcomes  Firm survival: this is a dummy variable coded as one if the firm is still operating (regardless of whether or  not it has the original owners), and 0 otherwise.   Interested in equity: this is an average of standardized z‐scores from the following variables:   Interested  in  equity  financing  for  the  business:  a  dummy  variable  which  takes  value  one  if  the  owners say they are interested in receiving new equity financing for the business.    Maximum equity share willing to have held by outside investors: this variable ranges from 0 to  100, and is the percent of equity the firm owner reports being willing to have held by an outside  investor. It is coded as 100 for individuals who have sold their whole firm, and as the share of  equity currently held by investors for those who are not interested in receiving new equity.   Have specific deal terms of offer outside investors: this is a dummy variable, coded as one if the  firm owner reports having specific deal terms (e.g. a draft term sheet) to offer outside investors,  and zero otherwise. It is coded as zero for firms that have closed.   Would consider a royalty‐based investment:  a dummy variable, coded as one if the firm owner  reports willingness to consider a royalty‐based investment, and zero otherwise. It is coded as zero  for firms that have closed.  47   General investability: this is an average of standardized z‐scores of the following variables:   Number of employees in the company: the number of employees in the company, coded as zero  for firms that are closed, and winsorized at the 99th percentile.   Founder/co‐founders work full‐time in the company:  a dummy variable that takes value one if at  least one of the founders works full‐time in the company, and zero otherwise.   Positive total sales for first quarter: this is a dummy variable which takes value one if the firm  made positive sales in the first quarter of 2016 (first follow‐up survey), or in the first quarter of  2017 (second follow‐up survey), and zero otherwise. It is coded as zero for firms that have closed.   Total sales for first quarter of at least 10,000 euros:  a dummy variable which takes value one if  the firm made sales of at least 10,000 euros in the first quarter of 2016 (first follow‐up survey), or  in the first quarter of 2017 (second follow‐up survey), and zero otherwise. It is coded as zero for  firms that have closed.   Business made  positive  profit in last year: a  dummy variable which  takes value one if  the firm  made a positive profit in 2015 (first follow‐up survey) or in 2016 (second follow‐up survey), and  zero otherwise. It is coded as zero for firms that are closed.   Sales made in Western Europe or U.S.: a dummy variable which takes value one if the firm makes  sales in European Union countries (excluding Croatia and Slovenia) or in the United States, and  zero otherwise. It is coded as zero for firms that are closed.  Meeting  the  specific  needs  of  investors:  this  is  an  average  of  standardized  z‐scores  of  the  following  variables:   Accounts of the business are separated from those of the owners:  a dummy variable that takes  value  one  if  the  business  accounts  are  kept  separately  from  those  of  the  owner,  and  zero  otherwise. It is coded as zero for closed firms.   Revenue projection made for the next 12 months:   a dummy variable that takes value one if the  firm has in place a revenue projection for the next 12 months, and zero otherwise. It is coded as  zero for closed firms.   Business  knows  customer  acquisition  costs:  a  dummy  variable  that  takes  value  one  if  the  firm  knows the cost of acquiring a customer, and zero otherwise. It is coded as zero for closed firms.   Number of key metrics (out of 11) being tracked: the number of key metrics being tracked such as  newsletter sign‐ups, pre‐orders, free user downloads, requests for samples or free trials, free pilot  projects  with  customers,  current  active  users,  new  sales  leads  per  month,  sales  meetings  per  month,  paid  pilot  projects  with  customers,  paid  customer  sign‐ups  or  paid  downloads,  and  customer life‐time value. This is coded as zero for closed firms.   Found out whether product or service can be covered by intellectual property protection: a dummy  variable that takes value one if the firm has found out whether their product or service can be  covered by some form of intellectual property protection, and zero otherwise. This is coded as  zero for closed firms.   Has at least one form of intellectual property protection or application pending: A dummy variable  that takes value one if the firm has, or has pending, a copyright, trademark, industrial design right,  patent, or other form of IP protection, and zero otherwise. This is coded as zero for closed firms.  Investment Steps: this is an average of standardized z‐scores of the following variables:  48    Has contacted an outside investor to see if they are interested in making an investment: A dummy  variable taking the value one if, in the last year, the firm has contacted an outside investor to see  if they are interested in making an investment, and zero otherwise. Firms that say they are not  interested in investment and that do not answer this question are assumed to have not contacted  an investor. Coded as zero for closed firms.   Has made a pitch to outside investors outside of our program: A dummy variable taking the value  one if, in the past year, the firm made a pitch to outside investors at an event. Firms were explicitly  asked to exclude pitches made during the semi‐finals and finals of the Pioneers program. Firms  that say they are not interested in outside investment are assumed not to have made a pitch.  It  is coded as zero otherwise, including if the firm is closed.   Have a mentor or external expert supporting them to obtain external financing: a dummy variable  that takes the value one if the firm has a mentor or external expert helping them to raise funding,  and is zero otherwise, including if the firm is closed.   Entered into negotiations with outside investor since August 2015: a dummy variable which takes  the value one if the firm has entered into negotiations with any outside investor since August  2015, and zero otherwise. It is coded as zero if the firm is closed. Firms which have been sold, or  which  have  received  outside  equity  investments,  and  which  did  not  answer  this  question,  are  assumed to have entered into negotiations.  External investment: this is an average of standardized z‐scores of the following variables26:   Taken on new debt since August 2015:  a dummy variable which takes value one if the firm has  taken  on  new  debt  since  August  2015,  and  zero  otherwise.  It  is  assumed  to  be  zero  for  firms  closed.   Have made a deal with an outside investor since August 2015: a dummy variable which takes value  one if the firm has made a deal with an outside investor (who is not family or friends) since August  2015, and zero otherwise. This takes value one if the firm has been sold, and zero if the firm has  closed before being sold.    Received at least 25,000 euros in new outside investment since August 2015:   a dummy variable  that takes value one if the firm has received at least 25,000 euros in outside investment since  August 2015, and zero otherwise. It is set at zero for firms that have closed and not been sold for  more than 25,000 euros.   Received incubator/accelerator grant since August 2015:  a dummy variable that takes the value  one if the firm has received a grant from an incubator or accelerator since August 2015, and zero  otherwise.  Have made a deal with an outside investor since August 2015: a dummy variable which takes value one if  the firm has made a deal with an outside investor (who is not family or friends) since August 2015, and  zero otherwise. This takes value one if the firm has been sold, and zero if the firm has closed before being  sold. Note that this is also considered  as part of the external investment index, but  given its role as a                                                               26   Our  pre‐analysis  plan  also  originally  added  a  fifth  variable  to  this  index:  total  amount  of  outside  investment  received. However, after our first follow‐up survey found firms were very reluctant to specify the exact amount of  funding received, this question was dropped from the second follow‐up survey, and so is not included in the overall  index.  49   summary  statistic  of  whether  investment  readiness  leads  to  new  investment,  is  also  considered  as  an  outcome by itself.    Appendix 4: Additional Details of the Treatment Program   Selection of Content  The  treatment  group  intervention  was  designed  to  reflect  best  international  standards  for  investment  readiness programs and guarantee quality of training and mentoring. One of the main concerns for us was  to find an implementer having the capacity to train more than one hundred firms across five countries in  the Western Balkans in a limited amount of time. This required the availability of a considerable quantity  of  mentors,  both  local  and  international,  willing  to  travel  to  the  region  and  with  a  wide‐ranging  background  of  skills  in  business  development.  We  also  needed  to  find  a  partner  with  demonstrated  capacity on organizing internationally renowned pitch events, where small and nascent enterprises have  the opportunity to pitch in front of international investors and opportunity to network their product and  ideas, witness successful stories from established young entrepreneurs and the investors’ community.   The selection procedure consisted in three phases: a call for an Expression of Interest (EOI), followed by  submissions of Technical Proposals (TP) and a final phase where we made a comprehensive assessment  of the technical proposals and their compatibility with the Terms of Reference (TOR).  The first phase saw  eight companies submitting their EOI. We selected five out of the nine companies that expressed their  interest for the second phase: all of them shared a few characteristics like an international focus, and a  team with experiences in the region and familiarity with the SMEs and VC eco‐systems of the Western  Balkans.   The World Bank team reviewed these technical proposals, and also sought an outside evaluation from  Professor  Josh  Lerner  and  his  team  at  the  Bella  Research  Group.  They  have  worldwide  experience  in  assessing venture capital eco‐systems and business accelerator programs. In addition, we referred to the  expert opinion of country officials in the Western Balkan region, experts in the local national innovation  agencies, familiar with the regional eco‐systems and hence able to detect incongruences of the technical  proposals with local conditions. The final overall assessments merged the feedbacks of these three main  sources:  it  listed  the  positives  and  negatives  of  each  proposal  and  identified  specific  questions  to  be  submitted  to  the  applicants  in  case  there  were  aspects  to  investigate  further.  The  final  ranking  that  emerged from the series of consultations and assessments identified the Austrian company Pioneers JFDI  GmbH as the best suitable candidate for the planned intervention.   Pioneers JFDI GmbH was the best candidate because of the experience of their team in the region and in  providing small businesses personalized training and advice,  the competences and logistical as well as  human capital capacity to deliver a widespread training program across five countries. Prior to 2011, the  Pioneers team was involved with STARTeurope, which offered the Startup Live events, a series of training  workshops  and  pitch  events.  Pioneers’  mentors  have  deep  experience  as  venture‐funded  startups  entrepreneurs  and  represent  the  countries  of  interest  in  the  Western  Balkan  region  and  in  addition  Austria,  Germany,  Greece,  Israel,  Lithuania,  Poland,  Slovak  Republic,  Turkey,  United  Kingdom  and  the  50   United States. Many of their mentors come through the Pioneers JFDI GmbH program already, so they  already know the curriculum and thus do not need to be trained.  Treatment Website  The treatment was operated under a separate brand to ensure separation and clearly communicate the  difference between the “Pioneers of the Balkans” competition and the investment readiness program for  the treated group. The “Startup Live Mini‐Accelerator” provided a dedicated website that also provides a  central  point  of  access  to  all  the  treatment  resources.  It  was  password‐protected  to  ensure  that  only  invitees (i.e., Treatment Group participants, mentors, the program management team and World Bank  Group team members) could access it.  At the beginning of the program each beneficiary of the treated group was provided with a starter kit  including  a  detailed  booklet  with  instructions  and  description  of  all  the  four  parts  of  the  investment  readiness program: qualification phase, mentoring phase, masterclasses, and pitch training; and details  of the Pioneers team and their contact details.  WhatAVenture  WhatAVenture asks a simple set of question about the business in order to i) match the entrepreneur with  the appropriate mentor ii) understand the phase of development and the preparation of the entrepreneur  in order to tailor to each firm the subsequent individual mentoring phase, iii) bring the treatment group  firms to a similar level of qualification before proceeding with individual mentoring in the second stage of  the training period.  The application WhatAVenture and the methodology therein was developed and tested in the context of  post‐graduate studies at the University of Economics and Business in Vienna, in close collaboration with  leading  academics  and  practitioners  from  the  innovation  and  entrepreneurship  field.  It  is  an  online  interactive course for start‐ups to put in words the details of their business idea, from the development  of the business plan to marketing strategy and their financing needs.27 The application is designed for self‐ paced progress along its steps. Once registered, startups assigned to the treatment group were granted  access  to  the  tool  until  31  December  2015  independent  of  their  progress  or  advancement.  After  completing each step, they had the opportunity to discuss their progress, findings and potential questions  or difficulties in short online mentoring sessions (typically 30‐45 minutes). The main questions addressed  with the WhatAVenture application are:  1. Customer Exploration: the first step requires the team to answer questions on the targeted  customers, to identify the customer segment and to customer needs related to their product  2. Solution: develop a solution to the problem and match it to the customers’ needs                                                               27  Since the beginning of its external commercialization in 2014, the WhatAVenture has already been rolled out at  several  academic  institutions  as  well  as  leading  European  corporates  like  Deutsche  Telekom  that  use  it  for  standardizing and professionalizing their intrapreneurship processes. Furthermore, several (corporate) accelerator  programs  like  Bayer’s  Grants4Apps  and  two  Austrian  governmental  equity  financing  and  R&  funding  institutions  (Austrian Federal Promotional Bank; Vienna Business Agency) have chosen the tool as their central application for  tracking startups progress and coordinating mentoring sessions throughout their programs.  51   3. Business model – frame a sound business model around the value proposition of the company  4. Competitor analysis – Elaborate on the competitive advantage of the firm, organize an idea of  marketing, sizing and competitive positioning  5. Market size: define the target size of the customer segment  6. Financials: quantify the costs and revenue structures, expected profitability and financing needs  until break‐even  In the first meetings of the WhatAVenture the mentor takes some time to ask questions and understand  in detail the product the company plans to market and the possible value generation. This is important  for providing a better mentorship in the successive acceleration phase.    Assignment to Mentors  In the qualification phase each company was assigned a lead mentor from the beginning who takes the  role of a direct contact person for getting started in the mentoring program. The lead mentors support  their mentees not only as their personal sparring partner during the qualification phase but also as primary  contact  person  and  advisor  during  the  acceleration  and  pitch  preparation  phases.  Match‐making  is  conducted based either on relevant professional experience (e.g., an entrepreneur in the dairy industry  might be assigned a lead mentor with an academic background in dairy product management), personal  interests (e.g., a participating business active in the area of  design might be assigned a lead mentor with  a passion for sailing), technical expertise (e.g., a team that lacks even a basic online presence might be  assigned a web‐/graphics designer as a lead mentor) or proximity.  In  addition,  a  “Mentors  Catalogue”  was  distributed  to  each  firm.  It  contains  relevant  biographical  and  professional information of the 100+ mentors forming the pool of regional and international experts from  where the participants can draw in addition to the assigned lead mentors. The catalogue was sent to the  treated group beneficiaries in the welcoming package just before the beginning of the program and they  were provided with an internet interface where they have access to the network of dedicated mentors,  and where they have the possibility to screen the qualifications and the field of expertise of the mentors  through a short CV and contact them directly to book a mentoring session.   In total the treatment group could benefit from 141 mentors, who came from 26 different countries. Most  of them live in Austria (43.3%) followed by Serbia (10.6%) and Germany (9.2%). They can be divided in  four  main  groups:  standard  teachers  and  mentors  (i.e.  business  consultants,  university  and  business  school professors), successful entrepreneurs (i.e. CEOs of their companies), successful young enterprise  investors  (e.g.  business  angel  investors,  venture  capitalists  etc.),  leading  public  speakers  and  pitch  trainers. All of them cover a wide range of expertise and have at least three years of mentoring experience,  while  more  about  half  of  mentors  have,  individually,  more  than  10  years  of  experience  in  business  mentoring.  The  majority  having  experience  in  business  development  and  management  in  the  IC&T  industry; there are more technical mentors with a science background as software or hardware experts,  payment  systems  and  financial  industry  experts.    Other  industries  were  also  covered,  as  for  instance  health  care  and  pharmaceuticals,  automotive  and  transportation,  shipping  and  apparel  sectors.  All  mentors have a good knowledge of business development, but a dedicated group of mentors was highly  52   specialized in sales, marketing and e‐commerce as well as intellectual property, competitive strategy and  marketing. A smaller subset has experience in human resources, relationships and team building.   Acceleration Phase  Upon successfully completing the qualification phase all beneficiaries are inducted into the acceleration  phase. The individual mentoring sessions were scheduled on the online dedicated website to the program  and  were  carried  out  either  remotely  via  phone,  video  call28  or  on‐site  mentoring  depending  on  the  availability of mentors in the cities where the entrepreneur is located. It is important to note that among  the  pool  of  100+  mentors  many  of  them  are  internationals  living  and  residing  in  the  Western  Balkan  region, hence there was still the possibility to get international mentoring in English within the city of  residence  of  the  entrepreneurs.  We  ensured  that  every  startup  in  the  program  gets  some  on‐site  mentoring exposure, partly also as an instrument to ensure their continued personal commitment to the  program and to allow for the development of personal relationships beyond voice and video calls.  Average  mentoring  sessions  typically  lasted  approximately  90  minutes  and  required  additional  work  between sessions from the entrepreneur to improve the business proposal before the next session. In  total we had more than 1800 hours of individual mentoring. Once a mentor submits his feedback to the  central database, the information entered into the first part of the form is be forwarded via e‐mail to the  mentored entrepreneur, along with the request to likewise provide feedback to the mentor in question.  This bidirectional feedback process not only serves the purpose of assessing mentees’ satisfaction with  the mentor and the benefit gained from a particular session, but also to validate the mentor’s feedback  and data entered by means of a counterparty review process.   Examples of the discussion in the acceleration phase were:   Some companies were developing more products so needed advice on what would be best to  focus on or whether to spin‐off part of their business.   Explore value proposition for different customer segments and how to structure it (i.e. B2B or  B2C), how to implement it and what channels of communication to use. When necessary narrow  down customer segment.   Some firms needed a market validation ‐ to take a prototype or mockup to target customers and  test the outcomes.   Formulating and analyzing the competitor’s matrix, set up a market research plan to investigate  competition in target markets.   Identifying local partners for collaboration and regional expansion.   Defining a clear pricing strategy for different markets (e.g., Western Balkans, Europe, U.S. etc.)   For companies in a more advanced stage discussions on possible financing options for current  expansions plans, the amount to be asked and the form of partnership.   Discuss  legal  ways  to  achieve  monetization:  early‐stage  selling,  licensing  to  interested  parties  worldwide for franchising etc.                                                               28  Remote session were arranged between both parties to take place either via Skype or, especially for remote group mentoring hosted on our software solutions to provide video calls (e.g. WebEx). 53    Making sure the startup product abides to and will operate according to existing regulations and  the  differences  in  regulations  between  the  EU  and  the  Western  Balkans  for  limited  liability  partnerships and equity financing.   Practice to present the company in 5 minutes and in an elevator pitch of 90 seconds; preparation  for the questions time to understand what investors want, and working on telling a clear story.  Masterclass Weekends  Additional training during the acceleration phase is delivered in from of classes and lectures, these take  place during 2.5 days “masterclass weekends” organized in the participating countries. At the masterclass  weekends general business education is taught, courses such as marketing, finances, team building, sales,  competition as well as rhetoric, body language and design.   A dedicated  website for  masterclasses and  the material was set up, the portal also offers information  about  and  access  to  a  dedicated  community  communication  channel  “ChatGrape”.  This  is  an  instant  communication tool available as a browser‐based application as well as native application for most mobile  devices and allows for private as well as group communication in a structured way by allowing all users to  set  up  and  join  subject‐specific  groups  and  to  tag  information  and  questions  posted  with  key  word  expressions.  Masterclasses took place in the following four locations and dates:  - 9 to 11 October ‐ Split, Croatia   - 16 to 18 October ‐ Novi Sad, Serbia   - 23 to 25 October ‐ Pristina, Kosovo   - 30 October to 1 November ‐ Skopje, Macedonia   Each  weekend  had  a  main  theme  but  was  not  exclusively  dedicated  to  it  with  lectures,  panels  and  presentations covering other topics as well. For example, the weekend in Split dealt with the business  model, while the masterclass in Novi Sad with sales and marketing. The Pristina masterclass had the main  lectures  on  team  building  and  human  resources  while  the  Skopje  weekend  dealt  with  investment  and  finance. The final program of each masterclass weekend was set up at the end of September and published  on  the  information  portal  so  that  beneficiaries  can  gather  information  and  decide  which  masterclass  weekends they want to attend. Before each masterclass the mentors and the beneficiaries are provided  with a guide that helps them to understand the organizational structure of the weekend and the benefits  of participating in the weekend. Some examples of the content of the masterclasses are described below.  The lecture on “Research and networking” introduced the importance of research and networking for the  best business model. It explained the difference between a business model and a business plan, how to  prepare an action plan and structure a business model canvas step by step. It urged entrepreneurs to  think about the weakest points of their plans and possible solutions. It then touched upon the importance  of customers, competition, sales, marketing, traction, business development and finances.  The  class  “Rapid  Prototyping”  described  how  to  move  from  an  idea  to  a  market  validated  product.  It  explained the concept of rapid prototyping, the importance and the methods of prototyping and using  examples  from  the  cinema,  cars  and  smartphone  applications  sectors.  It  then  covered  the  concept  of  minimum viable product (MVP) and the need to frame business hypotheses on the market reaction to  54   their product, the customers and financial hypotheses too. All of them should be tested in the market to  get feedback and fine tune product development.   The team building panel addressed questions on how to create a team and what are the most important  features  a  new  company  must  develop  in  order  to  have  the  investors’  attention.  Two  main  things  emerged:  the  first  is  that  a  successful  enterprise  has  to  form  an  eclectic  and  competent  team  encompassing  all  possible  functions  that  a  nascent  company  must  have.  The  spectrum  of  functions  proposed  ranges  from  not  only  having  a  developer  and  an  idea  but  also  in  having  a  good  lawyer,  a  technician, a person familiar with the financing. But the most important of all seem to be having a very  good member acting as a sales person. This figure should end up being most of the time the CTO of the  company if not a co‐founder because dedicated persons are really difficult to find, in those cases is the  founder itself that must acquire sales skills and complement them with partners acting as supporters in  this role. The importance of having a team with a wide‐ranging expertise that complement each other  turned out to be one of the best ways sending a positive message of confidence and investment readiness  to the investors, a message saying that if you put the money in my company you are minimizing the risk  of wasting your money.    The  traction  presentation  emphasized  the  importance  of  the  three  Ts:  team,  technology  and  traction.  Traction because it is strictly linked with the term growth, with the importance of scaling up and having  sustainable growth and having a “product‐market fit” which is another way of saying that the product  should be in line with the demand coming from the market. However, it was stated that one size does not  fit all and there are no general rules, what works for one company is not always good for others, as well  as a channel to gain traction today is not guaranteed that will work for the same company some time  down the road. The focus shifted then to the need to update the targets, reset the objectives forward  every time a target is reached. The channels to increase traction were also covered, 19 of those channels  were mentioned and briefly explained (social and display ads, offline ads, email marketing, targeting blogs,  direct sales, trade shows etc.).     The presentation “The quantified startup” delved into trying to use data driven decision frameworks into  strategic decision making of a startup. The presentation is directed mostly, but not exclusively, to web  service providers, that is companies that can track their users online. What kind of metric are important  to identify which stage your startup belongs at the moment, what metrics are important to scale up or  increase traction? The presentation provided references of papers and books the presenter recommends  to  identify  the  metrics  needed  for  every  stage  in  the  startup  development.  Measures  such  as  churn,  acceptance rate, viral coefficient, cost per user and similar were overviewed.     The  presentation  “How  to  sell  to  corporations”  covered  the  topic  of  how  to  get  access  to  established  corporates for nascent startups. How to ally with them and exploit the market potential and value they  have. One of the main points is that corporations, despite investing money in in‐house accelerators and  alliance partnership programs for startups, they do not really understand fully the value the startups that  approach them have. So, it is up to the startups to get ready for this kind of partnership, it is they that  have to explain and convince the corporations of the value of their idea. The presenter described a process  55   toward strategically thinking about approaching a corporation.  How to convince corporations? Set the  targets, find the best match, do your research, be well prepared, set our sales steps. An important aspect  touched upon was that, once arranged a meeting you need to frame the meeting in order to get the idea  convened, speak about concrete and clear things.    The  lecture  on  B2B  marketing  saw  a  short  introduction  on  the  history  of  marketing.  Some  general  information  was  given  and  the  difference  between  the  B2B  and  B2C  marketing  was  explained.  Introductions to new paradigms like the C2B and C2C was also described. The speaker explained processes  of customer decision making and affiliation with a brand, with few examples from the most established  companies and their marketing strategies. The importance of tradeshows for marketing was emphasized  despite being an expensive option. But it is one of the best ways to get in touch with professional buyers  informally.  The lecture on “EU funding” delved into the landscape of funding opportunities for startups and SMEs at  the institutional EU level through EU structural funds for development. Information on different type of  funding, the application process and the best way to approach these funding minimizing the load of work  for the application. The need of a consultant for the application was also pushed forward as a good idea  to develop these proposals and how much consultancy is needed.  The  presentation  “How  to  craft  a  pitch”  as  delivered  in  all  four  masterclasses  and  described  how  to  structure the pitch and what to  emphasize in it.  The second part of the  talk  dealt with  the 90 second  elevator pitch.  The emphasis for the 5 minute pitch was on seven main points to take into account: i)  product/service  what  it  is  and  explained  it  in  detail  to  make  the  audience  understand  it,  ii)  market  opportunity, what are the prospects, the vision and the demand for the product, iii) team, who are the  main  components  of  the  team,  what’s  their  expertise  and  role,  iv)  competition,  v)  finances  and  cost  structure, vi) development stage: where you are, at what stage, what you need, and vii) future, where you  will be, or expect to be, in 6 months to a year.   Pitch Preparation  The mentoring program transitions into the Pitch Preparation Phase after the last masterclass weekend.  This  phase  is  intended  to  ensure  that  all  beneficiaries  focus  their  attention  entirely  on  their  pitch  performance in the remaining two weeks before their appearance on stage in front of jury members in  the semi‐finals pitch event.   In the course of this phase, a standard pitch training approach was implemented, this was developed and  tested in the context of the annual Pioneers Festivals and consists of the following steps:   1. The entrepreneurs were asked to upload the pitch decks (tailored to a 5‐minute on‐stage presentation  followed  by  another  5  minutes  of  questions  and  answers  with  the  jury).  This  pitch  deck  is  then  made  available to the lead mentor for initial review.   2. The entrepreneurs schedule a video call with their lead mentor to begin practicing the pitch together.   3. During the sessions, the entrepreneur delivers his/her pitch and receive feedback on both the oral pitch  performance as well as the pitch deck.   56   4. Lead mentor and entrepreneur may schedule additional sessions bilaterally to review progress as the  entrepreneur implements recommendations.   5. In parallel, the program management team assigns each entrepreneur two additional mentors, one it  has worked with already and one new mentor.   6. Also these mentors are asked to schedule pitch training sessions with the entrepreneur and request the  latest version of the pitch deck.   7. The program management team collects and reviews feedback protocols to assess the entrepreneurs’  preparedness for their Semi‐finals appearance.  The entrepreneurs were encouraged to use the time between sessions to work on their pitch decks and  practice their oral delivery of the pitch further. Additionally, to this standard pitch training cycle and the  live "dress rehearsal" on the day prior to their pitch in the Semi‐finals, entrepreneurs can request further  support from specialists on rhetoric, body language or slide deck design by approaching relevant mentors  from  the  mentors’  catalogue  if  needed,  or  upon  recommendation  by  one  of  their  pitch  preparation  mentors.   Detailed Dost Greakdown  The cost of offering the program is provided in appendix Table 4.1  Appendix Table 4.1: Detailed Program Cost Breakdown    57   Investment readiness programs - Calculatory program cost A. Individual mentoring Unit Quantity Rate [USD] 1. Direct cost of individual mentoring hours per beneficiary 30 1'917 2. Overhead cost per mentor per beneficiary 1 326 3. Overhead cost of mentoring program per beneficiary 1 492 4. Online mentoring tool per beneficiary 1 338 Subtotal per beneficiary 3'072 B. Masterclasses 1. Organization per beneficiary 1 321 2. Venue & catering per beneficiary 1 107 3. Lectures per beneficiary 4 175 4. Travel and accomodation cost per beneficiary 1 191 Subtotal per beneficiary 793 C. Pitch training 1. Organization per finalist 1 170 2. Venue & catering per finalist 1 73 3. Pitch training per finalist 1 279 4. Travel and accomodation cost per finalist 1 168 Subtotal per finalist 690 Grand total per beneficiary A. Individual mentoring per beneficiary 1 3'072 B. Masterclasses per beneficiary 1 793 C. Pitch training per beneficiary 0.33 230 4'095 Grand total per investment readiness program A. Individual mentoring Number of beneficiaries 150 460'865 B. Masterclasses Number of masterclasses 4 118'932 C. Pitch training Number of finalists 50 34'513 614'310         58   Appendix 5: Additional Details of the Control Program   Selection of Content  We organized the control group intervention design around a few simple guidelines: i) an online course,  ii) relatively cheap or free to use, iii) offering general knowledge of simple investment readiness concepts  and iv) providing e‐guidance toward a start‐up pitching competition. The World Bank team conducted  market  research  together  with  Innovative  Ventures  Incorporated,  a  specialized  investment  advisor  to  international financial institutions and governments in private equity and venture capital funds. After this  initial screening of available alternatives, the decision was made to use a paid online course since the  alternatives  without  fee  did  not  offer  the  necessary  quality  standards.  For  the  paid  alternatives  we  carefully  evaluated  the  contents  and  undertook  the  full  demo  versions  to  understand  the  specific  differences among the candidate courses.   The program chosen is an e‐learning course developed and distributed by the Global Commercialization  Group (GCG) of the IC2 Institute at the University of Texas at Austin. The group is an internationally active  facilitator  for  growth  of  innovative  and  technology‐based  businesses  and  it  offers  a  wider  range  of  technology  commercialization  training  programs  for  managers  around  the  world.  The  Innovation  Readiness  SeriesTM  was  created  to  bring  the  work  of  the  Global  Commercialization  Group  to  a  global  customer  base  at  a  cheaper  price  vis‐à‐vis  delivering  training  and  international  business  development  programs in‐country. Since its launch in 2011, the Innovation Readiness SeriesTM has trained more than  two thousand entrepreneurs and students from 20 countries worldwide. The content can be offered in  three different languages: English, Spanish or Russian. For the Pioneers of the Balkans cohort we opted  for the English based course.  Course Details and Content  The  program  introduces  students  to  common  terminology  used  in  the  start‐up  eco‐system,  and  the  requirements  to  commercialize  innovations,    including  protecting  intellectual  property,  describing  an  innovation and the benefits it provides (vs. features), navigating development, understanding competition  (substitutes  and  direct  competitive  products),  market  validation,  creating  a  ‘pitch’  and  presenting  to  investors, customers and others.  This content is delivered online through 10 modules of 45‐60 minutes each. The modules have a set of  slides that are read and explained via a recorded voice. Each module has detailed steps to work through  for creating a business proposition and includes assignments in two formats: quizzes with multiple‐choice  answers beneficiaries can take to test their understanding of the material, and in the case of some of the  ten modules (i.e. technology brief and description, benefits, competition and presentation skills) written  exercises  to  be  voluntarily  handed  in.  Finally,  in  the  last  module  there  is  the  possibility  to  record  and  upload  a  video  sample  of  the  planned  pitch.  Nevertheless,  for  the  Pioneers  of  the  Balkans  cohort  the  program was customized to allow feedback only after the multiple‐choice quizzes in form of number of  correct answers. Written exercise and the video of the pitch were voluntarily uploaded on the platform  but were not commented or discussed with the participant.   While this program is not a substitute to one‐on‐one mentoring, it gives a basic introduction to business  planning  and  pitching,  is  well‐structured  and  cheap  alternative  to  a  mentorship‐based  investment  readiness  program,  it  is  comprehensive  and  allows  beneficiaries  to  create  a  sketch  of  business  model  59   which  can  be  presented  to  investors,  customers  and  other  interested  parties.  Moreover,  it  is  a  self‐ learning tool, beneficiaries can work at their own pace, the ten‐module series introduces the key concepts  of innovation, and explores each of the primary issues that impact bringing a technology to the market  allowing for a self‐paced learning environment.  In terms of curricular incentives, at the completion of all the modules, beneficiaries who answer correctly  at least 70% of quiz questions and take active part in all of them, receive a certification of Investment  Readiness from IC2 Institute at the Texas University through the World Bank Group program “Pioneers of  the Balkans”.   The list of the ten modules and short description of the content is provided.  Module 1 – Introduction: the introduction module explains how the Innovation Readiness Series works,  and the objectives for the course. It explains what commercialization is and helps distinguish between  innovation and invention.   Module  2  ‐  Technical  Description:  the  technology  description  module  helps  participants  describe  their  innovation using technical jargon and key words.  Module 3 – Benefits: the benefits module teaches how to articulate the benefits of an innovation in a way  that conveys value to customers and users.  Module  4  ‐  Development  Status:  the  development  status  module  delivers  an  overview  of  the  product  development cycle with an eye to the market.  Module 5 ‐ Intellectual Property, Part 1: explains what IP is, the different types of ownership, and what  can be protected. It also explains Trademarks and Copyrights.  Module  6  ‐  Intellectual  Property,  Part  2:  the  focus  is  on  Patents  and  Trade  Secrets,  and  provides  a  foundation to designing an individual IP strategy.  Module  7  –  Competition:  the  competition  module  will  help  the  participant  discover  and  compare  key  benefits to those of the competition.  Module 8 ‐ Market Validation: the market validation module explains the validation process and how to  discover exactly what the market expects from an innovation.  Module 9 ‐ Pitching Your Innovation: the planning and pitching module helps prepare a technology brief  of the innovation and can be used in the next steps to commercialization.  Module 10 ‐ Presentation Skills: the presentation module is taught by an internationally established and  experienced public speaker, demonstrates how to deliver presentations in an effective and captivating  way.  Depending on the previous experience of the participant and their commitment to hand in a written set  of answers, a minimum of four weeks is recommended to deliver a basic course and the total envisioned  time to complete the course lectures, answer the quizzes and compile the written exercises is 15‐30 hours.  However, recall that among the set of assignments only the quizzes after each session were graded and  participants receive feedback on the number of correct answers. In case of written exercises and uploaded  pitch video no feedback was offered so that the only incentive in that case was self‐motivation. Moreover,  60   only quizzes counted toward the receipt of the final completion certificate, given this incentive structure  we expect a lower usage of the written exercises and video pitch uploads than multiple‐choice quizzes.  Communication and Reminders  During  the  deployment  of  the  intervention  our  team  sent  weekly  motivational  announcements  to  the  students  on  the  platform  and  on  their  email  address,  the  aim  of  the  announcements  was  to  promote  learning and active participation. They were structured as progress reports where we showed the top ten  performing firms in the last week in terms of correct answers in submitted quizzes, and explained the  reasons why it is important to take part in the course. Firms were told that going through the modules  would both  help provide  matching of  their businesses with judges who had sectoral expertise in their  business,  and  that  going  through  the  contents  of  the  modules  would  likely  increase  their  chances  of  getting a higher score in the semi‐finals and getting selected for the finals.  Usage  Appendix Figure 5.1 summarizes the proportion of students that submitted assignments (either quizzes  or written exercises), each bar corresponds to an assignment. Out of the 120 participants that connected  at least once to the online platform, 63 (36.6% of the total) actively participated in one of the quizzes,  with 45 of them completing the threshold of 70% correct answers. For the non‐graded written exercises,  the technology description was completed by 40 participants, the technology brief by 20, benefits exercise  by 29 and the competition exercise by 22 participants. Lastly, only 8 students uploaded a video of their  pitch.  Appendix Figure 5.1: Participation of the Control Group in Online Course Content  Proportion of students submitted assignemnts Technology Brief Exercise Competition Exercise Benefits Exercise Technology Description Exercise Prepare a Pitch Competition Quiz Tech Description Quiz Market Validation Quiz Intellectual Property 2 Quiz Intellectual Property 1 Quiz Development Status Quiz Benefits Quiz % 20% 40% 60% 80% 100%     Satisfaction  A short survey was administered after the semi‐finals to assess their satisfaction with different elements  of the program. Respondents are therefore only the entrepreneurs that participated in the semi‐finals.  The survey was answered by 102 treated group firms (92.7% of the treated semifinalists) and 87 control  61   group firms (86.1% of control semifinalists).  Appendix Table 5.1 compares the overall satisfaction of the  treated and control group semifinalists over few dimensions on a scale from 1 to 6. The treated group  values  more  the  communication,  the  structure  and  design  and  the  training  materials  provided,  the  difference  is  statistically  significant.  However,  the  mean  grade  given  by  the  control  group  to  those  dimensions  is  well  above  4.  Recall  that  firms  were  blind  to  treatment  assignment.  Where  there  is  no  significant satisfaction difference between the treated and control group is in the feedback received from  the jury at the semifinals and the organization of the semifinals. These features were common to both  groups. As such, the satisfaction survey indicates the value added of the treatment also in the subjective  assessment of the program by participants.     Appendix Table 5.1:  Treated vs. Control Satisfaction survey – How satisfied are you with each of the following?      Treatment    Control      Dimension    Obs.  Mean  Std. dev.    Obs.  Mean  Std. dev.    p‐value                        Communication overall    102  5.17  .95    87  4.55  1.44    0.014  Structure and Design of PotB    101  5.00  1.14    86  4.43  1.26    0.005  Training Resources    102  5.31  1.02    84  4.36  1.25    0.000  Jury Feedback    101  4.45  1.43    86  4.11  1.68    0.486  Semi‐Finals (Belgrade Venture    101  4.45  1.42    83  4.34  1.36    0.861  Forum)                        Note: PotB denotes Pioneers of the Balkans program    Appendix 6: Additional Details on the Semi‐Finals and Finals   Appendix Table 6.1 summarizes the characteristics of judges used for the scoring.  Appendix Table 6.1: Semi‐Final Judge Characteristics Mean Std. Dev. Lives in the Western Balkans 0.37 0.49 Lives in European Union (except Croatia) 0.48 0.50 Male 0.88 0.33 Age 39.1 10.4 Has Founded a Company 0.75 0.43 Years of Experience in their industry 11.5 8.5 Company makes venture investments 0.64 0.48 Is an Angel Investor 0.37 0.49 Regularly Mentors Start‐ups 0.80 0.40 Sample Size 65 Note: data unavailable for one judge.   Appendix Figure 6.1 shows that the baseline distribution of investment readiness scores is similar for those  that participated in the semi‐finals (and therefore received judges’ scores) and those that did not.  62   Appendix  Figure  6.1:  Baseline  Investment  Readiness  Scores  by  Participation  in  the  Semi‐finals  and  Treatment Status  Baseline scores by participation in semi-finals .6 Proportion of Firms .2 0 .4 1 2 3 4 5 Baseline Investment Readiness Score Treatment Attend Treatment Don't Attend Control Attend Control Don't Attend   Robustness to Non‐Participation  Our pre‐analysis plan specified two approaches to examining the robustness of our results to the attrition  that results from not all participants attending the semi‐finals, and therefore not having judges’ scores for  all firms.  The first approach is to impute investment scores for firms which did not participate in the finals. We  pre‐specified that we would do this by estimating the following equation on the control group sample  who participated in the semi‐finals:    This yields a prediction of the semi‐finals investment readiness score as a function of the baseline scores  on the different components, the country of operation, and whether or not they had an outside private  investor at baseline. We replace missing scores for both treatment and control with these predicted values  and re‐estimate equation (1). The first column of Appendix Table 6.2 repeats our estimated impact on the  overall score from Table 3, which assumes scores are missing‐at‐random. Column 2 then shows the impact  on the score after imputing missing values. The impact is still positive and statistically significant, with an  estimated effect of 0.19 points.   63   The second approach is to compare the participation rates of treatment and control and use Lee (2009)  bounds to adjust for differential attrition. The participation rate in the semi‐finals was 63.2 percent for  the  treatment  group,  and  58.7  percent  for  the  control  group.  The  difference  of  4.5  percent  is  not  statistically  significant  (p=0.39,  or  0.37  after  controlling  for  strata  fixed  effects).  Nevertheless,  we  test  sensitivity to this difference in attrition rates by dropping the top or bottom eight (4.5% of 174) scores  from the treatment group. The next two columns of Appendix Table 6.2 then show the Lee upper and  lower  bounds  respectively  are  0.41  and  0.18.  Since  Table  1  and  appendix  Figure  1  shows  that  the  differential attrition is not coming from the tails of the baseline investment readiness score distribution,  we think it highly unlikely that it would be coming from either tail of the follow‐up distribution.   As a final robustness check, we show in the last two columns of Appendix Table 6.2 that our results are  not sensitive to how we aggregate the different sub‐scores. Column 5 aggregates the five sub‐scores using  equal weights instead of the different weights in our main specification, while Column 6 also includes the  presentation score. We see the estimated effects of 0.277 and 0.293 are very similar in sign, significance,  and magnitude to those using the unequal weights.  Taken together, these results show that the impact of treatment on the investment readiness score is  unlikely to be driven by differential participation patterns in the semi‐finals between the treatment and  control  groups,  nor  by  the  weighting,  and  so  our  finding  that  the  investment  readiness  program  has  improved investment readiness is robust.  Appendix Table 6.2: Robustness of Impact on Investment Readiness to Attrition and to how scores are weighted Imputed   Lee  Lee  Equally weighted Score Score Upper Lower 5 components 6 components Assigned to Treatment 0.284** 0.193*** 0.408*** 0.176 0.277** 0.293** (0.126) (0.065) (0.119) (0.130) (0.123) (0.124) Sample Size 211 343 203 203 211 211 Control Mean 2.908 2.865 2.908 2.908 2.950 2.966 Control Std. Dev 0.903 0.750 0.903 0.903 0.884 0.894 Notes:  Robust standard errors in parentheses. Regressions control for randomization strata.  *, **, *** indicate significance at the 10,5, and 1 percent levels respectively Score is the investment readiness score in the semi‐finals. Imputed score imputes missing scores based on regressing the score  for the control group on baseline team, traction, market readiness, product technology, country, and having an outside investor  and using predicted score for missing observations. Lee upper and Lee lower bounds trim the bottom 8 and top 8 scores respectively from the treatment group to adjust for higher attrition in the control group. Equally weighted scores weight the five (team, technology, traction, market and progress) or six (also presentation) sub‐scores equally.   Appendix 7: Impact on Web Traffic and Being Included on AngelList  Our  pre‐analysis  plan  also  noted  that  we  would  consider  several  measures  of  web  traffic  and  web  presence that have been used by other researchers (e.g. Kerr et al. (2014), Gonzales‐Uribe and Leatherbee  (2015)), but which may be less appropriate for firms in a less developed market: whether or not the firm  is  included  in  AngelList,  a  popular  web  platform  for  fundraising,  startup  jobs  and  investing  allowing  startups  to  raise  capital  from  angel  investors;  and  the  global  web‐traffic  rankings  of  the  company’s  webpage as collected by Alexa and SimilarWeb. We see no significant impacts on any of these measures.  64   Appendix Table 7.1: Impacts on Web Traffic and AngelList Appears Has  Alexa   Has  Similar   on Alexa Global Similar Web Angel List Rank Ranking Rank Ranking Panel A: Impact at Six Months Assigned to Treatment ‐0.022 ‐0.042 ‐304.7 0.013 447.5 (0.035) (0.043) (1412) (0.047) (2068) Sample Size 346 346 188 346 160 Control Mean 0.308 0.535 11161 0.442 12364 Control S.D. 0.463 0.500 8383 0.498 9434 Panel B: Impact at Eighteen months Assigned to Treatment ‐0.034 0.032 120.6 ‐0.008 1952.4 (0.041) (0.048) (1438) (0.048) (1945) Sample Size 346 346 132 346 156 Control Mean 0.372 0.343 7407 0.442 12614 Control S.D. 0.485 0.476 5431 0.498 9949 Notes: Robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively. All regressions include controls for baseline level of outcome, and for strata used in randomization. Alexa Global ranking and Similar Web ranking are expressed in 1000s, and are conditional on having a ranking at all.       Appendix 8: Follow‐up Survey Response Rates and Balance on Responders  Appendix Table 8.1 reports the completion rates by treatment status for three definitions of completion.  Initially we began with a longer follow‐up survey, which in addition to asking about our key outcomes,  also asked a series of process questions about the Pioneers of the Balkans program and their reasons for  participating or not participating. In order to encourage responses from more reluctant firms, we removed  these  questions  to  shorten  the  questionnaire  for  a  second  interviewing  phase,  with  the  short  survey  containing all the key outcomes in our pre‐analysis plan. Finally, for firms that we could not interview after  multiple attempts, we attempted to collect basic information in a few minutes from them, asking for their  current operating status, their number of employees, whether they had entered into negotiations with  an outside investor to make an investment in their firm since August 2015, and how much new investment  they had received since August 2015. In the second follow‐up, this basic information was restricted to  whether the firm was still operating, and whether it has received external financing, and also used web  searches and secondary contacts.  We see that the treatment group was more likely to respond to the full survey than the control group in  the first follow‐up survey (p=0.066), but there is no significant difference in response rates for having at  least  the  short  survey,  or  at  least  basic  information,  and  no  significant  treatment  differences  for  the  second follow‐up.  Appendix Table 8.1: Follow‐up Survey Completion Rates     65      Overall  Treatment  Control  p‐value  First Follow‐up Survey              Completed Full Survey  0.65  0.70  0.60  0.066  Completed at least Short Survey  0.79  0.80  0.78  0.781  At least basic information  0.92  0.93  0.91  0.520  Second Follow‐up Survey              Completed Full Survey  0.64  0.66  0.62  0.282  Completed at least Short Survey  0.85  0.86  0.84  0.504  At least basic information  0.95  0.95  0.94  0.873     Sample Size  346  174  172     Note: p‐value is for test of equality of treatment and control completion   rates after controlling for randomization strata.        At least basic information denotes that information on whether the firm  is operating and whether it has received external financing is available.  Appendix Table 8.2 compares baseline observables for the treatment and control groups, conditional on  completing at least the short survey. We cannot reject that these observables are orthogonal to treatment  status for either definition of survey completion. Given the lack of significant difference in response rates  by  treatment  status,  and  that  the  sample  responding  to  at  least  the  short  survey  is  balanced  on  observables, we treat attrition as missing at random in our analysis of the survey data.  66   Appendix 8.2: Balance Test on Sample Interviewed at Follow‐up Answered First Follow‐up Answered Second Follow‐up Treatment Control P‐value Treatment Control P‐value Variables stratified on Incorporated/Registered in Croatia 0.230 0.237 0.869 0.27 0.24 0.623 Incorporated/Registered in Serbia 0.446 0.481 0.619 0.48 0.50 0.637 Baseline Readiness Score 2.997 2.899 0.183 2.93 2.94 0.163 Has an outside private investor 0.122 0.067 0.145 0.10 0.10 0.227 Other variables Market attractiveness score 3.112 3.062 0.885 3.06 3.09 0.657 Product technology score 2.485 2.419 0.649 2.44 2.48 0.872 Traction score 3.433 3.233 0.818 3.32 3.17 0.135 Team score 3.090 3.008 0.971 3.00 3.11 0.630 Sector is business and productivity 0.460 0.393 0.435 0.47 0.38 0.172 Sector is lifestyle and entertainment 0.187 0.230 0.516 0.19 0.23 0.428 Uses Cloud Technology 0.201 0.252 0.617 0.19 0.26 0.187 Uses Big Data 0.187 0.222 0.959 0.19 0.24 0.186 Place in value chain is developer 0.647 0.533 0.056 0.63 0.57 0.270 Place in value chain is service provider 0.568 0.533 0.479 0.59 0.56 0.482 Age of firm (years) 2.712 2.622 0.445 2.55 2.50 0.951 Early stage firm 0.331 0.304 0.475 0.32 0.37 0.464 Revenues in 2014 197649 157401 0.955 181796 127478 0.630 Number of employees 6.856 5.467 0.341 6.08 5.35 0.218 Age of main founder 38.216 36.563 0.222 38.02 37.19 0.433 Main founder has post‐graduate education 0.525 0.496 0.934 0.50 0.50 0.770 At least one founder is female 0.137 0.222 0.066 0.15 0.22 0.063 Company has a global focus 0.583 0.578 0.850 0.59 0.60 0.815 Have accepted outside financing 0.374 0.348 0.559 0.35 0.39 0.614 Previously in mentoring/accelerator program  0.173 0.178 0.535 0.16 0.17 0.905 Sample Size 139 135 150 144 Joint test of orthogonality of treatment p‐value 0.417 0.167 Note: interviewed at follow‐up denote that firm completed at least the short survey Appendix 9: Treatment Effects on Individual Survey Outcomes  Appendix Tables 9.1, 9.2, 9.3, 9.4 and 9.5 report the treatment impacts estimated on each of the individual  outcomes that make up the aggregate indices presented in Table 7. Our main approach to multiple testing  is to use the standardized indices of z‐scores, which are contained in Table 7, and are presented again at  the end of each table. Alternatively, since there are 25 outcomes presented in these appendix tables for  each  time  period,  using  Holm’s  (1979)  step‐down  method  gives  an  adjusted  p‐value  for  the  most  significant of the individual outcomes  (the employment effect in round 2) of 0.425.  Thus, none of the  coefficients shown in these tables are individually significant after adjustments for multiple testing.  67   Appendix Table 9.1: Treatment Impacts on Willingness and Interest in Taking on Equity Investment Interested in   Maximum  Has specific   Would consider Aggregate   equity investment equity share deal terms Royalties Index Panel A: Impact at Six Months Assigned to Treatment ‐0.019 3.920 0.001 0.025 0.051 (0.066) (3.169) (0.061) (0.065) (0.094) Sample Size 278 264 271 268 278 Control Mean 0.603 23.155 0.331 0.508 ‐0.015 Control S.D. 0.491 23.439 0.472 0.502 0.764 Panel B: Impact at Two Years Assigned to Treatment ‐0.034 ‐2.175 0.050 0.105* 0.032 (0.055) (2.972) (0.051) (0.056) (0.084) Sample Size 309 285 309 303 309 Control Mean 0.575 25.066 0.242 0.487 ‐0.005 Control S.D. 0.496 26.591 0.430 0.501 0.783   Appendix Table 9.2: Impacts on General Investability Number   Founder  Positive Revenue Positive Sales Aggregate Employees full‐time Revenue >10,000 euros Profit US/Europe Index Panel A: Impact at Six Months Assigned to Treatment 1.100 0.061 0.008 0.035 ‐0.061 ‐0.019 0.026 (1.215) (0.052) (0.061) (0.068) (0.059) (0.060) (0.085) Sample Size 318 269 277 277 272 265 277 Control Mean 6.111 0.750 0.699 0.353 0.289 0.386 ‐0.039 Control S.D. 10.596 0.435 0.461 0.480 0.455 0.489 0.634 Panel B: Impact at Two Years Assigned to Treatment 4.554** 0.018 0.032 ‐0.017 0.085 0.051 0.089 (1.814) (0.061) (0.071) (0.068) (0.061) (0.055) (0.082) Sample Size 291 291 232 242 276 310 291 Control Mean 4.683 0.620 0.526 0.361 0.482 0.340 ‐0.058 Control S.D. 6.381 0.487 0.502 0.482 0.502 0.475 0.650   68   Appendix Table 9.3: Impacts on Meeting Specific Needs of Investors Separates Has revenue  Knows customer   Number key Found out if Has IP or Aggregate   Accounts projection acquisition cost metrics tracked can protect IP pending Index Panel A: Impact at Six Months Assigned to Treatment 0.060 0.066 0.009 ‐0.168 0.033 0.054 0.082 (0.053) (0.066) (0.064) (0.299) (0.065) (0.061) (0.080) Sample Size 268 268 268 268 269 269 269 Control Mean 0.742 0.561 0.409 2.106 0.439 0.364 ‐0.059 Control S.D. 0.439 0.498 0.494 2.598 0.498 0.483 0.682 Panel B: Impact at Two Years Assigned to Treatment 0.086 0.018 0.061 ‐0.092 0.059 0.049 0.084 (0.060) (0.061) (0.059) (0.361) (0.063) (0.057) (0.079) Sample Size 291 291 291 269 271 275 298 Control Mean 0.577 0.486 0.352 1.667 0.444 0.244 ‐0.059 Control S.D. 0.496 0.502 0.479 2.854 0.499 0.431 0.692   69   Appendix Table 9.4: Impacts on Steps Towards Investment Contacted   Has mentor   outside Made   helping raise Entered into   Aggregate    investor a pitch finance negotiations Index Panel A: Impact at Six Months Assigned to Treatment ‐0.082 0.016 0.078 ‐0.008 ‐0.017 (0.074) (0.068) (0.063) (0.057) (0.098) Sample Size 239 240 232 279 240 Control Mean 0.509 0.549 0.236 0.323 0.008 Control S.D. 0.502 0.500 0.427 0.470 0.720 Panel B: Impact at Two Years Assigned to Treatment ‐0.019 0.006 0.050 0.068 0.044 (0.057) (0.047) (0.040) (0.059) (0.092) Sample Size 282 282 279 279 282 Control Mean 0.324 0.184 0.097 0.328 ‐0.032 Control S.D. 0.470 0.389 0.297 0.471 0.760 70   Appendix Table 9.5: Impact on External Investment Taken on Made deal   Received   Amount of Received   new with at least new investment incubator Aggregate   debt investor 25,000 received grant index Panel A: Impact at Six Months Assigned to Treatment ‐0.118** ‐0.024 ‐0.032 ‐11232* ‐0.036 ‐0.152* (0.057) (0.033) (0.028) (6425.486) (0.037) (0.087) Sample Size 276 279 277 277 269 279 Control Mean 0.419 0.083 0.068 13452.273 0.090 0.084 Control S.D. 0.495 0.276 0.253 62358.927 0.288 0.741 Panel B: Impact at Two Years Assigned to Treatment ‐0.059 0.050 ‐0.024 n.m. ‐0.005 0.003 (0.048) (0.049) (0.041) (0.036) (0.080) Sample Size 278 330 317 268 330 Control Mean 0.182 0.244 0.168 0.076 0.018 Control S.D. 0.388 0.431 0.375 0.267 0.698 Note: n.m. denotes not measured in this survey round. 71