WPS8011 Policy Research Working Paper 8011 How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence David McKenzie Development Research Group Finance and Private Sector Development Team March 2017 Policy Research Working Paper 8011 Abstract Jobs are the number one policy concern of policy makers typically assume. Many of these evaluations find no sig- in many countries. The global financial crisis, rising nificant impacts on either employment or earnings. One demographic pressures, high unemployment rates, and reason is that urban labor markets appear to work reason- concerns over automation all make it seem imperative ably well in many cases, with fewer market failures than is that policy makers employ increasingly more active labor often thought. As a result, there is less of a role for many market policies. This paper critically examines recent traditional active labor market policies than is common evaluations of labor market policies that have provided practice. The review then discusses examples of job cre- vocational training, wage subsidies, job search assistance, ation policies that do seem to offer promise, and concludes and assistance moving to argue that many active labor with lessons for impact evaluation and policy is this area. market policies are much less effective than policymakers This paper is a product of the Finance and Private Sector Development Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at dmckenzie@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence# David McKenzie, World Bank JEL classification codes: O15; J08; J68 Keywords: Active Labor Market Policy; Jobs; Wage Subsidies; Vocational Training; Job Search. # I thank Asli Demirguc-Kunt for encouraging me to give a policy talk on this topic, and the authors of the different papers summarized for helpful clarifications and comments. Introduction In a well-functioning labor market, firms that want workers and workers who want jobs are able to find one another reasonably easily, and the only unemployment is low, frictional, and temporary. In such a world, the main area for government policy is passive policy, in which the government undertakes investments in infrastructure and provides the regulatory framework needed for the economy as a whole to grow and raise incomes, but does not intervene directly to help particular workers find jobs or particular firms to find workers. However, in practice, governments have long engaged in a variety of active labor market policies (ALMPs) that directly intervene in the labor market with the aim of generating more and better employment opportunities for workers. Examples include training programs that aim to increase the skills of the labor supply, wage subsidies that aim to increase firms’ demand for labor, and job search and matching assistance that aims to better enable firms and workers to find and contract with one another. Four recent global trends have increased the importance of jobs as a policy concern, and renewed interested in the effectiveness of ALMPs. The first was the global financial crisis of 2007-2008, which increased unemployment in many countries worldwide. Second, rising demographic pressures in some parts of the developing world have led to headlines of a “jobs time bomb” with claims like India needing to create 12 million new jobs annually (Kumar and Busvine, 2014), 10 to 12 million young people entering the job market annually in Africa (Mohammed, 2015), and the Middle East and North Africa (MENA) region needing to create 100 million new jobs by 2020 (World Bank, 2004). Third, high rates of youth unemployment, particularly in the MENA region, have raised fears of social unrest and large emigration flows (Kelly, 2016). Finally, enormous progress in automation may mean that manufacturing jobs, which were vital to the growth of East Asian countries, may no longer be available for poorer countries as they develop. This has led to headlines like “Robots could eat all of Ethiopia’s jobs; South Africa, Nigeria and Angola not safe either” (Mwiti, 2016). While policy makers have long employed ALMPs and interest in their effectiveness has increased, until recently most of the evidence as to their effectiveness came from developed and transition countries, and very little from experimental evidence. For example, Dar and Tzannatos (1999) 2    cover 72 evaluations but only find Hungary and Poland among non-developed countries, and have evaluations for their programs based on matching participants with non-participants. The heavily cited update of this work by Betcherman et al. (2004) added 39 additional evaluations from developing and transition countries, of which only four drew on randomized experiments, and of which only one (Galasso et al, 2004) was published in an academic journal. The typical evaluation during this period uses propensity score matching, attempting to compare participants in an ALMP to non-participants, using a relatively small number of cross-sectional observed characteristics to compare the two groups. There is continued debate about the extent to which matching can provide reliable estimates of program impacts, but estimates are likely to be more reliable when the selection process into programs is known and multiple periods of pre-program data are available for both treatment and control (Smith and Todd, 2005), conditions few of these non-experimental evaluations satisfy. The last decade has seen growth in the number of experimental evaluations of ALMPs in developing countries. These new studies provide more rigorous evidence for the impacts of these programs, but still suffer from some of the same problems faced by many non-experimental studies such as survey attrition, the difficulty in accounting for general equilibrium effects, and concerns with the right timing over which to measure impacts. I critically survey this recent literature and draw out lessons for the effectiveness of ALMPs.1 The general message is that traditional ALMPs that focus on skill training, wage subsidies, and job search assistance have at best modest impacts in most circumstances. I compare this to expectations of program impacts from participants and policy makers, and show that they tend to have over-optimistic expectations of how beneficial these programs can be. However, revealed preference also shows that many of the formal sector manufacturing jobs that these programs are intended to foster are not that highly valued by workers. I then turn to emerging evidence on the effectiveness of less traditional active labor market policy actions. I note the promise of policies which attempt to deal with sectoral and spatial mismatches, in which workers are stuck in occupations or locations that differ from where demand is. Finally, I attempt to draw out lessons for new impact evaluations in this area, as well as concluding lessons for policy makers. 3    The Rationale and Evidence for Traditional ALMPs as a Response Traditional ALMPs are divided into three main categories. The first set of programs operate on the labor supply side, aiming to increase the employability of workers through vocational training (see McKenzie and Woodruff (2014) for a separate review of programs that foster self-employment through business training). A second set of programs operates to increase the demand for labor, through subsidizing the cost of labor to firms with employment subsidies.2 Finally, search and matching assistance programs aim to lower frictions that prevent demand meeting supply in the labor market. I discuss the key economic rationale for each type of program, and the recent empirical evidence for each. Vocational Training Vocational training programs were the most common ALMP used by governments following the global financial crisis of 2007-08 (McKenzie and Robalino, 2010). Blattman and Ralston (2015) note that the World Bank and its client governments invested nearly US$1 billion per year between 2002 and 2012 on skills training programs. The premise of such programs is that a lack of certain technical skills is the reason that particular individuals are unemployed, and that these skills can be taught and learned in a relatively short period of time. In practice, these programs are typically used with two target groups of beneficiaries. The first is to offer the program to the general population of unemployed workers. Although this is a common policy option, the only evaluation of such a program in a developing country setting is Hirshleifer et al. (2016), who conduct a randomized experiment to evaluate Turkey’s program for the unemployed. Typical programs here last a duration of 3 months, and cover a wide range of occupations. The second approach is to more narrowly focus on low-income, or “at-risk” youth, where youth can run from 15 to 24 years old, or even to 29 years depending on the country. Programs focusing on youth have been particularly common in Latin America, and act as a substitute to the formal schooling system for youth who have dropped out. The standard model in the entra21, Jovenes, and Juventud y Empleo programs in Latin America has been to combine 3 months of classroom 4    training with 2 to 3 months of on-the-job training in the form of an internship. Some programs additionally provide life skills training. There has been rapid growth in the number of randomized experiments evaluating these programs. I focus on evaluations of traditional vocational training programs – there have also been several recent evaluations of bundled packages for adolescent girls which incorporate vocational training with other services such as business skills training, empowerment activities, and support in setting up businesses or finding jobs. An example is Adoho et al. (2014) who report early findings from Liberia. They find positive impacts of that program on employment, but estimate that it would take 12 years for participants to recoup the costs of the job skills training provided in that program. Table 1 summarizes the results from 12 evaluations from 8 countries. The typical evaluation measures impacts 12 to 18 months after the conclusion of the training program, using surveys administered to the treatment and control groups. The use of surveys to measure key employment outcomes raises several concerns. The most major one is that of attrition, with all, but one study, having attrition rates of 18 percent or higher, ranging up to 46 percent in Cho et al. (2013). This attrition is a problem, because we might expect the employment outcomes of individuals who refuse to be surveyed or who cannot be found to differ from those who are interviewed. A typical approach has been to compare attrition rates in the treatment and control groups, and then do a bounding exercise if the attrition rates vary (often the control group is slightly less likely to respond). But it is easy to think of problems which can arise even when the attrition rates are the same for both groups: for example, the attritors in the treatment group may be people who went through the training and did not find it useful and have still not found jobs, while those in the control group could be those who are too busy to answer surveys because they are employed in good jobs. This type of differential response would bias the estimated treatment effect upwards, overstating the impact of training. A second issue with the use of survey measures of employment is the possibility that those in the treatment groups over-report their employment outcomes to express their appreciation for being given the program, while those in the control group potentially underreport these outcomes because they maintain some hope of still being given the program. Good survey design and survey framing can mitigate these issues. An alternative approach is then to use administrative data on employment from national social security or labor databases. These databases capture formally 5    registered employment, and enable the trajectory of formal employment outcomes to be measured over longer time windows – including up to 10 years after treatment in the case of Attanasio et al. (2015). They are not subject to attrition, but because they only capture formal employment, may overestimate the employment impacts of the program if individuals simply shift from informal to formal employment. With these caveats in mind, Table 1 then provides an overview of the evidence from these recent studies. I consider two key outcomes: paid employment, and earnings. I report here the intention- to-treat (ITT) effects estimated in the different studies. These give the impact of offering vocational training to target participants. Even though most programs require individuals to express interest and sign up for the training, not all of those selected for training complete it. In most of the programs here, between 70 and 85 percent of those selected for training complete it. The local- average-treatment effect, which is the effect of taking up training when it is offered, can then be obtained by multiplying these ITT impacts by between 1.2 to 1.4 in most cases. We see that only 3 of 9 studies find a significant impact on employment. The simple unweighted average across the studies is a 2.3 percentage point increase in employment. That is, for every 100 people offered vocational training, fewer than 3 will find a job they would not have otherwise found. The last column of Table 1 shows the cost of these programs typically ranges from $500 to $1700 per person trained (the exception being the tailoring and stitching training in India studied by Maitra and Mani (2016) which cost a remarkably low $13 per person trained). The result is approximately $17,000-$60,000 cost per additional person employed. A number of the studies have also considered formal employment as an outcome. This is of interest in its own right, because of a belief that formal employment may offer additional benefits and stability to workers, as well as being a measure that can be obtained from administrative data. Studies which have measured both employment and formal employment have tended to find slightly larger impacts on formal employment, indicating that training helps shift workers towards more formal jobs. The average impact across the studies is 3.6 percentage points. I consider the impact on earnings in terms of two measures. The first is the percentage increase in earnings relative to the earnings levels of the control group. The second is the absolute level increase in earnings relative to the control group in terms of US dollars. We see that only 2 out of 9 studies find a statistically significant impact on earnings. However, all but two show positive 6    point estimates, with a mean of a 17 percent increase and median of 11 percent. The absolute change in monthly income ranges from -$5 to $83 per month, with a mean of $19. Taken together, these studies show the promise of vocational training to have some impact on employment, but also that these impacts are modest in many cases. In order to get a sense of how to view the size of these effects, I find two perspectives useful. The first is to consider vocational training as a substitute for schooling in building human capital. Standard estimates of the return to an additional year of schooling around the world show an average return of 10 percent, with returns to tertiary schooling averaging 21 percent in Sub-Saharan Africa (Montenegro and Patrinos, 2014). From this perspective, we might expect a 3 month course to result in 3-7 percent higher earnings, and six months to result in 5 to 10 percent higher earnings. The earnings impacts in Table 1 are largely within this order of magnitude, and are consistent with there being a return to human capital, but that vocational training should not be expected to deliver much different returns from schooling itself. An exception is Maitra and Mani (2012), where the increase in income represents a 95.7 percent increase on the control group’s mean. This reflects a situation where the women in their sample are unlikely to be working and have very low earnings, so this large relative increase is a small absolute increase of only $2.40 per month. The second, more standard, perspective is that of cost-benefit. Comparing the cost of providing these programs to the monthly income gains shows that the cost of these programs averages 50 times the monthly income gain. Even adjusting for incomplete take-up (which means not having to pay the full costs for people who drop-out), it will typically take three or four years at least for participants to recoup in income gains the cost of the program. This calculation also excludes the opportunity cost of income lost by the participants during the period they are trained. The result is that cost-benefit calculations for these programs are reliant on making assumptions of the trajectories of impact lasting for periods beyond which impacts have typically been measured. Some studies which have measured impacts over multiple time periods beyond a year after training (Hirshleifer et al, 2016; Alzúa et al. 2016; Acevedo et al. 2017)3 have tended to find impacts fall over time, making the assumption that short-term gains will necessarily persist problematic (although others have found sustained impacts on formal employment for certain subgroups (Attanasio et al, 2015; Ibarrarán et al., 2015)). Further adding the need to discount the future at 7    some rate, and it is easy to arrive at the conclusion of Blattman and Ralston (2015, p. ii) that “it is hard to find a skills training program that passes a simple cost-benefit test.” In search of a more positive role for vocational training, researchers have pursued two approaches. The first is to find training programs that can be provided much more cheaply, such as the NGO program of Maitra and Mani (2016). If skills training can be delivered much cheaper, it does not need to deliver as large an income gain to be cost effective. The second approach has been to investigate whether the returns to training might be different for some subgroups of the population or training types, to argue that targeted training might work. Foremost among this has been a focus on gender, and there appears to be a stylized fact in the literature that vocational training has higher returns for women (e.g. Blattman and Ralston, 2015). This appears to stem largely from the work in Colombia by Attanasio et al. (2011, 2015), who find significant impacts on employment for women but not for men. However, they never formally test for a difference in impact by gender, and indeed in their long-term follow-up, note the magnitudes are similar for both men and women, but only statistically significant for women. Moreover, as Table 2 shows, all of the studies which have formally tested for equality by gender can either not reject that impacts have been similar for men and women, or have found significantly higher impacts for men.4 Therefore, it should not be assumed that training is generally more effective for women. Hirshleifer et al. (2016) investigate treatment heterogeneity by key characteristics of the type of training provided. They find some evidence that training is more effective when given by private providers rather than government training institutes. This is consistent with increasing emphasis in policy towards better aligning training programs with private sector demand. However, they still find that even the impacts of privately-provided training are modest and fall off over time. Finally, an important point to note with all these evaluations is that they measure the private returns to vocational training, assuming that the treatment group and control group are not competing for the same jobs. Even if this is the case, and the estimates remain internally valid, the public policy question of whether to support such programs also depends on whether trained individuals get new jobs, or crowd out non-program participants who would have otherwise taken them. None of the studies were designed to look at this question, although Hirshleifer et al. (2016) and Attanasio et al. (2015) discuss it, and examine whether impacts differ by the tightness of the labor market, 8    finding no significant differences. This offers some comfort against the displacement concern, but it still seems likely that at least some of the modest gains shown by vocational training programs come from changing who gets particular jobs, rather than from generating new employment in the economy as a whole. Wage Subsidies In a simple model of the labor market, workers are paid their marginal product, and so if young workers are not very productive to begin with, they would simply be paid low wages. Indeed, in some African contexts under the apprenticeship system, workers actually receive negative wages, paying firms for the privilege of learning on the job. However, in many labor markets, minimum wages and subsistence constraints set a lower bar on the amount firms can pay for labor, and additionally the presence of hiring and firing frictions means that if there is uncertainty about the productivity of a worker, firms may prefer not to hire them. The result is that individuals who are willing to work may be unemployed, particularly youth who are inexperienced and untested, and less able to signal productivity. Wage subsidies are intended to help overcome these causes of unemployment. A temporary wage subsidy given to a worker lowers the cost to a firm of hiring that worker (although as Levinsohn and Pugatch (2014) show, workers may increase their reservation wages in response so the cost of labor need not fall by the full amount of the subsidy). This should then lead to an increase in employment for the period the subsidy is in effect. Moreover, there are several possible ways for this short‐term subsidy to have a lasting impact on employment: the experience gained may act as a stepping stone to longer‐term employment, workers may learn on the job and increase productivity to a level above minimum wages, and firms may learn about the quality of workers and be able to keep individuals who are good matches. Three studies have evaluated the impact of wage subsidies given to workers using randomized experiments in developing countries (Table 3). The earliest was Galasso et al. (2004), who offered welfare recipients a wage subsidy voucher that was valid for up to 18 months, paying the firm up to $150 per month. However, employers had to formally register 9    any workers hired with this subsidy, and would face severance charges if they fired the worker after the program, so only three workers in the treatment group were hired using the voucher. A similar situation arose in Levinsohn et al. (2014) in South Africa, in which youth were given vouchers that would pay the firm a monthly subsidy for up to 6 months, if the firm formally registered the worker. Only 22 firms used the voucher, hiring only 30 workers out of 1,500 given the voucher. Both studies show the reluctance of firms to face the labor regulations associated with hiring workers. In contrast, Groh et al. (2016a) in Jordan did not require firms to formally register the worker, following the norms of the labor market in which most employment was informal. Their subsidy was also valid for six months. Half of the individuals given the voucher in their study used it, and there was a 38 percentage point increase in employment during the period the subsidy was in effect. However, as detailed in Figure 1, once the subsidy ended this treatment effect disappeared quickly, as firms fired workers, other workers quit, and the control group caught up a little. The result was no long‐term significant impact on employment. Subsidies did not provide the stepping stone to additional work that theory might suggest. Despite the lack of use of the vouchers in the Argentina and South Africa experiments, both studies do report significant impacts on wage employment (although no overall impact on employment in the Argentina case). The authors of both studies speculate that having the voucher gave job‐seekers the confidence to approach more employers and exert more job search effort, which resulted in more employment, just of an informal sort. If true, this would make the policy very cost‐effective, since hardly anyone cashed in the voucher. However, note that the attrition rates are high in both studies (23 percent in Argentina, and 39 percent by the 2 year follow‐up in South Africa). The South African study has a higher point estimate at 2 years than 1 year, but then shows the treatment effect decreasing over time when restricted to the sample present in both follow‐up years. It seems highly likely that the employment outcomes of the attritors are different from those who responded to the survey, so that extreme caution should be used in interpreting these treatment effects. 10    Moreover, as with vocational training, a key concern is that any gains to those receiving the vouchers come at the expense of others in the economy who would have otherwise been hired. Groh et al. (2016a) find suggestive evidence of this in Jordan. When they examine impacts by region, and look at longer‐term time trends, they find a lasting impact of the subsidy on employment in the less populated labor markets outside of Amman. But the control group ends up with a lower employment rate than other cohorts of graduates had received in recent years, and direct survey evidence suggests that they were competing directly with the treatment group for some jobs. The result is that wage subsidies do not seem likely to have increased aggregate employment in this case. An alternative to giving subsidies to workers has been to give the subsidies to firms, to encourage them to hire more workers. De Mel et al. (2016) test the impact of wage subsidies given to microenterprises to encourage them to hire workers. They find 24 percent of firms use the subsidy to hire a worker, resulting in an increase in employment while the subsidy is in effect. But the dynamics then look reasonably similar to those in Figure 1, with much of this impact disappearing as soon as the subsidy is removed, and no long‐term impact after two years. A final use of subsidies is to use them to help prevent liquidity‐constrained firms from shedding workers during a temporary shock. This type of policy was another common response to the global financial crisis. The idea was that firms suffering a temporary demand shock and/or liquidity shock may fire workers who they would later want to hire back. A subsidy may prevent them from firing these workers in the first place, and hasten the recovery of these firms if hiring and firing is costly. Bruhn (2016) evaluates a wage subsidy program Mexico used during the global financial crisis, using difference‐in‐difference analysis to compare the employment trajectories of durable manufacturers in industries eligible for the program to those in industries ineligible for the program. She finds employment to be 6 to 13 percent higher in the affected industries during the program, and to grow faster after the crisis, suggesting the program helped firms to recover more quickly from the shock. 11    This accumulated evidence suggests that wage subsidies are unlikely to be very effective in generating additional employment under standard labor market conditions, and may also even not be very effective in playing a distributional role in determining which individuals get to access jobs. However, it also suggests two potential use cases. The first is during conditions of large, temporary shocks. Even if ALMPs like wage subsidies have no lasting impacts, from a social protection viewpoint if they help households smooth temporary shocks, then this might be justification enough for their use. The difficulty here, of course, is in knowing whether or not the shocks are temporary or structural in nature, since there is a danger in trying to maintain employment in industries that economic shocks make permanently less competitive. Secondly, the evidence suggests that wage subsidies may be useful for temporary employment creation. This might be important particularly in fragile economies, where large youth unemployment raises other concerns. In this vein, short‐term evidence from Yemen (McKenzie et al, 2016) showed positive impacts of a youth internship program which subsidized firms to take on interns, although the outbreak of war prevented analysis of any lasting impacts. Search and Matching Assistance Many governments provide employment services in the form of helping job-seekers with preparing resumes, hosting labor exchanges, and helping to match firms with workers seeking employment. The review of Betcherman et al. (2004) was relatively favorable of these types of programs, noting that since the costs are often low for providing such services, the cost-benefit ratios can be favorable. However, this recommendation was largely based on developing country evidence, and the review also noted, based on non-experimental evaluations from Brazil and Uruguay, that such programs may be less effective in countries with large informal sectors if workers typically use other channels to find jobs, and if they work at all, might work best for more educated job-seekers. A competing view to this concern is that search and matching frictions may be greater in developing countries, leaving more scope for improvements. The educational systems in many countries may not be very good at signaling quality, and may teach content that is very different from the skills employers are looking for. Information about vacancies may be more difficult to come by if workers and firms are not all online, and match quality may be worse if informal 12    networks are relied upon to fill vacancies. Improving this process could then reduce unemployment directly (by filling existing vacancies) as well as indirectly (by lowering hiring costs so firms create more vacancies). Table 4 summarizes the results of 9 recent randomized experiments which have tested various interventions designed to reduce information and search frictions, and to better match workers and firms together. These incorporate several types of specific interventions. The work that tests public intermediation services most directly is Dammert et al. (2015), who worked with the public service provider in Peru to test whether providing information about job vacancies to registered job seekers improves employment, and additionally whether sending these announcements by SMS message helps further. Another example of providing information about job opportunities and recruiting services is Jensen (2012), who connected rural villages in India to experienced recruiters at the start of the business process outsourcing boom in India, providing information about this new sector. Two studies (Beam, 2016; Abebe et al, 2016b) test the impact of job fairs which bring firms and workers together. The idea here is to give both firms and workers the opportunity to assess a large number of possible matches at the same time, and become better informed about the range of job opportunities and worker types. Two studies (Franklin (2015), Abebe et al. (2016a)) test the impact of reducing the monetary costs of search for job seekers by offering transport subsidies to allow them to travel to a different part of town where job opportunities are more commonly displayed. The final approach used in four studies is to try and reduce the information frictions faced by firms by providing more information about job-seekers. Abel et al. (2016) approach this by developing a standardized reference letter format, and encouraging job-seekers to get this reference from former employers. Groh et al. (2015), Abebe et al. (2016a), and Bassi and Nansamba (2017) instead develop their own tests of a variety of soft and hard skills that might otherwise be difficult for firms to observe, but which firms say they find valuable. Examples include information about math ability, creativity, teamwork, attendance rates, and communication skills. These types of programs tend to be much cheaper than vocational training and wage subsidies (if taken up) in terms of cost per person invited to participate. The last column shows that all but one of the studies that provide cost information have costs of $25 or lower per person assisted. That is, the costs are one-fiftieth to one-hundredth of the cost of vocational training programs. The 13    exception is Groh et al. (2015) who had a cost of $203 per person, since their enrollment and testing procedure was rather expensive. These lower costs certainly lower the bar in terms of treatment impacts needed in order for these programs to pass cost-benefit tests. However, as seen in Table 4, out of the 10 different interventions, only one (Jensen, 2012) finds a significant impact on employment, increasing employment by 2.4 percentage points over 3 years. Dammert et al. (2015) find their intermediation services tend to speed up the process of finding a job, with a significant employment impact after 1 month, but by 3 months the control group has caught up. Many of the other studies have small, but positive, point estimates, with an average impact across the studies of 2.7 percentage points. However, it is also worth noting that, apart from Jensen (2012), none of the studies measures impacts beyond a year, so cannot measure whether there is any sustained employment impact. A number of studies de-emphasize employment as an output, claiming their intervention helps in improving the quality of jobs. They examine quality in different ways, sometimes defining quality jobs as “permanent” or “formal” or simply as “wage employment” rather than self-employment. For example, Beam (2016) finds attending a job fair results in a 10 percentage point increase in formal employment, that is matched by a reduction in self-employment; Franklin (2015) finds more positive impacts on permanent employment and being employed in an office than on total employment; and Abebe et al. (2016a) find their job application workshop, which certified skills and provided interview preparation, led to a 6.9 percentage point increase in permanent employment. However, there are two problems with justifying these programs on the basis of improved “job quality.” The first is that there is a long and large literature that debates the extent to which informality and self-employment are choices made by individuals, that have benefits associated with them such as flexible labor hours and less taxation, rather than reflecting exclusion from formal wage jobs (e.g. Maloney, 2004). Indeed, Abebe et al. (2016a) find no significant change in job satisfaction from their treatment, despite the change to permanent employment. Secondly, as seen in Table 4, none of these interventions shows a significant impact in labor earnings. While the confidence intervals are wide in many cases, and therefore allow the possibility of these interventions passing cost-benefit tests, the short time horizons and lack of significant impact on earnings means that there is currently no evidence that they do. 14    A further point to note is how few direct hires occur through many of these interventions, and how an important share of job offers are turned down by job-seekers. Groh et al. (2015) made more than 1,000 matches between firms and workers. Youth rejected the opportunity of an interview 28 percent of the time, and when a job offer was received, they rejected the job offer or quit quickly 83 percent of time, resulting in only 9 hires that lasted one month. Bassi and Nansamba (2017) report that only 2-4 percent of their job matches resulted in a worker being hired, and few workers hired were still employed at the firm at their follow-up. Abebe et al. (2017b) invited 1,007 people to their job fairs, 606 attended, but only 76 job offers were made and 14 people were hired. Beam (2016) reports only two respondents from her job fair were working for one of the employers that attended the fair at endline. As such, while the cost per person invited to treatment can be low, the cost per individual actually placed in a job can be substantially higher – Groh et al. (2015) estimate a cost of $22,000 in their case. Several studies note that programs which allow workers to better certify their skill levels may have differential effects for those with low and high skills. Being able to signal your skills can be good if you have high skill levels, but disadvantageous if your skill levels are below those of other job- seekers. The result might be better quality workers for firms, but simply a reallocation of who gets work from less- to more-skilled workers. What Do Policy Makers Expect of Such Programs and What Does Revealed Preference Show? The above discussions show that traditional active labor market programs have had at most modest impacts on employment in most cases, with a typical intervention leading to a 2 percentage point increase in employment that is usually not statistically significantly different from zero. Cost- benefit calculations usually rely heavily on extrapolating statistically insignificant total earnings gains over periods well beyond the timeframe of the study. These impacts are much lower than expected by policy makers and program participants in many cases. Hirshleifer et al. (2016) show this formally in the context of their vocational training experiment in Turkey. There was strong demand for this training from participants, with courses oversubscribed by a factor of two or more. Subjective expectations of the employment impact of the program elicited from participants show that they expected a 32 percentage point increase in the likelihood of employment, while staff in the government employment office expected the 15    training to increase the likelihood of employment by 24 percentage points. These expectations far exceed the actual impact of 2 percentage points seen in Table 1. Groh et al. (2016b) likewise show that policy makers in Jordan expected the wage subsidy program to have lasting impacts on youth employment, in contrast to the realized impacts. Economists are also not immune to this tendency to think active labor market programs will be more effective than they typically are. A first testament to this comes from a number of the studies covered in this review being interventions designed by the researchers themselves, in addition to those evaluating programs that governments were already going to implement. Secondly, Groh et al. (2016b) carried out an expectations elicitation exercise when presenting the results of their Jordan wage subsidy research. They find that development economists on average expected a 10 percentage point increase in employment after the subsidy had ended, compared to the 2.8 percentage point increase seen in Table 3. However, while revealed preference shows that there are participants who think these programs will be effective and therefore choose to participate in them, revealed preference also suggests that the types of formal jobs and manufacturing jobs that many of these programs think of as “successful” outcomes are not that valued by job-seekers. For example, Blattman and Dercon (2016) randomize job-seekers into industrial jobs in large formal firms in Ethiopia, and find that almost a third of people offered a job quit in the first month, and 77 percent within the first year, and that workers experienced health problems from staying in this work. Similarly, Adhvaryu et al. (2016) find female garment workers in India to have very high quit rates, losing almost 80 percent of the workers in their study over two years. These high rates of turnover are not consistent with formal jobs being so valuable and desired that workers never want to leave once they attain such positions. The implicit assumption behind search and matching interventions in particular is often that search frictions make it costly and difficult for firms to find workers. Simple queries of firms often find firms saying that they find it hard to find the right workers. But one also sees firms being reluctant to raise wages or spend more money in getting better matches. Groh et al. (2015) conducted a survey in Jordan where they tracked firms as they opened up job vacancies, and found that only 6 percent of positions required more than 4 weeks to find a new employee, and most firms could, in fact, fill jobs quite quickly. De Mel et al. (2016) similarly find firms in Sri Lanka say it would take 16    7 days on average to fill positions. If it were particularly costly for firms to find and recruit workers, we might expect a range of market solutions to emerge to help them lower these costs. Indeed, there are a range of human resources consultants and executive talent firms that help firms fill skilled and unusual positions. But the lack of an existing market alternative to many of the interventions being trialed may suggest that firms do not face large search costs for other entry- level positions. What Types of Alternative Policies Show Promise? Given the continued pressure for governments to be seen to be doing something to help people find jobs, this lack of empirical evidence for the effectiveness of many traditional programs is unlikely to be enough to cause them to be abandoned unless better alternatives can be found instead. What might these alternatives be? One set of alternative policies is to move away from interventions on the labor supply side and focus more on policies to help firms overcome the obstacles they face in innovating, growing and creating more jobs. Such private sector development programs also have a mixed record of success, but there are examples (e.g. McKenzie, 2016 and the references therein) of programs that have generated new jobs. A related approach is to help firms overcoming onerous regulations and labor laws that limit their hiring. Bertrand and Crépon (2016) find that teaching South African firms about labor laws and providing legal support to help them deal with these laws spurred new employment generation. On the labor supply side, the most promising interventions appear to be ones that help workers access different labor markets, overcoming sectoral and, especially, spatial mismatches. Sectorial mismatches arise when people are trapped in the wrong occupations as trade and technology changes the demand for labor, or because of gender-segregation in society. Campos et al. (2016) show that in Uganda that women who cross-over into male-dominated industries make three times as much as women who remain in female-dominated industries. Hendra et al. (2016) report that a demand-driven training program in the U.S. that aimed to train the unemployed in sectors which were in demand resulted in a 14 percent income gain after two years. However, they also note that these programs can be complex to run and need experienced providers. 17    The largest market failures in labor markets occur across space, with very different employment opportunities for the same skills depending on where individuals are located. We have seen some of the more successful screening and matching interventions were ones that provided assistance with learning about job opportunities in a different location (Jensen, 2012), or subsidizing job search in different parts of the city (Franklin, 2015; Abebe et al. 2016a). More striking evidence comes from Bryan et al. (2014) who show a small subsidy equal to the cost of a bus ticket spurred new seasonal migration in Bangladesh, increasing household consumption by 30-35 percent during the hungry season (they do not measure household income). Even larger gains can be had from facilitating international migration. Gibson and McKenzie (2014) show that sending seasonal workers to New Zealand increased per-capita incomes in Tonga and Vanuatu by more than 30 percent. Luthria and Malaulau (2013) discuss the process of facilitation used by governments and the World Bank to allow this movement to happen. However, such facilitation is not always successful, especially if it focuses only on barriers on the worker side. For example, Beam et al. (2016) conducted several interventions in the rural Philippines to facilitate more international migration, and were unsuccessful in generating additional international employment. Concluding Lessons for Impact Evaluations The modal study surveyed in this review is from 2016, reflecting rapid recent growth in the body of evidence around active labor market interventions in developing countries. This body of work has generated substantial new knowledge, but also suffers from several limitations that future work can attempt to learn from: 1) Given the likely effect size of active labor market interventions, sample sizes may need to be a lot larger. Based on the current body of research, it seems many interventions may have only a modest impact on employment, such as a 2 percentage point increase. In some cases, such as expensive training programs, such an effect size is too small to be economically meaningful. But cheaper programs such as search and matching assistance could still deliver gains that exceed the costs with these modest impacts. Taking as an example the 13 percent employment rate in the control group of Abel et al. (2016), a study needs to have 6,424 individuals in the treatment group and 6,424 in the control group to detect a 2 percentage point employment impact with 80 percent statistical power. This is much larger than existing studies. 18    2) Measuring impacts over longer-time frames: the returns to these programs will differ substantially if they merely speed up the process of gaining employment versus having lasting impacts. Yet most studies measure impacts over at most 1 to 2 years, leaving them to speculate about cost-benefit on the basis of assumptions about how impacts vary over time. Tracking impacts over longer-time periods is therefore needed. Studies which link participants to administrative records (such as Attanasio et al, 2015) offer one promising way to do this. 3) Limiting attrition: When the likely impact on employment is only 2 to 3 percentage points, and attrition rates are 10, 20, or even more than 30 percent, any treatment effects are dwarfed by attrition, and bounds that incorporate this attrition will be completely uninformative. Limiting attrition is particularly difficult given that so many ALMPs focus on youth, who tend to be more mobile and difficult to track over time. Serious investment in limiting attrition, combined with the use of administrative data is needed. 4) Continued and improved careful measurement of costs: I was pleasantly surprised by the number of studies which did report the costs of the intervention, although a number still lack this key information. More work is needed to make clear average versus marginal costs in understanding the cost structure as pilot programs expand. 5) Pre-specifying outcomes and heterogeneity: a number of studies fail to find a significant impact on either employment or earnings, but then emphasize impacts on a particular subgroup (such as one gender, or one skill level) or for one outcome (such as formal employment). Pre-specification of the primary outcomes and key heterogeneity of interest lessens concerns about multiple hypothesis testing. 6) Testing placebo effects: Several studies find impacts despite almost no direct hires through the program they study (e.g. Beam (2016), Galasso et al. (2004), Levinsohn et al. (2014)). These studies raise the possibility that simply doing anything to support job-seekers may encourage them to keep exerting effort and search, so that what matters is their sense that someone wants them to succeed, not the particular policy pursued.5 Testing more formally this sort of placebo effect would be interesting in further work. 7) Understanding general equilibrium better: a key concern with many of these policies directed at particular job-seekers is that they merely change who gets the jobs firms are advertising, without increasing the total number of jobs available. The ideal would be 19    approaches like Crepón et al.’s (2013) experiment in France, which randomized at the local labor market level. Abebe et al. (2016a) attempt this within clusters in Ethiopia. A second approach is to randomize also at the firm level, as in Groh et al. (2015) and Abebe et al. (2016b) to attempt to measure if firms increase hiring. Further methodological work to develop additional ways to examine these spillovers is needed. Concluding Lessons for Policy Given the importance of jobs for poverty reduction, productivity growth, and social cohesion (World Bank, 2012), it is no surprise that policy makers have actively pursued policies to try to help job-seekers find jobs. But as this review has shown, an emerging body of evidence shows these policies to generally be far less effective than policy makers, program participants, and economists typically expect. It should be noted that this is not unique to ALMPs in developing countries: Crépon and van den Berg (2016) in their review of largely developed country evidence conclude that “the general outlook for ALMPs is rather grim.” One reason for this lack of effectiveness is a positive one: labor markets (at least in urban areas) in developing countries actually appear to work a lot better than is sometimes thought. It is easy to imagine all types of constraints that might inhibit the functioning of labor markets, but in practice firms appear to be able to fill many vacancies quite quickly, and workers turn down many job opportunities and quit jobs frequently in pursuit of better opportunities. These facts do not suggest workers and firms have great difficulties meeting one another, or that job-matches are so rare and scarce that workers cling to every job opportunity they receive. There may be other constraints that limit the number of jobs created, such as high minimum wages and inflexible labor laws, or lack of access to financing and infrastructure that prevent firm growth, but the solution to these issues lies outside of active labor market policies. Nevertheless, while this suggests less of a role for traditional active labor market policies, there still appears to be significant scope for improvements in dealing with structural and spatial mismatches in labor. As the evidence here has shown, not everything that policy makers try works, and so these new policy innovations should be piloted against competing alternatives and accompanied by rigorous impact evaluations in order to test different approaches. 20    References Abebe, Girum, Stefano Caria, Marcel Fafchamps, Paolo Falco, Simon Franklin and Simon Quinn (2016a) “Anonymity or Distance? Experimental Evidence on Obstacles to Youth Employment Opportunities”, Mimeo. Stanford University. Abebe, Girum, Stefano Caria, Marcel Fafchamps, Paolo Falco, Simon Franklin, Simon Quinn and Forhad Shilpi (2016b) “All the fun of the (job) fair: Matching firms and workers in a field experiment in Ethiopia”, Mimeo. Abel, Martin, Rulof Burger, and Patrizio Piraino (2016) “The Value of Reference Letters: Experimental Evidence from South Africa”, Mimeo. Harvard University. Acevedo, Paloma, Guillermo Cruces, Paul Gertler and Sebastián Martínez (2017) “Living Up to Expectations: How Job Training Made Women Better Off and Men Worse Off”, Mimeo. Adhvaryu, Achyuta, Namrata Kala and Anant Nyshadham (2016) “The Skills to Pay the Bills: Returns to On-the-job Soft Skills Training”, Mimeo. University of Michigan. Adoho, Franck, Shubha Chakravarty, Dala Korkoyah, Mattias Lundberg and Afia Tasneem (2014) “The Impact of an Adolescent Girls Employment Program: The EPAG Project in Liberia”, World Bank Policy Research Working Paper no. 6832. Alzúa, María Laura, Guillermo Cruces and Carolina Lopez (2016) “Long-run effects of youth training programs: Experimental Evidence from Argentina”, Economic Inquiry 54(4): 1839-59. Attanasio, Orazio Arlen Guarín, Carlos Medina and Costas Meghir (2015) “Long-term impacts of vouchers for vocational training: Experimental evidence for Colombia”, NBER working paper no. 21390. Attanasio, Orazio, Adriana Kugler, and Costas Meghir (2011). ‘Subsidizing vocational training for disadvantaged youth in Colombia: evidence from a randomized trial’, American Economic Journal: Applied Economics, 3(3): 188–220. Bassi, Vittorio and Aisha Nansamba (2017) “Information Frictions in the Labor Market: Evidence from a Field Experiment in Uganda”, Mimeo. UCL. Beam, Emily (2016) “Do job fairs matter? Experimental evidence on the impact of job fair attendance”, Journal of Development Economics 120: 32-40. Beam, Emily, David McKenzie and Dean Yang (2016) “Unilateral Facilitation Does Not Raise International Labor Migration from the Philippines”, Economic Development and Cultural Change, 64(2): 323-68. Bertrand, Marianne, and Bruno Crépon (2016) “Teaching labor laws: Results from a randomized control trial in South Africa”, Mimeo. CREST. Betcherman, Gordon, Karina Olivas and Amit Dar (2004) “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries”, World Bank Social Protection Discussion Paper Series no. 29142. 21    Blattman, Christopher and Stefan Dercon (2016) “Occupational choice in early industrializing societies: Experimental evidence on the income and health effects of industrial and entrepreneurial work”, Mimeo. University of Chicago. Blattman, Christopher and Laura Ralston (2015) “Generating employment in poor and fragile states: Evidence from labor market and entrepreneurship programs”, Mimeo. Chicago. Bruhn, Miriam (2016) “Can Wage Subsidies Boost Employment in the Wake of an Economic Crisis?”, World Bank Policy Research Working Paper no. 7607. Bryan, Gharad, Shyamal Chowdhury, and Ahmed Mushfiq Mobarak (2014) “Underinvestment in a Profitable Technology: The Case of Seasonal Migration in Bangladesh.” Econometrica 82(5):1671–1748 Campos, Francisco, Markus Goldstein, Laura McGorman, Ana Maria Boudet and Obert Pimhidzai (2016) “Breaking the Metal Ceiling: Female Entrepreneurs who Succeed in Male-Dominated Sectors”, World Bank Policy Research Working Paper no. 7503. Card, David, Pablo Ibarrarán, Ferdinando Regalia, David Rosas, and Yuri Soares (2011) “The Labor Market Impacts of Youth Training in the Dominican Republic.” Journal of Labor Economics, 29(2): 267–300. Card, David, Jochen Kluve and Andrea Weber (2015) “What Works? A Meta-Analysis of Recent Active Labor Market Program Evaluations”, NBER Working Paper no. 21431 Cho, Yoonyoung, Davie Kalomba, Ahmed Mushfiq Mobarak and Victor Orozco (2013) “Gender Differences in the Effects of Vocational Training: Constraints on Women and Drop-out Behavior”, IZA Discussion Paper no. 7408. Crepón, Bruno, Esther Duflo, Marc Gurgand, Roland Rathelot and Philippe Zamora (2013) “Do labor market policies have displacement effects? Evidence from a clustered randomized trial”, Quarterly Journal of Economics 128(2): 531-80. Crepón, Bruno and Gerard van den Berg (2016) “Active Labor Market Policies”, Annual Review of Economics 8: 521-46. Dammert, Ana, Jose Galdo and Virgilio Galdo (2015) “Integrating mobile phone technologies into labor-market intermediation: a multi-treatment experimental design”, IZA Journal of Labor and Development 4:11 Dar, Amit and Zafiris Tzannatos (1999) “Active Labor Market Programs: A Review of the Evidence from Evaluations”, World Bank Social Protection Working Paper. De Mel, Suresh, David Mckenzie and Christopher Woodruff (2016) “Labor Drops: Experimental Evidence on the Return to Additional Labor in Microenterprises”, World Bank Policy Research Working Paper no. 7924 22    Diaz, Juan Jose and David Rosas (2016) “Impact Evaluation of the Job Youth Training Program ProJoven”, IADB Working Paper no. 693. Franklin, Simon (2015) “Location, Search Costs and Youth Unemployment: A Randomized Trial of Transport Subsidies in Ethiopia”, CSAE Working Paper WPS/2015-11 Galasso, Emanuela, Martin Ravallion and Augustin Salvia (2004) “Assisting the Transition from Workfare to Work: A Randomized Experiment”, Industrial and Labor Relations Review 57(5): 128-42. Gibson, John and David McKenzie (2014) “The Development Impact of a Best Practice Seasonal Worker Policy”, Review of Economics and Statistics 96(2): 229-243. Groh, Matthew, Nandini Krishnan, David McKenzie and Tara Vishwanath (2016a) “Do wage subsidies provide a stepping stone to employment for recent college graduates? Evidence from a Randomized Experiment in Jordan” Review of Economics and Statistics 98(3): 488-502. Groh, Matthew, Nandini Krishnan, David McKenzie and Tara Vishwanath (2016b) “The impact of soft skills training on female youth employment: Evidence from a randomized experiment in Jordan” IZA Journal of Labor and Development 5:9. Groh, Matthew, David McKenzie, Nour Shammout, and Tara Vishwanath (2015) “Testing the importance of search frictions and matching through a randomized experiment in Jordan”, IZA Journal of Labor Economics, 4:7, 2015 Hendra, Richard, David Greenberg, Gayle Hamilton, Ari Oppenheim, Alexandra Pennington, Kelsey Schaberg, and Betsy Tessler (2016) “Encouraging Evidence on a Sector-Focused Advancement Strategy: Two-Year Impacts from the WorkAdvance Demonstration”, MDRC Washington D.C. Hirshleifer, Sarojini, David McKenzie, Rita Almeida and Cristobal Ridao-Cano (2016) “The Impact of Vocational Training for the Unemployed: Experimental Evidence from Turkey”, Economic Journal 126: 2115-2146. Honorati, Maddalena (2015) “The Impact of Private Sector Internship and Training on Urban Youth in Kenya”, World Bank Policy Research Working Paper no. 7404. Ibarrarán, Pablo, Jochen Kluve, Laura Ripani and David Rosas (2015) “Experimental Evidence on the Long-Term Impacts of a youth training program”, IDB Working Paper no. 657. Ibarrarán, Pablo, Laura Ripani, Bibiana Taboada, Juan Miguel Villa and Brigida Garcia (2014) “Lifeskills, employability, and training for disadvantaged youth: Evidence from a randomized evaluation design”, IZA Journal of Labor and Development 3:10 Jensen, Robert (2012) “Do labor market opportunities affect young women's work and family decisions? Experimental evidence from India”, Quarterly Journal of Economics 127 (2), 753–792. 23    Kelly, Tom (2016) “Middle East and North Africa jobs crisis ‘may import terror into Europe’”, Daily Mail, 26 May, http://www.dailymail.co.uk/news/article-3611803/Middle-East-North- Africa-jobs-crisis-import-terror-Europe.html Kumar, Manoj and Douglas Busvine (2014) “Business looks to Modi to defuse India’s jobs time bomb”, Reuters, May 4. http://www.reuters.com/article/us-india-election-employment- idUSBREA4309B20140504 Levinsohn, James and Todd Pugatch (2014) “Prospective Analysis of a Wage Subsidy for Cape Town Youth”, Journal of Development Economics 108:169-183. Levinsohn, James, Neil Rankin, Gareth Roberts and Volker Schöer (2014) “Wage Subsidies and Youth Employment in South Africa: Evidence from a Randomized Control Trial”, Stellenbosch Economic Working Paper 02/14. Luthria, Manjula and Mai Malaulau (2013) “Bilateral labor agreements in the Pacific: A Development-Friendly Case Study” pp. 129-147 in Sebastian Saez (ed.) Let Workers Move: Bilateral Labor Agreements to Increase Trade in Services. World Bank, Washington D.C. Maitra, Pushkar and Subha Mani (2016) “Learning and Earning: Evidence from a Randomized Trial in India”, Labour Economics, forthcoming. Maloney, William (2004) “Informality Revisited”, World Development 32(7): 1159-1178. McKenzie, David (2016) “Identifying and Spurring High-Growth Entrepreneurship: Experimental Evidence from a Business Plan Competition”, Mimeo. World Bank. McKenzie, David, Nabila Assaf and Ana Paula Cusolito (2016) “The Demand for, and Impact of, Youth Internships: Evidence from a Randomized Experiment in Yemen”, IZA Journal of Labor and Development, 5:1. McKenzie, David and David Robalino (2010) “Jobs and the Crisis: What has been done, and where to go from here?”, Viewpoint https://openknowledge.worldbank.org/bitstream/handle/10986/11076/585760VP03251j10BOX3 53808B01PUBLIC1.pdf?sequence=1 McKenzie, David and Christopher Woodruff (2014) “What are we learning from business training evaluations around the developing world?”, World Bank Research Observer, 29(1): 48-82. Mohammed, Omar (2015) “Africa has the World’s fastest-growing labor force but needs jobs growth to catch up” Quartz Africa, November 13, https://qz.com/547929/africa-has-the-worlds- fastest-growing-labor-force-but-needs-jobs-growth-to-catch-up/ Montenegro, Claudio and Harry Patrinos (2014) “Comparable Estimates of Returns to Schooling Around the World”, World Bank Policy Research Working paper no. 7020. Mwiti, Lee (2016) “Robots could eat all of Ethiopia’s jobs; South Africa, Nigeria, and Angola not safe either”, Mail and Guardian Africa, 31 January, http://mgafrica.com/article/2016-01-28-look- away-ethiopia-south-africa-and-nigeria-the-robots-are-coming-for-your-jobs 24    Smith, Jeffrey and Petra Todd (2005) “Does Matching Overcome Lalonde’s Critique of Non- experimental Evaluations”, Journal of Econometrics 125: 305-353. World Bank (2004) Unlocking the Employment Potential in the Middle East and North Africa: Toward a New Social Contract. World Bank, Washington D.C. http://documents.worldbank.org/curated/en/343121468753030506/pdf/288150PAPER0Unlockin g0employment.pdf World Bank (2012) World Development Report 2013: Jobs. World Bank: Washington, D.C. 25    Table 1: Summary of Vocational Training Program Impacts Impacts on: Country Study Population Sample Size Attrition Time Frame Employment Formal Employment Earnings Formal Earnings Monthly income Cost Turkey Hirshleifer et al. (2016) Unemployed 5,902 6% 1 year 2.0 2.0 5.8 8.6 US$11.5 US$1700 [‐0.5, 4.4] [‐0.4, 4.4] [‐2.3, 13.8] [‐0.5,17.7] Unemployed 0% 2.5 years n.r ‐0.1 n.r ‐0.8 ‐US$3 [‐3.3, 1.5] [‐7.9, 6.3] Argentina Alzúa et al. (2016) Low‐income 407 0% 18 months n.r. 8.0 n.r. 64.9 US$83 US$1722 Youth [0.7, 15.3] [17.1, 112.7] Low‐income  0% 33 months n.r. 4.3 n.r. 23.1 US$45 Youth [‐3.6, 12.1] [‐15.3, 61.5] Colombia Attanasio et al. (2011) Low‐income 4,350 18.5% 14 months 4.5 6.4 11.6 27.1 US$12.8 US$750 Youth [1.0, 8.0] [3.2, 9.6] [4.5, 18.7] [12.8, 41.3] Attanasio  et al. (2015) Low‐income  0% up to 10 years n.r. 4.2 n.r. 13.6 US$17.7 Youth [1.8, 6.6] [5.5, 21.8] Dominican Card et al. (2011) Low‐income  1,556 38% 12 months 0.7 2.2 10.8 n.r. US$10 US$330 Republic Youth [‐4.6, 6.0] [‐2.3, 6.7] [‐4.2, 25.7] Ibarrarán et al. (2014) Low‐income  5,000 20% 18 to 24 months ‐1.3 1.8 6.5 n.r. US$8.5 US$700 Youth [‐4.8, 2.2] [‐0.3, 3.9] [‐4.8, 17.9] Ibarrarán et al. (2015) Low‐income  5,000 34% 6 years ‐1.4 2.6 ‐1.9 n.r. ‐US$2.3 US$700 Youth [‐4.4, 1.6] [‐0.5, 5.5] [‐10.0,6.3] Acevedo et al. (2017) Low‐income  2,779 17.6% 3 years 0.7 n.r. n.r. (a) n.r. n.r. n.r. Youth [‐4.0, 5.3] India Maitra and Mani (2012) Low income 658 25% 18 months 8.1 n.r. 95.7 n.r. US$2.4 US$13 Women [2.2, 14.0] [5.6, 186.0] Kenya Honorati (2015) Low‐income 2,100 23% 14 months 5.6 n.r 29.7 n.r. US$47.5 US$1150 Youth [0.9, 10.3] [‐2.9, 62.3] Malawi Cho et al. (2013) Low‐income 1,900 46% 4 months n.r. n.r. ‐19.6 n.r. ‐US$5 n.r. Youth [‐63.9,24.7] Peru Diaz and Rosas (2016) Low‐income 4,509 35% 36 months 1.6 3.8 13.4 n.r. n.r. US$420 Youth [‐3.3,6.5] [0.3, 7.3] [‐17.6, 44.4] 7,151 0% 36 months n.r 4.5 n.r n.r [‐0.1, 9.0] Notes: Timeframe refers to time since the end of the intervention before measuring follow‐up outcomes. n.r. denotes not recorded. Estimates are the Intention‐to‐Treat estimates reported in different studies. 95 percent confidence intervals shown in parentheses. (a) no impact on unconditional earnings reported. A negative and statistically significant impact on earnings conditional on working is reported. Impacts on employment are in terms of percentage points, impacts on earnings in terms of percentage growth relative to control mean. When study reports results for subgroups only, a weighted average is used to present the overall effect. 26    Table 2: The Vocational Training Works Better for Women Myth Country Study Findings Turkey Hirshleifer et al. (2016) Can't reject equality of impacts by gender Impacts only significant for males aged 25 and older Argentina Alzúa et al. (2016) Impacts for men statistically different from women Impacts are only significant for men Colombia Attanasio et al. (2011) Does not test for equality by gender Impacts only significant for women Attanasio  et al. (2015) Does not test for equality by gender Impacts more significant for women Dominican Card et al. (2011) Can't reject equality of impacts by gender Republic No significant impact for either gender Ibarrarán et al. (2014, 2015) Does not test for equality by gender Significant impact on formal employment for men Acevedo et al. (2017) Does not test for equality by gender No significant long‐run impact for either gender Kenya Honorati (2015) Does not test for equality by gender Finds significant impacts for both men and women Malawi Cho et al. (2013) Can't reject equality of impacts by gender No significant impact for either gender Peru Diaz and Rosas (2016) Does not test for equality by gender Some significant impacts on formal employment for both 27    Table 3: Summary of Wage Subsidy Impacts Proportion using Impact on Country Study Population Sample Size Attrition Time Frame In Effect Subsidy Employment Earnings Argentina Galasso et al. (2004) Welfare recipients 548 22.5 18 months Yes 0.011 1.7 n.r. [a] Jordan Groh et al. (2016a) Female community 1349 8 6 months Yes 0.503 38.4 228.3 college graduates [33.3, 43.5] [197, 260] 4 14 months No 2.8 15.9 [‐3.4, 9.1] [‐14,46] 8 27 months No 1.4 14.0 [‐3.2, 9.8] [‐17,45] South Africa Levinsohn et al. (2014) Youth 3064 23.0 12 months No 0.02 7.4 14 [2.9, 11.9] [‐9,37] 39.2 24 months No 9.5 ‐19 [3.6,15.4] [‐72,34] Notes: a. Paper did not provide standard errors, but this impact was not statistically significant. Time Frame refers to timing since subsidy begun. In Effect denotes whether subsidy still being paid at time of survey. n.r. denotes not reported South African sample started with 4,009, but waited one year and re‐interviewed before starting intervention. Estimates here based on 2010 sample. Impacts on Employment are in terms of percentage points, on Earnings are Percent increase on Control Mean. 28    Table  4: Evidence  on Search and Matching Assistance Attrition Impact on Country Study Type of Assistance Population Sample Size (%) Time Frame Employment Earnings Cost Ethiopia Abebe  et al. (2016a) transport subsidy young 2097 6.5 8 months 4.0 0.1 $7.5 Job‐seekers [‐1.9,9.9] [‐ 7,7] certifying hard skills young 1778 6.5 8 months 2.0 5.4 n.r. + interview assistance Job‐seekers [‐4.0,8.0] [‐9, 20] Franklin (2015) transport subsidy Unemployed 877 31 10 months 6.8 n.r. (b) $9 Youth [‐ 0.8, 14.4] Abebe  et al. (2016b) job fair 18‐29 year old 4059 6.5 4 months ‐1.2 6.6 $14 Job‐seekers [‐8.0,5.6] [‐9,23] India Jensen (2012) connecting to  young  1534 6 3 years 2.4 n.r. $12 recruiters women [0.2,4.6] Jordan Groh et al. (2015) certifying soft and hardunemployed 1354 19 5 months 2.4 n.r. (a) $203 skills & matching  tertiary graduates [‐4.7,+9.4] Peru Dammert et al. (2015) information about Job‐seekers 1280 7 1 month 6.2 n.r. (a) $25 job vacancies [0.9, 11.5] 1280 7 3 months ‐0.2 n.r.  [‐6.7, 6.3] Philippines Beam (2016) Attending job fair 20‐35 year  865 3.5 10 months 2.7 n.r. $3.5 olds [‐13.6,19.0] South AfricaAbel  et al. (2016) encouraged to use Unemployed 1267 17 3 months 2.0 n.r. reference  letter Youth [‐2.3, 4.3] Uganda Bassi  and Nansamba (2017)certifying soft skills Young 515 15.2 12 months 5.8 ‐ 6.3 $19 Job‐seekers [‐ 3.6, 15.2] [ ‐42,29] Notes: Peru estimates are  for pooled treatment effect across three  subtreatments which all  provided job vacancy information. a. Study does not report unconditional  earnings estimates. No significant impact found on earnings conditional  on work. b. No significant impact on log earnings conditional  on work at 4 month follow‐up. ITT estimates reported except for Beam (2016) where  several  incentives used in encouragement design, and LATE impact of attending job fair after receiving attendance  voucher is reported. Impacts on employment are  in terms of percentage  points, impacts on earnings in terms of percentage  growth relative  to control  mean. 29    Figure 1: Trajectory of Impact from a Wage Subsidy Program in Jordan .4 Treatment Effect on Employment .1 .2 0 .3 jan2010 jul2010 jan2011 jul2011 jan2012 jul2012 jan2013 Time Source: Groh et al. (2016a). Figure shows month by month impacts of a wage subsidy on employment, along with 95 percent confidence intervals. The two vertical lines shows the start and end of the subsidy period. Endnotes                                                              1 A complementary approach is meta-analysis, with Card et al. (2016) pooling together ALMP estimates from both developed and developing countries, including both randomized and quasi-experimental evaluations. 2 Another category of ALMPs that aims to increase labor demand are public works programs. There have been fewer recent experimental evaluations of these programs, although evaluations are in progress in Côte d’Ivoire and Sierra Leone. Blattman and Ralston (2015) and World Bank (2012) survey the existing evidence. 3 In addition to the 33 month impact reported in Table 1, Alzúa et al. (2016) also report an impact on formal employment (but not earnings) at 48 months. This is smaller still, at 1.4 percent, and not statistically significant. Acevedo et al. (2017) also report 12 month impacts, which are positive and significant on employment for women, and negative and significant on employment for men. 4 Acevedo et al. (2017) find no impact for either gender at 36 months, but do find stronger impacts for women at 12 months. 5 Franklin (2015) tests whether merely surveying people about job search leads to changes in behavior, and finds it does not. 30