World Development xxx (2018) xxx–xxx Contents lists available at ScienceDirect World Development journal homepage: www.elsevier.com/locate/worlddev Optimal targeting under budget constraints in a humanitarian context Paolo Verme ⇑, Chiara Gigliarano The World Bank, 1818 H St., NW Washington DC 20433, USA Department of Economics, Università dell’Insubria, via Monte Generoso 71, 20100 Varese, Italy a r t i c l e i n f o a b s t r a c t Article history: The paper uses Receiver Operating Characteristic (ROC) curves and related indexes to determine the opti- Available online xxxx mal targeting strategy of a food voucher program for refugees. Estimations focus on the 2014 food vou- cher administered by the World Food Program to Syrian refugees in Jordan using data collected by the JEL classification: United Nations High Commissioner for Refugees. The paper shows how to use ROC curves to optimize I31 targeting using coverage rates, budgets, or poverty lines as guiding principles to increase the overall effi- I32 ciency of a program. As humanitarian organizations operate under increasing budget constraints and I38 increasing demands for efficiency, the proposed approach addresses both concerns. Keywords: Ó 2017 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND Food vouchers license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Refugees Receiver Operating Characteristics curves Targeting 1. Introduction In these countries, conflict has been the primary driver of forced displacement but conflict also contributes to other negative out- The United Nations has estimated the number of forcibly dis- comes such as food insecurity and poor health, factors that can placed people in the world at 65.6 million in 2016, most of them increase displacement in their own right.3 Moreover, the same located in the Middle East and Sub-Saharan Africa.1 Forced dis- countries that experienced mass forced displacement in recent years placement is largely the consequence of conflict as shown by coun- are also the same countries that are experiencing persistent tries such as Nigeria, South Sudan, Somalia and the Republic of droughts. The UN estimated that up to 20 million people may be Yemen. Northern Nigeria has over 2 million Internally Displaced at risk of famine in Sub-Saharan Africa and the Middle East by the Persons (IDPs) as a direct consequence of the conflict with Boko end of 2017 with countries such as Nigeria, South Sudan, Somalia Haram; South Sudan may now have up to 80 percent of its popula- and the Republic of Yemen being highly affected.4 This factor rein- tion either displaced or hosting displaced people as a consequence of forces the negative consequences of conflict including forced the civil war; Somalia has generated millions of refugees over dec- migration. ades of instability who settled in neighboring countries or moved Because of this combination of factors, humanitarian organiza- to third countries; the Republic of Yemen has almost 3 million tions have been facing severe budget constraints that made target- people counted as either refugees or IDPs as a consequence of the ing more compelling. Donors find themselves pulled between the civil war.2 necessity to mitigate starvation and save lives and provide for shel- 3 See, among others, Brück and d’Errico (2017) for a discussion on the relationships between food insecurity and various forms of violent conflict, insecurity and fragility. Mercier, Ngenzebukez, and Verwimpx (2017) show that Burundi households living in localities exposed to the 1993 civil war exhibit a significantly higher level of deprivation than non-exposed households. See Shemyakina (2017) on the health ⇑ Corresponding author. consequences of conflict. See also Martin-Shields and Stojetz (2017) for a recent E-mail addresses: pverme@worldbank.org (P. Verme), chiara.gigliarano@uninsu- review of the literature on conflict and food security, van Weezel (2017) for a global bria.it (C. Gigliarano). study of the impact of conflict on food security covering 106 countries and Brück, 1 http://www.unhcr.org/en-us/figures-at-a-glance.html.1. d’Errico, and Pietrelli (2017) for a study of the impact of conflict on food security in 2 For more details of these crises, see www.unhcr.org (country updates) and www. Gaza. For a discussion about the uncertainty that conflicts cause on agricultural iom.int (particularly the Displacement Tracking Matrix – DTM – at http://www.glob- production, see Arias, Ibáñez, and Zambrano (2017). 4 aldtm.info/). http://www.cnn.com/2017/03/11/africa/un-famine-starvation-aid/4. https://doi.org/10.1016/j.worlddev.2017.12.012 0305-750X/Ó 2017 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 2 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx ter, services and any other needs of displaced populations in coun- by the World Food Program (WFP) or cash programs administered tries where host governments have scarce means of their own to by the United Nations High Commissioner for Refugees (UNHCR). face these challenges. Budgets are very stretched and humanitarian These are important questions considering the experience with organizations face tough choices on the ground. Universal coverage targeting programs in development contexts. There are different of assistance programs such as cash or food assistance becomes the methodologies based on microdata that are used for targeting such exception rather than the rule and humanitarian organizations are as simple means tests, proxy means test, community based or geo- forced to ration and target resources. This is where targeting graphical targeting, categorical targeting or various types of self- becomes a very relevant activity for donors and humanitarian selection methods. In a classic study on targeting performances organizations alike. of these methods that covered 122 interventions from 48 coun- This paper is a contribution in the direction of making targeting tries, Coady, Grosh, and Hoddinott (2004) found that targeting per- more effective when budgets are stretched. It exploits Receiver formances varied very significantly across programs worldwide. Operating Characteristics (ROC) curves and related indices to The median program was found to transfer to the poor only 25 per- devise a relatively simple methodology for optimizing coverage, cent more resources than a random allocation of transfers whereas poverty reduction and leakage in the presence of budget con- 21 of the 85 programs evaluated were regressive, meaning that straints. We therefore focus on optimizing outcomes of social pro- they performed worse than a random allocation. We are not aware tection programs based on cash transfers. of a comparable study in humanitarian contexts but it is evident ROC curves are one of the most common statistical tools that improving the targeting capacity of programs is essential. employed to assess the performance of a diagnostic rule based This paper uses the same modelling used for proxy means test- on a predictive model (Lusted, 1971). They are generated by plot- ing, which is one of the best performing methodology according to ting the fraction of true positives out of the positives (true positive Coady et al. (2004). Our contribution is to improve on this method- rate) versus the fraction of false positives out of the negatives (false ology by adopting a more rigorous system to select the optimal positive rate), at various probability thresholds. If we are assessing probability threshold for classifying households as poor or non- a poverty reduction program, this would correspond to plotting the poor. The methodology proposed is not a substitute for a good wel- fraction of poor persons who benefit from the program (coverage fare or poverty model, which remains the cornerstone for a good rate) versus the fraction of non-poor persons who benefit from targeting performance, but helps to improve on targeting after the program (leakage rate). This curve can be used to determine the model has been defined. We show how to operationalize such the probability cut point that optimizes coverage and leakage. approach in the context of welfare improving humanitarian opera- For example, using a poverty prediction model, one can estimate tions using relatively simple visual devices suitable for policy mak- from microdata the probability of an individual to be poor. To clas- ers. In this respect, the paper is a complement to the relatively new sify these same individuals as poor or nonpoor based on these pre- literature on targeting food and cash programs in humanitarian dictions, one must then decide the probability threshold to use as contexts (Coll-Black et al., 2012; Maxwell, Young, Jaspars, Burns, cut point between poor and nonpoor. The standard approach is to & Frize, 2011; Morris, Levin, Armar-Klemesu, Maxwell, & Ruel, consider as poor those individuals who have a probability of being 1999; Tranchant et al., 2017). poor above 50 percent and as non poor those individuals who have The paper is organized as follows. The next section describes a probability equal or below 50 percent. However, this practice is the food voucher program that we use to illustrate the methodol- arbitrary and may result in coverage and leakage rates that are ogy proposed. Section 3 describes the data, Section 4 outlines the not optimal. By changing the probability cut point, one can plot models employed to caliber the methodology, Section 5 outlines the curve that describes all possible combinations of coverage and results and Section 6 concludes. leakage rates (the ROC curve) and choose the probability that corre- spond to the best combination of the two outcomes. The main applications of the ROC curves are in medicine (see, 2. The 2014 food voucher program for refugees in Jordan e.g., Hand, 2010) or more generally in diagnostics (see, e.g., Hand & Anagnostopoulos, 2013) and in credit risk analysis (see, e.g. As a case study, the paper uses the 2014 food voucher program Gigliarano, Figini, & Muliere, 2014; Thomas, 2009), and the Area administered by WFP to Syrian refugees in Jordan. It should be Under the Curve (AUC) is a popular measure to evaluate the discrim- noted upfront that Jordan is a middle-income country and has a inative power of a predictive model (see, e.g. Hand, 2009, 2012; very different capacity to provide for displaced populations as Krzanowski & Hand, 2009). The use of these curves in economics compared to countries such as South Sudan or Somalia. The fund- has been less frequent (see Wodon, 1997 for an early application) ing per refugee available in this country is higher than what is gen- and, to our knowledge, these curves have not been used in the con- erally available in poorer countries and the quality and quantity of text of humanitarian programs, with the exception of our previous refugee data is superior to those available in Sub Saharan Africa. contribution on which this paper builds (Verme et al., 2016). For example, the estimated UNHCR expenditure for the Middle In the context of a welfare program such as cash assistance or East and North Africa region in 2015 was higher than the estimated food vouchers, the approach proposed can be used when policy expenditure for Sub Saharan Africa, in spite of the fact that the pro- makers work with a coverage, poverty or budget target. 5 It jected needs and caseload was higher in Sub Saharan Africa.6 The answers questions such as: i) What is the budget required to reduce infrastructure and the education level of the local staff in a middle poverty by X percent? ii) What is the budget required to increase house- income country like Jordan also contributed to improve the quality hold coverage by Y percent? iii) What is the coverage or poverty reduc- of data collection, management and use. However, we will show that tion we can obtain with a given Z budget? iv) Can targeting be improved the approach proposed can be implemented using micro data that by shifting the poverty line? The answers to these questions can help the WFP and UNHCR routinely collect and that this is a method donors make funding decisions and humanitarian organizations not beyond the existing capacity of these organizations. make targeting choices. We show that this approach can be applied The material assistance provided to refugees in Jordan in 2014 to existing programs such as food voucher programs administered was essentially structured in two programs: i) a cash program 5 The technology proposed in this paper requires basic econometric skills and trained econometricians but the resulting graphs presented in the results section are 6 designed for policy makers and do not require any particular training. http://www.unhcr.org/56d7f9884.pdf6. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx 3 administered by the UNHCR and ii) a food voucher program The second data set is the Home Visits database (HV), which has administered by the WFP. In 2014, the UNHCR cash program pro- been administered in Jordan in successive rounds starting from vided 50 Jordanian Dinars (JD)7 per month to cases8 including one 2013 for the purpose of targeting the cash assistance program. or two members, 100 JD to cases with 3–5 members and 120 JD to The HV questionnaire results in almost 200 variables that can be cases with more than five members. The WFP program included used for analysis and includes questions on income and expendi- two bi-weekly vouchers for a total value of 24 JD per person per ture, which we use to construct our welfare aggregate. For the pur- month. The voucher was provided to the principal applicant and it pose of this paper, we use the second round implemented between could be spent via a network of 652 stores distributed across all gov- November 1, 2013 and September 30, 2014. ernorates of Jordan. In this paper, we focus on the food voucher pro- The home visits survey is administered to refugees living out- gram but the approach proposed can be equally used for any cash side camps, which is the focus group of this paper. In 2014, less program. Note that the two programs are not substitutes but com- than 18 percent of refugees were hosted in camps with almost plements. Refugees can receive both programs and neither of the 14 percent residing in the Zaatari camp alone. Syrian refugees liv- programs can exclude households from the other program. ing in Jordan at the time were not allowed to work although a few By the end of 2013, the food voucher program reached almost hundred refugees obtained work permits through special arrange- universal coverage and was assessed positively by both its admin- ments granted to selected employers. Despite this restriction, istrator and external organizations. In addition to the benefits about 44 percent of refugees living outside camps reported to have accrued to the direct beneficiaries, the WFP found that ‘‘The pro- some form of income suggesting that many worked informally. gram has also led to some US$2.5 million investment in physical Reported income was very low, below or close the minimum wage infrastructure by the participating retailers; created over 350 jobs in of Jordan and also typically occasional (Verme et al., 2016). the food retail sector; and generated about US$6 million in additional The unit of observation used by the study is the ‘‘case”. The tax receipts for the Jordanian government. In terms of indirect effects, UNHCR abides to the principle of family unity whereby a case this study finds a predictive multiplier ranging from 1.019 to 1.234. In should include family members but family members may have other words, WFP’s plan to distribute US$250 million in vouchers dur- been separated or several families may be living together. This ing 2014 would lead to some US$255-US$308 million of indirect ben- implies that a case may correspond to a set of related individuals, efits for the Jordanian economy.” (WFP, 2014, p. 1) The World Bank a family or a household made of several families.9 The case identi- and the UNHCR jointly evaluated the poverty reduction of the pro- fier is present in both PG and HV data and for this reason it was pos- gram and found it to reduce the poverty headcount from 69.2 to sible to merge the PG and HV data into one dataset. The final sample 32.3 percent of the refugee population living outside camps used for this paper included over 43,000 observations, about a third (Verme et al., 2016). of all cases registered in Jordan in 2014. This sample is not a random Towards the end of 2014, the humanitarian community started sample of the refugee population but Verme et al. (2016) showed to face budget shortages that forced the WFP to scale down its food that it is the closest approximation to a random sample that one voucher program. The organization found itself with the difficult could have with existing data. choice of how to prioritize beneficiaries and opted to follow a wel- The welfare aggregate was constructed using two questions on fare approach whereby income or consumption would be used to expenditure present in the HV data. The HV data contain questions assess welfare and target households. Targeting based on welfare on income and expenditure. The income question was clearly modeling – where the dependent variable is a measure of wellbe- underreported and, as customary when measuring welfare in poor ing such as income, expenditure or consumption and the indepen- and middle-income countries, expenditure or consumption is pre- dent variables are a set of individual and household socio- ferred as a measure of welfare. The two questions on expenditure economic characteristics –, is a well-developed methodology in were both based on a 30 days’ recall period but one had answers economics and has produced important tools like Proxy Means Tar- organized in six items and the second in ten items.10 The two ques- geting (PMT) that help organizations such as the World Bank or tions were aggregated by selecting the highest value between the governments to target cash-based programs. However, this is not two questions for items common to both questions and preserving an exact science and it is known that this approach invariably the rest of the items from both questions. This allowed to reduce results in sizeable leakage and undercoverage errors. underreporting to a minimum. The resulting welfare aggregate showed common properties to consumption aggregates in terms of distribution.11 We had also knowledge of the households (cases) that 3. Data received assistance via the UNHCR cash program. For these house- holds, the value of the programs was subtracted from the aggregate The paper uses a combination of the UNHCR proGres (PG) reg- to obtain expenditure net of social assistance. Most households were istry data and the UNHCR Home Visits (HV) data. The PG database also receiving food vouchers but interviewers and respondents to the is the official global registration system of the UNHCR. Much of home visits questionnaire were instructed to exclude these expendi- the published statistics on refugees derive from this database and tures when filling the questionnaire.12 Finally and for the purpose of any survey or home visit targeting refugees is usually based on this poverty modeling, the welfare aggregate was transformed into per database. ProGres contains information collected from refugees at capita basis. different stages starting from the first brief interview administered when refugees cross the border to more extended interviews car- 9 ried out when refugees are settled. Data are updated on a continu- See UNHCR, Operational Guidance Note on Resettlement Case Composition, June 2011. See also: http://www.unhcr.org/3d464ee37.pdf9. ous basis and they are principally used to identify beneficiaries of 10 The questionnaire contained two questions on expenditure because one was the various protection and cash programs that the UNHCR and designed for measuring general expenditure and the other for better understanding other organizations administer to refugees. It is, in short, the refu- food expenditure. 11 gee ‘‘census” and contains individual and household (case) socio- For more details on the welfare aggregate and its properties see Verme et al. economic information but does not contain information on welfare. (2016, p. 60). 12 Excluding the value of the food voucher from the expenditure aggregate was essential to make sure that cash assistance could be cumulated with food assistance. 7 A Jordanian Dinar in 2014 was roughly equivalent to one Euro. In other words, the UNHCR provided cash assistance to cases whose expenditure net 8 In the UNHCR jargon, a ‘‘case” is the household or family unit used to register of food assistance was found to be below the poverty line. This is also essential for our refugees. See also the data section. analysis where we evaluate the pre- and post-transfers poverty of beneficiaries. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 4 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx Table 1 Summary statistics of welfare aggregates monthly). Variable Mean (USD) Mean (JD) Std. Dev. (JD) Min (JD) Max (JD) Income per capita 49.63 34.95 64.41 0 3000 Income per capita with no zeroes 93.01 65.50 75.99 0.5 3000 Expenditure per capita 110.65 77.92 74.72 1 1675 Expenditure per capita net of UNHCR cash assistance 100.25 70.60 76.29 0 1675 Source: Authors’ estimations based on Jordan UNHCR data. Table 1 provides the summary statistics for income and expen- diture. It is clearly visible that income per capita is underreported .8 when compared to expenditure. It is also evident that the UNHCR cash program contributes, on average and per capita, with about 7 .6 JD per month, which is in line with what we should expect given the coverage of the program in 2014. Fig. 1 also shows how the shape of the welfare aggregate of choice (expenditure per capita .4 net of UNHCR cash) exhibits the expected characteristic of a bell- shaped distribution. The poverty line adopted for the study is 50 JD per capita per .2 month. This threshold was selected because it was the poverty line adopted by the UNHCR for providing cash assistance. At the end of 0 2013, this line was equivalent to 71 USD and 160 USD at Purchas- -2 0 2 4 6 8 ing Power Parity (PPP) per capita per month. This latter amount x corresponded to 5.25 USD PPP per day, a significantly higher kdensity Wexp_lncap kdensity Winc_lncap amount than the international poverty line of 1.9 USD PPP recom- kdensity Wexp_unhcr_lncap mended for poor countries but a lower amount as compared to the poverty line adopted by Jordan at the same time (8.2 USD/day/ Fig. 1. Income and expenditure distributions. Source: Authors’ estimations based PPP). Considering that Jordan is a middle-income country and that on Jordan UNHCR data. On the x-axis is the log of income or expenditure per capita. most refugees live in urban areas, this poverty line is reasonable.13 On the y-axis is the density (percentage of population). Based on the sample used for this study, a poverty line of 50 JD/cap- ita/month results in a poverty rate of 52.5 percent for cases and 69.2 percent for the population. In other words, 7 in 10 refugees in 2014 variables and then used this shortlist with a backward and forward were estimated to be poor based on a 50 JD per person poverty line. stepwise selection method to identify the best model. The optimal prediction model is then used to predict poverty as follows: 4. Models ^i ¼ a P ^þc ^X i ð2Þ The underlying model of the predictions and simulations that will follow is a simple poverty model described as follows: where P^ i can be interpreted as the expected probability of being poor given a set of X characteristics and based on the estimated Pi ¼ a þ cX i þ ei ð 1Þ parameters a ^ and c ^ from Eq. (1). where Pi = 1 if the case is under the poverty line and Pi = 0 if the To determine whether a household is expected (predicted) to be case is on the poverty line or above; Xi = vector of case characteris- poor or not, the standard approach is to assign a prediction of ‘‘i = tics that derive from merging the UNHCR proGres registry data and non-poor” to households with 0 6 P ^ i 6 0:5 and ‘‘i = poor” to house- the home visits data (see Table A in annex for the full list of vari- holds with 0:5 < P ^ i 6 1. This is not particularly efficient because a ables); ei = normally distributed error term with zero mean; and cut point of 0.5 may not be the cut point that maximizes the cov- i = case. erage rate (i.e. the fraction of poor correctly predicted as poor) and The model is estimated with a probit function and the main minimizes the leakage rate (i.e. the fraction of nonpoor predicted objective is to maximize its poverty prediction capacity. To achieve as poor), when these are the objectives of the program’s adminis- this objective, we followed a systematic procedure to optimize the trators. One approach to solve this problem is to use indices that construction of the independent variables and the explanatory promise to optimize coverage and leakage. Two popular indices power of the model. For the variables construction, we constructed are the Youden Index (YI) and the Distance Index (DI) described case indicators and turned some of the categorical variables into as follows: dichotomous variables.14 This avoided the question of discontinuity   between categorical variables. To maximize the explanatory power Pt NP t of the model, we first explored the explanatory capacity of each indi- YI ¼ max À ð3Þ P NP vidual variable with binomial models to prepare a first shortlist of sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  2  2 13 See also the discussion on poverty line in Verme et al. (2016, p. 64). Pnt NP t DI ¼ þ ð4Þ 14 The categorical variables that have been turned into dichotomous variables are: P NP House: electricity, House: sanitary, House: concrete house, House: for rent or owned, Receiving any kind of NFIs. This transformation, although affecting the variance of where N = population; P = number of poor; NP = Number of non- these explanatory variables, was due to the fact that some modalities of the original variables had very small frequencies, thus making the estimation of the regression poor; t = targeted; nt = non-targeted. Consequently: coverage rate coefficients unreliable. =PP t ; undercoverage rate = PP nt ; and leakage rate = NP NP t . Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx 5 The methodology that we propose has some limitations. First of all, it considers as equally important the misclassifications due to false negative and those due to false positive. Other approaches have been proposed in the literature, which assume, instead, that it is possible to quantify the different costs incurred in misclassify- ing individuals. These methods combine benefits and costs with the coverage and leakage in order to find the value on the ROC curve that minimizes the average cost (or maximizes the average benefit) of a poverty assessment. Another drawback of the method is that comparisons over time and across countries may be prob- lematic, since different ROC curves will provide different optimal probability thresholds. 5. Results As in any other econometric effort, the key to a successful implementation of the methodology proposed is to find a poverty Fig. 2. Coverage and leakage rates with different Probability Thresholds (ROC model with good explanatory power. Table A in the annex shows curve). Source: Authors’ estimations based on Jordan UNHCR data. the results of alternative poverty models based on the 2014 UNHCR Jordan data. The dependent variable is the dummy variable that indicates whether a case’s expenditure per capita is below the poverty line (equal to 50 JD),15 whereas the independent variables Similarly, one can draw the ROC curve, which plots for each are divided into groups of variables. In Model 1 we consider only possible cut point between 0 and 1 the corresponding leakage rate socio-demographic variables such as case size, proportion of chil- (x-axis) and coverage rate (y-axis), thus determining the optimal dren, age, education, employment and marital status of the Principal trade-off between these two indicators (see also Wodon, 1997). Applicant (PA),16 as well as place of destination and origin of the This is represented in Fig. 2 below. In such a graph, the diagonal household. In Model 2 we also add variables related to characteris- represents an equal probability of coverage and leakage, which is tics of the house and of the WASH17 system, and Model 3 includes what one would obtain if targeting was random (blind assignment also variables related to coping strategies implemented by the refu- of the program). All the points on the left of the diagonal show the gees as well as information on whether the case (household) receives performance of models that do better than random draws. The humanitarian assistance. Area Under the ROC Curve (AUC) becomes therefore an indicator In Table A, the three models are compared in terms of pseudo R- of the predicting capability of the model and it can be used in con- squared and Area Under the ROC Curve (AUC). These indicators junction with the R-Squared statistics of a model to assess its pre- show that Model 3 shows the best performance followed by mod- dicting capacity. Differently from the R-squared, the AUC value els 2 and 1 in this order. Regressors related to housing and WASH varies between 0.5 and 1 where 1 represents perfect prediction included in models 2 and 3 seem to make the difference with capacity. model 1 and contribute significantly in explaining the probability The example illustrated in Fig. 2 shows that the model does bet- of being poor. Fig. 3 shows graphically the same results using the ter than a random assignment. The AUC value is 0.8751, which ROC curves. The AUC is almost the same in models 2 and 3 and indicates that the model is capable of predicting poverty correctly the corresponding ROC curves practically overlap (the green line 87.5 percent of the time, while a random assignment would predict in Fig. 3), whereas the ROC curve corresponding to model 1 (the correctly 50 percent of the time. The curve also shows that there is blue line in Fig. 3) is clearly below. one point where the distance between the diagonal (random Considering Model 3 as our best poverty model, we can now assignment) and the curve (the model) is maximized (vertical line test its capacity to predict poverty correctly. The exercise consists in the figure). This is the point that offers the best coverage while of using the parameters estimated by the models to predict the minimizing leakage, given the selected model. To this point corre- dependent variable of the model (poor/non-poor) as if we did not sponds a probability threshold of 0.703, which is what can be used have information on this variable. Table 2 shows the results using to maximize the targeting performance of the model. This also cor- different probability thresholds. For any given cut point between 0 responds to the Youden index, which can be graphically repre- and 1, we can evaluate the corresponding coverage rate (that is the sented as the longest vertical distance between the ROC curve probability of correctly classifying the poor) and leakage rate (that and the diagonal. The Distance index, instead, corresponds to the is the probability of classifying as poor a non-poor). The table com- shortest Euclidean distance between errorless classification (the pares coverage and leakage rates for three arbitrary cut point val- point (0,1)) and the ROC curve. Therefore, in the example depicted ues (0.3, 0.5 and 0.6), revealing that in correspondence to a low cut in Fig. 2, the probability threshold that provides optimal targeting point we obtain a very high coverage rate but also a high leakage is well above 50 percent according to both indexes and the ROC rate. curve. In particular, the central panel of Table 2 shows that, using a This is precisely what ROC curves can help to do. Each threshold poverty line of 50 JD and a 50 percent threshold, the model would corresponds to a specific coverage rate and leakage rate and there be able to predict correctly if an individual is poor 90.4 percent of is clearly a different trade-off between coverage and leakage in cor- the time, which implies that 9.6 percent of the time the model respondence of each probability threshold. Changing the threshold would predict poor individuals as non-poor (under-coverage rate may improve on one of the two rates while making the other rate worse. By knowing preferences for minimizing leakage (save 15 See Verme et al. (2016) for a sensitivity analysis based on different poverty lines. money) or maximizing coverage (reduce poverty) and the corre- 16 The principal applicant is the reference case person for the UNHCR, similarly to sponding trade-offs depicted by the ROC curves, policy makers head of household in household budget surveys. can evaluate and compare alternative outcomes in anticipation of 17 WASH is an acronym frequently used by humanitarian organizations to describe implementing a program. the Water, Sanitation and Hygiene sectors. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 6 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx 1 1 Youden and Distance indices .8 .8 .6 Coverage .6 .4 .4 .2 .2 0 0 0 .2 .4 .6 .8 1 Leakage 0 .2 .4 .6 .8 1 Model 1 Model 2 Cutpoint Model 3 45 degree line youden dist Fig. 3. Comparing poverty models’ ROC curves. Source: Authors’ estimations based Fig. 4. Youden and distance indices for different cut point. Source: Authors’ on Jordan UNHCR data. estimations based on Jordan UNHCR data. Table 2 Coverage and leakage rates for different choices of cut point. indices curves. This falls indeed within the two optimal thresholds Observed poor indicated respectively by the two indices and could solve the issue No Yes Total of choosing between two optimal values. Predicted poor Cut point 30% As an alternative, it is possible to select the optimal probability No 37.64 2.92 13.59 threshold using the available budget for the food voucher program Yes 62.36 97.08 86.41 as baseline. The WFP would normally know the budget available Cut point 50% for targeting and this is what frequently drives targeting decisions. No 61.01 9.59 25.39 The WFP or the UNHCR do not have large regular annual budgets Yes 38.99 90.41 74.61 Cut point 60% and they rely on donors’ contributions pledged during humanitar- No 71.20 15.63 32.71 ian crises. Based on the effective contributions received following Yes 28.80 84.37 67.29 the pledges, these organizations take targeting decisions accord- Source: Authors’ estimations based on Jordan UNHCR data. ingly. Therefore, the targeting decisions are largely based on bud- Bold values refer to coverage and leakage rates as referred in the text where the get rather than coverage criteria. Indeed, if programs were to be table is discussed. administered based on purely legal and entitlements criteria, these organizations should cover the totality of refugees and coverage would be universal. Targeting is therefore a second best option or exclusion error). The model would also predict correctly if an determined by budget constraints. individual is non-poor 61 percent of the time, which means that As an example of targeting based on budget criteria, Fig. 5 plots 39 percent of the time the model would predict as poor nonpoor cut points (or probability thresholds) corresponding to different individuals (leakage rate or inclusion error). The first type of error mixes of coverage and leakage rates on the y-axis versus different (under-coverage) is more problematic from a policy and welfare budget scenarios on the x-axis. For simplicity, the budget on the perspective, whereas the second type of error (leakage) is more x-axis varies between 0 and 100, where 100 corresponds to the problematic from a budget perspective. budget needed for universal coverage. For example, if we assume The first and the last panels repeat the exercise with a probabil- that the WFP has a budget sufficient to cover only 80 percent of ity threshold of 30 percent and 60 percent, respectively. In the for- the refugee population (the vertical line in the figure), the proba- mer case, the under-coverage rate improves (2.9 percent) while the bility threshold that would achieve that outcome is 0.42 (the hor- leakage rate worsens (62.4 percent). In the latter case, the under- izontal line in the figure). coverage rate worsens (15.6 percent) while the leakage rate Alternatively, one can focus on optimizing coverage and leak- decreases considerably (28.8 percent). Clearly, changing probabil- age. This can serve organizations that do not have a sufficient bud- ity threshold affects targeting results. Hence, it is important to get to cover all potential beneficiaries but can choose between fine-tune the parameter related to the probability threshold to different levels of expenditure based on efficiency criteria. In this obtain results the closest as possible to the error we want to min- case, one can plot coverage and leakage rates against different bud- imize (under-coverage or leakage). get scenarios as shown in Fig. 6. By drawing a vertical line that cor- We now determine the optimal probability threshold that max- responds to the maximum distance between coverage and leakage imizes coverage and minimize leakage, by computing the Youden rates, we can derive the optimal budget. In the example proposed, and the Distance indices (Fig. 4). In general, the two indicators this budget is equivalent to 58.7% of the budget necessary for uni- may suggest different optimal thresholds (indicated in Fig. 4 by versal coverage. This is the budget that simultaneously maximizes the two vertical lines). In our case, we find the two values rather coverage and minimizes leakage. close: according to the Youden index, the optimal probability On the contrary, a decision maker may prefer to fix a priori the threshold should be 0.703 (corresponding coverage, 0.766; corre- percentage of coverage. This is shown in Table 3 where, for a given sponding leakage, 0.184), while according to the Distance index coverage rate, the corresponding percentage of the universal bud- the probability cutoff should be 0.685 (corresponding coverage, get, as well as the leakage rate and the cut point are given. For 0.782; corresponding leakage, 0.201). Also noteworthy and from example, in order to reach 80% of coverage, one needs to use a graphical perspective, one could consider as an optimal choice, 62.25% of the universal budget, which corresponds to 22% of leak- the maximum vertical distance between the Youden and Distance age and to a probability threshold equal to 66%. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx 7 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 14.6 23.3 30.9 38.5 46.8 57.5 73.3 100.0 155.0 1675.0 Poverty line (in JD) OpƟmal coverage OpƟmal leakage OpƟmal cutpoint CDF Fig. 7. Optimal coverage rate, leakage rate and cut point (%). Source: Authors’ estimations based on Jordan UNHCR data. Fig. 5. Probability threshold for different budget scenarios. Source: Authors’ Summarizing alternative options, Table 4 shows the change in estimations based on Jordan UNHCR data. coverage, leakage, probability threshold and poverty rates in corre- spondence of different budgets (represented as different percent- ages of the universal budget). The first line shows that, in case of universal budget, every case receives assistance (coverage and leakage are equal to one and cut point is equal to zero) and the poverty rate corresponds to 32.3%. Moreover, 69.2% of the overall budget is used for coverage whereas leakage costs 30.8%. When, instead, only 80% of the universal budget is used, the poverty rate increases to 38.4%, and the budget is split between 81.2% for cov- erage and 18.8% for leakage. Reducing again the percentage of the universal budget to 50%, the poverty rate rises to 50.1%, cover- age absorbs 93.1% of the budget and leakage 6.9%. Therefore, this approach provides a costing of leakage in addition to coverage and the corresponding outcomes in terms of poverty rates. A table like Table 4 is a useful tool for donors and decision makers called to make difficult budget choices. Adjusting the poverty line may be another device to optimize coverage and leakage. One can repeat the whole exercise of esti- mating the poverty model for a set of different poverty lines, thus obtaining the corresponding predicted probabilities and the cut point that optimizes coverage and leakage, and see whether mar- ginally adjusting the poverty line can result in significant gains in terms of coverage and/or leakage. This is illustrated in Fig. 7. Fig. 6. Coverage rate and leakage rate for different budget scenarios. Source: As an example, we use ten alternative poverty lines in correspon- Authors’ estimations based on Jordan UNHCR data. dence of each decile of expenditure per capita and estimate the optimal probability threshold and the corresponding coverage and leakage rate for each poverty line. We then connect these ten points to show the resulting curves. A policy maker would nor- Table 3 mally seek the maximum distance between the coverage and leak- Budget, leakage, probability threshold for different % of the universal coverage. age rate and, as one would expect, this distance grows as we % Budget (as % of universal Leakage Probability increase the poverty line. Coverage budget) threshold However, the increase in this distance is not linear. For example, 100% 100.00 1.00 0.00 using the second decile’s mean value as poverty line (23.3 JD), 80% 62.25 0.22 0.66 would result in a coverage rate of 72 percent and a leakage rate of 60% 43.81 0.07 0.83 26 percent. If the poverty line is increased to the third decile’s mean 50% 3.47 0.00 1.00 value (39.9 JD), this would not change the coverage rate and would Source: Authors’ estimations based on Jordan UNHCR data. decrease the leakage rate by only 1 percent. In other words, there is Table 4 Coverage, leakage, probability threshold and poverty rate for different% of the universal budget. % Universal budget Coverage rate Leakage rate Probability threshold Poverty rate (%) % Budget for coverage % Budget for leakage 100% 1.00 1.00 0.00 32.30 69.2% 30.8% 80% 0.94 0.49 0.42 38.39 81.2% 18.8% 60% 0.78 0.20 0.69 45.65 89.8% 10.2% 50% 0.67 0.11 0.78 50.10 93.1% 6.9% 0% 0.00 0.00 1.00 69.18 Source: Authors’ estimations based on Jordan UNHCR data. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 8 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx almost no gain in increasing the poverty line from 23.3 to 30.9 JD, be applied to food voucher programs as illustrated in this paper although some poor households would be excluded by the program. or to any other cash program. We also showed that such methods Instead, shifting the poverty line from the sixth to the seventh dec- can be applied using existing data collected by the UNHCR in the ile’s mean value would increase the coverage rate by 2 percentage framework of existing programs such as the WFP food voucher points while decreasing leakage by 4 percentage points, a signifi- program. In other words, these are tools that can be readily applied cant gain. Poverty lines established for targeting are generally of and do not necessarily require the collection of new data or the an absolute nature and based on basic needs assessments. However, administration of special programs. targeting or budget considerations may justify adjusting the It is also clear that the method proposed cannot be applied in all poverty line to optimize the use of resources. For this purpose, contexts. The Jordan example provided in this paper relies on a set Fig. 7 can be a useful instrument for policy makers. of data (proGres and home visits data) that is atypical in humani- Fig. 7 also shows the Cumulative Distribution Function (CDF) tarian contexts. While proGres data are available in most countries built on deciles of expenditure per capita (a straight line by con- where the UNHCR operates, the quality of these data is very vari- struction). The CDF is useful in that it shows the poverty rate on able (the Jordan data are known to be among the best quality data the (y-axis) for each possible poverty line (x-axis). For example, available). Also, home visits data that contain information on with a poverty line of approximately 42 JD, about 50 percent of income or expenditure are the exception rather than the rule in the population would be under the poverty line. Therefore, when humanitarian contexts. Therefore, at present, the replicability of one adjusts the poverty line to optimize coverage and budget, it the method proposed elsewhere is limited to selected areas and is also possible to monitor what the effect of this change would countries. However, all humanitarian operations with budget lim- be on the poverty rate and, consequently, on the number of people itations that use cash or food vouchers as a form of social protec- under the poverty line. Again, this is a relatively simple device that tion face the same targeting challenges described in this paper. supplies a set of critical information for policy makers aiming to In the absence of quality data on income or consumption, these fine-tune targeting decisions. operations have to rely on alternative and less accurate targeting criteria the outcomes of which (in terms of poverty, coverage and 6. Conclusions leakage) are non-measurable. This paper has implicitly shown that the collection of income or consumption data can lead to improve- Humanitarian organizations such as the WFP and the UNHCR ments in the measurement of outcomes and in the effectiveness of rely entirely on donors’ contributions to administer programs for targeting. refugees and IDPs. These contributions are increasingly hard to This paper has also ignored aspects of political economy, come by and, when crises are multi-dimensional, contributions administration and other outcomes that may be equally important are typically below the needs generated by the sum of these for beneficiaries. Governments have often a bias for geographical dimensions. This is the case, for example, of the recent crises in targeting covering areas relevant for selected constituencies rather Nigeria, South Sudan, Somalia and the Republic of Yemen. The con- than focusing on the poor. Administrative constraints such as com- flicts that affected these countries generated the displacement of plex logistics or lack of administrative budgets can make reaching millions of people who became IDPs or refugees. The more recent the poor impossible in some cases making geographical or other droughts that affected all these countries are contributing to fur- forms of targeting more appealing. Outcomes different from pov- ther misery and the combination of conflict and droughts led to erty reduction such as nutrition levels may be as important or food shortages that are resulting in famine. more important than poverty reduction in a humanitarian and The needs of the affected populations, whether refugees and emergency context. These are all elements to factor in when con- IDPs or host populations, grow disproportionally when these sidering the method proposed in this paper. calamities occur simultaneously, donors find themselves stretched between different calls for funds and humanitarian organizations Conflict of interest remain under-funded. In these cases, targeting is an obliged choice and optimal and efficient targeting can make the difference We hereby declare that neither authors (Paolo Verme or Chiara between life and death for some of the displaced populations. This Gigliarano) have any conflict of interest in relation to the paper justifies refining targeting techniques making the most of the submitted: ‘‘Optimal Targeting under Budget Constraints: The Case available technology. of a Refugee Food Voucher Program”. In addition to being technically sound, targeting requires tools that make choice and implementation relatively simple for policy Acknowledgement makers and field staff. One cannot expect all humanitarian staff working on the field to be knowledgeable in econometrics or This work is part of the program ‘‘Building the Evidence on Pro- statistics techniques. Hence the need for relatively simple visual tracted Forced Displacement: A Multi-Stakeholder Partnership”. devices that can be used for making normative targeting choices The program is funded by UK aid from the United Kingdom’s based on positive criteria without necessarily be cognizant of the Department for International Development (DFID), it is managed econometrics and statistics that lie behind these devices. by the World Bank Group (WBG) and was established in partner- This paper has proposed the use of ROC curves and related ship with the United Nations High Commissioner for Refugees” indices to refine targeting when budgets are constrained and has (UNHCR). The scope of the program is to expand the global knowl- developed relatively simple graphs that can be used by policy mak- edge on forced displacement by funding quality research and dis- ers to make decisions on coverage, budgets and poverty lines when seminating results for the use of practitioners and policy makers. targeting is based on welfare criteria. We show existing trade-offs This work does not necessarily reflect the views of DFID, the between optimal coverage and leakage rates, the optimal use of WBG or UNHCR. The authors would like to thank Lidia Ceriani, existing budgets and small adjustments that can result in large Xavier Devictor, Ugo Gentilini and Phillippe Leite for very useful gains in terms of poverty reduction. The methods proposed can comments. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx 9 Appendix A. Table A Poverty models. MODEL 1 MODEL 2 MODEL 3 Coef. z Coef. z Coef. z Case size 0.581 49.000 0.639 53.190 0.629 51.580 Case size squared À0.009 À10.450 À0.014 À16.890 À0.014 À15.800 Case size * Age of PA À0.003 À14.300 À0.003 À14.150 À0.003 À13.970 Age of PA 0.007 9.150 0.009 10.370 0.008 9.170 Proportion of children < 18 years 0.362 15.470 0.343 13.950 0.295 11.800 Highest education of PA (in years) À0.079 À20.310 À0.051 À12.420 À0.046 À11.250 Employment of PA (Ref. None) Low Skilled À0.001 À0.030 À0.043 À2.200 À0.039 À1.990 Skilled À0.053 À3.280 À0.045 À2.660 À0.047 À2.720 High Skilled À0.149 À9.170 À0.119 À7.000 À0.120 À7.020 Professional À0.239 À13.350 À0.213 À11.370 À0.213 À11.320 Marital status of PA (Ref. Married or engaged) Divorced or separated 0.243 7.100 0.215 5.950 0.189 5.200 Single 0.111 6.000 0.059 3.000 0.049 2.500 Widowed 0.132 7.400 0.094 4.970 0.080 4.230 Origin (Ref. Damascus) Al-hasakeh 0.032 0.620 À0.175 À3.060 À0.122 À2.100 Aleppo 0.085 4.650 0.087 4.570 0.105 5.460 Ar-raqqa À0.168 À5.100 À0.231 À6.690 À0.188 À5.370 Dar’a 0.054 3.610 0.076 4.920 0.064 4.080 Hama 0.609 28.130 0.083 3.330 0.082 3.270 Homs 0.140 9.230 0.192 12.100 0.158 9.830 Idleb 0.613 16.690 0.170 4.090 0.158 3.770 Rural Damascus 0.083 5.180 0.074 4.430 0.064 3.770 Tartous 0.288 1.770 0.263 1.570 0.248 1.470 As-sweida 0.225 2.010 0.056 0.460 0.024 0.200 Deir-ez-zor 0.072 1.370 À0.017 À0.310 À0.012 À0.210 Lattakia À0.100 À1.360 À0.058 À0.770 À0.059 À0.780 Quneitra À0.121 À2.040 À0.088 À1.450 À0.079 À1.290 Formal arrival À0.254 À23.720 À0.199 À17.770 À0.193 À17.070 Destination (Ref. Amman) Ajloun 0.472 14.530 0.549 16.260 0.553 16.250 Aqabah À0.139 À2.400 À0.161 À2.610 À0.136 À2.170 Balqa 0.121 6.400 0.063 3.190 0.081 4.060 Irbid 0.116 10.430 0.125 10.690 0.123 10.330 Jarash 0.434 15.160 0.486 16.370 0.502 16.840 Karak 0.422 15.910 0.477 17.130 0.448 15.980 Maan 0.303 8.250 0.376 9.560 0.340 8.580 Madaba 0.390 14.400 0.366 12.700 0.391 13.510 Mafraq 0.288 22.410 0.093 6.620 0.031 2.120 Tafilah 0.725 12.590 0.883 14.350 0.913 14.770 Zarqa 0.485 36.840 0.442 31.870 0.422 29.810 Border crossing point (Ref. Airport) Ruwaished-Hadallat 0.109 5.710 À0.019 À0.940 À0.026 À1.280 Tal Shihab 0.150 8.380 0.117 6.300 0.094 5.010 Nasib-official or unofficial 0.177 13.500 0.143 10.510 0.113 8.190 Other or no data 0.098 6.530 0.063 4.050 0.044 2.790 House: electricity À0.120 À8.920 À0.121 À8.930 House: sanitary À0.221 À19.850 À0.205 À18.270 House: concrete house À0.277 À12.410 À0.294 À13.040 House: for rent or owned À1.025 À42.580 À1.033 À42.620 House area: square meters per capita (Ref. < 10 sq meters) 10–15 sq meters À0.099 À9.840 À0.097 À9.570 >15 sq meters À0.285 À29.890 À0.280 À29.150 Wash: water through piped and piped sewerage À0.086 À8.700 À0.081 À8.180 Wash: having adequate and functioning latrine À0.109 À9.490 À0.114 À9.840 Receiving any kind of NFIs 0.093 9.710 Coping strategy: humanitarian assistance 0.297 25.970 Coping strategy: host community 0.091 10.510 Coping strategy: dropping children from school 0.095 7.180 UNHCR financial assistance 0.648 52.190 0.725 56.100 0.627 46.180 Constant À2.137 À46.740 À0.599 À11.390 À0.572 À10.790 N 174,125 174,125 166,616 Pseudo R-squared 0.314 0.314 0.348 AUC 0.858 0.873 0.875 Source: Authors’ estimations based on Jordan UNHCR data. Note: PA = Principal Applicant. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012 10 P. Verme, C. Gigliarano / World Development xxx (2018) xxx–xxx References Martin-Shields, C., & Stojetz, W. (2017). Framing the relationship between food security and violent conflict data: A clustering and typology approach. World Development (forthcoming). Arias, M. A., Ibáñez, A. M., & Zambrano, A. (2017). Agricultural production amid Maxwell, D., Young, H., Jaspars, S., Burns, J., & Frize, J. (2011). Targeting and conflict: Separating the effects of conflict into shocks and uncertainty. World distribution in complex emergencies: Participatory management of Development, forthcoming. humanitarian food assistance. Food Policy, 36(4), 535–543. Brück, T., & d’Errico, M. (2017). Food security and conflict: Introduction to the Mercier, M., Ngenzebukez, R. L., & Verwimpx, P. (2017). Violence exposure and special issue. World Development, forthcoming. deprivation: Evidence from the Burundi civil war. World Development Brück, T., d’Errico, M., & Pietrelli, R. (2017). The effects of violent conflict on (forthcoming). household resilience and food security: Evidence from the 2014 Gaza conflict. Morris, S., Levin, C., Armar-Klemesu, M., Maxwell, D., & Ruel, M. T. (1999). Does World Development. forthcoming. geographic targeting of nutrition interventions make sense in cities? Evidence Coady, D., Grosh, M., & Hoddinott, J. (2004). Targeting of transfers in developing from Abidjan and Accra. World Development, 27(11), 2011–2019. countries: Review of lessons and experience. Washington DC: World Bank and Shemyakina, O. N. (2017). Political violence and child health: Results from International Food Policy Research Institute (IFPRI). Zimbabwe. World Development (forthcoming). Coll-Black, S., Gilligan, D. O., Hoddinott, J. F., Kumar, N., Taffesse, A. S., & Wiseman, Thomas, L. C. (2009). Consumer credit models: Pricing, profit, and portfolios. Oxford W. (2012). In: Food and agriculture in Ethiopia: Progress and policy challenges, University Press. chapter 10 International Food Policy Research Institute (IFPRI). Tranchant, J.-P., Bliznashka, A. G. L., Diallo, A. S., Sacko, M., Assima, A., Siegel, E. H., Gigliarano, C., Figini, S., & Muliere, P. (2014). Making classifier performance et al. (2017). The impact of food assistance on food insecure populations during comparisons when ROC curves intersect. Computational Statistics and Data conflict: Evidence from a quasi-experiment in Mali. World Development, Analysis, 77, 300–312. forthcoming. Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the van Weezel, S. (2017). Food security and armed conflict: a cross-country analysis, area under the ROC curve. Machine Learning, 77, 103–123. School of Economics, University College Dublin, mimeo. Available at: Hand, D. J. (2010). Evaluating diagnostic tests: The area under the ROC curve and . the balance of errors. Statistics in Medicine, 29, 1502–1510. Verme, P., Gigliarano, C., Wieser, C., Hedlund, K., Petzoldt, M., & Santacroce, M. Hand, D. J. (2012). Assessing the performance of classification methods. (2016). The welfare of Syrian refugees: Evidence from Jordan and Lebanon. International Statistical Review, 80, 400–414. Washington, DC: Sage. https://openknowledge.worldbank.org/handle/10986/ Hand, D. J., & Anagnostopoulos, C. (2013). When is the area under the receiver 23228 License: CC BY 3.0 IGO. characteristic curve an appropriate measure of classifier performance? Pattern World Food Programme (WFP) (2014). Verification exercise, preliminary results, Recognition, 80, 400–414. March 2014. WFP, Rome. Available at: . CRC Press—Chapman and Hall. Wodon, Q. (1997). Targeting the poor using ROC curves. World Development, 25(12), Lusted, L. B. (1971). Signal detectability and medical decision-making. Science, 171, 2083–2092. 1217–1219. Please cite this article in press as: Verme, P., & Gigliarano, C. Optimal targeting under budget constraints in a humanitarian context. World Development (2018), https://doi.org/10.1016/j.worlddev.2017.12.012