NOTE NUMBER 352 viewpoint PUBLIC POLICY FOR THE PRIVATE SECTOR MARCH 2018 Trade Facilitation Reform MACROECONOMICS, TRADE & INVESTMENT GLOBAL PRACTICE Russell Hillberry, Ana The Impact of Targeted Sampling on Import Compliance Margarida Fernandes, and Alejandra Mendoza Alcántara The re sults of a n im p a c t e v a lua t io n o f t he int r o d uc t io n o f a r i s k - Russell Hillberry b ase d ap p roach t o s a m p ling im p o r t s hip m e nt s b y t he Ma c e d o n i a n (rhillber@purdue.edu) is an Associate Profes- Food and V eteri na r y Ag e nc y ind ic a t e t ha t t he s a m p ling p la n w a s sor in the Department of Agricultural Economics at not e f f e ctive ly i m p le m e nt e d , a nd t he r e f o r m d id no t im p r o v e t h e Purdue University. Ana targe ting of risk y s hip m e nt s . Ev id e nc e t ha t t he r e f o r m inc r ea s e d Margarida Fernandes (afernandes@worldbank. trad e f lows is inc o nc lus iv e . T he we a k r e s ult s s ug g e s t a ne e d t o org) is a Senior Economist in the Trade and Interna- conce ntrate ef f o r t s in im p r o v ing d a t a m a na g e m e nt in t e c hn i c a l tional Integration Unit of the Development Research age ncies, not only f o r a na ly t ic a l p ur p o s e s b ut e s p e c ia lly t o i m p r o v e Group at the World Bank. Alejandra Mendoza the d esign and e f f e c t iv e ne s s o f t he r e f o r m s . Alcántara (amendoza1@ worldbank.org) is an Im- pact Evaluation Specialist Every year hundreds of millions of dollars of former Yugoslav Republic of Macedonia (FYR in the Finance, Competi- development aid are spent on trade facilitation Macedonia). tiveness and Innovation Global Practice at the projects. Few projects are subjected to scrutiny via Prior to the reform, the World Bank Group’s World Bank Group. formal impact evaluations, and most of the stud- Western Balkans Trade Facilitation project iden- Research for this note was ies that have been conducted evaluate reforms tified lengthy and intrusive sampling and labora- supported by the govern- by customs agencies.1 tory testing regimes by technical agencies as a ments of Canada, the This note is part of a research series on trade significant constraint on trade in FYR Macedonia. United Kingdom, and the United States through facilitation reforms in the Western Balkans. It Laboratory tests of sampled shipments very rarely THE WORLD BANK GROUP the Impact Program of is the first study to evaluate a trade facilitation identified noncompliant shipments, suggesting the World Bank Group’s reform undertaken by a “technical agency” (an poor targeting and excessive sampling. In 2013, Trade and Competitive- ness Global Practice. agency tasked with oversight of food safety, for example, samples were taken from nearly 1 in Additional support was pro- environmental protection, or other respon- 10 import shipments. Less than 0.2 percent of the vided by the World Bank’s sibilities beyond the remit of the customs samples taken were found to be noncompliant Multi-Donor Trust Fund agency). The study evaluated the introduction (authors’ calculations based on FVA administra- for Trade and Develop- ment, the Strategic Research of a risk-based approach to sampling of import tive data for imports of food of animal origin (continued on back page) shipments for the purpose of laboratory testing and animal feeds). High rates of sampling sub- (Fernandes, Hillberry, and Mendoza 2017). jected imports to queuing at the border, causing The reform was undertaken in 2014 by the delays during the movement of goods. Shippers Food and Veterinary Authority (FVA) of the of goods that were sampled for testing also had AR T AR TT DA TA ECF CI IN GL IFTD IO AIT H RUECF N M OW HO D ROMESTH IN IM EV E PM ST CN AE T TOC F L IT M AART GE EA E TM DT T SEAR M ?P L I N G O N I M P O R T C O M P L I A N C E to wait for laboratory testing results before their sampling activity match the sampling activity goods could be released to the market. planned in the reform? Did the reform cause The research reviewed here used data on sampling activity to become better linked to the planned and actual sampling activity, labora- probability that samples do not comply with the tory test outcomes, and international trade flows FVA’s technical standards? How did the reform to assess the impact of the reform. It provides affect import flows? answers to three questions: How closely did actual Improving sampling procedures in the FVA The goal of the reform was to reduce administra- 2 Planned and actual number of import shipments sampled by the Food and tive and private sector costs through adoption of Figure Veterinary Authority, by product and country group, 2014 and 2015 1 a risk-based approach to sampling import ship- a. 2014 ments for laboratory testing. With World Bank Group support, in 2013 FVA staff received train- 600 Actual ing on risk-based approaches. Training included Planned the development of risk scores and guidance on constructing an annual sampling (or monitor- Number of Samples 400 ing) plan based on historical data. The objective of this process was to enable the FVA to better focus sampling and testing efforts on shipments 200 that were more likely to be noncompliant with technical standards and more dangerous if noncompliant. More focused sampling activity 0 would improve oversight of import shipments w EF A t – – C tic pr ne ica & – & A M EF U & TA Pe NA FTA – C SA – A Fis FTA rk oo SA U rk – A C NA er ) ll & ll ) & All xc ultr ts – l U B ll E il try try ll M A ll – E eat ) E ll nd A itz TA l d sh TA ne M FTA – S & FT U ilk U FT T vi U & A A ll Po duc – A . E y– A & raz ul ul A –N –A U –A ed M FT while reducing the number of shipments sent Pe ed Bal C E EF c. P EF sh Ho Afr la Bo – E ilk – –E d– si h – Po Po sc – ilk & Po t f & un Fi EF ilk TA E .E M E E – E E o y fe – C & o lu U & a for laboratory testing. The sampling plan out- ol ne M E sh fe lined the number of samples to be taken the i ex v Fi (e Fi Bo po d l( un Al om –A po following year for a given border post, product l– xc C ne om ffa (e vi O ll C Bo –A group, and source country or group of countries. ed fe Sampling plans were designed for three sets of d un po products under FVA oversight: food of animal om C b. 2015 origin, animal feeds, and food of nonanimal Actual origin and food contact materials. 1,000 Planned 800 Weak adherence to sampling plan Number of Samples The annual sampling plans that were designed as 600 part of the reform aim to optimize sampling activ- ity by border agents, taking into account expected 400 probabilities of noncompliance and potential degrees of harm, given resource constraints. 200 Each annual plan is intended to guide sampling activity over the subsequent year, although agents 0 can deviate from the sampling plan as necessary. a tic ea SA rk A fe FTA ul U & TA A A ey ) C il) Po sh – ll A l ll try l xc us All p ffa ll – B il & ll un & ll C ffal uc All – C ll si k – l SA Al ul Al – A Mil Al ne fric Po FT – N EFT Fi A –A O –A z A A po EU – A ilk – A z al Po – E EF Fi Bra ra – ll Mo d – . N c– O rod l – – M & E E The discovery of salmonella in products imported om – ts t ed EU B oo Bo & – un lk – xc. on & sh a (e ll tf H d po Mi l (e try vi into another country in the European Union, for Pe M ed Al sh sh fe – –A Fi ne Fi example, would likely generate additional FVA d vi try Bo ul sampling activity on similar shipments. Even Po om C Source: Fernandes, Hillberry, and Alcantara 2017. Note: Figures show information based on sampling plans; the FVA laboratory samples dataset; and the single window for import, absent news of a bacteria outbreak, FVA border export, and transit of goods and tariff quota (EXIM) dataset. The datasets are combined based on the broad product group and cover agents retain discretion to depart from the sam- food of animal origin and animal feeds. Group of countries: All countries (All), North America (NA), South America (SA), European Union (EU), Central European Free Trade pling plan when they suspect that a shipment Agreement members (CEFTA), European Free Trade Association members (EFTA). warrants additional scrutiny. Raw summary statistics reveal sizable differ- results. Oversampling in 2014 can be attributed ences between actual sampling activity and the to the fact that implementation of the reform sampling plans. The sampling plan for 2014 pro- was delayed until July of 2014. Actual sampling jected that 598 samples would be taken from two rates in 2014 therefore included six months of classes of products: food of animal origin and sampling activity that preceded the plan. In 2015, animal feeds. The actual number of samples taken a funding dispute between the FVA and the labo- surpassed the plan by a factor of three (figure 1a). ratories that test the samples dramatically limited In contrast, in 2015 the actual number of samples sampling activity. taken was 23 times smaller than the planned num- ber (figure 1b). FVA failed to target risky shipments A regression model was used to assess the cor- One of the objectives of the risk-based sampling relation between the actual samples taken in 2014 reform was to align sampling activity with the and 2015 and the corresponding sampling plans.2 likelihood that shipments would be judged non- The analysis was limited to the two sets of products compliant by subsequent laboratory tests. If the subject to FVA oversight for which complete data reform was effective, a stronger positive relation- are available: food of animal origin and animal ship between noncompliance rates and sampling feeds. The regression controlled for the inherent rates would be observed after the reform than risk characteristics associated with each group of before it. A regression model was used to test this products, group of source countries, and year.3 effect.4 The estimation controlled for unrelated The results indicate weak adherence to the factors, such as differences in product character- sampling plan by FVA border agents. A posi- istics or the potential for harm.5 tive coefficient found on the planned number A preliminary analysis showed that the num- of samples suggests that deviations from the ber of shipments that did not comply with sampling plan leaned in the direction of over- the FVA’s technical standards in 2013–15 was sampling shipments that had initially been extremely small (fewer than 40). In the second deemed most risky (those with higher numbers semester of 2014, sampling and noncompliance of planned samples). rates increased for food of animal origin, possibly Two unexpected challenges to implemen- because of the 2013 aflatoxin outbreak (table 1). tation of the reform plan help explain these Given the very small number of noncompliant Table Sampling and noncompliance rates of all imports, 2013–15 1 Number of sampled Non-compliance Number of import Total number of import shipments Sampling rate rate Year shipments sampled import shipments noncompliant (percent) (percent) 2013 First half 835 8,167 12 10.3 1.4 Second half 446 8,479 1 5.3 0.2 Total 1,281 16,646 13 8 1 2014 First half 451 7,932 2 5.7 0.4 Second half 780 8,643 21 9.1 2.7 Total 1,231 16,575 23 7 2 2015 First half 4 8,844 0 0 0 Second half 59 9,615 0 0.6 0 Total 63 18,459 0 0 0 Source: Fernandes, Hillberry, and Alcantara 2017. Note: Table shows information based on the FVA laboratory samples dataset and the EXIM dataset. The datasets are combined using the broadly defined product grouping and cover food of animal origin and animal feeds. shipments, the regression estimates were unable for FVA products that coincided with the reform to identify a significant relationship between the (though efforts were undertaken to control for likelihood of noncompliance and the sampling export supply and import demand shocks and to rate before the 2014 reform. Considering also restrict the sample of control products to be more data from the post-reform period, the analysis “similar” to the FVA products to rule out this reveals that the probability of a shipment being possibility). If the estimates of an additional 3–5 noncompliant was lower when sampling rates percent growth in imports of FVA products are were higher, especially after the reform. correct, trade response parameters suggest that This result is concerning, because risk-based the reform had an impact roughly equivalent to 4 targeting aims to focus sampling activities on ship- a 1-percentage point tariff cut on FVA products. ments that are more likely to be noncompliant. The second exercise used a regression model One would therefore expect the probability of to assess the relationship between changes in the noncompliance among sampled shipments to sampling rate and changes in import value. This be higher when sampling rates are higher.6 The exercise was complicated by the 2015 funding findings suggest that the targeting of sampling dispute with the laboratories, which caused a dra- activity associated with imports of food of animal matic drop in the sampling rate that is not attrib- origin and animal feeds was not effective in FYR uted to the reform. The relationship between Macedonia. sampling rates and import value is of interest to A caveat with these and other exercises on the trade facilitation literature, so it was estimated sampling activity is that the data needed to calcu- with and without the 2015 data, which include late sampling rates for less risky products (food the effects of the funding dispute. The exercise of nonanimal origin and food contact materials) found no evidence of an effect of reduced sam- were not collected. Given that those products pling on import value over the 12–18 months are less likely to pose health risks to the public immediately after the reform. if noncompliant, the FVA reform was designed Overall the estimates indicate inconclusive to reduce their sampling rates by larger magni- effects of reduced sampling rates on import value. tudes. The reform may have succeeded in ways Changes in the sampling rate had no immediate that could not be measured directly. effect on import value; it is possible that such effects emerge with a lag over longer time hori- Limited impact of reform on trade zons. The findings reveal that imports of FVA A central question typically asked in the small products grew slightly faster than various sets of literature evaluating trade facilitation reforms is control products following the reform, though it the degree to which such reforms affect trade. In is not possible to say conclusively that this growth this study, this question was addressed in two ways. was a result of the reform. These findings need First, an exercise examined whether imports of to be interpreted in light of the caveat that it was products subject to FVA oversight saw more rapid not possible to estimate relationships between growth than other products following reform. sampling rates and imports for less risky products, Second, an exercise tested whether larger reduc- for which larger changes in the sampling rates tions in the sampling rate led to larger increases were planned. in the import value of products subject to FVA oversight. Technical agencies need to improve data In the first exercise, a difference-in-difference collection efforts regression was used to compare import growth of This research highlights the challenges asso- products subject to FVA oversight with growth of ciated with an impact evaluation of reforms similar products. It found that the import value undertaken by technical agencies that play an of FVA products rose 3–5 percent following the important role at the border but are unable to reform. These estimates include the effects of any collect comprehensive data on variables of inter- other FVA policy changes implemented at the est. One reason data are lacking is that techni- same time as the reform. They may also be con- cal agencies sometimes collect data in ways that flated with unobservable changes in the market are not consistent with the international trade classification system. A lack of comprehensive Conclusion data on test outcomes also limited the evalua- This study is innovative for three reasons: (a) tion of substantial reforms that occurred in the it applies rigorous impact evaluation methods sets of products imposing smaller public health to trade facilitation reforms; b) it explores the risks. In the FVA, these data challenges not only impact of the reform on compliance with import undermine evaluation efforts, they also limit the regulations, an outcome the literature previously effectiveness of both the risk-based system and had ignored; and c) it assesses for the first time the construction of the sampling plans. a reform implemented by a technical agency. Recommendations for improving data man- The study highlights the need to concentrate 5 agement include: efforts in improving data management in such agencies, not only for analytical purposes but 1. The FVA should itself collect all test out- particularly to improve the effectiveness and comes—positive and negative—in an elec- design of subsequent reforms. In this case, a tronic format or platform, rather than relying data management reform would help track the on the laboratories to collect and provide this effectiveness of the sampling plan, monitor its information. The information collected from implementation, and improve targeting based each import shipment sampled should be on shipment characteristics. linked to the shipment identification number Many of the study’s findings are inconclusive, issued by the single window for import, export, because of the short sample period; the weak and transit of goods and tariff quota (EXIM), quality of the data; and specific implementation which collects information on the volume of issues, such as the funding dispute with the public imported products, the country of origin, and laboratories in 2015. But the research highlights the firm engaged in each import transaction. the fact that several underlying mechanisms— Electronic data collection would allow more including the design of the sampling plan, the precise tracking of the characteristics of non- testing process, and training for border agents— compliant shipments and the updating of the make trade facilitation reforms work. The con- sampling plan in real time. tributions of these subcomponents represent knowledge gaps that could be filled by a different 2. A more ambitious data management reform evaluation design. would seek to integrate information from (See back page for Notes and References.) laboratory test results with data recorded by other agencies, especially the customs agency. Data at the shipment level could be used to improve the targeting strategy and the sam- pling plan. Incorporating shipment-level data contained in the customs declaration would require high levels of coordination with the national customs agency. 3. Sampling plans should be updated more than once a year and take into account emerging reports of outbreaks or evidence of failed outcomes at other agencies. Emerging risks are currently communicated to border agents through other channels; updating the sam- pling plan accordingly makes sense. Frequent updating of the sampling plan would better tie actual sampling behavior to the plan and help the agency learn about the effectiveness of its approaches to updating the plan. TRADE FACILITATION REFORM THE IMPACT OF TARGETED SAMPLING ON IMPORT COMPLIANCE Notes References 1. See, for example, Fernandes, Hillberry, and Alcan- Carballo, Jerónimo, Georg Schaur, Alejandro Graziano, tara (2015); Fernandes, Hillberry, and Berg (2016); and Christian Volpe Martincus. 2016. “Transit Carballo, Schaur, Graziano, and Martincus (2016); and Trade.” IDB Working Paper IDB-WP-704, Inter- viewpoint Carballo, Schaur, and Martincus (2016). American Development Bank, Washington, DC. 2. The regression model estimated by ordinary least Carballo, Jerónimo, Georg Schaur, and Christian Volpe is an open forum to squares (OLS) explains the relationship between the Martincus. 2016. “Trust No One? Security and Inter- encourage dissemination of number of import shipments sampled and the number national Trade.” IDB Working Paper IDB-WP-703, public policy innovations of samples to be taken according to the sampling plan, Inter-American Development Bank, Washington, for private sector–led and at the product-group of country-year level. DC. market-based solutions for 3. This control was handled by including fixed effects Fernandes, Ana M., Russell H. Hillberry, and Alejandra development. The views for the product group, the country group, and the year Mendoza Alcantara. 2015. “Trade Effects of Customs published are those of the as regressors in some of the OLS regressions. The as- Reform: Evidence from Albania.” Policy Research authors and should not be sessment also excluded outliers, such as milk and milk Working Paper 7210, World Bank, Washington, DC. attributed to the World products, for which Europe set up a warning in 2013 ———. 2017. “An Evaluation of Border Management Bank or any other affiliated following the outbreak of aflatoxin, and pet food prod- Reforms in a Technical Agency.” Policy Research organizations. Nor do any ucts, for which the number of shipments to be sampled Working Paper 8208, World Bank, Washington, DC. of the conclusions represent increased dramatically in 2015. Fernandes, Ana M., Russell H. Hillberry, and Claudia N. official policy of the World 4. The regression model estimates a relationship Berg. 2016. “Expediting Trade: Impact Evaluation of Bank or of its Executive between an indicator for noncompliance of an import an In-House Clearance Program.” Policy Research Directors or the countries shipment and the associated sampling rate. Sampling Working Paper 7708, World Bank, Washington, DC. they represent. rates are calculated as the ratio between the number of shipments sampled and the total number of imported To order additional copies shipments for each broad product group and semester. contact Jenny Datoo, 5. This control was handled by including in the OLS managing editor, regressions fixed effects for the product group, the Room F 5P-504, The World Bank, country group, and the semester-year. 1818 H Street, NW, 6. The sampling and noncompliance rates were nega- Washington, DC 20433. tively correlated even in the second semester of 2014, when the funding dispute with the laboratories, which Telephone: undermined the analysis of 2015, was not a factor. 001 202 473 6649 Email: jdatoo@worldbank.org Produced by Carol Siegel Printed on recycled paper (continued from front page) Program on Economic Development, and the Impact Evaluation for Development Impact (i2i) Fund.