40207 An Evaluation of the Effectiveness and Outcomes of WBI FY01-03 Activities: Results from 12 Focus Countries Jaime Quizon Nidhi Khattri Heidi Zia Victoria Gunnarsson WBI Evaluation Studies EG05-108 The World Bank Institute The World Bank Washington, D.C. April 2005 Acknowledgements This evaluation report was prepared for WBI under the overall guidance of Nidhi Khattri (Acting Manager), World Bank Institute Evaluation Group (IEG). It was prepared by Jaime Quizon, Nidhi Khattri, Heidi Zia, and Victoria Gunnarsson. The report benefited greatly from discussions with Marlaine Lockheed (Adviser, IEG), Peter F. Orazem (Iowa State University) and Mario A. Sánchez (IADB), who served as peer reviewers. Humberto S. Díaz (IEG) contributed to the document production. We would also like to acknowledge all IEG team members who collected the data used in this study. The individual country studies based on these data are listed in the reference section of this report. WBI Evaluation Studies are produced by IEG to report evaluation results for staff, client, and joint learning activities. An objective of the studies is to release findings quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://web.worldbank.org/WBSITE/EXTERNAL/WBI/0,,contentMDK:20252874~pagePK :209023~piPK:335094~theSitePK:213799,00.html Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group (Acting) Ms. Nidhi Khattri Task Team Leader Ms. Heidi S. Zia ii ACRONYMS 2SLS Two-stage least squares CD Capacity Development CEO Chief Executive Officer CRS Client Registration System FGD Focus Groups Discussion FY Fiscal Year GNI Gross National Income IADB Inter-American Development Bank IEG World Bank Institute Evaluation Group K&S Knowledge & Skills LIC Low-Income Country LMIC Lower Middle-Income Country MGD Millennium Development Goals NGO Non-Governmental Organization OLS Ordinary Least Squares RCET Regional Capacity Enhancement Team VC Video Conference WBI World Bank Institute WBIEN World Bank Institute Environmentally & Socially Sustainable Development Division WBIFP World Bank Institute Finance & Private Sector Development Division WBIHD World Bank Institute Human Development Division WBIKL World Bank Institute Knowledge & Learning Services WBIPR World Bank Institute Poverty Reduction & Economic Management Division iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS.......................................................................................................ii ACRONYMS ........................................................................................................................iii TABLE OF CONTENTS .........................................................................................................v EXECUTIVE SUMMARY .....................................................................................................vii Objectives and Methodology ...................................................................................vii Key Findings............................................................................................................vii Conclusions and Recommendations .........................................................................ix 1. INTRODUCTION.............................................................................................................1 Background ................................................................................................................ 1 Evaluation Objectives and Questions......................................................................... 2 2. METHODOLOGY............................................................................................................3 Data Sources .............................................................................................................. 3 Analytical Approach and Evaluation Limitations...................................................... 5 3. RESPONDENT CHARACTERISTICS AND ACTIVITY FEATURES .....................................7 Respondent Characteristics........................................................................................ 7 Activity Features........................................................................................................ 9 4. ACTIVITY RELEVANCE, EFFECTIVENESS, AND USE OF KNOWLEDGE AND SKILLS...12 Activity Relevance................................................................................................... 12 Activity Effectiveness.............................................................................................. 14 Use of Information, Knowledge and Skills (K&S) Acquired in the WBI Activity..................................................................................................................... 15 5. DETERMINANTS OF WBI ACTIVITY EFFECTIVENESS AND PARTICIPANTS' USE OF K&S ACQUIRED FROM WBI LEARNING ACTIVITIES ...............................................18 The Basic Model ...................................................................................................... 18 Determinants of Activity Effectiveness ................................................................... 20 Determinants of Participants' Use of the Information, Knowledge and Skills (K&S) Acquired from the WBI Learning Activity........................................ 26 v 6. CONCLUSIONS AND RECOMMENDATIONS..................................................................30 Determinants of Activity Effectiveness and Impact ................................................ 30 WBI Effectiveness Over Time................................................................................. 32 WBI Data Recording and Reporting........................................................................ 32 Recommendations.................................................................................................... 33 References................................................................................................................ 34 Annexes.................................................................................................................... 37 LIST OF ANNEXES Annex 1: Survey Questionnaire for Participants from Brazil, Egypt, Russia, Sri Lanka and Thailand (Activities Held in FY01-02)............................................. 39 Annex 2: Survey Questionnaire for Participants from Guatemala, Indonesia, Kenya, Nigeria, Tajikistan and Yemen (Activities Held in FY02-03) .................... 46 Annex 3: Survey Questionnaire for Participants from Burkina Faso (Activities Held in FY02-03) ................................................................................... 52 Annex 4: 2002 Gross National Income (GNI) Per Capita by Country .................... 60 Annex 5: Respondent Participant Characteristics by Country................................. 61 Annex 6: Correlations Between Outcome Measures, Income and Time of Delivery ............................................................................................................... 62 Annex 7: Distribution of Effectiveness and Use Ratings......................................... 63 Annex 8: Ratings of Effectiveness and Use Components by Country GNI/capita................................................................................................................ 64 Annex 9: Variable Descriptions and Summary Statistics Regression Analysis Data ........................................................................................................... 67 Annex 10: Alternative Regression Specifications ................................................... 70 vi EXECUTIVE SUMMARY In FY02, the World Bank Institute developed a new country-focused, capacity development strategy. In response to and support of this strategy, in FY03, the Institute Evaluation Group (IEG) initiated country-focused evaluations in WBI focus countries and completed evaluations in 12 countries. This report builds on these evaluations, consolidates the data collected in the 12 countries, and assesses the effectiveness and outcomes of WBI programs over time (FY01-03) and by country economic status. OBJECTIVES AND METHODOLOGY The main questions we investigate in this report are: (a) What are the main determinants of WBI effectiveness and outcomes? (b) Has the effectiveness of WBI activities changed over time (i.e., between FY01 and FY03)? The study investigates these questions using two main sources of data: (a) WBI's Client Recording System data on WBI activities and participants; and (b) survey and focus group data collected by IEG during FY03 and FY04. The sample comprised a random selection of 11 percent of the participant population from the following countries: Brazil, Burkina Faso, Egypt, Guatemala, Indonesia, Kenya, Nigeria, Russia, Sri Lanka, Tajikistan, Thailand and Yemen. The overall response rate was 71 percent. Sixty percent of the survey respondents were male and 75 percent held middle- or senior-level positions. Forty- five percent of the respondents represented local, provincial or national governments and another 41 percent worked in the private sector or at universities. The evaluation uses a two-stage regression model to examine the influence of time of activity delivery (i.e., fiscal year), country-level factors (e.g., GNI per capita), and activity-level features (e.g., development of action plans) on participants' ratings of WBI activity effectiveness and use of knowledge and skills gained. The report also highlights differences in effectiveness and use of knowledge and skills gained, over time and by country economic level. KEY FINDINGS Effectiveness is higher in lower income countries. Participants from lower- income countries rate WBI activity effectiveness significantly higher than their lower- middle-income country counterparts do, controlling for participant characteristics, WBI activity features, and exogenous factors (including the time of activity delivery). This finding is true for all measures of activity effectiveness. vii More recent activities are more effective. There is an independent time of activity delivery effect. Activities delivered during the last six months of this evaluation (January- June 2003) obtained significantly higher ratings on overall effectiveness than those delivered in the first six months (July-December, 2000). The model used predicts an increase of 12 percent between the first and last six-month period, from an average rating of 4.72 to 5.49, based on a seven point scale. The effectiveness increase over time is the largest for providing strategies and approaches to development. The predicted increase in this area of effectiveness was 17 percent between the first and the last six-month period, from an average rating of 4.5 to 5.4 on a seven point scale. The effectiveness increase over time in building knowledge and skills is also significant. The increase in this area of effectiveness was 12 percent between the first and last six-month period, from an average rating of 4.8 to 5.7 on a seven point scale. The development of action plans as part of the WBI learning activity has the largest and most significant positive effect on activity effectiveness. Developing action plans during the learning activity can boost the overall effectiveness measure by a considerable 13 percent. This finding is consistent across a number of IEG studies of WBI activity effectiveness and outcomes. Other activity features, such as mode of activity delivery and the WBI thematic sponsor, do not influence ratings of effectiveness. Participants who are proficient in the language and terminology of instruction report higher effectiveness. However, on average, WBI participants rate themselves as being highly proficient in both the language of instruction and in the technical terminology of the activity. Targeting of learning to more proficient participants or using national languages as the medium of instruction are not, therefore, likely to yield additional gains in effectiveness. There are, of course, specific WBI learning activities and/or countries in which significant gains in effectiveness may be possible through attention to these issues. Increased effectiveness leads to increased use of the knowledge and skills acquired from the WBI activity. This finding implies that the same factors that determine overall effectiveness also affect, albeit indirectly, participants' overall use of the knowledge and skills acquired from WBI activities. Use of knowledge and skills acquired is higher in lower-income countries. GNI per capita is negatively related to participants' use of the knowledge and skills gained. This suggests that participants from lower-income countries benefit more from WBI learning activities than do their counterparts from more affluent countries. It also suggests that WBI's current activities may be better tailored to the needs of lower-income countries. Participants in high-level positions report higher effectiveness of WBI activities, and they use knowledge and skills more frequently than do participants in lower-level positions. This is particularly true with regard to the activity's effectiveness in providing strategies and approaches, and the use of acquired knowledge and skills for operational rather than academic purposes. This finding implies that WBI should continue to select participants carefully. viii Workplace factors significantly increase use of knowledge and skills gained through the WBI activity. Participants perceive positive workplace factors (e.g., work procedures, incentive systems, funding) to be helpful in their use of the knowledge and skills gained through the WBI activity. Country-related factors (e.g., political and social groups, country policies or the country's general readiness for reform), however, have no relationships with the frequency of participants' use of WBI-acquired knowledge and skills. Differences in effectiveness and use across WBI thematic groups are explained by their differential use of action plans. While there are, in fact, significant bivariate differences in the average overall effectiveness and use ratings among the WBI program participants, these differences are largely explained by the varying use of action plans among the thematic groups. The programs with the highest average ratings of overall effectiveness, WBIPR and WBIHD, also had the highest proportion of participants who reportedly prepared action plans. CONCLUSIONS AND RECOMMENDATIONS We conclude first that WBI performs more effectively in lower-income countries than in lower-middle-income countries. Second, we find that WBI activity effectiveness and utilization has increased over the period FY01-03. We also note an important finding that the WBI participant data recording and reporting system needs improvement. Of the total number of participants registered in the system, only 48 percent were participants with recorded names. The evaluation's three main recommendations are to: (1) Continue with the action planning approach, especially if the activity objective is to help participants apply WBI-acquired knowledge and skills to development. For task managers unfamiliar with this approach, it may be useful to organize a workshop featuring experienced task managers who can share their techniques and lessons learned. (2) Continue to include participants holding high-level positions, for greater impact. (3) Review the activity content and approach to activity implementation in lower- middle-income countries to assess whether or not changes are warranted. ix x 1. INTRODUCTION BACKGROUND 1.1 In FY02, the World Bank Institute (WBI) embarked on a new, country-focused capacity development (CD) strategy. The Institute established a country pillar and the Regional Capacity Enhancement Team (RCET) to give new prominence to country needs as key determinants for the design and delivery of WBI sector/thematic products and services. To take this initiative forward, WBI in collaboration with the Regions identified 33 focus countries in which to channel its country-focused services initially. Today, WBI continues to shift its strategy towards working more closely with World Bank country operations to define and prioritize country-level capacity needs in these countries, develop a more country-focused capacity development program in collaboration with them, and better align WBI's CD services with the Bank's overall country strategy. 1.2 In response to the Institute's country-focused strategy, the Institute Evaluation Group (IEG) initiated several country-focused evaluations in WBI focus countries to report on and support WBI's results-based agenda.1 The objectives of these evaluations were to assess the effectiveness and outcomes of past WBI programs in the selected countries and to establish initial baselines against which the results of the country- focused interventions could be assessed. Data were collected in two sets, in FY03 and FY04. The first set includes data on FY01-02 WBI activities for participants from lower- middle-income countries (Thailand, Russia, Brazil, Egypt, and Sri Lanka), while the second set includes data on FY02-03 WBI activities for participants from mostly low- income countries (Burkina Faso, Kenya, Nigeria, Indonesia, Tajikistan, Yemen and Guatemala). Table 1 lists IEG's evaluations completed in WBI's focus countries in FY03-04 and those to be completed in FY05-06. Table 1: Focus country evaluations completed and to be completed, by fiscal year FY03 FY04 FY05 Region Future Completed Completed Data collection AFR Burkina Faso, Kenya, Nigeria Ghana, Tanzania, Ethiopia Chad, Senegal, Madagascar EAP Thailand Indonesia China, Vietnam Laos ECA Russia Tajikistan Turkey Bosnia & Herzegovina LCR Brazil Guatemala Mexico, Bolivia MNA Egypt Yemen Iran, Morocco SAR Sri Lanka India, Bangladesh Afghanistan, Pakistan Total 5 7 12 7 1Quizon, Jaime B., "Measuring, Monitoring and Managing for Development Results: An Approach to Monitoring and Evaluation in World Bank Institute (WBI) Country Focused Activities" WBIEG Working Paper (final draft), November, 2002. 1 EVALUATION OBJECTIVES AND QUESTIONS 1.3 This report extends the analysis of the FY04 IEG report, "The Effectiveness and Use in FY03 of WBI FY01-02 Activities: A Baseline Assessment in Five Countries."2 It now covers twelve countries, and builds on the individual IEG retrospective evaluations already completed in these twelve WBI focus countries: Brazil, Burkina Faso, Egypt, Guatemala, Indonesia, Kenya, Nigeria, Russia, Sri Lanka, Tajikistan, Thailand, and Yemen.3 1.4 The key evaluation questions that this report attempts to answer are: (a) What are the main determinants of WBI effectiveness and outcomes? (b) Has the effectiveness of WBI activities changed over time, i.e., between FY01 and FY03? 1.5 In answering these questions, the study investigates the relationship of country level factors - such as per capita income - with participants' ratings of outcomes, in addition to activity-level features and participant characteristics. The study also provides aggregate baseline measures of the relevance and effectiveness of WBI activities and of participants' use of the knowledge and skills they acquired from attending these activities. 1.6 The report is organized as follows: Chapter 2 provides an overview of the evaluation methodology; Chapter 3 covers descriptive statistics on the participants and activities included in the sample; Chapter 4 discusses the relevance, effectiveness, and use of WBI activities; Chapter 5 presents the analyses on the determinants of activity effectiveness and use, including change over time; and the final chapter summarizes the study's conclusions and recommendations. 2Quizon, J. B., C.L. Chard, M.E. Lockheed. "The Effectiveness and Use in FY03 of WBI FY01-02 Activities: A Baseline Assessment in Five Countries." WBI Evaluation Studies, EG04-86, May 2004. 3These reports are authored by IEG staff and consultants and are available online at http://www.worldbank.org/wbi/evaluation/puball.htm. The reports are listed in the reference section. 2 2. METHODOLOGY DATA SOURCES 2.1 This evaluation is based on three main sources of data, including (a) participant surveys conducted in twelve focus countries covering WBI activities implemented in FY01 through FY03; (b) activity-related information from WBI's Client Recording System (CRS); and (c) country-level per capita gross national income (GNI) from the Bank's "Country at a Glance" sources. We supplement these data with qualitative information gathered from focus group discussions (FGDs) with participants of selected activities.4 Participant survey population, sample, and response rates 2.2 Participant data collected in FY03 (in the earlier set of five countries) cover learning activities delivered during FY01-02. The data collected in FY04 (in the later set of seven countries) cover WBI activities delivered during FY02­03 (see Table 2). Table 2: Countries, activity delivery times and survey times Region Earlier (FY03) survey: Later (FY04) survey: Covers activities in FY01-02 Covers activities in FY02-03 AFR Burkina Faso, Kenya, and Nigeria EAP Thailand Indonesia ECA Russia Tajikistan LCR Brazil Guatemala MNA Egypt Yemen SAR Sri Lanka Total 5 7 2.3 The earlier FY03 surveys cover countries classified, according to the World Bank, as lower middle-income countries (LMICs), whereas the FY04 surveys cover countries that are mostly classified as lower-income countries (LICs).5 Annex 4 provides the GNI per capita by country for 2002, the one year common among all countries. The implications of these differences in income categories are discussed further in Chapter 5, especially in Box 1. Pooled FY01-03 data confounds activity delivery time with country income level. Thus, observed changes over time in any single survey variable, could be 4Full descriptions of the methods used for sampling the survey and focus group participants are provided in each of the country-specific evaluations. 5Exception: Guatemala which is a LMIC but was surveyed in the later round. 3 due to the shift in sampled participants from LMICs to participants from LICs as much as due to time. This issue is discussed in Chapter 5. 2.4 Participant surveys in each of the twelve countries involved either the full population or a random sample of 18 to 35 percent of the participants who were nationals of that country (see Table 3). The studies employed a common questionnaire (translated for and pre-tested in each country) with some minor variations. Exemplar questionnaires are attached in Annexes 1, 2 and 3. 2.5 The initial source of WBI alumni sampling is WBI's Client Recording System (CRS) database. The second column in Table 3 shows the number of participants attending WBI activities that lasted more than one day.6 The third column shows the number of participants attending activities of more than one day whose names were actually entered in the CRS. The fourth column records the number of WBI participants in each country who were eligible to be surveyed, i.e., participants with contact information from one of four sources: e-mail address, fax number, telephone number or mailing address in the selected country. The final three columns indicate the numbers of reachable participants and respondents, and the response rate. Table 3: Participant population, sample size, and survey response rates by country Participants Participants in Participants in with names in activities Reachable activities activities greater than participants Adjusted greater than greater than one day eligible (out of response Countries one day* one day to be surveyed sampled) Respondents rate Brazil 1826 300 298 231 119 52% Burkina Faso 1975 597 408 284 145 51% Egypt 882 248 227 203 171 84% Guatemala 601 182 132 84 45 54% Indonesia 731 693 417 (240 sampled) 151 114 75% Kenya 917 727 444 (147 sampled) 144 133 92% Nigeria 1836 1175 854 (413 sampled) 165 70 42% Russia 3046 1628 1628 (300 sampled) 239 218 91% Sri Lanka 258 184 169 153 122 80% Tajikistan 181 162 86 65 57 88% Thailand 1536 734 650 (236 sampled) 186 160 86% Yemen 93 100 65 52 28 54% Total 13882 6730 5313 (2721) 1957 1382 71% Share of total 48% 38% (20%) 14% 10% participants * Excludes World Links activities. 6Excludes World Links activities. 4 2.6 Seventy nine percent of the sampled participants were eligible to be surveyed, based on some form of contact information recorded. Due to inaccurate or outdated CRS contact information, only about 72 percent of the eligible participants were reachable. The adjusted response rate varied from 42 in Nigeria to 92 percent in Kenya. 2.7 WBI's new approach to capacity development may require regular follow-up activities with its alumni, and therefore it will be crucial for the Institute to ensure the accuracy of its CRS data. As previously noted in almost all of the twelve individual country evaluation reports, invalid entries comprised approximately one-third of the CRS database on participants. Invalid entries include participants who never attended the learning activity, who were organizers/instructors/administrative assistants, or those who had completely incorrect and untraceable contact information. Furthermore, as is evident from Table 3, column 3, approximately 52 percent of the total number of participants lacked contact information. Participant focus group discussions 2.8 IEG held at least two focus group discussions (FGDs) in each of the twelve countries. In general, we randomly selected the WBI activities that were the topics of these in-country FGDs. ANALYTICAL APPROACH AND EVALUATION LIMITATIONS 2.9 We use a two-stage least squares (2SLS) quantitative approach to analyzing the data and assessing the effectiveness and use of WBI activities. In the next sections, we report several descriptive tables and the results from our quantitative model. Where relevant, we augment these quantitative findings with information derived from the FGDs and interviews we conducted in each country. This qualitative inquiry elaborates on issues that could not be covered in depth by the quantitative analysis because of the standard limits to collecting information from formal surveys. 2.10 We note three key limitations to our approach. First, although the response rate is high, our final respondent sample may not be representative of the population due to issues noted regarding the CRS data; we have no way of checking the severity of this problem. Our best assumption is that the problem is random with no biases inherent in missing data. Or, if biases exist, we assume that they work similarly across time and countries and, therefore, do not affect our key results. Second, the issue of effectiveness is explored from the perspective of those participants who chose to attend the WBI activity, and not those individuals who may have eschewed participating in such activities because of their understanding or belief that such activities are not effective or useful. However, our purpose is to assess whether, from the participants' perspective, the activity was useful, but we acknowledge the fact that the knowledge, skills, and information the participants gained from the activity could have been acquired elsewhere as well. The evaluation focuses more on the effectiveness of the WBI activity process and less on the specific knowledge and skills outcomes that could be acquired through other (perhaps more cost-effective) means. Thus, we have no external comparisons for 5 assessing the relative effectiveness of WBI activities. Finally, the evaluation does not attempt to measure development impact of these activities. 6 3. RESPONDENT CHARACTERISTICS AND ACTIVITY FEATURES 3.1 This chapter describes the key characteristics of the survey respondents and the main features of the WBI learning activities that they attended. The chapter also defines the factors that may influence participants' ratings of the effectiveness and usefulness of WBI activities. It describes the variables included in the regression analysis presented later in Chapter 5. RESPONDENT CHARACTERISTICS 3.2 For the purposes of this chapter, the twelve countries may be divided into two groups: lower-middle-income countries and lower-income countries. The lower-middle- income countries are countries with a 2002 GNI per capita above the average of all twelve countries (USD1,279).7 These consist of Brazil, Russia, Thailand, Guatemala and Egypt. The lower-income countries, with 2002 GNIs per capita below the twelve-country average, are: Sri Lanka, Indonesia, Yemen, Kenya, Nigeria, Burkina Faso and Tajikistan.8 3.3 Table 4 shows the distribution of respondents by four characteristics: gender, position, type of work and type of work organization. The majority is male (61 percent), but the share is lower in the lower-middle-income countries.9 For the highest income countries in the sample, Russia and Thailand, the shares of males are smaller: 56 and 42 percent, respectively. For the lowest income countries, Burkina Faso and Nigeria, the shares of male respondents are larger: 77 and 67 percent, respectively. 3.4 Most respondents held either a middle-level or senior-level position. Twenty-nine percent reported that they held a middle-level position, such as program manager, project leader, assistant professor or technical expert. Another 46 percent indicated that they were senior professionals, such as department heads, division heads, associate professors, or senior researchers. 3.5 The types of jobs that respondents were performing varied widely. Table 4 shows that 27 percent reported that they were primarily involved in management or administration, 12 percent indicated that they were policy makers or were involved in legislative issues. Twenty-seven percent were involved in research, teaching, or traditional academic activities, while 14 percent worked in the provision of other 7Source: World Bank "Country at a Glance" 2002. 8The World Bank officially classified Sri Lanka as a lower middle income country (see Annex 4). 9Annex 5 displays the characteristics of the respondents for each of the twelve countries. 7 services. The majority of respondents represented local, provincial or national governments (45 percent) and over 40 percent worked in the private sector or at a university. Participant characteristics by country are presented in Annex 5. Table 4: Respondent characteristics (unweighted) Respondent characteristics Distribution (%) Gender Female 39.2 Male 60.8 Position Highest 10.3 Senior 46.3 Middle 29.2 Junior 14.2 Primary type of work Management/administration 27.2 Policy making/legislative 12.2 Research or teaching 26.8 Provision of services 14.0 Other 19.8 Work organization Local/Provincial/National Gov. 44.5 Non-Governmental Org. (NGO) 5.5 Private Sector 21.4 University 19.8 Other (incl. Donor org. and media) 8.8 3.6 Focus group participants offered both positive and negative remarks on having participants from different professional backgrounds and/or with varying levels of content knowledge engaged in the same WBI learning activity. Many praised WBI for facilitating networking with participants of different backgrounds and for bringing together the various stakeholders in the same activity. On the other hand, FGD participants from the majority of countries expressed that the selection and targeting of participants was poor and that the content did not fit the specific needs of the participants. 3.7 The majority of participants rated themselves to be proficient in both the language and terminology of instruction, as shown in Figures 1a and 1b. However, language proficiency is significantly lower, while terminology proficiency is slightly higher, in lower-middle-income countries (language proficiency mean = 5.84, terminology proficiency mean = 5.73), as compared with proficiency in lower-income countries (language proficiency mean = 6.16, terminology proficiency mean = 5.65). Ratings of respondents' proficiency levels in the language and in the terminology of instruction are strongly and positively correlated (0.61, p<0.05). Eighty-nine percent of those with 8 "good" proficiency in the terminology of instruction also rated their language proficiency as "good". 3.8 Although survey respondents, on average, rated their language proficiency as high overall, FGD participants from lower-income countries and from Egypt and Thailand strongly expressed the need for activities to be held in-country and in their own languages. Figure 1a: Respondents' level of Figure 1b: Respondents' level of proficiency in the language of proficiency in the terminology used instruction (N=1347) (N=1335) Poor Poor 2% 3% Some Some 28% 36% Good Good 70% 61% ACTIVITY FEATURES 3.9 Table 5 displays the distribution of respondents participating in WBI activities with four types of features. Nearly forty percent of participants reported that they produced action plans10 in the WBI learning activities they attended. However, a larger share of participants from lower-income countries reported action plans (47 percent) than did participants from lower-middle-income countries (32 percent). Moreover, the share of respondents reporting that they prepared action plans increased from less than a third in FY01 to close to 50 percent in FY02 and FY03 (Figure 2). This latter finding may be due to the nature of the data. As we have already noted, the composition of our participant survey sample shifted from lower-middle-income countries to lower-income countries over time. The effects of this change are explored in Chapters 4 and 5. 10Action plans are a flexible pedagogic tool that helps participants operationalize and implement their new knowledge and skills in the context of their jobs and the needs of their countries, mainly by "learning by doing". Action plans and their effect on learning and learning outcomes are described in greater depth in "Pedagogy for Effectiveness and Impact: Evidence Regarding Action Learning and Course Duration." WBI Evaluation Brief, June 2003. 9 Figure 2: Share of respondents preparing action plans, by date of activity 0.6 0.49 0.5 0.47 s 0.43 0.43 dent 0.4 pon 0.30 esr 0.3 of e 0.2 0.17 harS 0.1 0 FY01Jul-Dec FY01Jan-Jun FY02 Jul-Dec FY02 Jan-Jun FY03 Jul-Dec FY03 Jan-Jun 3.10 Less than a third of the respondents reported that they had received follow-up from WBI or other partner organizers after the conclusion of the activity (Table 5). The data reveal no differences in the frequency of follow-up with the participants between the lower-income countries and lower-middle-income countries. Focus group participants from all twelve countries, however, strongly stated that they desired more follow-up from the organizer to help them implement what they learned and to continue to learn. Some FGD participants also mentioned that they lacked both contact and direct means of communication with WBI. Table 5: Distribution of respondents by activity features/WBI policy variables (unweighted) Activity features/WBI policy variables Distribution (%) Action plan Prepared 38.8 Not prepared 61.2 Follow-up Contacted 30.7 Not contacted 69.3 Delivery mode Classroom/F2F 65.0 VC/Global dialog 17.4 Blended/Multi-media 14.7 Web-based/Computer/Electronic 2.9 WBI thematic group WBIFP 38.8 WBIEN 21.9 WBIHD 20.6 WBIPR 16.7 WBIKL 2.1 10 3.11 The majority of respondents reported attending classroom or face-to-face activities. However a slightly higher proportion of respondents from lower-income countries (20 percent) participated in blended and web-based activities as compared with respondents from lower-middle-income countries (14 percent). Also, a higher share of respondents from lower-middle-income countries attended activities using videoconferencing or global dialogs than did respondents from lower-middle-income countries (21 percent versus 13 percent). 3.12 Respondents attended activities spread across WBIFP, WBIEN, WBIHD and WBIPR. The share of respondents in WBIFP activities was higher in the lower-middle- income countries (65 percent of respondents attending WBIFP activities were from lower-middle-income countries), while reverse was true for the WBIHD and WBIPR activities (63 and 54 percent of respondents attending WBIHD and WBIPR activities were from lower-income countries). 3.13 Only 37 percent of respondents could provide comparisons between WBI activities and activities of similar nature offered by other organizations. For those respondents who provided comparisons, WBI activities were rated as similar in usefulness and effectiveness to the non-WBI activities they had attended (Figure 3). Seventy-four percent of respondents rated the effectiveness and usefulness of the WBI activity they attended as "about the same" as non-WBI activities they had attended. However, this rating was higher in lower-income countries (seven percent rated WBI activities as much less effective/useful and 29 percent rated them much more effective/useful) than in lower-middle-income countries (12 percent versus six percent). Figure 3: Comparison of WBI activities with non-WBI activities for effectiveness/usefulness (N=515) Much more Much less effective/useful effective/useful 16% 10% About the same 74% 11 4. ACTIVITY RELEVANCE, EFFECTIVENESS, AND USE OF KNOWLEDGE AND SKILLS 4.1 This chapter presents the respondents' ratings (un-weighted) of WBI activity relevance to country needs, activity effectiveness, and their use of the knowledge and skills (K&S) they acquired from the WBI activity.11 We disaggregate the results by income level and by time. The disaggregated results must be interpreted with caution, however, due to the structure of the data discussed earlier in paragraph 2.3. More comprehensive results are discussed in Chapter 5. ACTIVITY RELEVANCE 4.2 Figure 4 ranks the countries included in our study according to their GNI per capita. As shown, except for Brazil, participants' ratings of relevance of WBI activities to their countries' needs (on a 7-point scale where 1 = "not relevant at all" and 7 = "extremely relevant") are generally higher in lower-income countries (see Table A1 in Annex 6).12 The correlation between GNI per capita and relevance ratings is negative and significant (­0.16, p<0.05), when Brazil is treated as though it were two separate countries ­ a Northeast Brazil with a lower GNI per capita income and the rest of Brazil with a higher GNI per capita.13 The average relevance rating across all twelve countries is 5.49. 11The 12-country average, as presented in Figure 4, is the simple unweighted average of the twelve countries. Average ratings, when weighted by the total population of participants eligible to be surveyed in each country (column 3 in Table 3), fall within two percent of the unweighted ratings. 12Paragraph 3.2 provides a definition of lower- and lower-middle-income countries in our data. 13On average, Brazil has the highest GNI per capita in our sample. However, a large share of the WBI activities in Brazil took place in the Northeast where per capita GNI is only about half of the national average. In the regression analysis in Chapter 5, we split Brazil into two separate "countries" ("Northeast" with 50 percent of the national average GNI per capita and "the rest" with the national average GNI per capita) and differentiate the Brazilian respondents accordingly, based on their residence. We realize that these income divisions are not completely adequate. We note that we would have done a similar "split" for other countries that appear to go against the general trend of declining WBI activity relevance with rising GNI per capita, such as Nigeria and Yemen (in Figure 4). Unfortunately, however, we lack similar per capita GNI data breakdowns by region for these countries. 12 Figure 4: Ratings on relevance of the activity to the participant's country needs, by country* 7 6.32 6.24 5.91 6 5.81 5.75 5.61 5.49 5.26 5.36 5.44 ngsitar 5.11 4.92 5 4.84 nae M 4 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) * The 12-country average is unweighted 4.3 Ratings of relevance are higher for the more recent activities as shown in Figure 5. Over the three fiscal years, relevance ratings increased from 5.15 in the first six months of FY01 to 5.84 in the last six months of FY03. Without controlling for other factors that may affect relevance ratings over the time period, this corresponds to a 13 percent total increase. The date the WBI learning activity was held (represented as the quarter of the fiscal year when the activity was delivered)14 and relevance ratings are significantly positively correlated (0.17, p<0.05). Figure 5: Ratings on relevance of the activity to the participant's country needs, by date of delivery 7 5.84 6 5.63 5.66 sgni 5.41 5.15 5.21 atr 5 eanM 4 3 FY01 FY01 FY02 FY02 FY03 FY03 July-Dec. Jan.-June July-Dec. Jan-June July-Dec. Jan-June 4.4 This positive result must be interpreted with caution, however. Although average ratings for relevance were on the higher end, focus group participants from all twelve 14Fiscal year quarter, when the activity was delivered is defined on a scale from 1-12 where: = 1 refers to delivery in the first quarter of FY01; = 2 refers to delivery in the second quarter of FY01, and so on to = 12 which refers to delivery of the activity in the fourth quarter of FY03. 13 countries stated that the content could be better targeted to the national context of their country, and that the case studies the activities used tended not to be relevant to their countries' specific development issues. ACTIVITY EFFECTIVENESS 4.5 The average overall effectiveness rating is 5.10 (on a scale of 1= "not effective at all" to 7 = "extremely effective"). Similar to ratings on activity relevance, participants from lower-income countries tended to provide higher overall effectiveness ratings, with the exception of Brazil (see Figure 6).15 The correlation between GNI per capita and overall effectiveness is negative (-0.20, p<0.05, see Table A2 in Annex 6). The distribution of the respondent ratings on effectiveness is presented in Figure A1 in Annex 7. Figure 6: Ratings on overall effectiveness, by country GNI/capita* 7 6 sgnitar 5.55 5.60 5.46 5.45 5.60 5.38 5.20 5.07 5.10 5.10 5 4.74 n 4.67 4.64 Mea 4 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) * The 12-country average is unweighted 4.6 To obtain a more detailed view of activity effectiveness, we disaggregated the overall effectiveness ratings into three separate components; (a) effectiveness in raising awareness and understanding country development issues; (b) effectiveness in building knowledge and skills; and (c) effectiveness in providing strategies and approaches16 for addressing organizational and country needs.17 The ratings for these three areas of effectiveness by country are displayed in Figures A3, A4 and A5 in Annex 8. 15As defined in the survey questionnaires in Annex 1, 2 and 3, effectiveness is specified in terms of six areas (in all country surveys except for Burkina Faso which asked about nine areas). Overall effectiveness refers to the simple average of these effectiveness areas. 16One should note that activity effectiveness in providing strategies and approaches measures the output of strategies and approaches and not the outcome (quality) thereof, which is also of importance when determining the developmental impact of the interventions. 17The scale reliability coefficients, Cronbach's alpha, for these three effectiveness components are 0.73 or higher. 14 4.7 The negative correlation between income and effectiveness ratings holds for each of these measures of effectiveness. The negative relationship is especially strong in the areas of building knowledge and skills and providing strategies and approaches (see Table A1 in Annex 6).18 On average, effectiveness in raising awareness is rated the highest at 5.34, followed by effectiveness in building knowledge ands skills at 5.25 and effectiveness in providing strategies and approaches at 4.93 on the 7-point scale. 4.8 More recent activities received higher ratings, but, as noted already, these differences over time must be considered with caution. As presented in Figure 7, in raw numbers effectiveness increased from the first six months in the period (overall effectiveness mean 4.72) to the last six months (overall effectiveness mean 5.49) by 16 percent. In Chapter 5 we investigate what factors contribute to this increase and predict the percent increase in effectiveness over the time period that is due to participant characteristics, WBI policy variables and country income. Figure 7: Ratings on effectiveness, by date of WBI activity delivery 7 Overall effectiveness 6 5.71 5.65 5.53 5.59 Effectiveness sgntiar 5.285.31 5.49 5.41 5.36 in raising 5.08 5.20 5.15 awareness 5.02 5.15 5.19 5.02 5.03 5.06 5 4.85 4.72 4.79 4.83 4.50 4.61 Effectiveness in building Mean knowledge and 4 skills Effectiveness in providing strategies and 3 approaches FY01 Jul-Dec FY01 Jan-Jun FY02 Jul-Dec FY02 Jan-Jun FY03 Jul-Dec FY03 Jan-Jun USE OF INFORMATION, KNOWLEDGE AND SKILLS (K&S) ACQUIRED IN THE WBI ACTIVITY 4.9 Survey respondents rated the frequency with which they used the information, knowledge and skills (K&S) they acquired from the WBI activity they attended. The rating scale ranged from 1 = "not used at all" to 7 = "used very often". The average overall use rating is 4.42 on this 7-point scale.19 Similar to effectiveness ratings and as displayed in Figure 8, overall use ratings are higher in lower-income countries. The correlation between GNI per capita and overall use is negative (-0.25, p<0.05). The distribution of the respondent ratings is presented in Figure A2 in Annex 7. 19As defined in the survey questionnaires in Annexes 1, 2 and 3, use is specified in terms of six to eight areas (depending on the version of the survey instrument used). Overall use refers to the simple average of all these different areas of use. 15 Figure 8: Ratings on overall use, by country GNI/capita* 7 6 sgnitar 5.28 5.16 4.91 5 4.80 4.86 4.85 4.62 4.36 4.42 4.40 4.52 Mean 4 3.81 3.44 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) * The 12-country average is unweighted 4.10 Overall use can be divided into two broad areas: (a) academic use and (b) operational use. Academic use refers to use in teaching, research, and awareness raising. Operational use, on the other hand, refers to use of K&S in implementing work strategies, influencing legislation, and organizing group initiatives. On average, academic use is higher (mean of 4.60) than operational use (mean of 4.25). However, the correlation between GNI per capita and operational use is slightly stronger (-0.27, p<0.05) than between GNI per capita and academic use (-0.21, p<0.05) (Tables A1 and A2 in Annex 6; Figures A6 and A7 in Annex 8). 4.11 With the exception of the last six months in FY03, use ratings increased over the three-year period (Figure 9). Ratings on overall use increased from an average of 3.68 in the first months of FY01 to 4.62 in the last six months of FY03, equivalent to an increase of 26 percent.20 Without controlling for factors that may influence use, we note that participants' operational use ratings increased by 39 percent from the first six month period to the sixth six month period, from an average of 3.26 to 4.53. Because this increase is larger than a similar increase in academic use ratings, it signals somewhat a subtle change in the general orientation of WBI learning events overall. WBI events are now becoming increasingly more partial to practical applications over academic-type uses. 4.12 The results suggest that participants' ratings of activity effectiveness and use vary across countries and over the date of delivery of the learning activity. In the next chapter, we describe our multivariate models that explore more rigorously the relationships between these variables and their contributions to WBI activity effectiveness and use. 20In Chapter 5 we investigate what factors contribute to this increase and predict the percent increase in use over the time period that is due to participant characteristics, WBI policy variables and country income. 16 Figure 9: Ratings on use of knowledge and skills, by date of delivery 7 6 sgnitar 5.00 5 4.87 4.94 4.89 4.76 4.70 4.67 4.62 4.53 4.50 4.39 4.46 Mean 4.18 3.97 4.01 4 3.87 3.68 3.26 3 FY01Jul-Dec FY01Jan-Jun FY02 Jul-Dec FY02 Jan-Jun FY03 Jul-Dec FY03 Jan-Jun Overall use Academic use Opearional use 17 5. DETERMINANTS OF WBI ACTIVITY EFFECTIVENESS AND PARTICIPANTS' USE OF K&S ACQUIRED FROM WBI LEARNINGACTIVITIES 5.1 In this chapter, we investigate the key factors that might explain: (a) the effectiveness of WBI learning activities; and (b) WBI participants' use of the knowledge and skills (K&S) that they acquired from these learning activities. We use a two-stage regression model to understand the direction and the magnitude of the effects of participant characteristics, WBI policy variables (or activity features), and exogenous factors (such as time, country income per capita, and country-specific facilitators and barriers) on alternative outcome measures of activity effectiveness and participants' use of the K&S gained from WBI learning activities. 5.2 It is not exactly clear whether (and the magnitude by which) the key factors (in particular, GNI per capita and time of delivery of the activity) contribute to participants' ratings because, due to the structure of the survey data, these effects may be confounded. In addition, some participant characteristics and/or activity features (that may also vary over time and/or by country) may be the real factors explaining the observed variances in our outcome measures of WBI performance. A key objective of this empirical exercise is to isolate the confounded effects of several factors, including of per capita income and of time, on outcome measures of effectiveness and use. This is useful because it ascertains what underlying factors ­ and thereby, what relevant recommendations ­ are most likely to improve desired outcomes from WBI-supported learning activities. Below we discuss our regression model and the key results from the regressions. THE BASIC MODEL 5.3 Figure 10 below describes the structure of the weighted 2SLS model we use in the analysis.21 In the first stage regression, we define three components of activity effectiveness, i.e., participants' ratings of the activity's effectiveness (ratings are from 1 = "not effective at all" to 7 = "extremely effective") in: (a) raising awareness and understanding of country development issues; (b) building new knowledge and skills; and (c) providing strategies and approaches for addressing organizational or country needs. We also define overall effectiveness as the simple average of these three components. 21This model follows the theoretical structure and modeling technique developed in Khattri, N., et al, "Impact Evaluations of WBI Client Programs, FY00-01." WBI Evaluation Studies, EG03-63, November 2002. For each country, we weight the responses by the total number of participants from that country (in activities greater than one day) eligible to be surveyed (column 3, Table 3). 18 We then explore the WBI participant characteristics, WBI policy (or activity) variables and exogenous factors that explain these individual measures of effectiveness. 5.4 In the second-stage regression, we use the predicted overall effectiveness from the first stage, as well as (a) participant characteristics (e.g., job position level), (b) WBI policy variables (e.g., follow-up) and (c) other exogenous variables (e.g., GNI per capita, workplace facilitators/barriers and country-level facilitators/barriers) to explain participants' ratings of their use of the K&S they acquired from the WBI learning activity (ratings are from 1 = "not at all" to 7 = "very often"). We define two components of participants' use, namely: academic use and operational use. We also define overall use as the simple average of these two components. The exact definitions and summary statistics for the dependent and independent variables are explained in Annex 9. Figure 10: Model for two-stage least squares (2SLS) regression analysis INDEPENDENT VARIABLES PARTICIPANT WBI POLICY EXOGENOUS CHARACTERISTICS VARIABLES FACTORS Gender Action plans GNI per capita Job position level Delivery mode Time of delivery Language proficiency WBI thematic group Terminology ACTIVITY EFFECTIVENESS (stage 1) a) Raising awareness and understanding of country development issues b) Building new knowledge and skills c) Providing strategies and approaches for addressing organizational or country needs INDEPENDENT VARIABLES FACILITATORS & BARRIERS Job position level Follow-up Organizational factors GNI per capita Country level factors UTILIZATION OF KNOWLEDGE AND SKILLS (stage 2) a) Academic use: research, teaching, raising awareness b) Operational use: implement practices, develop strategies, influence legislation, organize initiatives 5.5 Those variables that appear in the top "independent variables box" but not in the bottom in Figure 10 (i.e. gender, language and terminology proficiency (and the 19 interaction term), action plans, delivery mode, WBI thematic group and time of delivery) have effects only on our alternative measures of effectiveness, and have no direct effects on participants' use of the K&S gained from the WBI activity, other than through effectiveness.22 The model also includes two measures of the level of facilitators/barriers to applying the K&S acquired in the WBI activities. As shown, these two exogenous variables directly affect participant's use (the second-stage estimates) but not activity effectiveness (the first-stage regressions). 5.6 The model can also be expressed by the following equations where i symbolizes data on the participant level, j activity level and k country level: Stage 1: ijk = ijk eff gender ,ijk + ijk + ijk lang term lang*term + ijk action + mode income jk + thematic + k jk + time +ijk jk Stage 2: ijk = ^ijk + ijk use eff position+ ijkfollow-up+ k income+ ijk org. fac.+ ijk count. fac.+ijk DETERMINANTS OF ACTIVITY EFFECTIVENESS 5.7 What factors determine participants' ratings of the effectiveness of WBI activities? We tested several variables that might explain: (a) each of the three separate components of activity effectiveness and (b) overall effectiveness, as noted in Figure 10 above. Table 6 reports the results of the OLS regressions of these dependent variables on participant characteristics, WBI policy variables, and the exogenous factors (including country GNI per capita and time of delivery of the activity). Among all the OLS models that we tested, these regression estimates have about the highest explanatory power (i.e., R-square = 0.19 for the overall effectiveness regression). They also include all those explanatory variables that appear to be the most consistently significant and meaningful in the other model specifications.23 These Table 6 results are the most robust among all the other specifications that we explored. 5.8 The columns in Table 6 show the regression results for overall effectiveness (column 1), effectiveness in raising awareness (column 2), effectiveness in building knowledge and skills (column 3) and effectiveness in providing strategies and approaches (column 4). We note the following key results. 22These variables are therefore treated as instruments of overall effectiveness. 23We also explored regressions with other explanatory variables, such as participant age, duration of learning activity, differences in lag time between the activity and the participant survey, country dummies and so on, none of which were statistically significant in explaining overall effectiveness or any of the components of effectiveness. 20 Use of action plans in a WBI learning activity increases activity effectiveness. 5.9 Regression results show that the preparation of action plans in a WBI activity has the largest and most significant effect on activity effectiveness (i.e., = 0.611, p<0.05 for overall effectiveness). Other variations across WBI offerings, particularly in the mode used to deliver the learning activity and in the WBI thematic sponsor of the learning event, do not affect effectiveness. The inclusion of action plans in a learning delivery is the central activity feature explaining effectiveness. For WBI activities without action plans, where their mean overall effectiveness is 4.80 (based on a 7-point scale), the inclusion of action plans in the delivery of the learning activity can boost overall effectiveness by a considerable 13 percent.24 Table 6: OLS Regressions of effectiveness of WBI learning activities, FY01-03 (un-standardized regression coefficients with t-statistics in parenthesis) Overall Effectiveness Effectiveness in Effectiveness in (Aggregated) in Raising Building Knowledge Providing Strategies Effectiveness Awareness and Skills and Approaches Participant characteristics Female 0.124** 0.112 0.112 0.147* (2.01) (1.36) (1.42) (2.05) Position 0.074* 0.065 0.016 0.110* (1.79) (1.23) (0.29) (2.37) Language proficiency 0.316** 0.271* 0.372** 0.280** (3.04) (1.85) (3.28) (2.38) Terminology proficiency 0.337** 0.371** 0.400** 0.272** (2.94) (2.26) (3.01) (2.12) Language*terminology proficiency -0.054** -0.056** -0.069** -0.041** (-2.82) (-2.06) (-3.22) (-1.90) WBI policy variables/Activity features Follow-up 0.053 -0.029 0.124 0.067 (0.78) (-0.33) (1.42) (0.86) Action plan 0.611** 0.642** 0.566** 0.619** (8.76) (7.26) (6.49) (7.73) Blended/Multi-mediaa -0.054 -0.107 0.007 -0.057 (-0.51) (-0.84) (0.05) (-0.48) VC/Global Dialoga -0.120 -0.121 -0.187 -0.085 (-1.20) (-0.99) (-1.56) (-0.75) Web-based/Computer/Electronica -0.139 -0.179 -0.042 -0.188 (-0.61) (-0.58) (-0.17) (-0.73) WBIFPb -0.008 0.036 0.099 -0.082 (-0.11) (0.33) (0.95) (-0.91) WBIHD/WBIKLb -0.090 -0.060 0.014 -0.160 (-0.92) (-0.47) (0.11) (-1.39) (Table 6 continuous on next page) 24Different from the raw change in effectiveness over time presented in Chapter 4 this is the adjusted change in effectiveness predicted by the frequency of action plan development alone. We calculated the predicted percentage improvement by dividing the coefficient on action plans (0.611) by average effectiveness ratings for those who reported no use of action plans (4.80). 21 (Table 6 continued) Overall Effectiveness Effectiveness in Effectiveness in (Aggregated) in Raising Building Knowledge Providing Strategies Effectiveness Awareness and Skills and Approaches WBIPRb -0.059 -0.168 0.130 -0.093 (-0.54) (-1.20) (1.03) (-0.73) Income and time GNI per capita (1000s USD) -0.159** -0.193** -0.149** -0.128* (-2.48) (-2.46) (-1.96) (-1.73) Delivered in FY01 Jan.-Junec 0.133 -0.213 0.146 0.282** (1.48) (-1.57) (1.15) (2.77) Delivered in FY02 July-Dec. c 0.162 -0.192 0.097 0.360** (1.32) (-1.23) (0.60) (2.42) Delivered in FY02 Jan.-Junec 0.317** -0.102 0.304** 0.552** (2.80) (-0.71) (2.02) (4.33) Delivered in FY03 July-Dec. c 0.247* -0.139 0.493** 0.312 (1.46) (-0.59) (2.62) (1.50) Delivered in FY03 Jan.-Junec 0.543** 0.177 0.572** 0.745** (3.54) (0.90) (3.03) (4.10) Constant 2.691** 3.393** 2.709** 2.398** (4.62) (4.14) (4.02) (3.70) N 1073 1042 1065 1065 R-square 0.192 0.105 0.149 0.169 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aClassroom and face-to-face learning is the reference. bWBIEN thematic group is the reference. cFirst six months of FY01 is the reference. 5.10 By making more use of action plans, course content can be made more practically adaptable for participants who, in FGDs, explained that the activity content was otherwise too technical, theoretical or generic in nature. Participants from poorer countries provide higher overall effectiveness ratings. 5.11 Activity effectiveness is higher in lower-income countries. Regression results confirm this finding that we already observed in an earlier chapter (Figure 6). Controlling for participant characteristics, WBI activity features and exogenous factors (including the time of delivery of the activity), participants form the poorer countries rate WBI activity effectiveness significantly higher than their counterparts from more affluent countries. This finding is true for all measures of activity effectiveness. 5.12 Focus group participants emphasized that WBI activities were most effective in raising awareness but that they also provided some new technical knowledge and skills. However, the effectiveness of WBI activities could be improved by increasing the depth and breath of the activities and by offering training to a broader range of participants. More recent WBI activities are more effective. 5.13 In addition to the GNI per capita (or income) effect, there is an independent time of activity delivery effect that is not due to the structure of the data (see Box 1). As shown in Table 6, more recent activities received significantly higher effectiveness 22 ratings than activities delivered early in the FY01-03 period.25 Between the first six- month period (July-December, 2000) and the last six-month period (January-June, 2003) of the three fiscal years, the regression results for overall effectiveness ( = 0.543, p<0.05) suggests that the time factor alone increased this measure of effectiveness by 0.54 units. This is an adjusted 12 percent improvement from the starting overall effectiveness rating of 4.72 (based on a 7-point scale) in the first six-month period, holding constant participant characteristics and activity features.26 Or, put differently, this improvement is equivalent to 24 percent of the total gain in overall effectiveness that was possible from the first six-month period.27 Box 1: The effects of country GNI per capita and time of activity delivery on activity effectiveness One notable feature of the 12-country survey data is the confounding of country GNI/capita with activity delivery and survey collection. The first round of data collection covered activities offered in FY01 and FY02 and countries classified as LMICs. The second round of data collection covered activities offered in FY02 and FY03 and countries typically classified as LICs. Thus, special analyses are required to test the independence of time of activity delivery and country GNI per capita on activity effectiveness and use. Straightforward OLS regressions, estimated from pooling both data sets, may produce unclear results as to the real and independent effects of GNI per capita and of time of activity delivery (as explanatory variables) on the dependent variable (e.g., overall effectiveness). To explore whether our first-stage OLS regressions with the pooled data are able to capture separate GNI per capita and time effects on the dependent variables (or our separate measures of effectiveness as shown in Figure 10), we ran several regressions to test the independence of these effects. First, using data from FY02 only, or the common period for which we have data for all countries in the study. We then compared and tested whether there were any significant differences in the estimated coefficients for GNI per capita from these FY02-only regressions (i.e., holding time constant) with the same coefficient estimates that used the entire FY01-03 data set (i.e., where time of activity delivery is now included as an explanatory variable). Our comparisons indicate generally stable and statistically significant coefficients for GNI per capita in these regressions, suggesting that our OLS estimates from the full-data set with GNI per capita and time as dependent variables actually capture country income effects that are separate from the time effects. This finding is fairly robust to various changes in model specifications. Second, we also ran several specifications that included different variations of interaction terms between GNI per capita and time of activity delivery to testing whether these variables affected the results jointly or independently. The interaction terms were never significant in these specifications, nor in general were GNI per capita and time of activity delivery in regressions where these variables were included alongside the interaction terms. We therefore concluded that our model specification presented in Tables 6 and 7 correctly controls for both income and time effects and that GNI per capita and time of activity delivery individually have significant effects on the outcome measures. The alternative specifications for the FY02 period only, the FY01-03 period including interaction terms and with details on participants' positions are presented in Annex 10. 25We note that this result cannot be explained by the lag time between the delivery of the WBI activity and the conduct of the participants' surveys. Although shorter lag times may result in higher participant ratings of an activity's effectiveness, none of the time lag variables introduced in our several regressions were significant determinants of these effectiveness ratings. 26Different from the raw change in effectiveness over time presented in Chapter 4 this is the adjusted change in effectiveness predicted by the time factor alone. We computed the predicted percentage improvement at the mean by dividing the coefficient on the last six-month period (0.543) by the average effectiveness rating (4.72) in the first six-month period. The same principle is used for explaining the predicted percentage improvement at the mean for other explanatory variables. 27We computed the predicted percentage improvement of what is possible to improve by dividing the coefficient on the last six-month period (0.543) by the difference between the maximum effectiveness score (7) and the average effectiveness ratings (4.72). The same principle is used for explaining the predicted percentage improvement of what is possible for other explanatory variables. 23 Among the three components of overall effectiveness, the increase over time is largest for effectiveness in providing strategies and approaches, followed by effectiveness in building knowledge and skills. 5.14 Although raising awareness was rated as the area where WBI activities were the most effective, the effect of time of activity delivery on effectiveness ratings is especially strong for effectiveness in providing strategies and approaches ( = 0.745, p<0.05) and effectiveness in building knowledge and skills ( = 0.572, p<0.05). Effectiveness in providing strategies and approaches increased significantly by an adjusted 17 percent from the mean of the first six-month period (= 4.50) or by 30 percent of the maximum gain that was possible. Effectiveness in building knowledge and skills also increased by an adjusted 12 percent at the mean (= 4.79), which is an increase of 26 percent of the maximum possible increase. 5.15 The fact that the increase in effectiveness over the three-year period took place in the knowledge and skills and in the strategies and approaches areas rather than in the area of raising awareness (where the independent effect of time is not statistically significant) indicates an important and significant shift in WBI's effectiveness that favors the former rather than the latter effectiveness areas (or deeper rather than broader impacts). Having participants with better language proficiency and technical terminology proficiency improves the effectiveness of WBI activities. 5.16 Participants' proficiencies in the language of instruction of the WBI learning activity and in the technical terminology used in the instruction have significant and positive effects on activity effectiveness. However, WBI participants are, on average, already highly proficient in both language and terminology (means of 5.99 and 5.69, respectively, on a 7-point rating scale). Evaluating the language and terminology effects from the respective coefficients and at the mean of the coefficient of the interaction term indicates that being proficient in the language or terminology increases effectiveness but that for those who are already highly proficient in the language (the majority, as the mean proficiency levels are high) additional gains in terminology do not increase effectiveness and vice versa. Thus, additional gains in activity effectiveness from targeting learning to more language-proficient participants or choosing a national language as the language of instruction for a WBI activity are, on average, not likely to be high. 5.17 However, there are, of course, certain specific WBI learning activities and/or countries where significant gains in effectiveness may be possible, through attention to these issues. For instance, in focus group discussions and one-on-one interviews, participants from lower-income countries, Egypt and Thailand reported that the WBI learning activity they attended involved participants who were not proficient in either the language or the technical terminology used in the WBI activity. 24 Although regression results indicate no significant differences in participants' ratings of activity effectiveness across WBI thematic groups, there are significant differences in overall effectiveness across these groups that can be explained by disparities in their use of action plans. 5.18 The regression results find no statistically significant differences in effectiveness across the five WBI thematic groups, controlling for activity features. While there are, in fact, significant differences in the mean overall effectiveness and use ratings across the WBI programs, these differences can largely be explained by the varying use of action plans among the thematic groups.28 This finding has real implications for program preparation and development. The programs with the highest average ratings of overall effectiveness, WBIPR and WBIHD, also had the highest proportion of participants reporting preparation of action plans (43 percent and 54 percent respectively, see Figure 11).29 In addition, participants from lower-income countries are also the main attendees of the learning activities organized by these thematic groups. Figure 11: Overall effectiveness and use by thematic group Overall effectiveness Overall effectiveness and use by WBI thematic groups Overall use (share of sample ­ scale 1-7) Action planning 6 1.00 5.24 5.32 0.80 4.98 4.99 5 4.88 4.56 0.60 4.41 0.54 4.07 0.43 0.40 4 0.30 0.33 0.20 3 0.00 WBIEN (22%) WBIFP (39%) WBIPR (16%) WBIHD (21%) 28We tested for the differences among the thematic areas by first regressing effectiveness on the WBI thematic areas (or event sponsors). WBIHD and WBIPR were positively and significantly different from the base WBIEN. We then introduced other activity and participant characteristics into the equation. This erased the effect of thematic programs on effectiveness, indicating that the use of action plans is the main activity-level determinant of activity effectiveness. 29Action plans are a flexible pedagogic tool that helps participants operationalize and implement their new knowledge and skills in the context of their jobs and the needs of their countries by "learning by doing". Action plans and their effect on learning and learning outcomes are described in greater depth in "Pedagogy for Effectiveness and Impact: Evidence Regarding Action learning and Course Duration." WBI Evaluation Briefs, June 2003. 25 5.19 We note here too that from the regression results, there also appear to be no significant differences in activity effectiveness across modes of delivery (classroom/face- to-face, blended/multi-media, VC/global dialog, and web-based/computer/electronic). But this might also be the case, because any underlying differences across these modes of delivery might be explained by their varying use of action plans. Women and participants occupying higher positions in their work places are more likely to find WBI activities more effective, compared to men and to participants in lower-level positions. 5.20 Regression results suggest that women and high-level participants (e.g. minister, top government official, full professor or CEO) report more benefits than men and lower- level counterparts.30 This is particularly true with the respect to activity's effectiveness in providing strategies and approaches. DETERMINANTS OF PARTICIPANTS' USE OF THE INFORMATION, KNOWLEDGE AND SKILLS (K&S) ACQUIRED FROM THE WBI LEARNING ACTIVITY 5.21 Table 7 shows the results of the second stage regressions of the independent variables on participants' use of the K&S they obtained from the WBI learning activity. For all regression results shown in this table, we use the regression results for overall effectiveness in Table 6 as the first stage. The variables gender, language and terminology proficiency (and their interaction term), action plans, delivery mode and WBI thematic group serve as instrumental variables and are excluded from the second-stage use regressions.31 30The regression results in Tables A12 and A13 in Annex 9 display the results when position is modeled as three dummy variables with the lowest position (Entry/junior level) as baseline. These regressions reveal that participants in top-level positions rate activity effectiveness in providing strategies and approaches higher and are able to make significantly more use of the knowledge and skills acquired than participants in all other position levels. 31The instruments used to predict effectiveness in the two-stage model are relatively adequate as correlation between the second stage residuals and the instruments is minimal. We can hence refute a reverse causal relationship between effectiveness and use. 26 Table 7: Instrumental variable regressions of participants' use of the K&S acquired from WBI learning activities, FY01-03 (un-standardized regression coefficients with t-statistics in parenthesis) Overall (Aggregated) Use Academic Use Operational Use Effectiveness Predicted effectiveness 0.785** 0.791** 0.767** (5.34) (4.46) (4.61) Participant characteristics Position 0.107** 0.070 0.156** (2.26) (1.30) (2.70) WBI policy variables/Activity features Follow-up 0.036 0.012 0.059 (0.45) (0.12) (0.60) Facilitators and barriers Work-place factors 0.168** 0.123** 0.225** (3.08) (2.10) (3.52) Country-level factors 0.084 0.078 0.090 (1.62) (1.30) (1.52) Income GNI per capita (1000s USD) -0.193** -0.160** -0.249** (-3.39) (-2.46) (-3.73) Constant -0.876* -0.443 -1.323 (-1.65) (-0.71) (-2.15) N 953 905 912 R-squarea 0.454 0.352 0.434 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aR-square from OLS regression of use on effectiveness and all other independent variables. 5.22 Table 7 shows the regression results for overall use (column 2), academic use (column 3) and operational use (column 4). As already explained elsewhere, overall use is the simple average of its two components: academic use and operational use. We note the following key results: Activity effectiveness increases participants' use of the K&S they acquire from the WBI learning activity. 5.23 The main and most significant factor explaining all our measures of use is overall effectiveness. These findings imply that the same factors that significantly determine overall effectiveness noted earlier also affect, albeit indirectly, participants' use of the K&S they gained from the WBI activity. 5.24 Higher overall effectiveness ratings significantly increase participants' ratings of academic use, operational use and overall use. For instance, for each of these three measures of participants' use, a 10 percent improvement in participants' overall effectiveness ratings increases the frequency with which they use the K&S they acquired 27 from the WBI activity by about nine percent.32 On average, this corresponds to an improvement of 16 percent of the total possible increase in overall use ( = 0.785, p<0.05). The separate effects on academic use and operational use are of similar magnitudes. Participants' use is higher in poorer countries. 5.25 Participants' use of the K&S acquired from WBI activities is higher in poorer countries (Table 7). More than just indirectly through overall effectiveness, GNI per capita has a direct, negative and significant effect on overall use and its two components. This is not a very surprising result. It suggests, however, that participants from the poorest countries appear to benefit more from WBI learning activities than do their counterparts from more affluent countries. It suggests too that perhaps WBI's current activities are better suited to lower-income countries. Further improvements in WBI's effectiveness might come from designing new learning activities that are more relevant and better tailored to the needs of lower-middle-income countries. Participants in top-level positions use their WBI-acquired K&S more frequently. 5.26 On the whole, participants in high-level positions make more frequent use of the K&S they acquired from WBI activities. Operational use especially increased among these higher-placed participants ( = 0.107, p<0.05). This is not surprising, given that these participants, because of their positions, are likely to exercise more influence over the actual implementation of any new organizational or national strategies/approaches. Workplace and country-level factors facilitate (rather than hinder) participants' use of the K&S they obtained from the learning activity. 5.27 In the survey, we asked participants to rate whether their work-place factors (such as incentive systems, peers and resources and funding) and country-level factors (such as political and social groups and the country's readiness to reform) were helpful or unhelpful in facilitating/impeding their use of what they learned from WBI activities. The average participant ratings for work-place and country-level factors were 4.53 and 4.61, respectively, where 1 = "not helpful" at all to 7 = "very helpful" (see Annex 9). We denote factors that on average were rated over 4.5 as facilitators to use, those rated below 3.5 as barriers to use, and those factors rated between 3.5 and 4.5 as neither helpful nor unhelpful in facilitating/impeding participants' use of the acquired knowledge. In seven of the twelve countries in our survey, respondents rated both the work and country level factors higher than 4.5. The exceptions were Burkina Faso, Egypt, Guatemala, Thailand and Russia, where work and country level factors were rated as neither helpful nor unhelpful. There were no countries where average ratings fell below 3.5, causing factors to be identified as barriers. 32Different from the raw change in use over time presented in Chapter 4 this is the adjusted change in use predicted by the effectiveness ratings alone. This was calculated by deriving the elasticity of use with respect to overall effectiveness [(y/x)(x / y)]. 28 5.28 Table 7 results suggest that work-related factors serve as significant facilitators to participants' application of the knowledge and skills obtained from WBI activities ( = 0.168, p<0.05). Country-related factors have no significant effects on the participants' frequency of use of their WBI-acquired K&S. 5.29 Some of the most significant barriers to impact were noted in the FGDs. Participants mentioned that, particularly in lower-income countries, the oftentimes very rigid and hierarchical structures of relevant governments (national and regional) and of participants' organizations hinder the impact of WBI activities. The impacts of any WBI activity depends heavily on the will of decision makers who are in positions that are often above those of the activity participants. Impact may be improved by training entire teams (or of different levels of participants within the same organizations), thereby institutionalizing learning and impact. In addition, some lower-income FGD participants mentioned that addressing practical implementation issues could be useful too, particularly in the context of their environments where change is usually slow. 29 6. CONCLUSIONS AND RECOMMENDATIONS 6.1 The main questions we investigated in this report are: (a) What are the main determinants of WBI activity effectiveness and impact? (b) Has WBI effectiveness changed over time (i.e. between FY01 and FY03)? 6.2 This report provides a baseline for evaluating the effectiveness and impact of WBI activities, based on data from activities implemented between FY01 and FY03. This span of time covers the period just preceding WBI's shift to a more country­focused approach in the provision of capacity development services (FY01 and FY02) as well as the period covering just the very beginning of this new approach (FY03). 6.3 We also review issues related to data recording and reporting in WBI, as it has implications for WBI's capacity development strategy going forward, as well as for reporting and monitoring outputs and results. We focus on the key findings and their implications. DETERMINANTS OF ACTIVITY EFFECTIVENESS AND IMPACT 6.4 In the twelve countries included in this study, average activity effectiveness varied from 4.6 in Russia, barely above the midpoint on a scale of 1 to 7, to 5.6 in Kenya and Brazil. Average use (our indicator of impact) ranged from 3.4 in Russia, below the scale midpoint on a scale of 1 to 7, to 5.3 in Tajikistan. We used a two-stage regression model to predict participants' use of information, knowledge, and skills, based on a theoretical framework we developed in earlier studies. This first stage predicts effectiveness and the second stage predicts use, with effectiveness as an instrument. Determinants of Effectiveness 6.5 Our main findings regarding the determinants of WBI activity effectiveness reveal that in our model only a handful of key policy-related variables affect activity effectiveness. The main findings are all drawn from the robust regression analysis results presented in Chapter 5. 6.6 Development of action plans during activity implementation increases activity effectiveness dramatically. Participants who reported developing action plans as part of the learning activity they attended rated the activity effectiveness to be much higher than those who did not. The development of action plans during the activity can boost activity effectiveness by 13 percent. This result has been replicated across a number of IEG studies of WBI activity effectiveness and outcomes. The share of participants preparing 30 actions plans increased from only 17 percent in the first half of FY01 to more than 40 percent in FY02 and FY03. 6.7 Participants with good proficiency in the language of instruction (or discussion, as the case may be) and in the technical subject matter of the activity provided higher effectiveness ratings. Relatively few study participants lacked language skills overall however, and were already highly proficient in both the language of instruction and in the technical terminology of the activity (means of 5.99 and 5.69, respectively, on a 7-point rating scale). 6.9 The implication of this finding is that WBI and its partners must continue to select participants carefully, paying attention to both their language skills and to their technical capabilities. It also may require providing activities in national languages to meet the needs of participants who may not be skilled in English, as was noted in the focus group discussions. 6.9 Activity effectiveness is higher in lower-income countries. Participants from lower-income countries rate activity effectiveness higher than do their counterparts from lower-middle-income countries controlling for participant characteristics, WBI activity features and exogenous factors (including time of activity delivery). The current dataset does not allow us to determine precisely why effectiveness is lower in lower-middle- income countries. One clue to this disparity is that participants from lower-middle- income countries also rate WBI activities as less relevant as compared to those from lower-income countries. This result suggests that WBI should examine both the features and content of the activities to improve their relevance to the lower-middle-income countries. Determinants of Use 6.10 Activity effectiveness is the strongest predictor of participants' reported frequency of use of information, knowledge and skills they acquired from the activity. Calculation of elasticities shows that a 10 percent improvement in participants' predicted overall effectiveness ratings increases the frequency with which they use the knowledge and skills by nine percent. 6.11 The implication of this finding is that WBI should continue to select participants carefully. Task managers should focus on identifying higher-level participants, particularly where the focus of the activity is less on awareness raising and more on helping the participants with the "nuts and bolts" of developing strategies and approaches to solving development issues. 6.12 Participants holding higher-level positions in their work make more use of WBI-acquired knowledge and skills compared to participants holding lower-level positions. The finding is especially strong for WBI activity effectiveness in providing strategies and approaches for development. This is not an unexpected result; high-level participants are more strategically placed to contextualize and grasp the information, 31 knowledge and skills for use in their work place. We note that in our sample, about 57 percent of the study participants comprised individuals occupying senior level positions. 6.13 Use of knowledge and skills is higher in lower-income countries. Income levels have a direct influence on the degree to which participants use knowledge and skills gained from WBI activities, over and above their influence through the effectiveness measure. 6.14 This last result again implies that perhaps the content of WBI activities may not be as relevant to the lower-middle-income countries and the content and approach need to be reviewed, or that perhaps there is more of a market place of ideas, with WBI providing only one input. WBI EFFECTIVENESS OVER TIME 6.15 A clear finding from our analyses is that more recent WBI activities are more effective and the knowledge and skills gained from more recent WBI activities therefore get more frequently used. Activities delivered during the last six-month period covered in this evaluation (January-June, 2003) were significantly more effective than those delivered in the first the first six-month period (July-December, 2000). Furthermore, participants attending the most recent activities also made significantly more use of what they learned than did participants in the earliest activities. 6.16 The increase in WBI effectiveness is largest for providing participants strategies and approaches for development. The increase in this area of predicted effectiveness was 17 percent from the first six months to the last six months of the period covered in this study, from an average of 4.5 to 5.4 on a scale of 1 to 7. 6.17 The increase in WBI effectiveness in building knowledge and skills is also significant. The increase in this aspect of predicted effectiveness was 12 percent, from an average of 4.8 in the first six months to 5.7 in the last six months. Effectiveness with regard to raising awareness did not increase significantly. 6.18 These results suggest that there has been a fundamental shift in WBI activities' approach and content and that WBI began focusing much more on helping clients gain hands-on, problem-solving kinds of knowledge and skills, than it had previously done. The averages for building knowledge and skills and for developing strategies and approaches measured on par with awareness raising in the last six months of FY03. WBI DATA RECORDING AND REPORTING 6.19 One clear finding is that the WBI participant data recording and reporting will need to be improved. Of the total number of participants reported, only 48 percent were participants with recorded names. The new approach to capacity development activities may require regular follow-ups with WBI activity alumni, and therefore, it will be crucial for the Institute to ensure the accuracy of its CRS data. 32 RECOMMENDATIONS 6.20 Our three main recommendations are to: (a) Continue with the action planning approach, especially if the activity objective is to help participants apply WBI-acquired knowledge and skills to development. For task managers unfamiliar with this approach, it may be useful to organize a workshop featuring experienced task managers who can share their techniques and lessons learned. (b) Continue to include participants holding high-level positions for greater impact. (c) Review the activity content and approach to activity implementation in middle-income countries to assess whether changes are warranted. 33 REFERENCES Bardini, Mark, D., Victoria Gunnarsson, Elina M. Manjieva and Yekaterina Narozhnaya. "The Impact of WBI Activities, FY01-02, on Participants from Russia: A Baseline Assessment." WBI Evaluation Studies, EG04-81, December, 2003. Chard, Cristina Ling and Diana J. Arango. "Evaluating the Impacts of FY02-03 WBI Activities on Nigerian Participants." WBI Evaluation Studies, EG04-87, June, 2004. Eckert, William, Graça Sousa and Victoria Gunnarsson. "The Impact of WBI Activities, FY01-02, on Participants from Brazil: A Baseline Assessment." WBI Evaluation Studies, EG04-82, January, 2004. Khattri, Nidhi, et al, "Impact Evaluations of WBI Client Programs, FY00-01." WBI Evaluation Studies, EG03-63, November, 2002. ___________, Shreyasi Jha and Denis Nikitin. "The Impact of WBI Activities, FY02-03, on Participants from Kenya: A Baseline Assessment." WBI Evaluation Studies, EG04- 89, June 2004. ___________, Peter Bachrach and Tao Jiang. "The Impact of WBI Activities, FY01-02, on Participants from Sri Lanka: A Baseline Assessment." WBI Evaluation Studies, EG04-73, October, 2003. Le Rouzic, Violaine. "The Impact of WBI Activities, FY02-03, on Participants from Burkina Faso: A Baseline Assessment." WBI Evaluation Studies, EG05-102, September, 2004. Liu, Chaoying, Jonathan Richter and Laurie Choi. "Country Focus Evaluation of WBI Activities in FY02-03, Guatemala: A Baseline Assessment." WBI Evaluation Studies, EG05-93, July, 2004. Quizon, Jaime, B. "Measuring, Monitoring and Managing for Development Results: An Approach to Monitoring and Evaluation in World Bank Institute (WBI) Country Focused Activities." WBIEG Working Paper (final draft), November, 2002. _____________. "Evaluation of FY02-03 WBI Activities with Indonesian Participants: Relevance, Effectiveness and Impact." WBI Evaluation Studies, EG05-96, July, 2004. _____________., Cristina Ling Chard and Marlaine E. Lockheed. "The Effectiveness and Use in FY03 of WBI FY01-02 Activities: A Baseline Assessment in Five Countries." WBI Evaluation Studies, EG04-86, May, 2004. 34 _____________ and Cristina Ling Chard. "The Impact of WBI Activities, FY01-02, on Participants from Thailand: A Baseline Assessment." WBI Evaluation Studies, EG04-77, October, 2003. Zia, Heidi, S. and Shreyasi Jha. "The Impact of WBI Activities, FY02-03, on Participants from Tajikistan: A Baseline Assessment." WBI Evaluation Studies, EG05-95, July, 2004. __________. and Shreyasi Jha. "The Impact of WBI Activities, FY02-03, on Participants from Yemen: A Baseline Assessment." WBI Evaluation Studies, EG04-92, July, 2004. __________., Mustapha Kamel Al-Sayyid, Sahar Tawila and Victoria Gunnarsson. "The Impact of WBI Activities, FY01-02, on Participants from Egypt: A Baseline Assessment." WBI Evaluation Studies, EG04-78, October, 2003. "Pedagogy for Effectiveness and Impact: Evidence Regarding Action Learning and Course Duration." WBI Evaluation Briefs, June, 2003. 35 36 ANNEXES 37 38 ANNEX 1: SURVEY QUESTIONNAIRE FOR PARTICIPANTS FROM BRAZIL, EGYPT, RUSSIA, SRI LANKA AND THAILAND (ACTIVITIES HELD IN FY01-02) W O R L D B A N K I N S T I T U T E Promoting Knowledge and learning for a better world Evaluation Participant Questionnaire Section I. Usefulness of the WBI Learning Activity 1. Thinking about the WBI learning activity you attended, would you say that the activity has been effective or ineffective in the areas noted below. Not Areas effective 2 3 4 5 6 Extremely Don't at all Effective Know a. Raising your awareness and understanding of the development issues important to your country b. Updating or refining the knowledge or skills you already had c. Providing you with new knowledge or skills d. Helping you get to know people interested in the subject matter of the learning activity e. Providing you with strategies or approaches to address the development needs of your country f. Providing you with strategies or approaches to address the needs of your organization 2. Thinking about the WBI learning activity you attended, what was the main theme or area it addressed? 3. Did the WBI learning activity you attended lead to any overall changes in that Yes No Don't specific area? (If you answer "No," or "Don't Know," skip to question #5.) know 4. How would you rate the nature of the overall change in that specific area? Strong Strong Do negative No positive Not change change change Know 1 2 3 4 5 6 7 8 39 5. Thinking about the WBI activity you attended, how often have you used the knowledge and skills you acquired in the following areas? Please mark "Not Applicable" if you do not work in the given area. Areas Not at all 2 3 4 5 6 Very Often N/A a. Conducting research b. Developing country development strategies c. Implementing country development strategies. d. Changing or influencing legislation or regulations e. Teaching f. Raising others' awareness in development issues g. Organizing collective community initiatives h. Implementing new practices within your work organization 6. How helpful or unhelpful are the following factors in actually using the new knowledge or skills that you acquired from the WBI learning activity? Not Neither Factors helpful 2 3 helpful 5 6 Extremely Don't at all nor Helpful Know unhelpful a. Practices and procedures in your work organization b. Resources and funding availability in your work organization c. Incentive system in your work organization d. Your supervisor e. Your colleagues 7. How helpful or unhelpful are the following factors in implementing the ideas covered in the learning activity? Not Neither Items helpful 2 3 helpful 5 6 Extremely Don't at all nor Helpful Know unhelpful a. Your country's policies b. Social groups in your country c. Political groups in your country d. Your country's general mood of "readiness" for reform and innovation 40 8. How has the WBI activity that you attended, influenced or led to changes in the following areas? (Please mark N/A if the particular area is not relevant to the activity.) Areas Negative Positive Influence 2 3 4 5 6 Influence N/A a. Research methodology or publications b. Legislation or regulations c. Teaching materials for courses d. Consensus building for change e. Community-based initiatives f. Work practices in your organization g. Other (Please specify) 9. Have the issues raised in WBI learning activities been discussed at work, with local partners, government officials or NGOs? Never discussed Discussed Don't at all thoroughly Know 1 2 3 4 5 6 7 10. Thinking back to the time immediately following the learning activity you attended, Yes No were you aware of any follow-up activities (e.g., meetings, workshops, or E-mail discussion groups) arranged by participants? (If you answer "No," skip to question #15) 11. Did you participate in any of these follow-up activities? Yes No 12. In how many of these follow-up activities did you participate? 13. Are these activities still taking place? Yes No 14. If not, why? II. Relevance of WBI Learning Activities to the Needs of Your Country Recall, you are being asked to think about the activity you attended between the summer of 2000 and the summer of 2002. 15. Was the activity you attended designed specifically for participants from your country? Yes No Don't know 16. Thinking about the WBI learning activity, to what degree were the topics covered in the activity relevant to your country's specific needs? Extremely Extremely Don't Irrelevant Relevant Know 1 2 3 4 5 6 7 8 41 (First review the development goals under column "A." Next, under Column "B," indicate with an "X" the relevant goal(s), then rate the extent to which the activity addressed key issues in reaching that goal(s) under Column "C.") B Mark an "x" here if WBI C A Activity was If you marked "X," please rate the extent to which related to each the activity addressed key issues in achieving that goal goal. Goals for Development Mark with an "X' Low 2 3 4 5 6 High N/A a. Eradicate Extreme Poverty b. Achieve universal primary education c. Promote gender equality and empower women d. Reduce child mortality e. Improve maternal health f. Combat HIV/AIDS, malaria, and other diseases g. Ensure environmental sustainability h. Develop global partnerships for development i. Ensure water sanitation and supply j. Improve investment climate and finance k. Promote trade III. Comparison of WBI Learning Activities with Similar Activities Offered by Other Organizations 18. Thinking about the WBI learning activity you attended, are you aware of any learning Yes No activities with similar content offered by organizations other than WBI, in your country? (If you answer "No," skip to question #21.): 19. If yes, please provide the name(s) of the organization(s): 1. 2. 3. 42 20. In comparing the activities offered by WBI and non-WBI, would you say that, overall, the non-WBI learning activities were more effective or less effective than WBI training? Much About Much Do more the less Not effective Same effective Know 1 2 3 4 5 6 7 8 IV. Characteristics of the Learning Event and Your Background Now, we would like to ask you to think about some of the things that took place during the activity you attended and to recall some of the logistics. 21. How many of your colleagues, or others with whom you work closely (either within or outside of your organization), attended the same activity? None 1 to 2 3 to 4 5 to 10 Over 10 22. During the learning activity, did you develop an action plan/strategy (e.g., work Yes No plans, strategy papers, policy documents, assessment of country needs, assessment of sectoral needs) to apply the new ideas you learned? (If you answer "No," please skip to question #25) 23. If yes, did you work as a team with your colleagues to produce the action plan? Yes No 24. Did you actually use parts or all of the action plan in your work? Yes No 25. To your knowledge was the learning activity a part of a series of related Yes No activities? 26. If yes, did you participate in other parts of the series? Yes No 27. Were you provided with the contact information of other participants in the Yes No activity you attended such as email addresses, telephone numbers or mailing addresses? 28. If yes, please indicate whether the contact information was: Not useful Extremely Do Not at all useful Know 1 2 3 4 5 6 7 8 43 29. After the WBI learning activity, were there any follow-up contacts made by the Yes No organizers? 30. If yes, please indicate the nature of the follow-up: Face-to-Face Web-based E-mail Other (please specify) meetings/activities discussions discussion __________________ 31. After the learning activity, did you receive any newsletters related to the learning Yes No activity? 32. If yes, please indicate whether it was: Not Do useful at Extremely Not all useful Know 1 2 3 4 5 6 7 8 33. In retrospect, what aspect(s) of the learning activity did you find most useful for your work, organization, or country? 34. In retrospect, what aspect(s) of the learning activity did you find least useful for your work, organization, or country? 35. What was the primary language used during the learning activity? (For each item below, select one answer on the Not scale, where 1=Not proficient at all and 7=Highly proficient 2 3 4 5 6 Highly proficiency.) at all Proficient 36. At the time of the learning activity, what was your level of proficiency in the language of instruction of the learning activity? 37. At the time of the learning activity, what was your level of proficiency in the technical terminology utilized in the learning activity 38. How long was the learning activity? Please indicate number of days 39. What proportion of the learning activity did you attend? A quarter Less More All or or less than half Half than almost half all 44 40. At the time of the learning activity, which of the following best characterizes the organization in which you worked? (Select one.) University /Research Institution Donor Agency Non-Governmental Organization (NGO) (not-for- profit) National/Central Government Private Sector (for profit) Provincial/Regional Government Other ____________________________________ Local Government 41. At the time of the learning activity, which of the following best describes the primary type of work you did? (Select one.) Research Management/Administration Teaching Policymaking / Legislative Research / Teaching Provide Services (e.g. financial, health, etc.) Other ____________________________________ 42. At the time of the learning activity, how would you best characterize your position in your organization? Top management (e.g. Minister, Deputy Minister, Top government official) Senior management/administration (e.g. Department Head, Division Head) Middle management/administration (e.g. Program Manager, Project Leader) Senior professional/technical/research staff Junior professional/technical/research staff Other (Please specify) ____________________________________ 43. Please indicate your gender Male Female 44. In which year were you born? (Please fill in the year in the boxes provided.) 1 9 Thank you for your feedback. We greatly appreciate your cooperation. 45 ANNEX 2: SURVEY QUESTIONNAIRE FOR PARTICIPANTS FROM GUATEMALA, INDONESIA, KENYA, NIGERIA, TAJIKISTAN AND YEMEN (ACTIVITIES HELD IN FY02-03) World Bank Institute (WBI) Country Focus Evaluation Questionnaire Questionnaire to be completed by former participants I. Relevance of the Activity The activity that you are asked to evaluate is mentioned on the first page of this questionnaire. 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 3. Was the activity designed specifically for participants from your country? Yes No Don't know 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Yes No Don't know b. Achieve universal primary education Yes No Don't know c. Promote gender equality and empower women Yes No Don't know d. Reduce child mortality Yes No Don't know e. Improve maternal health Yes No Don't know f. Combat HIV/AIDS, malaria, and other diseases Yes No Don't know g. Ensure environmental sustainability Yes No Don't know h. Develop global partnerships for development Yes No Don't know i. Ensure water sanitation and supply Yes No Don't know j. Improve investment climate and finance Yes No Don't know k. Promote trade Yes No Don't know 46 II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not Areas effective Extremely Not at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country b. Providing you with knowledge or skills c. Helping you better understand your role as an agent of change in your country's development d. Helping you develop strategies or approaches to address the needs of your organization e. Helping you develop strategies or approaches to address the needs of your country f. Helping you develop contacts, develop partnerships and build coalitions in the field 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong Strong negative positive change No change change Don't know 1 2 3 4 5 6 7 DK 7. How often have you used the knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Purposes Not Very Not at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research b. Teaching c. Raising public awareness in development issues d. Implementing new practices within your work organization e. Organizing collective initiatives f. Influencing legislation and regulation g. Implementing country development strategies 47 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, etc.) b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research b. Teaching c. Public awareness in development issues d. New practices within your work organization e. Collective initiatives f. Legislation and regulation g. Country development strategies 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) Yes No 12. If yes, please provide the name(s) of the organization(s): 1. ................................................................................................................................................................................................................... 2. ................................................................................................................................................................................................................... 3. ................................................................................................................................................................................................................... 48 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI WBI much less much more no useful About the same useful opinion 1 2 3 4 5 6 7 IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Video Sessions (Distance Class room Mix of Video and Web-based Learning) (Face to Face) Face to Face Conference Learning Study tour 1 2 3 4 5 6 15. How effective was this type of learning activity in helping you learn? Not effective at Extremely No all effective opinion 1 2 3 4 5 6 7 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) Yes No 17. If yes, did you use part or all of the action plan in your work? Yes No 18. Were you provided with the contact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) Yes No 19. If yes, how did you use it? Used it to Used it to Other uses Never used it continue activity organize joint follow- (Please specify briefly) related discussions up activities _____________ 20. Was the language of instruction used during the activity the same language you use at work? Yes No 49 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient Highly at all proficient 1 2 3 4 5 6 7 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient Highly at all proficient 1 2 3 4 5 6 7 23. After the activity, did WBI contact you for follow-up issues regarding the activity? Yes No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) Yes No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI WBI I did not responded, responded have WBI did but was not and was follow-up not helpful extremely requests respond at all helpful for WBI 0 1 2 3 4 5 6 7 NA 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) University/research institution National/central government Non-governmental organization (not-for-profit) Provincial/regional government Media Local/municipal government Private sector Other, specify: _________________ 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) Research Teaching Policymaking/legislation Provision of services (e.g., financial, health, etc) Management/administration Other, specify: __________________ 50 28. How would you best describe the level of the position you have held the longest since the activity? Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) Junior level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) Entry level (e.g., Intern, assistant) Other, Please specify:______________________________________________ 29. What is your gender? Male Female Thank you for your feedback. We appreciate very much your cooperation. 51 ANNEX 3: SURVEY QUESTIONNAIRE FOR PARTICIPANTS FROM BURKINA FASO (ACTIVITIES HELD IN FY02-03) I N S T I T U T D E L A B A N Q U E M O N D I A L E Promouvoir l'éducation et le savoir pour un monde meilleur PARTICIPANT SURVEY French version of the participant survey completed by alumni Evaluation de l'impact de l'Institut de la Banque Mondiale (IBM) par pays Questionnaire à remplir par les anciens participants aux formations Burkina Faso Instructions D'après nos fichiers, l'Institut de la Banque Mondiale a eu le plaisir de votre participation à la formation intitulée : Titre de l'activité tenue du Date de début au Date de fin au/en Pays où l'activité s'est déroulée Avez-vous effectivement participé à cette formation? (Veuillez entourer votre réponse) a Oui Veuillez poursuivre ce questionnaire. b Non Veuillez le signaler à Mme Lamien/Ouando (contact en bas de page) et ne pas remplir ce questionnaire. Maintenant que vous avez assez de recul, recueillir votre opinion sur la formation sus- mentionnée est très important à l'Institut de la Banque Mondiale pour améliorer ses programmes. C'est pourquoi nous vous prions de bien vouloir remplir ce questionnaire. Il comprend quatre sections et vous prendra environ une demi-heure à remplir. Nous comptons sur votre franc jugement. Ayez à l'esprit que vos réponses seront tenues confidentielles. Seule l'équipe d'évaluation de l'Institut de la Banque Mondiale ­ chargée de cette évaluation ­ aura accès aux réponses individuelles. Vos réponses seront utilisées aux seules fins d'améliorer les programmes de l'Institut de la Banque Mondiale. Tout au long du questionnaire, nous vous prions d'entourer une seule réponse par question. Veuillez indiquer votre réponse en entourant le chiffre ou la lettre correspondant comme sous indiqué : 1er exemple : 1 2 3 4 5 6 7 X 2ème exemple : a Oui b Non c Ne sais pas Si vous vous trompez en entourant une réponse (et si vous ne pouvez pas l'effacer), veuillez la corriger de la manière suivante: 1) encerclez la réponse de votre choix, et 2) rayez la mauvaise réponse. Cette image signale que l'échelle va du négatif au positif avec une position médiane neutre. Les autres échelles sur sept sont des barèmes classiques allant du plus petit au plus grand, sans valeurs négatives. Veuillez lire attentivement chaque question et son échelle, et la noter en conséquence. Si vous avez besoin de précisions sur ce questionnaire, veuillez contacter Mme Adiza Lamien/Ouando à l'adresse suivante : zen_nepia@yahoo.fr ou aux n° de téléphone suivants : 50 36 25 44 / 70 25 38 01. 52 I. Pertinence de la formation La formation que l'on vous demande d'évaluer est indiquée en page de couverture. 1. Depuis la fin de la formation, à quel point celle-ci a-t-elle été pertinente pour votre travail ? Pas du tout Très pertinente pertinente 1 2 3 4 5 6 7 2. A quel point les sujets traités pendant la formation ont-ils été en rapport avec les besoins du Burkina Faso ? Pas du tout Très en rapport en rapport 1 2 3 4 5 6 7 3. La formation était-elle spécialement conçue pour des participants du Burkina Faso ? a Oui b Non c Ne sais pas 4. La formation était-elle conçue pour vous aider à résoudre certains problèmes que le Burkina Faso rencontre ? a Oui b Non c Ne sais pas 5. La formation était-elle en rapport avec les buts de développement du Pays sous-cités ? a. Eradiquer l'extrême pauvreté a Oui b Non c Ne sais pas b. Assurer l'éducation de base pour tous a Oui b Non c Ne sais pas c. Promouvoir l'égalité des sexes et l'habilitation des femmes a Oui b Non c Ne sais pas d. Réduire la mortalité infantile a Oui b Non c Ne sais pas e. Améliorer la santé maternelle a Oui b Non c Ne sais pas f. Lutter contre le VIH/SIDA, le paludisme et autres maladies a Oui b Non c Ne sais pas g. Assurer la pérennité de l'environnement a Oui b Non c Ne sais pas h. Développer des partenariats globaux pour le développement a Oui b Non c Ne sais pas i. Assurer l'alimentation en eau potable a Oui b Non c Ne sais pas j. Améliorer l'environnement des investissements et des finances a Oui b Non c Ne sais pas k. Promouvoir le commerce a Oui b Non c Ne sais pas 53 II. Utilité de la formation 6. Veuillez apprécier le degré d'efficacité de la formation dans chaque domaine ci-dessous. (Si le domaine ne constituait pas un objectif de la formation, veuillez marquer « sans objet » .) Domaines Pas du tout Très Sans efficace efficace objet 1 2 3 4 5 6 7 X a. Augmenter votre prise de conscience et compréhension des grands enjeux de développement au Burkina Faso 1 2 3 4 5 6 7 X b. Vous apporter des connaissances et aptitudes 1 2 3 4 5 6 7 X c. Vous aider à mieux comprendre votre rôle en tant qu'agent de changement pour le développement du Burkina Faso 1 2 3 4 5 6 7 X d. Vous doter de connaissances et d'aptitudes requises pour assumer ce rôle 1 2 3 4 5 6 7 X e. Vous aider à développer des stratégies et des approches pour répondre aux besoins de vos proches collaborateurs 1 2 3 4 5 6 7 X f. Vous aider à développer des stratégies et des approches pour répondre aux besoins de votre organisation 1 2 3 4 5 6 7 X g. Vous aider à développer des stratégies et des approches pour répondre aux besoins du Burkina Faso 1 2 3 4 5 6 7 X h. Vous aider à développer des contacts, des partenariats et à mettre en place des coalitions dans le domaine 1 2 3 4 5 6 7 X i. Augmenter votre prise de conscience et compréhension des questions de genre liées au sujet de la formation 1 2 3 4 5 6 7 X 7. Comment appréciez-vous le changement -- induit par la formation -- au Burkina Faso sur le thème principal de la session ? Changement Changement fortement Pas de fortement Ne sais négatif changement positif pas 1 2 3 4 5 6 7 X 54 8. A quelle fréquence avez-vous utilisé les connaissances et aptitudes acquises pendant la formation pour atteindre les objectifs suivants ? (Si vous n'avez pas travaillé dans un domaine cité depuis la formation, veuillez marquer « sans objet » .) Objectifs Pas Très Sans du tout souvent objet 1 2 3 4 5 6 7 X a. Mener une recherche 1 2 3 4 5 6 7 X b. Enseigner 1 2 3 4 5 6 7 X c. Sensibiliser les populations aux questions de développement 1 2 3 4 5 6 7 X d. Mettre en oeuvre des nouvelles pratiques dans votre organisation 1 2 3 4 5 6 7 X e. Etablir des partenariats entre agents de développement 1 2 3 4 5 6 7 X f. Influencer la législation et la régulation 1 2 3 4 5 6 7 X g. Mettre en oeuvre des stratégies de développement pour le Burkina Faso 1 2 3 4 5 6 7 X h. Intégrer le genre dans votre travail 1 2 3 4 5 6 7 X 9. A quel point les facteurs suivants ont-t-ils aidé ou gêné le processus d'utilisation des connaissances et aptitudes acquises pendant la formation ? Facteurs A N'a A beaucoup ni aidé beaucoup Sans gêné ni gêné aidé objet 1 2 3 4 5 6 7 X a. L'environnement de travail (ex. procédures de travail, collègues, système de motivation, financement, etc.) 1 2 3 4 5 6 7 X b. L'environnement général de développement du Burkina Faso (ex. politiques du pays, groupes sociaux, groupes politiques, volonté de réforme, etc.) 1 2 3 4 5 6 7 X 10. De quelle manière la formation a-t-elle influencé ou induit des changements dans les domaines suivants ? (Si le domaine n'a pas de relation avec la formation, veuillez marquer « sans objet » .) Domaines Influence Pas Influence Sans négative d'influence positive objet 1 2 3 4 5 6 7 X a. Recherche 1 2 3 4 5 6 7 X b. Enseignement 1 2 3 4 5 6 7 X c. Sensibilisation du public aux questions de développement 1 2 3 4 5 6 7 X d. Nouvelles pratiques dans la façon d'organiser votre travail 1 2 3 4 5 6 7 X e. Etablissement de partenariats 1 2 3 4 5 6 7 X 55 a. Stratégies de développement du Burkina Faso 1 2 3 4 5 6 7 X 11. Depuis la formation avez-vous discuté des questions soulevées pendant la session, au travail, avec des partenaires locaux, des fonctionnaires de l'Etat, des Organisations Non- Gouvernementales ou dans les médias ? Jamais Largement discuté discuté 1 2 3 4 5 6 7 III. Comparaison de la formation de l'Institut de la Banque Mondiale (IBM) avec des formations similaires données par d'autres organisations 12. Avez-vous participé à des activités d'apprentissage similaires données par des organisations autres que l'Institut de la Banque Mondiale dans votre pays ? (Si non, marquez « non », puis allez directement à la question 15.) a Oui b Non 13. Si oui, bien vouloir donner le nom de ces organisations : 1. __________________________________________________________________________________ 2. __________________________________________________________________________________ 3. __________________________________________________________________________________ 14. Comment appréciez-vous l'utilité de la formation de l'Institut de la Banque Mondiale (IBM) comparée aux formations des autres organisations ? Celle de Celle de l'IBM moins A peu près l'IBM plus Pas utile pareille utile d'opinion 1 2 3 4 5 6 7 X IV. Caractéristiques de la formation de l'Institut de la Banque Mondiale, son suivi et votre profil La formation mentionnée dans les questions suivantes est celle indiquée en première page. 15. Comment décririez-vous le mode de la formation de l'IBM à laquelle vous avez participé ? a Sessions vidéo (apprentissage à distance) d Conférence b En salle (face à face) e Apprentissage par Internet c Mélange de vidéo et de face à face f Voyage d'études 16. A quel point ce mode de formation a-t-il été efficace pour vous aider à apprendre ? Pas du tout Très Pas efficace efficace d'opinion 1 2 3 4 5 6 7 X 56 17. Pendant la formation, avez-vous élaboré un plan d'action/une stratégie (ex: plan de travail, stratégie, documents de politique) pour appliquer les connaissances et aptitudes que vous avez acquises ? (Si non, veuillez marquer « non » en dessous, puis passer à la question 19.) a Oui b Non 18. Si oui, avez-vous utilisé ce plan d'action (ne serait-ce qu'en partie) dans votre travail ? a Oui b Non 19. L'Institut de la Banque Mondiale vous avait-il communiqué les coordonnées des autres participants à la formation, telles que leurs adresses e-mail, numéros de téléphone ou adresses postales ? (Si non, veuillez marquer « non » en dessous, puis passer à la question 21.) a Oui b Non 20. Si oui, comment les avez-vous utilisées ? a Jamais utilisées b Utilisées pour continuer les discussions relatives à la formation c Utilisées pour organiser des activités conjointes de suivi d Autre utilisation, spécifiez : _________________________________________________________ 21. La langue utilisée pour dispenser la formation était-elle la même langue que vous utilisiez au travail ? a Oui b Pour une partie de la formation seulement c Non 22. Lors de la formation, dans quelle mesure maîtrisiez-vous la langue d'enseignement ? Aucunement Parfaitement 1 2 3 4 5 6 7 23. Lors de la formation, dans quelle mesure maîtrisiez-vous les termes techniques utilisés ? Aucunement Parfaitement 1 2 3 4 5 6 7 24. Après la formation, l'Institut de la Banque Mondiale vous a-t-il contacté pour des questions de suivi en relation avec la formation ? (Veuillez ne pas considérer les suivis d'ordre logistique.) a Oui b Non 25. Après la formation, avez-VOUS contacté l'Institut de la Banque Mondiale pour des questions en dehors de celles ayant trait à la logistique ? (Si non, veuillez marquer « non » ci-dessous, et passer à la question 27.) a Oui b Non 26. Si oui, veuillez évaluer l'utilité de la réponse de l'Institut de la Banque Mondiale (IBM) à vos questions. (Veuillez ne pas prendre en compte les questions d'ordre logistique.) L'IBM a L'IBM a répondu, répondu, L'IBM mais n'a et a été n'a pas pas du tout très répondu été utile utile 0 1 2 3 4 5 6 7 57 27. Parmi les domaines suivants, lequel décrit le mieux l'organisation où vous avez le plus longtemps travaillé après la formation ? (Veuillez choisir une seule réponse.) a Université/Institut de recherche e Administration nationale/centrale b Média f Collectivité provinciale/régionale c Secteur privé (à but lucratif) g Collectivité locale/municipale d Organisation non-gouvernementale (à but non lucratif) h Autre, spécifiez : _______________________ 28. Parmi les tâches suivantes, laquelle décrit le mieux celle que vous avez le plus longtemps assumée après la formation ? (Veuillez choisir une seule réponse.) a Recherche e Management/administration b Enseignement f Information/sensibilisation c Elaboration de politique/législation g Prestation de services (ex : financiers, de santé...) d Plaidoyer h Autre, spécifiez : _______________________ 29. Comment décririez-vous la position que vous avez le plus longtemps occupée depuis la formation ? a Haut niveau (ex : Ministre, Secrétaire d'Etat, Doyen d'Université, Président d'Organisation) b Niveau supérieur (ex : Haut Fonctionnaire, Chef de Département, Professeur Titulaire, Directeur) c Niveau intermédiaire (ex : Chef de Projet, Professeur de Faculté, Expert) d Niveau exécutant (ex : Chercheur, Analyste, Maître Assistant, Technicien) e Niveau débutant (ex : Stagiaire, Etudiant, Assistant) f Autre, veuillez spécifier : ___________________________________________________________ 30. Depuis la formation, comment décririez-vous le pouvoir que vous avez le plus longtemps assumé dans le domaine de la session ? a Pouvoir décisionnaire sur les grands objectifs b Pouvoir d'influence sur les grands objectifs c Pouvoir décisionnaire sur les choix techniques de mise en oeuvre des grands objectifs d Pouvoir d'influence sur les choix techniques de mise en oeuvre des grands objectifs e Pouvoir de mise en exécution f Pouvoir d'information, sensibilisation, mobilisation g Autre, veuillez spécifier : ___________________________________________________________ 31. De quel sexe êtes-vous ? a Masculin b Féminin 32. Résidez-vous encore au Burkina Faso ? a Oui b Non 58 33. Accepteriez-vous de participer à une discussion en petit groupe pour examiner l'impact des actions de l'IBM sur le Burkina Faso et comment l'améliorer ? a Oui b Non 34. Quelle est la date d'aujourd'hui (jour/mois/année) ? _______ / _______ / 200__ 35. Veuillez indiquer en bas toute information/clarification que vous souhaitez porter à la connaissance du Groupe d'Evaluation de l'IBM dans le cadre de cette évaluation. _____________________________________________________________________ _____________________________________________________________________ _____________________________________________________________________ _____________________________________________________________________ _____________________________________________________________________ Merci beaucoup pour vos réponses. Nous vous sommes très reconnaissants pour votre coopération. 59 ANNEX 4: 2002 GROSS NATIONAL INCOME (GNI) PER CAPITA BY COUNTRY 2004 World Country 2002 GNI per capita Development Indicators classification Brazil 2,830 LMIC Burkina Faso 220 LIC Egypt 1,470 LMIC Guatemala 1,740 LMIC Indonesia 710 LIC Kenya 360 LIC Nigeria 290 LIC Russia 2,140 LMIC Sri Lanka 851 LMIC Tajikistan 180 LIC Thailand 1,980 LMIC Yemen 490 LIC All countries (average) 1,279 60 ANNEX 5: RESPONDENT PARTICIPANT CHARACTERISTICS BY COUNTRY Participant Characteristics Distribution (%) Brazil Burkina Faso Egypt Guatemala Indonesia Kenya Nigeria Russia Sri Lanka Tajikistan Thailand Yemen Gender Female 40 23 37 43 41 36 33 44 29 44 58 36 Male 60 77 63 57 59 64 67 56 71 56 42 64 Position Highest 11 3 12 17 11 12 13 1 16 31 6 26 Senior 60 38 56 44 30 35 47 64 13 38 26 37 Middle 19 32 15 29 36 41 34 33 54 25 44 26 Junior 10 27 17 10 23 12 6 2 17 6 24 11 Primary type of work Management/administration 49 34 30 32 36 50 22 12 35 14 29 53 Policy making/legislative 08 23 7 5 15 14 15 6 20 16 24 18 Research or teaching 22 14 40 57 38 28 49 35 24 37 17 11 Provision of services 7 6 15 4 2 8 10 46 11 26 8 18 Other 14 23 8 2 9 0 4 1 10 7 28 0 Work organization Local/Provincial/National Gov. 79 66 37 29 66 37 36 19 52 9 58 64 Non-Governmental Org. (NGO) 1 12 6 18 11 23 12 4 17 46 3 11 Private Sector 6 7 12 9 6 16 11 48 3 2 18 4 University 7 7 26 38 12 17 30 28 15 9 14 11 Other (incl. donor org. and media) 7 8 20 7 5 7 11 1 14 34 7 10 61 ANNEX 6: CORRELATIONS BETWEEN OUTCOME MEASURES, INCOME AND TIME OF DELIVERY Table A1: Correlations between outcome measures and income Correlation with 2002 GNI per capita N Correlation* Spearman p-value Relevance to country needs 1376 -0.162 0.000 Overall effectiveness 1373 -0.201 0.000 Effectiveness in awareness raising 1325 -0.136 0.000 Effectiveness in building knowledge and skills 1358 -0.179 0.000 Effectiveness in providing strategies and approaches 1355 -0.181 0.000 Overall use 1301 -0.252 0.000 Academic use 1223 -0.208 0.000 Operational use 1228 -0.273 0.000 * Spearman correlation coefficient for these data is almost identical to the more common Pearson correlation coefficient reported. Table A2: Correlations between outcome measures and quarter of the fiscal year when the activity was delivered Correlation with quarter of the fiscal year when the activity was delivered N Correlation* Spearman p-value Relevance to country needs 1363 0.174 0.000 Overall effectiveness 1360 0.222 0.000 Effectiveness in awareness raising 1314 0.135 0.000 Effectiveness in building knowledge and skills 1345 0.203 0.000 Effectiveness in providing strategies and approaches 1342 0.213 0.000 Overall use 1288 0.204 0.000 Academic use 1210 0.172 0.000 Operational use 1216 0.226 0.000 * Spearman correlation coefficient for these data is almost identical to the more common Pearson correlation coefficient reported. 62 ANNEX 7: DISTRIBUTION OF EFFECTIVENESS AND USE RATINGS Figure A1: Distribution of respondents' ratings of activity effectiveness Overall effectiveness 1=not effective at all, 7=extremely effective Effectiveness in raising awareness Effectiveness in building knowledge and skills Effectiveness in providing strategies and approaches 40% stn 30% de ponser 20% of er 10% haS 0% <2 >=2 but <3 >=3 but <4 >=4 but <5 >=5 but <6 >=6 but <7 7 Effectiveness rating Figure A2: Distribution of respondents' ratings of use of acquired knowledge and skills Overall use 1=not used at all, 7=used very often Academic use Operational use 40% s nt 30% de ponser 20% of er haS 10% 0% <2 >=2 but <3 >=3 but <4 >=4 but <5 >=5 but <6 >=6 but <7 7 Use rating 63 ANNEX 8: RATINGS OF EFFECTIVENESS AND USE COMPONENTS BY COUNTRY GNI/CAPITA Figure A3: Effectiveness in raising awareness ratings, by country GNI/capita* 7 6 5.77 5.68 5.52 5.57 5.61 5.43 ngs 5.29 5.34 5.36 atir 5.06 5.12 5.15 5 4.85 Mean 4 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) *The 12-country average is unweighted Figure A4: Effectiveness in building knowledge and skills ratings, by country GNI/capita* 7 6 5.77 5.73 5.83 5.64 5.70 s ngi 5.36 5.46 5.25 atr 5.09 5.18 5 4.90 4.74 4.75 Mean 4 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) *The 12-country average is unweighted 64 Figure A5: Effectiveness in providing strategies and approaches ratings, by country GNI/capita* 7 6 5.56 5.42 ngsi 5.24 5.36 5.21 5.32 atr 4.95 4.91 4.93 5 4.90 4.53 4.56 4.49 Mean 4 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) *The 12-country average is unweighted Figure A6: Academic use ratings, by country GNI/capita* 7 6 5.32 5.33 ngs atir 5.03 4.93 5.02 5.03 5 4.77 4.73 4.76 4.49 4.60 Mean 4 3.89 3.74 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) *The 12-country average is unweighted 65 Figure A7: Operational use ratings, by country GNI/capita* 7 6 ngs 5.27 atir 4.99 5 4.85 4.72 4.76 4.55 4.65 Mean 4.20 4.25 4.27 4.14 4 3.76 3.07 3 Tajikistan Burkina Nigeria Kenya Yemen Indonesia Sri Lanka 12 country Egypt Guatemala Thailand Russia Brazil ($180) Faso ($290) ($360) ($490) ($710) ($851) average ($1470) ($1740) ($1980) ($2140) ($2830) ($220) ($1279) *The 12-country average is unweighted 66 ANNEX 9: VARIABLE DESCRIPTIONS AND SUMMARY STATISTICS REGRESSION ANALYSIS DATA Independent Variable Descriptions and Data Sources (Data Source in Parenthesis) Participant Characteristics Female Dummy taking a 1 if the participant is female (survey). Position (reference: Entry/junior level) Mid-level position Dummy taking a 1 if the job position within the participant's organization is mid-level such as program manager, project leader, assistant professor or technical expert (survey). Senior-level position dummy taking a 1 if the job position within the participant's organization is senior-level such as department head, division head, associate professor or senior researcher (survey). Top-level position dummy taking a 1 if the job position within the participant's organization is top-level such as minister, top government official, full professor or CEO (survey). Language proficiency level of proficiency in the language of instruction of the learning activity 1=not proficient at all, 7=highly proficient (survey). Terminology proficiency level of proficiency in the language of instruction of the learning activity 1=not proficient at all, 7=highly proficient (survey). Language*terminology interaction term between language and terminology proficiency 1=not proficient at all in proficiency either the language or technical terminology, 49=highly proficient in both the language and terminology (survey). WBI Policy Variables/Activity Features Follow-up dummy taking a 1 of the participant perceived that he/she was contacted by the activity organizer after attending the activity (survey). Action plan dummy variable taking a 1 of the participant reported that the activity used action planning (survey). Delivery mode (reference: Classroom/F2F) Blended/Multi-media Dummy taking a 1 if the activity used a blended/multi-media mode (CRS). VC/Global dialog Dummy taking a 1 if the activity was a videoconference of global dialog (CRS). Web-based/ dummy taking a 1 if the activity used a web-based or electronic computer format to Computer/Electronic access the activity (CRS). WBI thematic program (reference: WBIEN) WBIFP Dummy taking a 1 if the activity falls under WBIFP (CRS). WBIHD/KL Dummy taking a 1 if the activity falls under WBIHD or WBIKL (CRS). WBIPR Dummy taking a 1 if the activity falls under WBIPR (CRS). Exogenous Factors GNI per capita Gross National Income of the country of the participant (World Bank Country at A Glance). (Annex 9 continues on next page.) 67 (Annex 9 continued.) Independent Variable Descriptions and Data Sources (Data Source in Parenthesis) Participant characteristics Time of delivery (reference: July-Dec.FY01) Delivered in FY01 Jan.-June Dummy taking a 1 if the activity was delivered between January-June, 2001 (surveyor). Delivered in FY02 July-Dec Dummy taking a 1 if the activity was delivered between July-December, 2001 (surveyor). Delivered in FY02 Jan.-June Dummy taking a 1 if the activity was delivered between January-June, 2002 (surveyor). Delivered in FY03 July-Dec Dummy taking a 1 if the activity was delivered between July-December, 2002 (surveyor). Delivered in FY03 Jan.-June Dummy taking a 1 if the activity was delivered between January-June, 2003 (surveyor). Facilitators and Barriers Organizational factors level of helpfulness of organizational factors such as practices, procedures and incentive system in the organization, available resources and funding, supervisors and peers in helping the participant apply the knowledge and skills acquired in the activity, 1=not helpful at all, 4=neither helpful nor unhelpful, 7=extremely helpful (survey). Country level factors level of helpfulness of country level factors such as country policies, social and political groups and the country's readiness for reform and innovation in helping the participant apply the knowledge and skills acquired in the activity, 1=not helpful at all, 4=neither helpful nor unhelpful, 7=extremely helpful (survey). Dependent Variable Description And Data Sources (Data Source In Parenthesis) Activity Effectiveness Participant's perception of the extent of effectiveness of the activity attended in three areas and overall. scale: 1=not effective at all, 7=extremely effective (survey) Overall effectiveness average of all areas of activity effectiveness that coincide for the first round and second round of surveys Raising awareness Effectiveness in raising awareness and understanding of country development issues Building new knowledge and Effectiveness in building new knowledge and skills skills Providing strategies and Effectiveness in providing strategies and approaches for addressing organizational approaches or country needs Use of Knowledge and Skills Participant's report of the frequency of use of knowledge and skills acquired in the activity in two areas and overall scale: 1=not at all, 7=very often (survey) Overall use Average of all areas of use of knowledge and skills that coincide for the first round and second round of surveys Academic use Use in research, teaching, raising awareness Operational use Use in implementing practices, developing strategies, influencing legislation, organizing initiatives 68 Summary Statistics for Variables Used in Regression Analysis N Mean Std. Dev. Min Max Effectiveness (1=not al all, 7=extremely effective) Overall effectiveness 1373 5.105 1.172 1 7 Awareness in raising awareness 1325 5.263 1.383 1 7 Effectiveness in building knowledge and skills 1358 5.249 1.380 1 7 Effectiveness in providing strategies and approaches 1355 4.931 1.341 1 7 Use (1=not at all, 7= use very often) Overall use 1301 4.419 1.501 1 7 Academic use 1223 4.595 1.567 1 7 Operational use 1228 4.249 1.650 1 7 Participant characteristics Female (1=female, 0=male) 1371 0.392 0.488 0 1 Position (1=entry/junior, 2=middle, 3=senior, 4=top level) 1197 2.526 0.860 1 4 Mid-level position 1197 0.292 0.455 0 1 Senior-level position 1197 0.463 0.499 0 1 Top-level position 1197 0.103 0.304 0 1 Language proficiency (1=poor, 7=high) 1347 5.992 1.307 1 7 Terminology proficiency (1=poor, 7=high) 1335 5.692 1.350 1 7 Language* terminology proficiency (1=poor, 49=high) 1325 35.134 12.899 1 49 WBI policy variables/Activity features (1=yes, 0=no) Follow-up 1339 0.307 0.461 0 1 Action plan 1367 0.388 0.487 0 1 Classroom/F2F 1377 0.638 0.481 0 1 Blended / Multi-media 1377 0.144 0.351 0 1 VC/Global dialog 1377 0.171 0.376 0 1 Web-based/Computer/Electronic 1377 0.028 0.166 0 1 WBIEN 1306 0.216 0.412 0 1 WBIFP 1306 0.383 0.486 0 1 WBIHD 1306 0.204 0.403 0 1 WBIKL 1306 0.021 0.142 0 1 WBIPR 1306 0.165 0.371 0 1 Facilitators and barriers (1=not helpful at all, 7= very helpful) Work-place factors 1311 4.528 1.398 1 7 Country-level factors 1279 4.605 1.372 1 7 Income and time trend GNI per capita (1000s USD) 1377 1.248 0.831 0.180 2.830 Delivered in FY01 Jan.-June 1364 0.126 0.332 0 1 Delivered in FY01 July-Dec. 1364 0.171 0.376 0 1 Delivered in FY02 Jan.-June 1364 0.158 0.365 0 1 Delivered in FY02 July-Dec. 1364 0.290 0.454 0 1 Delivered in FY03 Jan.-June 1364 0.080 0.271 0 1 Delivered in FY03 July-Dec. 1364 0.174 0.380 0 1 69 ANNEX 10: ALTERNATIVE REGRESSION SPECIFICATIONS Table A3: OLS regressions of the effectiveness of WBI learning activities, FY02 (un-standardized regression coefficients with t-statistics in parenthesis) Effectiveness Effectiveness Overall Effectiveness in Building in Providing (Aggregated) in Raising Knowledge Strategies and Effectiveness Awareness and Skills Approaches Participant characteristics Female 0.252** 0.216 0.411** 0.214 (2.23) (1.58) (2.93) (1.63) Position 0.087 0.006 0.012 0.165** (1.31) (0.07) (0.13) (2.28) Language proficiency 0.412** 0.397** 0.389** 0.417** (2.80) (2.32) (2.66) (2.27) Terminology proficiency 0.436** 0.454** 0.290* 0.481** (2.52) (2.26) (1.73) (2.30) Language*terminology proficiency -0.074** -0.078** -0.063** -0.076** (-2.58) (-2.32) (-2.20) (-2.14) WBI policy variables/Activity features Follow-up -0.075 -0.172 -0.086 -0.140 (-1.13) (-1.17) (-0.56) (-0.97) Action plan 0.726** 0.680** 0.640** 0.793** (6.16) (4.93) (4.54) (5.78) Blended/Multi-mediaa -0.103 -0.029 0.051 -0.206 (-0.67) (-0.16) (0.24) (-1.19) VC/Global Dialoga -0.529** -0.512** -0.447** -0.559** (-3.16) (-2.57) (-2.29) (-2.74) Web-based/Computer/Electronica -0.726 -0.340 -0.556 -0.995* (-1.60) (-0.76) (-1.22) (-1.95) WBIFPb -0.097 -0.244 0.127 -0.107 (-0.63) (-1.24) (0.63) (-0.57) WBIHD/WBIKLb -0.138 -0.150 0.104 -0.216 (-0.72) (-0.69) (0.46) (-1.00) WBIPRb -0.020 -0.361 0.194 0.054 (-0.11) (-1.62) (0.83) (0.24) Income and time trend GNI per capita (1000s USD) -0.117 -0.217** -0.184* -0.032 (-1.28) (-1.99) (-1.66) (-0.31) Delivered in FY02 Q2c 0.137 0.048 0.253 0.082 (0.78) (0.22) (1.06) (0.37) Delivered in FY02 Q3c 0.019 -0.046 0.283 -0.047 (0.09) (-0.19) (1.04) (-0.18) Delivered in FY02 Q4c 0.173 0.035 0.293 0.178 (1.03) (0.18) (1.43) (0.88) Constant 2.390** 3.134** 2.953** 1.780* (2.72) (3.07) (3.37) (1.66) N 498 488 494 495 R-square 0.182 0.134 0.133 0.174 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aClassroom and face-to-face learning is the reference. bWBIEN thematic group is the reference. cFirst quarter of FY02 is the reference. 70 Table A4: Instrumental variable regressions of participants' use of the K&S acquired from WBI learning activities, FY02 (un-standardized regression coefficients with t-statistics in parenthesis) Overall (Aggregated) Use Academic Use Operational Use Effectiveness Predicted effectiveness 0.837** 0.629** 0.778** (3.97) (3.41) (3.20) Participant characteristics Position 0.181** 0.185** 0.039 (2.36) (2.55) (0.49) WBI policy variables/ Activity features Follow-up 0.060 0.051 0.207 (0.45) (0.41) (1.28) Facilitators and barriers Work-place factors 0.061 0.119 0.008 (0.63) (1.34) (0.09) Country-level factors 0.141 0.177** 0.170* (1.47) (2.13) (1.94) Income GNI per capita (1000s USD) -0.301** -0.323** -0.302** (-3.34) (-3.85) (-2.97) Constant -0.794 -0.145 0.154 (-1.13) (-0.24) (0.20) N 452 452 429 R-squarea 0.426 0.334 0.396 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aR-square from OLS regression of use on effectiveness and all other independent variables. 71 Table A5: OLS regressions of the effectiveness of WBI learning activities including income and time interactions, FY01-FY03 (un-standardized regression coefficients with t-statistics in parenthesis) Effectiveness Effectiveness Overall Effectiveness in Building in Providing (Aggregated) in Raising Knowledge Strategies and Effectiveness Awareness and Skills Approaches Participant characteristics Female 0.128** 0.128 0.123 0.141** (2.10) (1.58) (1.55) (1.97) Position 0.074* 0.064 0.015 0.110** (1.79) (1.21) (0.28) (2.38) Language proficiency 0.308** 0.253* 0.344** 0.296** (2.99) (1.74) (3.01) (2.57) Terminology proficiency 0.352** 0.363** 0.406** 0.309** (3.14) (2.23) (3.04) (2.46) Language*terminology proficiency -0.056** -0.054** -0.068** -0.047** (-2.96) (-2.00) (-3.15) (-2.24) WBI policy variables/Activity features Action plan 0.611** 0.607** 0.583** 0.623** (8.87) (6.92) (6.73) (7.95) Blended/Multi-mediaa -0.041 -0.111 0.027 -0.044 (-0.38) (-0.87) (0.19) (-0.37) VC/Global Dialoga -0.125 -0.117 -0.194 -0.093 (-1.24) (-0.95) (-1.60) (-0.83) Web-based/Computer/Electronica -0.210 -0.274 -0.080 -0.252 (-0.91) (-0.89) (-0.32) (-0.97) WBIFPb -0.029 0.032 0.076 -0.112 (-0.38) (0.28) (0.73) (-1.24) WBIHD/WBIKLb -0.137 -0.085 -0.050 -0.204* (-1.40) (-0.65) (-0.40) (-1.77) WBIPRb -0.110 -0.168 0.064 -0.160 (-0.97) (-1.15) (0.48) (-1.22) Income and time trend Low-income groupc 0.329 -0.255 0.545 -0.074 (0.74) (-0.58) (1.41) (-0.12) Mid-income groupc -0.161 -0.245 -0.119 -0.061 (-0.50) (-0.62) (-0.32) (-0.16) Delivered in FY01, July-December -0.639** -0.428 -0.678** -0.659** (-2.45) (-1.31) (-2.21) (-2.53) Delivered in FY01, January-June -0.490* -0.630** -0.523* -0.344 (-1.90) (-1.96) (-1.75) (-1.35) Delivered in FY02, July-December -0.396 -0.709** -0.578* -0.079 (-1.39) (-2.07) (-1.72) (-0.26) Delivered in FY02, January-June -0.255 -0.554* -0.331 -0.009 (-0.93) (-1.68) (-1.04) (-0.03) Delivered in FY03, July-December -0.552 -0.564 -0.227 -0.689 (-1.28) (-0.97) (-0.49) (-1.14) Delivered in FY01, July-December *Mid-income 0.831** 0.726 0.751 0.843* (2.13) (1.44) (1.57) (1.87) Delivered in FY01, January-June *Mid-income 0.502 0.800 0.525 0.274 (1.26) (1.62) (1.14) (0.61) Delivered in FY02, July-December *Low-income 0.165 0.666 0.124 0.386 (0.34) (1.28) (0.26) (0.58) Delivered in FY02, July-December *Mid-income 0.178 0.728* 0.220 -0.200 (0.49) (1.65) (0.50) (-0.46) Delivered in FY02, January-June *Low-income -0.116 0.572 -0.323 0.267 (-0.22) (1.03) (-0.63) (0.39) Delivered in FY02, January-June *Mid-income 0.274 0.606 0.277 0.056 (0.79) (1.42) (0.67) (0.14) Delivered in FY03, July-December *Low-income 0.377 0.939 -0.019 0.881 (0.65) (1.36) (-0.04) (1.04) (Table A5 continuous on next page) 72 (Table A5 continued) Overall Effectivenes Effectivenes Effectiveness (Aggregated) s in Raising s in Building in Providing Effectiveness Awareness Knowledge Strategies and Skills and Approaches Delivered in FY03, July-December *Mid-income 0.343 0.070 0.223 0.492 (0.66) (0.10) (0.39) (0.68) Delivered in FY03, January-June *Low-income 0.002 0.465 -0.262 0.571 (0.00) (0.84) (-0.53) (0.86) Constant 3.022** 3.469** 3.179** 2.725** (4.88) (3.91) (4.39) (4.10) N 1094 1062 1086 1085 R-square 0.202 0.117 0.154 0.182 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aClassroom and face-to-face learning is the reference. bWBIEN thematic group is the reference. cHigh-income countries (GNI/capita >USD1, 740) is the reference. Table A6: Instrumental variable regressions of participants' use of the K&S acquired from WBI learning activities including income and time interactions in the first stage, FY01-FY03 (un-standardized regression coefficients with t-statistics in parenthesis) Overall (Aggregated) Use Academic Use Operational Use Effectiveness Predicted effectiveness 1.149** 1.201** 1.155** (7.33) (6.40) (6.58) Participant characteristics Position 0.079 0.040 0.118* (1.52) (0.68) (1.89) WBI policy variables/ Activity features Follow-up 0.012 -0.017 0.027 (0.12) (-0.15) (0.25) Facilitators and barriers Work-place factors 0.106* 0.043 0.162** (1.65) (0.63) (2.17) Country-level factors 0.030 0.008 0.041 (0.51) (0.13) (0.59) Income Constant -2.410** -1.993** -3.058** (-4.79) (-3.40) (-5.40) N 953 905 912 R-squarea 0.435 0.343 0.408 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aR-square from OLS regression of use on effectiveness and all other independent variables. 73 Table A7: OLS regressions of effectiveness of WBI learning activities, FY01-03 with detail on participant position (un-standardized regression coefficients with t-statistics in parenthesis) Effectiveness Effectiveness in Overall Effectiveness in Building Providing (Aggregated) in Raising Knowledge Strategies and Effectiveness Awareness and Skills Approaches Participant characteristics Female 0.129** 0.113 0.122 0.151* (2.09) (1.38) (1.54) (2.09) Mid-level positiona -0.085 0.004 -0.188 -0.072 (-0.76) (0.03) (-1.39) (-0.57) Senior-level positiona 0.036 0.090 -0.044 0.059 (0.32) (0.64) (-0.31) (0.47) Top-level positiona 0.225 0.190 -0.111 0.412** (1.54) (1.03) (-0.59) (2.53) Language proficiency 0.313** 0.270* 0.364** 0.280** (3.01) (1.84) (3.18) (2.39) Terminology proficiency 0.337** 0.371** 0.402** 0.272** (2.94) (2.26) (3.01) (2.13) Language*terminology proficiency -0.053** -0.056** -0.069** -0.040** (-2.79) (-2.05) (-3.18) (-1.88) WBI policy variables/activity features Follow-up 0.058 -0.027 0.130 0.074 (0.87) (-0.31) (1.48) (0.96) Action plan 0.607** 0.641** 0.575** 0.607** (8.72) (7.18) (6.61) (7.59) Blended/Multi-mediab -0.057 -0.109 -0.002 -0.059 (-0.55) (-0.85) (-0.01) (-0.50) VC/Global dialogb -0.141 -0.129 -0.205 -0.112 (-1.41) (-1.04) (-1.71) (-1.00) Web-based/Computer/Electronicb -0.131 -0.176 -0.027 -0.182 (-0.57) (-0.57) (-0.11) (-0.71) WBIFPc -0.000 0.039 0.104 -0.069 (-0.00) (0.35) (1.00) (-0.77) WBIHD/WBIKLc -0.082 -0.058 0.016 -0.146 (-0.84) (-0.45) (0.13) (-1.27) WBIPRc -0.059 -0.169 0.136 -0.095 (-0.55) (-1.20) (1.07) (-0.74) Income and time trend GNI per capita (1000s USD) -0.154** -0.191** -0.146** -0.120 (-2.41) (-2.43) (-1.91) (-1.62) Delivered in FY01 Jan.-Juned 0.135 -0.212 0.157 0.280** (1.51) (-1.56) (1.24) (2.75) Delivered in FY02 July-Dec.d 0.158 -0.193 0.107 0.349** (1.29) (-1.23) (0.66) (2.35) Delivered in FY02 Jan.-Juned 0.304** -0.106 0.306** 0.531** (2.70) (-0.73) (2.04) (4.16) Delivered in FY03 July-Dec.d 0.238 -0.138 0.520** 0.284 (1.41) (-0.58) (2.76) (1.38) Delivered in FY03 Jan.-Juned 0.536** 0.177 0.580** 0.732** (3.49) (0.89) (3.06) (4.04) Constant 2.859** 3.493** 2.834** 2.618** (4.92) (4.26) (4.22) (4.07) N 1073 1042 1065 1065 R-square 0.195 0.106 0.153 0.173 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aEntry/junior level is the reference. bClassroom and face-to-face learning is the reference. cWBIEN thematic group is the reference. dFirst six months of FY01 is the reference. 74 Table A8: Instrumental variable regressions of participants' use of the K&S acquired from WBI learning activities, FY01-03 with detail on participant position (un-standardized regression coefficients with t-statistics in parenthesis) Overall (Aggregated) Use Academic Use Operational Use Effectiveness Predicted effectiveness 0.884** 0.902** 0.895** (5.71) (4.88) (5.07) Participant characteristics Mid-level positiona 0.102 0.128 0.158 (0.76) (0.89) (0.94) Senior-level positiona 0.143 0.111 0.199 (1.12) (0.84) (1.21) Top-level positiona 0.459** 0.343* 0.705** (2.70) (1.87) (3.48) WBI policy variables/ Activity features Follow-up 0.030 0.001 0.048 (0.36) (0.01) (0.48) Facilitators and barriers Work-place factors 0.148** 0.099 0.199** (2.61) (1.63) (3.00) Country-level factors 0.062 0.053 0.059 (1.14) (0.86) (0.94) Income GNI per capita (1000s USD) -0.160** -0.129* -0.203** (-2.71) (-1.92) (-2.96) Constant -1.099* -0.768 -1.587** (-1.93) (-1.15) (-2.44) N 953 905 912 R-squareb 0.456 0.354 0.436 * Significant at the 0.1 significance level. ** Significant at the 0.05 significance level. aEntry/junior level is the reference. bR-square from OLS regression of use on effectiveness and all other independent variables. 75