31299 Evaluating the Impacts of FY02-03 WBI Activities on Nigerian Participants Cristina Ling Chard Diana J. Arango WBI Evaluation Group EG04-87 The World Bank Institute The World Bank Washington, DC June 27, 2004 ACKNOWLEDGEMENTS This report was prepared for the World Bank Institute (WBI) under the overall guidance of Marlaine Lockheed, Manager, WBI Evaluation Group. The team was led by Cristina Ling Chard of the Institute Evaluation Group (IEG). The author thanks Mark Tomlinson, the Country Director of Nigeria, and his staff Okintunde Adekole for his support; Kofi Anani for his valuable guidance; Bankole Oni, Akin Opatola, and Isiaka Balogun for administering the participant surveys. I would also like to thank Guy Darlan and Kofi Anani for kindly serving as peer reviewers of this evaluation report; Denis Nikitin for conducting preliminary analyses; and Humberto Diaz for his help with formatting this document. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=getPubs&cat egory=Publications&Intro=yes&instructions=no&showDetails=no&ID=. Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Cristina L. Chard ii TABLE OF CONTENTS TABLE OF CONTENTS ...................................................................................................iii EXECUTIVE SUMMARY.................................................................................................v 1. INTRODUCTION........................................................................................................1 Background .............................................................................................................1 Objectives................................................................................................................3 2. METHODOLOGY .......................................................................................................4 Survey Sampling Procedure....................................................................................4 Survey Participants .................................................................................................6 Activities Sampled ..................................................................................................7 Operational Staff Interviews ...................................................................................7 Focus Group Discussions........................................................................................7 3. PARTICIPANT SURVEY RESULTS.........................................................................9 Relevance of WBI Activities to Nigeria's Needs ...................................................9 Assessing the Influence of WBI Activities in Key Impact Domains....................10 Activity Sustainability: Follow-up by Organizers and Participants .....................11 Activity Features...................................................................................................11 WBI's Role in Nigeria ..........................................................................................12 4. ACTIVITY EFFECTIVENESS .................................................................................13 Level of Effectiveness...........................................................................................13 A Model of Activity Effectiveness .......................................................................14 5. USE OF KNOWLEDGE AND SKILLS....................................................................18 Level of Impact .....................................................................................................18 A Model of Impact: Use of Acquired Knowledge and Skills...............................19 6. CONCLUSIONS AND RECOMMENDATIONS.....................................................22 Recommendations about Evaluation Methods......................................................22 Recommendations for Activities...........................................................................23 iii ANNEXES Annex A: WBI Activities for Nigerian Participants FY02-03..............................27 Annex B: World Bank Institute (WBI) Country Focus Evaluation Questionnaire Nigeria...........................................................................33 Annex C: Nigeria WBI Program Survey FY02-03...............................................41 Annex D: Interview and Focus Group Questions for Country-Focused Impact Evaluation.................................................................................44 Annex E: Operational Staff Survey......................................................................45 Annex F: Focus Group Discussion.......................................................................50 Annex I: Millennium Development Goals (MDG)...............................................52 Annex H: Descriptive Statistics ............................................................................53 iv EXECUTIVE SUMMARY This report evaluates WBI activities offered to Nigerian participants in FY02-03. The study uses both quantitative survey data based on participant responses (N=70), and qualitative data from interviews with country staff and focus group discussions to assess the effectiveness and impacts of past WBI learning events. The overall purpose of this paper is to establish benchmarks against which WBI's upcoming country-focus activities can be compared. In this evaluation, we investigate the factors relating to WBI activities': · effectiveness, conceptualized as individual level benefits from the activity: 1) academic effectiveness, the activities' effectiveness in broadening individual knowledge through enhancing participants' skills, raising their awareness in Nigeria's development issues and their individual roles as "agents of change,"; 2) operational effectiveness, WBI's effectiveness in enhancing individual actions through teaching strategies for approaching the needs of participants' work organizations and Nigeria as a whole, as well as, developing contacts and partnerships to build coalitions in the field; and 3) overall effectiveness, which is WBI's effectiveness in both these domains; and · impact, or the use of the knowledge and skills acquired from the WBI-sponsored activity: 1) academic use, which is the utilization of WBI knowledge and skills for raising public awareness in development issues generally, and more specifically through research and teaching; 2) operational use, the application of WBI knowledge and skills to take action at the community level (i.e. organizing collective initiatives), institutional level (i.e., applying new skills at work organization); and country level (i.e. policy and legislation); and 3) overall use, which is the application of WBI-acquired learnings in various areas across the board. Evaluation results point to the need for WBI activities that are country focused (i.e. more relevant to Nigeria's needs). Our models predicting activity effectiveness v consistently show that activities rated as relevant to Nigeria are also deemed to be more effective by participants. In fact, activity relevance to the country far outweighed activity relevance to participants' work. Hence, we conclude that WBI's country focus approach is a crucial strategy for successful training. Proxy measures of country focus such as whether the activity was located in Nigeria also supported this conclusion. The only other activity feature that predicts effectiveness was activity duration ­ increasing the length of the activity (up to seven days, after which there are marginal diminishing returns) enhanced effectiveness ratings. As one would expect, our model of impact shows that the more effective the activities are, the more likely participants are to report using lessons from them. We also find that participant use of WBI-acquired knowledge and skills is significantly more related to work factors (e.g. work procedures, colleagues, incentive systems, funding etc.) than country factors (e.g. country policies, social groups, political groups, Nigeria's readiness for reform). This suggests that working individuals are the catalyst for change and that even if an activity is engaging and focused on the country, in order for participants to apply activity learnings, individuals' working conditions must be amenable to change. Hence, institutional structures are highly important to participants' implementation of change. This conclusion is further supported by our finding that participants who discuss WBI issues at work, or are in academic jobs, are also more likely to use their WBI-acquired knowledge and skills. Overall, our results suggest that WBI's FY02-03 activities have had positive effects in Nigeria. As the country-focus agenda in Nigeria advances, we expect WBI's future evaluations, to produce significantly more positive results than the benchmarks we establish in this report. Already, our analysis predicts that as activities focus more on Nigerian issues, and become more applicable to people's work, learning events will become more effective and have greater impact. vi 1. INTRODUCTION BACKGROUND 1.1 According to the Nigeria Joint Interim Strategy (JIS), the Bank's role in Nigeria is one of capacity-building.1 The overarching goal, as outlined in the JIS, is to help Nigerians manage their own resources more effectively. To that end, the Bank established a framework for assisting Nigeria centered around three pillars: a)improving economic governance; b) creating conditions for rapid private sector-led, poverty-reducing growth in the non-oil economy; and c) improving basic service delivery and empowerment (enabling local communities to take charge of their own development). 1.2 In FY 2002, WBI unveiled a new country pillar to spearhead a "country-focused" approach to building and developing capacity in client countries. The immediate goal is to enhance capacity in 12 priority countries in key areas as identified in the country assistance strategies (CASs) for these countries. WBI selected these priority countries in consultation with the regions, purposefully selecting a low-income and a middle-income focus country from each of the six World Bank (WB) regions. For the Africa Region, WBI identified Nigeria as its low-income focus country. 1.3 WBI's new country-focused initiative is a response to questions that have been raised regarding on-the-ground impacts of WBI's past interventions and their relevance for the Bank's goals set for its individual client countries. WBI's country-focus pillar is supposed to reorient WBI learning activities in favor of greater alignment with the Bank's country assistance programs and towards broader and more sustainable impacts, including in Nigeria. 1.4 Attachment 3 of the JIS details the role for WBI in Nigeria as it embarks on this country-focused approach. The capacity enhancement objectives are currently organized around four desired outcomes: (a) stability; (b) growth; (c) empowerment; and (d) governance. WBI will offer learning activities for Nigerians in various programs to provide support for each of these four objectives. (a) Stability: One indicator for stability will be whether senior teams at the Federal and State levels are responsible for setting the economic policy agenda and management of an established and functioning economy. Both Poverty Reduction Strategy Paper support and Social Protection and Risk Assessment training will be provided by WBI. The other indicator will be the "National Action Plan for Addressing Violent Conflict." It should be put in place and progressing towards implementation. To that end, WBI will 1Memorandum of the President of the International Development Association and The International Finance Corporation to the Executive Directors of an Interim Strategy Update for the Federal Republic of Nigeria. February 13, 2002.Report No. 23633-UNI. 1 provide programs on the media and on consensus building for natural resource management. (b) Growth: Growth indicators are defined as improving the financial infrastructure and increasing the availability of commercial medium-term projects and corporate financing. WBI is planning a financial sector capacity enhancement program to "strengthen the capacity of key players in the financial sector to make informed decisions and solve financial sector problems." Additionally, the JIS cites the plan for a national WTO workshop. (c) Empowerment: Indicators of empowerment would show: (a) an improvement in human development including an end to the growth of HIV/AIDs; (b) increase in school enrollments and completion rates; (c) availability of health care; and (d) social protection for the most vulnerable members of society. In response to these needs, WBI will offer programs on: HIV/AIDS, "Using Media and Technology to Combat HIV/AIDs" and options for children and youth employment; "Respect for Diversity through Education"; "EFA/Basic Education Reform Strategies and Teaching Quality"; "Building Capacity for Health Sector Reform for Achieving the Health MDGs"; "Social Protection Risk and Vulnerability." Other indicators of empowerment outlined in the JIS are: improved availability of community-level infrastructure; access to knowledge on economics including women; natural resource management; and improved access to microfinance strategy. WBI activities include help in developing a poverty reduction strategy that focuses on community empowerment and builds upon active local level participant which feeds into the CAS; the Community Driven Development program; and Building Capacity in Nigerian Youth Organizations. (d) Governance: Governance indicators are defined as the development and implementation of strategic approaches to curbing corruption by the ICPC. This will be supported by a series of WBI activities designed to improve the impact of internal and external auditing on the use of public financial resources, and learning offerings on judicial reform for improving governance, media, law, and regulation. The second indicator of good governance is a close alignment between federal and state government policy making and Nigeria's development priorities. WBI will offer programs for parliamentarians for strengthening oversight committees and establishing a legislative budget research office; strengthening parliamentary staff; and establishing a PAC association. 1.5 Having briefly listed WBI's programs in Nigeria for FY04-05, as stated in the JIS, we can now step back to examine WBI's activities in FY02-03. That is, the work taking place before the country focus plan was implemented. In this paper, we seek to establish 2 baselines against which future comparisons can be made after the country focus work outlined above is implemented. 1.6 In FY02-03, 2,046 Nigerian participants received training in 145 WBI learning activities (See Annex A for complete list). Activities represent most WBI programs: Attacking Poverty; Community Empowerment for Social Development (CESI); Decentralization and Public Finance; Education; Finance; Globalization and Macroeconomics; Good Governance; Health and Population; AIDS; Infrastructure; Urban; Private Sector Development; Rural Poverty and Development; Social Protection; Sustainable Development: Natural Resource and Environmental Management; and Trade. 1.7 This study is one of twelve country WBIEG retrospective evaluations of the impacts of WBI's interventions in country-focus priority countries (Brazil, Burkina Faso, Egypt, Guatemala, Indonesia, Kenya, Russia, Sri Lanka, Thailand, Tajikistan, and Yemen). OBJECTIVES 1.8 This study has two main objectives. This first is to assess the impacts of past (FY02 and FY03) WBI interventions in Nigeria. This retrospective appraisal aims to provide immediate lessons on how, in the Nigeria context, the beneficial impacts of WBI interventions might be broadened and deepened. The second objective is to establish benchmarks and other important data for subsequent evaluations of WBI's upcoming country-focused initiatives in Nigeria. 1.9 The retrospective evaluation investigates two major research issues that relate to the effectiveness and impact of WBI activities. First, we examine whether WBI activities were effective in building Nigerian capacity and providing benefits to participants. Second, we investigate whether these activities had an impact through participants utilizing what they learn. 3 2. METHODOLOGY 2.1 We use a mixed-methods approach to data collection and analysis in this evaluation, employing a mixture of quantitative and qualitative methods. Our methods include: (a) a quantitative survey of a random sample of Nigerian alumni from WBI activities (see Annex B); (b) survey of WB operational staff members who worked in Nigeria on projects related to WBI activities during the time period under examination (see Annex C); and (c) a focus group discussion (or FGD) with five Nigerian participants who took part in a distance learning activity (see Annex D). SURVEY SAMPLING PROCEDURE 2.2 In FY02-03, 1,838 Nigerian participants were reported as taking part in 145 WBI activities lasting more than one day in duration (see Table 1). However, only 959 unique names were documented in WBI's Client Registration System (CRS), and of these, only 854 had any of four pieces of contact information (e-mail, work phone, fax phone, work address). Table 1. FY02-03 WBI Participants from Nigeria, as documented in CRS N % Reported at activity level 1,838 Reported at participant level 959 52 With contact information 854 46 Sampled 413 48 Traced 165 40 Responded 70 42 2.3 Based on our assumption of random inclusion in the CRS2, we selected a stratified random sample of 200 participants and two groups of replacements (N=213) from the 854 participants with contact information. Stratification of the sample by program included purposefully selecting participants from Attacking Poverty, CESI, Social Protection, and Urban.3 Only 165 participants were traceable out of a total of 413.4 In 2As there is no strong reason to believe otherwise, we assume that the participants for whom information exists do not systematically differ from the participants for whom information is missing. In other words, whether or not participants and participant-related information were included in the CRS database are random occurrences. A problem would arise if all participants listed in the CRS were from activities within the same WBI program, because surveying these participants would result in findings that are program- specific instead of generally applicable to WBI as a whole. Fortunately, the CRS-recorded participant data represent diverse programs and activities. This suggests that record keeping was done at random and thereby excludes the likelihood of a systematic bias in our sample. 3Weighting the responses correct for the unequal probability selection procedure. This is described in the analysis and results section. 4For example, the address listed for a participant in the CRS led the interviewers to a restaurant. 4 the end, there were 70 completed surveys out of the 165 potential respondents who could be contacted, for an adjusted response rate of 42 percent. These outcomes again point to serious deficiencies in the maintenance of participant information in WBI's CRS database.5 The response rate is comparable to some other lower income countries (Burkina Faso, Guatemala and Yemen ) evaluated by IEG which ranged between 38 percent and 58 percent. Country conditions contributing to Nigeria's response rate are addressed below. 2.4 A local consultant from the Nigerian Institute of Social and Economic Research in Ibadan led the survey team which included two other interviewers. Nigeria's infrastructure and socio-economic climate requires adapting the standard interviewing methodology to the country conditions. The telecommunications infrastructures services are accessible to a limited few, and work erratically at best. Telephone, email and fax is only easily accessible by a minority of participants. Most people rely on computer cafes where they pay to access the Internet and email. Even when computers are accessible it is still necessary to rely on the sporadic electricity supply. Thus, given the infrastructure, conducting surveys on the ground in Nigeria are challenging to say the least. 2.5 Consequently, the methodology was to go door to door. The standard protocol was to drop off the questionnaire in person and then to make up to two more visits to collect the completed survey.6 This caused another set of challenges related to transportation. For instance, the high traffic problems, particularly in Lagos, made interviewing former participants difficult. Hence, relying on in-person interviews in Nigeria is significantly more challenging than conducting surveys in countries where email and fax are readily available such as countries in which response rates were higher i.e. Indonesia or where commuting is easier. 2.6 Additionally, the local consultant pointed out the challenges of tracer studies in Nigeria because of what he called the "migrant nature of the population." Many of the people he tried to contact had already moved without forwarding information. Given the fluid nature of Nigerians, it is difficult to maintain their most current contact information. 2.7 Nevertheless, the fact remains that the majority of participants who actually received their questionnaires (58 percent) did not respond. The consultant reported that participants were simply not interested in participating due to a general air of social frustration among Nigerians. 2.8 Non-responses were sometimes more subtle. Some respondents claimed to have misplaced their questionnaires, while others were not available when called upon. Again, the constraints of having to go door-to-door played a major factor in the response rates. 5 Given that actual names were documented for only 52 percent of total participants reported at the activity level, it is clear that WBI-wide participant data collection efforts need to dramatically improve. Participant names are simply not being documented generally half the time. For the individuals that were documented ten percent did not have any type of contact information reported. More importantly, the contact information that was documented was incomplete or no longer accurate. Documentation efforts must improve, particularly in the context of "country focus" where the goal is local capacity enhancement 6Some participants submitted their questionnaires directly to the World Bank office in Abuja. 5 2.9 This disinterest was underscored by the failure of two focus group meetings scheduled to take place at the Sheraton Hotel and Towers in Abuja. Invitations were delivered by hand to twelve participants and four were mailed by post. Although all were residents in Abuja, only one individual showed up. Finally, a third focus group was planned and successfully conducted in Ibadan (details reported in later sections). 2.10 It seems that participants are much less eager to participate in evaluation surveys than WBI's activities. This is undoubtedly partially due to there being no inherent benefits for former participants to take the time to fill out our questionnaire. We do not offer a monetary incentive or learning gains. Rather, we ask them to do this as a favor. Future response rates might be improved if WBI's instructors or facilitators notify participants of the possibility that they might be contacted in the future to partake in an evaluation survey and encourage them to participate. This could therefore be viewed by participants as an obligation for taking the course. Nevertheless, country conditions would still play a major role in the response rates obtained. SURVEY PARTICIPANTS 2.11 Of the 70 sample respondents, women represent a majority (67 percent). Respondents' main work affiliations included work in: universities and research institutions (30 percent); national, regional, and local governments (26 percent, nine percent, and two percent respectively); the private sector (11 percent); not-for-profit organizations (12 percent) and other organizations (three percent). Nearly half of respondents were in somewhat academically-oriented positions such as research (37 percent) and teaching (12 percent). Additionally, respondents' careers included: management and administration (22 percent); policymaking and legislation (15 percent); services (10 percent); and other not for profit and work organizations (four percent). Respondents primarily held higher level positions. While a few held entry/junior level positions (nine percent), the bulk of respondents fell in the mid to upper range, with 30 percent at the middle level and 54 percent in senior and uppermost level positions.7 Table 2: Distribution of Nigeria Participants by Program FY02-03 Program Survey Program Participants Participants Health & Population 6% 7% Social Protection 17% 16% Education Program 0% 2% Good Governance 3% 29% Finance & Banking 1% 3% Attacking Poverty 26% 3% CESI 27% 18% Macroeconomics and Policy Assessment 1% 5% Private Sector Development 3% 8% Sustainable Development 7% 8% No Associated Program 0% 1% 7Three percent fell in the Other category. 6 ACTIVITIES SAMPLED 2.12 The survey respondents underrepresented participants in Good Governance activities and over-represented participants from Attacking Poverty and Community Empowerment for Social Development (CESI). Our sample respondents participated in the following nine programs: Attacking Poverty (26 percent); Community Empowerment for Social Development (CESI) (27 percent); Finance and Banking (one percent); Macroeconomics and Policy Assessment (one percent); Good Governance (three percent); Health/AIDS (six percent); Private Sector Development (three percent); Social Protection Risk Management Learning (seventeen percent); and Sustainable Development Environmental Management (seven percent). The actual distribution of WBI programs for Nigerian participants in FY02-03 with contact information differed from the sample particularly in the programs for which we used a stratified sampling process. In the population distribution, CESI represents 18 percent, Attacking Poverty represents three percent, and Social Protection represents 16 percent. Thus, we weighted the sample to correct for the non-randomness of the stratified selection procedure. OPERATIONAL STAFF INTERVIEWS 2.13 We compiled data on WBI activities from operational staff based on written responses to our country staff questionnaire (see Annex C). The data we gathered focused on five main issues: (a) results from WBI activities, (b) barriers to impact or the use of WBI-acquired knowledge and skills (c) WBI's role in Nigeria as a learning provider, (d) alignment between WBI activities and the CAS, and (e) suggestions for future WBI offerings. The general purpose of the interviews was to obtain feedback on past and future WBI activities.8 2.14 We identified 28 WB staff involved with Nigeria in FY02-03 who might be familiar with WBI's work. We contacted these staff members initially by email. We followed up with two waves of emails, the first two weeks later and the second one week later. For those who still did not reply, we telephoned the individuals directly asking them to respond. After several attempts, we obtained a total of eleven completed questionnaires from Bank country staff (39 percent). Results are reported throughout the report and summarized in their entirety in Annex E. FOCUS GROUP DISCUSSIONS 2.15 We conducted a focus group discussion with five WBI alumni who participated in one or more WBI activities. Participants represented three WBI programs: Attacking Poverty, Social Protection, and Environment. The participants were selected based on program and location. Because the meeting was to be held in Ibadan, only former participants living in Ibadan were invited for the discussion.9 8Annex E summarizes the key results of the resident staff interviews. Throughout this paper, we also refer occasionally to these results to elaborate on the findings from the participants' survey. 9Recall, two focus group meetings scheduled to be held in the capital of Abuja were unsuccessful in attracting participants. 7 2.16 The focus group discussion took place on March 25, 2004. The local consultants facilitated the event. One hundred percent of the focus group participants were professors and research fellows. The activities they attended were from the following programs: Attacking Poverty, Environment, and Social Protection, The findings from the focus group discussion are summarized around four major themes: relevance to the activity; usefulness of the program; participation in similar activities and follow-up. Annex F reports the key findings of the focus group discussion to which we will refer as necessary to elaborate on the main findings from the participant survey. 8 3. PARTICIPANT SURVEY RESULTS 3.1 One purpose of this evaluation is to establish a baseline for activities held for Nigerian participants against which WBI's future country-focus work can be assessed. Accordingly, we asked survey respondents a series of questions measuring the relevance, effectiveness, and impact of WBI's FY02-03 activities to participants' work and Nigeria, that is, before WBI's country focus approach was established. RELEVANCEOF WBI ACTIVITIES TO NIGERIA'S NEEDS 3.2 Survey results show interesting findings. First, as expected, the majority of respondents reported that the activities they attended were not solely for Nigerians. Fewer than half of respondents, 42 percent, reported that the learning event was designed specifically for Nigerian participants. Reports from focus group participants supported this finding. Focus group discussions of the activities indicated that events are primarily regionally focused on Africa, the continent.10 3.3 Second, although not designed specifically for Nigerians, activity topics were assessed by participants as pertinent to Nigeria. The average rating was 5.37 on a scale from one (not at all relevant) to seven (extremely relevant). While half of respondents (50 percent) rated the activity as "relevant" or "extremely relevant" to Nigeria, even more respondents reported the activity was relevant to their work. That is, 69 percent of respondents gave high ratings on relevance to work.11 3.4 The focus group discussions (FGD) indicated that activities were relevant to both Nigeria and participants' work. It is important to preface this conclusion by recalling that the FGD participants were social science researchers who are trained to think about the "big picture." One would expect that it would be easier for social science researchers to make the connection between a course and its implications for Nigeria compared to non- academics who may be more task-oriented. Overall, FGD participants reported that the seminar topics were important to their work and research in social science and highly pertinent to Nigeria's development needs. 3.5 Third, respondents reported the frequency with which issues raised in WBI activities were discussed at work, with local partners, government officials, and or NGOs. The mean score was five, on a scale from one (never discussed at all) to seven (discussed thoroughly). While issues raised in WBI events were not discussed at great length as perhaps, the main topics of conversations, they also were not entirely neglected in participants' work. 10The exception was the Environmental Economic program. Focus group participants reported this was designed specifically for Nigeria. 11The average rating of activity relevance to work was 5.77. 9 ASSESSING THE INFLUENCEOF WBI ACTIVITIES INKEYIMPACT DOMAINS 3.6 In order to determine the nature and extent of influence that WBI learning activities may have had in Nigeria we asked respondents how positively (or negatively) the activity may have contributed to outcomes in six development areas. Almost all areas were rated above average: raising public awareness (5.55); research (5.31); teaching (5.14); country development strategies (5.11); new practices in work organizations (5.22); collective initiatives (4.87); legislations or regulations (4.07). 3.7 Figure 1 illustrates the percentage of participants who reported positive impacts from WBI in various domains. WBI's impact was observed by less than half of respondents in: raising public awareness (48 percent); research (42 percent); teaching (40 percent); work organizations (39 percent); country development strategies (39 percent); and collective initiatives (34 percent). There was not much perceived influence in legislation or regulation (8 percent). Although legislation was rated lowest by participants, focus group members indicated that their policy memos influenced by WBI were used to shape policy. In other words, the memos written by researchers in the focus group are intended to influence policy decisions, however there is no direct evidence for this. Researchers hope their work is used but a multitude of factors play important roles in policy making. Figure 1. Perceived Influence of WBI in Key Impact Domains (Percent rating 6 or 7) 80% 60% 48% 42% 40% 39% 39% 34% 40% 20% 8% 0% at Public work devpt awareness Research Teaching strategies Collective initiatives regulation practices untry Legislation/ Co New 3.8 The Bank operational staff we interviewed observed concrete "on the ground" evidence of impact from WBI activities including: a) increases in investigative reporting and more balanced news coverage; b) workshops in auditing organized by the federation of auditors from all states, and events arranged by the accountant general's office; c) increases in transparency and accountability in project outcomes; d) use of CDD principles by clients; e) participatory planning in PRSP process; f) increased policy dialogue; g) proposed implementation of the sustainable development action program and reform of the Niger Basin Authority ; and h) preparation of the national water resources policy by the Federal Ministry of Water. 10 3.9 Some participants were not able to give any examples of "on the ground" impact (36 percent). Some stated that it was too soon to see the impacts of WBI's efforts, e.g. "results so far are only on paper," "projects involved have not yet become effective, so no results yet." ACTIVITY SUSTAINABILITY: FOLLOW-UPBY ORGANIZERS AND PARTICIPANTS 3.10 Sustainable development is the ultimate goal for clients of the World Bank. Thus, in order for WBI's activities to have an impact and help clients achieve this goal, it is necessary for there to be follow-up that directly involve participants. Follow-up can be executed in various ways such as e-mail discussions, face-to-face meetings, newsletters, etc. and can be initiated by organizers (WBI) or participants themselves. The key for change is that new approaches continue to be discussed and supported after the formal learning event is over. 3.11 The survey responses show disappointing results regarding follow-up to WBI activities. Only 21 percent of respondents were contacted by organizers and only 34 percent contacted WBI. Among participants who were contacted, WBI was rated as somewhat helpful, i.e. 4.4 on a scale ranging from one (not helpful at all) to seven (extremely helpful). While follow-up seems low, Nigeria falls in line with other countries where participants reported lack of follow up. We expect that reported follow- up would be much higher in future evaluations after the country focus approach is established. 3.12 One of the objectives for WBI activities is to encourage its alumni to make and maintain contacts with others interested in the same development issues. A majority of respondents, or 62 percent, reported that they were provided with contact information (e.g. e-mail, telephone, address) of other participants who attended the same activity. Among those who were provided with contact information for other participants, only 24 percent used it to continue activity related discussions. 3.13 This pattern was also demonstrated by the findings in our FGD. Focus group participants reported that they were provided with detailed contact information for other participants, however none of the respondents used the information to contact their classmates. In fact the only follow-up cited by participants was a WBI email conferencing facility, "PAC TALK." Participants did not indicate how often they used the facility, however, Nigeria's infrastructure makes it challenging to maintain email correspondence. 3.14 Interviews with Bank staff indicated other types of follow up activities. One Bank staff stated that WBI helped clients organize follow up workshops and public health campaigns. ACTIVITY FEATURES 3.15 Fifty percent of participants reported developing an action plan/strategy such as work plans, strategy papers, or policy documents. Among those who reported developing an action plan in the activity, 72 percent used it at work. Past research shows that action 11 plans are effective tools for training (e.g. in Brazil, Egypt, Russia, Sri Lanka, and Thailand). 3.16 Respondents also rated the delivery method used in the WBI learning activity in which they participated on a scale from one (not effective at all) to seven (extremely effective). There was not much difference in ratings of delivery method effectiveness according to type of learning activities in which the majority of respondents participated. Blended methods using a mix of video conference and face to face were rated the highest, 5.67 (N=21), followed by conferences, 5.36 (N=25), and classroom, i.e. face to face only, 5.29 (N=14). Fewer respondents participated in web-based learning 4.0 (N=3), distance learning sessions i.e. video only 5.75 (N=4), and study tours 5.0 (N=2). The focus group results supported the conclusion that there is no perceived difference in the effectiveness of activities based on delivery method. 3.17 Respondents in the survey reported that they were proficient in the language used for instruction (5.81) and the technical terminology (5.71) used in the activity.12 This suggests that on the whole, participants were proficient in both and therefore able to comprehend everything in the activity clearly.13 WBI'S ROLE IN NIGERIA 3.18 A remaining question we attempt to understand is the role that WBI plays for Nigerian participants. In other words, are WBI's activities unique? Or, are other organizations providing the same kind of learning opportunities, thereby rendering WBI's services redundant? We asked respondents to report whether they participated in any learning activities with similar content offered by other organizations in Nigeria. Their answer appears to be a resounding no. Only 17 percent of respondents reported being aware of comparable learning events offered by other organizations. Of the groups who were familiar with non-WBI training activities, average ratings indicated that WBI's programs are more useful, with a mean score of 5.08, where one is "much less useful" and seven is "much more useful." 3.19 Focus group participants reported that the content in WBI's activity was unique compared to training provided by other agencies. Training providers cited as offering similar learning activities included: the African Economic Research Consortium; the British Council and Department for International Development (DFID); the International Labor Organization; the Central Bank of Nigeria; and the National Planning Commission. 3.20 Most country Bank staff (64 percent) agreed that WBI plays a unique role in Nigeria because of its role as a "global knowledge bank." WBI's vast international knowledge in a variety of development areas is considered to be its main asset by most staff. However, one staff member suggested that WBI was not unique and that it would be more beneficial to Nigerians if the Bank's training was offered through local institutes with more experience and better knowledge of Nigeria. 12Respondents rated their proficiency from one (not proficient at all) to seven (highly proficient). 13However, we should note that in the small focus group discussion the issue of instructional language was a major concern. Participants in the focus group meeting explained that the facilitator was speaking in French, rather than English which made it difficult for the English-speaking audience to understand. 12 4. ACTIVITY EFFECTIVENESS 4.1 This chapter reports the overall level of WBI activity effectiveness in Nigeria and presents a model of the determinants of effectiveness. LEVELOF EFFECTIVENESS 4.2 We measured "effectiveness" with a series of questions that asked respondents to self-rate the impacts of WBI activities on respondents. Specifically, respondents were asked to assess the "effectiveness" of the learning event in five areas: enhancing understanding and awareness of development issues in Nigeria (5.96); increasing their knowledge and skills (5.84); helping participants understand their roles as agents of change in Nigeria's development (5.73); providing strategies for addressing Nigeria's development needs (5.05); offering approaches for addressing the needs of their organization (5.13); and introducing respondents to others also interested in the theme of the learning activity (5.07).14 Overall, responses are positive with mean scores indicating that activities were rated as "somewhat effective" and "effective." However, it is important to note that responses ranged across the spectrum, from "ineffective" (two) to "extremely effective" (seven). 4.3 Figure 2 illustrates the proportion of respondents rating WBI activities as "effective" or "extremely effective." Overall, the majority of respondents, or 55 percent, rated WBI as effective. However, a comparison of the various areas in which WBI was rated on effectiveness shows that WBI was more successful in academic effectiveness [knowledge and skills (K&S) building (72 percent), awareness raising (74 percent) and teaching people about their role as an agent of change in Nigeria's development (75 percent)] than operational effectiveness. Less than half of participants rated WBI as effective or extremely effective in developing strategies for their organization (48 percent) or country (35 percent). Likewise, a minority of respondents (36 percent) indicated that WBI was effective in helping them develop contacts and partnerships or build coalitions in the field. This finding was supported in the focus group discussions where participants described the activities to be most effective in raising their awareness of development issues and teaching them new skills. 4.4 Additionally, interviews with Bank country staff also point to WBI's academic effectiveness. A little more than half of the staff we interviewed (55 percent) described outcomes from WBI learning events in areas of awareness raising, K&S building, and training of trainers i.e. capacity building in: methods for modernization government auditing practices; investigative journalism skills; resources for information in the Bank; facilitators trained in community driven development (CDD) and public health information campaigns. The remaining staff (45 percent) cited outcomes associated with strategies and approaches for development such as PRSP process for HD officials; approaches for Niger River Basin development; approaches for CDD; participatory M&E 14The response options ranged from one, "not effective at all" to seven, "extremely effective." 13 plans for communities by Food International; Risk and Vulnerability assessment framework for Nigeria; and Social Protection policy note. Figure 2. Perceived Effectiveness of FY02-03 WBI Activities (Percent rating 6 or 7) 100% 75% 74% 80% 72% 55% 60% 48% 36% 35% 40% 20% 0% skills change Raising Overall of awareness Effectivenss strategies strategies Organizational Agent Knowledge/ Contacts/partnerships Country A MODEL OF ACTIVITY EFFECTIVENESS 4.5 A primary evaluation question for Nigeria is: "What determines activity effectiveness among Nigerian participants?" To understand what factors determine the various types of WBI effectiveness, we relate participants' self-measures of effectiveness to WBI activity features and participant-level characteristics. We estimated several models of effectiveness. The most meaningful results are based on the model outlined in equation 4.1 predicting overall effectiveness (Y1), academic effectiveness (Y2), and operational effectiveness (Y3): Activity effectiveness1,2,3 = 0 + 1 Nigeria relevance + 2 Nigeria work relevance + 3 Nigerian location + 4 action plan + 5 DL + 6 activity duration+ 7 activity duration2 + 8 technical terminology + 9 female + (4.1) 4.6 Three dependent variables are: a) overall effectiveness, which is a summary score for effectiveness in increasing awareness in development and one's role as an agent of change, enhancing knowledge and skills, building coalitions for development, strategies and approaches for organization and country (as explained previously). Effectiveness is a continuous variable that is recalibrated (from the original seven-point scale) to range between zero (ineffective) and one (effective); b) academic effectiveness, which is a summary score of effectiveness in the areas of learning only (awareness, knowledge and skills, agent of change); and c) operational effectiveness, which is a summary score of effectiveness in the operational areas: building coalitions, strategies and approaches for work organization and also for Nigeria. 14 4.7 Based on the philosophy underlying WBI's new country focus approach, we expect activities with a Nigerian focus to be more effective. The independent variables capturing this Nigerian focus are: Nigeria relevance, Nigeria work relevance, and Nigerian location. Nigeria relevance measures the extent to which the activity was rated as relevant and applicable to Nigeria as a country and work in Nigeria. Ratings of this variable was originally on a seven-point scale, but was recalibrated to range between zero (not relevant at all) and one (highly relevant) for easier interpretation of regression results. Nigerian location is a dummy variable that is equal to one if the activity location was Nigeria and zero otherwise. 4.8 WBIEG's impact study of program activities offered in FY00-01 (Khattri, Quizon, et al., 2002) notes that WBI activity characteristics can influence ratings of activity effectiveness. We test the significance of various activity features by including in the regression model a number of activity-related variables defined as follows: action plan (equal to one if the WBI learning activity included action plans; zero otherwise), DL (equal to one if distance learning was used; zero otherwise), activity duration which is the number of days the activity lasted (along with the variable activity duration squared, taking into account the likely quadratic nature of the relationship, i.e., more days are better to a point of "diminishing marginal returns"). We also include female as a demographic control for gender. 4.9 We estimated the models using Ordinary Least Squares (OLS) regression.15 Table 3 presents the OLS results for the three measures of effectiveness: overall effectiveness (first column), academic effectiveness (second column), and operational effectiveness (third column). The models predict the three measures of effectiveness very well with an adjustedR2 =.68 for overall effectiveness, R2=.58 for academic effectiveness, and R2=.59 for operational effectiveness. Table 3: Activity Effectiveness as a Function of Country Focus, Activity and Participant Characteristics DV= Operational DV= Effect. DV= Academic Effect. Effect. Variable Estimate Estimate Estimate Nigeria relevance .568*** .569*** .565*** Nigeria work relevance .071 .002 .173* Nigeria location .079* .056 .115** Action plan .010 .018 -.002 Distance Learning .000 .020 -.029 Activity Duration .071** .042 .115*** Activity Duration² -.005** -.003 -.009*** Technical terminology .033 .059 -.007 Female -.073** -.083** -.058 Constant .038 .171 -.161 N=70 N=70 N=70 ***=p<.01,**=p<.05, *=p<.10 R2=.68 R2=.58 R2=.59 15Missing values were replaced using multiple imputations based on a multivariate regression-based method (Allison, Paul D. 2004 working paper "Multiple Imputation for Missing Data: A Cautionary Tale.") 15 4.10 We find some support for the country-focus hypothesis. Specifically, the statistical significance and large magnitude of the effect for Nigeria country relevance (=.57, p<.001) indicates that assessments of the effectiveness of WBI activities become more positive as ratings of activity relevance to Nigeria increase. Activities rated as extremely relevant to Nigeria have significantly higher effectiveness ratings (or by up to 57 percent more) than those activities that are not perceived as relevant. This is true for all types of effectiveness: overall effectiveness, academic effectiveness, and operational effectiveness. 4.11 Activity relevance to the participant's work, Nigeria work relevance, is marginally significant in predicting operational effectiveness (=.002, p<.10). Activities that relate to participants' work help to increase ratings of the activity's effectiveness in the areas of developing strategies and approaches for participants' work organizations and country needs. 4.12 We learn from this comparison that among Nigerians, a learning activity's relevance to Nigeria is the most important factor in making it effective. Thus, WBI should focus on making activities relevant in the larger context of Nigeria and not just for the participants' work tasks at hand. This suggests that Nigerian participants are engaged by topics related to their country's development and that they respond well to activities that concentrate on Nigeria. 4.13 Nigeria location, is a proxy measure of country focus based on data collected from the CRS rather than respondent survey data. Its statistically significant impact shows that activities that took place in Nigeria were rated as more effective overall (=.079, p<.10), and more effective in operational areas (=.115, p<.05), but not academic areas. In other words, country focus was important even when loosely defined as the activity being held in the home country of participants. Further, this result is impressive because it stems from a data source that is not based on participant responses. The lack of relationship between Nigeria location and academic effectiveness might be partly explained by the generic nature of learning content for teaching participants certain skills. For instance, teaching econometrics does not require a country focus in order for it to be effective. 4.14 Our second hypothesis explores the importance of WBI activity features in predicting effectiveness ratings. The number of days during which an activity took place (duration and duration2) was significantly related to WBI activity effectiveness overall (=.07 p<.05; = -.005 p<.05) and effectiveness in operations (=.115 p<.05; =-.008 p<.05). This suggests that increasing the length of the activity improves effectiveness ratings to a point, 7 days, afterwards there being a marginal rate of diminishing returns. In contrast to previous WBIEG findings, there were no other significant activity features in predicting effectiveness. There were no significant effects for the use of action plans in WBI activities nor for activity delivery method. It seems that country focus (relevance) trumps all other variables in predicting activity effectiveness. 4.15 Participant characteristics were also not highly predictive. Respondents' language abilities and technical terminology levels were not significant factors in ratings of WBI's 16 effectiveness. However, gender had a significant effect in predicting overall effectiveness. Women rated the activity as less effective overall (=-.07 p<.05), but not in the specialized cases of operational effectiveness or academic effectiveness. 4.16 Thus, we did not find much support for the third hypothesis that participant characteristics are related to WBI effectiveness. As mentioned previously, an earlier WBIEG program impact study (Khattri, Quizon et al., 2002) with global coverage noted that some individual-level factors were significant predictors of WBI's effectiveness and impact. In our study, which is based solely on Nigerian alumni, participant characteristics do not appear to explain their self-ratings of activity effectiveness, with the exception of female in one model. One possible explanation for these seemingly differing results is that the larger global sample of the earlier study was picking up country differences among participants that seemed to fall along lines of participant characteristics. For instance, the past study reported that participant proficiency in technical terminology was an important predictor of activity effectiveness. However, if technical proficiency is highly correlated with participant country, then the findings related to participant characteristics might be spuriously correlated. Indeed, this seemed to be the case in other individual focus country studies where participant characteristics were not predictive of effectives e.g. in Thailand (Quizon and Chard 2003). 4.17 Overall, our test of the country focus hypothesis shows that the relevance and applicability of the activity's content to the Nigerian context is the driving force in explaining an activity's effectiveness. It outweighs all other variables both in magnitude and statistical significance. These results underscore the importance of WBI's country focus approach in Nigeria. 17 5. USE OF KNOWLEDGE AND SKILLS 5.1 Previously, we explored whether WBI was effective in providing respondents with knowledge and skills. Here, we investigate whether Nigerian respondents were actually able to utilize these new concepts and methods. Do participants actually use what they learn? And if so, how frequently do they use their newly acquired knowledge and skills (K&S)? LEVELOF IMPACT 5.2 Average ratings of how often respondents apply their WBI-acquired K&S on a seven point scale from one (not at all) to seven (very often) ranged from a low of 4.16, for legislation and regulation, to a high of 5.27, for raising awareness of development issues. On average, respondents used the skills and knowledge learned in WBI activities "somewhat often." 5.3 Figure 3 illustrates the proportion of survey respondents who reported using their WBI-acquired K&S "often" or "very often." Most participants used WBI-acquired knowledge frequently to raise public awareness in development issues (57 percent). Nearly half used WBI-acquired skills in implementing new work practices (41 percent) and conducting research (41 percent). Fewer participants reported frequently using their K&S for organizing collective initiatives (32 percent) and teaching (29 percent). Less than a quarter used their WBI-acquired K&S in implementing country development strategies in Nigeria (17 percent), or influencing legislation and regulation (14 percent). Figure 3. Reported Use of Knowledge and Skills (Percent rating 6 or 7) 80% 57% 60% 44% 41% 41% 40% 32% 29% 26% 17% 17% 20% 14% 0% in at in Use Use Use action Public learning New work Research Collective initiatives Teaching Country dev'nt awareness strategies practices Overall Legislation regulation/ 18 5.4 On the contrary, participants in our focus group reported that they used what they learned in the activities, for instance, their improved research skills in writing policy memos and reports, to influence the government's agenda. This is supported by Bank staff interviews as well. Operational staff cited policy dialogues on poverty and vulnerability issues as evidence of "on the ground" impact from WBI. A MODEL OF IMPACT: USE OF ACQUIRED KNOWLEDGE AND SKILLS 5.5 To understand the factors that affect how frequently participants use their WBI- acquired K&S, we estimated several regression models relating participants' use to: (a) WBI activity effectiveness, (b) work and country factors that may act as facilitators or barriers to use (c) WBI follow up after the activity, and (d) participant characteristics. Equation 4.2 describes our OLS model of impact: Use of K&S123 = 0 + 1effectiveness + 2F&B work + 3F&B country + 4follow-up + 5org govt + 6job_academic + 7job_policy+ 98 job_management + 9 job_service + 10 positionlevel + 11discussed + (4.2) 5.6 The dependent variable, is a composite score of the frequency with which WBI- acquired K&S is used in various domains.16 We operationalize participants' K&S use as: a) overall use (summary score of use in all domains); b) academic use (raising awareness, teaching, and research); and c) operational use (organizing collective initiatives, creating development strategies for Nigeria, implementing new practices at work, and influencing legislation and/or regulation). 5.7 In this model, effectiveness is a summary scale of participants' perceptions of the degree to which activities are successful in increasing participants' knowledge of development issues, enhancing their research and teaching, improving their skills, strategies, and approaches for development, and broadening participants' network and building partnerships. 5.8 The facilitators and barriers (F&B) that enable or prohibit the utilization of knowledge, skills, and strategies learned in WBI activities at work and in one's country are captured in two variables, F&B work and F&B country. F&B Work is the helpfulness (unhelpfulness) score given by respondents for work factors, such as, work policies and procedures, resources and incentive systems, attitudes of supervisors and behaviors of colleagues that may enable or prohibit the use of WBI-acquired K&S. F&B country is the respondent's rating of country factors that can affect implementation of WBI- acquired ideas/awareness such as country policies, behaviors of social groups and local political groups and the state of Nigeria's "readiness" for reform and innovation. 5.9 WBI discussed is whether or not the issues raised in the activity were discussed at work, with local partners, government officials, NGOs or in the media. This score is 16The DV is recalibrated from an original seven-point scale to a zero-one range where zero means "not used at all" and one means "used very often". 19 based on a scale from one (never discussed) to seven (thoroughly discussed). We expect that the more WBI issues are discussed, the more likely they are applied. 5.10 WBI follow-up with participants is a measure based on whether participants were contacted by WBI for follow-up issues regarding the activity. We anticipated that participants who've been contacted by WBI might be more likely to make use of WBI acquired K&S. 5.11 Participant characteristics include a variety of factors including the type of job in which respondents work, the organization to which they are affiliated, and their position level. The area in which one works or the type of organization with whom one is affiliated may affect a participant's ability to utilize new ideas and implement change. For instance, governmental units with their rigid rules and regulations might be less free to implement changes compared with non-governmental organizations with more leeway to overhaul their policies (Quizon and Chard 2004). Likewise, it might be more or less challenging to implement change depending on one's seniority level. Past research showed that higher level participants were more likely to use what they learned (Quizon, Chard, and Lockheed 2004). 5.12 Table 4 shows the OLS estimates for the three regressions explaining use of K&S. All three models have strong explanatory power. The results for each model are displayed in three columns respectively for overall use (¯R2 =.45), academic use (¯R2 =.52) and operational use (¯R2 =.28). Table 4: Use of Knowledge and Skills as a Function of Country and Work factors, and Participant Characteristics DV= Use DV= Academic Use DV= Operational Use Variable Estimate Estimate Estimate Effectiveness .541*** .704*** .419*** F&B Work .136* .091 .170* F&B Country -.023 -.087 .025 Follow-up -.068 .084 -.056 Government organization -.002 -.058 .040 Academic job .047 .209* -.074 Policy job .063 .155 -.006 Management job .014 .107 -.056 Service job .025 .101 -.031 Position level .011 .029 -.002 Discussed .145* .235** .078 Constant .014 -.192 .167 N=70 N=70 N=70 ***=p<.01,**=p<.05, *=p<.10 R2=.45 R2=.52 R2=.28 20 5.13 Activity effectiveness is the strongest predictor of use of K&S: overall use (=.54, p<.001), academic use (=.70, p<.01) and operational use (=.42, p<.01). Academic use increases by as much as 70 percent when respondents rate activities as highly effective (likewise 42 percent for operational use, and 54 percent for academic use). In sum, if activity effectiveness is rated highly, reported use of WBI acquired K&S is more frequent. Thus, not surprisingly, we find that in order for an activity to be applied later on, it must first be effective. 5.14 Respondents' rating of work factors as facilitators (very helpful) versus obstacles (very unhelpful) increases use of K&S. F&B: Work has a significant effect on overall use (=.14, p<.10) and operational use (=.17, p<.10).17 Respondents' ratings of work factors as facilitators (very helpful) versus obstacles (very unhelpful) increases use of K&S by approximately 14-17 percent of the scale. If we compare the effect of F&B: Work with that of F&B: Country, it is clear that work factors are more important predictors of impact than country factors. Country factors do not significantly influence participant use of what they learn in WBI activities. These findings suggest that working individuals are strong catalysts for change when existing work factors favor change. 5.15 Participant characteristics are generally not significant predictors of use of K&S. The results show that participants' position level, organizational affiliation, and job type are usually unrelated to using WBI-acquired knowledge and skills. Only in the case of academic use was job type significant. Respondents who worked in academic jobs were significantly more likely to use what they learned in WBI for raising public awareness, conducting research and teaching (=.21, p<.10). The positive effect for working in academe is expected because academics tend to have a good degree of autonomy in their research agendas and teaching materials. In other words, it is not surprising that professors are able to apply the new concepts and skills they learned in activities to their work. 5.16 Discussing the subject of the activity at work, with local partners, or government officials, NGOs, increases participants' overall use of the WBI-acquired K&S (=.15, p<.10) and academic use (=.24, p<.05), but not operational use. This is logical if we consider that discussions are the key to frequent academic use. For example, lecturing on the topics learned from WBI are the means to teaching and raising public awareness. Likewise, one can imagine that discussing new skills or theories learned in training with colleagues or during brown bags could potentially inspire research or grant proposals in the subject area, in academic settings where professors are free to research topics at their discretion. 17However, work factors are not significantly related to academic use. This suggests that the work environment is not as influential a factor in people's ability to use their K&S to research, teach, and raise awareness. 21 6. CONCLUSIONS AND RECOMMENDATIONS 6.1 Overall, our models confirm the importance of effectiveness at the participant level. In other words, WBI activity effectiveness at the participant level is the first criteria for activity impact, when impact is defined as the application of lessons learned. In the Nigerian context, WBI can boost effectiveness by making its activities relevant for Nigeria. The importance of effectiveness in predicting use indicates that WBI activities in FY02-03 were successful in establishing the connection between issues learned in the WBI activity and their application in the workplace. 6.2 Work factors were also significantly related to participants' use of K&S. Work factors that are rated as facilitators (such as supervisors, colleagues, work policies and procedures) significantly increase participants' use of WBI-acquired K&S. On the other hand, country factors (such as Nigeria policies, political and social groups, and Nigeria's general mood or readiness for reform) are not related to use when work factors are taken into account. That is to say that work factors are more important than country factors in participants' abilities to use K&S. Positive relationships exist with the use of knowledge and skills when work factors were rated as facilitators (very helpful) rather than obstacles. In terms of participant factors that influence use of K&S, our results note that individuals who work in academics are more frequent users of K&S in academic areas: raising awareness, conducting research and teaching. This suggests that among Nigerians, academics are most easily able to use the K&S learned in WBI. RECOMMENDATIONS ABOUT EVALUATION METHODS 6.3 WBI-wide participant data collection efforts need to dramatically improve, particularly in the context of "country focus" where the goal is local capacity enhancement. A serious problem is that contact information is not available for over half of the WBI FY02-03 Nigerian alumni.18 Additionally, approximately 60 percent of the available contact information was no longer correct for our sample of 400. These conditions drastically reduce the possibility of reaching the majority of Nigerian WBI graduates for any follow-up activities (i.e. for additional courses, or for evaluations of the learning activity in which they participated). WBI's Client Registration System is established for these purposes, however it is only as useful as it is comprehensive, current and actually used. In other words, WBI task team leaders (TTLs) need to document and, for country-focused activities, regularly update participant information in order for the CRS to serve its purpose as a comprehensive database of WBI activities and alumni. Perhaps, these responsibilities might be linked to performance measures with clearly tied incentives/disincentives for TTLs. 18On average, participant names exist for only 47 percent of total participants entered into the CRS for participants from countries evaluated by IEG (Brazil, Burkina Faso, Egypt, Guatemala, Indonesia, Kenya Nigeria, Russia, Sri Lanka, Tajikistan, Thailand, and Yemen). 22 6.4 Moreover, it is important that activity leaders impress upon WBI alumni that part of their obligations for the activity is to cooperate in future evaluations of activity impact. In other words, monitoring and evaluation must be instituted at the activity level as part of the activity and responsibility of the participants. Of course, mentioning that WBI might follow up with an evaluation survey will not guarantee higher response rates. However, it might help to improve cooperation by the people who are able to be reached. 6.5 While the biggest challenge for interviewing respondents was tracking participants down, the second was obtaining completed questionnaires. Because of Nigeria's poor infrastructure, relying on telephone, fax and email were not plausible methods for collecting surveys. The survey methodology in Nigeria required going door- to-door as was the case for Burkina Faso and Yemen. Consequently, response rates are inherently lower in these countries because of the added obstacles of travel and timing. For instance, in Nigeria, we were not always able to meet someone twice, even if we were able to drop off the survey for the participant, they were not available when we tried to pick them up. Additionally, the large size of Nigeria was a constraint in that it required a bit of travel to obtain respondents in the second capital of Lagos. The traffic in Lagos made it very difficult for interviewers to hand deliver questionnaires and invitations to focus groups. RECOMMENDATIONS FOR ACTIVITIES 6.6 Survey responses indicate that activities in FY02 and 03 were perceived to be very relevant to Nigeria by half of respondents. Substantially more respondents, 67 percent, reported that the activity was relevant to their work. This difference is important because our model results show that Nigeria relevance is consistently a statistically significant indicator of overall activity effectiveness, academic effectiveness, and operational effectiveness. On the other hand, work relevance was only marginally significant in predicting operational effectiveness. Thus, we find that while activity relevance to Nigeria is the driving force in activity effectiveness, the survey results show that WBI activities are more relevant to participants' work. Our conclusions suggest that WBI's country focus approach in Nigeria is the right path for the future. Further, WBI activities should strive to present activity topics within the context of Nigeria. 6.7 Supporting this conclusion, effectiveness ratings are also influenced by the activity location, the event being held in Nigeria. In other words, activities that were held in Nigeria were rated more effectively than those that were not. This additional measure for country focus is a proxy that indirectly captures a dimension of the concept underlying "Nigeria-focused" activities. A better measure would be a survey of task managers where they could indicate if the activity was Nigeria-focused. We hope to explore this in our future evaluation work. However, because the time period under study is before WBI's country-focus approach was established, we would not expect the majority of activities to be designed especially for Nigerians. 6.8 The only other activity feature that influenced effectiveness was activity duration. The results showed that increasing the length of learning events up to seven days in 23 duration enhanced activity effectiveness. However, increasing days indefinitely would lead to marginal diminishing returns.19 6.9 A little more than half of respondents (57 percent) frequently used their WBI- acquired K&S in raising public awareness. Overall, mean ratings by respondents show that WBI K&S are used by participants "somewhat often." 6.10 Factors perceived as facilitating the use of WBI acquired K&S were primarily work-related (e.g. work policies, incentive systems at work, supervisors, colleagues). Work factors are significant predictors of overall use and operational use. This suggests that work factors play a major role in the ability of participants to apply the K&S from activities focusing on development of legislation, regulation, and work policies and procedures. 6.11 The magnitude of the effect for work factors is so large that it suppresses the relationship between country-related factors (e.g. country policies, social groups, political groups) and use. This is a surprising finding for Nigeria where country factors were expected to have a strong impact. The statistical reason for this is that work factors are so strong that they outweigh country factors in our models when included them together. In the presence of work factors, country-related factors have no significant effect on use.20 6.12 Alternatively, discussing WBI issues at work, with local partners, government officials, and NGOs led to increased academic use of K&S. Given that discussions are the heart of teaching and raising public awareness, it is logical that this should be highly significant for participant use in these activities. Thus, activities concentrating on teaching people to raise public awareness in development issues, reaching and research, should gear their discussions in a way to encourage follow up discussions about the topics in other settings. 6.13 Participant characteristics were unrelated to activity impact in Nigeria. For instance, affiliations with governmental organization or working in academics, management and administration, or policy and legislation did not influence use WBI K&S. Previously reported country studies have shown varying results for these characteristics. For instance, while government affiliation had a negative effect in Thailand, it had a positive effect in Brazil. Participant characteristic effects should be explored in future analyses with larger more diverse samples to reach firm conclusions for Nigerians. 6.14 The survey responses show disappointing results regarding follow-up to WBI activities. Less than a quarter of respondents reported some type of follow-up activity (21 percent). Although a majority of respondents received contact information for other participants (62 percent), only 24 percent reported any activity-related communications with participants. Both focus group discussions and interviews with country staff also highlight the importance of more follow-up with participants and for WBI to establish even more partnerships with local organizations for broader and deeper impacts. 19Activity length was six days on average, lasting a maximum of ten days. 20Estimating the equation without work factors shows significant effects for country factors. 24 However, follow up was not a significant predictor in our models of use. Most likely this will be addressed by the implementation of WBI's country focus approach where capacity building is the target of WBI's agenda. WBI should continue in its unique role in Nigeria 6.15 There is general agreement that WBI plays a unique role in Nigeria as a provider of learning opportunities. Respondents were not aware of comparable learning events offered by other organizations. Of the few who were familiar with equivalent activities, they noted that WBI was as good if not slightly more effective than its competitors. 6.16 Likewise, focus group participants reported that the content in WBI's activity was not redundant to other events offered by other organizations. The respondents who found WBI's activity useful also reported that WBI's offering was equally as good as these other learning activities theyattended. 6.17 Interviews with country Bank staff also indicated that WBI plays a unique role in Nigeria because of the role the Institute plays as a "global knowledge bank." WBI brings an important international perspective to its offerings. Staff suggested that WBI's expertise in a variety of development issues in so many countries is WBI's most important asset. Further, staff suggested that WBI could enhance Nigerian capacity by partnering with local institutions and conducting more "Training of trainer"activities. 6.18 Looking forward, WBI will play a key role in the Bank's future work in Nigeria. One needs to look no further than the Joint Interim Strategy where WBI's role in Nigeria is documented. WBI's RCET team for Nigeria sets an exemplary example for other focus countries in its close working relationship with the Nigeria country team. Moreover, Bank staff suggest that WBI establish a local representative in the Abuja office. 25 ANNEXES 26 ANNEX A: WBI ACTIVITIES FOR NIGERIAN PARTICIPANTS FY02-03 ACTIVITY TITLE START END Adapting to Change: Core Course on Population- Reproductive Health and Health Sector Reform 09/10/01 09/28/01 ATTACKING POVERTY PROGRAM (BUNDLE MASTER) 09/10/01 09/13/01 SECOND AFRICA FORUM ON POVERTY REDUCTION STRATEGIES 09/10/01 09/13/01 Second Africa Forum on Poverty Reduction Strategies 09/10/01 09/13/01 SECOND AFRICA FORUM ON POVERTY REDUCTION STRATEGIES 09/10/01 09/13/01 Public Pension Fund Management 09/24/01 09/26/01 Mainstreaming environment in Poverty Reduction Planning-Abuja 09/25/01 09/27/01 Pan African Capacity Building Forum 10/20/01 10/24/01 Flagship Core Course on Health Sector Reform & Sustainable Financing 10/29/01 11/16/01 WTO Course for Trade Policy Officials 11/06/01 11/07/01 Cours de Microfinance pour les non-spécialistes 11/19/01 12/05/01 CDD Implementation and Its Challenges 11/19/01 11/22/01 Developing a Spatial Poverty Map in Africa: Principles and Practice 11/26/01 11/30/01 Flagship Capacity Buidling in Anglophone Africa -- 1st Regional Course on PRSP 12/03/01 12/07/01 Africa Local Government Action Forum (Algaf) 12/07/01 12/07/01 AIDS Leadership: Human Capacity Development for Effective Responses to HIV/AIDS 12/14/01 12/14/01 Journalism: Covering Budgets and Government Spending 01/10/02 03/30/02 Anglophone Africa Investigative Journalism Core Course 01/15/02 03/19/02 Poverty Reduction Strategies Forum for Bank/Fund Staff. Challenges Ahead and Operational Implications 01/18/02 01/18/02 Interactive Internet Course for Investigative Journalists 01/28/02 02/24/02 Human Resources and Medical Education in Africa 01/29/02 01/31/02 AFRICA: Course on Water Sector Reform 02/11/02 02/15/02 Poverty Seminar For Policy Makers Africa 02/12/02 02/17/02 Corporate Governance 02/15/02 05/31/02 Global Dialogue on Africa Roundtable (NEPAD) - Ghana 02/28/02 02/28/02 Strategic Choices for Education Reform Global Core Course 03/04/02 03/22/02 Capital Flows - Monetary Policy and the International Financial System 03/04/02 03/07/02 Social Risk Management - DL 03/18/02 04/05/02 (Annex A continues on next page.) 27 (Annex continued.) ACTIVITY TITLE START END FUTURE LEADERS E-CONFERENCE ON CORPORATE SOCIAL RESPONSIBILITY 03/18/02 04/26/02 Decentralization and Community Empowerment: Sharing Lessons and Designing Actions 03/25/02 03/28/02 IF/PRSP capacity-building workshop for parliamentarians 03/25/02 03/26/02 Financial Markets and Development Conference: In Whom We Trust 04/17/02 04/19/02 Global Integration and the New Trade Agenda for Southern Africa 04/18/02 04/21/02 Quantitative Macroeconomic Analysis for Growth and Poverty Reduction Strategies 04/22/02 05/02/02 Social Risk Management Workshop - I 04/22/02 04/26/02 Africa Open and Transparent Municipal Governance Program: Enabling corruption free governance at the local level 04/23/02 06/25/02 Workshop on Improving Public Expenditure and Financial Accountability for Nigeria 04/24/02 04/26/02 HD - PRSP Training 04/29/02 05/03/02 Water Forum 2002 05/06/02 05/08/02 Third Annual Conference of the Parliamentary Network on the World Bank 05/09/02 05/11/02 Risk Management Workshop for Regulators 05/20/02 05/24/02 Rural Poverty Reduction Through Food Security and Agricultural Growth in Francophone Africa 05/23/02 06/27/02 Fighting the HIV/AIDS Pandemic through Information and Strategic Communication. 06/06/02 06/27/02 Ghana: Lectures in Economics 06/06/02 06/06/02 Enhancing Good Governance by Curbing Corruption - the Role of Parliaments 06/10/02 06/13/02 Americas E-Forum on Corporate Social Responsibility 06/10/02 07/05/02 The Americas E-Conference on Corporate Social Responsibility 06/10/02 07/05/02 Decentralization and Community Empowerment: Training of Trainers Workshop 06/11/02 05/14/02 Community Based Monitoring & Evaluation in Nigeria - WB & Food Basket Foundation Intl 06/11/02 06/14/02 Market Creation for Biodiversity 06/25/02 06/27/02 Adolescent Health and Development in Africa 06/26/02 06/26/02 Adolescent Health and Development in Africa 06/26/02 06/26/02 AIDS Leadership: Communication Strategy 06/27/02 11/21/02 Basic Poverty Measurement and Diagnostics Course 07/02/02 07/10/02 SP/SRM - DL - QER - Anglophone Africa 07/02/02 07/17/02 (Annex A continues on next page.) 28 (Annex continued.) ACTIVITY TITLE START END Integrated Water Resources Management 07/08/02 07/12/02 An Introduction to CSR for Future Leaders (Web-Based Course) 07/11/02 07/24/02 Introduction to Corporate Social Responsibility for Future Leaders 07/11/02 07/23/02 Environmental Economics and Development Policy Core Course-V 07/15/02 07/26/02 Global Judges Symposium on Sustainable Development and the Role of Law 08/18/02 08/20/02 Adapting to Change: Core Course on Population- Reproductive Health and Health Sector Reform 08/19/02 08/30/02 CDD Training of Trainers Workshop-Kampala- Uganda 08/25/02 08/31/02 Accountability in Business and Government 08/27/02 08/27/02 WSSD/Water Dome Press Briefings 08/28/02 09/03/02 Business Ethics and Corporate Accountability: The Search for Standards 09/09/02 09/27/02 CPA Annual Conference : Parliament- Governance + the PRSP 09/10/02 09/12/02 Global Dialogue on Combating Money Laundering and Terrorist Financing - Anglophone Africa Region 09/10/02 09/10/02 Evidence-Based Governance in the Electronic Age 09/16/02 10/19/02 Journalism: Diversity and Tolerance - Nigeria 09/23/02 09/27/02 Introduction to Corporate Social Responsibility for Staff of Multilateral Organizations 09/26/02 10/20/02 Introduction to Corporate Social Responsibility for NGOs 09/26/02 10/20/02 Berlin Summer School on PPI: Transport- Water- Energy 10/06/02 10/13/02 Global E-Conference on Business- Democracy- and Peace 10/07/02 11/01/02 Global Organization of Parliamentarians Against Corruption 10/13/02 10/16/02 African Economics Editors Forum 10/27/02 10/30/02 Community Driven Development Course 10/31/02 11/22/02 Niger Delta Insitutional Development Project 11/04/02 11/08/02 Comparing rivers: the Mississippi and the Niger 11/07/02 11/08/02 HD/SP PRSP - Africa 11/11/02 11/15/02 Flagship Capacity Buidling in Anglophone Africa -- HD PRSP 11/11/02 11/15/02 Internet Core Course on Investigative Journalism 11/11/02 12/20/02 Global Dialogue on Access and Connectivity (Anglophone) 11/11/02 11/11/02 HD/EDU PRSP - Africa 11/11/02 11/15/02 HIV/AIDS and Strategic Communication in Nigeria 11/12/02 11/15/02 Poverty-Natural Resource Linkages in Nigeria 11/12/02 11/12/02 WTO2000 Program Management and Handbook 11/13/02 11/16/02 Business- Democracy and Peace Web-Based Course 11/13/02 12/20/02 (Annex A continues on next page.) 29 (Annex continued.) ACTIVITY TITLE START END Introduction to Corporate Social Responsibility - 10 11/15/02 11/29/02 Global E-Conference on CSR- Education and Technology: How Can Corporations Contribute to Bridging Educational and Digital Divides? 11/18/02 12/07/02 Protecting the Vulnerable: The Design and Implementation of Effective Safety Nets 12/02/02 12/13/02 Parliamentary Staff Training - Africa -CPA- 12/02/02 12/07/02 New Technologies for Small and Medium -Size Enterprise Finance 12/04/02 12/06/02 Glocal Forum Think Tank Meeting 12/10/02 12/10/02 National Governance and Anti-Corruption 01/08/03 06/27/03 Flagship Core Course on Health Sector Reform and Sustainable Financing 01/13/03 01/31/03 Introduction to Corporate Social Responsibility - 11 01/15/03 03/13/03 Global Dialogue on ICT Education: Policy and Outcomes (Anglophone) 01/16/03 01/16/03 National Governance and Anti-Corruption 01/20/03 01/22/03 Social Protection in Nigeria - Part 1 01/22/03 01/22/03 On the Road to Quality Education for All 01/27/03 02/07/03 A Regional Learning Workshop on Preventing and Combating Corruption in Africa 01/27/03 01/31/03 Community Based Monitoring and Evaluation - Training of Trainers 1 01/28/03 01/31/03 Population and Reproductive Health Curriculum Networking Workshop: Anglophone Africa 01/30/03 02/01/03 Training of Trainers - Basic Poverty Course- Anglo Africa 02/03/03 02/14/03 Participatory Monitoring and Evaluation 02/14/03 02/16/03 Media + Governance -CJA-WB Conference- 02/17/03 02/22/03 Pension Reform Action Plan - Africa 1: South Africa 02/24/03 02/28/03 WBI/UNDP Pilot Initiative for Community Capacity Building in Ethiopia- Rwanda- and Sudan 02/26/03 02/28/03 Labor Market Policies Core Course 03/03/03 03/14/03 Health Outcomes and the Poor 03/03/03 04/11/03 Youth for Good Governance--Phase Two 03/03/03 06/30/03 Investigative Journalism - Anglophone Africa 03/03/03 05/31/03 Housing Finance in Emerging Markets: Policy and Regulatory Challenges 03/10/03 03/14/03 Introduction to Corporate Social Responsibility and Sustainable Competitiveness 03/12/03 03/26/03 Regional Sem for Managers of M E 03/19/03 03/21/03 Nigerian Media Workshops: Covering Elections 03/24/03 03/25/03 Youth for Good Governance--Phase Three 04/01/03 06/30/03 (Annex A continues on next page.) 30 (Annex continued.) ACTIVITY TITLE START END Introduction to Corporate Social Responsibility and Sustainable Competitiveness 04/02/03 04/24/03 Internet-based Macroeconomic Management: Analytical Foundations and Case Studies 04/14/03 06/09/03 Annual Financial Markets and Development Conference - Brookings Institute- International Monetary Fund- and World Bank 04/14/03 04/16/03 Introduction to Corporate Social Responsibility - Nigeria 04/14/03 05/06/03 African course on trade and WTO 04/17/03 06/23/03 Discussion Forum on Pension Reforms for Africa 04/18/03 06/20/03 CC Distance Learning Intergovernmental Fiscal Relations and Local Financial Management/Central European University 04/20/03 06/28/03 Global E-Conference on The Role Of Business In Fighting HIV/AIDS 04/21/03 05/09/03 Risk Management Workshop for Regulators: Assessing- Managing- and Supervising Financial Risk 04/27/03 05/02/03 Botswana Conference on ICT 04/28/03 05/02/03 Introduction to Corporate Social Responsibility and Sustainable Competitiveness 05/01/03 05/22/03 Evidence-Based Governance in the Electronic Age--Phase Three 05/03/03 06/01/03 Policy and Institutional Reform for Sustainable Rural Development 05/05/03 05/07/03 Public Pension Fund Management Conference - World Bank 05/05/03 05/07/03 Regional Workshop on Effective Use of Auditing Resources 05/12/03 05/15/03 Global Dialogue on ICT & Education: Professional Development - Anglophone- 05/19/03 05/19/03 Social Protection in Nigeria - Part 2 05/20/03 05/21/03 Improving Internal Auditing in Nigeria 06/02/03 06/05/03 Funding Natural Disasters: A New Perspective on Country Risk Management 06/02/03 06/03/03 Financing Health Services for the Poor in Africa 06/02/03 06/06/03 SEIA Secondary Education Seminar 06/09/03 06/13/03 A Regional Workshop on Challenges to Implementing Governance Reforms in Africa 06/09/03 06/11/03 Advanced Impact Evaluation Course 06/09/03 06/12/03 Workshop on Improving External Auditing in Nigeria 06/09/03 06/11/03 Government Bond Market Development Workshop in Africa 06/17/03 06/19/03 IWRM - Training of Trainers with UNESCO-CapNet 06/23/03 06/28/03 Investigative Journalism in Nigeria 06/23/03 06/27/03 Strengthening the Linkages between Reproductive Health and HIV/AIDS 06/25/03 06/25/03 31 32 ANNEX B: WORLD BANK INSTITUTE (WBI) COUNTRY FOCUS EVALUATION QUESTIONNAIRE NIGERIA Instructions Thank you for agreeing to complete this questionnaire. The purpose of this questionnaire is to obtain your opinions on the performance of WBI learning activities in your country. We are going to ask you to think about the WBI learning activity you attended between the summer of 2000 and the summer of 2002. During this time period, if you participated in more than one WBI activity we would like for you to answer the following questions with the most recent activity in mind. According to our records the most recent WBI activity you participated in was entitled ______________________________________________________________________ and was held from _____________ to _______________. The questionnaire has four sections and should take approximately 15 to 20 minutes to complete. · Section 1 asks about the usefulness of the learning activity · Section 2 asks about the relevance of the learning activity to the specific needs of your country · Section 3 asks you to compare WBI learning activities with similar ones offered by other organizations · Section 4 asks about the characteristics of the learning event and your background. We would like to emphasize that your views are critical to us in our goal of continually improving WBI's learning activities. We need your honest feedback in order to get a clear picture of the effectiveness of the activity you attended. Please keep in mind that your responses will never be associated with you individually and that your participation in this survey will be kept confidential. If you have any questions or concerns about the questionnaire, please send us an e- mail to cchard@worldbank.org, or send us a fax to 202 522 1655, or give us a call at 202 473 2310. We will respond to you promptly. 33 WORLD BANK INSTITUTE (WBI) EVALUATION Participant Questionnaire World Bank Institute (WBI) Country Focus Evaluation Questionnaire Questionnaire to be completed by former participants Nigeria Instructions WBI had the pleasure to have you participate in the following learning activity: Title: _____________________________________________________________ Held from: ________________________ to: ________________________ In: _______________________________________________________________ Getting your opinion of the above-mentioned activity--now that you have had time to reflect on it--is very important to help WBI improve its programs. For this, we ask you to complete this questionnaire. The questionnaire has four sections and should take approximately 20 minutes to complete. · Section 1 asks about the relevance of the activity. · Section 2 asks about the usefulness of the activity. · Section 3 asks you to compare this activity with similar learning activities offered by other organizations. · Section 4 asks about the characteristics of the activity, its follow-up and your background. We need your honest feedback. Please keep in mind that your responses will be kept confidential, and will be used for the sole purpose of improving WBI programs. If you have any questions about the questionnaire please call or send a message to _____________________________________________________________. Thank you for agreeing to complete this questionnaire! 34 I. Relevance of the Activity The activity that you are asked to evaluate is mentioned on the first page of this questionnaire. 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 3. Was the activity designed specifically for participants from your country? Don't s Yes s No s know 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Don't s Yes s No s know b. Achieve universal primary education Don't s Yes s No s know c. Promote gender equality and empower women Don't s Yes s No s know d. Reduce child mortality Don't s Yes s No s know e. Improve maternal health Don't s Yes s No s know f. Combat HIV/AIDS, malaria, and other diseases Don't s Yes s No s know g. Ensure environmental sustainability Don't s Yes s No s know h. Develop global partnerships for development Don't s Yes s No s know i. Ensure water sanitation and supply Don't s Yes s No s know j. Improve investment climate and finance Don't s Yes s No s know k. Promote trade Don't s Yes s No s know 35 II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not effective at Extremely Not Areas all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country s s s s s s s s b. Providing you with knowledge or skills s s s s s s s s c. Helping you better understand your role as an agent of change in your country's development s s s s s s s s d. Helping you develop strategies or approaches to address the needs of your organization s s s s s s s s e. Helping you develop strategies or approaches to address the needs of your country s s s s s s s s f. Helping you develop contacts, develop partnerships and build coalitions in the field s s s s s s s s 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong Strong negative positive Don't change change know 1 2 3 4 5 6 7 DK s s s s s s s s 7. How often have you used the knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Not Very Not Purposes at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research s s s s s s s s b. Teaching s s s s s s s s c. Raising public awareness in development issues s s s s s s s s d. Implementing new practices within your work organization s s s s s s s s e. Organizing collective initiatives s s s s s s s s f. Influencing legislation and regulation s s s s s s s s g. Implementing country development strategies s s s s s s s s 36 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Neither Greatly helped Greatly Not Factors hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, etc.) s s s s s s s s b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) s s s s s s s s 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research s s s s s s s s b. Teaching s s s s s s s s c. Public awareness in development issues s s s s s s s s d. New practices within your work organization s s s s s s s s e. Collective initiatives s s s s s s s s f. Legislation and regulation s s s s s s s s g. Country development strategies s s s s s s s s 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 s s s s s s s 37 III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) s Yes s No 12. If yes, please provide the name(s) of the organization(s): 1. .................................................................................................................................................... 2. .................................................................................................................................................... 3. .................................................................................................................................................... 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI WBI much less About the same much more No useful useful opinion 1 2 3 4 5 6 7 s s s s s s s s IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Video Sessions Class room Mix of Video and Web-based (Distance Learning) (Face to Face) Face to Face Conference Learning Study tour 1 2 3 4 5 6 s s s s s s 15. How effective was this type of learning activity in helping you learn? Not effective Extremely No at all effective opinion 1 2 3 4 5 6 7 s s s s s s s s 38 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) s Yes s No 17. If yes, did you use part or all of the action plan in your work? s Yes s No 18. Were you provided with the contact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) s Yes s No 19. If yes, how did you use it? Used it to Used it to Other uses Never used it continue activity related organize joint (Please specify briefly) discussions follow-up activities _______________ s s s s ______________ 20. Was the language of instruction used during the activity the same language you use at work? s Yes s No 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient Highly at all proficient 1 2 3 4 5 6 7 s s s s s s s 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient Highly at all proficient 1 2 3 4 5 6 7 s s s s s s s 23. After the activity, did WBI contact you for follow-up issues regarding the activity? s Yes s No 39 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) s Yes s No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI WBI I did not responded, responded have but was not and was follow-up WBI did not helpful extremely requests respond at all helpful for WBI 0 1 2 3 4 5 6 7 NA s s s s s s s s s 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) s University/research institution s National/central government s Non-governmental organization (not-for- s profit) Provincial/regional government s Media s Local/municipal government s Private sector s Other, specify: ____________________________ 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) s Research s Teaching s Policymaking/legislation s Provision of services (e.g., financial, health, etc) s Management/administration s Other, specify: ________________________ 28. How would you best describe the level of the position you have held the longest since the activity? s Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) s Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) s Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) s Junior Level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) s Entry level (e.g., Intern, assistant) s Other, Please specify: ________________________________________________________ 29. What is your gender? s Male s Female Thank you for your feedback. We very much appreciate your cooperation. 40 ANNEX C: NIGERIA WBI PROGRAM SURVEY FY02-03 This questionnaire asks about the impact of WBI program activities that took place in FY02 and FY03. Please feel free to review the list of activities and programs to remind you of the various activities offered by WBI between July 2002 and June 2003 . Thank you in advance for your cooperation! Name : ________________________________________________________________ Area of expertise: ___________________________Date:________________________ Generally speaking, are you familiar with WBI's activities in FY02-03? (Feel free to review the list of 145 activities to help jog your memory.) Yes _____ No ____ 1. We'd like to ask you about your involvement in WBI's training activities over the past two fiscal years, FY02-03 . · Were you involved in WBI activities at that time? Yes_____ No____ If you answered No, please skip to question #3 . If yes, what was the nature of your involvement in WBI activities? (e.g., development of course material, identifying participants, providing instructors, providing funding for participants etc.)_____________________________________________________________________ ________________________________________________________________________ 2. How would you rate the extent of your involvement in WBI activities on a scale from 1(low) to 10 (high)? Please circle. Low ß à High 1 2 3 4 5 6 7 8 9 10 3. We'd like to ask you about the results, if any, from WBI's training activities over the past two fiscal years, FY02-03. · Can you think of any outcomes that resulted from these WBI learning events? Please describe._______________________________________________________ _____________________________________________________________________ · If yes, what sort of evidence "on the ground" have you seen from these outcomes or results? Please describe._____________________________________________ ______________________________________________________________________ 4. Over the past two years, how would you rate the effectiveness of WBI learning activities in achieving the MDGs in Nigeria? Looking at the table below, please mark an "x" in the box next to each goal that is related to any of the WBI activities you are familiar with. Next, rate the extent to which the goal is addressed by the WBI activity in achieving that goal, by filling in the appropriate bubble (labeled low to high). Mark an "X" here if WBI Activities that If you marked "X," please rate the extent to which WBI addressed the you know about was related to the key issues in achieving that goal goal Goals for Low 2 3 4 5 6 High N/A Development Mark with an "X' 41 a. Eradicate extreme poverty s s s s s s s s b. Achieve universal primary education s s s s s s s s c. Promote gender equality and s s s s s s s s empower women d. Reduce child mortality s s s s s s s s e. Improve maternal health s s s s s s s s f. Combat HIV/AIDS, malaria, s s s s s s s s and other diseases g. Ensure environmental s s s s s s s s sustainability h. Develop global partnerships for s s s s s s s s development i. Ensure water sanitation and s s s s s s s s supply j. Improve investment climate s s s s s s s s and finance k. Promote trade s s s s s s s s 42 5. Overall, to what degree do you think WBI activities are aligned with the Country Assistance Strategy (CAS)? If you were to rate the degree of alignment on a scale from 1 to 10, (where 1 is no alignment and 10 is complete alignment), how would you rate it? No ß à Complete Alignment Alignment 1 2 3 4 5 6 7 8 9 10 6. What would you say are the three main barriers preventing participants from applying what they learn toward making change in Nigeria? 1. ________________________________________________________ 2. _______________________________________________________ 3. ________________________________________________________ 7. In your opinion was WBI uniquely positioned to deliver learning activities for capacity building purposes in Nigeria? Yes_____ No____ · If yes, what single reason do you think makes WBI uniquely positioned to deliver learning activities for capacity building in Nigeria? ______________________________________________________ · In your opinion, what single thing does WBI bring to Nigeria that other organizations do not? _______________________________________________________ 8. From your perspective, as the operational counterpart to WBI activities, do you think there are things that should have been done differently by WBI to better serve the needs of Nigeria? Yes_____ No_____ If you answered No, please skip to question #8 . If you answered "Yes", list up to three things that could have been done differently: 1_____________________________________________________ 2_____________________________________________________ 3_____________________________________________________ · Have you communicated any such "lessons learned" to the WBI task manager(s) in charge? Yes_____ No_____ 9. Taking a look forward, please list three ways that you think WBI could assist Nigeria in moving closer to its development goals? 1__________________________________________________________ 2______________________________________________________ 3______________________________________________________ 43 ANNEX D: INTERVIEW AND FOCUS GROUP QUESTIONS FOR COUNTRY-FOCUSED IMPACT EVALUATION 1. What did you gain from the activity? 2. Was this activity part of a larger program? 3. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Probe 1: For what reasons? Please describe in detail. 4. Can you offer specific examples of how you applied what you acquired in the activity? Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? Probe 2: What were these changes? 5. In your view, how has the activity benefited your country? Probe 1: In what way? Please provide examples. 6. When you used the information that you acquired from the activity, what were some of the obstacles that you encountered? Probe 1: Please explain and give specific examples. Probe 2: In what ways were these obstacles? 7. Since attending the activity, have you attended other courses addressing the same topics? Probe 1: What were they? Probe 2: How useful were they? 8. Were the knowledge, skills or new ways of thinking that you obtained in the activity available from another source (e.g., institution, self-study materials, Internet site, etc.)? Probe 1: If yes, what were these sources and what were the names of the activities? Probe 2: How useful were they? 9. How can we improve similar WBI activities in the future? Probe 1: Please give specific examples. Probe 2: In particular, what types of courses would you like to see offered? 10. Is there any other assistance that WBI could provide in future activities to meet your specific professional needs? Probe 1: If so, what would it be? 44 ANNEX E: OPERATIONAL STAFF SURVEY We interviewed and sought comments from World Bank (WB) country staff members, who worked in Nigeria in FY02-03, about WBI's activities offered to Nigerian participants during this period. Our overall purpose was to learn about the impacts of WBI from the Bank staff who observe and experience first hand the consequences of WBI's learning offerings. To that end, we received eleven completed questionnaires from twenty eight staff members contacted. Staff Familiarity and Involvement with WBI Activities Most WB staff were not familiar with WBI's learning activities for Nigerian participants in FY02-03. Sixty-one percent of the staff contacted were not familiar with WBI activities (29 percent), or did not fill out the questionnaire because they were unaware of WBI's work (32 percent) during the time period. Thirty-nine percent reported that they were familiar with WBI's learning activities. Figure 4. Familiarity with WBI Activities 100% 82.0% 80% 60% YES NO 39.3% 40% No Reply 32.1% 28.6% 18.0% 20% 0% % Familiar with WBI activity of staff % Involved in WBI activities that responded The majority of country staff (82 percent) we interviewed were involved with WBI activities in various methods. Staff involvement in WBI events included: identifying participants (56 percent); funding participants (33 percent); actually being a participant (22 percent); acting as a facilitator (22 percent); developing course material (11 percent); planning workshops (22 percent); and identifying training needs (11 percent). However, 45 the extent of country staff involvement averaged a 5.5 on a scale from one (low) to ten (high), indicating that WB resident staff are not extensively involved with WBI events.21 How do staff assess the results of WBI's training activities? We asked staff to describe any concrete "on the ground" evidence of impact from WBI activities in Nigeria. Sixty-four percent reported that they observed actual outcomes from WBI's activities including: 1) increases in investigative reporting and more balanced news coverage; 2) auditing workshops organized by the federation of auditors from all states, and events arranged by the accountant general's office; 3) increases in transparency and accountability in project outcomes; 4) use of CDD principles by clients; participatory planning in PRSP process; increased policy dialogue; 5) proposed implementation of the sustainable development action program by the Niger Basin Authority and the Basin nine riparian countries; and 6) preparation of the national water resources policy by the Federal Ministry of Water, as well as reform of national river basin authorities in Nigeria. The remaining 36 percent who were not able to give any examples of "on the ground" impact (36 percent) suggested that it was too soon to see the impacts of WBI's efforts, e.g. "results so far are only on paper," "projects involved have not yet become effective, so no results yet." What are the obstacles to WBI impact? We asked staff to define the factors that may inhibit WBI participants from applying what they learn in training. Staff named a variety of country related barriers preventing participant implementation of WBI-acquired K&S including: cultural factors (i.e. strict hierarchical system); the political climate (e.g. one staff member stated that "political will does not exist in the country for participants to be able to implement."); government bureaucracy; institutional barriers (i.e. non-merit based staff transfers); infrastructure problems (e.g. sporadic Internet and email access), and a lack of resources and necessary tools (e.g. finance, books, reference material). Additionally, staff cited WBI-related factors that contribute to a lack of participant use of WBI-acquired K&S such as participant targeting and lack of follow up. In other words, they cited issues over which WBI has control. For example, regarding participant selection, one staff member stated that a four week seminar can only attract lower level staff who can afford the time off, thereby reducing the chance of implementation. Other staff mentioned that participants are not selected based on merit. One contributing factor to this is that the criteria for participants is not clear and therefore, the wrong person is sent to training. On the second issue, Bank staff mentioned that WBI does not provide enough follow up training for participants. They explain that participants could benefit from additional learning events to help them implement what they learn. 21Responses ranged from three to eight. 46 Is WBI unique as a learning provider? As illustrated in Figure 5, most of the country staff indicated that WBI is a unique training provider in Nigeria (sixty-four percent). The primary reason given for WBI's crucial role in Nigeria is that WBI brings international perspective and expertise into the picture. WBI is described as "the global knowledge bank." In fact, many staff cited participant knowledge of WBI's wealth of information as a notable outcome from alumni's participation in WBI's activities. More importantly for WBI's country focus approach, staff cited WBI's ability to customize and tailor training to the needs of individual countries, and Nigeria in particular as the single most important feature that makes WBI a unique learning provider. Additionally, staff cited WBI's ability to attract participants who might not otherwise attend local events because WBI's name holds a high level of prestige. Finally, Bank staff mentioned WBI's ability to develop partnerships with national training institutions. These partnerships are critical for capacity building of Nigeria. Figure 5. WBI as a Capacity Building Institution 100% 80% 77.8% 63.6% 60% 50.0% 50.0% Yes No 40% 36.4% 22.2% 20% 0% % Uniquely positioned to % WBI should have done % communicated "lessons deliver capacity building things differently to better learned" to WBI staff activities serve Nigeria What can WBI do differently to better serve Nigeria's needs? The majority of staff we interviewed (78 percent) suggested numerous areas where WBI could improve. Half of staff reported that they communicated these issues to WBI. Overall, WBI could be more effective if it worked together with WB country staff, and worked locally with the country staff team. The primary recommendation was that WBI establish a staff member in the Abuja office. 47 Additionally, WBI should better target the participant lists for their activities including government employees at both federal and state levels. WBI should consult with the local office when they develop their participants lists. Secondly, WBI's partnerships with local institutions should be continued and increased. Local institutional partners could help WBI build capacity locally and regionally. Related to that, WBI should do more "training of trainers" of local facilitators. Specific recommendations include: 1a) greater participation by WBI in country team meetings (for instance, the country director was kind enough to invite WBI's Nigeria RCET team to the country retreat both in FY03 and FY04), and 1b) more cross-support from WBI to country projects; tailor made training (i.e. activities designed for Nigeria); 2) regional training programs offered as a follow up to learning activities held in HQ; 3) outsourcing of training to local institutions (i.e. partnering with local organizations); 4) more "training of trainers" e.g. in the health and education sector learning and capacity building could continue after WBI; 5) well-defined long term training agendas such as documented in the joint interim strategy; and 6) participant selection from the federal and state government. Do WBI activities address the Millennium Development Goals (MDGs)? We asked staff to identify which of the MDGs WBI addressed with its Nigerian activities and to rate WBI's effectiveness in addressing these MDGs on a scale from one (low) to seven (high). Figure 6 illustrates the proportion of staff who said WBI addressed the MDG and the average score, indicating the extent to which WBI addressed the issue. Figure 6. MDGs Allignment and Extent to which WBI has addresses it 100.0% 4.20 6.00 4.71 4.83 4.60 5.00 80.0% 4.30 4.33 4.00 5.00 3.75 4.00 4.25 4.00 Alligned 60.0% 3.00 Average 40.0% 2.00 Percent 20.0% 1.00 0.0% 0.00 Addressed for Trade Health Poverty Equality Education Diseases Climate Mortality al Sanitation Development Sustainability Promote Child and Extreme Gender Partnerships Matern Primary Combat Investment Reduce Supply Improve Eradicate Universal Environmental Improve Water MDGs YES Average 48 According to staff, the MDGs addressed by WBI are: eradicating poverty (80 percent); promoting gender equality and empowering women (70 percent); partnerships for According to staff, the MDGs addressed by WBI are: eradicating poverty (80 percent); promoting gender equality and empowering women (70 percent); partnerships for development (70 percent); universal primary education (70 percent); combat disease (60 percent); environmental sustainability (60 percent); investment climate (50); promote trade (50); reduce child mortality (40 percent); improve maternal health (40 percent); and water supply and sanitation (40 percent). However, the extent to which WBI addressed the key issues in obtaining the goal was not very high. Average scores ranged from below average (3.75 for reducing child mortality) to above average (5.0 for improving investment climate). Most MDGs were rated around four, indicating that more in-depth treatment of MDGs is required. Were WBI activities in FY02-03 aligned with Nigeria's CAS? Resident staff rated the alignment between FY02-03 activities and the CAS as only slightly aligned. The average score for alignment was 6.6 on a scale where ten equals "completely aligned" and one equals "not aligned at all." It is important to point out that these ratings of alignment are based on past activities before WBI's country focus approach was implemented. Consequently, we would not expect higher ratings of alignment ­ the average score for Nigeria falls in line with baselines for other focus countries. How can WBI help Nigeria in the Future? Nigeria stands out as a model for other focus countries evaluated by WBI because of the close working relationship between the country office and WBI. The plan for future WBI work in Nigeria is laid out in detail in Nigeria's Joint Interim Strategy. The specifics are summarized in the introduction section of this report. 49 ANNEX F: FOCUS GROUP DISCUSSION The focus group discussion was held in Ibadan and included five WBI alumni who participated in multiple programs: Attacking Poverty, Social Protection, and Environment. The results of the focus group centered around four major areas: relevance of the activity; usefulness of the activity; participation in other training programs; and follow up. Activity Relevance, Effectiveness, and Use Across programs, focus group members overwhelmingly believed the activities they attended were relevant to their work and to Nigeria. They reported that the activities were directly associated with their work as social science researchers. Further, the topics of the seminars were pertinent to Nigeria's development needs. Participants of the Environmental Economics program reported that the course was designed specifically for Nigeria, while the rest of the programs focused regionally on Africa, as a continent. FGD respondents rated the activities most strongly in helping to raise their awareness of development issues and in building their knowledge and skills. This is particularly important in the context of Nigeria where the libraries are not maintained with the most current research and information. Participants stressed the importance of this given the lack of Internet facilities in the country. Participants perceived themselves to be "agents of change" through their abilities to set the agenda of the government based on their research and policy reports. They reported that the activities they attended helped them tremendously in their research and in writing policy memorandums to governments on critical national issues. Specifically, participants improved their research skills in the areas of: conducting empirical research, model building, and in measuring poverty. Infrastructure problems are a major barrier to use. In other words, participants explained that while WBI learnings were helpful in enhancing their skills the socio-economic climate often times prevents them from being able to fully utilize their new K&S. For instance, power failures are a source of disruption to people's work on the regular basis. Participation in Other Trainers Organization All focus group respondents reported having engaged in training programs with other agencies ­ both within Nigeria and in other countries. However, they did not feel comfortable comparing these agencies with WBI , saying that "the areas of focus are different." The organizations they cited as offering similar learning activities included: the African Economic Research Consortium; the British Council and the Department for International Development (DFID); the International Labor Organization; the Central Bank of Nigeria; and the National Planning Commission. 50 Characteristics of WBI Activities: Method of Course Delivery, Language, Class size, and Follow-up Participants engaged in both face to face activities and distance learning courses. However, participants did not perceive any differences between the effectiveness of the activity due to delivery method. Rather than difficulties with video-conferencing or technology (which was often cited in other countries), the participants cited the main barrier to effectiveness as the lack of language abilities of the facilitator. For instance, the resource person spoke French rather than English. Participants stressed that Anglophone resource people should be used for English-speaking audiences. Participants also cited small class sizes as important to their learning experience. Participants suggested that activities with smaller classes were more effective regardless of the delivery method. WBI provided detailed contact information for participants. However, no one in the FGD reported actually using it to contact anyone. The only follow-up that participants could recall was a WBI email conferencing facility named "PAC TALK." Based on previous comments about Nigeria's electric infrastructure, email as the primary means for follow- up might adversely affect participant continual use. 51 ANNEX I: MILLENNIUM DEVELOPMENT GOALS (MDG) What MDGs did WBI's FY02-03 activities directly address according to participants? According to Nigerian survey respondents, WBI activities primarily addressed the MDG of eradicating extreme poverty. A large majority of respondents (78 percent) reported that the activity they attended was relevant to this overarching development objective. A little more than half of respondents (57 percent) cited global partnerships for development as related to the activity. Nearly half (49 percent) identified combating HIV/Aids, malaria and other diseases as having been addressed by WBI.22 Figure 7. Respondents Reporting that Activity Addressed Millenium Development Goals (MDGs) eradicate extreme poverty 90 78 develop global partnerships for 80 development combat HIV/AIDS, malaria, and other 70 diseases improve maternal health 60 57 49 achieve universal primary education 50 44 41 41 39 40 37 36 ensure environmental sustainability Respondents 29 of 30 27 improve investment climate and finance % 20 reduce child mortality 10 promote trade 0 ensure water sanitation and supply MDG promote gender equality and empower women 22Differences in ratings of MDGs between participants and country staff reflect the differences in the activities which respondents assessed. WBI participants rated their experience in the course, and country staff were asked to rate all the activities in which Nigerians participated during FY02-03. 52 Annex H: Descriptive Statistics Variable Mean Standard Deviation Nigeria relevance 5.37 .205 Nigeria work relevance 5.77 .183 Nigeria location .42 .095 Action plan .50 .086 Distance Learning .19 .39 Activity Duration 5.6 2.56 Technical terminology 5.76 .139 Female .31 .47 Effectiveness Awareness 5.97 .174 Knowledge & Skills 5.84 .179 Awareness of role as agent of change 5.73 .181 Organization Needs 5.13 .267 Helping Country 5.05 .262 Develop Contacts 5.07 .208 F&B Work 4.69 .302 F&B Country 4.46 .286 Follow-up .21 .064 Use Conducting Research 5.22 .24 Teaching 4.76 .266 Raising Awareness 5.27 .29 Implementing new practices 4.94 .26 Organizing collective initiatives 4.77 .256 Influencing Legislation 4.16 .198 Implementing country development strategies 4.49 .204 Government organization .37 .088 Academic job .45 .07 Policy job .05 .018 Management job .20 .072 Service job .15 .070 Position level .61 .039 Discussed 5.1 .25 53