31289 The Impact of WBI Client Activities, FY01-02, on Participants from Thailand: A Baseline Assessment Jaime B. Quizon Cristina Ling Chard WBI Evaluation Studies No. EG04 -77 The World Bank Institute The World Bank Washington, DC October 2003 ACKNOWLEDGEMENTS This report was prepared for the World Bank Institute (WBI) under the overall guidance of Marlaine Lockheed, Manager, WBI Evaluation Group. The team was led by Jaime Quizon and Cristina Ling Chard of IEG. The authors thank Ian Porter, the Country Director of Thailand, and his staff Tinnakorn Sareenun and Apisit Charitsue for their exemplary support with getting the surveys under way; Ejaz Ghani for his valuable guidance; Bhumsaran Amthong for his excellent work in administering the participant surveys; Ejaz Ghani, Blanca Moreno-Dodson and Heidi Zia for kindly serving as peer reviewers of this evaluation report; and Humberto Diaz for his help with formatting this document. WBI Evaluation Studies are produced by the Institute Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: World Bank Institute Evaluation Group Publications and Briefs Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Managers Mr.. Jaime B. Quizon Ms. Cristina L. Chard ii ACRONYMS CAS Country Assistance Strategies CRS Client Registration System DL Distance Learning EPA Environmental Protection Agency F&B Knowledge/Skills FGD Focus Group Discussion FY01-02 Fiscal Year 2001-2002 GDLN Global Distance Learning Network HD Human Development HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency ICT Information Communication Technologies IEG Institute Evaluation Group K&S Knowledge and Skills MDG Millennium Development Goal NGO Nongovernmental Organizations PCD Pollution Control Department PREM Poverty Reduction & Economic Management RECT Regional Capacity Enhancement Team TTL Task Team Leader WB World Bank WBI World Bank Institute WBIEN World Bank Institute Environmentally & Socially Sustainable Development Division WBIFP World Bank Institute Finance & Private Sector Development Division iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS.................................................................................................ii ACRONYMS......................................................................................................................iii TABLE OF CONTENTS ....................................................................................................v EXECUTIVE SUMMARY...............................................................................................vii 1. INTRODUCTION........................................................................................................1 Background...................................................................................................................1 Objectives .....................................................................................................................1 2. METHODOLOGY .......................................................................................................3 Data collection..............................................................................................................3 Survey Sampling Procedure .........................................................................................3 Survey Participants.......................................................................................................4 Activities Sampled........................................................................................................5 Country staff interviews...............................................................................................5 Focus Group Discussions .............................................................................................6 3. EVALUATION DESCRIPTIVE RESULTS ...............................................................7 Relevance of WBI Activities to Thailand's Needs.......................................................7 Activity Effectiveness ..................................................................................................8 Assessing Impact, use of WBI-acquired knowledge and skills....................................9 Facilitators and Barriers to Activity Impact ...............................................................11 Assessing the Influence of WBI Activities ................................................................12 Influence at the Country Level...................................................................................12 Influence in Key Impact Domains..............................................................................13 Activity Sustainability: Follow up by Organizers and Participants ...........................14 WBI's Role in Thailand..............................................................................................15 4. EVALUATION: SOME MODELING RESULTS ....................................................17 Model of Activity Effectiveness.................................................................................17 A Model of Impact: Use of Acquired Knowledge and Skills ....................................19 What Factors Determine the Likely Influence or Effects of a WBI Learning Activity?.....................................................................................................................23 v 5. CONCLUSIONS AND RECOMMENDATIONS.....................................................25 ANNEXES Annex A: WBI Activities for Thai Participants FY01-02..........................................31 Annex B: World Bank Institute (WBI) Country Focus Evaluation Questionnaire - THAILAND...............................................................................................34 Annex C: Thailand WBI Program Survey FY01-02..................................................44 Annex D: Interview and Focus Group Questions for Country-Focused Impact Evaluation..................................................................................................47 Annex E: Country Staff Interviews ............................................................................48 Annex F: Focus Group Discussion.............................................................................52 Annex G: Two Stage Least Squares Results Predicting Impact.................................54 Annex H: Two Stage Least Squares Results Predicting Influence Of Impact...........55 Annex I: Millennium Development Goals (MDG)....................................................56 vi EXECUTIVE SUMMARY This report evaluates WBI activities offered to Thai participants in FY01 and 02. The study uses both quantitative survey data based on participant responses (N=160), and qualitative data from interviews with country staff and focus group discussions to assess the effectiveness and impacts of past WBI learning events. Additionally, this paper establishes useful benchmarks for future evaluations of WBI's upcoming country-focus activities. In this evaluation, we investigate three factors that relate to the WBI activity: · its effectiveness, operationalized as individual benefits from the activity (e.g., improvement of knowledge and skills (K&S), raised awareness, etc.); · its impact, or the use of the K&S acquired from the WBI-sponsored activity (i.e., applying new skills at work); and · WBI's influence, or participant assessments of WBI's impact. Evaluation results point to the need for WBI activities that are more relevant to Thailand and that involve participants' involvement in the development of action plans. The model predicting activity effectiveness shows that activities rated as relevant to Thailand are also deemed to be more effective by participants. Effectiveness is also related to the use of action plans in activities. Utilizing exercises where participants outline how they will apply lessons to work plans increases ratings of activity effectiveness. As one would expect, the more effective the activities are, the more likely participants are to report using lessons from them. Impact or use of WBI-acquired K&S is also related to the type of work and organization in which a participant is engaged. Participants who worked in the areas of management and administration, policy and legislation, and academe are more likely to apply their newly acquired K&S at work. However, participants affiliated with governmental organizations are less likely to report such use. This information is pertinent for future participant selection into WBI programs for Thailand. Finally, assessments of WBI's influence indicate that participants' ratings are more positive among participants who apply their K&S and/or discuss them at work. Hence, WBI's program planners should also consider enhancing the substance of activities to increase country and work relevance. Overall, our results suggest that WBI's most recent activities have had positive, albeit unremarkable, effects in Thailand. As the country-focus agenda in Thailand advances, we expect WBI's future evaluations, to produce significantly more positive results than the benchmarks we establish in this report. Already, our analysis predicts that as activities focus more on Thai issues, and become more applicable to people's work, vii learning events will become more effective and have greater impact, which will ultimately augment WBI's influence. viii 1. INTRODUCTION BACKGROUND 1.1 In FY 2002, WBI unveiled a new country pillar to spearhead a "country-focused" approach to building and developing capacity in client countries. The immediate goal is to enhance capacity in 12 priority countries in key areas as identified in the country assistance strategies (CASs) for these countries. WBI selected these priority countries in consultation with the regions, but purposefully selecting a low-income and a middle- income focus country from each of the six World Bank (WB) regions. For the East Asia Region, WBI identified Thailand as its middle-income focus country. 1.2 WBI's new country-focused initiative is a response to questions that have been raised regarding on-the-ground impacts of WBI's past interventions and their relevance for the Bank's goals for its individual client countries. WBI's new country-focus pillar is supposed to reorient WBI learning activities in favor of greater alignment with the Bank's country assistance programs and towards broader and more sustainable impacts, including in Thailand. 1.3 In FY01-02, 1,536 Thai participants received training in 96 WBI learning activities that were longer than one day's duration (see Annex A for a list of these activities). Activities represent various WBI program divisions and/or thematic sectors, including: WBIFP, WBIEN, HD, and PREM. Like most other WBI learning events held over this recent period, the majority of these activities were subject-matter oriented and not tailored specifically to immediate country-specific needs, like in the case, for Thailand. 1.4 This study forms one piece of a five-part IEG retrospective evaluation of the impacts of WBI's interventions in five (Thailand, Sri Lanka, Egypt, Russia, Brazil) of its 12 country-focus priority countries. Thailand was selected for evaluation because the number of Thai participants in FY01-02 WBI events ranks third highest among the 12 WBI priority countries. OBJECTIVES 1.5 This study has two main objectives. This first is to assess the impacts of past (FY01 and FY02) WBI interventions in Thailand. This retrospective appraisal aims to provide immediately useful lessons on (a) how, in the Thai context, the beneficial impacts of WBI interventions might be broadened and deepened, and (b) what pitfalls and/or constraints to these impacts might be addressed. The second objective is to gather useful benchmarks and other important data for subsequent evaluations of WBI's upcoming country-focused initiatives in Thailand. 1 1.6 The retrospective evaluation seeks to answer three major research issues that relate to the effectiveness, impact, and influence of WBI activities. First, we examine whether WBI activities were effective in building Thai capacity and providing benefits to participants. Next, we investigate whether these activities had an impact through participants using what they learn in learning events. Finally, we explore the influence of WBI's impact in Thailand, based on respondents' self-assessments. 2 2. METHODOLOGY DATA COLLECTION 2.1 We use a meta-analytic approach to data collection and analysis in this evaluation, employing a mixture of quantitative and qualitative methods. Our methods include: (a) a quantitative survey of a random sample of Thai alumni from WBI activities (see Annex B); (b) qualitative interviews with WB resident staff members who worked in Thailand during the time period under examination (see Annex C); and (c) a focus group discussion (or FGD) with seven Thai participants who took part in a distance learning activity (see Annex D). In general, we use the results from the interviews with WB resident staff and the focus group discussion to elaborate on the findings from the participants' survey. SURVEY SAMPLING PROCEDURE 2.2 In FY01-02, WBI offered 96 learning events to 1,536 Thai participants. Of this total, useful documentation in WBI's Client Registration System (CRS) exists for only 734 participants (48 percent). Among this group there were 27 repeats, or participants who had attended more than one WBI offering. This low number indicates that WBI generally targets unique audiences for its activities instead of inviting the same individuals to different events. Of the participants listed in the CRS database, only 650 had one of the four key pieces of information to reach them (i.e., e-mail, work phone, fax phone, work address). In short, less than half of the FY01-02 Thai participants, as reported in WBI's CRS database, have useful contact information for any follow-up activity. 2.3 Overall, it is clear that WBI-wide participant data collection efforts need to dramatically improve, particularly in the context of "country focus" where the goal is local capacity enhancement. In over half the cases for FY01-02, there is no contact information available for WBI graduates, thereby excluding the possibility of reaching them for any follow-up or other activities. Even if WBI thematic teams keep their own records, this information is not readily available to other task teams within the institute who may want to target a similar audience. Better WBI-wide efforts at collecting and updating participant data, perhaps tied to incentives, are necessary in order to ensure that WBI is able to maintain useful engagement with the majority of its alumni. 2.4 As there is no strong reason to believe otherwise, we assume that the participants for whom information exists do not systematically differ from the participants for whom information is missing. In other words, whether or not participants and participant- related information were included in the CRS database are random occurrences. A problem would arise if all participants listed in the CRS were from activities within the same WBI program, because surveying these participants would result in findings that are 3 program-specific instead of generally applicable to WBI as a whole. Fortunately, the CRS-recorded participant data represent diverse programs and activities. This suggests that record keeping was done at random and thereby excludes the likelihood of a systematic bias in our sample. 2.5 With this assumption of random inclusion in the CRS, we selected a random sample of 200 participants and 36 replacements from the group of 650 for whom at least one of the four vital pieces of contact information was available.1 We stratified the sample by program in order to include purposefully those participants from the eleven programs that were underrepresented in this list of participants with viable contact information.2 Of these 236 participants, 23 (or 10 percent) ended up having bad contact information and were not traceable. Another 27 (or 11 percent) reported that they did not attend the learning activity. These outcomes again point to serious deficiencies in WBI's CRS database. Some 26 (or 11 percent) sampled participants refused to take part in this study. In the end, there were 160 completed surveys out of the 186 potential respondents who could be contacted, for an adjusted response rate of 86 percent. 2.6 A local consultant led the survey team which included two other interviewers who were fluent in both Thai and English and had educational backgrounds in public relations and linguistics. The survey methodology involved undertaking up to ten separate attempts to obtain a completed survey from a respondent. These attempts included telephone calls, faxes, and emails to: track down respondents for whom the contact information was no longer valid; reach people who were out of the office and/or busy; remind individuals who agreed to comply to return their questionnaires, and persuade potential respondents to comply. While the standard procedure was for interviewers to first call the respondents and then fax or email them the survey questionnaires, interviews were conducted over the telephone as a final resort to avoid non-response and/or missing data. Only after a maximum of ten unsuccessful attempts were sampled respondents designated as a non-response. Upon receiving the completed questionnaires, the interviewers manually entered the data onto a website from the filled-in hard copies.3 SURVEY PARTICIPANTS 2.7 Of our 160 sample respondents, women represent a slight majority (58 percent). Participants' ages range from 24 to 68, with the average age of 42 years. Respondents' main work affiliations included work in: national, regional, and local governments (50 percent, two percent, and eight percent respectively); the private sector (18 percent); universities and research institutions (14 percent); not-for-profit organizations (three percent) and other organizations (seven percent). The sample was diverse in terms of 1Of the original 40 selected as replacements, four were duplicate names. 2These programs include: Community Empowerment and Social Inclusion, Flagship Program on Health Sector Reform and Sustainable Financing, Good Governance, Governance and Public Sector Reform, Health and Population Program, Infrastructure: Finance and Regulation, Leadership Program on AIDS, Learning Design and Capacity Building, Public Finance Decentralization and Local Financial Management, Social Protection Core Program, and Social Protection Learning Program. 3 IEG's verification of a random sample of 10 percent of the questionnaires determined that the survey questionnaires reported by the local survey team were 100 percent accurate. 4 respondents' careers including in: management and administration (29 percent); policymaking and legislation (24 percent); academics (17 percent); services (eight percent); and other not for profit and work organizations (28 percent). Respondents were pretty evenly distributed across position levels. While a few held entry and high level positions (four and five percent respectively), the bulk of respondents fell in between, with 24 percent at the senior level, 41 percent at the mid-level, and 29 percent at the junior level.4 ACTIVITIES SAMPLED 2.8 Our sample respondents participated in programs from WBIFP (46 percent), WBIEN (29 percent), and Other (HD and PREM, 25 percent).5 The actual distribution of WBI programs for Thai participants in FY01-02 comprised 28 percent WBIFP, 28 percent WBIEN, and 35 percent HD and PREM. Thus, the distributions of respondents by activity program were relatively close, despite the difficulties encountered with obtaining participants with valid contact information. 2.9 Under its country focus initiative, WBI is committed to align itself with the Thailand CAS. Although the WBI's FY04 priorities for the country is still under review, the latest Thailand CAS calls for capacity enhancement initiatives (and thereby, direct assistance from WBI) in the program areas of governance and knowledge for development. This suggests some reorientation in the sectoral composition of WBI activities because past efforts in Thailand were directed mainly at sectors other than these two new CAS priority areas. We will return to this issue of alignment of WBI activities with the Thailand CAS in a later section. COUNTRY STAFF INTERVIEWS 2.10 We compiled data on WBI activities from country staff based on face-to face interviews, written responses to our country staff questionnaire (see Annex C), and comments made during RCET meetings between WBI and the World Bank (WB) Bangkok office. The data we gathered focused on five main issues: (a) results from WBI activities, (b) barriers to impact or the use of WBI-acquired knowledge and skills, (c) WBI's role in Thailand as a learning provider, (d) alignment between WBI activities and the CAS, and (e) suggestions for future WBI offerings. The general purpose of the interviews was to obtain feedback on past and future WBI activities.6 2.11 We identified twelve WB staff involved with Thailand in FY01-02 who might be familiar with WBI's work. During a IEG mission to Bangkok, we conducted face-to-face interviews with three members of the Bangkok office staff.7 We contacted the remaining nine operational staff who were unavailable for interviews through letters, verbal requests 4Seven percent fell in the Other category. 5We combined the HD and PREM activities in order to reach a statistically valid number of respondents to allow comparisons with this category. 6Annex E summarizes the key results of the resident staff interviews. Throughout this paper, we also refer occasionally to these results to elaborate of the findings from the participants' survey. 7Many staff were in Washington DC for PREM week or away on mission. 5 from the local assistant, and e-mails. After several attempts, we obtained a total of five completed questionnaires. Additionally, we were able to obtain data pertinent to issues c, d, and e from three staff members who did not respond to the survey but attended an RCET meeting between WBI and WB. FOCUS GROUP DISCUSSIONS 2.12 We conducted a focus group discussion with seven WBI alumni who participated in a distance learning activity, "Environmental Compliance and Enforcement: Practical Case Studies." The activity was selected because it was delivered via distance learning (DL courses are the majority of all FY01-02 WBI learning events with Thai participants) and offered by a program that delivered a substantial portion of activities in Thailand, WBIEN.8 Also, most of the participants were Bangkok-based and therefore easy to organize into a focus group. 2.13 The focus group discussion took place in Bangkok in mid-April 2003. Two IEG evaluators and the local Thai consultant facilitated this event. Six of the seven Thai participants were female. All were university graduates and working full-time. Half of the participants worked in areas related to environmental regulation. Two participants worked for the pollution inspection division of Thailand's Pollution Control Department (PCD), one was a professor and deputy director of the Environmental Research Institute of Chulalongkorn University. The remaining four participants did not work in environmental regulations but had attended the WBI-sponsored DL workshop. This leads to one of the main points raised by the focus group participants (as well as WB resident staff members), the need to be more strategic in targeting WBI learning event participants. Annex F summarizes the key findings of the focus group discussion which we will refer to when necessary to elaborate on the main findings from the participants' survey. 8It is important to note that although the participants were all from a single activity, the results presented here are not to be interpreted as an evaluation of the activity. The comments surmised from the focus group discussions are used to illustrate and provide more in-depth insight into the quantitative survey results. 6 3. EVALUATION DESCRIPTIVE RESULTS RELEVANCEOF WBI ACTIVITIES TO THAILAND'S NEEDS 3.1 One purpose of this evaluation is to establish a baseline for activities held for Thai participants against which WBI's future country-focus work can be assessed. Accordingly, we asked survey respondents a series of questions measuring the relevance of WBI's FY01-02 activities to Thailand, that is, before WBI's country focus approach was established. 3.2 Survey results show a number of interesting findings. First, as expected, the majority of respondents did not believe the activities they attended were solely for Thailand. Less than half of respondents, 43 percent, reported that the learning event was designed specifically for Thai participants.9 3.3 Second, although not designed specifically for Thailand, the WBI activities' topics were assessed as somewhat relevant to Thailand's needs. The average rating was 4.92 on a scale from one (extremely irrelevant) to seven (extremely relevant).10 3.4 Third, this result is supported by respondents' reports of the frequency with which issues raised in WBI activities were discussed at work, with local partners, government officials, and or NGOs. The mean score was 4, indicating that the issues were discussed an average amount (1=never discussed at all, 7=discussed thoroughly). While issues raised in WBI events were not discussed at great length as perhaps, the main topics of conversations, they also were not entirely neglected in participants' work. 3.5 These survey findings are supported by our focus discussions (FGD) with participants. In the FGD, participants indicated that the activity they attended was not designed specifically for Thailand. The activity our FGD members attended was a regional event delivered via distance learning (DL). However, only two of the seven focus group participants discussed the issues from the learning activity in great detail at work. In fact, they were asked to present what they learned to their colleagues. But this was clearly an exception rather than the overall trend (see Annex F for more on the FGD results). 9Was the activity you attended designed specifically for participants from your country? (Question number 15 in Annex B.) 10Thinking about the themes covered in the WBI learning activity you attended, to what degree were the topics relevant to your country's specific needs? (Question number 16 in Annex B.) 7 ACTIVITY EFFECTIVENESS 3.6 We measured "effectiveness" with a series of questions that asked respondents to self-rate the impacts of their participation in WBI activities. Specifically, respondents were asked to assess the "effectiveness" of the learning event in five areas: increasing their knowledge and skills; enhancing understanding and awareness of development issues in Thailand; providing strategies for addressing Thailand's development needs; offering approaches for addressing the needs of their organization; and introducing respondents to others also interested in the theme of the learning activity (Self-ratings were on a seven-point scale, see Annex B for the exact wording of each question).11 3.7 Overall, the responses were positive. Figure 1 shows that respondents rated WBI activities above average in all five areas of effectiveness. WBI activities were rated to be effective in raising awareness of Thailand's development needs (4.81), offering networking opportunities (4.79), increasing knowledge and skills (4.74), providing strategies for Thailand's development needs (4.61), and giving approaches for addressing the needs of participants' organization (4.51). While mean ratings registered between average (four) and somewhat effective (five), it is important to note that responses ranged across the spectrum, from ineffective (two) to extremely effective (seven). 3.8 Again, this survey result is supported by our focus group discussion results. Most of the FGD participants found the material presented in the learning event to be interesting, and consequently felt that the activity had raised their awareness of Thailand's environmental needs for development. Only two of the seven participants in our focus group reported that participation in the activity enhanced their knowledge and skills and provided them with strategies for their organizations. Figure 1: Perceived Effectiveness of FY01-02 WBI ACTIVITIES AWARENESS OF THAILAND'S DEVELOPMENT NEEDS 4.81 4.79 NETWORKING 4.74 4.61 KNOWLEDGE AND SKILLS 4.51 STRATEGIES FOR THAILAND'S DEVELOPMENET NEEDS 4.30 4.40 4.50 4.60 4.70 4.80 4.90 Average Ratings STRATEGIES FOR ORGANIZATION 1(not effective at all) to 7(extremely effective) 11The response options ranged from one, "not effective at all" to seven, "extremely effective." 8 3.9 In order to obtain a general sense of the effectiveness of WBI activities, we created a summary scale of effectiveness by averaging the scores across the individual effectiveness items (=.92). Results show that respondents rated WBI activities as effective when all the dimensions of effectiveness are taken together. As illustrated in Figure 2, the majority of respondents, or 73 percent, rated WBI as effective ("somewhat effective", "effective", and "extremely effective"). Only a small minority of respondents, or 21 percent, scored WBI as ineffective across the various areas. Figure 2: Distribution of Overall Perceived Effectiveness of WBI Activities FY01-02 50 39 40 34 30 20 Respondents 20 of % 10 6 1 0 INEFFECTIVE SOMEWHAT AVERAGE (4) SOMEWHAT EFFECTIVE (6- (1-2) INEFFECTIVE EFFECTIVE (5) 7) (3) ASSESSING IMPACT, USE OF WBI-ACQUIRED KNOWLEDGE AND SKILLS 3.10 Previously, we explored whether WBI was effective in providing respondents with knowledge and skills. Here, we investigate whether Thai respondents were actually able to utilize these new concepts and methods. Do WBI Thai participants actually use what they learn? And if so, how frequently do they use their newly acquired knowledge and skills (K&S)? 3.11 Figure 3 illustrates the proportion of survey respondents who acknowledge using their WBI-acquired K&S. A majority of respondents reported applying K&S in creating development strategies in Thailand (88 percent); implementing new work practices (85 percent); and raising awareness in development issues (73 percent); and implementing country development strategies (67 percent). Approximately half of respondents reported organizing community initiatives (52 percent), influencing legislation and regulation (50 percent); and teaching (50 percent). A minority of respondents applied what they learned to conducting research (41 percent). We have no benchmarks with which to compare the above survey results. Even so, these overall findings seem to point to only very middle- of-the-road types of effects for WBI's Thai activities -- i.e., while the outcomes of WBI events are positive, there is nothing exceptional with the frequency of Thai participants' use of WBI-acquired K&S from what one might ordinarily expect a priori. These results do not appear very exciting. Nonetheless, they provide in themselves a good baseline 9 against which future assessments of WBI's impacts (under its upcoming country-focused initiative for Thailand) might be measured. Figure 3. Percentage of Respondents Who Reported Applying Knowledge and Skills from WBI Training 100 88 85 80 73 67 60 52 50 50 41 Respondents 40 of % 20 0 Create Implement Raise Implement Organize Influence Teaching Conduct development country awareness in new work community legislation Research strategies in development development practices initiatives and Thailand strategies issues regulation 3.12 Average ratings of how frequently respondents apply their WBI-acquired K&S also indicate that this use is not very frequent. On a seven point scale from one (not at all) to seven (very often), mean ratings ranged from a low of 3.29 for conducting research to a high of 4.25 for raising awareness of development issues (Figure 4). In short, mean scores indicate seldom to average use of skills and knowledge learned in WBI activities. Figure 4: Reported Use of Knowledge and Skills (1=Not at all, 7=Very often) 4.40 4.32 4.20 4.00 3.86 3.78 Ratings 3.80 3.63 3.63 3.63 Mean 3.60 3.50 3.40 3.29 3.20 3.00 Raise Teaching Create Implement Implement Organize Influence Conduct Awareness in Development New Country Community Legislation/ Research Development Strategies for Practices in Development Iniciatives Regulations Issues Thailand Work Org. Strategies 3.13 Nonetheless, in separate interviews with WB resident staff, they cited networking and new approaches to work (based on ideas provided by WBI) as intangible positive outcomes, resulting from WBI's learning events. For instance, WB staff described a 10 learning event on journalism as inspirational to local writers, due to the extraordinarily well-selected international and local speakers, who related their experiences in battling and overcoming the challenges journalists typically face. 3.14 Additionally, from our focus group discussion, there was a strong impact in the work of the FGD participants who worked for the Pollution Control Department (PCD). They were able to apply what they learned in their respective positions. Moreover, their learnings from the WBI activity affected their organization. Their supervisors instructed them to teach what they learned to their colleagues, and to implement these new practices immediately. FACILITATORS AND BARRIERS TO ACTIVITY IMPACT 3.15 In order to determine whether certain factors systematically inhibit or facilitate participants' abilities to use the knowledge, skills, and theories they learned in WBI events, we asked respondents to rate work-related and country factors that could affect applicability of lessons learned in various areas (see questions no. 6 and 7 in Annex B). The potential areas we examined are in: conducting research, creating country development strategies, implementing country development strategies, changing and influencing legislation and regulation, teaching, raising awareness in development issues, organizing collective community initiatives, and implementing new practices at work organizations. 3.16 Specifically, we asked respondents to rate specific factors on a seven point scale meant to capture degrees of helpfulness and unhelpfulness (where one is "not helpful at all" and seven is "extremely helpful"). The first column in Table 1 illustrates the average ratings for each factor for all areas. Respondents deemed most factors somewhat helpful or leaning toward facilitating the use of new skills and knowledge. These factors included: work policies( 4.6), incentive systems at work (4.09), supervisors (4.29), colleagues (4.41), country policies (4.59), social groups (4.46), political groups (4.18), country readiness for reform and innovation (4.59). However, "resource and funding availability at work" appears to be viewed as unhelpful, or as a barrier, preventing participants from applying their knowledge and skills (mean=3.89). While the 3.89 rating suggests that resources are a barrier to implementation, given the narrow range of average responses and the likelihood that resources are likely to be a problem in some areas and not others, we analyze helpfulness ratings broken down by the areas where WBI training could potentially have an effect. 3.17 Looking across the columns in Table 1, we can see that that there are no real differences between factor ratings by area. Thus, facilitators/barriers are not area specific. Resources are viewed as somewhat of a barrier across all areas while all other factors are viewed as somewhat of a facilitator. 11 Table 1. Perceived Helpfulness Of Facilitators And Barriers In Applying Knowledge And Skills And Implementing Strategies, By Area, (1=Barrier, 7=Facilitator) Area Facilitator/ Overall Work Raise Create Implement Organize Legislation Barrier Mean Org. awareness strategiesstrategies community regulation Teaching Research Work policies 4.6 4.74 4.69 4.68 4.70 4.66 4.65 4.65 4.57 Resources 3.89 3.90 3.89 3.96 3.98 3.88 3.95 3.96 3.77 Incentive system at work 4.09 4.17 4.15 4.18 4.22 4.23 4.19 4.23 4.13 Supervisors 4.29 4.35 4.36 4.40 4.38 4.34 4.33 4.43 4.31 Colleagues 4.41 4.50 4.47 4.51 4.54 4.51 4.50 4.57 4.47 Country policies 4.59 4.57 4.54 4.68 4.72 4.52 4.59 4.62 4.58 Social groups 4.46 4.54 4.47 4.55 4.62 4.55 4.47 4.55 4.42 Political groups 4.18 4.19 4.17 4.30 4.34 4.17 4.22 4.24 4.16 Country's readiness for reform and innovation 4.59 4.57 4.58 4.70 4.70 4.55 4.5* 4.64 4.57 3.18 The consensus among WB staff was that many of the lessons learned from WBI events are not translated into actions due to a lack of follow up by WBI and no institutional support locally. Another barrier cited was the readiness for reform, i.e., the "lack of local leadership and commitment to reform in certain areas." A barrier related to the journalism course was the technical limitation regarding access to the internet. The instructor demonstrated how to gather data for news stories on the internet but internet access is not available in every office. 3.19 Our focus group discussions did not turn up any barriers. The participants who worked in areas that were relevant to the activity did implement the lessons they learned. The other participants did not work in a related area or simply did not find it useful. For instance, the professor we spoke with did not incorporate anything from the activity in teaching materials or her research. ASSESSING THE INFLUENCEOF WBI ACTIVITIES 3.20 In order to determine the nature and extent of influence that WBI learning activities may have had in Thailand, we asked respondents not only whether the WBI activity had any local impact, but also how positively (or negatively) the activity had a bearing on certain local outcomes, In other words, the evaluation question we seek to answer in this section is: "What influences (positive and negative) have FY01-02 WBI activities had in Thailand?" INFLUENCE AT THE COUNTRY LEVEL 3.21 The survey asked respondents to report whether they observed any changes that may be attributed to WBI interventions, and to rate the direction (positive or negative) and the intensity of these changes (see questions no. 3 and 4 in Annex B). The survey results show that 45 percent of respondents observed changes in Thailand that may have been swayed by WBI learning activities in FY01-02. Further, the average ratings 12 indicate that respondents viewed these changes somewhat positively, 4.65 on a scale from one (strong negative change) to seven (strong positive change). INFLUENCE IN KEYIMPACTDOMAINS 3.22 The survey also asked respondents to rate the degree to which WBI activities may have contributed to outcomes in six development or impact areas: consensus building for change; legislations or regulations; community-based initiatives; research methods and publications; work practices in organizations; and teaching materials for courses. Overall, the perceived contributions of WBI's activities on these areas were somewhat positive (see Figure 6). Based on a summary scale (where one is extremely negative and seven is extremely positive), the mean score across all six impact areas was 5.04. (alpha=.86). 3.23 The individual domain-specific ratings indicate that respondents evaluated the influence of WBI activities in these specific areas more positively than at the country level (see questions 8a-f in the Annex B). These mean ratings range narrowly between a maximum of 5.19 to a minimum of 4.88, with the highest ratings afforded to academic- related outcomes, i.e., teaching materials for courses (5.19) and research methods and publications (5.04). Respondents also rated positively the impact of WBI activities, on work practices (5.15); community based initiatives (5.02); legislation and regulations (4.96); and consensus building (4.88). Figure 6 Perceived Impact of WBI: Evaluation of WBI's Influence on Thailand (1=negative 7=positive) Concensus Building 4.88 for Change Legislation or 4.96 Regulations Community-Based 5.02 Initatives Research Methods 5.04 or Publications Work Practices in 5.15 Organization Teaching Materials for Courses 5.19 4.70 4.80 4.90 5.00 5.10 5.20 5.30 Mean ratings 3.24 The positive influence reported in the survey results for "work practices in organization" were reflected in our focus group discussions. The two PCD workers reported that the WBI and EPA activities had a significantly positive impact in their work place because they learned "cutting-edge" techniques for their inspection work. 13 3.25 In general, the WB resident staff we interviewed did not observe concrete "on the ground" evidence of impact from WBI activities and consequently could not evaluate it positively or negatively. (This is not surprising since most staff were not very familiar with WBI activities during this time.) However, one resident staff was able to report a positive influence from a WBI activity. Local practitioners published a manual that was based on WBI course materials, written in Thai, and adapted throughout Thailand. ACTIVITY SUSTAINABILITY: FOLLOW UP BY ORGANIZERS AND PARTICIPANTS 3.26 Sustainable development is the ultimate goal for clients of the World Bank. Thus, in order for WBI's activities to have an impact and help clients achieve this goal, it is necessary for there to be follow-up that directly involve participants. Follow-up can be executed in various ways such as e-mail discussions, face-to-face meetings, newsletters, etc. and can be initiated by organizers (WBI) or participants themselves. The key for change is that new approaches continue to be discussed and supported after the formal learning event is over. 3.27 The survey responses show disappointing results regarding follow-up to WBI activities. Less than half of respondents, an overall group of 43 percent reported some type of follow-up activity either by other participants or WBI. Only 33 percent of respondents were contacted by organizers and only 31 percent were contacted by participants after the learning event. It is important to note that both focus group discussions and interviews with country staff highlight the importance of more follow-up with participants and for WBI to establish even more partnerships with local organizations for broader and deeper impacts. 3.28 One of the objectives for WBI activities is to encourage its alumni to make and maintain contacts with others interested in the same development issues. However, only a small majority of respondents, or 55 percent, reported that they were provided with contact information (e.g. e-mail, telephone, address) of other participants who attended the same activity.12 Consequently and not surprisingly, only 31 percent of respondents reported being aware of follow-up activities arranged by participants. The most frequent type of follow-up activities were e-mail discussions followed by face-to-face meetings. 3.29 On a slightly positive note, of the survey participants who were aware of such activities taking place, 68 percent reported that they participated in follow-up activities. Additionally, 62 percent report that these activities still continue. Contact information was rated as somewhat useful a mean of 4.96 on a scale from one (not useful at all) to seven (extremely useful) by respondents who reported receiving such information from WBI organizers. 3.30 These survey findings of having only very halfhearted activities following WBI- sponsored events were confirmed also in our focus group discussion. Individuals were not provided with contact information for other participants, nor were they contacted 12Likewise, only 24 percent of respondents reported receiving newsletter follow-ups after the learning activity was over. They were rated by those who received them as somewhat useful, with a mean score of 4.92 on a scale of one (not useful at all) to seven (extremely useful). 14 again by WBI after the learning event. Consequently, their personal networks were not enhanced by their workshop participation. They strongly suggested that WBI e-mail participants immediately after the event with: (a) contact information on other participants, and (b) information on available WBI and other resources to broaden and deepen their WBI-acquired knowledge and skills. Likewise, they suggested that WBI organize follow-up activities even if only virtually, three to six months after the initial WBI activity. 3.31 WB resident staff also underscored the need for more follow-up by WBI. The consensus among WB staff was that WBI does not go beyond delivery of the training activity but that such follow-up would be much encouraged. WBI is viewed by resident staff as deserting the country after the activity is delivered. WB staff recognize that it is not always easy for WBI to follow-up on the regular basis from Washington DC, hence they recommend delivering activities through partnerships with local organizations, particularly for regional WBI offerings where Thai organizations and nationals can be involved (e.g. Cambodia and Laos). WBI'S ROLE IN THAILAND 3.32 A remaining question we attempt to understand is the role that WBI plays for Thai participants. In other words, are WBI's activities unique? Or, are other organizations providing the same kind of learning opportunities, thereby rendering WBI's services redundant? We asked respondents to report whether they were aware of any learning activities with similar content offered by other organizations, whether in Thailand or elsewhere. Their answer appears to be a resounding no. Only 27 percent of respondents reported being aware of comparable learning events offered by other organizations.13 Of the groups who were familiar with non-WBI training activities, average ratings indicated that WBI's programs are somewhat more effective, with a mean score of 3.83, where one is "much more effective" and seven is "much less effective." 3.33 Focus group participants reported that the content in WBI's activity was not redundant to offerings by other organizations. The respondents who found WBI's activity useful also reported that, in comparing WBI to other organizations, WBI's offering was equally as good as the other activities they attended. 3.34 Interviews with country Bank staff indicated that WBI plays a unique role in Thailand. The consensus among country staff was that WBI brings an important international perspective to its offerings. That is, the speakers and material that WBI can provide are unmatched by local organizations in terms of teaching the lessons of other 13Participants cited the following organizations as training-providers in Thailand: Asian Combustion Technology; Bank of Thailand; Department of Insurance; Department of Pollution Control; Green Peace; IMF; Institute of Directors; Japan International Cooperation Agency; Karolinska Institute; Ministry of Public Health; Project Finance International; Royal Forest Department; Securities and Exchange Commission of Thailand; Shinawas University; Thai Rating Information Services; Thailand Environment Institute; Thamasart University; The Forest Industry Organization; Transparency International; Transparency Thailand; University of Thailand; United Nations; and United States Environmental Protection Agency. 15 developing countries. The point made by staff was that challenges for development surpass political boundaries and that WBI's ability to share lessons learned across regions can be very useful. Thus, the experiences of journalists in Latin America can be similar to challenges faced by writers in Asia. Moreover, staff suggested that WBI play an instrumental role in regional development in the future by using Thailand's experiences as case studies from which neighboring countries could learn in regional activities. 16 4. EVALUATION: SOME MODELING RESULTS MODELOF ACTIVITY EFFECTIVENESS 4.1 A primary evaluation question for Thailand is: "What determines activity effectiveness among Thai participants?" To understand what factors determine participants' ratings of effectiveness, we relate participant's self-measures of effectiveness to WBI activity features and participant-level characteristics. We estimate the following OLS regression model of overall effectiveness: effectiveness = 0 + 1 Thailand relevance + 2 designed4Thai + 3 Thai location & Ps + 4 action plan + 5 DL + 6 activity duration+ 7 activity duration2 + 8 follow-up + 9 no. colleagues Attended + 10 technical terminology + 11 percent activity attended + 12 position level + 13 female + 14 age + 15 age2 + (4.1) The dependent variable is effectiveness, which is a summary score for effectiveness in increasing awareness, knowledge, skills, networking, strategies and approaches (as explained previously).14 Effectiveness is a continuous variable that is recalibrated (from the original seven-point scale) to range from zero (ineffective) to one (effective). 4.2 Based on the philosophy underlying WBI's new country focus approach, we expect activities with a Thai focus to be more effective. The independent variables capturing this Thai focus are: Thailand relevance, designed4Thai and Thai location&Ps. Thailand relevance measures the extent to which the activity was rated as relevant and applicable to the Thai context. Ratings of this variable was originally on a seven-point scale, but was recalibrated to range between zero (not relevant at all) and one (highly relevant) for easier interpretation of regression results. Designed4Thai is a dichotomous variable of participants' assessments of whether or not the activity they attended was designed specifically for Thai participants where one is yes and zero is no. Thai location&Ps is a dummy variable that is equal to one if the activity location was Thailand and that the majority of participants (80 percent or greater) were from Thailand ; and zero otherwise. 4.3 IEG's impact study of program activities offered in FY00-01 (Khattri, Quizon, et al., 2002) notes that WBI activity characteristics can influence ratings of activity effectiveness. We test the significance of various activity features by including in the regression model a number of activity-related variables defined as follows: action plan (equal to one if the WBI learning activity included action plans; zero otherwise), DL 14See question numbers 1a-1f in the questionnaire, Annex B. 17 (equal to one if distance learning was used; zero otherwise), activity duration which is the number of days the activity lasted (along with the variable activity duration squared, taking into account the likely quadratic nature of the relationship, i.e., more days are better to a point of "diminishing marginal returns"), and follow-up (summary scale of the items measuring different kinds of activity follow-up where one indicates follow up and zero means there was no follow up15). 4.4 The IEG program impact study also showed that certain participant characteristics are important predictors of WBI's effectiveness. In our model, we investigate the likely impacts of the following factors: no. colleagues attended (average number of people from the same office with whom the participant attended the activity), technical terminology (continuous variable encompassing self-ratings of proficiency in technical vocabulary ranging from low (zero) to high (one) proficiency), percent activity attended (proportion of WBI activity attended), position level (continuous variable where zero is entry level and one is the highest level), gender designated by a dummy variable for female (one), and age in years (along with age squared). 4.5 We estimated the model using Ordinary Least Squares (OLS) regression. Table 2 presents the OLS results for equation 2. The first column lists the beta coefficients and the second column lists the standard errors. Given the cross-sectional nature of the data, the model predicted effectiveness fairly well with an R2 of .29 and an adjusted R2 of .22. 4.6 The results presented in Table 2 point to two important findings that relate to (a) our country focus hypothesis and (b) our expectations regarding activity features as based on past research on program impact. 4.7 We find some support for the country-focus hypothesis. Specifically, Thailand relevance indicates that ratings of activity relevance to Thailand are significantly related to assessments of the effectiveness of WBI activities (=.21, p<.001). Activities rated as extremely relevant to Thailand have significantly higher effectiveness ratings (or by up to 21 percent more) than those rated as less relevant. The two other Thai focus variables designed4Thai and Thai location & Ps are not statistically significant determinants of overall effectiveness. This suggests that perceptions of whether or not the activity was designed specifically for Thai participants do not influence ratings of effectiveness. Likewise, the location of the activity and the percentage of Thai participants in the activity are not related to effectiveness scores. Overall, our test of the country focus hypothesis shows that the relevance and applicability of the activity's content to the Thai context is the main determinant of an activity's effectiveness. 4.8 Our second hypothesis explores the importance of WBI activity features in predicting effectiveness ratings. Our results replicate previous IEG findings that the use of action plans in WBI activities enhances perceptions of activity effectiveness. Action plans are in-class action-learning activities, that can be assigned individually or as a group project. Participants are required to implement what they learn in the context of 15The follow-up items included variables measuring whether or not activity follow-up took place (initiated by WBI, or initiated by the participants themselves) and whether or not WBI provided attendees' contact information to participants. 18 their jobs and the needs of their country. Action planning varies according to activity. Examples include: outlining implementation plans, conducting assessments of country and sector needs, or developing work-related products. Our model results indicate that the inclusion of action planning in WBI learning events significantly increased Thai participants' self ratings of effectiveness by about eight percent (=.08, p<.01). Table 2: Effectiveness As A Function Of Activity And Participant Characteristics (OLS Coefficients) Variable Estimate Standard Error Activity features Country focus: Thailand relevance .2100** .0600 designed 4 Thai .0200 .0200 Thai location & Ps .0400 .0600 General activity factors: action plan .0800* .0300 DL -.0100 .0300 activity duration .0300 .0100 Activity duration2 -.0001 .0001 follow-up .0100 .0400 Participant characteristics no. colleagues attended -.003 .0040 technical terminology .0500 .0700 percent activity attended .0200 .0500 position level -.1100 .0700 female .0100 .0200 age .0100 .0100 age2 -.0001 .0001 constant .1500 .2000 N=160 R2=.29, R2=.29 **=p<.001, *=p<.01 4.9 We find no support for the third hypothesis that participant characteristics are related to WBI effectiveness. As mentioned previously, an earlier IEG program impact study (Khattri, Quizon et al., 2002) with a global coverage noted that some individual- level factors were significant predictors of WBI's effectiveness and impact. In our study, which is based solely on Thai participants, participant characteristics do not appear to explain their self-ratings of activity effectiveness. One possible explanation for these seemingly different results is that the larger global sample of the earlier study was picking up country differences among participants that seemed to fall along lines of participant characteristics. For instance, the past study reported that participant's proficiency with the technical terminology used in WBI learning events was an important predictor of activity effectiveness. However, if technical terminology is highly correlated with participant country, then the findings related to participant characteristics might be spuriously correlated. A MODEL OF IMPACT: USE OF ACQUIRED KNOWLEDGE AND SKILLS 4.10 To understand the factors that affect how frequently participants use their WBI- acquired K&S, we estimated several regression models relating participants' use to: (a) measures of WBI activity effectiveness in raising participants' K&S, (b) participants' 19 characteristics, (c) WBI activity features, and (d) facilitators and barriers to participants' use of their WBI-acquired K&S. Our initial regression estimates made use of Two Stage Least Squares (2SLS) techniques, where the first-stage regression was as shown for equation 4.1 in Table 2 and the second-stage regression related participants' use of K&S with several explanatory variables: the stage one results predicting WBI activity effectiveness from various participant characteristics (e.g. technical terminology, attending with colleagues) and activity features (e.g. DL, activity follow-up).16 In all, these 2SLS results did not hold up as well as simpler OLS regression estimates shown below. 4.11 We relate participants' K&S use to activity effectiveness in educating participants, a variety of factors that facilitate (or prohibit) the implementation of new skills and strategies, and the type of work organization to which people belong. While we estimate this as an OLS model here, we also report the 2SLS results in the Annex G. Equation 4.2 describes our model of impact: use of K&S = 0 + 1effectiveness + 2F&B knowledge/skills + 3F&B implement Strategy + 4governmental org + 5management/administration + 6 policy/legislation + 7academic + (4.2) 4.12 The dependent variable, is a summary scale of the frequency with which WBI- acquired K&S is used in various domains: raising awareness, teaching, creating development strategies for Thailand, implement new practices in work organization, country development. Impact is recalibrated from an original seven-point scale to a zero- one range where zero means "not used at all" and one means "used very often". 4.13 Effectiveness is a summary scale of participants' perceptions of the degree to which activities were successful in increasing participants' knowledge of development issues, enhancing and teaching new skills, strategies, and approaches for development, and broadening participants' networking circles. This variable is as defined in equation F&B Knowledge/Skills and F&B Implement Strategy represent the facilitators and barriers (F&B) that enable or prohibit the utilization of knowledge, skills, and strategies learned in WBI activities. F&B Knowledge/Skills is the average helpfulness (unhelpfulness) score given by respondents for work factors, such as, work policies and procedures, resources and incentive systems at work, attitudes of supervisors and behaviors of colleagues (= .89) that may enable or prohibit the use of WBI-acquired K&S. F&B Implement Strategy is the respondent's summary rating of country factors that can affect implementation of WBI-acquired ideas/awareness such as country policies, behaviors of certain social groups, conduct of local political groups and the state of Thailand's "readiness" for reform and innovation (=.90).17 16Results of these initial 2SLS regressions are shown in Annex G. 17F&B Knowledge/Skills and F&B Implement Strategy are based on question numbers 8 and 9 in the survey (See Annex B). 20 4.14 Governmental Org, Management/Administration, Policy/Legislation, Academic are dummy variables controlling for the type of organization in which respondents work and to which they are affiliated. The area in which one works or the type of organization in which one works may affect a participant's ability to utilize new ideas and implement change. For instance, governmental units with their rigid rules and regulations might be less free to implement changes compared with non-governmental organizations with more leeway to overhaul their policies. 4.15 Table 3 shows the OLS estimate of equation (4.2). The regression explains use of K&S rather well with an R2 =.40 and adjusted R2 =.37. Table 3: Use Of WBI-Acquired K&S As A Function Of Effectiveness, Facilitators And Barriers, Organizational Affiliation And Job Type (OLS Coefficients) Variable Estimate Standard Error Effectiveness .39*** .09 F&B: Knowledge/Skills .32*** .11 F&B: Implement Strategy .19* .11 governmental org -.06** .03 management/administration .09** .03 policy/legislation .07*** .04 academe .09* .04 constant -.06** .07 N=160 R2=.40 R=.37 *** = p < .001, ** = p < .05 *=p<.10 4.16 As shown, all the independent variables are statistically significant in explaining use of K&S. Effectiveness is the strongest predictor of impact (=.39, p<.001). Utilization of knowledge and skills increases by as much as 40 percent when respondents rate activities as highly effective. 4.17 F&B: Knowledge/Skills is almost equally strong in explaining impact, =.32, p<.001. Respondents' rating of work factors as facilitators (very helpful) versus barriers (very unhelpful) increases use of K&S by approximately 30 percent of the scale. What participants rate as facilitating factors for applying K&S at work include: work policies and procedures, colleagues, and supervisors. If we compare the effect of F&B: Knowledge/Skills with that of F&B: Implement Strategy (=.19, p<.10), it is clear that work factors are more important predictors of impact than country factors. Country factors (national policies, social and political groups, and country's general mood or "readiness" for reform and innovation) have a smaller and less significant influence on impact. These findings suggest that working individuals are strong catalysts for change, and they are even stronger catalysts when existing work and country factors favor change. 4.18 Participant characteristics are also significant predictors of use of K&S. The results show that participants' organizational affiliation and job type are significantly related to using learned knowledge and skills. Affiliation with a governmental org has a 21 negative effect on impact (= -.06, p<.05). On the other hand, working in academe (=.09, p<.10), policy/legislation (=.07, p<.001), and management/administration (=.09, p<.05) is positively related with impact.18 This suggests that new skills are easier to apply when working in these areas. 4.19 The statistically significant and negative effect of governmental org shows that being affiliated with a government organization (including national/central provincial/regional or local government) results in less use of the skills learned in WBI activities. Perhaps this result is on account of the greater difficulties, in the Thai context, with introducing new practices into government bureaucracies compared with their NGO counterparts. 4.20 The positive effect for working in academe is expected because academics tend to have a good degree of autonomy in their research agendas and teaching materials. In other words, it is not surprising that professors are able to apply the new concepts and skills they learned in activities to their work. 4.21 A priori, we did not expect that applying WBI-acquired knowledge would be more common among policy makers or legislators (relative to non-policymakers/non- legislators) primarily because legislation and policymaking are usually long contentious processes that are very difficult to get going. However, as the model shows, participants who work in policy/legislation report significant use of WBI-acquired knowledge and skills in their policymaking work. 4.22 The effect from working in management/administration is also somewhat surprising because this area of work usually has standard protocols making it potentially difficult to apply new knowledge and skills. However, we found that new learnings can be utilized in management and administration and that working in this area actually facilitates application of new skills. 4.23 The results for participant job types are encouraging for WBI because they show that the potential impacts of WBI activities are far-reaching across many types of work and participants. Program planners can use these findings in developing lists of Thai participant invitees for maximizing WBI's impact. Individuals who work in academics, policy/legislation and management/administration are more likely to use what they learn in their positions. Also to be kept in mind when developing target audiences is that individuals who work in Thai government organizations are slightly less likely to apply knowledge and skills from WBI learning events. 4.24 Overall, our estimated model confirms the importance of effectiveness at the participant level. In other words, effectiveness at the participant level is the first criteria for activity impact, when impact is defined as the application of lessons learned. In the Thai context, WBI can boost effectiveness by tailoring its learning activities to the immediate needs of Thailand and by encouraging the use of action learning, particularly 18The excluded category is "other" including the service sector. 22 the development of action plans, for enhancing participants' knowledge about development issues and their understanding and use of the new concepts and skills in the context of their own work. The significant coefficient for effectiveness indicates that WBI activities in FY01-02 were successful in establishing the connection between issues learned in the WBI activity and their application in the workplace. 4.25 Work and country factors also significantly determine participants' use of K&S.. Work factors that are rated as facilitators (such as supervisors, colleagues, work policies and procedures) significantly increase participants' use of WBI-acquired K&S. Likewise country factors, such as Thailand policies, political and social groups, and Thailand's general mood or readiness for reform, are also positively related to the use of knowledge and skills. Both work and country factors show positive relationships with the use of knowledge and skills when these factors were rated as facilitators rather than barriers. In terms of participant factors that influence use of K&S, our results note that belonging to a government organization (versus non-governmental) results in less utilization of the theories and skills obtained in activities. On the other hand, there is a significantly positive increase in the frequency of skills used for those working in management or administration, in a policy and legislative forum, and/or in academe. WHAT FACTORS DETERMINE THE LIKELY INFLUENCE OR EFFECTS OF A WBI LEARNING ACTIVITY? 4.26 The previous section notes that, on average, survey respondents rate the influences of WBI learning events somewhat positively across key domain areas. In this section, we attempt to better understand respondents' assessments of WBI's influence by relating this factor to participants' self-assessments of their behaviors and estimating a simple regression model of perceived WBI impacts. As illustrated in equation 4.3, we relate respondents' summary ratings of WBI's influence across the six impact domains as a function of their reported frequency of use of WBI-acquired knowledge and skills and the frequency of their discussions of WBI-acquired knowledge and ideas on key issues.19 WBI Influence = 0 + 1 use+ 2 WBI issues discussed + (4.3) 4.27 The dependent variable, WBI Influence, is measured as the average influence rating across the six areas (noted above) in which WBI activities were likely to have some sway. This variable is recalibrated and ranges from zero (negative influence) to one (positive influence). The independent variables, use and WBI issues discussed, are operationalized respectively as the frequency with which respondents: (a) applied their WBI-acquired knowledge and skills (summary scale of question numbers 5a-h in Annex 19 Similar to how we finally settled for an OLS estimate of our equation on Use (equation (4.2), we initially tried a 2SLS approach to estimate (4.3). This approach tested the relationship of the dependent variable WBI Influence to the independent variables listed in (4.3) and participant-related factors (e.g., the variables government org; management/administration; policy/legislation; and others as described in equation 4.2). We also tried a number of alternative OLS specifications for (4.3). Some of these results are reported in Annex H. In the final analysis, the simple OLS relationship shown in Table 4.3 proved to be quite robust and the most meaningful, given the data. 23 B), and (b) discussed the issues raised at the WBI learning event with work colleagues, local partners, government officials and/or NGOs (see question number 9 in Annex B). 4.28 The underlying hypothesis of (4.3) is that the more frequently respondents report use of their WBI-acquired knowledge and skills, the more positively they would rate the influence of WBI activities. Likewise, the more frequently and thoroughly respondents discussed WBI-raised issues in their own work situations, the more likely they would rate positively the influence of WBI activities. 4.29 We estimated the model using OLS regression. Results are shown in Table 4. Given that the dependent variable is a continuous variable that falls between zero (negative influence) and one (positive influence), the beta coefficients shown in Table 4 represent larger changes than what one might expect at first glance. Overall, the model predicted WBI influence reasonably well with an R2 =.20. 4.30 The results show that both use and WBI issues discussed were significant predictors of WBI's influence ratings. Specifically, use has a strong positive effect on ratings of WBI's influence, =.19, p<.03. In other words, the more frequently people reported using their WBI-acquired knowledge and skills, the more positively they rated WBI's impact. These effects are quite strong, sufficient to increase influence ratings by a whole level on the seven point scale (e.g. from somewhat positive to positive). 4.31 Likewise, WBI issues discussed has positive and significant effects on WBI Influence, =.02, p<.01. While the magnitude of the effect is not as large as with use, this result indicates that the more highly respondents rated their discussions at work about issues addressed in WBI learning events, the more positively they assessed WBI's impact. 4.32 In sum, the model shows the importance for WBI of engaging participants in practical activities and relevant issues that directly relate to their day-to-day work affairs. WBI activities should focus on the applicability of the lessons to participants' everyday work. As our previous results also indicate, WBI's influence lies with how frequently WBI-event participants are able to use their newly acquired knowledge and skills in their daily dealings at their workplace.20 Table 4: WBI Influence as a Function of Use and Discussion (OLS Coefficients) DV= influence rating from zero (negative) to one (positive) Variable Estimate Standard Error use .19** .03 WBI issues discussed .02* .01 Constant .55** .03 N=160 R2=.20 R2=.19 ** = p < .001 *=p<.01 20This falls in line with our finding that the use of action plans (e.g. developing outlines of how concepts learned in event can be applied at work) significantly increases activity effectiveness. 24 5. CONCLUSIONS AND RECOMMENDATIONS WBI's Client Registration System (CRS) needs to be improved. 5.1 WBI-wide participant data collection efforts need to dramatically improve, particularly in the context of "country focus" where the goal is local capacity enhancement. A serious problem is that contact information is not available for over half of the WBI FY01-02 alumni. Additionally, at least four percent of the available contact information is no longer correct. These conditions drastically reduce the possibility of reaching the majority of Thai WBI graduates for any follow-up activities (i.e. for additional courses, or for evaluations of the learning activity in which they participated). WBI's Client Registration System is established for these purposes, however it is only as useful as it is comprehensive, current and actually used. In other words, WBI task team leaders (TTLs) need to document and, for country-focused activities, regularly update participant information in order for the CRS to serve its purpose as a comprehensive database of WBI activities and alumni. Perhaps, these responsibilities might be linked to performance measures with clearly tied incentives/disincentives for TTLs. 5.2 Moreover, it is important that activity leaders impress upon WBI alumni that part of their obligations for the activity is to cooperate in future evaluations of activity impact. In other words, monitoring and evaluation must be instituted at the activity level as part of the activity and responsibility of the participants. Survey results demonstrate the importance of Thai relevant agendas, and of relating lessons taught to participants' work for producing successful activities. 5.3 Survey responses indicate that activities in FY01 and 02 were not perceived to be very relevant to Thailand, according to the majority of respondents (57 percent). This is important because model results show that Thai relevance is a statistically significant indicator of activity effectiveness. That is, the more relevant to Thailand activities are deemed to be, the higher are the effectiveness ratings of the activity. 5.4 Nonetheless, activities were rated as being above average in the various dimensions of effectiveness. Aggregating the individual items into one scale of effectiveness shows that greater than 70 percent of respondents gave scores indicating "somewhat effective" or better. 5.5 Effectiveness ratings are also influenced by how well activities teach participants to apply what they learn. That is, activities that utilize action plans in their learning events were significantly more effective. Action plans are important tools that compel participants to determine how they can apply WBI-acquired K&S in the context of their own work. 25 5.6 Most respondents use their WBI-acquired K&S in implementing new work practices (73 percent) and raising awareness in development issues (73 percent). However, mean ratings of how frequently respondents apply these K&S indicate that this use is not very frequent. 5.7 Factors perceived as facilitating the use of WBI acquired K&S include: work policies, incentive systems at work, supervisors, colleagues, country policies, social groups, political groups, country readiness for reform and innovation. However, the availability of resources and funding is viewed as unhelpful, or as a barrier that prevents participants from applying new K&S. Overall, these factors are significant predictors of use, though work factors are even more important than country factors. 5.8 Participant characteristics influencing use of K&S relate to work type and organization. Affiliation with governmental organizations decreases use but working in academics, management and administration, or policy and legislation increases use of K&S. This information is informative for participant selection. Impact of WBI activities have been positive but not extraordinary. 5.9 Overall, while average survey ratings indicate that respondents viewed changes somewhat positively, the same responses indicate only a middling impact from WBI's intervention in Thailand. Results show that less than half (45 percent) of respondents observed actual changes that may have been swayed by WBI learning activities in FY01- 02. Work practices and teaching materials received the highest scores among impact areas. 5.10 Our models of WBI influence indicate that impact would have been even more positively rated if respondents were able to use and discuss their WBI-acquired K&S at work. In other words, assessments of WBI's impact are related to the frequency with which respondents used their new K&S and how often respondents reported discussing issues addressed in WBI events with colleagues, local partners, government officials and/ or NGOs. WBI needs to improve its follow-up activities. 5.11 The survey responses show very disappointing results regarding follow-up to WBI activities. Less than half of respondents reported some type of follow-up activity either by other participants or WBI. This is in part because most respondents reported never having received contact information for other participants. Both focus group discussions and interviews with country staff also highlight the importance of more follow-up with participants, and the need for WBI to establish even more partnerships with local organizations for broader and deeper impacts. WBI communications with the WB resident staff have to improve. 5.12 WB Thai resident staff are not very aware of WBI's work in Thailand. In interviews, Thai resident staff were surprised to learn of the large number and varied nature of FY01-02 WBI activities with Thai participants (96 activities in 18 programs) 26 and of the number of Thai participants in these activities (1,536). Additionally, resident staff were not fully aware of the follow-up activities linked to these WBI events. Resident staff recommend that they be consulted and engaged more actively than previously, particularly with the country-focused WBI activities that are forthcoming. WBI plays a unique role in Thailand 5.13 There is general agreement that WBI plays a unique role in Thailand as a provider of hard-to-replicate learning opportunities. Respondents were not aware of comparable learning events offered by other organizations. Of the few who were familiar with equivalent activities, they noted that WBI was as good, if not slightly more effective than its competitors. 5.14 Likewise, focus group participants reported that the content of WBI's activity was not redundant to other events offered by other organizations. The respondents who found WBI's activity useful also reported that WBI's offering was equally as good as these other learning activities they attended. 5.15 Interviews with country Bank staff also indicated that WBI plays a unique role in Thailand because the Institute brings an important international perspective to its offerings. The speakers and material that WBI can provide are unmatched by local organizations in terms of teaching the lessons of other developing countries. Further, staff suggested that WBI could potentially play an instrumental role in regional development by using not only Thai development expertise (as partners) in delivering WBI learning events, but also Thai experiences as case studies from which neighboring countries could learn in regional activities. 27 28 ANNEXES 29 30 ANNEX A: WBI ACTIVITIES FORTHAIPARTICIPANTSFY01-02 Activity Title Begin End Civic Participation in Local Government: Thailand 8/6/00 8/11/00 UNAIDS Global Dialogue Series - Thailand 8/31/00 8/31/00 Quality Assurance through ISO-9000 Certification 9/4/00 9/9/00 E-Learning Seminar - Thailand 9/7/00 9/7/00 Anti-corruption Summit 9/21/00 9/23/00 Pension and Provident Funds Course - Thailand 9/22/00 9/26/00 Health Issues Teleseminar Series- Evidence Based Medicine 9/27/00 9/27/00 East Asia and Pacific Regional Workshop on Poverty Reduction and Environmental Management: Emerging Sustainable Development Strategies for East Asia 10/4/00 10/6/00 Workshop on Urban Air Quality Management in Bangkok 11/13/00 11/16/00 Environmental Compliance and Enforcement: Practical Case Studies 11/30/00 12/7/00 Safety Nets Core Course 12/4/00 12/15/00 Planning Session - Core Course for Parliamentary Strengthening 12/11/00 12/14/00 Urban Transport Sector Strategy Asian Consultation Workshop 12/11/00 12/13/00 Technology Development and Competitiveness in East Asia 12/14/00 12/15/00 Structural Adjustment Policies and Poverty 12/18/00 12/21/00 Global Workshop on Implementing Emerging Financial Sector Standards: A Public/Private Sector Challenge 12/18/00 12/19/00 Implementing International Financial Standards - Thailand 12/20/00 12/20/00 South Asia - Policy and Institutional Reform for Sustainable Rural Development- Part I 1/15/01 1/19/01 WDR Dissemination East Asia: Thailand 1/22/01 1/22/01 WDR Dissemination East Asia: Thailand 1/23/01 1/23/01 South Asia - Policy and Institutional Reform for Sustainable Rural Development Core Course - Part II 1/29/01 3/13/01 Expert Workshop with Donors 2/12/01 2/16/01 Regional Workshop on Clean Air Initiative in East Asian Cities - Workshop I 2/12/01 2/14/01 Development Debates-Thailand 2/12/01 4/2/01 Strategic Management and Marketing of Training 2/19/01 3/2/01 Frontiers in Infrastructure Finance Distance Learning Course Bangkok Thailand 2/20/01 4/24/01 Thailand Corporate Governance and Strategy (DL) - Bangkok Thailand 2/21/01 2/23/01 Thailand Corporate Governance and Strategy Part I (DL) - China 2/21/01 2/23/01 Decentralization Dialogues for East Asia: Thailand 2/21/01 2/21/01 Non-Bank Financial Institutions Development and Regulation 2/26/01 3/2/01 Financial Sector Policy Dialogue Series 2/28/01 2/28/01 East Asia Decentralization Dialogues 3/21/01 3/21/01 Forest Law Enforcement: A Preministerial Meeting 4/2/01 4/3/01 Profiting from Small Business Lending: The Role of Credit Information and Credit Scoring in a Winning Small and Micro Enterprise Credit Strategy 4/2/01 4/3/01 Insurance Supervision Program 4/9/01 5/3/01 Health Sector Reform : Flagship Capacity Building in Asia-1st Regional Course in Malaysia 4/17/01 4/20/01 East Asia Decentralization Dialogues 4/17/01 4/17/01 Judicial Reform: Accountability and Performance 4/24/01 6/5/01 Poverty Analysis for Policymaking: East Asia 4/30/01 5/3/01 East Asia Decentralization Dialogues:Thailand 5/15/01 5/15/01 Health Sector Reform Capacity Building in Asia: Second Regional Course in Thailand 5/21/01 6/1/01 Parliaments and the PRSP Process in Asia - Thailand 5/24/01 5/24/01 Asia Development Forum 6/4/01 6/7/01 Fighting Poverty through Sustainable Development 6/6/01 6/8/01 (Annex A continues on nexr page.) 31 (Annex A continued) Activity Title Begin End Southeast Asia Investigative Journalism - Manila Philippines 6/10/01 6/14/01 Ethics in Government and Business - Washington DC USA 6/12/01 6/12/01 The Japanese Model for Achieving Intergovernmental Reform: A Dialogue for East Asia 6/15/01 6/16/01 Urban and City Management for East Asia: Training of Distance Learning Trainers - Singapore 6/18/01 6/22/01 East Asia Decentralization Dialogues: Thailand 6/19/01 6/19/01 Corporate Governance and Strategy -Thailand 6/20/01 6/20/01 East Asia Decentralization Dialogues:Thailand 7/17/01 7/17/01 Global Dialogue on Mainstreaming Social Development in the National Development Agenda 7/25/01 7/25/01 Global Dialogue on Mainstreaming Social Development in the National Development Agenda 8/22/01 8/22/01 Corporate Governance in East Asia 9/4/01 9/5/01 CAI-Asia Clean Air Regional Workshop and Second Coordination Meeting in Hanoi 9/5/01 9/7/01 Forest Law Enforcement and Governance 9/11/01 9/13/01 Frontiers in Infrastructure Finance Distance Learning Course 10/9/01 11/8/01 Flagship Core Course on Health Sector Reform & Sustainable Financing 10/29/01 11/16/01 The 3rd Clean Air Regional Workshop in Asia 11/7/01 11/9/01 Introduction to Program Evaluation 11/11/01 11/18/01 Youth and Corporate Citizenship: Future Leader's Forum 11/13/01 11/17/01 Corporate and National Governance Nexus: ASEAN Solidarity Network 11/15/01 11/17/01 City Strategies to Reduce Urban Poverty through Local Economic Development 11/19/01 12/14/01 Leadership Program on AIDS: Distance Learning Series for East Asia: Cross- Border Transmission and Related Issues- VC and Elec. Dis. Forum 12/1/01 6/1/02 Regional Forum - Hanoi 12/5/01 12/7/01 Parliamentary-Executive Relations 12/10/01 12/13/01 Globalization Economic Reform and Poverty Reduction 12/10/01 12/13/01 Thailand Governance and Anti-Corruption 12/13/01 12/13/01 Workshop on Intergovernmental Fiscal Relations in East Asia 1/10/02 1/11/02 Mainstreaming Social Development in the National Development Agenda 1/16/02 1/18/02 East Asian Decentralization Dialogues - II Global Distance Learning Network (GDLN) Programming 2/6/02 2/6/02 Training of Trainers Course on Basic Poverty Analysis (C101) 2/18/02 3/1/02 Transport Air Quality Workshop 2/26/02 2/27/02 CAI-Asia Regional Workshops in Manila on Reducing Vehicle Emissions 2/28/02 3/1/02 CAI-Asia Regional Workshops in Manila - Coordination Meeting 3/2/02 3/2/02 East Asian Decentralization Dialogues - II Global Distance Learning Network (GDLN) Programming 4/10/02 4/10/02 International Conference on Environmental Compliance and Enforcement (INECE) 4/15/02 4/20/02 Financial Markets and Development Conference: In Whom We Trust 4/17/02 4/19/02 Irrigation Institutional Reforms Initiative 4/21/02 4/26/02 Strategic Management and Marketing of Training (SMMT-5): Ensuring Sustainability and Financial Health of Training Institutions 4/22/02 5/3/02 Labor Market Policies Core Course 4/22/02 5/3/02 Corporate Governance and Strategy in East Asia 4/23/02 6/25/02 Health Sector Reform and Sustainable Financing: Asia Regional Course 4/30/02 5/3/02 Clean Air Initiative Asia Regional Workshop: Household Energy Indoor Air Pollution and Health 5/9/02 5/10/02 Mainstreaming Social Development in the National Development Agenda: Global Dialogue 5/15/02 5/15/02 Risk Management Workshop for Regulators 5/20/02 5/24/02 Asia Regional Flagship Course on Health Sector Reform and Sustainable Financing 5/20/02 5/31/02 (Annex A continues on next page) 32 (Annex A continued) Activity Title Begin End Asia Regional Air Quality Training Consortium Stakeholder Meeting-Clean Air Asia 5/28/02 5/31/02 Reaching the Rural Poor: Strategies for Rural Development 6/3/02 6/6/02 Enhancing Training Quality Through Customer Service (QCS-1): Caring for Clients Before During and After Training 6/5/02 6/11/02 WB Staff and Client: Intergovernmental Relations Local Financial 6/6/02 6/7/02 Basic Poverty Measurement and Diagnostics 6/10/02 6/21/02 East Asian Decentralization Dialogues 6/12/02 6/12/02 Promoting Practical Environmental Compliance and Enforcement Approaches in East Asia 6/24/02 6/28/02 Second World Congress of Environmental and Resource Economics 6/24/02 6/27/02 33 ANNEX B: WORLDBANK INSTITUTE (WBI) COUNTRYFOCUSEVALUATION QUESTIONNAIRE - THAILAND Instructions Thank you for agreeing to complete this questionnaire. The purpose of this questionnaire is to obtain your opinions on the performance of WBI learning activities in your country. We are going to ask you to think about the WBI learning activity you attended between the summer of 2000 and the summer of 2002. During this time period, if you participated in more than one WBI activity we would like for you to answer the following questions with the most recent activity in mind. According to our records the most recent WBI activity you participated in was entitled ______________________________________________________________________ and was held from _____________ to _______________. The questionnaire has four sections and should take approximately 15 to 20 minutes to complete. · Section 1 asks about the usefulness of the learning activity · Section 2 asks about the relevance of the learning activity to the specific needs of your country · Section 3 asks you to compare WBI learning activities with similar ones offered by other organizations · Section 4 asks about the characteristics of the learning event and your background. We would like to emphasize that your views are critical to us in our goal of continually improving WBI's learning activities. We need your honest feedback in order to get a clear picture of the effectiveness of the activity you attended. Please keep in mind that your responses will never be associated with you individually and that your participation in this survey will be kept confidential. If you have any questions or concerns about the questionnaire, please send us an e- mail to cchard@worldbank.org, or send us a fax to 202 522 1655, or give us a call at 202 473 2310. We will respond to you promptly. 34 WORLD BANK INSTITUTE (WBI) EVALUATION PARTICIPANT QUESTIONNAIRE Section I. Impact of WBI Learning Activities 1. Thinking about the WBI learning activity you attended, would you say that the activity has been effective or ineffective in the following areas: Not Extremely Don't Areas effective 2 3 4 5 6 Know at all Effective a. Increasing your awareness and understanding of the development issues important to your country b. Updating your skills and knowledge c. Providing you with new skills or knowledge d. Helping you get to know others also interested in the subject of the learning activity e. Providing you with strategies or approaches to address the development needs of your country f. Providing you with strategies or approaches to address the needs of your organization 2. Thinking about the WBI learning activity you attended, what was the main area or theme it addressed? 3. Thinking about this theme, do you think that the WBI learning activity led to any overall changes in that area within your Yes No Don't know country? (If "No," or "Don't Know," skip to question 5.) 35 4. How would you rate the nature of the overall change in that specific area? Strong negative 2 3 4 5 6 Strong positive Don't change change Know 5. Thinking about the WBI activity you attended, how often have you used the knowledge and skills you acquired in the following areas? Please mark "N/A" if you do not work in the given area. Areas Not at Very all 2 3 4 5 6 Often N/A a. Conducting research b. Developing country development strategies c. Implementing country development strategies. d. Changing or influencing legislation or regulations e. Teaching f. Raising others' awareness in development issues g. Organizing collective community initiatives h. Implementing new practices within your work organization 6. How helpful or unhelpful are the following factors in actually using the new knowledge or skills that you acquired from the WBI learning activity? Not Neither Extremely Don't Items helpful at 2 3 helpful 5 6 all nor Helpful Know unhelpful a. Your country's policies b. Social groups in your country c. Political groups in your country d. Your country's general "readiness" for reform and innovation 36 7. How helpful or unhelpful are the following in implementing the ideas covered in the learning activity? Not Neither Extremely Items helpful at 2 3 helpful nor 5 6 Don't Know all unhelpful Helpful a. Your country's policies b. Social groups in your country c. Political groups in your country d. Your country's general mood or "readiness" for reform and innovation 8. Did the WBI activity you attended have a positive influence, negative influence, or no influence in the following areas? (By influence we mean any changes resulting from your participation in the WBI activity.) Please mark "N/A" if the particular area is not relevant. Areas Negative No Positive Influence 2 3 Influence 5 6 Influence N/A a. Research methodology or publications b. Legislation or regulations c. Teaching materials for courses d. Consensus building for change e. Community-based initiatives f. Work practices in your organization g. Other (Please specify) ____________________ 9. Have the issues raised in the WBI learning activity been discussed at work, with local partners, government officials, or NGOs? Never Discussed Don't discussed at 2 3 4 5 6 all thoroughly Know 37 10. Thinking back to the time immediately following the learning activity you attended, were you aware of any follow-up activities (meetings, events, or Don't E-mail discussions) arranged by participants? (If you answer "No," skip to Yes No know question #15) 11. Did you participate in any of these follow-up activities? Yes No Don't know 12. In how many of these follow-up activities did you participate? 13. Are these activities still taking place? Yes No Don't know 14. If not, why? SECTION II. Relevance of WBI Learning Activities to the Needs of Your Country Recall, you are being asked to think about the activity you attended between the summer of 2000 and the summer of 2002. 15. Was the activity you attended designed specifically for participants from Don't your country? Yes No know 16. Thinking about the themes covered in the WBI learning activity, to what degree were the topics relevant to your country's specific needs? Extremely Extremely Don't Irrelevant 2 3 4 5 6 Relevant Know 38 17. Keeping in mind the WBI activity you attended, please review the list of country development objectives below in column A and mark those which were relevant to the activity you attended in column B. Next, for each one you marked as relevant, please rate whether the activity addressed the key issues in achieving that objective in your country in column C. A B C Mark an If you marked "X" , "x" here if please rate the degree to which the activity activity addressed the key issues in achieving that was related objective. Mark Development Objectives with an Low 2 3 4 5 6 High N/A "X' a. Eradicate Extreme Poverty b. Achieve universal primary education c. Promote gender equality and empower women d. Reduce child mortality e. Improve maternal health f. Combat HIV/AIDS, malaria, and other diseases g. Ensure environmental sustainability h. Develop global partnerships for development i. Ensure water sanitation and supply j. Improve investment climate and finance k. Promote trade SECTION III. Comparison of WBI Learning Activities with Similar Activities Offered by Other Organizations 18. Thinking about the WBI learning activity you attended, are you aware of any learning activities with similar content offered by organizations other than WBI, in Yes No your country? (If you answer "No," skip to question #21.): 39 19. If yes, please list the name(s) of the organization(s): 20. In comparing the activities offered by WBI and other organizations, would you say that, overall, the Non-WBI training activities were more effective or less effective than WBI training? Much About Much less Don't more 2 3 the 5 6 effective same effective Know SECTION IV. Characteristics of the Learning Event and Your Background Now, we would like to ask you to think about some of the things that took place during the activity you attended. 21. How many of your colleagues, or others with whom you work closely (both within or outside of your organization) attended the same activity? None 1 to 2 3 to 4 5 to 10 Over 10 22. During the learning activity, did you develop an action plan (e.g. work plans, strategy papers, policy documents, assessment of country needs)? Yes No (If you answer "No", please skip to question #25) 23. If yes, did you work as a team with your colleagues to produce the action plan? Yes No 24. Did you actually use the action plan, or parts of it, in your work? Yes No 40 25. To your knowledge was the learning activity a part of a series of related activities? Yes No 26. If yes, did you participate in other parts of the series? Yes No 27. Were you provided with contact information for other participants in the activity you attended such as email addresses, telephone numbers or mailing Yes No addresses? 28. If yes, please indicate whether it was useful or not useful: Not useful Extremely Do Not at all 2 3 4 5 6 useful Know 29. After the WBI learning activity, did the organizers contact you as a followup to the activity? Yes No 30. If yes, please indicate the nature of the follow-up: Face- to- Face Web-based E-mail Other Meetings discussions discussions (Please specify) 31. Did you receive any newsletters related to the learning activity after it was over? Yes No 32. If yes, please indicate whether it was useful or not useful: Not useful Extremely Do at all 2 3 4 5 6 useful Not Know 41 33. In retrospect, what aspect(s) of the learning activity did you find most useful for your work, organization, or country? 34. In retrospect, what aspect(s) of the learning activity did you find least useful for your work, organization, or country? 35. What was the primary language used during the learning activity? (For each item below, select one answer on the scale, Not Highly where 1=Not proficient at all and 7=Highly proficient.) proficient 2 3 4 5 6 at all Proficient 36. At the time of the learning event, what was your level of proficiency in the language of instruction? 37. At the time of the learning event, what was your level of proficiency in the technical terminology used in the learning activity? 38. How long was the learning activity? Please indicate number of days 39. What proportion of the learning activity did you attend? A quarter Less than More All or or less half Half than half almost all 40. At the time of the learning activity, which of the following best characterizes the organization in which you worked? (Select one.) 42 University /Research Institution Donor Agency Non-Governmental Organization (NGO) (not-for- profit) National/Central Government Private Sector (for profit) Provincial/Regional Government Other ____________________________________ Local Government 41. At the time of the learning activity, which of the following best describes your primary type of work? (Select one.) Research Management/Administration Teaching Policymaking / Legislative Research / Teaching Provide Services (e.g. financial, health, etc.) Other ____________________________________ 42. At the time of the learning event, how would you best characterize your position in your organization? Highest level (e.g. Minister, Deputy Minister, Top government official, Full Professor, President, CEO) Senior level (e.g. Department Head, Division Head, Associate Professor, Sr. Researcher) Mid level (e.g. Program Manager, Project Leader, Assistant Professor) Junior level (e.g. Research Associate, Ph.D. level graduate student) Entry level (e.g. Intern) Other (Please specify) ____________________________________ 43. Please indicate your gender Male Female 44. In which year were you born? (Please fill in the year in the boxes provided.) 1 9 Thank you for your feedback. We greatly appreciate your cooperation. 43 ANNEX C: THAILANDWBI PROGRAM SURVEY FY01-02 This questionnaire asks about the impact of WBI program activities that took place in FY01 and FY02. Please feel free to review the list of activities and programs to remind you of the various activities offered by WBI between July 2000 and June 2002. Thank you in advance for your cooperation! Name : _______________________________________________________________ Area of expertise: _________________________________Date:________________________ 1. Generally speaking, are you familiar with WBI's activities in FY01-02? (Feel free to review the list of 96 activities to help jog your memory.) Yes _____ No _____ If you answered No, please skip to question #8 . 2. We'd like to ask you about your involvement in WBI's training activities over the past two fiscal years, FY01-02 . · Were you involved in WBI activities at that time? Yes_____ No____ If you answered No, please skip to question #3 . If yes, what was the nature of your involvement in WBI activities? (e.g., development of course material, identifying participants, providing instructors, providing funding for participants etc.)__________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ How would you rate the extent of your involvementin WBI activities on a scale from 1(low) to 10 (high)? Please circle. Low ß à High 1 2 3 4 5 6 7 8 9 10 3. We'd like to ask you about the results, if any, from WBI's training activities over the past two fiscal years, FY01-02. · Can you think of any outcomes that resulted from these WBI learning events? Please describe. _________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ · If yes, what sort of evidence "on the ground" have you seen from these outcomes or results? Please describe.____________________________________________ ______________________________________________________________________ ______________________________________________________________________ 44 4. Over the past two years, how would you rate the effectiveness of WBI learning activities in achieving the MDGs in Thailand? Looking at the table below, please mark an "x" in the box next to each goal that is related to any of the WBI activities you are familiar with. Next, rate the extent to which the goal is addressed by the WBI activity in achieving that goal, by filling in the appropriate bubble (labeled low to high). Mark an "X" here if WBI Activities that If you marked "X," please rate the extent to which WBI addressed the you know about was related to the key issues in achieving that goal goal Goals for Mark with an Low 2 3 4 5 6 High N/A Development "X' a. Eradicate extreme poverty s s s s s s s s b. Achieve universal primary education s s s s s s s s c. Promote gender equality and s s s s s s s s empower women d. Reduce child mortality s s s s s s s s e. Improve maternal health s s s s s s s s f. Combat HIV/AIDS, malaria, s s s s s s s s and other diseases g. Ensure environmental s s s s s s s s sustainability h. Develop global partnerships for s s s s s s s s development i. Ensure water sanitation and s s s s s s s s supply j. Improve investment climate s s s s s s s s and finance k. Promote trade s s s s s s s s 5. Overall, to what degree do you think WBI activities are aligned with the Country Assistance Strategy (CAS)? If you were to rate the degree of alignment on a scale from 1 to 10, (where 1 is no alignment and 10 is complete alignment), how would you rate it? 45 No ß à Complete Alignment Alignment 1 2 3 4 5 6 7 8 9 10 6. What would you say are the three main barriers preventing participants from applying what they learn toward making change in Thailand? 1. ________________________________________________________ 2. _______________________________________________________ 3. ________________________________________________________ 7. In your opinion was WBI uniquely positioned to deliver learning activities for capacity building purposes in Thailand? Yes_____ No____ · If yes, what single reason do you think makes WBI uniquely positioned to deliver learning activities for capacity building in Thailand? ______________________________________________________ · In your opinion, what single thing does WBI bring to Thailand that other organizations do not? _______________________________________________________ 8. From your perspective, as the operational counterpart to WBI activities, do you think there are things that should have been done differently by WBI to better serve the needs of Thailand? Yes_____ No_____ If you answered No, please skip to question #8 . · If you answered "Yes", list up to three things that could have been done differently: 1_____________________________________________________ 2_____________________________________________________ 3_____________________________________________________ · Have you communicated any such "lessons learned" to the WBI task manager(s) in charge? Yes_____ No_____ 9. Taking a look forward, please list three ways that you think WBI could assist Thailand in moving closer to its development goals? 1______________________________________________________ 2______________________________________________________ 3______________________________________________________ 46 ANNEX D: INTERVIEW ANDFOCUSGROUPQUESTIONS FORCOUNTRY- FOCUSED IMPACT EVALUATION 1. What did you gain from the activity? 2. Was this activity part of a larger program? 3. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Probe 1: For what reasons? Please describe in detail. 4. Can you offer specific examples of how you applied what you acquired in the activity? Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? Probe 2: What were these changes? 5. In your view, how has the activity benefited your country? Probe 1: In what way? Please provide examples. 6. When you used the information that you acquired from the activity, what were some of the obstacles that you encountered? Probe 1: Please explain and give specific examples. Probe 2: In what ways were these obstacles? 7. Since attending the activity, have you attended other courses addressing the same topics? Probe 1: What were they? Probe 2: How useful were they? 8. Were the knowledge, skills or new ways of thinking that you obtained in the activity available from another source (e.g., institution, self-study materials, Internet site, etc.)? Probe 1: If yes, what were these sources and what were the names of the activities? Probe 2: How useful were they? 9. How can we improve similar WBI activities in the future? Probe 1: Please give specific examples. Probe 2: In particular, what types of courses would you like to see offered? 10. Is there any other assistance that WBI could provide in future activities to meet your specific professional needs? Probe 1: If so, what would it be? 47 ANNEX E: COUNTRYSTAFF INTERVIEWS We interviewed and sought comments from eight World Bank (WB) country staff members, who worked in Thailand in FY01-02, about WBI's activities offered to Thai participants during this period Our purpose was to learn about the impacts of WBI from the resident WB staff who observe and experience first hand the consequences of WBI's learning offerings. Staff familiarity and involvement with WBI activities Most WB staff interviewed were surprised to learn that WBI had 96 learning activities with Thai participants in FY01-02. The WB resident staff were not fully aware of the extent of WBI's work in Thailand. This was true even among those who provided support to, or were familiar with, some WBI offerings. Country staff are not very involved with WBI activities, even though they are the principal representatives of the Bank in the country. The extent of country staff involvement on a scale from one (low) to ten (high) ranged from one to eight, with an average of four, indicating that WB resident staff are not too involved with WBI events. The nature of staff involvement, when this occurred, varied tremendously according to the nature of the WBI learning event. In some instances, some Thai-resident staff provided tremendous support to WBI activities such as moderating courses and identifying local experts as speakers. However, in most instance, country staff involvement centered around identifying local participants. Some staff also reported attending WBI's learning events themselves. How do staff assess the results of WBI's training activities? Generally speaking, staff did not observe concrete "on the ground" evidence of impact from WBI activities. This is not surprising since most staff were not very familiar with WBI activities during this time. However, one hard result that was pointed out was a manual on a subject matter that was translated and adapted throughout Thailand based on WBI course materials. Additionally, resident staff cited networking and new approaches to work (based on ideas provided by WBI) as intangible positive outcomes, resulting from WBI's learning events. For instance, staff described a learning event on journalism as inspirational to local writers, due to the extraordinary international and local speakers, who related their experiences in battling and overcoming the challenges journalists typically face. 48 What are the barriers to WBI impact? We asked staff to define the factors that may inhibit WBI participants from applying what they learn. There seemed to be consensus that many of the lessons learned from WBI events are not translated into actions due to a lack of follow up by WBI and no institutional support locally. Partnerships between WBI and local institutions would help to foster follow-up and continued engagement of participants. Another barrier cited was the readiness for reform, i.e., "lack of local leadership and commitment to reform in certain areas." A barrier related to the journalism course was the technical limitation regarding access to the internet. The instructor demonstrated how to gather data for news stories on the internet but internet access is not available in every office. Is WBI unique as a learning provider? As mentioned previously, most of the country staff indicated that WBI is a unique training provider in Thailand because WBI brings international perspective and expertise into the picture. WBI has the ability to attract well--known keynote speakers which attracts participants who might not otherwise attend local events. Likewise, in some instances, WBI's name holds a certain level of prestige that attracts participants. However, this last point is two-sided. One staff member indicated that some participants may not attend World Bank events unless they were co-sponsored by local organizations. What can WBI do differently to better serve Thailand's needs? Staff suggested numerous areas where WBI could improve. The primary recommendation was that WBI better target the participant lists for their activities. WBI should consult with the local office when they develop their participants lists. Secondly, WBI's partnerships with local institutions should be continued and increased. Local institutional partners could help WBI build capacity locally and regionally. Related to that, WBI should utilize local facilitators to decrease language barriers as translations may be necessary for participants.21 Additionally, activities need to be announced on a broader scale. Staff believe that WBI could better advertise their course offerings if announcements were organized with the support of the country office. Overall, WBI could be more effective if it worked together with WB country staff and utilized its resources and staff. Country staff reported that WBI sometime undertakes activities that appear to separate itself from the World Bank. For example, one resident staff member described an instance when WBI hired an NGO to represent it instead of asking someone from the WB field office for help with this task. Because clients do not perceive a difference between WBI and the World Bank, it is more difficult for the WB local staff to manage its own public relations. For instance, staff complained that the country office was not able to facilitate WBI's reimbursement processes (which can take over six months) and ended up taking the blame for the delay from participants, hurting its own credibility. 21This concern was also noted in focus group discussions but it is important to note that proficiency in English was not a significant factor in activity effectiveness. 49 Do WBI activities address the Millennium Development Goals (MDGs)? We asked staff to identify which of the MDGs did WBI address with its Thai activities and to rate WBI's effectiveness in addressing these MDGs on a scale from one (low) to seven (high). According to staff, the MDGs addressed by WBI are: eradicating poverty (mean = 3.67), promoting gender equality and empowering women (mean = 5), combating HIV/AIDS, malaria and other diseases (mean = 6), ensuring environmental sustainability (mean = 4), developing global partnerships for development (mean = 5.67), improving the investment climate (mean = 4), and promoting trade (mean = 6).22 Were WBI activities in FY01-02 aligned with the Country Assistance Strategy for Thailand? Resident staff rated the alignment between WBI's Thai activities and the CAS as only slightly aligned. The average score for alignment was 6.25 on a scale where ten meant "completely aligned" and one meant "not aligned at all". The general consensus was that WBI specialized in certain sectors/themes and offered only those activities, rather than tailoring activities specifically to Thailand's needs. Staff urged WBI to try to develop activities that relate to operational goals as described in the latest CAS. This is explored in more detail below. How can WBI help Thailand in the future? The country office urges WBI to concentrate its FY04 efforts in two general areas: governance work and knowledge sharing, as outlined in the CAS. The Thai country office rejects the idea of delivering activities from diverse sectors, stating that WBI's offerings are broader than the World Bank's agenda for Thailand. Instead, WBI's country-focused interventions should concentrate on issues including: government decentralization, E-government, ICT, and GDLN. Additionally, WBI should strive to develop new partnerships within Thailand, particularly with resident staff and local counterparts in order to better tailor the WBI learning experience to the local context. Local Thai partners are also important for expanding WBI's influence to other country's in the Southeast Asia region. The Bangkok office suggests that WBI programs deliver two types of courses: "training of trainers" and "nuts and bolts" courses. The former is meant to promote sustainability. For example, government training conducted in conjunction with Chulalongkorn University would allow the university to continue training programs after WBI leaves. Formal courses that teach practical methods, such as how to perform internal audits, are also necessary. An example cited of a good WBI "nuts and bolts" activity with peer learning was a course for parliamentarians where the Minister of Commerce in Cambodia participated actively, in particular, with identifying the different government bodies that should be invited to certain organized learning events. On distance learning, country staff suggest that WBI focus specifically on customizing course contents and other materials for GDLN activities. Resident staff encourage WBI 22It is important to note that the means are based on the small number of staff we interviewed and should be interpreted in a qualitative descriptive sense. 50 to: (a) establish partnerships in content development with local institutions, and (b) employ Thais more widely as resource persons, moderators, and/or presenters. These would serve to develop teaching materials based on Thai and regional experiences and to encourage sustainability within the region with GDLN as a key vehicle for extending WB's reach to Thailand's neighboring countries . 51 ANNEX F: FOCUSGROUPDISCUSSION We conducted a focus group discussion with seven WBI alumni who participated in a distance learning activity, "Environmental Compliance and Enforcement: Practical Case Studies." The activity was selected because it was delivered via distance learning (DL courses are the majority of all FY01-02 WBI learning events with Thai participants) and offered by a program that delivered a substantial portion of activities in Thailand, WBIEN. The focus group took place in Bangkok. Two evaluators and the local Thai consultant facilitated this event. Six of the seven participants were female. All were university graduates and working full-time. Half of the participants worked in areas related to environmental regulation. Two participants worked for the pollution inspection division of Thailand's Pollution Control Department (PCD), one was a professor and deputy director of the Environmental Research Institute of Chulalongkorn University. The remaining four participants did not work in environmental regulations but had attended the WBI-sponsored DL workshop. This leads to one of the main points raised by the focus group participants (as well as WB resident staff members), the need to be more strategic in targeting WBI learning event participants. The results from the focus group discussion split along career lines. Only two people found the training very useful and applicable to their jobs. Not surprisingly, these were the individuals who worked for the pollution inspection division of the government. The others noted that while they learned new things, they did not find the activity useful in their own work currently. These latter participants were not well chosen for the learning activity. In fact, participants related that some were there to fulfill an assignment by their professor or boss, and one came because a non-English speaking friend brought her along to better understand the proceedings. Further, the two government workers explained that they attended the activity with a group from their office but served as translators for their colleagues during the activity because their English was better. One participant noted that the potential beneficiaries from such training could not attend because they speak only Thai.23 While written hand-outs were provided to participants, the materials were in English and participants complained that the outlines were too general. These WBI alumni also indicated the need for more follow-up activities. Individuals were not provided with the contact information for other participants, nor were they contacted by WBI again. Consequently, their personal networks skills were not enhanced by the workshop participation. Only one follow-up, conducted by the Environmental Protection Agency of the US, WBI's partner in offering the course, took place. The two PCD participants found this extremely useful. One participant suggested activity follow- ups to take place three to six months after the activity. The PCD participants raved in detail about the follow-up activities provided by the EPA. The EPA invited them to go on 23It is important to note here that the statistical analyses did not show a significant impact due to level of language proficiency. 52 a field trip to inspect a leather tanning factory. Additionally, EPA representatives came to their office twice to present videos of pollution inspections, which they found to be very relevant. Additional areas for improvement were on the technical aspects of the DL delivery. There were problems in the transmission of the video conference and certain portions were difficult to understand. Also, because the activity was a regional event, it was offered at odd times, six to eight in the morning, over a number of days, which made it difficult for some to attend all DL sessions. 53 ANNEX G: TWO STAGE LEAST SQUARESRESULTS PREDICTING IMPACT Table 5: Predicting Impact as a Function of Predicted Effectiveness, Facilitators and Barriers, Organizational Affiliation and Job Type Variable Estimate Standard Error Predicted Effectiveness (^Y) .33* .09 F&B: Knowledge/Skills .39*** .11 F&B: Implement Strategy .24** .11 Governmental Org -.07** .03 Management/Administration .08** .03 Policy/Legislation .07* .04 Academic .09* .04 Constant .09** .07 N=160 R2=.34 R=.31 *** = p < .001, ** = p < .05 *=p<.10 54 ANNEX H: TWO STAGE LEAST SQUARESRESULTS PREDICTING INFLUENCE OF IMPACT Table 6: Two Stage Least Squares Results, Influence of Impact as a Function of Predicted Use, Discussion, and Job Types Variable Estimate Standard Error Predicted Use .4940* .0499 WBI issues discussed .0095 .1133 Management/Administration -.0288 .0266 Policy/Legislation -.0238 .0293 Academic .0237 .0366 Constant .44* .05 N=160 R2=.29 R2=.26 * = p < .001 DV= influence rating from zero (negative) to one (positive) 55 ANNEX I: MILLENNIUMDEVELOPMENT GOALS (MDG) What MDGs did WBI's FY01-02 activities directly address? According to Thai survey respondents, WBI activities primarily addressed two of the MDGs: (1) improving investment climate and finance, and (2) ensuring environmental sustainability. A majority of respondents (55 and 54 percent, respectively) reported that the activity they attended was relevant to these two development objectives (See question numbers 17a-k, Annex B). Almost half of respondents (49 percent) cited global partnerships for development as related to the activity. Fifty-six percent of Thai participants attended activities delivered by WBIFP and WBIEN. Figure 7: Respondents Reporting that Activity Addressed Millennium Development Goals (MDG) 70 improve investment climate 59 60 54 enivornmental sustainability 49 global partnerships for development 50 41 promote trade 40 34 34 29 28 universal primary education 27 25 25 Respondents 30 eradicate poverty of 20 combat HIV/Aids, malaria % 10 ensure water sanitation & supply empower women 0 reduce child mortality MDG improve maternal health Respondents who identified MDGs as relevant to their activity were asked also to rate -- on a scale from one (low) to seven (high) -- the degree to which the issue was addressed in the activity. Average ratings point to the same two MDGs, improving investment climate (mean=5.01) and ensuring environmental sustainability (mean=5.20). Figure 8: Mean Ratings of Activity Concentration on MDGs (1=low, 7=high) reduce child mortality 4.1 improve maternal health 4.2 empower women 4.21 ensure water sanitation & supply 4.23 universal primary education 4.5 eradicate poverty 4.65 promote trade 4.69 combat HIV/Aids, malaria 4.91 global partnerships for development 4.92 improve investment climate 5.01 enivornmental sustainability 5.2 0 1 2 3 4 5 6 56 Both our survey results and participant CRS data indicate that WBI's concentration was on two millennium development goals for Thai participants in FY01-02: enhancing the investment climate and ensuring environmental sustainability. However, these results conflict with assessments by WB resident staff. According to staff, the MDGs addressed by WBI are: eradicating poverty (mean = 3.67), promoting gender equality and empowering women (mean = 5), combating HIV/AIDS, malaria and other diseases (mean = 6), ensuring environmental sustainability (mean = 4), developing global partnerships for development (mean = 5.67), improving the investment climate (mean = 4), and promoting trade (mean = 6). It is important to note that the means are based on the small number of staff we interviewed and should be interpreted in a qualitative descriptive sense. Although, these results also indicate the extent of the wide discrepancies between what resident WB staff perceive as the nature of WBI main activities in Thailand vs. what WBI actually delivers. There is certainly more room for WBI to improve on its dealings/communications with the Bank's Thailand resident mission. 57