Annual Review of World Bank Staff Learning FY03 Cristina Ling Chard Diana J. Arango WBI Evaluation Studies No. EG04-74 The World Bank Institute The World Bank Washington, D.C. December, 2003 ACKNOWLEDGEMENTS This evaluation report was prepared for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, Institute Evaluation Group. The report was discussed by the Learning Board at a meeting chaired by Ms. Phyllis Pomerantz, Chief Learning Officer, on October 27, 2003. This report would not have been possible without the careful and hard work of Maria Beatriz Camargo Cardenas, Paul Date, and Katherine O'Connor. The authors also gratefully acknowledge the special efforts by Humberto S. Díaz for picking up evaluation questionnaires whenever necessary, and for his help in formatting. Finally, we thank Jaime Quizon for his thoughtful peer review and Elizabeth De Meuter for carefully editing the final report. WBI Evaluation Studies are produced by the WBI Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the name of the author and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=Home&Intro=yes Vice President, World Bank Institute Ms. Frannie Léautier Chief, Learning Officer Ms. Phyllis Pomerantz Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Team Leader Ms. Cristina Ling Chard ii ACRONYMS ACS Administrative and Client Support AFR Africa Region ASTD American Society for Training and Development BO Bank Operations BS Behavioral and Social DEC Development Economics EAP East Asia and Pacific Region ECA Europe and Central Asia Region ESD Environmentally and Socially Sustainable Development Vice Presidency ESSD Environmentally and Socially Sustainable Development Vice Presidency EXT External Affairs FSE Financial Sector FY02 Fiscal Year 2002 FY03 Fiscal Year 2003 FY04 Fiscal Year 2004 GA-GD Staff in grades A through D GE Staff in grade E GF Staff in grade F GG Staff in grade G GH Staff in grade H GSD General Services Department HDN Human Development Network HQ Headquarters HRS Human Resources IEG World Bank Institute Evaluation Group IFC International Finance Corporation ISG Information Solutions Group IT Information Technology JPA Junior Professional Associate K&S Knowledge and Skills LCR Latin America and the Caribbean LP Learning Plan LEG Legal MIGA Multilateral Investment Guarantee Agency MNA Middle East and North Africa OPCS Operational Policy and Country Services P&I Process and Interaction PT Professional and Technical PREM Poverty Reduction and Economic Management iii PSD Private Sector Development PSI Infrastructure RM Resource Management SAR South Africa Region STC Short-Term Consultant STT Short-Term Temporary VPU Vice-Presidency Unit WB World Bank WBI World Bank Institute iv TABLE OF CONTENTS ACKNOWLEDGEMENTS ......................................................................ii ACRONYMS.........................................................................................iii TABLE OF CONTENTS........................................................................... v EXECUTIVE SUMMARY........................................................................vii 1. INTRODUCTION................................................................................1 2. METHODOLOGY...............................................................................3 Evaluation Administration.........................................................................................3 Evaluation Instruments...............................................................................................4 Activity Drop Rates....................................................................................................4 Participant Response Rates ........................................................................................5 3. ACTIVITY AND PARTICIPANT SAMPLES ..............................................7 Activity Sample ..........................................................................................................7 Participant Sample......................................................................................................9 4. DESCRIPTIVE STATISTICS................................................................13 Knowledge and Skills, Applicability to Job.............................................................13 Relevance, Quality and Usefulness Ratings.............................................................16 Summary of Training Rating Comparisons across Sponsors...................................17 5. MODEL RESULTS ...........................................................................19 Relevance to the Bank's Mission.............................................................................19 A Model of Activity Value: Quality and Usefulness ..............................................21 A Model of Knowledge and Skills...........................................................................23 A Model Predicting Applicability to Job .................................................................26 6. TESTING OUR MODEL RESULTS ........................................................29 7. QUALITATIVE RESULTS:PARTICIPANT FEEDBACK .............................31 8. RECOMMENDATIONS AND CONCLUSIONS ........................................33 Conclusions and Recommendations for Trainers.....................................................33 Recommendations for Participants...........................................................................33 Recommendations for IEG.......................................................................................33 v ANNEXES ..........................................................................................35 Annex A: Activity Questionnaire............................................................................37 Annex B: Participant Questionnaire........................................................................38 Annex C: Distribution of Respondent Unit, by Activity Sponsor ..........................40 Annex D: Distribution of Respondents by Grade and Course Content...................44 Annex E: Distribution of Respondents by Unit in Activity Content Types............45 Annex F: Percentage of Respondents Rating Courses 4 or 5..................................46 Annex G: Variable Definitions ...............................................................................47 Annex H: Models Predicting Relevance to Bank's Mission...................................48 Annex I: Models Predicting Quality and Usefulness Ratings................................49 Annex J: Models Predicting Perceived Knowledge and Skills Increase................51 Annex K: Models Predicting Training Applicability to Job ...................................53 Annex L: Models Controlling for Additional Variables .........................................55 Annex M: IEG Policy on Self-Conducted Level 1 Staff Evaluations.....................58 vi EXECUTIVE SUMMARY This report evaluates staff learning activities offered in FY03. The study is based on a compilation of level one evaluation questionnaires completed by 3,326 respondents in 242 staff learning activities. We examine the results of these surveys to monitor staff training and evaluate which factors can improve it. We analyze four basic dimensions of training quality: relevance to the Bank's mission; overall quality and usefulness of training; applicability of course to job; and perceived increases in knowledge and skills (K&S). Overall, participant ratings of training in FY03 remained the same as in FY02. However, this can and should be viewed as an improvement because of the positive sample selection bias in FY02. Our multivariate analyses demonstrate the following evaluation results: · Lengthier courses were perceived as increasing participant knowledge and skills. · Smaller class sizes improve reported increases in knowledge and skills. · Course content, materials, and order of presentation are significant to all dimensions of course quality. · Course topic is important for different dimensions: (a) Bank Operations, Behavioral Social, and Executive Management courses were deemed more relevant to Bank's goals, and (b) Bank Operations and Professional Technical courses were perceived to be less applicable to participants' jobs. The annual report points to two recommendations for IEG. First, IEG should improve the level one questionnaire by extending the five point response option scale to seven point. This would increase variance in participants' responses and therefore enhance the statistical power necessary for analyses conducted in the annual report. However, the transition would be seamless to trainers and the Learning Board because all course evaluations will report results based on a five point scale. Second, to meet the needs of the Learning Board to compare course ratings by activity sponsor, IEG should enhance the evaluation sample by employing a stratified random sampling procedure in selecting FY04 nominations. A stratified random sample would create a more even distribution of evaluations, based on activity sponsor. Weights can be applied to create a representative aggregate sample for the annual report. vii viii 1. INTRODUCTION 1.1 The Learning Board reports that almost three thousand staff learning activities took place in FY03 (N=2,880), with a total cost of over $69 million.1 These activities include a variety of learning opportunities such as formal face-to-face courses as well as e-learning events and informal brown-bag lunches. These also include all joint staff and client learning activities. 1.2 Given the enormous resources allocated to staff learning, the Learning Board has requested the World Bank Institute Evaluation Group (IEG) to monitor the progress of these learning activities. As part of this mandate, IEG conducts level one evaluations of staff learning on an on-going basis.2 In FY03, IEG evaluated a random sample of 242 courses, representing approximately ten percent of formal staff learning events for the year, and nearly 50 percent of unique activities offered.3 1.3 These level one evaluations contain a wealth of data. While level one evaluation is limited in that it relies solely on participant assessments of learning, it is nonetheless informative because it allows direct comparisons across a multitude of variables. Because of the large number of respondents, 3,326 in FY03, we can estimate complex statistical models. Moreover, self-reported satisfaction immediately after the course is not hindered by "recall effects," because respondents are not being asked to remember events that took place months or even years ago. 1.4 In this report, we examine participants' assessments of staff training. Specifically, we examine respondents' evaluations of training quality based on four dimensions: relevance to the Bank's mission; overall quality and usefulness; applicability to job; and perceived increases in knowledge and skills. We also estimate the determinants of these perceptions with multivariate statistical models. 1"Building Staff Capacity for Development: Update on the Staff Learning Program- FY03." Learning Board Executive Summary, FY03. 2IEG also conducts staff learning evaluations at levels two, three and four, the results of which are published in a series of separate reports. 3There were 514 unique events registered in the learning catalog as workshops, clinics, or courses under "event type." 1 2 2. METHODOLOGY 2.1 In FY03, we developed a science-based approach for sampling courses for level one evaluation. We obtained general course information from training coordinators and the Learning Catalog. Also, we gathered data for our evaluations through standard questionnaires administered in a uniform method to course participants. We discuss the methods for data collection in detail below. SELECTION PROCEDURE 2.2 The selection procedure used in FY03 was to draw a random sample of 30 staff training events from the learning catalog on a bi-weekly basis.4 If there were more than 30 courses eligible for evaluation within each period, the courses were selected randomly by a computerized random number generator. Course eligibility was based on the following criteria: · Activities were registered as formal events (i.e. no brown bags) in the learning catalog two weeks before the activity begins; · Bank staff comprised at least half of attendees; · Course delivery mode was face to face or distance learning; and · The course had not been previously evaluated by IEG in FY03. 2.3 We believe selecting courses produces less biased results than those produced from the self-nomination procedure used in FY02.5 Because the FY02 sample is based on courses that self-nominated into the evaluation sample, the ratings should be higher. Generally, courses requesting nominations receive higher evaluations. Thus, scores are potentially biased upwards because data was gathered only by those courses that volunteered to be evaluated. EVALUATION ADMINISTRATION 2.4 For each selected course, we asked training coordinators to submit the course agenda and fill out a short questionnaire about the event logistics (e.g. start and end dates, number of participants expected). Upon receiving the information necessary for coordinating the evaluation, we scheduled the course for evaluation and prepared the 4We selected 30 courses during each selection period in order to meet our target of evaluating 15 activities (we explore later why all selected courses are not evaluated). 5The self-nomination procedure was used because entering the activities into the learning catalog had not yet become mandatory. 3 course questionnaires with the title and dates.6 The survey administrator delivered the evaluations to participants on the first day of the learning event. While the standard procedure was to introduce the survey to participants in person, explain how to fill out the form, and to emphasize the importance of participant feedback, there were instances where instructors refused to comply, and the survey administrators were forced to leave the questionnaires for participants to complete without the benefit of instruction. Completed forms were collected at the end of each course by an IEG staff member. EVALUATION INSTRUMENTS 2.5 The data for the analyses presented in this paper are compiled from the level one evaluations and information provided by course administrators about the activity. The activity questionnaire asks training coordinators to detail the logistics of the event as mentioned above as well as give some background information on the activity such as sponsor, training content (i.e. Bank Operations, Behavioral and Social, Executive Management, Information and Technology, Professional and Technical), the learning approach (i.e. interactive versus formal), and the history, design, and delivery of the training. The activity questionnaire is presented in Annex A. 2.6 The evaluation questionnaires focus on participant satisfaction with the course overall as well as with course specifics: course delivery, design, and substance. In terms of individual benefit, respondents reported the extent to which the course fulfilled their learning needs, increased their knowledge and skills, and how much it related directly to their jobs. Perceptions of course quality, usefulness, and relevance to the Bank's mission were also asked to determine participant satisfaction. Assessments of the activity include participants' ratings of subject matter, order in which the content is presented, the materials and the balance of time spent on theoretical content, practical content, instructors' presentation, audience participation and overall pace of the training. The participant questionnaire is presented in Annex B. 2.7 In addition to quantitative ratings of the course, the survey asked open-ended questions about how the participants will apply what they learned on the job and the type of support they need to do so. Also the questionnaire asked respondents to provide suggestions for improving the course and to describe what they thought worked best. ACTIVITY DROP RATES 2.8 We evaluated 242 activities out of a total of 445 selected for evaluation in FY03. We "dropped" 203 courses from the evaluation sample.7 The percentage of "dropped" courses was slightly higher in FY02 (50 percent) compared to FY03 (46 percent). 6Activities for which the course administrators do not respond with the necessary information after several requests and reminders are dropped. 7Drop rates were highest among activities sponsored by: WBI (82 percent), ECA (76 percent), HRS (70 percent), PSI (68 percent), ESD (55 percent), and RM (52 percent). 4 2.9 In FY03, approximately one third of these "dropped" courses (N=69) were cancelled, postponed, or had inaccurate course dates published in the learning catalog. Another third (N=59) did not provide the essential information necessary for evaluation. We dropped the remaining courses (N=75) for the following reasons: · The sponsor or facilitator did not agree to evaluation being administered. · Another previously scheduled evaluation was to be delivered that date. · Problems were incurred during overseas mail delivery of questionnaires. · Local sponsor did not administer the evaluation survey. · Activity was actually for clients rather than Bank staff (advertised as both in the learning catalog). · Activity was not a formal training session but was advertised as such in the learning catalog (e.g. conference). PARTICIPANT RESPONSE RATES 2.10 Four thousand nine hundred and eighty-two participants attended the courses evaluated in FY03. Of this total, 3,326 completed our questionnaire, for an unadjusted response rate of 67 percent. However, controlling for class size (calculating the response rate without the two outlying courses that had more than 100 participants) the adjusted response rate rose to 78 percent. This adjusted response rate is higher than the response rate of 70 percent reported in FY02. 2.11 The trend in these data show that response rates decrease as class sizes increase above 50 participants. It appears that the social pressure to comply with evaluations tends to be stronger in smaller classes. Further, participants are less likely to leave smaller classes early, as their absence is more noticeable. Generally speaking, the likelihood of remaining until the end of the course and completing the evaluation is greater in smaller classes. 5 6 3. ACTIVITY AND PARTICIPANT SAMPLES 3.1 Our data is actually two-dimensional and can be measured at the course level and participant level. Thus, we discuss our samples from both the activity and participant perspectives. ACTIVITY SAMPLE 3.2 We evaluated a total of 242 staff learning activities in FY03. The World Bank designed and delivered a majority of courses (76 and 77 percent respectively). External consultants designed and delivered the other 25 percent. Overall, the Bank offered "seasoned" activities (courses delivered two or more times in the past) 53 percent of the time. Less than half of courses (40 percent) utilized "interactive learning" as the primary training approach. Instead, more than half of FY03 activities (60 percent) used conventional instructional methods (i.e. lecture). The average course ran for 2.2 days, while activity duration ranged from one to ten days. Course Sponsor 3.3 We categorized staff learning activities according to their institutional activity sponsor (see Figure 1). The largest proportion of courses evaluated came from HRS (19 percent); PREM (18 percent); ISG (13 percent); and OPCS (10 percent). Our evaluation sample also included courses from ACS (6 percent), PSI (5.8 percent), ESD (5.4 percent), HDN (5.4 percent), RM(4.1 percent), LCR (2.9 percent), WBI (2.5 percent), ECA (2 percent), AFR ( 1.7 percent), SAR (1.7 percent), FPS (1.2 percent) EXT, GSD, LEG (all less than one percent each). Figure. 1 Distributions of Course Evaluations by Sponsor 20 19.0118.18 18 16 14 13.22 12 9.50 10 courses 8 6.20 % 5.79 5.37 5.37 6 4.13 4 2.89 2.48 2.07 1.65 1.65 2 1.24 0.41 0.41 0.41 0 HRS ISG ACS PSI ESD RM HDN LCR WBI ECA AFR SAR FPS EXT GSD LEG PREM OPCS Sponsor 7 Activity Content 3.4 The largest share of courses covered are Professional and Technical (31 percent) and Bank Operations (31 percent) content (Figure 2). The share of Bank Operations courses increased sharply from FY02 (17 percent), and the share of Professional and Technical courses declined (from 53 percent). Figure 2. Percentage of Activities Evaluated by Content Type in FY03 Professional and 1% Technical 5% 1% Bank Operations 14% 31% Information Technology Behavioral and Social Executive 17% Management Other Trust Funds 31% 3.5 In comparison to Bank Operations and Professional Technical, Information Technology (17 percent) represented a smaller portion of courses in FY03; however IT course evaluations still increased from FY02 (13 percent). Likewise, Behavioral and Social courses comprised 14 percent of courses, which was slightly less than the percentage evaluated in FY02 (16 percent). The Executive Management courses made up 5 percent of our evaluation sample in FY03. This is a great improvement over FY02, when no managerial courses were evaluated. Finally, Trust Fund courses represented the same small proportion of courses in FY03, one percent, as in FY02, two percent.8 Activity Participants 3.6 Two sponsors accounted for 40 percent of participants in FY03 (Figure 3): PREM (22 percent) and HRS (16 percent). OPCS, ISG, ESD, ACS, PSI and HDN were in the middle with respondents making up anywhere from 11 percent to 5 percent of the sample. ECA, EXT, FPS, GSD, and LEG had the fewest respondents. 3.7 In both FY03 and FY02, PREM and OPCS provided courses for large shares of total respondents. PREM had the most respondents in both samples (15 percent in FY02 and 22 percent in FY03). Likewise, OPCS was second highest in FY02 (14 percent) and third highest in FY03 (11 percent). However, HRS gained significantly more respondents in FY03 (17 percent), ranking as the second highest compared to FY02, when it was in tenth place, with only 4 percent. ECA remained relatively low in FY03 as in FY02, with one percent or less in both years. 8One percent of training coordinators indicated their activity did not fit in any of these content types. 8 Figure 3. Distribution of Respondents, by Course Sponsor in FY03 25 22.31 20 16.87 15 10.73 10 8.24 7.88 Respondents 6.19 6.10 4.81 % 5 3.13 3.01 2.59 2.56 1.86 0.90 0.87 0.87 0.75 0.33 0 HRS ISG ESD ACS PSI HDN AFR LCR SAR WBI RM ECA EXT FPS GSD LEG PREM OPCS Sponsor 3.8 Sponsors with the most activities did not necessarily train the most participants. For instance, while HRS offered the highest proportion of courses evaluated (19 percent), PREM actually trained more respondents (22 percent) than HRS (15.6 percent). Likewise, ISG represented more evaluated courses (13 percent) than did OPCS (9.5 percent), but OPCS had more respondents (11 percent) than ISG (8 percent). This suggests that PREM and OPCS register slightly more participants in their courses. 3.9 The Learning Board emphasizes the importance of staff learning across units.9 In other words, staff training should not be constrained to participants taking courses offered only by their own units. To provide a picture of participant course interest and attendance, we present several tables in Annex C illustrating the distribution of participants by activity sponsor and participant unit. The regions were more likely than networks and others to have participants from their unit dominate their participant distribution. This suggests that region-sponsored courses are geared specifically for staff working in those areas, whereas non-regional courses are designed for broader audiences that include regional staff as well. PARTICIPANT SAMPLE 3.10 Our participant sample is based on 3,326 respondents. Most of our respondents identified themselves as World Bank Staff (87 percent). However, clients who responded to our questionnaires in the joint staff and client courses made up a small portion of respondents (13 percent).10 Bank staff respondents work mainly in headquarters (76 percent), with approximately one quarter (24 percent) stationed in the field. Most respondents are from the regions: AFR (N=472); ECA (N=340); LCR (N=243); EAP (N=219); and SAR (N=220) (with the exception of MNA (N=96)). The networks with the most respondents were PREM (N=181); PSI (N=114); IFC (N=111); HDN (N=100). Units with fewer than 100 respondents include: ESSD; OPCS; HRS; 9Moria Deborah Sutherland, the Learning Catalog Coordinator, explains that course offerings are not published according to sponsor in order to avoid staff constraining their training to their respective units. 10Recall the criteria for evaluating joint courses is that at least half of the participants are staff. 9 3.11 LEG; FSE and MIGA. The total number of respondents by unit are illustrated in Figure 4.11 Figure 4. Total Respondents, by Unit 500 472 450 411 400 340 350 300 243 239 250 220 219 200 181 Respondents 153 150 114 111 100 96 96 80 78 77 No.100 53 50 23 20 0 AFR OTH ECA LCR SAR EAP WBI PSI IFC HDN LEG FSE PREM ESSD MNA DEC HRS OPCS MIGA CLIENT Unit Participant Motivation 3.12 Respondents' primary motivation for participating in training activities was "to enhance performance in current/planned assignment" (56 percent). "Professional interest and growth" was the next most frequently cited reason for participation (35 percent). On the other hand, "network and sharing information" was mentioned by only very few respondents (5 percent) as their main motivation. This is surprising given the importance of networking within the Bank for future career moves. These results indicate that for more than the majority of respondents, motivation for course selection is narrowly confined to people's current work. Staff learning opportunities are generally not used to broaden individuals' understandings of Bank work in areas they are unfamiliar. 3.13 Interestingly, while more than half of respondents indicated that their intention was to enhance their current performance at work, only 32 percent reported registering these activities in their learning plan. Given, participants' reported motivations, more of their courses should be emphasized in the training section of their OPE. Participant Grade Levels 3.14 Participants in staff learning come from all grade levels (Figure 5): ACS level staff in grades GA-GD comprised almost a quarter (23 percent) of respondents; while there were 11 percent in GE, 24 percent in GF, and 18 percent in GG. Staff in grades GH and above made up seven percent of respondents. Additionally, our sample 11The "Other" category refers to respondents who checked off "other" on the survey for their work unit. "Client" refers to participants who stated they were "clients" on the questionnaire and participated in WBI courses that are jointly offered to staff and client. 10 included short term consultants (8 percent) and respondents who identified themselves as "other" (7 percent).12 3.15 The distribution of respondent grade levels in FY03 reflects the same patterns as in FY02. The two largest groups in FY02 were staff in grades GA-GD (31 percent) and GF (25 percent), as was the case in FY03 (23 and 24 percent respectively). Likewise, in both FY02 and FY03, staff in grades GG and GE represented mid-size groups (21 and 15 percent in FY02; 18 and 11 percent in FY03). Managerial staff in grades GH and above were the smallest group in both fiscal years (5 percent in FY02 and 7 percent in FY03). Figure 5. FY03 Respondents by Grade Level 7% GA - GD 8% 23% GE 1% GF 7% GG GH+ 11% STT 18% STC OTHER 24% 3.16 To get a picture of what types of courses respondents from different grade levels attended, we present a figure illustrating the proportion of respondents in each course type by grade level, in Annex D. In each grade level, Bank Operations and Professional and Technical courses were the most highly attended. Bank Operations were most popular with respondents in grade levels GA-GD, GE, and GG. On the other hand, Professional and Technical courses had the highest percentage of respondents among grade levels GF and GH. It appears that staff in higher grade levels select to learn about technical topics while staff in lower grades choose training in operational work. Participant Unit and Course Type 3.17 To gain insight into the audience for certain course types, we analyzed the proportion of respondents by VPU. Annex E illustrates the percentage of respondents by VPU in various types of courses.13 Bank Operations courses captured 20 percent of its participants from Africa region, followed by ECA (14 percent); LCR (11 percent); EAP(9 percent) and SAR (8 percent). Executive Management courses attracted a large proportion of individuals from the Africa region (26 percent). Information Technology courses were attended the most frequently by participants from WBI (16 percent). Professional and Technical courses attracted a high proportion of respondents from the Africa region (15 percent) and clients (15 percent). Behavior courses did not attract a significantly higher proportion of respondents from one particular unit. In fact, 12Respondents in the "other" category include JPAs etc. 13Respondents who did not identify themselves with a Bank unit are excluded. 11 attendance in behavior courses dropped uniformly by one to two percentage points per unit, ranging from the maximum in ECA (12 percent) to a minimum in ESD (one percent). 12 4. DESCRIPTIVE STATISTICS 4.1 Based on our survey of 3,326 participants, we established new Bank benchmarks for evaluating staff training. Specifically, we determined these benchmark standards by calculating the average scores given by participants for quality and course characteristics. Training quality is evaluated by: (a) assessments of the course's applicability to the participant's job; (b) perceptions of the degree to which knowledge and skills (K&S) increased; (c) evaluations of the course's relevance to the Bank's mission; and (d) ratings of overall usefulness and quality of training. Activity characteristics are assessed by respondents' ratings of: (a) materials used in the course; (b) the order in which the content was presented; (c) training content quality; (d) the extent to which the training achieved its announced objective and (e) fulfilled respondents' learning needs. The question items are rated on a scale from one (low) to five (high). Respondents' mean ratings are presented in Figure 6 with the items relating to course quality presented on top, and those relating to course characteristics presented on the bottom half of the figure.14 Figure 6. FY03 Respondents' Mean Ratings of Activities (1=low; 5=high) Applicability to Job 4.14 Knowledge and Skills 4.23 Relavance Bank's Mission 4.44 Overall Usefulness 4.26 Overall Quality 4.26 Materials used 4.14 Presentation Order 4.28 Content 4.28 Announced objectives 4.27 Learning needs met 4.18 3.95 4 4.05 4.1 4.15 4.2 4.25 4.3 4.35 4.4 4.45 4.5 Mean KNOWLEDGE AND SKILLS, APPLICABILITY TO JOB 4.2 Participants' assessments of training applicability to job and the level of increase in participants' knowledge and skills (the first two items in Figure 6) are the benchmark questions used by the American Society for Training and Development (ASTD). Our 14 Annex F presents the percentage of respondents who rated the items as 4 or 5. 13 respondents' ratings of applicability (4.14) and K&S (4.23) are statistically the same as the ASTD benchmark (4.21). This means that the Bank is on par with other organizations around the world that offer training. Next, we compare the Bank's ratings to its own benchmarks, established in FY02. Comparing Bank Averages in FY02 and FY03 4.3 To understand how the Bank's learning programs progressed in FY03 compared to FY02, we examine the average training ratings for these years. The applicability of courses to participants' jobs (4.14) generally stayed the same compared to last year (4.15). Perceptions of increases in knowledge and skills were also statistically the same (4.23 in FY03 and 4.27 in FY02). 4.4 The lack of statistically significant changes in ratings between FY02 and FY03 does not necessarily mean there was no improvement in staff training. The results from FY02 were likely biased upward because the sample included only self-selected courses, which tend to get higher ratings. In contrast, the FY03 sample is based strictly on courses nominated by IEG. Thus, it is difficult to make direct comparisons between FY02 and FY03 ratings. We must take into account that ratings from FY02 were inherently biased upward without knowing the extent of the bias. Having said that, we interpret that the lack of decrease in mean ratings indicates at least a small improvement in FY03. 4.5 Figure 7 illustrates activity sponsors in relation to the overall Bank average on knowledge and skills; job applicability; relevance to the Bank's Mission; overall quality and usefulness. Sponsors of activities that are rated significantly above the Bank average are indicated by solid circles, whereas, sponsors that are rated significantly below the Bank average are indicated byempty circles. Sponsors of activities that are equal to the Bank average are indicated by half-filled circles. 14 Figure 7. Comparing Activity Sponsors to Bank Benchmark Increase Knowledge and Job Relevance to Bank's Overall Overall Sponsor Skills Applicability Mission Quality Usefulness ACS ( ( ( ( ( LCR ( ( ( ( ( RM ( ( ) ( ( SAR ( ( ( ( ( FPS ( * ( ( ( ISG ( ) * ( ( HRS ) ) ) ( ( AFR ) ) ) ) ) HDN ) ) ) ) ) OPCS ) ) ) ) ) WBI ) ) ) ) ) ECA * ) ) ) ) ESD ) ) ) * ) EXT * ) ) * ) PSI * ) * * * PREM * * ) * * LEG * * * * * GSD * * * * * ( = Significantly above Bank Average ) = Same as Bank Average * = Below Bank Average Comparing Knowledge and Skills across Sponsors in FY02 and FY03 4.6 The first column of Figure 7 shows ratings of Knowledge and Skills by activity sponsor. Thirty-nine percent of sponsors (FPS, ACS, RM, ISG, LCR, SAR) were rated statistically higher than the Bank average, 4.23. Twenty-eight percent of sponsors (HRS, AFR, OPCS, HDN, WBI, ESD) did not differ significantly from the Bank Average. On the other hand, 33 percent of sponsors (PREM, PSI, EXT, ECA, LEG, GSD) were rated statistically below the Bank average. 15 4.7 We find that many sponsors that held above benchmark ratings in knowledge and skills in FY03 were the same sponsors that were rated highly in FY02 including: ACS (4.64), SAR/EAP (4.58), ISG (4.38), and FPS (4.27). RM was the exception. While RM was not rated highly in FY02 with an average of 3.89, it rated well above average in FY03, with an average score of 4.52.15 4.8 The same held true for courses that were rated below the bank average. The same sponsors that had low scores in FY03 were also rated low in FY02 including: HDN (4.04), ESD (3.99), PREM (4.15), and PSI (3.74). While these ratings are well below benchmark, HDN, ESD, and PSI demonstrated improvement in FY03. HDN's ratings increased by .14 (from 4.04 to 4.20); ESD improved by .13 (from 3.99 to 4.12); and PSI increased by .26 (from 3.74 to 4.01). 4.9 On the other hand, PREM and LEG course ratings showed no improvement in FY03. Not only were PREM's scores below average in FY02, they decreased by .12 in FY03 (from 4.15 to 4.03). Further, LEG's course ratings decreased from FY02 when it was above the benchmark (4.36) to significantly below the benchmark (3.91) in FY03.16 Comparing Applicability to Job across Sponsors in FY02 and FY03 4.10 The second column of Figure 7 illustrates the assessments of training applicability to job according to course sponsor. Twenty-two percent of sponsors (RM, ACS, LCR, SAR) were rated statistically higher than the Bank average, 4.14. Fifty percent of sponsors (ECA, WBI, EXT, ISG, AFR, HDN, HRS, OPCS, ESD, PSI) met the Bank benchmark. However, 28 percent of sponsors (PREM, FPS, LEG, GSD) were statistically significantly below the Bank average. RELEVANCE, QUALITY ANDUSEFULNESS RATINGS 4.11 Participants felt that training was very relevant to the Bank's mission (mean rating=4.44). In fact, comparing the ratings of all items on which we assessed Bank training (in Figure 6), the Bank was most highly rated on courses being relevant to the Bank's mission. There is no comparison score from FY02 for relevance to the Bank's mission because the question was not asked previously. 4.12 Participants assessed the quality and usefulness of Bank training equally positively, with the exact same average rating of 4.26. Scores were statistically the same in FY02, 4.28 for quality and 4.27 for usefulness. Given the self-selected sample from FY02 discussed previously, the lack of change in quality and usefulness ratings should be viewed positively. Because of the likely upward bias of average ratings in FY02, the fact that FY03 scores are as high can be considered an achievement. 15The varying sample sizes between years for each of the activity sponsors does not permit statistical significance tests. 16Again, it is important to note that LEG courses make up only a small portion of our sample (<3% in FY02 and <1% in FY03). 16 Relevance to Bank's Mission 4.13 The third column of Figure 7 illustrates training relevance to the Bank's mission by activity sponsor. Twenty-two percent of sponsors (FPS,ACS, LCR, SAR) were rated significantly higher than the Bank average, 4.44. Fifty-six percent (AFR, HDN, ECA, RM, OPCS, EXT, ESD, HRS, PREM, WBI) of sponsors had ratings that were statistically the same as the Bank average. Twenty-two percent of sponsors were significantly below the Bank benchmark (ISG, PSI, GSD, LEG). Quality of Training 4.14 The fourth column of Figure 7 illustrates training quality by course sponsors. One third of activity sponsors (FPS, LCR, ACS, RM, ISG, SAR, HRS) were rated above the bank average. Another third (AFR, WBI, HDN, OPCS, and ECA) were evaluated as equal to the bank average. The remaining third (ESD, PREM, EXT, PSI, LEG, GSD) were rated significantly below the Bank average. Usefulness of Training 4.15 The fifth column of Figure 7 presents of training usefulness. Thirty-nine percent of sponsors (ACS, LCR, RM, SAR, FPS, ISG, HRS) were significantly higher than the Bank average. Likewise, 39 percent (OPCS, ECA, HDN, WBI, AFR, EXT, ESD) met the Bank benchmark, 4.26. However, 22 percent (PSI, PREM, GSD, LEG) were rated significantly below the Bank benchmark. SUMMARYOF TRAINING RATING COMPARISONS ACROSSSPONSORS 4.16 Overall, the Bank is doing well with less than a third of course sponsors falling below the Bank and ASTD benchmarks. As comprehensive look at the patterns of training ratings in FY03 by activity sponsor (figure 7) shows that SAR/EAP, LCR and ACS are always rated statistically higher than the Bank average. On the other hand, PREM, GSD, LEG, and PSI are almost always rated significantly below the Bank benchmark. 4.17 Examining quality ratings across course sponsors provides an interesting descriptive look at training quality ratings and allows stakeholders to compare their ratings with their colleagues. However, in terms of improving training and serving the interests of Bank staff, it is critical to investigate potential differences according to course content because content cuts across sponsors. Thus, sponsors can focus their activity content to match the needs of participants. Simply looking at which sponsors were rated higher does not provide direction in how to improve staff training. In order to gain such insight, we must analyze the determinants of training quality. 17 18 5. MODEL RESULTS 5.1 In the following analyses, we attempt to understand the factors that affect participants' assessments of training quality. Specifically, we investigate which adaptable characteristics of training activities enhance (or decrease) assessments of four key training dimensions: (1) relevance to the Bank's mission; (2) overall quality and usefulness; (3) training applicability to job; and (4) knowledge and skill levels. RELEVANCE TO THE BANK'S MISSION 5.2 The ultimate goal of the Bank's staff learning offerings is to enhance the abilities of staff to achieve the Bank's mission. Consequently, as part of the level one evaluation, we ask respondents to evaluate the extent to which the activity they attended was relevant to the Bank's mission. This is a potentially difficult question to answer in some instances because the connection between a learning event and the goal of a world free of poverty, may be indirect in some instances. Nevertheless, some participants should understand that the courses they are taking will improve their abilities in their jobs, that is, the capacity in which they work towards achieving the Bank's mission. The overarching objective of staff courses is to build participant capacity in working towards the Bank's mission.17 5.3 We attempt to identify the prototypical characteristics of learning activities that are deemed relevant to the Bank's mission. Specifically, we develop a model predicting ratings of activity relevance to the Bank's mission as a function of learning activity characteristics including: course type (i.e. Bank Operations, Professional Technical, Behavioral Social, Executive Management, and Information Technology); activity offering history (whether or not the course was previously offered two or more times, Prevoffer2), and course design (whether or not the learning activity was designed by the Bank, WBdesign). Information Technology is the omitted variable. Relevance= f(Bank Operations + Professional Technical + Behavioral Social + Executive Management + Prevoffer2 + WBdesign) + (eq.1) 5.4 The primary purpose of this model is to identify which types of activities were deemed most relevant to the Bank's mission. Our hypothesis is that activities focusing on Bank operations would be rated as significantly more relevant to the Bank's mission. As mentioned previously, while information technology courses make people more efficient at their work, it might be more difficult for people to establish the connection 17See Annex G for variable definitions. 19 between increased efficiency in their current work and achieving the Bank's overarching mission. 5.5 Further, we expected that well-established courses (in contrast to courses offered for the first or second time) would be deemed significantly more relevant to the Bank's mission. The rationale is that, due to lack of experience, newer courses are not as effective at impressing the importance of the substance being taught in the larger context of the Bank's mission. 5.6 Finally, we anticipated that activities designed by the World Bank would be rated as more relevant to the Bank's mission than learning events designed by external consultants. This hypothesis is based on the notion that internal staff are more apt than external consultants at designing courses with the Bank's mission in mind. 5.7 Due to the highly skewed distribution of relevance ratings (with 85 percent of participants giving ratings of four and five), we use various estimation procedures to test our models. Specifically, we estimate the models based on a gamma distribution as well as a logistic distribution. Gamma regression is used for skewed data with positive values. The scale of the dependent variable ranges from one (not at all relevant) to five (very relevant). The logistic regression uses a dichotomous dependent variable where ratings of four and five are coded as one and zero otherwise. Results for both sets of estimates are reported in Annex H. Table 1 presents a comparison of results for the two estimation procedures predicting respondents' ratings of activity relevance to the Bank's mission. The results are consistent across estimation procedures. Table 1. Comparing Effects across Estimation Procedures: Models Predicting Relevance to Bank's Mission Variable Gamma Logit Bank Operations +, p<.05 +, p<.01 Professional Technical N.S. N.S. Behavioral Social +, p.001 +, p.001 Executive Management +, p.001 +, p.001 Prevoffer2+ +, p.001 +, p.01 WBdesign N.S. N.S. Note: + =positive effect; N.S.= no significant effect 5.8 Indeed, we find support for our theory that Bank Operations courses are positively related to higher ratings of relevance to the Bank's mission. Both models indicate significantly more positive ratings of relevance to the Bank's mission for Bank Operations courses in comparison to Information Technology courses, the excluded category. Additionally, Behavioral/Social activities and Executive Management learning events were also significantly more likely to be rated as relevant to the Bank's mission. 5.9 Likewise, we find support for our hypothesis that seasoned courses are more likely to be viewed relevant to the Bank's mission than new activities. Results from both models indicate a significant positive effect for activities that were offered two 20 times or more. These findings are good news, indicating that with experience, activities improve their ability to impress upon participants the importance of what they are learning to the Bank's mission. 5.10 On the other hand, there was no support for the notion that courses designed by the World Bank would be rated as significantly more relevant to the Banks' mission.18 It may be that externally designed courses are directed closely by activity sponsors, who emphasize the importance of the Bank's mission. Consequently, externally designed courses do not differ significantly from internally designed activities. A MODEL OF ACTIVITY VALUE: QUALITY AND USEFULNESS 5.11 Next, we examine respondents' perceptions of activity value. Value is a combined measure of the overall quality and usefulness of the activity. Theoretically, these items are tapping similar, if not the same, underlying dimensions of the activity's value to participants. Thus, we combine these fields into one variable due to their theoretical and statistical likeness. In fact, the strong statistical correlation between the two variables (r=.79, p<.01) indicates that the variables are essentially measuring the same thing (i.e. high quality courses are useful, and useful courses are high quality). We collapse the two measures by taking the mean of the items at the respondent level. 5.12 In order to understand the main determinants of a valuable (high quality and useful) activity, we estimate a model of Value as in Equation 2. Value depends on activity characteristics such as class size, activity duration, whether or not the course was part of a learning plan, activity offering history, as well as participant's assessments of course content, topic order, and materials, instructional approach, course design.19 Value= f(TotPart + Duration + LearnPlan + Content + Order +Materials + Interactive + WBdesign + Prevoffer2) + (eq.2) 5.13 We hypothesized that as class size (TotPart) increases activity ratings would decrease because instructors can give more individualized attention to participants when there are fewer participants by spending more time with each participant. For instance, instructors are able to answer more questions specific to participants' unique circumstances. 5.14 Past research shows that longer courses are generally assessed more positively than shorter courses (Ouchi and Le Rouzic 2001). Based on the findings from FY02, our expectation was to observe a significantly positive relationship between number of course days (Duration) and quality/usefulness ratings (Value). 5.15 We also theorized that participants would evaluate activities more positively when the course was listed in their learning plans (LearnPlan). In other words, 18As noted previously, the Bank deigned and delivered 85 percent of activities in our sample. 19Annex G defines the variables. 21 participants should find courses more useful and higher quality when those courses were identified as important in their official Learning Plans. 5.16 Participant ratings of training content, order in which the materials are presented, and the materials used in the activity should be indicative of participants' assessments of the overall activity. Thus, if Content, Order and Materials are assessed positively, the overall training value should be high. 5.17 Instructional approach should also be important for assessments of activity quality/usefulness. Activities that use the interactive learning approach should be rated more positively than activities that use conventional lecture approaches because interactive learning has been demonstrated to help people understand how to apply what they learn in training to their daily work. Thus, we anticipated that participants would rate course quality and usefulness higher in courses where Interactive learning approach is employed. 5.18 We hypothesized that courses designed by the World Bank (WBdesign) would be rated as more useful and higher quality. That is, internally designed courses should be deemed more valuable by respondents than externally designed courses. 5.19 Likewise, we expected established courses to be rated more highly (PrevOffer2). Specifically, activities that have been offered previously two or more times should have a better likelihood of being rated highly than newer courses that have not benefited from experience. 5.20 We tested these hypotheses in the model specified in equation 2 using three different estimation procedures. Again, because of the lack of variance in the dependent variable (Value), we estimate the model using gamma regression, logistic regression, and OLS with a logarithmic dependent variable. The model was specified three ways based on: (1) the gamma distribution with the continuous positive dependent variable on the five point scale; (2) the logistic distribution with a dichotomous dependent variable where a one indicates scores of 4 or 5 are observed; and (3) a normal distribution where the five point dependent variable is expressed in logarithmic equivalents (see Annex H). The results of the three estimation procedures are presented in Table 2. Table 2. Comparing Effects across Estimation Procedures: Models Predicting Value (Quality/Usefulness Ratings) Variable Gamma Logit OLS with Logged DV TotPart N.S. -, p<.01 N.S. Duration N.S. N.S. N.S. LearnPlan +, p<.05 +, p<.01 +, p<.05 Content +, p<.01 +, p<.001 +, p<.001 Order +, p<.001 +, p<.001 +, p<.001 Materials +, p<.001 +, p<.001 +, p<.001 Interactive N.S. N.S. +, p<.05 WBdesign N.S. N.S. N.S. Prevoffer2 +, p<.05 N.S. +, p<.05 Note: + = a positive effect;- = a negative effect; N.S.= no significant effect 22 5.21 Table 2 illustrates the results for our model predicting activity Value (the average of quality and usefulness ratings). The estimates from the varying procedures are generally consistent across all three methods. We find support for our hypotheses regarding the positive relationship between activity value and activity characteristics such as learning plan, content, order, materials and course offering history. On the other hand, class size, duration, instructional approach, and course design by WB were not significantly related to perceptions of activity value. 5.22 Our model confirmed our hypotheses based on the following findings: · Activities are perceived to be more valuable when participants identify the courses in their learning plans (LearnPlan). · Similar to results reported for ratings of relevance to the Bank's mission, previously offered courses (Prevoffer2) are significantly more positively rated than courses that have only been offered once before. However, the effect is not significant in models controlling for course topic (see Annex L for results of these estimations). · The more positively courses are viewed by participants in terms of training content, order in which the material is presented, and the overall quality of the training materials, the higher are overall ratings of activity value.20 · There is some evidence that interactive learning methods enhance the learning experience for participants by making the skills easier to apply in the workplace. Interactive learning was highly significant in the OLS model (p<.05) and almost reached statistical significance in the gamma and logit models (p<.10).21 5.23 On the other hand, our model also showed some results contrary to our theories. Value ratings were not related to activity logistics such as class size, duration or course design. Perhaps, when people answer this question they are thinking of their general impression of the course in a sort of "big picture" framework, in which case activity logistics are not important. A MODEL OF KNOWLEDGE AND SKILLS 5.24 The short-term objective for the Bank's staff learning activities is to enhance participants' knowledge and skills (K&S). While the overarching goal for staff training is to achieve the Bank's mission, the immediate objective is to increase participants' K&S. To better understand how self-rated K&S can be increased, we model K&S as a function of activity and participant characteristics (Interactive, Grade_AD, WBDesign, 20It is important to note that the effect of content is twice that of order and materials. Hence, trainers should focus on content first if they are interested in increasing their overall quality and usefulness ratings. 21The interactive learning effect is not significant in the larger models controlling for course topic (Annex L, Table L1). 23 DurationDays, and TotPart) as well as participant ratings of activity characteristics (LearnPlan, Materials, Order, Content) in equation 3.22 K&S= 0 + 1Interactive + 2Learnplan + 3Materials + 4Order + 5Content + 6Grade_AD + 7WBDesign + 8Duration + 9TotPart + (eq.3) 5.25 We hypothesized that Interactive learning would enhance K&S. In other words, participants in courses where interactive learning is employed would learn more than individuals in traditional lectures, and therefore, rate their level of the knowledge and skills increase higher. Interactive learning is a good technique for teaching people how to apply what they are learning to their daily work, thereby also enhancing their K&S as well. 5.26 Registering activities in learning plans suggests that participants are very interested in the training substance because they make plans to take the course in advance. Given this intense interest, we assume that their counterparts who did not plan the activity as part of their learning plans may therefore not be as intent on the course and learn less. That is to say, people who plan to take a course in advance may be more motivated to learn and therefore be more likely to increase their K&S. 5.27 Ratings of training materials, order of presentation and learning content should all predict ratings of knowledge and skills increases. The higher these training items are rated, the more likely respondents should be to learn, and hence increase their K&S. 5.28 Participant grade level is an important demographic variable that is included in this model of increasing K&S. The purpose of this variable is to gage whether groups of staff are more or less likely to increase their skills. Additionally, grade level provides a proxy for individuals' skill levels. The purpose of this variable therefore was not only to control for skill level differences but also exploratory to test whether people from ACS grades were more or less likely to learn in training events. 5.29 Additionally, we included our standard activity characteristic variables: WBdesign, Duration, and TotPart. We test whether internally designed course increases perceived learning. Our expectation for Duration is critical in this model of knowledge and skills increase. Individuals should definitely learn more form longer courses. The more time they spend in training, the more their knowledge and skills should increase. With each additional day respondents spend in training, their levels of knowledge and skills should increase. Likewise, more learning should take place when there are fewer participants. Thus, there should be an inverse relationship between number of participants and reported increases in knowledge and skills. 5.30 We again rely on various estimation procedures in our analyses because responses to the knowledge and skills question are unevenly distributed (with most ratings being a four or five). Table 3 presents the results for the various estimation procedures used to explain perceived knowledge and skills: gamma regression; logit 22See Annex G for variable definitions. 24 regression and OLS regression using a logged dependent variable (see Annex J for results). As illustrated in Table 3, the results are rather consistent across the variously specified models. The most important variables explaining knowledge and skills are LearnPlan, Materials, Order, Content, Duration, and TotPart. Table 3 Comparing Effects Across Estimation Procedures: Models Predicting Increases in Knowledge and Skills Variable Gamma Logit OLS with Logged DV Interactive N.S. N.S. N.S. LearnPlan +, p<.05 N.S. +, p<.05 Materials +, p<.001 +, p<.001 +, p<.001 Order +, p<.001 +, p<.001 +, p<.001 Content +, p<.001 +, p<.001 +, p<.001 Grade_AD +, p<.001 N.S. N.S. WBdesign N.S. N.S. N.S. DurationDays +, p<.001 +, p<.001 +, p<.001 TotPart -, p<.001 -, p<.001 -, p<.001 Note: + = a positive effect; - = a negative effect; N.S.= no significant effect 5.31 Our hypotheses are supported in the following findings: · Similar to the results for our model predicting value, participants who registered their activities in their learning plans were also more likely to report increases in K&S. · Assessments of training materials, presentation order, and learning content are all highly significant and positively related to perceptions of increased K&S. · People learn more in longer courses. Individuals who participated in courses with longer duration are more likely to report increased K&S.23 · Class size is a significant predictor of reported increases in K&S. As total participants (TotPart) increase, perceived increases in K&S decrease, however not by much.24 5.32 On the other hand, the results show no significant effects for grade, WBdesign or Interactive. · There were no significant differences in perceived learning between ACS staff and non-ACS staff. Participants were not more likely to report increases in K&S based on their grade level. · Likewise, it made no difference whether the course was designed by the World Bank. 23The effect of duration remains significant even when the model controls for course topic, however there are no significant interactions between course topic and duration (see Annex L, Table L2). 24The effect of class size is significant in the larger model controlling for course topic as well, however there are no significant interactions between course topic and class size (see Annex L, Table L3). 25 · It appears that interactive learning does not directly increase how much participants learn. However, in our next model, we will test whether interactive learning helps people apply what they do learn. A MODEL PREDICTING APPLICABILITY TO JOB 5.33 To this point, we have examined the relevance of staff learning to the Bank's mission along with overall perceived value and increases in knowledge and skill, but we have yet to explore the connection between the two. In the following analyses, we will explore the concept that links training and achieving the Bank's mission by investigating the determinants of staff learning applicability to participants' jobs. 5.34 In other words, to achieve the Bank's mission through staff training it is not only important to deliver high quality, useful courses that increase knowledge and skills, it is also important that participant learning is related to their jobs. Thus, we develop a model predicting applicability of K&S to jobs in equation 4. Applicability= 0 + 1Interactive + 2Grade_AD + 3Motiv_currentwork + 4Motiv_profgrowth +5LearnPlan + 6BankOperations + 7BehavioralSocial + 8ProfessionalTechnology + (eq.4) 5.35 In our model, we attempt to discover how to improve course applicability to jobs through varying activity characteristics and pinpointing participant level differences. Thus, the dependent variable is self-assessments of how the knowledge and skills gained through training are directly applicable to participants' jobs. We predict applicability as a function of instruction approach (interactive learning), grade (ACS or not), participant motivation (Motiv_currentwork, Motiv_profgrowth) and course content type (Bank Operations, Behavioral Social, Professional Technology). 5.36 We hypothesize that the use of interactive planning in an activity would help participants understand how to apply what they learn to their current work. Consequently, we expected interactive planning would positively increase perceptions of the applicability of K&S to people's jobs. 5.37 We also anticipated that individuals who are motivated to take the course to "enhance performance in current/planned assignment" (Motiv_currentwork) would also more be likely to rate the K&S gained through training as directly applicable to their jobs. Likewise, individuals who planned to take the course in their learning plans (LearnPlan) would link the K&S in the activity to their jobs. On the other hand, participants who took the course for professional interest and growth (Motiv_profgrowth), would not be significantly more likely to perceive a link between their jobs and K&S learned in training. 5.38 Additionally, we explore differences in applicability to job by grade grouping. In other words, we test whether staff in ACS grades are more or less likely to draw a connection between the K&S gained in the course and their jobs. 26 5.39 We also investigate whether certain types of courses provide K&S that are more applicable to people's daily work. Our model compares the various types of activity (Bank Operations, Behavioral Social, Professional Technology) to a comparison group (Information Technology).25 The expectation was that Bank Operations and Professional Technology courses would be the most applicable to participants' jobs. 5.40 We estimate the model using three different estimation procedures (See Annex L). This was necessary again to be confident in our estimates, because of the skewed dependent variable. Table 4 shows that our results are consistent across the three estimation procedures. Table 4. Comparing Effects across Estimation Procedures: Models Predicting Applicability to Job Variable Gamma Logit OLS with Logged DV Interactive +, p<.01 +, p<.01 +, p<.01 Grade_AD N.S. N.S. N.S. Motiv_currentwork +, p<.001 +, p<.001 +, p<.001 Motiv_profgrowth N.S. N.S. N.S. LearnPlan +, p<.001 +, p<.001 +, p<.001 Bank Operations -, p<.01 -, p<.01 -, p<.01 Behavioral Social N.S. N.S. N.S. Professional Technology -, p<.05 -, p<.01 -, p<.01 5.41 Recall, our hypotheses were based on the premise that the mindset for taking the course was highly predictive of perceiving the activity to be directly applicable to one's job. Our results show that reported motivation was the strongest predictor of applicability. That is, identifying one's motivation for taking the course as "enhancing performance in current or planned assignment" significantly increase perceived applicability of K&S to job. On the other hand, if participants' motivations were "professional growth or networking," they were less likely to report the K&S as applicable to their job. Following motivation, registering the course in one's learning plan was highly indicative of reporting a direct application of K&S to job. 5.42 Moreover, interactive learning was an instrumental factor in predicting applicability to job. Participants who took courses where the interactive learning approach was used were significantly more likely to report that the K&S they learned were directly applicable to their job.26 5.43 A surprising finding was that Information Technology courses were more likely to be applicable to participants' jobs than Bank Operations courses and Professional Technology courses.27 While this seems counter-intuitive at first glance, it is actually a 25This includes a few executive management courses as well. 26However, the effect of interactive learning is not significant in larger models controlling for interactions with course topic (Annex L, Table L4). 27However, it is important to note that the effect sizes are not very large. 27 logical finding. The methods learned in I.T. courses such as EXCEL can be used on the daily bases whereas concepts learned in a Professional Technology class are not necessarily used on the regular basis, e.g. macro-economic modeling techniques can be used only with the appropriate data after it has been collected, which is often a timely process. Likewise, Bank operations courses may give best practices on what to do on mission but staff are not on mission everyday. On the other hand, staff use Lotus Notes all day long. 5.44 As with the other models, we found no significant differences between participants based on grade level. There is no difference in perceived applicability to job as a result of participants working in ACS. 28 6. TESTING OUR MODEL RESULTS 6.1 The model results described in the previous section point to several key features that enhance perceived training results: duration, class size, activity features, and participant assessments of course features. If these variables are indeed significant predictors of training success, then courses that are rated highly should comprise the characteristics predicted by our models. Thus, in order to test the validity of our models, we compare the characteristics of the ten highest quality and ten lowest quality activities.28 6.2 The ten highest quality activities exemplified the patterns predicted by our models in terms of course maturity, learning plans, and use of an interactive learning approach (Figure 8). They had a higher percentage of previously offered courses (80 percent versus 30 percent). Almost twice as many respondents in the highest quality activities registered the course in their learning plans (56 percent versus 26 percent). Interactive learning was used more often in the highest quality activities than in the lowest (50 percent versus 40 percent). Figure 8. Frequency of course maturity, learning plans, and interactive learning among the 10 highest quality and 10 lowest quality courses, WB staff learning, FY03 90% 80% 70% 56% 50% 50% 40% 30% 26% 30% 10% Lowest Highest Lowest Highest Lowest Highest 10 10 10 10 10 10 Course Maturity Learning Plan Interactive learning 6.3 Additionally, longer courses were characteristic of high quality courses (3.37 days versus 1.25 days; see Figure 9). On average, course duration was two days longer in the ten highest quality activities compared to the ten lowest. 28The highest and lowest activities were selected based on their T-values in a model predicting the average rating on the five quality dimensions (relevance, quality, usefulness, knowledge, and applicability). The independent variables included dummy variables for each course. The excluded category was the median course. Additionally, the model controlled for participant intake variables such as motivation, learning plan, grade level, bank assignment - headquarters, regions, and networks. 29 Figure 9. Average course duration for the 10 highest quality and 10 lowest quality courses, WB staff learning, FY03 4 3.37 3 2 1.25 1 0 Lowest 10 Highest 10 Number of Days 6.4 Further, smaller class sizes were prevalent among high quality courses (Figure 10). Classes were larger among the ten lowest quality courses with an average of 26 participants per activity, versus only 15 participants in the highest quality activities. Figure 10. Average class size for the 10 highest quality and 10 lowest quality courses, WB staff learning, FY03 30 26 20 15 10 0 Lowest 10 Highest 10 Number of Participants 6.5 The ten highest quality courses were rated significantly better than the ten lowest quality activities on course content (4.73 vs. 3.58), order of presentations (4.76 vs. 3.71), and materials used (4.71 vs. 3.45) (Figure 11). Figure 11. Average rating of content, order, and materials by participants in the 10 highest quality and 10 lowest quality courses, WB staff learning, FY03 6.00 4.73 4.77 4.71 5.00 3.71 4.00 3.58 3.45 Rating 3.00 2.00 Average1.00 0.00 Content Order Materials Lowest 10 Highest 10 30 7. QUALITATIVE RESULTS: PARTICIPANT FEEDBACK 7.1 We asked participants open-ended questions in order to get an understanding of what worked best in the training, and how to improve future learning activities. The results of this qualitative feedback point to the importance of interactive learning, circulating preparatory materials in advance of the course, and increasing activity duration. Interactive Learning 7.2 Interactive learning activities worked best in FY03 and should be used to improve training in FY04. Participant feedback confirmed our statistical findings that interactive learning is critical for increasing K&S and understanding how K&S are applicable to participants' jobs. Specifically, participants cited interactive activities, team-based learning, and practical sessions (see examples below). Interactive Activities Small Group Activities Hands-on Exercises "Interaction "Teamwork ­ break out groups" "practical exercises (computer-based) between especially when dealing with modules and attendees and estimation techniques" presenters" "Discussion, "Case study and group work make "would be good to have some hands-on Q&A period, us recall and think of what we have exercises to be able to know how to apply forced learned these days and help us to the tools in a real world" participation" apply the knowledge into our work "Role playing" "In-class group assignments "case studies" Course Duration 7.3 Course duration should be increased to enhance training according to participants' comments. The qualitative results support our findings showed that increasing course days increases self-reported levels of K&S. Quotes from participants' state that: · "The course was two short" · "The course was only one day but requires at least two weeks to be properly covered" 31 Course Materials 7.4 Providing course materials in advance would improve future learning. Participants reported that they wanted to be more prepared for the course to maximize their benefits. Respondents indicated the need for receiving information to study in advance of the activity in some quotes below: · "Assign reading prior to seminar" · "More theoretical and background information including policies and guidelines in preparing grants" · "Prior preparation i.e. background material, research and bring notes" · "Course materials should be distributed in advance if possible" · "Advance circulation of materials" Learning Support 7.5 Participants suggested various types of support for applying their new knowledge and skills including: computer software, reading materials, follow-up training, and managerial encouragement. Suggestions for improving support to participants include: · "Quantitative tools i.e. statistical programs and software and access to databases" · "Providing additional background materials and references for future use" · "Continued access to instructors for expertise via email or Web Site" · "Advanced follow-up training offered via e-classroom, video-clinics" · "Managerial support in the way of recognition and encouragement" · "More challenging assignments to apply skills" · "Developmental assignment to apply skills" 32 8. RECOMMENDATIONS AND CONCLUSIONS 8.1 Based on the results of our evaluation, we draw the following conclusions for FY03 and subsequently make recommendations for FY04. We outline our recommendations for trainers, participants, and IEG below. CONCLUSIONS AND RECOMMENDATIONS FORTRAINERS 8.2 On the whole, participant ratings of training in FY03 remained the same as in FY02. Given the positively biased sample in FY02, the fact that ratings stayed as high in FY03 as they were in FY02 suggests that courses in FY03 improved from FY02. 8.3 Characteristics among highly rated activities included: smaller classes sizes; longer course duration; higher use of interactive learning methods; and more participants who registered their courses in the learning plan. 8.4 Specifically, our analyses show that Information Technology courses were perceived as less relevant to the Bank's mission than Bank Operations, Professional Technical, and Management courses. Thus, IT trainers should make greater attempts to impress the importance of their training to Bank's goals. 8.5 We recommend that trainers review their courses to determine whether there is enough time to cover the content thoroughly and consider capping registration, based on our finding that K&S can be increased through longer courses and smaller classes (along with attention to course materials, course content and presentation order). RECOMMENDATIONS FOR PARTICIPANTS 8.6 Participants should use their learning plans. Since the main motivation for taking courses is to enhance performance in current work, participants should really be registering the courses they take in their learning plans. This is important because our analyses indicate that participants who do register their courses in their learning plans report learning more K&S. Additionally, we find that these participants are more likely to report that the K&S they obtained were applicable to their job. RECOMMENDATIONS FOR IEG Extend Response Option Scale on Questionnaire 8.7 We recommend that IEG extend the response option scale to seven points in the level one questionnaire, while continuing to report individual evaluation results on the 33 one to five scale. The data collected at the detailed seven point level will be collapsed into the five point scale so activity sponsors will receive the level one results and ratings with which they are familiar, and the same comparisons can still be made with the ASTD benchmarks.29 8.8 However, the annual report will be based on data using the seven point scale. Data based on the longer scale would provide more variance for data analysis, thereby, allowing us to examine shades of gray instead of just black and white results. In other words, we will be able to examine what enhances training from better to great, not just from bad to good because our dependent variable would no longer be forced into a dichotomous scale (ratings of 4 and 5 versus 3 and below). Implement Stratified Random Sample Selection Procedure 8.9 Based on our experience with a simple random sample selection process used in FY03, we recommend moving to a stratified random sampling design. Stratifying by activity sponsor should produce sub-samples of learning events that are large enough in size to make comparisons requested by the Learning Board. These data can then be weighed to match the distribution of actual course offerings for the analyses conducted in the annual report. 8.10 Because of our concentrated effort to obtain a scientifically selected sample of courses for evaluation, we cannot include self-nominated evaluations in our sample (See Annex M for IEG policy on self-nominated level one staff evaluations). 29Collapsing the data from seven to five points simply means recoding 7 to 5; 6 and 5 to 4; 4 to 3; 3 and 2 to 2; and 1 would remain one. 34 ANNEXES 35 36 ANNEX A: ACTIVITY QUESTIONNAIRE Level 1 Evaluation of Formal Training Training Offering Description Instructions for Training Administrators Please complete this form in consultation with the training Task Manager for the learning offering to be delivered as part of the World Bank Staff Training Program. Kindly return via e-mail the completed form along with the training announcement and agenda before the training to Diana Arango, WBIEG. We will then produce the evaluation forms for you and (time and location permitting) collect the evaluation data. 1 Training title: 2 Training code (from Learning Catalogue, e.g., OPS201-02-007): Start (mm/dd/yyyy) End (mm/dd/yyyy) 3 Training date(s): 4 Number of days: 5 Which of the following is the primary training sponsor? (Select ONE) Networks Regions Others ACS AFR DEC ESD EAP EXT FSD/FSE ECA HRS HDN LCR LEG ISN/ISG MNA P&I OPCS SAS RM PREM Other, specify here --> PSI 6 Training history (Select one) First time this training will be delivered Second time this training will be delivered Previously delivered twice or more 7 In which one of the following categories does the training content best fit? Bank Operations Executive Management Professional & Technical Behavioral & Social ( interpersonal communication skills) Information & Technology Trust Funds 8 How many participants are expected in this session? 9 Is this training mainly for Managers (Unit, Division or above)? Yes No 10 Was this training mostlydesignedby the World Bank? Yes No 11 Will this trainingmostlybe delivered by the World Bank? Yes No 12 Would you like to use the optional instructors' evaluation form? Yes No 13 At the end of the session, will there be 1 Internet-connected PC per participant? Yes No 14 Will the training approach primarily employ "Formal" or "Action" learning? Formal Action 37 ANNEX B: PARTICIPANT QUESTIONNAIRE Level 1 Evaluation Questionnaire for Formal Training Please assess the course to help to improve future training. To answer, please fill in the circle like thisl. If you wish to change an answer, fully erase it or draw an X over the unwanted mark and fill in the circle indicating your preferred answer. Please fill in only one circle per question. 1. Training Title: 2. Training Dates: 3. What was your main reason for taking this training? (Fill only one circle.) ? To enhance performance in current/planned assignment ? To network and share information ? For professional interest and growth ? Other, specify: ______________________________________________________________________________ 4. Do you work for the World Bank? ? Yes, at HQ ? Yes, in the Field ? No If no, go to Q8. 5. For what part of the World Bank Group do you primarily work? (Fill only one circle.) Regional Vice-Presidencies Network Anchors Other ? AFR ? LCR ? ESSD ? OPCS ? DEC ? IFC ? EAP ? MNA ? FSE ? PREM ? HRS ? MIGA ? ECA ? SAR ? HDN ? PSI ? LEG ? Other, specify: __________ ? ISG ? WBI 6. What is your grade level? ? GA-GD ? GE ? GG ? STC ? STT ? GF ? GH or above ? Other, specify:______ 7. Was this training agreed upon in your Learning Plan (LP)? ? Yes ? No ? I don't have a LP ? I don't know ____________ Using the scale on the right, please rate each question/statement below. Not Very at all much N/A 8. To what extent did the training fulfill your learning needs? ? ? ? ? ? ? 9. To what extent did the training achieve its announced objectives? ? ? ? ? ? ? Very Very poor good N/A 10. How would you rate the training content or subject matter? ? ? ? ? ? ? 11. How would you rate the order in which the content was presented? ? ? ? ? ? ? 12. How would you rate thematerials used during the training? ? ? ? ? ? ? Very Very low high N/A 13. How would you rate the overall quality of the training? ? ? ? ? ? ? 14. How would you rate the overall usefulness of the training? ? ? ? ? ? ? 15. How would you rate therelevance of this training to the Bank's mission? ? ? ? ? ? ? Strongly Strongly disagree Neither agree 16. My knowledge/skills increased as a result of this training. ? ? ? ? ? 17. The knowledge/skills gained through this training aredirectly applicable to my job. ? ? ? ? ? 38 N Please note the change in scale and adjust your rating pattern accordingly. · To assess how balanced the training was, please rate each aspect below with respect to quantity. · Attention to theoretical content ? ? ? ? ? ? · Attention to practical content ? ? ? ? ? ? · Time for instructors' presentations ? ? ? ? ? ? · Time for your participation ? ? ? ? ? ? · Pace of the training(insufficient = too slow; excessive=too ? ? ? ? ? ? fast) Please print or writeclearly. · What knowledge/skills acquired from the training will you nowapply on the job? (List the most important three.) a. ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ __________ b. ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ __________ c. ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ __________ · What type of support would you need to apply the newly acquired knowledge/skills? ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ · What workedbest in this training? ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ · What would you recommend toimprove this training in the future? ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________ _______________ Thanks for completing the questionnaire. Please leave it in the questionnaire box before leaving or send it to "L1 Eval" MSN J4 -401. 39 ANNEX C: COURSE AUDIENCE: DISTRIBUTION OF RESPONDENT UNIT, BY ACTIVITY SPONSOR NETWORKS Respondents by Unit in Courses Sponsored by ACS Respondents by Unit in Courses Sponsored by ESSD 20 30 17.24 15.52 23.53 13.79 13.79 20.00 20 10 8.05 7.47 Respondents Respondents 10.00 % % 10 4.60 7.65 7.06 6.47 6.47 3.45 3.45 2.87 2.87 5.29 4.71 4.12 1.72 1.72 1.15 1.15 2.35 0.57 0.57 0.59 0.59 0.59 0.59 0 0 IFC EAP Other SAR WBI ECA LCR MNA PSI ESSD HDN LEG OPCS DEC HRS FSE AFR LCR LEG PREM ESSD ECA EAP SAR MNA WBI PSI DEC HDN HRS MIGA OPCS ParticipantUnit ParticipantUnit Respondents by Unit in Courses Sponsored by ISG Respondents by Unit in Courses Sponsored by OPCS 30 30 28.04 20.65 20 20 16.22 12.9812.39 10.03 Respondents Respondents % 10 10 8.49 7.01 7.01 % 6.78 5.60 6.64 5.54 5.17 4.80 4.43 4.43 2.95 4.06 2.65 2.36 2.06 1.18 1.18 1.18 2.58 2.21 2.21 2.21 2.21 1.85 0.880.29 0.290.29 0.74 0.37 0 0 PSI AFR EAP LCR PSI IFC WBI LEG FSE Other AFR EAP IFC WBI ECA SAR HDN DEC HRS ECA HRS LCR Other MNA OPCS ESSD SAR FSE HDN MNA LEG DEC PREM MIGA OPCS ESSD PREM ParticipantUnit Participant Unit Respondents by Unit in Courses Sponsored by PREM Respondents by Unit in Courses Sponsored by PSI 30 40 30.72 20.15 30 20 17.16 20 12.09 Respondents 9.85 10 8.66 Respondents 11.4510.8410.24 % 6.27 % 7.23 5.22 10 7.83 4.18 5.42 3.58 3.43 2.39 2.24 2.24 3.01 2.41 1.34 0.60 1.811.811.811.811.811.200.60 0.30 0.15 0.15 0 0 AFR ECA SAR PSI IFC PSI IFC PREM Other DEC EAP MNA WBI LCR FSE WBI ESSD HDN HRS OPCS MIGA AFR Other SAR ECA EAP LCR ESSD MIGA MNA HDN HRS OPCS PREM Participant Unit Participant Unit 40 Respondents by Unit in Courses Sponsored by HDN 40 35.17 30 25.52 20 Respondents % 9.66 8.28 10 4.83 3.45 2.76 2.76 2.07 1.38 1.38 1.38 1.38 0 HDN AFR ECA LCR EAP MNA DEC WBI IFC SAR Other OPCS PREM Participant Unit REGIONS Respondents by Unit in Courses Sponsored by AFR Respondents by Unit in Courses Sponsored by ECA 70 100 65.91 90.00 90 60 80 50 70 60 40 50 30 Respondents Respondents 40 % % 20 30 10.23 20 7.95 10 3.41 3.41 2.27 2.27 10 1.14 1.14 1.14 1.14 3.33 3.33 3.33 0 0 AFR DEC ECA PSI PREM Other ESSD OPCS HDN MNA SAR ECA EAP HDN SAR Participant Unit Participant Unit Respondents by Unit in Courses Sponsored by LCR Respondents by Unit in Courses Sponsored by SAR 80 69.70 60 70 48.84 50 60 36.05 50 40 40 30 30 Respondents 20 20 % 8.14 10 10.10 2.33 2.33 2.33 10 3.03 3.03 3.03 3.03 3.03 1.01 1.01 1.01 1.01 1.01 0 0 SAR EAP PREM ESSD OPCS Other Participant Unit Participant Unit 41 OTHERS Respondents by Unit in Courses Sponsored by EXT Respondents by Unit in Courses Sponsored by FPS 40 60 55.17 34.48 50 30 40 20.69 20 30 Respondents Respondents 20 % 10 6.90 6.90 6.90 6.90 % 13.79 3.45 3.45 3.45 3.45 3.45 10 6.90 3.45 3.45 3.45 3.45 3.45 3.45 3.45 0 0 IFC Other AFR ECA LEG SAR EAP HRS PSI MNA PREM Other ECA AFR EAP FSE IFC LCR PSI WBI Participant Unit PREM Participant Unit Respondents by Unit in Courses Sponsored by GSD Respondents by Unit in Courses Sponsored by HRS 96.00 20 100 18.30 90 15.04 80 70 11.23 60 9.24 50 10 7.25 6.52 Respondents 40 Respondents 5.62 % 30 % 4.17 20 3.08 2.72 2.72 2.54 2.36 1.99 1.81 1.63 1.63 1.27 10 4.00 0.91 0 0 Other ECA AFR HRS LCR SAR EAP IFC MNA WBI DEC LEG HDN PSI FSE Other WBI OPCS MIGA ESSD PREM Participant Unit Participant Unit Respondents by Unit in Courses Sponsored by LEG Respondents by Unit in Courses Sponsored by RM 110 100.00 20 100 16.13 90 14.52 12.90 80 11.29 70 60 10 6.45 50 4.84 3.23 3.23 Respondents 40 Respondents 4.844.844.84 3.23 % 30 % 3.23 1.61 1.61 1.611.61 20 10 0 0 LCR AFR IFC PSI LEG Other DEC ECA FSE HRS HDN LEG WBI EAP SAR PREM ESSD OPCS Participant Unit Participant Unit 42 Respondents by Unit in Courses Sponsored by WBI 80 69.88 70 60 50 40 Respondents 30 %20 10 7.23 6.02 4.82 3.61 2.41 1.20 1.20 1.20 1.20 1.20 0 WBI ECA Other LCR FSE AFR DEC HRS IFC OPCS ESSD Participant Unit 43 ANNEX D: DISTRIBUTION OF RESPONDENTS BY GRADE AND COURSE CONTENT Percent of Respondentds by Grade Level and Content Type 60.00 54.10 50.00 45.65 46.51 44.54 43.49 41.63 40.25 39.59 40.00 37.40 35.28 33.64 30.13 30.22 30.00 27.91 26.60 26.64 Respondents 24.45 23.26 21.38 % 20.00 15.12 15.28 11.94 11.08 9.69 9.53 10.20 10.00 7.76 6.37 4.42 4.184.35 3.13 3.49 3.73 2.33 1.39 0.13 0.28 0.49 0.61 1.00 0.82 0.00 GA-GD GE GF GG GH or above Other STC STT Grade Level BANK OPERATIONS BEHAVIORAL EXECUTIVE MANAGEMENT INFO.TECHNOLOGY OTHER PROFESSIONAL/TECHNICAL 44 ANNEX E: DISTRIBUTION OF RESPONDENTS BY UNIT IN ACTIVITY CONTENT TYPES Respondents in Bank Operations Courses by Unit Respondents in Behavioral Courses by Unit 2 5 14 19.78 12 11.49 10.9210.92 2 0 10.3410.34 10 15 13.73 7.76 8 10.54 5.755.46 10 9.24 6 8.20 Respondents 6.656.30 Respondents 3.45 % 5.27 3.16 3.16 % 4 2.87 2.872.59 2.59 5 3.37 2.30 3.112.682.332.16 1.44 1.731.55 1.30 2 1.15 1.15 1.21 0.350.350.17 0.29 0 0 Respondent's Unit Respondent's Unit Respondents in Executive Management Courses by Respondents in Information Technology Courses by Unit Unit 3 0 18 26.23 16 15.47 2 5 14 2 0 12 10.57 10.19 10 15 7.92 11.48 8 7.17 9.84 6.42 6.42 Respondents 10 8.20 Respondents 6 4.91 4.91 % 6.56 6.56 6.56 % 4.53 4.15 4 3.40 3.02 3.02 5 3.28 3.28 3.28 3.28 3.28 2.64 1.64 1.64 1.64 1.64 1.64 1.89 1.89 2 1.13 0.38 0 0 Respondent's Unit Respondent's Unit Respondents in Professional / Technical Courses by Unit 18 16 15.4215.13 14 12 9.77 10 7.95 8 7.09 6.90 Respondents 5.65 6 % 4.69 4.694.31 3.743.64 4 3.45 2.87 1.82 2 1.44 0.57 0.48 0.38 0 Respondent's Unit 45 ANNEX F: PERCENTAGE OF RESPONDENTS RATING COURSES 4 OR 5 Percentage of Respondents Rating Courses 4 or 5 Materials used 76% Applicability to Job 78% Learning needs met 79% Announced objectives 82% Overall Usefulness 83% Presentation Order 84% Content 84% Relavance Bank's Mission 85% Overall Quality 85% Knowledge and Skills 87% 50% 60% 70% 80% 90% 100% 46 ANNEX G: VARIABLE DEFINITIONS Variable Name Definition Coding Relevance Participant ratings of the training's relevance to Higher scores indicate stronger relevance to Bank's mission; Bank's mission Value Average of participant ratings of training quality Higher scores indicate higher and usefulness quality/usefulness K&S Participant ratings of Knowledge and Skills Higher scores indicate higher ratings of K&S Applicability Participant rating of the training's applicability Higher scores indicate higher ratings of to job applicability Bank Operations Course content focuses on Bank operational 1 if Bank Operations; zero otherwise work Professional Course content focuses on professional and 1 if Professional Technical; zero otherwise Technology technical topics (e.g. macro-economics) Behavioral Social Course content type is behavioral and social 1 if Behavioral Social; zero otherwise (e.g. communication etc.) Executive Course content focuses on management 1 if Management; zero otherwise Management PrevOffer2 Course was previously offered two or more 1 if offered two or more times; zero otherwise times WBdesign World Bank designed the course 1 if designed by WB; zero otherwise TotPart Total participants in the course Number of participants Duration Course duration measured in days Number of days in training LearnPlan Course is registered in participant's learning 1 if in learning plan; zero otherwise plan Content Participant rating of course content Higher scores indicate higher course content quality Order Participant rating of the order in which topics Higher scores indicate better ratings of the were presented sequence in which the material was presented Materials Participant rating of the course materials Higher scores indicate higher ratings of course material Interactive Interactive learning was used in course 1 if interactive learning was used; zero otherwise Grade_AD Participant grade level A, B, C, or D 1 if participant was in grades A -D; zero otherwise Motiv_currentwork Participants' motivation for taking the course 1 if motivation was current or planned work was to enhance performance in current work assignment; zero otherwise load Motiv_profgrowth Participants' motivation for taking the course 1 if motivation was professional growth; zero was professional growth otherwise 47 ANNEX H: MODELS PREDICTING RELEVANCE TO BANK'S MISSION Table H1. Gamma Coefficients Predicting Relevance to Bank's Mission Transformed Variable Standard Error e ß Percentage Change from the Mean Bank Operations 0.0258* .0116 1.0261 2.61% Professional Technology 0.0102 .0117 1.0103 1.03% Behavioral Social 0.0431** .0125 1.0440 4.40% Executive Management 0.0591** .0211 1.0609 6.09% Prevoffer2+ 0.0250** .0065 1.0253 2.53% WBdesign 0.0102 .7957 1.0103 1.03% Intercept 1.4517** .0109 µ=4.2704 N= 3,326 Chi2= 83.1264 Scaled Chi2= 2711.0369 Deviance= 102.5037 Scaled Deviance=2711.0369 Log Likelihood= -3848.65221 ** = p < .001, * = p < .05 Table H2. Logit Coefficients Predicting the Probability of Relevance to Bank's Mission (scores 4 and 5) Transformed Variable Standard Error eß/(1+eß) Increased Probability of 4 or 5 Rating Bank Operations 0.5203* .2127 0.63 62.72% Professional Technology 0.2951 .2084 0.57 57.32% Behavioral Social 1.0779** .2643 0.75 74.61% Executive Management 1.9146** .7375 0.87 87.15% Prevoffer2+ 0.4962* .1353 0.62 62.16% WBdesign 0.1918 .1677 0.55 54.78% Intercept 1.4425** .1959 N= 3,326 Chi2= 42.3063 -2 Log Likelihood=2124.411 ** = p < .001, * = p < .01 48 ANNEX I: MODELS PREDICTING QUALITY AND USEFULNESS RATINGS Table I1. Gamma Coefficients Predicting Quality/Usefulness Ratings Transformed Variable Standard Error e ß Percentage Change from the Mean TotPart -0.0002 .0001 0.9998 -0.02% Duration 0.0014 .0012 1.0014 0.14% LearnPlan 0.0100* .0045 1.0101 1.01% Content 0.4957** .0147 1.6416 64.16% Order 0.1860** .0144 1.2044 20.44% Materials 0.1795** .0127 1.1966 19.66% Interactive 0.0082 .0043 1.0082 0.82% WBdesign 0.0067 .0050 1.0067 0.67% Prevoffer2 0.0102* .0049 1.0103 1.03% Intercept 0.7269** .0121 µ=2.0687 N= 3,326 Chi2= 41.62 Scaled Chi2= 3108.8529 Deviance= 44.6233 Scaled Deviance=3333.4205 Log Likelihood= -2321.1397 ** = p < .001, * = p < .05 Table I2. Logit Coefficients Predicting the Probability of Positive Ratings of Quality/Usefulness (scores 4 and 5) Increased Transformed Probability of Variable Standard Error eß/(1+eß) 4 or 5 Rating TotPart -0.0070* .0028 0.50 49.83% Duration 0.0823 .0446 0.52 52.06% LearnPlan 0.3695* .1438 0.59 59.13% Content 7.0563** .4505 1.00 99.91% Order 3.5315** .4058 0.97 97.16% Materials 3.7596** .3760 0.98 97.72% Interactive 0.2292 .1340 0.56 55.71% WBdesign 0.2510 .1566 0.56 56.24% Prevoffer2 0.2499 .1526 0.56 56.22% Intercept -9.4320** .4689 N= 3,326 Chi2= 1465.0039** -2 Log Likelihood=1740.331 ** = p < .001, * = p < .01 49 Table I3. OLS Coefficients Predicting Quality/ Usefulness Ratings Logged Variable Standard Error TotPart -0.0002 .0001 Duration 0.0011 .0012 LearnPlan 0.0097* .0046 Content 0.5159** .0150 Order 0.1955** .0148 Materials 0.1837** .0131 Interactive 0.0091* .0045 WBdesign 0.0077 .0051 Prevoffer2 0.0114* .0050 Intercept 0.6912** .0123 N= 3,326 R2=.62 F=606.51 P<.0001 ** =p < .001, * =p<.05 50 ANNEX J: MODELS PREDICTING PERCEIVED KNOWLEDGE AND SKILLS (K&S) INCREASE Table J1. Gamma Coefficients Predicting Increases in Knowledge and Skills Transformed Percentage Change Variable Standard Error e ß from the Mean Interactive 0.0010 .0061 1.0010 0.10% LearnPlan 0.0132* .0063 1.0133 1.33% Materials 0.0924** .0180 1.0968 9.68% Order 0.1621** .0201 1.1760 17.60% Content 0.4032** .0208 1.4966 49.66% Grade_AD 0.0189** .0071 1.0191 1.91% WBdesign 0.0119 .0070 1.0120 1.20% DurationDays 0.0068** .0015 1.0068 0.68% TotPart -0.004** .0001 0.9960 -0.04% Intercept .8769** .0169 µ=2.40 N= 3,326 Chi2= 74.3390 Scaled Chi2= 2817.74 Deviance= 88.13 Scaled Deviance=3340.62 Log Likelihood= -3416.49 ** = p < .001, * = p < .05 Table J2. Logit Coefficients Predicting the Probability of Increases in Knowledge and Skills (scores 4 and 5) Transformed Increased Probability of Variable Standard Error eß/(1+eß) 4 or 5 Rating Interactive -0.1763 .1341 0.46 45.60% LearnPlan 0.1308 .1461 0.53 53.27% Materials 1.0474** .3606 0.74 74.03% Order 2.1507** .3877 0.89 89.57% Content 4.7336** .4049 0.99 99.13% Grade_AD 0.2275 .1853 0.55 55.66% WBdesign 0.1619 .1578 0.54 54.04% DurationDays 0.2014** .0475 0.55 55.02% TotPart -0.0065* .0027 0.50 49.84% Intercept -4.1395** .3481 N= 3,326 Chi2= 625.5964** -2 Log Likelihood=1790.694 ** = p < .001, * = p < .01 51 Table J3. OLS Coefficients Predicting Increasesin Knowledge and Skills Logged Variable Standard Error Interactive 0.0012 .0065 LearnPlan 0.0151t .0066 Materials 0.0978** .0191 Order 0.1624** .0214 Content 0.4603** .0218 Grade_AD 0.0184* .0075 WBdesign 0.0133 .0074 DurationDays 0.0069** .0016 TotPart -0.0004* .0002 Intercept .81042** .0177 N= 3,326 R2=.35 F=203.61 P<.0001 ** =p < .001, * =p < .01, =p<.05 t 52 ANNEX K: MODELS PREDICTING TRAINING APPLICABILITY TO JOB Table K1. Gamma Coefficients Predicting Applicability of New Skills to Job Transformed Variable Standard Error e ß Percentage Change from the Mean Interactive 0.0265* .0092 1.0268 2.68% Grade_AD 0.0058 .0099 1.0058 0.58% Motiv_currentwork 0.1154** .0132 1.1223 12.23% Motiv_profgrowth -0.0036 .0137 0.9937 -0.63% LearnPlan 0.0397** .0089 1.0404 4.04% Bank Operations -0.0349* .0127 0.9657 -3.43% Behavioral Social 0.0068 .0163 1.0068 0.68% Professional Technology -0.0292t .0131 0.9712 -2.88% Intercept 1.3545** .4576 µ=3.87 N= 3,326 Chi2= 138.8658 Scaled Chi2= 2614.2072 Deviance= 178.2397 Scaled Deviance=3355.4376 Log Likelihood= -4496.5531 **=p< .001, *=p<.01, =p<.05 t Table K2. Logit Coefficients Predicting the Probability of New Skills Being Applicable to Job (scores 4 and 5) Transformed Increased Probability of Variable Standard Error eß/(1+eß) 4 or 5 Rating Interactive 0.2892* .1050 0.57 57.18% Grade_AD -0.1840 .1172 0.45 45.41% Motiv_currentwork 1.1711** .1392 0.76 76.33% Motiv_profgrowth -.1216 .1323 0.47 46.96% LearnPlan 0.5621** .1118 0.64 63.69% Bank Operations -0.4726* .1625 0.38 38.40% Behavioral Social 0.1050 .2169 0.53 52.62% Professional Technology -0.4338* .1652 0.39 39.32% Intercept 1.0677** .1927 0.74 N= 3,326 Chi2= 280.6233** -2 Log Likelihood=3280.429 ** = p < .001, * = p < .01 53 Table K3. OLS Coefficients Predicting Applicability of New Skills to Job, Logged Variable Standard Error Interactive 0.0305* .0099 Grade_AD 0.0053 .0108 Motiv_currentwork 0.1450** .0143 Motiv_profgrowth 0.0111 .0150 LearnPlan 0.0474** .0097 Bank Operations -0.0407* .0138 Behavioral Social -0.0009 .0178 Professional Technical -0.0395* .0143 Intercept 1.3093** .0184 N= 3,326 R2=.10 F=43.76 P<.0001 ** =p < .001, * =p < .01 54 ANNEX L: MODELS CONTROLLING FOR ADDITIONAL VARIABLES Table L1. Logit Coefficients Predicting the Probability of Positive Ratings of Quality/Usefulness (scores 4 and 5) Controlling for Course Topic Variable Standard Error TotPart -0.0033 .0028 Duration 0.1444** .0491 LearnPlan 0.2672* .1471 Content 7.0651*** .4550 Order 3.5398*** .4095 Materials 3.7590*** .3799 Interactive 0.1444 .1444 WBdesign 0.5468** .1830 Prevoffer2 0.0626 .1650 Professional Technical -1.3252*** .4111 Bank Operations -1.1607** .4072 Behavioral Social -0.3005 .4558 Information Technology -0.1703 .4974 Intercept -8.7532*** .6192 N= 3,326 Chi2= 1497.3120** -2 Log Likelihood=1708.023 *** = p < .001, ** = p < .01, *p<.10 Table L2. Logit Coefficients Predicting the Probability of Increases in Knowledge and Skills (scores 4 and 5) Controlling for Course Topic and Duration Interactions Variable Standard Error Interactive -0.0938 .1448 LearnPlan 0.1308 .1483 Materials 0.9783** .3635 Order 2.1144*** .3904 Content 4.7817*** .4092 Grade_AD 0.1645 .1926 WBdesign 0.4246* .1881 DurationDays 0.7628* .4631 TotPart -0.0051* .0028 Professional Technical (PT) 0.9115 .7369 Bank Operations (BO) 0.8225 .7303 Behavioral Social (BS) 1.3712* .7858 Information Technology (IT) 1.9077* .8550 Duration Days * PT -0.4976 .4720 Duration Days * BO -0.5397 .4677 Duration Days * BS -0.6808 .4807 Duration Days * IT -0.8887 .5512 Intercept -5.3809** .8201 N= 3,326 Chi2= 635.0365*** -2 Log Likelihood=1781.254 *** = p < .001, ** p < .01. *P<.10 55 Table L3. Logit Coefficients Predicting the Probability of Increases in Knowledge and Skills (scores 4 and 5) Controlling for Course Topic and Class Size Interactions Variable Standard Error Interactive -0.0654 .1494 LearnPlan 0.1819 .1519 Materials 0.9978** .3662 Order 2.0543*** .3944 Content 4.8142*** .4116 Grade_AD 0.1453 .1939 WBdesign 0.3191 .1974 DurationDays 0.2119*** .0497 TotPart -0.0365* .0225 Professional Technical (PT) -0.7192 .6219 Bank Operations (BO) -0.5078 .6218 Behavioral Social (BS) -0.2416 .7265 Information Technology (IT) 0.9582 .8422 TotPart * PT 0.0374 .0250 TotPart * BO 0.0278 .0229 TotPart * BS 0.0236 .0292 TotPart * IT -0.0448 .0409 WB -0.5343* .2548 HQ 0.1895 .1621 Region 0.0066 .1630 Network 0.2920 .2004 Motiv_currentwork 0.3676* .1929 Motiv_profgrowth 0.2717 .1970 Intercept -3.7983*** .7476 N= 3,326 Chi2= 652.7359*** -2 Log Likelihood=1763.555 *** = p < .001, ** p < .01. *p<.10 56 Table L4. Logit Coefficients Predicting the Probability of New Skills Being Applicable to Job (scores 4 and 5) Controlling for Course Topic and Class Size Interactions Variable Standard Error Interactive -0.3757 .6834 Grade_AD -0.0918 .1891 Motiv_currentwork 0.1411 .3092 Motiv_profgrowth -0.0952 .3254 LearnPlan -0.0421 .1720 Bank Operations (BO) -0.6068 .5474 Behavioral Social (BS) 0.5602 .8350 Professional Technology (PT) -0.7165 .5320 Information Technology (IT) -0.3290 .5639 Interactive * BO 0.3602 .7472 Interactive * BS 0.4624 .9832 Interactive * PT 0.6191 .7750 Interactive * IT 0.6066 .8589 WBdesign 0.1297 .2266 PrevIOffer 0.0273 .1993 Content 0.0486 .6098 Order 0.5928 .6119 Materials 0.3524 .5242 TotPart -0.0395** .0083 DurationDays 0.0128 .0523 WB -0.0896 .3573 HQ 0.0728 .2465 Region 0.0481 .1904 Network -0.0343 .2380 Intercept -2.4035** .8239 N= 3,326 Chi2= 73.6833** -2 Log Likelihood=1314.892 ** = p < .001, * = p < .01 57 ANNEX M: IEG POLICY ON SELF-CONDUCTED LEVEL 1 STAFF EVALUATIONS IEG has developed a methodology that provides results for a random sample of courses; these results, therefore, can generalize to the total population of courses. This makes for a robust presentation to the Learning Board. Thus, IEG will continue to support trainers in their individual endeavors to improve their courses, by: · providing trainers with evaluation tools (WORD documents with questionnaires and EXCEL workbook that automatically tabulates responses after they are entered) · teaching course administrators one-on-one how to process their own evaluations. · providing support via email and telephone in answering data processing questions. Data from self-evaluations cannot be added to the data base of the random sample of courses because it would defeat the purpose of this level of evaluation, deny robust data necessary for reporting purposes, and render IEG's actual data suspect. The reasons for this is as follows: 1. A self-selected sample is not representative of the total population. 2. The data integrity could be comprised if evaluations are not conducted under IEG's direct supervision · In the worst-case scenario, data could be manipulated to reflect more positive responses than were actually reported in the data. · A likely case would be simple data entry mistakes. IEG's analysts are hired specifically for their data processing abilities and attention to accuracy in processing data. · Further, IEG's procedure for data processing has put in place several checkpoints to guarantee the accuracy of its data (e.g. cross-checking); 3. The sample would be biased by introducing self-nominated course evaluations into the data base. · If sponsors choose which courses are evaluated, it is likely that evaluation results will be skewed due to (i) the inclusion of all "good" courses (frequently offered, evaluated positively in the past) and (ii) the exclusion of all courses that need improvement or are relatively new · The courses evaluated would no longer represent the courses offered, which would effectively remove the meaning of evaluation (that is, its ability to generalize to non-evaluated courses). 58 · The final picture of staff learning activities will be skewed, putting us back to the FY01 situation that we have been moving away from. 4. IEG's credibility and reputation as an objective evaluation unit could suffer from reporting results on data that are provided by those being evaluated, regardless of the circumstances. (this may not be important to the LB, but is important to us) 5. The resources provided by the LB for L-1 evaluations could be used inefficiently, diverting resources away from a random sample of courses to "self nominated" courses. Ultimately, this could lead to fewer courses being evaluated by IEG overall. · ?Fewer course evaluations would mean a less representative sample (resulting in more grouping of sponsors e.g. PREM into PT). 59