CGIAR GENDER PROGRAM WORKING PAPER, NO 23 23683 SUMMAR Y REPORT CIMMYT PILOT ON MULTI-SOURCE PERFORMANCE ASSESSMENT ALE COPY Prepared by Linda Spink Deborah Merrill-Sands Krista Baldini Marisa de la 0 May 1999 CGIAR Secretariat Center for Gender in Organizations (CGO) World Bank Simmons Graduate School of Management Washington, D.C. Boston, MA, USA LIST OF GENDER STAFFING PROGRAM WORKING PAPERS Working Paper, No. 1 Status of Internationally-Recruited Women in the ][nternational Agricultural Research Centers of the CGIAR, Deborah Merrill-Sands and Pammi Sachdeva; October 1992. Working Paper, No. 2 Spouse Employment in Organizations Around the World: A Toolkit for Developing Policies and Practices; Madelyn Blair; December 1992. Working Paper, No. 3 Spouse Employment at IRRI: A Case Study; Deborah Merrill-Sands; March 1993. Working Paper, No. 4 Strengthening the Recruitment of Women Scientists and Professionals at the International Agricultural Research Centers: A Guidelines Paper; Sarah Ladbury; October 1993. Working Paper, No. 5 Recruitment Resources in Europe: A List of Professional Organizations; Stella Mascarenhas-Keys and Sarah Ladbury; October 1993. Working Paper, No. 6 Filipino Women Scientists: A Potential Recruitment Pool for International Agricultural Research Centers; ISNAR and PCARRD; October 1993. Working Paper, No. 7 Recruitment Resources in the United States: A List of Professional Organizations; Bonnie Folger McClafferty and Deborah Merrill-Sands; January 1994. Working Paper, No. 9 CGLAR Human Resources Survey: 1991, 1994, Key Observations on International Staffing with a Focus on Gender; Deborah Merrill-Sands; October 1995. Working Paper, No. 12 Gender Staffing in the CGIAR: Achievements, Constraints, and a Framework for Future Action; October 1]995. Working Paper, No. 13 Sexual Harassment in the Workplace: How to Recognize It; How to Deal With It; Joan Joshi and Jodie Nachison; October 1996. Working Paper, No. 14 Maximizing Recruitment Resources: Using the World Wide Web; Bonnie Folger McClafferty; January 1997. Working Paper, No. 15 1997 CGIAR Human Resources Survey: International Staffing at the CGIAR Centers with a Focus on Gender; Deborah Merrill-Sands; October 1997. Working Paper, No. 16 Role of Boards in Addressing Gender Staffing Issues; Joan Joshi and Deborah Merrill-Sands; January 1998. Working Paper, No. 17 Strangers in a Strange Land: A Literature Review, of Women in Science; Bridgette Sheridan; April 1998. Working Paper, No. 18 Toward Gender Equity: Model Policies; Joan Joshi, Elizabeth Goldberg, Sara J. Scherr, Deborah Merrill-Sands; September 1998. Working Paper, No. 19 Gender Staffing in the CGIAR: Lessons Learned and Future Direction: Report of an Inter-Center Consultation; Bonnie Folger McClafferty; December 1998. Working Paper, No. 20 Taking Stock of Gender Staffing in the CGIAR, 1998; Sara J. Scherr, Deborah Merrill-Sands; May 1999. Working Paper, No. 21 Engendering Organizational Change: A Case Study of Strengthening Gender- Equity and Organizational Effectiveness in an International Agricultural Research Institute; Deborah Merrill-Sands, Joyce Fletcher, Anne Acosta, Nancy Andrews, and Maureen Harvey; May 1999. Working Paper, No. 23 Summary Report, CIMMYT Pilot on Multi-Source Performance Assessment; Linda Spink, Deborah Merrill-Sands, Krista Baldini, Marisa de la 0; May 1999. CGIAR GENER PROGRAM WORKING PAPER, NO 23 SUMMAR Y REPORT CIMMYT PILOT ON MULTI-SOURCE PERFORMANCE ASSESSMENT Prepared by Linda Spink Deborah Merrill-Sands Krista Baldini Marisa de la 0. May 1999 CGLAR Secretariat Center for Gender in Organizations (CGO) World Bank Simmons Graduate School of Management Washington, D.C. Boston, MA, USA CC*AR Gender Staffig Program The Comltative Groap on bntenaiomd Agricltu Reseah (COIAR) aim tobams modem scieceto h sustainable devlopmen t dagrle in por owds. h tU is jity ssorw d by the Wold Bank, te Food wa 4iukro Orgizaton (FAOX tt Th4aN aos Developxent Poga (UND)?, nd the Thtited Natos Eninmental. Pros (UNEP}. It i meda uipof 16 intenatioa agricultura msearcctti locatxed in 12 *deveo sitgmd 3 deiteopcd vxamais. Thes resarch conters specializ in staraxgi reeatdhmagdeuhuxc foodpolicy and scientitsU of 60 diffrent nationalities. CGIAR Gender Staffing Prmgranm 'he Gender Stafng Program supports effbrts ofthe OGLAR*supportcd centers to sfttrnhen th recrhuitet and roteutto of hihly qualifed women scientists and professionals and to ctmaft work enioments that are equally supportive otte produCgvtilyr dawe axnd jo satisactic of inanaenient consltig, traiin, and infnamtioa sorvicea. The Program, which bega in 19, is wordint by the CGLAR Secretariat, supportd by the membe ofthe CGLIA and ipemeted by fth Center for Gendler in Organiizatios (COO). at the Simmn Graduate SAhMo ofl mangcwcn at Simmons Collg in Boston, Massahbsets, T he nmissionoft Center for Gender in Organiatis is to ser as a natioal and international resource for soas and practitiofns WDo work at the intesection of gender ad strategic orgaizationa issues. The Center's wois based on te beheeftha organzational peforance is etcd b ger eqgitable work eavinmments that allow bothmen and wom to be ae-and prcduive ontributors. T Center pursu h* agenda throug education, collaboiative rearch, eonernces, and dissmnati of iform . CGAR Centers CIAT Centro htenacional de Agrctura T-ropica (Columbia) CUFOR Cent for Iteatioa Foresty Research &Tndesia) 4C1MMY Cntra Inmaciona de Mjoraninto 4de Maiz yTdg (Mexc) CIP Centre Internacional de la Papa (Per) ICARDA Intationa Ceter for AgricultUr Research in t Dry Areas (Syia) ICLARM Itematxonal Cener for Living Aqac Resources Management (Phipes) ICRAP Internaton Cnter for Researh in AgroO,esuiy (Kenya) ICRISAT Internationa Crops Research istitute, fr the Semi-Add Tropics (India) FPRI International Food Poicy Reseaich Istitut (USA) M1 nematonal Irigatin Managxent tliute (gSi L=nka) IfTA Inrmationa institute of Troicl Agricutu (Nigeria) ILRI Internationa Livestock Researh Ititute (Kenya) iPGRI Intentina Plat Genetis Resources institute (Italy) bitoatinal ieResearch Istite (Pwhlines) ISNAR Intematia Service for Nationa Agcral Research (he Ne rlans) WARDA West Afria Rice Development Association (Cote dIvoire) TABLE OF CONTENTS Table of Contents ............................... .i Acknowledgements ................................. ii Executive Summary ................................ iii I. Introduction ................................1 II. Overview of the Pilot ................................3 A. Design of Pilot .................................3 B. Participants ................................. .4 C. Communicating with CJIMMYT Staff ........................... ...4 D. Feedback Criteria ................................ .5 E. Selection of Respondent Teams ...................... ................. .............5 F. Preparation of Feedback Disks . . . 6 G. Individual Feedback Reports . . . 6 H. Administration of the Pilot . . . 6 I. Evaluation of the Pilot ....................................................................................................7 III. Results ....................................................9 A. Overview ................................................... 9 B. Objectives and Quality of Information ....................................................9 C. Instrument and Process .................................................. 10 D. Future Use of 3600 Feedback and Assessment within CIMMYT ..11.................................. 11 IV Recommendations for CIM M YT .................................................. 13 A. Use of Multi-Source Assessment .................................................. 13 B. Implementation of 3600 Feedback ............................................ 13 C. Criteria Development ................................ 14 D. Informing Staff ................................ 15 E. Selection of Respondents ................................ 15 F. Preparation of Respondents ................................ 16 G. Administration of the System ................................ 16 H. Follow up ................................ 17 Annex 1: Respondents ................................ 19 Annex 2: Survey Data Tables. ................................. 21 Annex 3: Staff Feedback on Summary Report ................................ 35 Annex 4: Comparison of Software Programs ................................ 37 ACKNOWLEDGEMENTS This pilot project with multi-source, or 3600, performance assessment has been a collaborative effort between the Centro Internacional de Mejoramiento de Maiz y Trigo (CIMMYT) [The International Center for Improvement of Maize and Wheat], the CGIAR Gender Staffing Program, and the Ford Support Program for Organizational Change for the CGIAR Centers. Funding for the project has been provided by these sources as well as the Center for Gender in Organizations at the Simmons College Graduate School of Management. We gratefully acknowledge the time and effort that CIMMYT staff invested in participating in the pilot project and evaluating the outcome. We would also like to acknowledge the important role that Dr. Joyce Fletcher, Professor of Management at the Center for Gender in Organizations, played in helping to design and launch this pilot project. The CGIAR Gender Staffing Program is funded by several members of the CGIAR, including The Australian Council for International Agricultural Research, The Ford Foundation, The International Development and Research Center, The Netherlands Ministry of Foreign Affairs, and the Swiss Development Corporation. AUTHORS' Linda Spink is a Senior Consultant with Training Resources Group, Inc. in Alexandria; VA, USA. Deborah Merrill-Sands is the Associate Director of tie. Center for Gender in Organzations at the Simmons Grduate School of Management, Boston, MA, USA and.tbe Co-Leader of the CCG Gendr. Staffing Program. Kista Baldini is the Manager of Human Rewurces at CIMMYT nd:Marisa de ia O is Head of Intemational Personnel in Human Resources at CINMYT. . .~~~~~~~~~~ EXECUTIVE SUMMARY A. Background and Objectives In response to staff recommendations generated during the Gender in the Workplace consultancy carried out in 1996, CIMMYT undertook a year-long effort to develop and pilot a multi-source, or 3600, performance assessment process. The experiment with multi-source assessment was intended to introduce practices that would interrupt and challenge several of the deeply held assumptions in CIMv:MYT's organizational culture that had been identified as having unintended consequences both for gender equity and organizational performance. It was expected that multi-source performance assessment would contribute to organizational effectiveness by: 1) Reinforcing values and skills considered important for CIMMYT's new strategic directions; 2) Giving staff an opportunity to receive fair and accurate feedback from coworkers who are most knowledgeable about their work; 3) Providing a means for staff to channel feedback up the hierarchy and provide input on supervisors' and managers' performance; 4) Giving greater visibility to intermediate work products and inputs; and 5) Focusing explicit attention on behaviors that foster collaboration, efficiency, and enabling of others, but often remain invisible and undervalued in performance appraisals that focus solely on individual achievement. Although seemingly gender neutral, this experiment also had the potential to affect gender equity in a significant way. Research indicates that multi-source performance assessment is often more gender equitable than traditional single-source systems. It lessens the potential for managerial bias and discomfort with providing feedback to women. It also provides a way of making visible many of the support functions and work skills that women routinely provide in organizations, such as facilitation, problem prevention, support, and coordination. B. 3600 Organizational Experiment A total of 239 staff participated in giving feedback to 55 "subjects" (those receiving feedback) from External Relations, the Software Development Division, the Management Advisory Committee, and the GP3 Wheat Project. The 3600 approach focused on behaviors and skills that are essential for strong work performance at CIMMYT and collected quantitative data on selected criteria. Each pilot group generated its own criteria for assessment. The feedback given in the pilots was used for staff development purposes. This meant that only the "subjects" received the data and they controlled who else had access to the results. Each "subject" selected a team of respondents (ranging between 4-16) that included their direct supervisors, colleagues/peers, direct and indirect reports, and external internal clients. Respondents received training in using the instrument and giving feedback. They used a 1-10 rating scale to provide feedback on selected criteria, answered three open-ended questions, and had the option of providing comments on each criteria rating. The feedback data was compiled and presented to the "subject" in a report. Workshops were held to train staff how to receive the feedback and interpret their individual reports. C. Results Focus groups and an assessment survey were used to capture participants' reactions to the 3600 pilot. Staff assessments indicate clearly that they found the feedback generated through the multi- source assessment to be accurate, fair, and credible. It was seen as useful and relevant to their work and as motivating them to improve their work performance. Staff reported that the 3600 feedback provided a more useful assessment of performance than that afforded by focusing on work outputs alone. Very importantly, staff found the feedback sufficiently useful to merit the time they invested. Staff sees the 3600 feedback as an important complement to, rather than substitute for, the management by objective (MBO) performance assessment system. They would like CIMMYT to implement the 3600 feedback for both staff development purposes and as part of the formal review process within two years. Staff recommended that CIMMYT adopt the current approach to 3600 and fiurther refine the assessment criteria as well as the instrument and process used. Among the subjects who responded to the survey, women expressed greater appreciation than men for the utility and relevance of the 3600 feedback for strengthening their work performance. Among all participants in the experiment, managers and staff in both program and administrative service roles expressed stronger support for the adoption of the 3600 approach than did scientists. All, however, agreed that CIMM4YT should adopt the 360° approach for staff development purposes and eventually as part of the formal performance appraisal system. D. Recommendations Based on the pilot experience, the assessment survey results, and consultations with staff and managers about next steps, the following recommendations are offered for the further development of multi-source performance at CIMMYT: 1. Adopt the quantitative 3600 approach used in the pilot center-wide, but continue to refine the instrument and process to ensure optimal utility. Include some criteria that refer to the quality of products and services, timeliness of delivery, and productivity. 2. Develop an approach to performance assessment that integrates 1) broader qualitative input from colleagues and managers into the current process of assessing work outputs and the attainment of work objectives (management by objectives) with 2) quantitative 3600 feedback that focuses on skills and behaviors important to CIMMYT's mission, organizational performance, and individual work performance. The integrated approach will be particularly relevant for scientists. iv 3. Use the 3600 process for staff development purposes for at least 1-2 years. Once trust and confidence in the system has been established, integrate the 3600 assessment into the formal appraisal process. 4. Implement the 3600 feedback system in phases, with the first year involving international and national staff in managerial, supervisory, and professional roles. Expand to other staff categories in subsequent years. Include one further year of experimentation with scientists to determine the optimal means for integrating qualitative multi-source assessment of scientific outputs with quantitative 3600 feedback on skills and behaviors. 5. Use a working group comprised of staff representing major job categories/functions to develop assessment criteria (no more than 15 items) that represent the core values and behaviors that CIMMYT wants to reinforce for all staff. These should be supplemented by criteria (no more than 15) reflecting the critical behaviors and skills relevant for specific job categories (e.g. scientist, manager, administrator, support staff). Develop criteria and surveys in both English and Spanish. 6. Introduce multi-source assessment using a concerted effort to educate staff about multi- source feedback, how to give and receive feedback, the process and safeguards of the system used, and the expected benefit of using multi-source assessment as a developmental tool. This education of staff is an investment in the future success of multi-source assessments at CIMMYT. 7. Have respondent teams selected jointly by the subject and his/her direct supervisor. Teams should have no fewer than 9 and no more than 16 respondents in the following categories: supervisor, colleagues/other, direct/indirect reports, and external/internal clients (recipients of work). Take steps to explain the safeguards that maintain anonymity of respondents and increase trust in the overall process. 8. Ensure optimal impact in improving work performance by providing support to staff interested in strengthening specific skills as a result of the 360° feedback. 9. Determine whether to use an external or internal administration based on criteria of costs and manageability, given staffs apparent confidence in CIvvIMYT's ability to maintain confidentiality and administer the system internally. 10. Seek to implement a system that will permit the generation of aggregate statistics on ratings for the core criteria and the job category criteria (called a "roll up") so that staff have a baseline with which to compare their ratings. I1. Select software that is able to handle: multiple mediums for surveys, i.e., paper, electronic disks, and WEB-base; unlimited number of subjects and respondents; customized criteria; multiple surveys; flexible reporting options; and safeguards for small sampling methods and assuring anonymity of respondents. The Intelligent Consensus' software meets the above criteria but should be compared against other available software for cost comparisons. A proprietary software developed by TEAMS Inc., in Tempe Arizona. v IA I. INTRODUCTION The pilot on multi-source assessment, or 3600, derives from the 1996 analysis of Gender Issues in the Workplace - a collaborative action research and learning project carried out by CIMMYT and the CGIAR Gender Staffing Program. Staff expressed a strong interest in this organizational expefiment as an intervention designed to address several central issues identified in the analysis. The experiment with multi-source assessment was intended to introduce practices that would disrupt and challenge several deeply held assumptions in CIMMYT's organizational culture that were rooted in CIMMYT's past, but were having unintended consequences both for gender equity and organizational performance in the CIMMYT of today. These mental models included "default to hierarchy" or the assumption that hierarchy is the best way to organize and that expertise resides at the top; the belief in individual accomplishments as the most effective means for achieving scientific breakthroughs; and the belief that tangible products are the best measure of success. 2 It was expected that multi-source performance assessment would contribute to organizational effectiveness by: I) Reinforcing values and skills considered important for CIMMYT's new strategic directions; 2) Giving staff an opportunity to receive fair and accurate feedback from coworkers who are most knowledgeable about their work; 3) Providing a means for staff to channel feedback up the hierarchy and provide input on supervisors' and managers' performance; 4) Giving greater visibility to intermediate work products and inputs; and 5) Focusing explicit attention on behaviors that foster collaboration, efficiency, and enabling of others, but often remain invisible and undervalued in performance appraisals that focus solely on individual achievement. Although seemingly gender neutral, this experiment also had the potential to affect gender equity in a significant way. Research indicates that multi-source performance assessment is often more gender equitable than traditional single-source systems.3 Not only does it provide a way of lessening managerial bias and discomfort in providing feedback to women, it also provides a way See Merrill-Sands, D.. J. Fletcher, A. Acosta, N. Andrews, and M. Harvey (1999). Engendering Organizational Change: A Case Study of Strengthening Gender-Equity and Organizational Effectiveness in an International Agricultural Research Institute. CGIAR Gender Staffing Working Paper, No. 21. Washington, D.C.: CGIAR Secretariat, World Bank: 3 See Edwards, M. and A. Ewen. 3600 Feedback: The Powerful New Modelfor Employee Assessment and Performance Improvement. New York: American Management Association. See also Edwards, M., A. Ewen, and W. Verdini. 1995. "Fair Performance Management and Pay Practices for Diverse Work Forces." ACA Journal, vol 4., no. 4, Spring 1995. of making visible many of the support functions that women routinely provide in organizations, both formally and informally, such as facilitation, problem prevention, support, and coordination.4 The following sections in this report describe the pilot experiment and its outcome, summarize the evaluation of the experiment, and present the recommendations developed for CIMMYT based on careful monitoring of the pilot and feedback from staff and managers. 4 See Fletcher, J. K. 1998. "Relational practice: A feminist reconstruction of work," Journal of Management Inquiry, 7, 163-186; and Fletcher, J.K. (1999). Relational Practice at Work: Gender, power and the "new " organization. Boston, MA: MIT press. 2 II. OVERVIEW OF THE PILOT A. Design of Pilot The pilot was designed in collaboration with CIMMYT's Manager of Human Resources. It received funding and technical support from the CGIAR Gender Staffing Program and the Ford Foundation Support Program for Organizational Change, the Center for Gender in Organizations at Simmons College Graduate School of Management, and Training Resources Group, Inc., which was the consulting agency helping to implement the pilot. Five assumptions underlay the pilot strategy: * Participation would be voluntary. * Lessons learned from the pilot would be used in the final discussion and decision regarding a center-wide use of 360 feedback. * Feedback would be "developmental" with the feedback going to the subjects only. * Various groups within CIMMYT would be asked to participate, including both international and national staff, scientific and non-scientific staff, and managerial and non-managerial staff. * Lessons learned from CEIVMYT would be shared with other CGIAR centers interested in multi-source assessment. Based on a review of the literature' and available software programs, a decision was made to use the Insight Profiles software.6 This software was developed by Mark Edwards, a leading authority on 3600 feedback, and was considered to have several important features not readily available in other software. Specifically, the Insight Profiles software offered the following statistical and technological safeguards which are critical to small sample surveys typical of multi- source assessments: * Respondent anonymity is protected by a feature that closes and seals the survey on the disk. Once completed, neither the respondent nor any other person is allowed to re-open the survey. Data is imported and no one sees individual respondent information. * Error avoidance from importing data directly without having to re-enter data from a paper inventory. * Trimmed mean scoring (also known as Olympic scoring) removes the most extreme high and low ratings that might skew the small sample. * Agreement of scores, a measure of inter-rater agreement, shows the degree to which respondents were consistent with one another. The agreement rating shows variation without indicating the lowest rating. 5 Gonnfley, W. and L. Spink (1997). Exploring Multi-Source Feedback and Assessment Systems. Organizational Change Briefing Note, No. 4., Ford Support Program for Organizational Change in the CGIAR-Supported Research Centers. Boston, MA.: Simmons College. Simmons Institute for Leadership and Change. 6 Developed by TEAMS Inc. and administered in partnership with TRG, Inc. 3 * Intelligent scoring has the potential of identi ing respondents' who are statistically providing ratings more than 20% different from all other respondents. Insight Profiles is a relatively simple and basic software program designed for smaller organizations with fewer than 300 participants. Its cost is reasonable: $100 for the software license and $50/person for individual subject licenses. The subject licenses are a one-time fee that allows individuals to participate in repeated assessments without additional expenses. The Insight Profiles had several other advantages: it had technical safeguards not found in other software; it allowed for customized criteria; and it permitted the use of either paper or electronic disk for the data collection. An advantage of the electronic disks as a medium for data collection is that it reduces the time and effort required to re-enter the data from the paper survey into a database. The ability to also use paper surveys was useful given that some clients and staff did not have access to or use computers. B. Participants Four different types of work groups participated in the pilot: a Program Support unit (External Relations), a service unit with predominately national staff group (SDD), the senior management team (MAC), and a scientific group (GP3). This diversity was desired as a means to identify questions and concerns that might emerge from different staff groups within CIMMYT. A total of 55 "subjects," those people receiving feedback, from four units within CIMMYT participated in the 3600 feedback pilot. Table 1 shows the number and gender breakdown of subjects in the pilot groups. In terms of respondents, 239 staff participated in giving feedback to the 55 "subjects." Table 1: Pilot Groups - Number of Subjects Pilot Groups Females Males Total External Relations (ER) 5 4 9 Software Development Division (SDD) 3 6 9 Management Advisory Committee (MAC) 3 11 14 Wheat Project Team (GP3) 4 19 23 Total 15 (27%) 40 (73%) 55 C. Communicating with CIMMYT Staff It was important to inform C1MMYT staff about multi-source feedback and the pilot effort before enlisting subjects and respondents. A variety of educational events and written materials were employed. A seminar was presented to staff from the Biotechnology Program, which had initially expressed interest in experimenting with multi-source assessment. The seminar presented information on the approach, background to CIMMYT's interest in 3600 feedback, the research on what other organizations had done with 3600 feedback, how the INSIGHT Profiles software worked and 4 answered questions of staff.7 The second event was a center-wide brown bag presentation for all interested staff. This brown bag presentation covered 3600 feedback in general and described the process for the CIIMIMYT pilot. Approximately 100 staff attended the seminar and received materials in Spanish and English. In addition to the discussions, information about the 3600 feedback pilot was regularly included in the center's weekly newsletter, the INFORMA, and referred to by the senior management in formal and informal meetings. D. Feedback Criteria It was decided to involve staff in the development of criteria so that the criteria would be tailored to reflect the work in the specific pilot work units. A different process was used with each group to determine the level of facilitation and external input that is necessary and helpful. The first group, External Relations, participated in a two-day facilitated workshop that identified the core values and behaviors they associated with excellent work in their unit and on which they wished to be rated. The facilitator took the work generated by the group, refined the criteria and returned it for final review and approval. A consultant from the action research team also participated in the workshop to ensure that the gender equity goals of the experiment did not get lost. The second group, SDD, worked without an external facilitator, but was supported by a staff member from the Human Resources Office. This group reviewed and modified the criteria developed by External Relations. This process was completed in a couple of internal meetings and through email. The MAC group asked the external consultant/facilitator to provide suggested criteria based on literature of essential management and leadership skills and behaviors. They reviewed the suggested criteria and made slight modifications to the language of the rating scale and items. The GP3 group participated in a 2-hour facilitated meeting with the external consultant that introduced the feedback process and generated a collective list of possible criteria. The collective list was circulated and revised by members of the GP3 project without further assistance from the facilitator. E. Selection of Respondent Teams Each person participating in the 3600 pilot identified respondents from whom they would seek feedback. The number of respondents selected ranged from 4-16 (see Annex 1). Each person was asked to work with their immediate supervisor to select their respondent team. The external consultant and staff from the Human Resources Office also helped staff to select respondents. The intent was to have agreement that the respondents would be people who knew the work of the staff person and who could provide relevant feedback. For the MAC group, the members sought feedback from other members of the MAC and at least three direct reports. 7 The Biotechnology Program eventually decided not to participate in the pilot because there was not a full consensus among staff to proceed. 5 The first two groups to implement the process, External Relations and SDD, were asked to select between 5-9 respondents. These groups concluded, however, that limiting their total respondents to 9 was too prohibitive and reduced the overall number of responses. Building on this experience, the MAC and the GP3 groups were asked to select more respondents. The overall respondent response rate was 75% with the lowest rate being 68% for the SDD and the highest rate of 87% for the MAC. Annex 1 contains details on the respondent team categories, size and response rates. F. Preparation of Feedback Disks The distribution of feedback questionnaires was done using a proprietary software program called Insight Profiles. Each respondent received an electronic disk (IBM compatible only) that included questionnaires for the subjects to whom they were giving feedback. The disks were prepared by the consultant from TRG, Inc. and distributed by the CIIMMYT Human Resources Department. With each disk, respondents received a personalized letter and instructions for completing the disks. The turn around time for the completion of the disks varied from two weeks (for External Relations) to two months (MAC & GP3). G. Individual Feedback Reports Once the disks were completed, they were sent to the external consultant for compilation. A confidential report was developed for each subject. These reports were distributed during a facilitated meeting that ranged from 2-4 hours. The meeting provided information on how to make the most of feedback, how to interpret the data, and how to develop personal action plans based on the feedback. The external consultant/facilitator conducted these meetings. An additional "roll-up" report was generated, upon request, for the MAC group. The roll-up report provided aggregate data on the overall ratings for all MAC members. This information allowed individual members to compare their own ratings to the group ratings and to identify group strengths and weaknesses as well as systemic factors within the organization affecting their performance. This report required the purchase of an additional reporting software at a cost of $500. H. Administration of the Pilot The Human Resources Office administered the pilot and coordinated the work with the external consultant/facilitator. Working together with the consultant, the Human Resources Office developed the objectives and guidelines for conducting the assessment process; developed guidelines for developing performance criteria and selecting respondents; trained and coached receivers of feedback; helped subjects to develop concrete action plans; and monitored development and outcome of the pilots. The Human Resources Office had lead responsibility for coordinating work between the pilot groups and the external consultant/facilitator and managing the logistics of distributing information and disks and following up with respondents to ensure adequate response rates. The Information Services Unit and the Human Resources Office developed the Spanish version of the questionnaire and translated all the information, presentation and materials used during the process. 6 The total cost of the pilot, not including staff time, is estimated at $22,000, somewhat more than half of which was provided by small grants. I. Evaluation of the Pilot After each meeting with the pilot groups, subjects and respondents were asked to provide overall reactions and comments on the multi-source assessment process. These comments were captured and are integrated into the discussion of results in the next section of this report. A focus group to get feedback on the pilot was held with representatives from the first three pilots, a member of the action research team, and the Manager of Human Resources. In addition, a survey was used to collect quantitative data on subjects' and respondents' reactions to the multi-source assessment process. The survey sought to capture staffs perceptions on: 1) the quality and utility of the information provided through the 3600 assessment; 2) the degree to which the objectives of the 3600 process were met; 3) the appropriateness of the specific instrument used; and 4) recommendations for future use of 3600 assessment at CLMIvIYT. Staff were asked to indicate their level of agreement with survey statements on a scale of 1-10 with 1 being strongly disagree and 10 being strongly agree. Annex 2, Table 1 summarizes the average ratings received from subjects (those who received the feedback), respondents (those who gave the feedback), and for the total sample. It also provides tables of responses disaggregated by sex (Annex 2, Table 2) and hiring category (Annex 2, Table 3). A summary of staff feedback at the end of the pilot is presented in Annex 3. In total, 78 staff responded to the survey representing 26% of all staff and external partners/clients who participated in the pilot. The survey respondents included 51 of the staff who been asked to give feedback (20% of total) and 27 (49%) of the staff who had received feedback (subjects). Women comprised 37% of the subjects who responded, compared to 27% of the population who participated, and 24% of the respondent group who completed the survey. With respect to intemationally- and nationally-recruited staff, the response rate among subjects reflected the relative proportion of international (66%) and national (34%) staff included among the subjects participating in the experiment. National staff also comprised 56% of the 3600 feedback respondents who completed the assessment survey. Table 2 shows the response rate by urit. The results of the surveys are reported in the following Section III. Table 2. Summary of response rate to assessment survey by pilot units Uinit No. of subjects No. of surveys No. of No. of surveys evaluated received from respondents received from subjects participating respondents* GP3 24 11 140 30 MAC 14 3 56 12 External Relations 9 6 37 3 Software Development 9 7 30 3 * 3 respondents did not indicate their work unit 7 8 III. RESULTS A. Overview To summarize, the staffs' qualitative and quantitative assessments of the pilot indicate clearly that they found the feedback collected through the multi-source assessment useful and relevant to their work. Staff agreed that they would like to see CIMMYT implement the 3600 feedback for both staff development purposes and as part of the formal review process within two years. Staff sees the 3600 feedback as an important complement to the management by objective (MBO) performance assessment system and as affording a more useful assessment of performance than that provided by focusing solely on work outputs. Staff responding to the assessment survey (n=78) recommend that the current approach to 3600 be adopted, but with further refinement of the assessment criteria and the instrument and process used (see Annex 2, Table 1). B. Objectives and Quality of Information In terms of the objectives of the 3600 and quality of information provided, respondents to the survey and staff participating in the focus groups had positive reactions. Staff who had received feedback in the pilots (e.g. subjects) indicated in the survey that they believed that the information generated is fair and credible. They agreed that the 3600 feedback focused on behaviors that were important for successful work performance both within CIMMYT and in their specific work group/unit. They further agreed that the 3600 feedback provided accurate information and that it motivated them to improve their work performance and to practice skills and behaviors that would strengthen their contribution to CIMMYT. They found the 3600 assessment to be useful for assessing their competencies in collaboration and team work and in enabling others to work efficiently and effectively-work skills that, although important for CIMvMYT's success, were perceived to be "invisible" in CIMvYT's organizational culture and undervalued in the current MBO performance assessment system. Subjects responding in the assessment survey indicated clearly that they thought that the 3600 approach offers a more useful assessment of performance than that provided by focusing on work outputs alone. They indicated that they thought that the information generated through the 360° feedback supplements in useful ways the feedback received from their supervisors under the existing system. Both subjects and respondents who participated in the assessment survey indicated that they think the 3600 feedback offers greater potential for fairness and more honest and frank feedback than the supervisor-only system. Very importantly, survey respondents indicated that they thought that the feedback received was sufficiently useful to warrant the time invested in the process. There was an interesting difference in reactions of male (n=16) and female (n=10) subjects responding to the assessment survey in terms of the degree to which the objectives of the 3600 were met and quality and utility of information generated (Annex 2, Table 2). Taking a composite score of the 16 indicators in the assessment survey on objectives and quality of information, the average rating by women was 8.0 compared to 6.7 by men.' Women indicated more than men All differences in averages reported here are statistically significant at the .05 level. 9 that they found the 3600 feedback to offer a more useful assessment of performance than that provided by focusing on work outputs alone (8.7 compared to 6.9). They also agreed more strongly that the 3600 feedback supplements that received by their supervisor in useful ways (8.7 compared to 6.1) and offers greater potential for fairness than the supervisor-only approach to performance appraisal (8.7 compared to 7.2). Women agreed more strongly that the 3600 feedback provided information that motivated them to improve their work performance (8.5 compared to 7.0). Women also expressed stronger support than men for the adoption of the 3600 approach to performance assessment, at least for staff development purposes (7.9 compared to 6.6) There were also interesting differences between the reactions of intemationally-recruited staff (n=18) and nationally-recruited staff (n=9) who were subjects in the 3600 feedback and responded to the survey (Annex 2, Table 3). Taking a composite score of the 16 indicators in the assessment survey on the degree to which the 3600 met its stated objectives and on the quality of information, the average rating by national staff was 8.1 compared to 6.8 for international staff. The most striking differences related to the degree to which the 3600 was useful in assessing skills and behaviors important for successful work performance at CIMMYT (8.4 for national staff, compared to 6.3 for international staff), and in the degree to which the information received motivated improvements in work performance (8.8 for national, compared to 7.0 for international staff). While both international and national staff agreed that the 3600 approach should be adopted as part of the formal assessment process, the national staff supported this more strongly. Among all participants in the experiment, staff in service roles and managers expressed greater appreciation for the utility and relevance of the 3600 approach and stronger support for the adoption of the 3600 than did scientists. This likely reflects the ability of the 3600 approach to capture aspects of "invisible work" of facilitating, enabling, and collaborating which is essential for meeting CIMMYT's mission, but is not recognized in the current formal appraisal system that focuses on work outputs. C. Instrument and Process In general, staff responding to the assessment survey and participating in the focus groups found the instrument and process used for the 3600 appropriate for CIAMMYT. They were satisfied that the process protected the anonymity of the respondents and the confidentiality of the subjects. The training and instructions for participating in the process were sufficient for staff to participate effectively. Staff endorses the method of including both the subject and his/her supervisor in the selection of respondents. Staff found the survey questions to be clear and the forms easy to fill in, generally taking between 20 - 90 minutes to complete depending on the extent of comments given. They appreciated that the questionnaires were prepared in both English and Spanish. Participants in the pilots found the reports easy to understand and use. They agreed that the comments included in the reports were useful supplements to the quantitative ratings on the criteria (rating = 8). Staff in the focus groups indicated that they would like a "roll-up," or aggregate report, of the scores from their work unit so that they could have a baseline against which to compare their own ratings. This was done in the MAC pilot and was found to be useful. The relevance of the criteria used in the 3600 feedback instruments is the primary area of concern with a diversity of opinions expressed by staff in both the assessment survey and the focus groups. 10 The average rating of agreement that the criteria were relevant was 6 for subjects and 7 for respondents, but there was a wide range of opinions among participants and work units. Subjects in GP3 expressed the most concern about the relevance of the criteria (rating = 4.9), although the respondents participating in GP3 found the criteria to be relevant (rating = 6.9). In contrast, subjects in the two service units (External Relations and Software Development) found the criteria to be the most relevant and tailored to their work units (rating=7.3). They also found it to be the most motivational in terms of strengthening work performance. These differences in reactions to the relevance of the criteria likely reflect the different processes used to generate the criteria. In External Relations more time was invested in a broadly participatory approach with support from the consultant throughout. A continuing question is the degree to which the performance criteria included in the multi-source assessment should reflect core values important for achieving CIMMYT's mission or whether the criteria should be tailored to specific job categories (such as scientist, administrator, manager, or service provider). The responses to the assessment survey and comments collected in the focus groups provide useful insights for improving the performance criteria. First, staff would like an instrument that includes a set of core criteria for behaviors and skills important for achieving CIM[MYT's mission (rating = 7) as well as a set of criteria reflecting discrete skills and behaviors required for specific job categories (rating = 8). Second, the staff felt that the criteria needed to be carefully crafted and limited in number. Feedback from the focus groups suggests that it is important to allocate sufficient time and have a facilitator help with the process of developing criteria. Most participants in the pilot felt that it would take at least two iterations to develop an appropriate set of criteria. Third, some staff, particularly scientists, want the 3600 to capture feedback on the quality of work outputs as well as on important behaviors and skills. D. Future Use of 3600 Feedback and Assessment within CIMMYT On the future use of 3600 feedback, the survey responses and feedback from the focus groups indicate clearly that staff wants CIIMMYT to adopt a multi-source assessment process. The majority of staff recommends that the approach used within the pilot be adopted, but that CIMMYT should invest in further refinement of the instrument and process. More attention needs to be given to developing appropriate assessment criteria. The use of the diskette technology also needs to be improved since the rate of disk failure was too high. Staff does not think that the 3600 feedback should replace the Management by Objective (MO) performance assessment process. Rather it should be used as a complement, with the MBO focusing on work outputs and the 3600 focusing on critical behaviors and skills. Staff responding to the assessment survey agree that they want to see the adoption of 3600 feedback center-wide for both staff development and formal performance appraisal purposes within two years (see tables in Annex 2). 11 12 IV. RECOMMENDATIONS FOR CIMMYT Below we outline recommendations for CIMMYT on how to proceed with the multi-source assessment approach. The recommendations are developed from feedback from staff and managers collected through the focus groups and the assessment survey, our observations of the process, discussions with staff and managers about the results of the assessment survey9, and the experiences from other organizations reported in the literature. A. Use of Multi-Source Assessment Staff feedback indicates clearly that there is a strong interest in CIMMIYT continuing to develop the multi-source assessment as a complement to the current performance appraisal system. We recommend that CIiMMYT adopt the 360°approach used in the pilot center-wide, but continue to refine the instrument and process as outlined below to ensure optimal utility. It will be important to include some criteria that refer to the quality of products and services, timeliness of delivery, and productivity. The goal, we suggest, is to develop an approach to performance assessment that combines 1) qualitative input from managers and colleagues close to a staff member's work into the current process of assessing work outputs and attainment of objectives (management by objectives) with 2) quantitative 3600 feedback that focuses on skills and behaviors important to CIMMYT's mission, organizational performance, and individual work performance. This approach is particularly relevant for performance appraisal of scientists. In discussing the integrated approach with staff, they stressed the importance of reviewing the MBO system to improve its effectiveness and efficiency and its consistency with the 3600 approach. They highlighted the principles of consistency, fairness, accountability, and assessment by those who are closest to and most knowledgeable of the staff member's work. Staff also stressed that in an integrated model it would be important that the 3600 assessment of skills and behaviors carry the same weight as the focus on attainment of objectives (see Annex 3). B. Implementation of 360° Feedback Research has shown that participants are most comfortable and receive more usefrul feedback when the process is seen as "developmental," where the feedback data is given only to the subject, rather than as a "performance appraisal" process where the data is given to both the subject and her/his supervisor. Organizations have found that with time, staff confidence and trust in the 3600 process increases and they begin to request that the 3600 feedback become a contributing factor to their performance appraisal. It is, therefore, recommended that CIMMYT start using the 3600 feedback first as a developmental tool. This developmental use should continue for 1-2 years with any modifications being made in the process between the first and second year. After the second 9 D. Merrill-Sands and K. Baldini discussed the results and implications of the assessment survey with the Director General, a group of 25-30 staff who had participated in the pilots, and members of the Management Advisory Committee on March 11-12, 1999. Feedback from these discussions has been incorporated into this final draft of the report (also see Annex 3). 13 year, staff could be polled to ensure that there is sufficient confidence in the 3600 feedback system for it to be used as part of the appraisal process. We recommend that CIMMYT implement the 3600 feedback system in phases, with the first year involving international and national staff in managerial, supervisory, and professional roles. The approach can then be extended to other staff categories in subsequent years. We believe that it will be important to include one further year of experimentation with scientists to determine the optimal means for integrating qualitative multi-source assessment of scientific outputs with quantitative 3600 feedback on skills and behaviors. C. Criteria Development During the pilot, each group was involved in developing their own unit-specific criteria with varying degrees of support from the external consultant/facilitator. All four groups felt that having gone through the process once, they would redo the criteria used in the survey instrument. There were several commonly experienced problems: respondents did not understand the language of the criteria; several items were included in one statement and thus subjects could not discern which item was being rated by the respondents; too many criteria were used; some criteria were duplicative; some criteria did not seem relevant to all subjects; staff agreed that there were generic core criteria appropriate for all staff, but they also wanted criteria tailored to specific job categories. Staff also felt that instructions for respondents needed to be clearer about using the N/A option when they did not have knowledge of the subject with respect to specific skills or behaviors. We recommend that the following considerations be taken into account as CIMMYT moves forward in developing the 3600 feedback system: * Total criteria should be limited to fewer than 30 questions. The criteria need to represent discrete observable behaviors and not include several behaviors in one statement. * Instruments should include some criteria that refer to the quality of products and services, timeliness of delivery, and productivity. * CIMMYT should develop a set of instruments that combine core criteria that reflect the central values and expectations for all staff working at CIMIMYT (no more than 15 items) and specific criteria tailored for primary job categories, i.e., managers, scientists, administrators, service providers, and scientists (no more than 15 items). This will provide enough flexibility to reflect the major differences in work done by staff in different functions, but enough consistency as to not become unwieldy or prohibitively expensive. It is important to note that with this approach "roll ups", or aggregate reports, can only be provided for the primary job categories, not for CIMMYT as a whole nor for all staff in one department, i.e., all SDD staff. * A Task Force, comprised of staff representing different primary job categories/functions, should be convened to develop the assessment criteria for the core CIMMYT values, skills, and behaviors. They should also organize representatives from major job categories to develop category-specific criteria. The Task Force should be responsible for reviewing the criteria and ensuring coherence and consistency across instruments. The Task Force should get feedback from staff on the proposed criteria and then make a recommendation to the Management Advisory Committee. Development of a solid and compelling set of criteria may 14 take several iterations of consultation and revision. To ensure robust and relevant criteria, it is very important that the diverse perspectives of staff from different job functions as well as from different identity groups are represented on the Task Force. We believe that it will be important to have an external facilitator/consultant support the work of the Task Force as a resource person. Each question in the instrument should be carefully worded so as to be clearly understood by non-native English speakers. All materials should be written in both English and Spanish as is consistent with CIMMYT's working norm of allowing staff to participate in the language which best facilitates their contribution. D. Informing Staff While the pilot included several "educational" events for subjects and respondents there were some respondents who did not fully understand nor feel confident about the process. Therefore, it is essential, if CvIMMYT moves to a center-wide usage of 3600, that a concerted educationalU communication campaign be a part of the implementation plan. This cannot be overstated, as an informed, trusting staff is critical for a successful implementation of the 3600 feedback. All means of communicating with staff will be necessary, i.e., formal seminars, informal discussions/brown bag gatherings, written articles in INFORMA, consistent messages and references to the process by management in all appropriate meetings and individual conversations. As one person put it "you can't say enough." It will also be important to see the implementation as an educational process, requiring strategic thinking and planning. The actual implementation of a 3600 feedback program cannot be started before the educational effort has been underway and most staff are aware of and comfortable with how the process will work. E. Selection of Respondents The process of selecting respondent teams varied across pilot groups. The first two groups were limited to fewer than 9 respondents while the last two groups were allowed many more. The increased number did not significantly increase the percentage of returned questionnaires. The latter two groups also increased the number of respondent categories (i.e., they distinguished between internal and external clients). It appears that as the number of respondent categories increased, the number of responses per category decreased. Some subjects had only two respondents per category. This reduced perceived anonymity and reduced the ability to disaggregate the data. Given the experience in the pilots, we recommend that CIMMYT should consider the following guidelines regarding selection of the respondents: * To ensure quality of feedback, subjects need to receive training and support in the appropriate means for selecting their respondent teams. * The subjects' immediate supervisors should discuss with staff whom to include on their respondent team and the final decision should be reached jointly. * Total number of respondents should be limited to no more than 16 and no less than 9. 15 * The categories of respondents should be limited to immediate supervisor, colleagues/others, direct/indirect reports, and external/internal clients (recipients of work or services). There should be a minimum of three respondents in each category, although not every subject would need to use all four categories. Where subjects do not have three respondents in a particular category, he/she should be encouraged to combine respondents into other sub-categories. For example the staff with only 2 direct reports might combine direct reports with customers. * Subjects belonging to more than one work team should be encouraged to select respondents from each team to ensure that all of their work is represented in the feedback. F. Preparation of Respondents As mentioned earlier, a thorough education/communication campaign should be part of any 3600 feedback program. In addition, given the concerns regarding anonymity of respondents, special efforts should be made to: * Ensure that respondents understand the safeguards inherent in the 3600 process. * Provide guidelines for how respondents can give specific comments without identifying themselves, e.g., don't use slang that is recognized as your way of speaking, don't type in all caps if you normally type emails in all caps. * Explain that the respondent is one of many who are providing feedback. The aim is to provide a sense of safety in numbers that can be achieved by either providing the total number of respondents involved or providing the names of the respondent team. * Special attention should be given to informing respondents from national research systems of the safeguards for anonymity of responses. G. Administration of the System There are two key components in the administration of any multi-source assessment process: the administrator and the software. Our recommendations for both aspects are outlined below. Usually a critical question to consider is whether to use an internal or external administrator. The administrator is the person or persons responsible for the actual development of the feedback questionnaire, contacting and tracking respondent replies, importing the data, and producing individual confidential reports. The data from the assessment survey indicates general acceptance of either an internal or external administrator, although there is a slight preference for intemal administration. Given this apparent confidence in CIMM'YT's ability to protect the confidentiality of the inforrnation, CIMMYT has greater flexibility to look at the costs and benefits to outsourcing the administration or developing the internal capacity to run the assessment process. Since the external consultant has worked closely with the Human Resources staff, internal expertise to administer the process has been developed within CIMMYT. The issue will clearly be one of staffing and cost. Regardless of the option selected, it is recommended that measures be taken to communicate to staff any and all safeguards used to address concerns of anonymnity and confidentiality of the data. With respect to the software system, the Insight Profiles software proved satisfactory for the pilot. The one concern was the higher than expected rate of disk error which caused frustration 16 as staff had to redo assessment forms. The exact nature of the disk failures is not known. However, any of the following can contribute to disk failure: incompatible machines (the system does not work on Apple Computers), respondent error in taking the disk out prior to the completion of the application, faulty original disks, damage in shipping and handling internationally, and virus contamination. While the Insight Profiles has some distinct advantages, e.g., lower costs, it also has significant limitations, e.g., cannot handle more than 300 subjects, that make it less appropriate for multi- source assessment of all CIMMYT staff. Another more sophisticated software called Intelligent Consensus'° appears more appropriate for CIMMYT's needs. Intelligent Consensus is designed for a larger organization and with the "Enterprise" version can handle unlimited numbers of subjects, surveys, and respondents. It provides data collection through any of three mediums: 1) paper surveys, 2) electronic disk surveys, or 3) WEB-based surveys (inter- or intra-net). The WEB-based option is a tremendous advantage that would allow CIMMYT to manage the selection of respondent teams, the notification of respondents, the collection of data, and follow- up to be done via an electronic mail system; virtually eliminating much of the paper administration of the process. It would also cut down on the time required to develop disks, send disks overseas, reduce mailing costs, and eliminate disk failures. The Intelligent Consensus System would also allow CiIvIYT to prepare aggregate statistics by job category/function, i.e. "roll ups," so that staff have a base line with which to compare their results. The WEB-based application and other features clearly out weigh the more limited Insight Profiles system. It is recommended that CIMIMYT consider the investment in the Intelligent Consensus system or compare it to other systems with the same features and safeguards. Intelligent Consensus is significantly more expensive, ranging from $20-40,000 for the software license and installation plus a one-time cost of $65/subject fee and an annual maintenance fee. Administration of the system by TEAMS' service bureau would mean additional costs of $3000- $4000 for each survey developed and a per-subject per-report fee of between $125-130. These costs could be reduced or eliminated through the use of an internal administrator or by designating a different administrator from TEAMS, Inc. The use of Intelligent Consensus or any other comparable software needs to been considered as a capital investment in a Human Resources Development process that will continue to serve CIMMYT well into the future. A more detailed comparison of the two software programs is provided in Annex 4. H. Follow up To ensure optimal impact in improving work performance, we recommend that CIMMYT should be prepared to provide support and opportunities to staff who are interested in strengthening specific skills as a result of the 3600 feedback. 10 Developed by TEAMS Inc. 17 18 ANNEX 1: RESPONDENTS RESPONDENT TEAMS, CATEGORIES, AND RESPONSE RATES Pilot Groups Respondent Categories Respondent Teams External Relations 1. Supervisor Range: 4-8 2. Colleague Average: 6.6 3. Client 4. Direct Report SDD 5. Supervisor Range: 6-8 6. Colleague Average: 7.5 7. Client r MAC 8. Supervisor Range: 4-15 9. Client Average: 10 10. Direct Report/Other GP3 11. Program Director Range: 8-16 12. Client Average: 11.43 13. Direct Report 14. Indirect Report 15. Ex/Internal Client RESPONSE RATES BY PILOT GROUP Pilot Groups Respondents Respondents Percentage Requested Completed of Responses External Relations 43 30 70% SDD 33 25 76% MAC 63 55 87% GP3 145 103 71% Totals 284* 213 75% * Total number of staff participating as respondents is 239. The total respondents requested included duplicate respondents in the four separate surveys. 19 20 ANNEX 2: SURVEY DATA TABLES ASSESSMENT SURVEY OF 3600 PILOT: DATA TABLES Table 1: Summary of responses to assessment survey: Responses by subjects, respondents, and total participants in pilot Table 2: Summary of responses to assessment survey: Responses by sex Table 3: Summary of responses to assessment survey: Responses by hiring category (internationally and nationally-recruited staff) 21 22 Table 1: Summary of responses to assessment survey: Responses by subjects, respondents, and total participants in pilot Respon7dents were asked to indicate their level of agreement with each of the following statements by circling the appropriate number uising the scale given below. If they did not feet they had siufficienit iniformationt to answer the questiotn, they were asked to mark N/A. I Don't Strongly Disagree Moderately Agree Strongly know disagree Agree . agree N/A 1 2 3 4 5 6 7 8 9 10 OBJECTIVES AND QUALITY OF INFORMATION SUBJECTS RESPON- TOTAL (N=27) DENTs (N=78) The 3600 degree feedback process provided me with information that:' 1. Is fair and credible. 8 2. Is useful for assessing skills and behaviors important for successful work performance at 7 CIMMYT. 3. Gives an accurate assessment of behaviors and skills important for work performance in mny 7 work group/unit. 4. Is useful for assessing my competencies in collaboration and teamwork. 7 5. Is useful for assessing my competencies in enabling others to work more effectively and 7 efficiently. ._.. 6. Provides information that motivates me to improve my work performance. 8 7. Supplements in useful ways the feedback received from my supervisor under the current 7 appraisal system. Questions 1 through 11 asked of subjects only 23 OBJECTIVES AND QUALITY OF INFORMATION SUBIECTS RESPON- TOTAL (N=27) DENTS (N=78) (N=5 1) 8 Has motivated me to practice specific skills and behaviors that I believe will strengthen nv 7 work contribution to CINIMIYT _ 9 Has miotivated ine to develop an action plan for improving my performance 7 10 Is useful For identif'ing elenients in the work environment that hinder mn perforimiance 7 11. Is sufficiently useful to wai arant the limne I invested in the process 7 The 3600 feedback approach: 12. Offers the potential for staff to receive feedback from coworkers who are most 7 7 7 knowledgeable about work and skills. 13. Offers a more useful assessment of performance than that provided by focusing on work 8 outputs alone. 14. Has greater potential for fairness than the single rater approach. (e.g. supervisor only). 8 8 8 15. Gives staff a more honest and frank appraisal of their work skills and behaviors than the 8 8 8 single rater approach. 16. Offers potential of helping staff to better align their work skills and behaviors with the core 7 7 7 values necessary for CIMMYT to achieve its mission. INSTRUTIENT AND PROCESS The specific 3600 process/inslrumnent used:_ 17. Ensured anonymity of respondents. 6 7 7 18 Ensured confidentialitv for the recipient. 8 24 INSTRUNMENT AND PROCESS SUBBJECTS RESPON- TOTAL (N=47) DENTS (N=78) 19. Used relevant criteria. 6 7 7 20. Used questions for which the meaning was clear. 7 7 7 21. Used forms with clear instructions for respondents. 7 8 8 22 Pro' ided stitliciext explanation and trainingy so that statfvre e,li-/t.eedhack conld 7 panicipate efTectively in the process 23. Provided sufficient explanation and training so that staff givingfeedback (e.g. respondents) 7 7 7 could participate effectively in the process. 24 Collated and reported the quaniltative data in a %kav that w,as easN. to understand and use 7 25 Generaied comimlents that pro% ided uLseftil supplemenitary information to tlhe quantitatik e 8 ratings RECONINIENDATIONS FOR FLrTLrRE LISE Ba5sed on nil ex,Jerience in the pilot, I recom,nie,,(l that CIAIIAI T: 26. Not use 3600 feedback in its current or modified form. 4 4 4 27. Adopt the current approach to 3600. 6 6 6 28. Adopt the current approach, but continue to refine the instrument and process. 7 8 8 29. Adopt the 360° concept, but explore different approaches. 6 7 7 30. Continue to develop the 360° feedback process for use throughout the Center. 8 7 7 25 RECONIMENDATIONS FOR FL;TURE LISE SUBJECTS RESPON- TOTAL (N=27) DENTS (N=78) (N=51) 31. Develop the 3600 feedback as a complement to the MBO performance assessment process. 7 7 7 32. Develop the 360° feedback as a substitute for the MBO performance assessment process. 4 5 5 33. Continue to use an external party to administer the analysis and generate reports. 6 6 6 34. Develop the internal capacity to administer 3600. 7 7 7 35. Continue to involve the person being reviewed in the selection of respondent teams. 7 7 7 36. Continue to involve the supervisor in the selection of respondent teams. 8 7 7 37. Develop a common set of performance criteria that reflect core values important for 8 7 7 achieving CIMMYT's mission rather than using criteria tailored to specific work groups. 38. Develop a range of instruments with criteria tailored specific job categories. 7 8 8 39. CIMMYT should seek to implement 3600 feedback for staff development center wide 7 7 7 within two years. 40. CIMMYT should seek to implement 3600 feedback for as part of the formal performance 7 7 7 appraisal system within two years. 26 Table 2: Summary of responses to assessment survey: Responses by sex Respondetnts were asked to indicate their level of agreement with each of the followinig statements by circling the appropriate number iusin1g the scale given below. If they did not feel they had si,fficient it?formation to answer the question, they were asked to mark N/A. Doll'| StronglN Disagree Mloderateh' A| g,ree Si rongly know I disagree I Agree I I agree l N/A 1 2 3 4 5 6 7 8 9 10 ORlEcrIVEs AN9D QUALITY' OF INFORMlATION NIALES FENIALES (N=54) ( N =22) The 360)' degreefeedback procmess proilded mtie wit/ inforination thi,u: I. Isfair and credible. 7 8 2. Is useful for assessing skills and behaviors important for successful work performance at 7 8 CIMMYT. 3. Gives an accurate assessment of behaviors and skills important for work performance in my 6 8* work group/unit. 4. Is useful for assessing my competencies in collaboration and teamwork. 7 8* 5. Is useful for assessing my competencies in enabling others to work more effectively and 7 7 efficiently. 6. Provides information that motivates me to improve my work performance. 7 9* 7. Supplements in useful ways the feedback received from my supervisor under the current 6 9* appraisal system. *=difference in means is statistically significant at .05 Questions I through 11 answered by subjects only (n=26; males= 16, females= 10). 27 OBJECTIVES AND QUALITY OF INFORIATIONQ MALES FEMALES (N=54) (N=22) 8. Has motivated me to practice specific skills and behaviors that I believe will strengthen my work 7 8* contribution to CIMMYT. 9. Has motivated me to develop an action plan for improving my performance. 6 8* 10. Is useful for identifying elements in the work environment that hinder my performance. 7 7 11. Is sufficiently useful to warrant the time I invested in the process. 6 8* The 360" feedbuhak approach: 12. Offers the potential for staff to receive feedback from coworkers who are most 7 8 knowledgeable about work and skills. 13. Offers a more useful assessment of performance than that provided by focusing on work 7 9* outputs alone. 14. Has greater potential for fairness than the single rater approach (e.g. supervisor only). 7 8* 15. Gives staff a more honest and frank appraisal of their work skills and behaviors than the 7 8 single rater approach. 16. Offers potential of helping staff to better align their work skills and behaviors with the core 7 8 values necessary for CIMMYT to achieve its mission. INSTRUMENT AND PROCESS The specific 3600 process/instrument used: 17. Ensured anonymity of respondents. 7 6 18. Ensured confidentiality for the recipient. 8 8 28 INSTRLUMTENT AND PROCESS NIALES FENIALES (N=54) (N=22) 19. Used relevant criteria. 7 7 20. Used questions for which the meaning was clear. 6 7* 21. Used forms with clear instructions for respondents. 8 8 22. Provided sufficient explanation and training so that staff receivingfeedback could 7 8 participate effectively in the process. 23. Provided sufficient explanation and training so that staff givingfeedback (e.g. respondents) 7 8 could participate effectively in the process. 24. Collated and reported the quantitative data in a way that was easy to understand and use. 7 8 25. Generated comments that provided useful supplementary information to the quantitative 8 9* ratings. RECOrINIENDATIONS FOR FUTURE USE Blusel on ity eperrience in fhe pilr(f. I recontmend1 thlat C(1111 T.: 26. Not use 3600 feedback in its current or modified form. 4 3* 27. Adopt the current approach to 3600. 6 5 28. Adopt the current approach, but continue to refine the instrument and process. 7 8* 29. Adopt the 3600 concept, but explore different approaches. 7 6 30. Continue to develop the 3600 feedback process for use throughout the Center. 7 8 29 REcohiNIEN1DATIONS FOR FLITURE USE AMALES FENIALES (N=54) (N=22) 31. Develop the 3600 feedback as a complement to the MBO performance assessment process. 7 6 32. Develop the 3600 feedback as a substitute for the MBO performance assessment process. 5 5 33. Continue to use an external party to administer the analysis and generate reports. 6 6 34. Develop the internal capacity to administer 3600. 7 6 35. Continue to involve the person being reviewed in the selection of respondent teams. 7 8 36. Continue to involve the supervisor in the selection of respondent teams. 7 7 37. Develop a common set of performance criteria that reflect core values important for 7 7 achieving CIMMYT's mission rather than using criteria tailored to specific work groups. 38. Develop a range of instruments with criteria tailored specific job categories. 8 7 39. CIMMYT should seek to implement 3600 feedback for staff development center wide 7 8* within two years. _ 40. CIMMYT should seek to implement 3600 feedback for as part of the formal performance 7 7 appraisal system within two years. 30 Table 3: Summary of responses to assessment survey: Responses by hiring category (international and national staff) Respondents were asked to ilidicate their level of agreemenit with each of the following statements by circlinig the appropriate nlumber using the scale given below. If they did niotfeel they had sufficient information to answer the question, they were asked to mark N/A. Don't Stronlyl Disagree NModerateIN' Agree Strongly I kio% disagree Agree -agree N/A 1 2 3 4 5 6 7 8 9 l10 OBJECTIVES AND QUALITY OF INFORNIATION IRS' NRS (N=4 1)- (N=3 1) The .60" degree feelback proces% prov ided mitoe irith information thant: 1. Is fair and credible. 7 8* 2. Is useful for assessing skills and behaviors important for successful work performance at 6 8* CIMMYT. 3. Gives an accurate assessment of behaviors and skills important for work performance in my 6 8* work group/unit. 4. Is useful for assessing my competencies in collaboration and teamwork. 7 8* 5 Is useful for assessing my competencies in enabling others to work more effectively and 6 8* efficiently. 6. Provides information that motivates me to improve my work performance. 7 9* 7. Supplements in useful ways the feedback received from my supervisor under the current 7 8 appraisal system. *=difference in means is statistically significant at .05 level ' IRS=internationally-recruited staff- NRS=nationally-recruited staff 2 Questions 1 through 11 answered by subjects only (n=27; IRS= 18; NRS=9) 31 OBJECTIVES AND QUALITY OF INFORMATION IRS NRS (N=4 0 (N=3 1) 8. Has motivated me to practice specific skills and behaviors that I believe will strengthen my 6 8* work contribution to CIMMYT. 9. Has motivated me to develop an action plan for improving my performance. 6 8* 10. Is useful for identifying elements in the work environment that hinder my performance. 6 8* 11. Is sufficiently useful to warrant the time I invested in the process. 7 8* The 3600 feedlback aippronch: 12. Offers the potential for staff to receive feedback from coworkers who are most 7 8* knowledgeable about work and skills. 13. Offers a more useful assessment of performance than that provided by focusing on work 7 8 outputs alone. 14. Has greater potential for fairness than the single rater approach (e.g. supervisor only). 8 8 15. Gives staff a more honest and frank appraisal of their work skills and behaviors than the 7 8 single rater approach. 16. Offers potential of helping staff to better align their work skills and behaviors with the core 7 8* values necessary for CIMMYT to achieve its mission. INSTRUMENT AND PROCESS The speeific 3600 processlinstrunient. 17. Ensured anonymity of respondents. 6 7* 18. Ensured confidentiality for the recipient. 8 8 32 IN.STRURlEN IrAND PROCESS IRS NRS (N=4 1) (N=3 1) 19. Used relevant criteria. 6 8* 20. Used questions for which the meaning was clear. 6 8* 21. Used forms with clear instructions for respondents. 7 8* 22. Provided sufficient explanation and training so that staff receiving feedback could 7 8* participate effectively in the process. 23. Provided sufficient explanation and training so that staff givingfeedback (e.g. respondents) 7 8* could participate effectively in the process. 24. Collated and reported the quantitative data in a way that was easy to understand and use. 7 8* 25. Generated comments that provided useful supplementary information to the quantitative 8 9* ratings. RECOMMENDATIONS FOR FlILTURE IISE lfOsed oti mY ctvperientee in the/ pilot, I rec'onintentl thiit CIA 11 I'T 26. Not use 360° feedback in its current or modified form. 4 4 27. Adopt the current approach to 3600. 5 7* 28. Adopt the current approach, but continue to refine the instrument and process. 7 8* 29. Adopt the 3600 concept, but explore different approaches. 6 7 30. Continue to develop the 3600 feedback process for use throughout the Center. 7 8* 33 RECONiNIENDATIONS FOR FUTtURE USE IRS NRS (N=4 1) (N=3 1) 31. Develop the 3600 feedback as a complement to the MBO performance assessment process. 6 7 32. Develop the 3600 feedback as a substitute for the MBO performance assessment process. 4 6* 33. Continue to use an external party to administer the analysis and generate reports. 5 7* 34. Develop the internal capacity to administer 3600. 6 7* 35. Continue to involve the person being reviewed in the selection of respondent teams. 7 8 36. Continue to involve the supervisor in the selection of respondent teams. 7 7 37. Develop a common set of performance criteria that reflect core values important for 6 8* achieving CIMMYT's mission rather than using criteria tailored to specific work groups. 38. Develop a range of instruments with criteria tailored specific job categories. 7 7* 39. CIMMYT should seek to implement 3600 feedback for staff development center wide 7 8 within two years. 40. CIMMYT should seek to implement 3600 feedback for as part of the formalperformance 6 8* appraisal system within two years. 34 ANNEX 3: STAFF FEEDBACK ON SUMmARY REPORT 3600 FEEDBACK REPORT - DIsCUSSION WITH STAFF At the end of the pilots a meeting was held with staff who had participated in the pilots to present the results assessment survey of the 3600 pilots and to get feedback from staff on the preliminary recommendations laid out in the draft report. About 25 staff attended from the 4 pilot projects. Staff found the results of the assessment of the pilots very positive and encouraging. Staff recommended strongly that the results of the 3600 pilot be widely disseminated in CIMMYT and shared with the Board. Staff supported adoption of a quantitative approach to 3600 focusing on skills and behaviors important to strong work performance at CIMMYT. They agreed that this approach should be integrated with the MBO system (at least for scientists), but encouraged a thorough review and refinement of the MBO approach as well. Staff agreed with the recommendations laid out in the report and made the following additions and clarifications. I . Staff stressed the important role that integrating 3600 into the performance assessment system can have in terms of reinforcing the cultural changes CIMMYT is trying to bring about. With this objective, they supported the idea of having half of the criteria reflecting core values/skills of CIMMYT and half reflecting critical skills and behaviors of specific job categories. 2. Staff indicated their support of an integrated performance appraisal system which would include both MBO and the quantitative 3600 assessment focusing on behaviors and skills, although some thought that this might only be needed for scientists. The quantitative 3600 approach with conmnents may be adequate for other staff groups. They stressed the importance of reviewing the MBO system to improve its effectiveness and efficiency and its consistency with the 3600 approach. They raised issues of consistency, faimess, accountability, and assessment by those closest to the staff member's work. They stressed the importance of having multiple sources of assessment in the MBO process as well, arguing that for scientists the primary input should be from the project coordinator. Staff also stressed that in an integrated model, it would be important that the 3600 assessment of skills and behaviors carried the same weight as the focus objectives. There was general agreement that the quantitative 3600 instrument should include some indicators on quality, relevance, and timeliness in delivery of outputs. 3. Staff stressed the importance of CIMMYT being prepared to invest in skills building as a follow up and reinforcement of the learning gained through the 3600. This is important if the 3600 is going to have a real impact in helping staff to improve work performance. 4. Staff stressed the importance of building the 3600 feedback into the formal appraisal system and having accountability mechanisms to reinforce performance objectives. They felt this was as important for managers as it was for staff. 5. Staff recommended strongly that a Task Force, composed of staff and managers representing different staff groups (e.g. scientists, program support, administrative) should be formed to 1) make recommendations on implementation; and 2) to develop the core criteria for 3600 assessment. This composition on the Task Force is important for ensuring that the needs and interests of all staff groups are addressed by the reformed performance assessment process. 6. Staff suggested that the assessment process be sequenced throughout the year so that staff appraisal would not become burdensome and rote. A schedule could be developed, for example, such that one major program or department could do its staff evaluations each month. 7. Staff suggested that the introduction of 3600 be phased in over a 2-3 year period and, very importantly. that the process begin at the top of the hierarchy. The first year could include international staff and national staff in managerial and professional roles. Once they had experience with the process, it could be extended to support staff. Staff felt it was important that managers and supervisors gain experience of the process first so that they could help institute it effectively center-wide. 8. Staff suggested that a further experiment be run to capture the costs and benefits of including feedback from extemal clients in the 360° process. Staffnoted that this could be an important mechanism for strengthening partnerships, but also raised more complex issues of confidentiality and trust. [GP3 was the only pilot to include extemal clients; but only 2 responded to the assessment survey.] 36 ANNEX 4: COMPARISON OF SOFTWARE PROGRAMS COMPARISON OF INSIGHT PROFILES AND INTELLIGENT CONSENSUS SOFTWARE Insight Profiles Intelligent Consensus Costs $100 initial software license fee, $20-40,000 initial software plus $50/subject fee license fee, plus $65/subject fee Survey Collection Size Fewer than 300 Up to 1000 in the standard version Unlimited in the "Enterprise" -version Data Collection Medium Paper and/or electronic disk. The Paper, electronic disk, and WEB- disks involves significant time for based. The WEB-based option the creation, mailing, and eliminates the need for disk importing of the data. creation, mailing, and importing of data. Criteria Rating Scale 1-10 or 1-5 only 1-3, 1-5, 1-7, and 1-10 scales available. Also allows two ratings to be given, i.e., how well the behavior is done and how important the behavior is to the job .performance. Respondent Selection Paper process that is then entered Respondent teams can be selected by the system administrator. by subjects on line. The Notification of respondents is proposed respondent teams can done via a separate letter or other be approved by supervisors on tracking process. line as well. Notification of respondents can be done via bulk email and on line tracking of completion rates. Reporting Capacity Limited to pre-established Is compatible with other formats for the reports. Does not reporting software that will allow produce any aggregated reports more creative and specific reports across subjects without the to be produced. purchase of an additional "roll- up" report feature. Even with the More aggregated reports can be additional report feature the produced. system will not provide data on subject/respondents' gender, completion rates, trends etc. Administration Can be administered by either an Can be administered by either an internal or external administrator. internal or external administrator. 37