88847 v2 RES-Research (Resilience in Education Settings Research) TRAINING FACILITATOR GUIDANCE NOTES (2) Education Resilience Approaches PART OF THE: SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 Contents About RES-Research Authors and Acknowledgements Guidance Note Getting started Guidance Note: Module 5B Field work and Feasibility planning Guidance Note: Module 6 Analysis and interpretation for policy and practice Guidance Note: Module 7 Knowledge mobilization I: Policy and reporting Guidance Note: Module 8 Knowledge mobilization II: external engagement, giving back and critique Guidance Note: Module 9 Open reflection 3 About RES-Research Development practitioners in fragile and conflict affected contexts are demanding better support of research, evaluation and assessments: whether it's conducting an exploratory needs assessment for an emergency intervention, monitoring and evaluating ongoing project impact or building the evidence base to design a post-conflict or violence mitigation program. In contexts of overwhelming adversity it is crucial not only to get reliable and valid data but to also ensure that we are going about this in the right way. Doing research “right” in these contexts require asking the right questions, talking to the relevant participants and stakeholders, using the most pertinent methods, and paying particular attention to ethics and power differentials. To address these concerns, the ERA program has developed the Resilience in Education Settings (RES)-RESEARCH training. It brings As with all SABER tools, together resilience theory and a transformative research paradigm. the RES-RESEARCH training module is Resilience theory seeks to understand the process by which individuals, openly available for communities and organizations recover from crisis, continue to perform education practitioners in the midst of adversities and even radically change to prevent future within the World Bank, risk exposure and continue their development process (Reyes, 2013). as well as other The transformative research paradigm provides methodological agencies. The module guidance to conduct studies with vulnerable populations, while consists of a research recognizing both their exposure to overwhelming threats but also their manual and handouts, assets such as strengths, opportunities and available services (Mertens, power point 2009). presentations and additional guidance Through a nine-month training program, RES-RESEARCH builds on the materials. For capacities of academics and education practitioners in fragile, conflict information on the ERA and violence affected contexts to undertake locally relevant and program please consult rigorous education resilience research. First piloted in Central America, www.worldbank/educat the training program was improved and recently implemented in the ion/resilience or write to South Asia region as part of a the Multi-Donor Trust Fund for the educationresilience@w orldbank.org System Assessment for Better Education Results (SABER) support by DfID-UKAID, AUSAID and the World Bank. The “Facilitator Guidance Notes” were prepared to guide universities, local researchers, and development and humanitarian agencies that would like to use and teach this approach, in its entirety or partially. This is only a guide, and the themes, sequence and materials can be adjusted and complemented as needed. 4 Authors and acknowledgements This training package was developed by Joel Reyes, Linda Liebenberg and Jo Kelcey, with the technical support and input of Donna M. Mertens and Michael Ungar. It has also greatly benefited from the reflections and input of Education Resilience researchers in Central America and Colombia, South Asia and the Middle East. We are especially thankful for the peer review comments from Abdul Hai Sofizada (Education Specialist, Afghanistan), Francisco Marmolejo (Higher Education Specialist, HDNED ), and Donna M. Mertens (Mixed Methods Research Specialist, Consultant). 1 1 The presentation format for this training Guidance Note is derived from the design of the “Education and Fragility” Inter-Agency Training (Nairobi, 2011) developed by Nina Papadopoulos (USAID) and Rachel McKinney (Save the Children). 5 GUIDANCE NOTE Getting Started 6 At-a-glance This section provides a summary of the overall RES-Research structure, discusses its objectives, intended audience and associated participant expectations. Whereas the other sections of the RES-Research users guide provide content and pedagogical guidance, this section is intended to support planning and project management. It is based upon two regional applications of the RES-Research module in Central America and the South Asia, during the period 2011-2014. Content 1. Target audience 2. Rationale 3. Objectives 4. Expectations 5. Administrative and logistical considerations 6. Materials 7. Training Structure 8. Module structure 9. How to use the Guidance Notes 10. A final request Target audience RES-Research is intended for the professional research and evaluation communities in countries affected by violence and conflict. Specifically, it targets university students at the research Master's and Doctoral levels, University faculty, and the professional evaluation community working with local and international civil society organizations interested in education policy and practice. Bringing these groups together needs to be seen as an important strength of the overall approach as it provides a crucial link between policy and research. Outreach to these communities is an important first step in planning your RES-Research. Several institutions, networks and professional bodies may support the applications process. These include: Universities and colleges providing tertiary education social sciences and humanities Counterparts in UN agencies, (I)NGOs The Inter-Agency Network for Education in Emergencies (INEE) Scholars at Risk Network 7 Scholar Rescue Fund and its parent organization the Institute of International Education Professional evaluation communities Rationale RES-Research is an experiential learning program. It provides technical guidance for conducting education resilience research in difficult contexts through a 9-12 month approach that combines theoretical and practical learning. As an experiential approach RES-Research embraces the challenges that Education Resilience research presents, recognizing that the lessons learned through this are an important part of the process. A key rationale for RES-Research is to bring together the worlds of research and policy making and practice. Notably, while university actors are used to conducting research on varied topics, not all researchers from the Global South are provided with the tools to access and influence domestic and international education development policy making and practice; nor is the evaluation community always provided with professional development opportunities to contribute applied knowledge in new and influential ways. RES-Research intends to bridge these gaps, in the process strengthening indigenous research capacities and promoting more sustainable evidence generation and use. RES-Research brings researchers and evaluators together in a South-to-South learning process. However, it takes knowledge exchange a step further than traditional “sharing of good practices.” Instead it promotes knowledge generation precisely from researchers and evaluators themselves living in contexts of adversity, supporting them to influence and inform current debates not only in the Global South but also within the international education development community. The table in the next page presents the core objectives of RES-Research. 8 Resilience in Education Settings Research (RES-Research) Training Program Objective Contribution to south to south Transformative objective learning Build on RES-Research is premised upon Through careful participant selection, capacities in recognition of existing capacities mutual learning objectives and the countries within countries affected by violence sustainability of capacity building affected by and conflict. It strategically targets approaches are better ensured. The violence local researchers and evaluators who approach aims to create local networks and conflict are well positioned to connect for change. evidence with policy. Learning by RES-Research combines theory and Two challenges face research in doing practice for a more holistic teaching countries affected by violence and and learning approach. It is grounded conflict. First, the culture of outsourcing in an empowering pedagogical capacity to external organizations can approach that supports local result in studies that are less familiar researchers and evaluators to with the context and complexities than influence internal and external local actors would be. Second, research debates on education resilience. By in countries affected by violence and actively engaging local actors to conflict presents significant operational undertake pilot studies the ERA and ethical challenges. By working with program seeks to improve the local actors and ensuring that capacity sustainability of knowledge through building approaches are grounded in locally relevant studies and appropriate methodologies RES- recommendations. Research aims to improve the quality of research by ensuring better representation of vulnerable communities in education policy and practice and ultimately providing more targeted and relevant recommendations. Research By focusing on the skills and RES-Research actively engages with for public strategies needed to influence and policy and practice. By including sessions policy and bring about change, and by working on translating research and the influencing through the networks implicit in the discussion and application of strategies practice target participants, RES-Research to influence policy and practice it promotes the generation of demand supports knowledge generation not only driven knowledge and knowledge for professional development but also sharing. more systematic education led change processes: education system pro-social transformation. 9 Expectations RES-Research is a practical training program. A practical component (the design, pilot and implementation of an Education Resilience study) runs throughout the training program. Pilot studies provide an opportunity for data collection, analysis, interpretation and report writing. Workshop participants are expected to form country or sub-national teams and work together towards the implementation of a transformative education resilience study. The focus, topics and research questions of each study are selected solely by the research participants and their in-country stakeholders. In between workshops, in-country research teams work together on the development and implementation of their study. During the face-to-face workshops, they reunite to discuss their progress and challenges they have had and to learn more about data analysis, interpretation, and report writing. Ultimately, the teams are provided with guidance regarding the use of data to influence policies and program for positive change in contexts of adversity. The ERA program supports the dissemination of the education resilience findings within each country case study, in addition to other publication sources that the authors may identify, such as agency reports or academic journals. This strong practical element has several implications: First, teams of researchers are encouraged to apply for RES-Research. Although the teams do not need to comprise individuals who have worked together closely beforehand, they should be made up of individuals who are willing and able to do this. Second, institutional commitment is important for research teams. This means that teams should have the support of an education oriented NGO or Agency or their university. This support does not necessarily need to be financial. It should however allow the participants better access to their target population and provide them with the in kind support needed to implement a study. Finally, RES-Research is above all a program that looks to identify and build on local capacities to conduct education resilience research. Participants and their supporting institutions therefore need to share these goals and believe in the crucial contributions they have to make towards positive educational change for vulnerable communities. 10 Administrative and logistical considerations for international- based training Any domestic or international institution (universities, research tanks, evaluation communities) can use the RES-Research materials for their own education resilience-based research and evaluation training. However, for international agencies that wish to provide training in one location with participants from different countries, the mechanism utilized by the World Bank’s RES-Research team is detailed next. This provides some initial guidance including financial support for participants—such as travel, accommodation and ‘seed funding” for resilience case studies. Application process: Potential participants can be identified through local universities, evaluation communities and think tanks. This broad identification of sources of interested participants should be followed by an application process. This should be started minimally 6-8 weeks before the date of the first workshop, in order to leave enough time for visa processing, travel bookings etc. As well as comprising an expression of interest in the training, the application form should also ask potential participants whether or not they have funding to cover the costs of participation through their university or agency. There are however no workshop fees. The costs of participation require travel to the workshop locations, accommodation during the workshop dates and meals and other incidentals. Travel support: To promote maximum access to researchers and evaluators from the Global South, RES-Research prioritizes having funding available for selected participants who would be otherwise unable to attend. Economy class airfare to the workshop destination, accommodation, visa costs, hotel transfer and three meals a day are included as part of the RES-Research funding package. In cases where road travel is feasible then a coach and driver may be rented to transport participants (as was the case in Central America for some participants). It is important to be clear to participants that they will not be paid per diem for the days of the workshop as all of their related expenses are covered directly. Visas: Participants are responsible for applying for their own visas although the training team agency may cover visa costs and also should issue participants requiring visas with a ‘comfort letter’ to facilitate visa issuance. Seed funding: Funding is also provided to facilitate the required practical component of RES- Research. It is called “seed” funding as it is not intended to finance the full cost of a research endeavor. In recent applications of the RES-Research training, this ‘seed’ funding has been for up to 10,000USD per team. It is intended to complement basic researcher costs over the 9-12 11 month period of a pilot Education Resilience case study, which complements the RES-Research training workshops. RES-Research participant teams are required to apply for ‘seed funding’ only if without this support the practice resilience study would not be possible (given the associated costs of carrying out research such as transportation to and from research sites, refreshments for focus groups or the costs of statistical software as well as some research team labor costs). Participants apply for funding at the end of the first workshop. They are required to submit their draft research plans as well as a budget and timeframe. They must also nominate one member of their team as the administrative ‘coordinator’ for the budget (see below for more information on this). Teams do not need to apply for the full 10,000USD. They may instead apply for a smaller amount to conduct an initial phase within what will (eventually) be a larger study. It’s more important for each team to submit a feasible proposal that will allow them to undertake a study that amounts to a quality practical learning experience within the timeframe of the program. Study coordinator: Seed funding for the practical studies can be disbursed through only one or two of the research team members, selected as ‘coordinator.” Funding can be in the form of short term consultancy, based on a “research coordinator” terms of reference, including two or three deliverables over the 6 -9 month period covered by their research proposal. In other words the coordinator is responsible for managing and submitting the team’s deliverables and for then disbursing the payments as agreed in the team budget. Templates for the coordinator terms of reference are available. The 6-9 months period for the practical education resilience case study does not - and should not - preclude the sustainability of the research into additional research cycles. Rather it is just an incentive and practice-based learning for local research teams to conduct their own studies. However, at the end of the RES-Research training, a preliminary study progress report is required. Templates for these preliminary reports are available as part of this open source RES- Research package. 12 Materials As mentioned at the outset, RES-Research is an open source package of education resilience- based research training content and resources. These materials can be adjusted as needed and complemented based on the needs and objectives of each training program and stakeholders. Based on the experience of the World Bank’s ERA team, the following are guidelines to share documentation with participants. Documentation: Prior to the first workshop, participants should be provided with background resources. In previous applications, drop box files have been created to share this documentation. These folders served as an online repository of useful information that could support not only their understanding of the program but also more specific aspects related to data collection, analysis and reporting writing. Participants should also be provided with printed copies of all presentations and activities. To facilitate the learning process the use of laptops is encouraged during the workshop. This not only allows participants to take notes throughout but also electronic templates for various group exercises form an important part of the workshops. Training structure The following table provides the overall structure for the RES-Research program, which is organized around 9 core modules that cover the lifespan of an education resilience research study. It is recommended to implement the program through three face-to-face workshops during which technical assistance is provided over a twelve-month period. This is the most enriching approach as it ensures participants have enough time to collect data and it provides them with guidance in 'real time' for each stage of their research. It is however possible to implement RES-Research in a more condensed format over 9 months with two workshops. In 13 both cases it is essential that the implementing team is available for technical assistance to the teams. This may be done virtually or, if resources allow, in person. The ERA team has developed several reporting templates (consisting of guiding questions) that can be used to facilitate this. Module structure Almost all modules consist of 20 - 30 power point slides. The first and last modules as well as Module 5B are slightly shorter as they devote a significant amount of time to participant led discussion and presentations. Module 5 bridges planning and practice and is therefore divided into two sections - one that takes place during the first workshop to support the research preparation, and a second part which elicits feedback once the studies are underway. Other modules are directed by facilitators, although all modules should plan for adequate time for participant check-ins and feedback. All modules are designed to promote team discussion among the larger group and within country teams. Most modules include at least one team or group exercise though facilitators may choose to add additional exercises and should also insert energizer activities as needed. It is crucial to leave enough time for these exercises as they are pivotal for the experiential learning process: most of the exercises directly contribute to the design of the studies and subsequently to their data analysis and interpretation. Table two below provides the exercise breakdown over the course of the 9 modules for which it is recommended that implementation occurs over 8 - 9 days. Table 2: learning activities and exercises Module Activities and exercises 1. Introduction and Group exercise: individual participant led feedback on expectations expectations and contexts of interest 2. Resilience Theory Activity: resilience navigation and negotiation (in pairs) Group exercise: individual participant reflections on experiences with resilience and examples from their contexts. Group feedback optional 3. Positioning the study Team exercise: defining the purpose of their team's transformative mixed methods resilience study. Feedback to the group 14 4. Designing the Study Team exercise: developing a sampling strategy Feedback to the group 5a. Field work and Team exercise: developing a data collection and data analysis plan Feasibility planning Team led exercise of presenting initial findings and field experience back to the wider group (see presentation template in RES-Research 5b. Field work and package) Feasibility planning 6. Analysis and Team exercise: practicing data coding interpretation for Feedback to the group policy and practice Team exercise: determining a statistical analysis plan Feedback to the group 7. Knowledge mobilization I: Policy and reporting 8. Knowledge Team exercise: developing a reporting and engagement plan mobilization II: Feedback to the group external engagement, Group exercise: participant feedback and thoughts on critiques of giving back and resilience in theory and practice critique 9. Open reflection Group exercise: open technical support and participant led feedback on RES-Research process and options for sustainability. If RES-Research is implemented over a 12 month period it is recommended that the following modular structure is adopted: workshop 1: modules 1 - 5a workshop 2: modules 5b - 6 (this could also be implemented through in country support to teams, by facilitators, and by extending module 6 so that teams and facilitators work more closely with data that has been collected) workshop 3: modules 7-9 If the 9 month structure is used then the breakdown is as follows: workshop 1: modules 1 - 5a workshop 2: modules 5b - 9 (with the understanding that reporting and knowledge mobilization will need to be discussed in less tangible terms as the teams are less likely to have completed data collection) 15 How to use the Guidance Notes These Guidance Notes provide operational support to deliver training aligned to the RES- Research Manual (Reyes and Liebenberg 2013). They should be used in conjunction with the Manual and the power point presentations. Each Guidance Note corresponds to a different RES-Research learning module. All are structured in the same way, setting out key messages, learning objectives and core content. Suggested timing for each component of the module is also provided. It is recommended that facilitators do not dedicate less time than is indicated but may choose to expand some sessions with additional explanations or team activities and discussion. Resources (academic papers and relevant websites) have also been included to provide facilitators with more robust references, as well as activities which determine the teaching and learning approaches in each module. These reference lists should also be provided to participants to allow them to delve more deeply into the subject matter. Where handouts have been developed for different modules, these are also indicated. The Guidance Notes are intended as just that - a guide. Facilitators may and should adapt and add to the content and the direction provided in these notes, as best suits the learning needs of RES-Research participants. Finally, the Guidance Notes should be used in conjunction with the RES-Research Manual. The two resources are complementary: the Guidance Notes are intended to support RES-Research facilitators, while the Manual is intended as a reference and resource for participants as they plan, implement and use their Education Resilience Research. A final request RES-Research is a open source resource for both World Bank staff, as well as those from other agencies that are engaged with policy and practice. In order that the ERA program may reflect the rich and diverse learning that an open source policy supports, we would appreciate your feedback and knowing about any lessons learned from implementing RES-Research. This includes reflections and reports regarding the RES-Research process as well as the outputs from the experiential model. Notably, the ERA team is committed to building an evidence base around education resilience: one that is driven by local researchers who are striving to improve education policy and practice in contexts of violence and conflict. We request therefore that you share your thoughts and feedback with us. 16 Table 3: RES-Research Content Schema Module Core content 1 Introduction and expectations - Background to the ERA program and RES-Research - Education in contexts of violence and conflict and the role of the World Bank - Workshop expectations and learning approach 2 Resilience Theory - Waves of resilience research - The social ecological perspective and resilience as a process - Capturing complexity in resilience research - Resilience and education settings and systems 3 Positioning the study - Research paradigms and the transformative research ontology and epistemology - Mixed methods in human and social research - Axiology in the transformative paradigm - Transformative resilience research purpose 4 Designing your Study - Research question and sub questions - Samples and sampling strategies - Data collection tools and data analysis plans 5 Planning And Practice: Feasibility, Field - Participant access and local advisory committees Work And Analysis - Timeframe, budget, research teams and resources - Field practice - Lessons learned, practical challenges and sharing findings 6 Analysis and interpretation for policy and - Qualitative data analysis and interpretation practice - Quantitative data analysis and interpretation - Mixed methods analysis and transitions (qualitative to quantitative and quantitative to qualitative) - Validating data findings with the community 7 Knowledge mobilization I: Reporting - Translating research to policy and practice - Current debates: resilience in international education development - Writing an education resilience report 8 Knowledge mobilization II: policy, external - Influencing policy engagement and giving back - Engaging with external organizations - Social justice and giving back 9 Open reflection - Local researcher empowerment - Constructive critique - Sustainability of local evidence for policy and practice 17 18 GUIDANCE NOTE FIVE B Planning and Practice: Feasibility, Field Work and Analysis 1 Module-at-a-glance Content Time Instructional Activity Field practice 30 minutes Brief welcome and introduction by facilitators Lessons Learned, practical challenges and 2 hours and 30 Team led presentations: sharing findings minutes Sharing your findings slide 3 Total Time Half a day Learning Objectives Each team is provided with an opportunity to report on the status of their study and to reflect upon opportunities, challenges and the process thus far Teams benefit from the advice and technical support of expert facilitators as well as their peers who are in similar stages of their research Key Messages Field work presents unforeseen challenges and opportunities Conducting a transformative education resilience study is an iterative process: initial plans may need to change and adjust to realities on the ground. Core content Module 5B is participant led. Participants are sent an electronic power point template several weeks before the second workshop, that they should use and adapt to present their findings back to the group. Each team has 10-15 minutes to present and then a 20 minute open discussion promoted by facilitator questions ensues. The module is expected to set the scene for the rest of the second workshop which is structured around the analysis and use of actual data collected by the teams. This part of module 5 therefore serves as a briefing for the facilitators (and wider group) to set the scene for the second workshop. 2 Bibliography and Support Material Greene, J. 2007, Mixed Methods in Social Inquiry, (San Francisco: Jossey-Bass). Liebenberg, L. 2006. The “us” and “them” in research: Can we get around it? Qualitative research in organizations and management, 1(2), 138-140. Liebenberg, L., and M. Ungar. 2009, Introduction: The challenges of researching resilience, In L. Liebenberg and M. Ungar. (eds.) Researching Resilience, (Toronto: University of Toronto Press) 2011. Ethical concerns regarding participation of marginalised youth in research. Bulletin of the International Society for the Study of Behavioural Development, 1(59), 24-27. Mertens, D. 2009. Transformative Research and Evaluation. New York and London: The Guildford Press. 2014. Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods. SAGE Publications, Inc. Fourth Edition. Mertens, D., F. Cram and B. Chilisa. (Eds). 2013. Indigenous Pathways into Social Research. Voices of a New Generation. Left Coast Press: California. Handouts Power point template to facilitate virtual feedback during implementation phase. Circulated electronically prior to the second workshop. Session Activities 1. This module is participant led. Facilitators do not present during this session but rather serve as timekeepers for the presentations and question and answer periods. During the Q&A, facilitators should prompt discussion on the team's data collection. This may include questions on how the team addressed any challenges; how initial plans may have altered, as well as new opportunities that have arisen since the field work began. 3 GUIDANCE NOTE 6 Analysis and Interpretation for Policy and Practice 1 Module-at-a-glance Content Time Instructional Activity 1. Qualitative Data Analysis and 2 hours Interpretation Facilitator presentation 2. Quantitative Data Analysis and Qualitative data coding slide 12 2 hours Interpretation Statistical analysis plan slide 22 3. Mixed methods analysis (transitions 1 hour 30 and integration) minutes 4. Validating data findings with the 1 hour community Total Time 1 day 30 SLIDES Learning Objectives Be aware of the range of ways in which qualitative and quantitative data sets can be analyzed and interpreted Understand that analyzing qualitative data draws upon context, researcher perspectives and theoretical frameworks Be aware of the different types of statistics that can be used to analyze qualitative data and understand what these statistics allow researchers to deduce about a given sample and where their limitations lie. Understand that different mixed methods designs have implications for how qualitative and quantitative data analysis are brought together Learn how to transition between qualitative and quantitative data sets (and vice versa) Understand the transformative rationale for community involvement in data analysis and interpretation and identify appropriate ways of doing this related to the team's study Key Messages Following data collection it is important to prepare data and define a systematic process for its analysis Data analysis should be structured around the core components of a resilience framework - risks, assets (protective and promotive) and relationships Coding of qualitative data is an inductive process that helps to summaries and ultimate theorize around key education resilience themes and concepts. 2 Education Resilience analysis and interpretation may also be complemented and enriched by other theoretical frameworks to be selected depending on the particular focus of each study. Three main types of statistical analysis offer different insights to explore, describe and explain data sets. Statistical analysis must be used cautiously however so as not to present findings based on incorrect assumptions about the vulnerable population, or that are devoid of important information related to group differences. Mixed methods involves combining qualitative and quantitative data during analysis as well as interpretation stages to provide different insights and enhance the transformative potential. Participant engagement should also be sought during data analysis and interpretation with partnership strategies continuing into the reporting and use of data. Core content The module begins with a presentation on how to analyze qualitative data. It is important to stress to participants the need to have a systematic process for managing qualitative data (from its preparation (such as transcriptions) through to the analytical framework that is selected. During the analytical process it is important to revert back to our foundational resilience concepts (discussed in Module 2). Facilitators may want to provide a summary recap on the core components of resilience studies as this should form the basis for any analytical framework. Although qualitative data may be analysed through a range of approaches this module focuses on coding and provides participants with an opportunity to practice coding. Participants are therefore provided with the opportunity to apply resilience theory to the thematic analysis of a practice data set in order to experience the qualitative data analysis process. Participants should be aware of what different analytical approaches allows them to say and their limitations. As such facilitators should also revisit the early discussion of data analysis and the differences between exploring, describing and explaining phenomena. At the same time that resilience theory provides the core thread for analysing the data, participants should also be made aware of other analytical and theoretical frameworks that may 3 supplement resilience theory and provide additional more in-depth insights into the particular issues that are emerging from their data (slide 9). The section on qualitative data then concludes with brief discussion of manual versus computer based analysis. Of note, given the experiential learning focus of RES-Research, it is recommended that the teams begin with manual analysis in order to familiarise themselves with the various steps in the process of preparing data. The module then shifts to quantitative data analysis and interpretation. During the data analysis planning (previous workshop) facilitators should have ascertained the level of comfort and experience within the group with regards to qualitative and quantitative analysis (thus far the majority of participants have been less comfortable with quantitative analysis). Participants should understand that statistical analysis is about reducing their data to obtain summarized information, to see a relationship between variables or to compare differences within their data sets. Participants are first introduced to the three types of statistics that can support the analysis of quantitative. Given the number of technical terms which will be presented (to audiences for whom English is likely a second language) it is important to begin with definitions and ensure clarity over what different types of statistics mean and offer to the mixed methods analysis. Participants should also be aware that statistical software is required for higher end quantitative analysis. For teams looking to initiate themselves with STATA and SPSS a brief introduction is included in the RES-360 manual. Participants should refer back to their statistical analysis plans, developed in module 5A, considering in more detail the specific purpose of their analysis. In all cases, descriptive statistics will be the appropriate starting point for most quantitative analysis. Teams must also consider the value added of other data manipulations as it relates to the purpose of their study. In addition to understanding the role and purpose of different statistical tests, the transformative paradigm also however requires critical consideration of results that will be obtained from statistical analysis (Mertens, 2009; 299). Facilitators should therefore also discuss with participants the concepts of statistical 'significance' and the potential harm that can come from generalized, aggregated results when dealing with complex contexts within which the diversity of risks and assets are crucial dimensions. Notably, statistical significance as an estimated measure of the degree to which a result is 'true' and reliable needs to be carefully weighed against the erosion of group 4 differences and possibility of false assumptions that arise regarding the size of the sample, characteristics of the population and the degree to which the sample can represent the population. Chapter 9 of Mertens (2009) discusses these issues and provides guidance on several statistical approaches that can be used to help offset over generalizations and maintain a transformative perspective throughout quantitative analysis and interpretation. The module then brings qualitative and quantitative analysis and interpretation together. This section begins with a discussion of the types of transitions that can be made between qualitative and quantitative data sets. There is then a slide on integrating results within the given mixed methods design of the study. This begins with an overview of the qualitative and quantitative analytical methods that relate to research sub-questions (providing an overview of the analytical methods that the studies will be using). The next slide reviews the rationale for mixing methods (see also Mertens, 2009; 309). Facilitators should recall the different types of mixed methods design and the levels at which integration can occur. Participants should be aware of the purpose of integrating analysis and interpretation; including to complement findings, expand upon findings from one cycle of the research to the next, add clarity and even raise additional questions. This discussion should be encouraged by having participants revisit their research purpose and research questions to consider how they will integrate findings during analysis and not only interpretive phases to enhance the transformative potential (Mertens, 2009; 310). The module ends with a discussion on validating findings with the community. The two slides in this section are intended to spur discussion on the importance of community validation at all stages of the research - including during analysis and interpretation - as well as the types of strategies that may be appropriate in different contexts. This section also serves as a transition into the next modules (7 and 8) that deal with knowledge mobilization for which community involvement remains paramount and should play a significant role in guiding reporting and use of data if we are to maximize the positive contributions the studies can make to supporting social justice. 5 Bibliography and Support Material American Psychological Association. 2014. Publication Manual of the American Psychological Association, Sixth Edition. Washington DC: American Psychological Association. Mertens, D. 2009. Transformative Research and Evaluation. New York and London: The Guildford Press. 2014. Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods. SAGE Publications, Inc. Fourth Edition. Reyes, J., L. Liebenberg and D. Meza. 2013. RES-360°: Resilience Rapid Assessment Manual. Washington DC: The World Bank. 2013. RES-360°: Resilience Rapid Assessment Tool Kit. Washington DC: The World Bank. Reyes, J., D. Mertens, and L. Liebenberg. 2013. "RES-Research Research Methods Manual." Washington DC: The World Bank. Ungar, M. and Liebenberg, L. 2013. “Ethnocultural factors, resilience, and school engagement.” In School Psychology International 2013 34: 514. Sage Publications. 2011. Assessing Resilience Across Cultures Using Mixed Methods: Construction of the Child and Youth Resilience Measure. Journal of Mixed Methods Research, 5(2), pp.126- 149. Handouts Education resilience interview transcript (Female Palestinian refugee student in Gaza) Coding template for the interview transcript (Female Palestinian refugee student in Gaza) Summary of codes for the interview transcript (Female Palestinian refugee student in Gaza) Handout on creating questionnaires 6 Session Activities Qualitative data coding (slide 12): This exercise is based on an early education resilience study conducted by the ERA team. This example is from an interview conducted with a female Palestinian refugee student living in Gaza. Participants are provided with three handouts which also offer templates for preparing qualitative data. The templates are intended to guide the participants through first identifying references to the core resilience components (risks, assets and outcomes) and then moving from these initial codes to categorising the data into overarching themes, to finally transposing the data to themes that reflect resilience theory and core concepts during the axial coding stage. Participants should work in their country teams and facilitators should underscore that coding is not an exact science but rather an interpretative process that draw upon the understanding of context and the particular focus of the study. This can also be highlighted through the commonalities and differences that arise during the team feedback on the coding process and their determination of categories and themes. Statistical analysis plan (slide 22): The second activity of the module does not require participants to quantitative manipulate the data, but instead to plan for it. Based on their research purpose and questions, teams should spend time deciding which types of statistics will reveal the exploratory, descriptive or explanatory information required from their studies. In line with the importance of adopting a critical lens they should also identify the limitations of this analysis and the assumptions upon which this is based. Facilitators should make clear in this exercise that especially for large scale data sets, quantitative data is often outsources to statisticians, however teams considering this will still need to know how to effectively guide the process as well as what different types of analysis mean. 7 GUIDANCE NOTE 7 Knowledge mobilization 1: Policy and reporting 1 Module-at-a-glance Content Time Instructional Activities 1. Translating Research to Policy and Facilitator presentation Practice 3 hours 2. Influencing policy 3. Writing an Education Resilience Report 2 hours 30 minutes Total Time 1 day 27 SLIDES Learning Objectives Understand the policy-making environment and consider the political economy and its implications in each team’s context Learn about the engagement process and the main ways that researchers can influence policy and programming Understand the needs of education policy makers and the implications of this for resilience research process, findings and recommendations Know the core components and basic structure for a written education resilience report Key Messages Education Policy makers have particular needs and concerns and to effectively promote change, resilience researchers must be aware of this and actively look to address them Influencing policy is a process that requires capturing policy makers attention, presenting compelling evidence and good practice examples and using the data collected from the study to propose policy alternatives Reporting does not begin after data collection is over. Both the redaction and dissemination must be planned for Reporting is not only a question of documenting findings, but of engaging with questions of power and privilege. It implicitly requires a stakeholder analysis and identification of those who will gain and those who may lose out from the findings. Just as at other stages of the study design and implementation, reporting raises important ethical considerations including questions of data ownership, authorship and representation. 2 Core content A transformative resilience approach recognizes that for research to be effective it must address both technical as well as political considerations. Although policy and practice considerations have been present throughout the resilience study positioning and design, this module goes into much greater detail around the needs and concerns of policy makers, in order to promote effective strategies to bring about change. The module begins by setting out the core needs of education policy makers. It is important however to recognize that these are premised on political regimes that are assumed to be somewhat responsive to public pressure and democratic in functioning. Participant teams should therefore be encouraged to discuss the particular political economy that drives their political context so that they can better identify relevant pressure points for change. Understanding the change climate is an essential first step when planning how to use data. When communicating to / with policy makers and those with influence, it is also important to recap on the value added of a resilience approach. This slide sets out what a resilience approach offers, which makes it unique and differentiates itself from other research frameworks. It is especially important to stress to policy makers the ways in which the study recognizes and values what is already in place at which ever level this exists. At the same time, the resilience studies do not shy away from complexity nor do they focus on quick fix linear solutions. Policy maker needs and the particular aspects of a resilience approach need to come together if change is to be achieved. This diagrams demonstrates the points at which resilience findings can intersect and respond to policy makers needs. The process that facilitates this - the research - policy translation - is an important dimension of knowledge mobilization and will be discussed over the remainder of module 7 and in module 8. 3 Policies should be understood as institutionally agreed aims and programs and projects as the formal means to achieve this. This slide presents a process for influence: identifying the core components that each team needs to present through an analysis and interpretation of their findings. This will also require drawing up secondary data sources (for example evidence of good practices elsewhere that can be adapted to the context in question). The next slides deal with each component of the process in turn. They discuss the topic in terms of the lessons that the Resilience Research Centre (in Canada) has learned over the course of their work. As such these lessons will need to be considered and ultimately adapted to the particular policy environment and context discussed in the previous module. Facilitators should ensure enough time for the teams to discuss this. Especially important to note within this process is the assumed tradeoff between scientific rigor and community ownership. As we move up the research pyramid towards more generalizable evidence, research methods tend to see a concomitant decrease in community ownership. This is a challenge that the transformative resilience research studies takes on - identifying ways in which community ownership can be maximized while also scaling up our evidence base through mixed methods and retaining rigor and validity of results (including through its adoption of a more holistic understanding of these concepts). Policy messages will also benefit from certain core content. In particular, developing a theory of change and indicators will allow policy makers and policy implementers to track progress and meet their short term change and impact objectives. This should not however detract from the longer term nature of transformative impact which remains the main objective and should be complemented through building on capacities and longer term planning. 4 The module closes with a discussion of report writing and effective display of data. Participants have been planning for data utilization throughout their study positioning and design and this section should reinforce what has gone before. The first thing to determine who needs the information to bring about change and social action. Different audiences may be selected in order to provide multiple pathways to change. It should be recognized however that different audiences will require different reporting strategies. Although reports present to the reader the activities undertaken as part of the study they must do much more than this. As Mertens (2009; 314) discusses, they must manage issues of power and inclusion in their timing and format as well as critically consider dissemination and utilization strategies. Thus even when the audience is one policy makers, teams should be considering who will be affected by their report and how the information presented will support positive change for them. The following slides present ways in which qualitative and quantitative data may be displayed. Equally important is the dissemination strategy for the report which raises questions of who is representing the perspectives of the vulnerable community at the centre of the study. Issues of shared authorship, data ownership and resistance to findings all need to be considered and planned for during report planning (see also Mertens 2009, chapter 10 for a more in-depth discussion). Strategies that promote discussion and transparency should be encouraged. The module concludes with set of guiding ethical questions that need to be integrated throughout the reporting phase. The questions are designed to allow the teams to think ahead to the various challenges - and power struggles - they may encounter (and therefore plan for them). This serves to reinforce the fact that reporting is not only undertaken at the end of the study data collection, but in fact needs to be planned for, if it is to be most effective. 5 Bibliography and Support Material Maton, K. I. 2000. Making a Difference: The Social Ecology of Social Transformation, American Journal of Community Psychology, Volume 28, Issue 1, pp. 25-57 2005. The Social Transformation of Environments and the Promotion of Resilience in Children. In R. Peters, B. Leadbeater, and R. Mc.Mahon (eds.), Resilience in Children, Families and Communities, (New York: Kluwer Academic/Plenum Publishers) 2008. Empowering Community Settings: Agents of Individual Development, Community Betterment, and Positive Social Change, American Journal of Community Psychology, Volume 41, Issue 1, pp. 4-21 2008. Community Resilience as a Metaphor: Theory, Set of Capacities and Strategy for Disaster Readiness, American Journal of Community Psychology, Volume 41, Issue 1, pp. 127-150 Maton, K.I, Schellenbach, J., Leadbeater, B. and Solarz, A. 2004. Investing in Children, Youth, Families and Communities: Strengths-Based Research and Policy, (Washington, D.C.: American Psychological Association) 86. Mertens, D. 2009. Transformative Research and Evaluation. New York and London: The Guildford Press. Peters, R. D., Leadbeater, B. and McMahon, R. 2005, Resilience in Children, Families and Communities: Linking Context to Practice and Policy, (New York: Kluwer Academic/Plenum Publisher) Steiner-Khamsi, G. Ed. 2004. The Global politics of educational borrowing and lending. Teachers College, Columbia University: New York. 6 GUIDANCE NOTE 8 Knowledge mobilization 2: External Engagement, Giving Back and Critique 1 Module-at-a-glance Content Time Instructional Activity Engaging with External Organizations: Reporting and engagement 1 hour large and small plan slide 16 Social Justice and Giving Back 2 hours Critiques of resilience slide 22 Current debates: Resilience in 2 hours 30 International Education Development minutes Total Time 1 day 22 SLIDES Learning Objectives Know that there are important differences when influencing for change between large and small organizations and be able to interpret lessons learned relative to each team’s context Be aware of current debates regarding the use of resilience in international development Identify critiques that may be relevant in each team's context and discuss how to engage and address these Identify ways in which the study will actively seek to give back to the vulnerable community it intends to support Key Messages Messages and approaches to engage with policy makers and practitioners must be tailored to the particular context (political economy and policy making environment) as well as the type of organization that is being engaged through the study. Transformative research defines itself through its accountability not only to policy makers but to the vulnerable communities it is concerned with: this requires explicit thought and action in order to 'give-back'. Critically engaging with the concept of resilience is essential to maximize the transformative potential of a study. Critique can guide researchers towards better considered arguments and more explicit recommendations. 2 Core content In addition to adjusting the lessons learned to the specific context of each team's study, the strategies for influence should be adapted to the type of organization that can support policy, program and practice change. In particular, smaller organizations that work more at the community level appreciate proximal evidence and to hear about promising practices from individuals who are operating programs. Building partnerships and coalitions between stakeholders as well as advocating for change are possible approaches. On the other hand, larger organizations and institutions tend to prefer larger data sets that support generalizable findings and change processes. The module the shifts to a section on social justice and giving back as it relates to ethical responsibility, quality purposes and for pragmatic reasons. This includes ways in which to ground the lessons of your research among the various stakeholders and to make connections and alliances between stakeholders that can promote transparency and accountability as part of the momentum for change. Here, facilitators may recall that the difference between a transformative paradigm and the pragmatic one is the responsibility not only to policy makers but the wider constituency of stakeholders. Especially relevant is to consider for transformative purposes, is how each study will give back to the vulnerable community that participated in the study. During this slide teams should be given plenty of time to discuss and report back on their planned approaches for giving back. This may include activities such as ethno-drama that allow young people to express their perspectives, community town halls to spur discussion and action, or if visual methods are used exhibits of pictures and photos. Facilitators may wish to explicitly draw upon the empowerment education models developed by Freire that promote community based critical thinking and engagement for reform. 3 The need for critical analysis runs throughout the transformative resilience approach and knowledge mobilization is no exception. Complementing their political economy analysis which is essential to the contextual relevance of their findings, teams should also critically engage with the concept of resilience. While resilience has gained much traction in terms of rhetoric within international education development is also has a growing number of detractors. To positively contribute to both in country and wider debates it is important for the teams to be aware of and engage with these critiques. Many of these critiques require revisiting content of Module 2 (during workshop one). In particular the importance of social ecology perspectives in situating resilience as a dynamic process that draws not only on assets among vulnerable individuals and communities but which by definition requires the support and resourcing of service providers (thereby stressing their role and responsibility in promoting and, importantly, sustaining resilience). The table of resilience debates should be discussed in plenary guided by the activity questions - see below). Bibliography and Support Material Bene, C., Wood, R., Newsham, A., Davies, M. (2012), Resilience: New Utopia or New Tyranny? Reflection about the Potential and Limits of the Concept of Resilience in Relation to Vulnerability Reduction Programs, IDS Working Paper 405, (Brighton, UK: Institute of Development Studies). 78 Davoudi, S. (2012). Resilience: A bridging concept of a dead end? Planning Theory & Practice, June 2012; 13 (2): 299-307. Freire, P. 1994. Pedagogy of Hope. New York: Continuum. 1996. Pedagogy of the Oppressed. London: Penguin Books. 4 2001. Pedagogy of Freedom. Lanham/Boulder/New York/Oxford: Rowman & Littlefield Publisher, Inc. IDS. February 2013. "Making the Most of Resilience". IDS in Focus Policy Briefing, Issue 32. Luthar, S.S., D. Cicchetti and B. Becker. 2000. “The construct of resilience: A critical evaluation and guidelines for future work.” Child Development 71(3): 543-562. Maton, K. I. 2000. Making a Difference: The Social Ecology of Social Transformation, American Journal of Community Psychology, Volume 28, Issue 1, pp. 25-57 2005. The Social Transformation of Environments and the Promotion of Resilience in Children. In R. Peters, B. Leadbeater, and R. Mc.Mahon (eds.), Resilience in Children, Families and Communities, (New York: Kluwer Academic/Plenum Publishers) 2008. Empowering Community Settings: Agents of Individual Development, Community Betterment, and Positive Social Change, American Journal of Community Psychology, Volume 41, Issue 1, pp. 4-21 2008. Community Resilience as a Metaphor: Theory, Set of Capacities and Strategy for Disaster Readiness, American Journal of Community Psychology, Volume 41, Issue 1, pp. 127-150 Maton, K.I, Schellenbach, J., Leadbeater, B. and Solarz, A. 2004. Investing in Children, Youth, Families and Communities: Strengths-Based Research and Policy. Washington, D.C.: American Psychological Association. 86 Pathways to Resilience project. Website. http://www.resilienceproject.org/research/projects/pathways-to-resilience “The Pathways to Resilience Research Project (PTR) is a mixed methods research study that examines service use patterns, personal and ecological risk factors, and aspects of resilience of youth across different cultures, contexts, and with complex service histories. It began in Canada and now includes partners in at least five countries: Canada, New Zealand, South Africa, Colombia, and China.” Peters, R. D., Leadbeater, B. and McMahon, R. (2005), Resilience in Children, Families and Communities: Linking Context to Practice and Policy, (New York: Kluwer Academic/Plenum Publisher) 5 Session Activities Reporting and engagement plan (slide 16): Country teams should work through a series of questions that serve to structure the reporting, engagement and dissemination stages of their research. Critiques of resilience (slide 22): As a prelude to the discussion that will occur during module 9, this activity is designed to support a plenary discussion in which participants and participant teams offer critiques that they have heard from others as well as critiques that they may be personally considering based on their experience. Facilitators should take this activity as another opportunity to contextualize findings by encouraging discussion on the particular policy environment (challenges and opportunities) that each research team will face, and working together to identify possible strategies to overcome them. 6 GUIDANCE NOTE 9 Open reflections 1 Module-at-a-glance Content Suggested time Instructional Activity The value of local perspectives 30 minutes Brief facilitator presentation Freedom to critique and grow Open discussion Open reflections 2 hours 30 minutes Total Time Half a day 9 SLIDES Learning Objectives To identify ways to better support sustainable local evidence for a wide range of different policy and practice contexts (in adversity) To promote an open debate that allows researchers to critical reflect on the RES- Research learning process among their peers To promote constructive critique that can be documented and used to improve the program Key Messages Research implicitly and explicitly evokes power relations between those 'being researched' and those with the power to document, measure and convey their life experiences. Local researchers have much to contribute to the International Education Development agenda, not least of which are cultural validity and local perspectives which respect and validate As part of this process it is important to critically engage with the theories and methodologies used and to critically reflect on the RES-Research process. 2 Core content The module begins with a brief presentation by the facilitators (to frame the subsequent discussion). Building on the 'current debates' that ended the previous module, facilitators should turn the critical reflection towards research more generally, highlighting the possible power dynamics that must be considered in any study. Facilitators can remind the group that the transformative agenda seeks to positively engage with this and reverse them towards empowering vulnerable communities. It does this through the ethical dimensions that guide the research as well as by empowering local researchers to conduct transformative research. We must therefore also apply the value of critique to the RES-Research process. Not only the theory and concept of resilience as was discussed the previous day, but also the methodology and philosophy of the training program. Part of this empowerment is being able to identify the opportunities that a transformative resilience approach has opened up and where the challenges remains. Especially important if we are to break down the dominant power dynamics is the need to adopt a longer term perspective and sustain the positive impacts. Facilitators can also use part of this time for more specific team based technical support and trouble-shooting. Bibliography and Support Material Kirkhart, K. 2010. Eyes on the Prize: Multicultural Validity and Evaluation Theory. American Journal of Evaluation 2010 31: 400. Sage Publications. - 2005. Through a cultural lens: Reflections on validity and theory in evaluation. In S. Hood, R. K. Hopson, and H. T. Frierson (eds.). The role of culture and cultural context: A mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: Information Age Publishing. 3 Liebenberg, L. 2006. The “us” and “them” in research: Can we get around it? Qualitative research in organizations and management, 1(2), 138-140. McCumber. 2000. Philosophy and freedom: Derrida, Rorty, Habermas, Foucault. Indiana: Indiana University Press. Mertens, D., F. Cram and B. Chilisa. Eds. 2013. Indigenous Pathways into Social Research. Voices of a New Generation. Left Coast Press: California. Mertens, D. M. 2009. Transformative Research and Evaluation. New York, NY: Guilford Press Mita, M. 1989. "Merata Mita On...". In The New Zealand Listener, 14 October, 1989. Session Activities The bulk of module 9 is dedicated to open discussion and is structured around four guiding questions: What are the contributions that only academics, researchers and evaluators who themselves live in contexts of chronic violence and conflict can provide? From your perspective, what elements would challenge from the resilience definitions that have evolved to-date? What are the challenges to sustain the relation between local evidence and national policy and practice in your country and do you have recommendations to overcome this? What do you take away from this workshop and the importance of resilience-based research? Facilitators should guide a discussion through these questions ensuring that all teams and team members have a chance to contribute and participate in the discussion. The reflections and ideas that are presented during the discussion should be noted down as they may be fed back into the RES-Research design and content over later iterations. The module concludes with the final evaluation forms which provide written feedback on the content and process. 4