80067 Punjab Province, Pakistan SABER Province Report STUDENT ASSESSMENT 2012 Key Policy Areas for Student Assessment Status 1. Classroom Assessment The 2006 National Curriculum provides very general, non-comprehensive guidelines on classroom assessment. Although available online, this document is not offered through other channels, such as teacher training courses or libraries, which teachers and others can easily access. There are some system-level mechanisms in place, including pre- and in-service teacher training opportunities, which are meant to ensure that teachers develop appropriate skills and expertise in classroom assessment. However, on-the- ground classroom assessment practices tend to vary from school to school, provide little feedback to students and parents, and are generally considered weak. There are no required uses of classroom assessment information to support student learning, and there are no mechanisms in place to systematically monitor the quality of classroom assessment practices. 2. Examinations The Secondary School Certificate exam is administered to students in Grades 9 and 10. The Punjab Boards of Intermediate and Secondary Education Act of 1976 authorizes the exam. The Exam Rules and Procedures document (known as the “Calendar”) for each Punjab Board covers key aspects of the exam. Officially, the examination is meant to measure student attainment of the knowledge and skills outlined in the 2006 National Curriculum. However, in practice, it assesses textbook knowledge. Moreover, the textbooks are based on an earlier (2002) curriculum, not the more recent (2006) one. There is a permanent staff dedicated to the exam, but it is insufficient to meet the needs of the examination due to staff members’ lack of technical capacity. Funding for the examination comes from student fees. Funding covers all core examination activities, but not research and development. There is also no technical documentation on the examination. 3. National Large-Scale Assessment (NLSA) The Punjab Education Assessment System (PEAS) was administered for the first time in 2011 as an independent, provincial-level assessment exercise. Previously, PEAS was part of the National Education Assessment System (NEAS). Under NEAS, Punjab administered the assessment to a provincially representative sample of students and published a provincial report separate from the national report. Although the PEAS office is a permanent agency, it is inadequately staffed to effectively carry out PEAS activities. This also was the case when PEAS was part of NEAS. PEAS results have been poorly disseminated and inadequately used. 4. International Large-Scale Assessment (ILSA) Punjab has not participated in an ILSA, and it does not have plans to do so in the near future. THE WORLD BANK Introduction SABER-Student Assessment methodology The Punjab province in Pakistan has focused on increasing student learning outcomes by improving the The SABER-Student Assessment framework is built on quality of education in the province. An effective the available evidence base for what an effective student assessment system is an important component assessment system looks like. The framework provides to improving education quality and learning outcomes guidance on how countries can build more effective as it provides the necessary information to meet student assessment systems. The framework is stakeholders’ decision-making needs. In order to gain a structured around two main dimensions of assessment better understanding of the strengths and weaknesses systems: the types/purposes of assessment activities of its existing student assessment system within the and the quality of those activities. context of the province’s Public Sector Education system, the Punjab province decided to benchmark this Assessment types and purposes system using standardized tools developed under The World Bank’s Systems Approach for Better Education Assessment systems tend to be comprised of three Results (SABER) program. SABER is an evidence-based main types of assessment activities, each of which program to help countries systematically examine and serves a different purpose and addresses different strengthen the performance of different aspects of their information needs. These three main types are: education systems. classroom assessment, examinations, and large-scale, system level assessments. What is SABER-Student Assessment? Classroom assessment provides real-time information to support ongoing teaching and learning in individual SABER-Student Assessment is a component of the classrooms. Classroom assessments use a variety of SABER program that focuses specifically on formats, including observation, questioning, and paper- benchmarking student assessment policies and systems. and-pencil tests, to evaluate student learning, generally The goal of SABER-Student Assessment is to promote on a daily basis. stronger assessment systems that contribute to improved education quality and learning for all. Examinations provide a basis for selecting or certifying students as they move from one level of the education system to the next (or into the workforce). All eligible National governments and international agencies are students are tested on an annual basis (or more often if increasingly recognizing the key role that assessment of the system allows for repeat testing). Examinations student learning plays in an effective education system. cover the main subject areas in the curriculum and The importance of assessment is linked to its role in: usually involve essays and multiple-choice questions. (i) providing information on levels of student learning and achievement in the system; (ii) monitoring trends in education quality over Large-scale, system-level assessments provide feedback time; on the overall performance of the education system at (iii) supporting educators and students with real- particular grades or age levels. These assessments time information to improve teaching and typically cover a few subjects on a regular basis (such as learning; and every 3 to 5 years), are often sample based, and use (iv) holding stakeholders accountable for results. multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. 2 Quality drivers of an assessment system Table 1: Framework for building an effective The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or assessment system, with indicator areas policy framework for assessment activities; institutional The indicators are identified based on a combination of and organizational structures for designing, carrying criteria, including: out, or using results from the assessment; the availability of sufficient and stable sources of funding; x professional standards for assessment; and the presence of trained assessment staff. x empirical research on the characteristics of effective assessment systems, including analysis of the System alignment refers to the extent to which the characteristics that differentiate between the assessment is aligned with the rest of the education assessment systems of low- versus high-performing system. This includes the degree of congruence nations; and between assessment activities and system learning x theory — that is, general consensus among goals, standards, curriculum, and pre- and in-service experts that it contributes to effective teacher training. assessment. Assessment quality refers to the psychometric quality of Levels of development the instruments, processes, and procedures for the assessment activity. It covers such issues as design and The World Bank has developed a set of implementation of assessment activities, analysis and standardized questionnaires and rubrics for collecting interpretation of student responses to those activities, and evaluating data on the three assessment types and the appropriateness of how assessment results are and related quality drivers. reported and used. The questionnaires are used to collect data on the Crossing the quality drivers with the different characteristics of the assessment system in a particular assessment types/purposes provides the framework country or system. The information from the and broad indicator areas shown in Table 1. This questionnaires is then applied to the rubrics in order to framework is a starting point for identifying indicators judge the development level of the country’s that can be used to review assessment systems and assessment system in different areas. plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the 3 development level of its assessment activities compared Education in the Punjab province to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, and Advanced. Pakistan is a lower-middle-income country in South These levels are artificially constructed categories Asia. Current GDP per capita (US, 2012) is $1,257 with chosen to represent key stages on the underlying annual growth of 4 percent. Economic growth and continuum for each indicator. Each level is progress in poverty reduction have slowed significantly accompanied by a description of what performance on over the past five years due to a weak macroeconomic the indicator looks like at that level. environment, political and security conditions, and catastrophic natural shocks. x Latent is the lowest level of performance; it Punjab is Pakistan’s largest province (out of four), represents absence of, or deviation from, the accounting for approximately three-fifths of the desired attribute. country’s population and income. In 2010/2011, x Emerging is the next level; it represents partial primary net enrollment rate was 70 percent, and presence of the attribute. secondary net enrollment was 37 percent. In particular, x Established represents the acceptable minimum there are low enrollment rates among girls in rural standard. areas and children from poor households. In addition, x Advanced represents the ideal or current best the overall level of student achievement tends to be practice. low; various assessments and surveys find that students perform significantly below curricular standards for key A summary of the development levels for each subjects and concepts at their grade level. assessment type is presented in Appendix 3. In order to increase both school participation and student achievement, Punjab has decided to focus In reality, assessment systems are likely to be at specifically on improving teacher quality and different levels of development in different areas. For performance. This decision is based on the example, a system may be Established in the area of government’s recognition that: 1) the teacher is the examinations, but Emerging in the area of large- main instrument for student learning, and that many scale, system-level assessment, and vice versa. While initiatives can affect the child only through the teacher; intuition suggests that it is probably better to be and 2) improved student learning helps retain students further along in as many areas as possible, the in school and attract new children to school. As a result, evidence is unclear as to whether it is necessary to Punjab plans to promote teacher quality and be functioning at Advanced levels in all areas. performance both directly and indirectly through Therefore, one might view the Established level as a initiatives such as: (i) strengthening the system of field- desirable minimum outcome to achieve in all areas, but based advisory support to teachers; (ii) introducing test- only aspire beyond that in those areas that most based recruitment of teachers; (iii) tying teacher contribute to the national vision or priorities for compensation more closely to school performance; (iv) education. In line with these considerations, the ratings improving the regular collection of credible information generated by the rubrics are not meant to be additive on school, teacher, and student performance, and across assessment types (that is, they are not meant to feeding this information back to internal and external be added to create an overall rating for an assessment stakeholders; and others. system; they are only meant to produce an overall rating for each assessment type). The methodology for assigning development levels is summarized in Appendix 4. 4 Detailed information was collected on the Punjab assessment activities, including textbooks which contain province’s student assessment system using the SABER- questions at the end of the chapters. The Taleemi Student Assessment questionnaires and rubrics in 2012. Calendar is a document that contains student learning This SABER-Student Assessment report reflects the outcomes for each grade and subject. In addition, the status of the student assessment system in the Punjab Directorate of Staff Development (DSD) has developed province as of 2012. It is important to remember that Teacher’s Guides which contain Student Learning these tools primarily focus on benchmarking a system’s Outcome (SLO)- based lesson plans for each subject and policies and arrangements for assessment activities at grade in primary school. These guides also include a the system or macro level. Additional data would need section on the kinds of activities and homework that to be collected to determine actual, on-the-ground teachers can use to assess students on the particular practices in the Punjab province, particularly by SLO. teachers and students in schools. The following sections discuss the findings by each assessment type, There are some system-level mechanisms, such as pre- accompanied by suggested policy options. The and in-service teacher training opportunities, in place suggested policy options were determined in that are intended to ensure that teachers develop the collaboration with key local stakeholders based on the appropriate skills and expertise in classroom Punjab province’s immediate interests and needs. assessment. School inspection and teacher supervision Detailed, completed rubrics for each assessment type in also include a component focused on classroom the Punjab province are provided in Appendix 5. assessment. However, on-the-ground classroom assessment Classroom Assessment practices tend to be mainly about recalling information, provide little feedback to students and parents, and are generally considered weak. Teachers primarily develop Level of development: EMERGING the questions directly from the textbook and use the exercises at the end of the chapters; they do not use In Punjab, the 2006 National Curriculum document explicit criteria for scoring or grading students’ work. In provides general, non-comprehensive guidelines on general, classroom assessment activities tend to vary classroom assessment. The document provides from school to school and are usually based on the information on assessment purposes, types and textbooks and not on the curricula. However, textbooks questions, and includes some sample test content and tend to not be aligned with the curriculum because they scoring criteria. Although available online, this are based on a previous version of the curriculum document is not offered through other channels that (developed in 2002) and not on the most recent are more readily accessible by teachers and other curricula that went into effect in 2006. stakeholders, such as through teacher training courses or libraries. The Punjab Curriculum Authority Act 2012 There are no required uses of classroom assessment established the Curriculum Authority. The Curriculum information to support student learning, and there are Authority is responsible for overseeing the supervision no mechanisms in place to systematically monitor the of curricula, preparing and approving textbooks, and quality of classroom assessment practices. maintaining standards of education in the Punjab province. Suggested policy options: The 2006 National Curriculum document has not been provided to all practicing classroom teachers, and 1. Make the curriculum document more easily because not all teachers have access to the internet, accessible to stakeholders through, for example, many teachers are not aware of the guidelines provided libraries, teacher training courses, and schools. in the document. In addition to the 2006 National Curriculum document, there are some system-wide resources available for teachers to engage in classroom 5 2. Develop detailed guidelines for teachers on how to carry out classroom assessment activities to support student learning. 3. Ensure more effective uses of classroom assessment information by: (i) communicating to teachers appropriate uses of the data; and (ii) making it compulsory for classroom assessment information to be disseminated to key stakeholders. 6 Examinations computerizing the scoring of the multiple-choice questions, and swapping examination papers between boards. The Aga Khan University Examination Board and Level of development: EMERGING USAID have also supported efforts to improve the quality of the examination items and curriculum The Secondary School Certificate exam is administered coverage of the Boards across Pakistan. to students in Grades 9 and 10. The Secondary School Certificate exam is a major examination for graduation Funding for the examination is provided by student from secondary school, with approximately 1.7 million fees, and covers all core examination activities including students taking it every year, and is therefore the focus examination design, administration, data processing, of this report. The Punjab Boards of Intermediate and and data reporting. Secondary Education Act of 1976 authorizes the exam and outlines the governance, distribution of power, and There is a permanent staff responsible for running the responsibilities among key stakeholders. Additionally, examination, but it is insufficient to meet the needs of the Examination Rules and Procedures document the examination due to staff members’ lack of technical (known as the “Calendar”) for each of the Punjab capacity. Issues identified with the performance of the Boards covers key aspects of the exam including the human resources responsible for the examination rules for preparation, the format of the examination, include errors in scoring, weakness in test design, and the procedures to investigate inappropriate omission of curricular topics, and errors in the behavior. examination questions and data processing. The Secondary School Certificate examination is In general, comprehensive material to help prepare for intended to measure student attainment of the the examination is accessible to most students. Such knowledge and skills outlined in the 2006 National items include the textbook (which is considered the Curriculum. In practice, the examination tests textbook main resource, as the examination is textbook-based) knowledge. At the same time, textbooks are not well and previous examination papers (as there is little aligned with the curricula due to two main issues: (1) repetition of examination questions from year to year). some textbooks in circulation are based on a previous version of the national curricula developed in 2002 There is no technical documentation on the (new textbooks based on the new curricula developed examination and there are no mechanisms in place to in 2006 are in various stages of development), and (2) ensure the quality of the examination. Inappropriate textbooks are not perceived to adequately cover all behavior surrounding the examination process is aspects of the curricula, such as higher order thinking moderate and includes leakage of the content of the skills. Thus, there is no clear link between the paper prior to the examination, copying, and using examination and the student learning outcomes or the unauthorized materials, such as prepared answers and skills identified in the curriculum, resulting in notes. As a result, the credibility of the examination and inadequate coverage and the lack of higher order skills its results tends to be called into question by many tested. stakeholders. Efforts to improve the examination have been led by the government and by donors. For example, several efforts to improve aspects of the examinations have Suggested policy options: been made by the Boards along with the Education Department, including the introduction of multiple- 1. Build the capacity of individuals responsible for choice and short answer questions. The Boards have examination activities, especially in the areas of test also sought to address the issues of standardization design, curriculum alignment, and report writing. between the exam boards and the variation in the This could be done by: (i) making training scoring of papers by collectively setting papers, opportunities available within the Boards, and (ii) 7 allocating funding for attending international courses or workshops. Introduce a mandate for capacity development of staff responsible for the examination. Identify which individuals or organizations will be responsible for the trainings, and provide them with regular funding for and access to the necessary training resources and key experts (at the provincial, national, and international levels). 2. Develop a technical report on the examination and disseminate it to key stakeholders. 3. Identify and agree on the official purposes and possible uses of examination results, and introduce mechanisms, such as an oversight committee, to monitor actual uses. 8 National Large-Scale Assessment (NLSA) the University of Melbourne to prepare PEAS staff. However, none of these courses has been sustained, and Punjab province currently offers no opportunities Level of development: EMERGING to prepare individuals for work on the PEAS. The Punjab province carries out several national large- The PEAS assessment measures competencies identified scale assessments to students at the primary and in the curriculum. However, since assessments are secondary levels. The Punjab Examination Commission based on the 2006 curricula and the textbooks available (PEC) was established in 2006 and awarded a statutory are based on the previous curricula (developed in 2002), status in 2010 through an act of parliament. PEC is some skills assessed on PEAS are not available in the responsible for designing, implementing, and reporting textbooks and could cause some confusion amongst on assessments for all Grade 5 (primary school) and teachers who only use the textbook to teach students. Grade 8 (middle school) students in Punjab. PEC conducts Grade 5 and Grade 8 annual assessments Various mechanisms are in place to ensure the quality (known as “examinations”) in all subjects for all the of PEAS including: (i) all administrators are trained public and private schools in Punjab. PEC provides according to a protocol, (ii) there is a standardized individual student reports which grade the student’s manual for the administrators, (iii) a pilot is conducted performance on each of the expected student learning before the main data collection takes place, (iv) all outcomes (SLOs) by subject. The main objective of booklets are numbered, and (v) external and internal these examinations is to develop a key database about reviewers and observers are employed. Coding and children’s performance and as a result measure the scoring is a centralized activity, conducted in one place performance of the teachers and schools. The Punjab and overseen by persons involved in data analysis. A province also carries out the Punjab Education two-day training is held for scorers, and scorers are also Assessment System (PEAS). PEAS was administered for provided with detailed codebooks and rubrics for open- the first time in 2011 as an independent, provincial- ended questions. 10 percent of the scored assessment level assessment exercise administered to students in papers are also scored by another person. Protocols are public sector schools. PEAS is viewed as the most also in place for ensuring the quality of data entry and important assessment for use by policymakers. data cleaning. Previously, PEAS was part of the National Education Assessment System (NEAS). Under NEAS, starting in At the time of data collection for this report, the 2011 2005, Punjab administered the assessment to a PEAS results had not been finalized and thus provincially representative sample of students and dissemination had not yet taken place. There are plans, published a provincial report separate from the national however, to prepare and disseminate three types of report. reports (technical, thematic for different audiences, and a simplified report for a general audience) as well as to Regular funding for PEAS is allocated by the government carry out district-based workshops to communicate the and donors. Funding covers some core PEAS activities, results. but not research and development. Although the PEAS office is a permanent agency, it is Suggested policy options: inadequately staffed to effectively carry out all PEAS activities. In general, there are issues in test design, 1. Develop the capacity of PEAS staff by (i) introducing frequent errors in data processing, and delays due to workshops and trainings on education the dependence on external human resources, including measurement and evaluation, including, in for paper development, data entry, and analysis. particular, data analysis and report writing; (ii) providing funding to PEAS staff to attend In the past, the University of Education (Lahore) offered international programs on educational short courses and a specialized course was offered by measurement and evaluation; and (iii) identifying 9 external human resources for conducting translation verifications between Urdu and Sindhi. 2. Identify the intended uses of PEAS results; develop and implement a plan to disseminate the results to stakeholders; introduce mechanisms, such as an oversight committee, to monitor actual uses of PEAS results. . 10 International Large-Scale Assessment (ILSA) Level of development: LATENT The Punjab province has not participated in an ILSA, and it does not have plans to do so in the near future. Suggested policy options: 1. Create an opportunity for high-level discussion among key stakeholders on key education policy questions or problems for which ILSA data could be useful. 2. Determine the need for, and possible next steps in relation to, participation in an ILSA exercise. 11 12 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques 13 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses 14 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. 15 Appendix 4: Methodology for Assigning 6. Scores for certain key dimensions under ‘Enabling Development Levels Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the 1. The country team or consultant collects information particular assessment type cannot be greater than the about the assessment system in the country or system. score for these key dimensions. These key variables include formal policy, regular funding, having a 2. Based on the collected information, a level of permanent assessment unit, and the quality of development and score is assigned to each dimension in assessment practices. the rubrics: x Latent = 1 score point x Emerging = 2 score points x Established = 3 score points x Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country or system receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s or system’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country or system has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 16 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 Appendix 5: SABER-Student Assessment Rubrics Punjab Province, Pakistan This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in the Punjab province, Pakistan. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development-level rating justifications” section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. PUNJAB PROVINCE, PAKISTAN Classroom Assessment SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 17 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. 2 dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 3 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. 4 clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom 5 assessment. assessment. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 18 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 6 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 7 practices. practices. assessment practices. quality of classroom assessment practices. * ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 8 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support student student learning. student learning, excluding its use as an learning, including its use as an input for input for external examination results. external examination results. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 19 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 Classroom Assessment: Development-level rating justifications 1. The only system-level document that provides some recommendations on assessment practices is the National Curricula 2006. It provides general information on assessment purposes, types, and questions and some sample test content and scoring criteria, but these are not comprehensive guidelines. Additionally, most teachers have not been provided with the curriculum document, and are therefore not aware of these guidelines. The Punjab Curriculum Authority Act 2012 established the Curriculum Authority. The Curriculum Authority is responsible for overseeing the supervision of curriculum, preparing and approving textbooks, and maintaining standards of education in the Punjab province. 2. The National Curricula 2006 is available online but not through other channels, such as libraries and teacher training courses and colleges, that teachers and other stakeholders can easily access. 3. The Taleemi Calendar is a document that contains student learning outcomes for each grade and subject. There are also textbooks and workbooks that provide support for classroom assessment. They contain questions at the end of the chapter, as well as guidelines for teachers on conducting classroom activities for assessment purposes, such as suggesting to teachers to hold speech competitions. Directorate of Staff Development (DSD) has developed Teacher’s Guides that contain Student Learning Outcome (SLO)-based lesson plans for each subject and grade in primary school. These include a section on the kinds of activities/homework teachers can use to assess students on that SLO. 4. There is an official national curriculum which went into effect in 2006. After the 18th constitutional amendment, the authority to develop curriculum has been transferred to the provinces and some provinces are developing their own curricula, but it is not clear whether the Punjab province is doing so. 5. System-level mechanisms include pre- and in-service teacher training, as well as school inspection or teacher supervision, which includes a component that is focused on classroom assessment. 6. It is not common for classroom assessment activities to rely mainly on multiple-choice, selection-type questions as short- and long-answer questions. Classroom assessment activities are, however, mainly about recalling information because teachers develop questions directly from the textbook. Teachers also do not use a priori criteria for grading students' work because there are no system-level guidelines or scoring criteria. Additionally, classroom assessment activities vary from school to school and generally provide little useful feedback to students and parents. In general, classroom assessment activities are based on the textbooks not the curricula. (Textbooks are generally not aligned with the curricula because the textbooks in circulation are based on a previous version of the national curricula that was developed in 2002. There are no new textbooks that are based on the new curricula that went into effect in 2006.) 7. There do not appear to be any system-level mechanisms that monitor the quality of assessment activities. The focus is primarily on test results. It appears that formal teacher performance evaluation is only a regular practice at the secondary level and in that case, usually with relation to Board exam results. Generally, district staff visits tend to focus on more administrative aspects, enrollment, or teacher presence. At the key grades and the secondary level where a board exam will be taken, it appears there is an interest in the student results. Finally the District Teacher Educators (DTE) visits also focus on results of their monthly tests, not the teachers’ own practices. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 20 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 8. Teachers are required to report on individual students’ performance to school district and Ministry of Education officials. Specifically, student results are reported to the district education department (usually for annual tests). In some cases for those classes preparing for board exams, particularly at the secondary level (i.e., grades 9 and 10), it appears that term tests are also reported. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 21 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 PUNJAB PROVINCE, PAKISTAN Examinations SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 22 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this place. operating on an irregular basis. has been operating regularly. dimension There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 1 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 2 dimension. the public. public. dimension. This option does not apply to this This option does not apply to this * The policy document addresses some The policy document addresses all key 3 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 4 the examination or are indifferent to it. examination. examination. examination. There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 5 groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 6 charge of the examination. charge of the examination. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 23 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 7 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 8 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 9 dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 10 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 11 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 12 country/system. another country/system. system in another country/system. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 13 examination. examination. examination. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 24 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 14 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country/system does not offer This option does not apply to this The country/system offers some The country/system offers a wide range opportunities that prepare for work on dimension. opportunities that prepare for work on of opportunities that prepare for work the examination. the examination. on the examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 25 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 15 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. 16 * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 17 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for teachers. to teachers. teachers. Teachers are excluded from * all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 18 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 26 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 19 examination process is high. examination process is moderate. examination process is low. examination process is marginal. The examination results lack credibility * The examination results are credible for The examination results are credible for This option does not apply to this 20 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. 21 barriers. equivalent barriers. equivalent barriers. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 27 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 22 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 23 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the examination. examination. examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 28 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 Examinations: Development-level rating justifications 1. The formal policy document is the Punjab Boards of Intermediate and Secondary Education Act of 1976. Additionally, there are "Examination Rules and Procedures" documents that are known as the "Calendar" for each of the Punjab Boards, which covers key areas. 2. All acts, ordinances, and the Calendar are technically public documents, but they are not necessarily easily accessible. The full Calendar is available in hard copy at the Board on request, and it must be purchased. Parts of the Calendar are easily accessible on the website, but internet is not easily accessible to all stakeholders. 3. The Punjab Boards of Intermediate and Secondary Education Act outline the governance, distribution of power, and responsibilities among key stakeholders. The Calendar outlines the procedures to investigate and address security breaches, cheating, or other forms of inappropriate behavior, the procedures for special/disadvantaged students, and specifies who can sit for the examination. It also identifies the rules about the preparation and format of the examination. 4. There is little publicly available evidence on stakeholder support or opposition, and there is very little public debate on this issue. 5. Efforts to improve the examination have been led by the government and, most recently, by donors. For example, there have been several efforts to improve different aspects of the examinations by the Boards along with the Education Department. The Inter Board Committee of Chairmen (IBCC) has changed the paper pattern in recent years to include multiple-choice questions and short answer questions. Under the guidance of the Chief Minister last year, they created a talent pool of best paper setters and provided capacity development to them. The Boards have sought to address issues of standardization between the exam boards by setting papers collectively. They have also sought to deal with variations in the scoring of the papers through processes such as computerizing the scoring of the multiple-choice question sections, swapping papers between Boards and syndicated marking of open- ended questions (in which one person marks one question and the other the next to minimize effects of the marker on the overall score). The Boards have also sought to computerize the system from registration to scoring to result production; although there are issues with this system currently, this may in the future reduce errors in the result and instances of unfair means. The Punjab Education Department and the Boards have instituted several measures to curb cheating and other inappropriate behaviors. Finally, there appear to be efforts by the Aga Khan University Examination Board (AKUEB) and a USAID- supported EDLINKS project to improve the quality of the exam paper items and curriculum coverage of the Boards across Pakistan. However this is a fairly new initiative and apart from a workshop with paper setters—individuals who set the examination question papers—from the Boards, it is not clear what other activities have been implemented. 6. The Boards at the higher levels appear to be aware of the various issues in the examination system and as a result have taken steps to address the issues themselves. However, since efforts to improve the examination from outside the government, such as those by the AKU-EB, have not occurred, it is difficult to say whether they would be receptive to such changes. 7. The funding for the examination are from student fees collected in previous years and other fees for issuing or validating certificates, conducting a re- evaluation of papers, etc. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 29 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 8. Data analysis and reporting are limited to development and reporting of exam results for individual candidates. 9. Only the Lahore Board has a research section; however, this section does not necessarily conduct research. Its functions are usually to develop descriptive analysis of exam results (i.e., by gender or location) and handle non-research activities such as scholarships, debates, and essay competitions. There is no staff development. 10. The oldest Board, the Lahore Board of Secondary & Intermediate Education, was established in 1954, while the most recent Board, the Dera Ghazi Khan Board of Secondary & Intermediate Education, was established in 1989. 11. Accountable to the Department of Higher Education, Punjab. 12. The IBCC/Foreign Ministry attests to the use of the SSC exam certificates abroad. Although the SSC exam would not be used for admission into university per se, it would presumably form part of the secondary school education record for admissions. 13. The consultant only visited one Lahore BISE physically, and the selections for this question are based on this Board, either through observation or respondent comments. Given that the Boards are located in very different areas, it is difficult to determine whether these responses in the questionnaire, particularly for technological facilities, are representative of the other Boards as well. 14. While there is permanent or full-time staff, it is insufficient to meet the needs of the examination. For example, there is lack of staff who can perform research and data analysis, and lack of technical staff with skills in examination design to guide the paper development process. Issues that have been identified with the performance of the human resources responsible for the examination include errors in scoring that have led to delays in results being reported, weakness in test design, omission of curricular topics, and frequent errors in the examination questions and data processing. 15. Officially, the examination seeks to measure the national curriculum. However, there is no clear link to the student learning outcomes or the skills identified in the curriculum, often resulting in inadequate coverage and the lack of higher order skills tested as noted by research studies. In practice, it is essentially the textbook content that is measured by the exam (in Punjab, there is one set of provincially approved textbooks in schools). Textbooks cannot be said to be well aligned with the curricula: (1) Some textbooks in circulation are based on a previous version of the national curricula developed in 2002 (new textbooks based on the new curricula, i.e. 2006, are in various stages of development), and (2) they are not perceived to adequately cover all aspects of the curricula, such as higher order thinking skills. 16. Different stakeholders have different concerns. Many stakeholders (such as parents, students, general public) are more concerned with issues related to cheating and speed, and accuracy of scoring. However, those who recognize that there are inadequacies in the curriculum coverage and skills tested do not accept what is measured. Such respondents for this study have included the Chairman of a Board, staff at NGOs and professional development institutions, and some educators. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 30 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 17. Materials required for preparing for the exam include the textbook (the main resource, as the exam is essentially textbook-based) and past papers (as there is apparently not a lot of repetition in exam questions from year to year). Textbooks in government schools are now provided free of cost. Respondents feel that both these resources should be readily available in the market even in rural areas, and cost is not too high even for those who have to pay. Thus only a very small percentage may have difficulty accessing these items. The material on the examination includes examples of the types of questions that are on the examination, past examination papers, and examples of examination papers developed by the Boards. 18. Retired teachers select or create examination questions, while active teachers administer and score the examination, supervise the examination procedures, and act as a judge in, for example, orals and practical. 19. Various inappropriate behaviors diminish the credibility of the examinations. Such behaviors include leakage of the examination paper content or part of a paper prior to the examination, impersonation, copying from other candidates, using unauthorized materials such as prepared answers and notes, collusion among candidates via mobile phones, passing of paper, or equivalent, and provision of external assistance via the supervisor, mobile phone, etc. To reduce inappropriate behavior, an Examination Monitoring Cell was established within the Education Department, which, in addition to the monitoring done by the Boards themselves and the monitoring by retired government officials, has strengthened the monitoring of exams. Press conferences are also used to raise awareness about the consequences of inappropriate behavior during the examination. 20. The credibility of the exams and its results are called into question with regards to the need for quality improvement at all levels (exam design, administration, scoring, and determining results) and the perception of the general public that cheating is still widespread. The views of stakeholders tend to vary in terms of their area of concern. Media reports tend to focus on the conduct (i.e., occurrence of cheating) and accuracy or delays in results, as do parents and students. This year, there has been a substantial amount of criticism over the computerization of the process. In addition to these issues, staff within the Boards, government departments, and teachers appears to be concerned with issues in marking. Finally, from reports and interviews, stakeholders such as those in teacher education institutions, NGOs, and selected government officials and educators point out the poor quality of the exam paper and its questions. Universities do not appear to trust the quality of the exams and have their own entrance exams. 21. Respondents have noted that if a student manages to make it to secondary school (i.e., does not drop out at earlier stages or face access-related barriers), then fees or location should not serve as a barrier to take the examinations as it is the key certification for proving they’ve completed secondary school. Exam centers are usually close to the students' schools and there no registration barriers based on ethnicity or gender. With regards to fees, as of 2010, the fees have been waived by the Punjab government, stating that fees were a cause of dropout at secondary level. For the last two years, the fees have not served as a barrier. When looking at the numbers of students who appeared for the exams in 2010, news reports estimate about 1.1 million for all Boards and this is similar to the enrollment figures for Grade 10 from 2008 of about 1 million (PSLM). One could assume that even with a population increase, the number is quite similar. 22. Because it is unclear what the official purposes and uses of examination results are, it is difficult to identify "improper" uses. Generally, however, exam results are not used beyond determining pass/fail and admission into the next level of education, which are in line with the intended purposes. 23. Students may retake the examination, may opt for less selective schools/universities/tracks, or they may repeat the grade. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 31 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 PUNJAB PROVINCE, PAKISTAN National (or System-Level) Large-Scale Assessment (NLSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 32 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. * There is no policy document pertaining There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the 3 dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 4 the NLSA or are indifferent to it. NLSA. NLSA. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 33 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 5 NLSA. the NLSA. NLSA. dimension. This option does not apply to this Funding covers some core NLSA * Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 6 and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 7 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this * 8 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 34 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to 9 NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. * The country/system does not offer This option does not apply to this The country/system offers some The country/system offers a wide range opportunities that prepare individuals dimension. opportunities to prepare individuals for of opportunities to prepare individuals 10 for work on NLSA. work on the NLSA. for work on the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 35 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 11 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 12 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 36 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 13 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 14 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality * documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general 15 not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 16 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. * NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and 17 assessment. technical characteristics of the technical characteristics of the assessment. assessment. There are no mechanisms in place to * This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of 18 the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 37 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 National (or System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. The Punjab Education Assessment System (PEAS) was administered in 2011 independently. Prior to that, PEAS was part of the National Assessment (National Education Assessment System). Under NEAS, the province implemented the assessment to a provincially representative sample of students and published a provincial report separate from the national report. The Punjab Examination Commission (PEC) was established in 2006 and awarded a statutory status in 2010 through an act of parliament. PEC is responsible for designing, implementing and reporting on assessments for all Grade 5 (primary school) and Grade 8 (middle school) students in Punjab. PEC conducts Grade 5 and Grade 8 annual examinations in all subjects for all the public and private schools in Punjab. PEC provides individual student reports that grade the student’s performance on each of the expected student learning outcomes (SLOs) by subject. The main objective of these examinations is to develop a key database about children’s performance and, as a result, measure the performance of the teachers and schools. 2. Notification of Punjab Education Assessment System (PEAS) covers the period 2005-2008, issued in 2005 by Government of Punjab, Education Department (while PEAS was still a part of NEAS). This notification has not been updated to include the current period. This is an issue pointed out by staff members themselves, who recognize that it needs to be raised with the Secretary, but no timeline was provided for when this will happen. 3. The plan for the next two years has been developed. The written plan, unofficially approved by the Secretary of Education, is currently under development, and they expect to receive official approval in the next few months. 4. There does not appear to be any publicly available evidence for stakeholder perceptions of the large-scale assessments. 5. There is regular funding allocated by the government, as well as technical assistance from the World Bank and DFID. 6. PEAS receives single-line funding that is not allocated to any one activity, then it seeks approval for planned activities. Planning does not appear as a separate activity in the work plan, however planning occurs twice a year as per government rules, in which funding can be reallocated to existing or new activities. 7. The NLSA office is a permanent institution created for running the NLSA. It was first established in 2002 as the Provincial Education Assessment Centre (PEACE), Punjab, and in 2005 it was renamed to the Punjab Education Assessment System (PEAS). 8. It is accountable to the Punjab Department of Education. 9. There is permanent or full-time staff, but it is insufficient to meet the needs of the assessment. Issues that have been identified with the performance of the human resources that are responsible for the NLSA include weaknesses in test design, frequent errors in data processing, and delays due to dependence on external human resources for, for example, paper development, data entry, and analysis. With regard to issues with item development, it has been difficult to find relevant/good quality specialists to develop items, especially the kind of items that go beyond the textbook and that assess higher order thinking SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 38 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 skills. There is also no in-house capacity to develop the sample. Additionally, in-house staff lack the required skills to benefit from the technical assistance that was provided. Effective data management and storage is also an issue. 10. In the past, the University of Education (Lahore) offered short courses, and a specialized course was offered by the University of Melbourne in collaboration with the Punjab University Institute of Educational Research (IER) for NEAS staff. However none of these have been sustained. 11. PEAS staff note that there has been some concern, particularly from teachers, over what the assessment covers because they are used to textbook-based exams and an assessment that links itself to the curricula and asks questions in a different manner is unfamiliar to them (textbooks cannot be said to be well aligned with the current curricula as they are not only based on an older version of the curricula, developed in 2002, but they also do not adequately cover all aspects of the curricula such as higher order thinking skills). However, this sentiment has decreased over the years with teachers’ increased familiarity of the assessment and their realizing that it is not a high stakes test. 12. For the PEAS 2011 assessment, there have been several reviews at different stages, during the development of the assessment framework, item development and after the pilot. These are internal reviews that include external experts from the universities, teacher professional development colleges, and schools. 13. The NLSA is offered in the language of instruction for almost all student groups and in various geographical regions of Punjab. 14. A variety of mechanisms are in place to ensure the quality of the NLSA instrument. Specifically, all proctors or administrators are trained according to a protocol, and there is a standardized manual for large-scale assessment administrators. Additionally, a pilot is conducted before the main data collection takes place and all booklets are numbered. There is also internal and external review by observers. With regard to scoring, there is a two-day training of scorers, provision of detailed codebooks and rubrics for open-ended items, and 10-15 percent check of scoring by another person. In terms of processing of data, there are protocols for ensuring the quality of data entry, 10 percent data check and cleaning. There does not appear to be double processing of data. 15. The technical report for PEAS has been developed and is currently being printed. For NEAS there were regular technical reports but with limited circulation. 16. The PEAS 2011 Assessment results and reports have not been finalized and thus there has been no dissemination at this stage. Plans include three types of reports (technical, thematic for different audiences, and simplified for a general audience), as well as district-based workshops. Under NEAS there were yearly reports and newsletters, stakeholder conferences and coverage in the media. However, dissemination was noted as one of the weakest areas in the entire program by a former staff member. Due to the lack of capacity and dedicated staff throughout most of the project, developing reports often took a great deal of time and caused delays in dissemination. 17. Given that the current PEAS assessment results have yet to be determined it is not possible to say how the information from this particular assessment will be used. In previous experiences, there has been some, although sporadic, use of results. For example the National Education Policy 2009 mentions NEAS results, and the Directorate of Staff Development (DSD) in Punjab claims to have used results in training. However, for the most part very little use is found, and implementation of the assessment recommendations has been almost non-existent. This is due to lack of a mechanism for ensuring the use of results SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 39 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 and to some extent lack of motivation to ensure implementation on the part of policymakers. It also has partly to do with the lack of a research-based decision making practices and understanding of the uses of assessment results. 18. The need is recognized by PEAS and this may happen in the future provided there is funding. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 40 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 PUNJAB PROVINCE, PAKISTAN International Large-Scale Assessment (ILSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 41 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. the next 5 years. the next 5 years. * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors. discretion. decree or norm. This option does not apply to this Funding covers some core activities of Funding covers all core activities of the This option does not apply to this dimension. the ILSA. ILSA. dimension. Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and development activities. dimension. dimension. development activities. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 42 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. activities. activities. activities. This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, effectively. with minimal issues. with no issues. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 43 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this The country/system offers some The country/system offers a wide range opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition team members only. to the country's/system's ILSA team members. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 44 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical This option does not apply to this published. standards to have its data presented standards required to have its data dimension. beneath the main display of the presented in the main displays of the international report or in an annex. international report. The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new new knowledge on ILSA. dimension. dimension. knowledge on ILSA. ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely the country/system. in the country/system. the country/system. disseminated in the country/system. Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are not made available. sometimes made available. systematically made available. There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA results. ILSA results. ILSA results. results. If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision used to inform decision making in the the country/system. country/system. making in the country/system. country/system. It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' on students' achievement levels. achievement levels. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 45 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT |2012 International Large Scale Assessment (ILSA): Development-level rating justifications 1. The Punjab province has not participated in an ILSA, and it does not have plans to do so in the near future. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 46 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Acknowledgements References This report was prepared by The World Bank SABER- Clarke, M. 2012. “What Matters Most for Student Student Assessment team in collaboration with Assessment Systems: A Framework Paper.” Umbreen Arif, World Bank Senior Education Specialist READ/SABER Working Paper Series. Washington, D.C.: and Task Team Leader for education projects in World Bank. Pakistan. The Government of the Punjab Province. 2012. The Punjab Curriculum Authority Act 2012. Punjab province: The Government of the Punjab Province, Pakistan. World Bank. 2012. Pakistan Country Indicator Data. Washington, DC: World Bank. Data retrieved from http://data.worldbank.org/ on March 10, 2014. World Bank. 2012. Pakistan - Second Punjab Education Sector Project. Report No. 66291-PK. Washington D.C.: The World Bank. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 46 PUNJAB PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country’s education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 46