ECS Staff Help Address UN Sustainable Development Goals, Show OECD Novel Approach to Validate Assessments like PISA

Dr Melissa Glackin, Dr David Pepper and Kate Greer developed a new approach to validate assessments, which involves the use of an eye tracker with students and interviews with teacher. In July, Melissa and David travelled to the OECD’s headquarters in Paris, France to show the OECD how their novel approach validates assessments of 21^st Century learning outcomes

Melissa Glackin, David Pepper and Kate Greer are using their research to develop a novel approach to validating assessments. The researchers recently gave a seminar on this approach at the Organisation for Economic Co-operation and Development (OECD), with twenty-seven officials at various levels of seniority from a range of education programmes including the Programme for International Student Assessment (PISA), PISA for Schools, PISA for Development, the Programme for the International Assessment of Adult Competencies (PIAAC) and the Teaching And Learning International Survey (TALIS) attending. Several officials asked questions and contributed to the resulting discussion.

Melissa, David, and Kate’s approach generates validity evidence by combining the use of cognitive interviewing with an eye-tracker device to explore what students do in assessments and what teachers think about those assessments. They are developing their approach using the OECD’s PISA of global competence, which is a new assessment that draws on the fields of intercultural education and environmental education. Moreover, this assessment is important because it aims to address the UN Sustainable Development Goals as a response to pressing global issues of poverty and climate change.

In September, Melissa and David showed officials from the OECD in Paris, France that their approach is innovative, evidence-based, and applicable to their international assessments. Indeed, they showed the OECD how the tool can be used to validate not just the PISA assessment of global competence, but other PISA assessments and other OECD assessments such as PIAAC and TALIS. OECD officials were interested in both the cognitive interviewing with students using an eye-tracker and the interviews with teachers about the PISA items, with the research alerting the officials to the value of involving teachers in assessment validation.

Melissa, David, and Kate’s work has significant implications for assessment validation because there is growing recognition that evidence of what people think, feel and do during assessments is crucial in assessments of so-called 21^st Century learning outcomes. Their approach also has potentially wide ramifications since OECD assessments such as PISA are designed to assess the performance of 15-year-old students in mathematics, science, and reading in real-world situations. The OECD is an intergovernmental organisation made up of 36 mostly high-income countries, with many more countries participating in PISA. The OECD’s aim is to create a platform on which countries share best practices, exchange experiences, and set global standards and public policy. The OECD’s PISA tests are thus used to gauge how prepared the world’s youth are for situations in adulthood. As such, countries’ PISA test scores are highly influential in the way policy makers reform education policy and allocate resources.

Melissa and David also raised how the OECD might use validity evidence to inform countries’ decisions about whether to participate in their international assessments. Melissa, David, and Kate hope to develop the use of this approach at scale to generate validity evidence for the diverse populations participating in large-scale assessments such as PISA.