Kan man räkna med PISA och TIMSS?: Relevansen hos internationella storskaliga mätningar i matematik i en nationell kontext

Can we count on PISA and TIMSS? : The relevance of international large-scale assessments in mathematics in a national context

The overall purpose of the thesis is to contribute to the knowledge of relevance of international large-scale assessments (ILSAs) in mathematics, when these are to be used and interpreted in a national context. If a nation is to engage in ILSAs, draw conclusions from them and act on them, one should be aware of what they are measuring and how well it fits in with the national mathematics education and the mathematics they are trying to teach their students.

The relevance is linked to the validity of the ILSA, which means that the assessment should measure the right things and do that in such a way that the results are as informative and useful as possible. In this thesis, the Swedish context is used as an example of a national context, against which the ILSAs PISA and TIMSS are contrasted.

One way to study validation is to use an argumentation-based validation method, Assessment Utilization Argument. In this model, an argument consists of making claims on the basis of data and warrants. The claim is an interpretation of assessment results and the assertion of a claim carries with it the duty to support the claim and, if challenged, to defend it. Warrants are created and functions as propositions to justify the inferences from the data that lead to the claim.

The ILSAs have been contrasted with the Swedish context through studies in three areas; content of the assessments, implementation of the assessments and results from the assessments. Based on these areas, the frameworks and tests from ILSAs were analyzed and compared with policy document and national tests from the Swedish context. Warrants were created based on these three areas.

The analysis of the warrants showed that the ILSAs had a high level of conformity with the Swedish context and the ILSAs coincide in such a way that the results from these are relevant for studying the development of performance in a Swedish context. The analysis also showed that certain content and abilities in the Swedish context were not covered by the ILSAs. The Swedish students do not have the opportunity to show all those skills that the Swedish mathematical context covers and there were also indications that they do not make full efforts to show their skills. In order to create a more complete picture of Swedish students’ mathematical skills, the assessment of students’ skills in mathematics should be complemented with other assessments.

The results are discussed in relation to the development of national policy documents, including the risk if a nation decides to adapt their policy document to the ILSAs. In the case of Sweden there are indications that policy document develops in the same direction and closer to the frameworks of ILSAs. It becomes important to thoroughly examine the relationship between the ILSAs and a national context so that one can be aware of how, if and it what way the large-scale assessment impact a national context.

This thesis contributes to a holistic approach based on a national context, with an overall method and covers the areas content, implementation and results of the ILSAs. It shows that opportunities and limitations can be found in all of these three areas. When the results of ILSAs are to be used in a national context, it is important to thoroughly examine what is meant to be assessed and what is really being assessed in these studies and to compare it with the purposes and content of the national context, so that suitable and valid conclusions could be drawn.