How much can we really deduce about academies and free schools from the Organisation for Economic Development’s (OECD) international education rankings? When the OECD’s deputy director for education Andreas Schleicher gave evidence recently to parliament’s select committee inquiry into academies and free schools, he made a number of points, including that its Programme for International Student Assessment (PISA) test was fair and gave “robust” results. It could, in his opinion, therefore be used to draw valid conclusions for policy-making.
Among the conclusions he proposed were that academies and free schools were a promising trend because school autonomy from local authority control was linked to better results. Also linked to better results were larger classes, better teaching, and strong national accountability systems. Since the latter claims rest on the first, we must first consider the quality of the PISA tests as evidence.
Limitations of international tests
The PISA programme which tests and collects data on schools and students internationally is a model of its kind, designed and conducted by serious social scientists with the best of intentions. However, the evidence it produces is necessarily limited in quality.
For example, it is not possible to devise a test that is equally suitable for the curriculum of every country. Some countries will have covered materials in the test more recently or more fully than others. Some countries treat PISA as another low-stakes test, others as an educational Olympics with stirring speeches and brass bands.
Some results are for rich coastal city areas like Hong Kong, most are for whole countries. Despite having a lower limit for response rates, not all schools selected to take part do so, and not all students in each school complete the test.
The response rates can vary considerably between countries. A school that made a big effort to ensure that all appropriate students, including absentees and those with learning challenges, completed the test would generally be rewarded by getting worse results than a school that included only the easy-to-reach (and easy-to-teach) students.
The actual difference in scores between countries is generally small, but this is obscured by the volatile way in which the results are presented as ranks. Statistically, the majority of countries are indistinguishable from each other. This means that the results should be used not so much on a country-by-country basis, based on one test occasion, but more in terms of trying to identify the common characteristics of schools and countries that repeatedly do well, or badly.
Patterns in PISA
Over repeated testing, one clear outcome from PISA studies has been that school systems with the highest levels of attainment tend to have mostly uniform schooling, with “comprehensive” or unselected intakes, little social and economic segregation between schools, and low or reducing gaps in attainment between different sectors of their society. Finland is the obvious example.
When it comes to autonomy, school leaders and teachers can be flexible and allow professional discretion without making the schools themselves independent of local or regional control. Schleicher was in danger of contradicting his support for the autonomy of academies when he provided examples such as the US where autonomy was part of the problem rather than any uniform solution to raising standards, or Sweden which has fallen in the recent PISA rankings.
A high proportion of academies in England are now part of a larger chain of providers, with these chains now playing a similar role to local authorities in some ways. What is clear is that academies and free schools are linked to much higher levels of socio-economic segregation between schools with all of the damage that ensues for the poverty gradient in attainment, for aspirational role models, and even social cohesion.
Schleicher is right that PISA, and other studies, have repeatedly shown that class size is not a major determinant of success. Some of the most successful Pacific-rim states have large average class sizes. The reduction of class sizes is an expensive and largely ineffective policy.
PISA is largely a test of knowledge, rather than of reasoning or of learning to learn, or any other outcome of education that might be more valuable than information transmission (in rote-learning settings). Schleicher is again right to suggest that the money could be spent better, but it is almost a tautology to say that good schools have better teaching. And better teaching is identifiable by better results.
PISA is much clearer on the nature of national school systems, with the conclusion that selective systems do not fare well. Selecting students into tracks, as in Germany, Italy or the Czech Republic, or allowing selection by schools as in Trafford and Kent in England is no better than zero-sum. No system with these kind of attributes does consistently well in PISA, however strong its accountability systems are.
And the evidence is mounting that academies in their current format are playing the same role as selective schools in England. They are separating children from different backgrounds by even more than what is also made necessary by region or housing.