On Measuring Global Reading Progress

Reflections and comments on ACER’s Next steps: measuring reading progress

Strong governments and strong global institutions are important for defining, monitoring and addressing inequality in education.  These three policy activities are linked by policy narratives that need to be strong, coherent and consistent to garner global legitimacy. The efforts of ACER’s Centre for Global Education and Monitoring is to be commended on this front.

Work on the United Nation’s Sustainable Global Development (SGD) goals goes back to 1972, and its 2030 agenda focusing on Poverty, Food, Health, Education, Gender and Water is laudable and an agenda to which we could all agree.  However agreement on implementation requires the legitimation of a stronger narrative, and there are elements of the ACER approach that I would like to explore in this blog.

There are many things to like about ACER’s approach; the use of Item Response Theory (IRT) to develop a commonly agreed scale is one of them. IRT is a proven methodology for system and national evaluation, even though the methodology becomes suspect at the school, class and student levels.  The other welcome element is the use of pairwise comparisons in the development of content. The recent increase in the use of teacher-based pairwise comparison is welcome because it reengages the teaching profession with scale formation, an engagement that has atrophied over recent decades due to the use of IRT scaling methodologies. However, in the ACER proposal teacher engagement seems limited to pairwise comparisons in preliminary item selection, and does not seem to extend to international agreement on content.

Where the proposal is likely to encounter legitimation issues relate to the hypothesis that educational skills are universal across the target countries and able to be described on a common scale.  Sure, technically this can be done, I’ve rarely seen any test data that doesn’t scale, and where some items don’t scale properly these can be removed for ‘mysterious item reasons’. However there is bound to be concern around the legitimacy of claims about the universality of scales developed in this manner.

As I have argued elsewhere [on a unifying principle], the notion of being able to universally ‘identify where a student is’ is problematic. There are many ways of describing this issue. One way is to say that it’s too Kantian and ignores the work of Hegel in showing that knowledge is historically and socially located, and the work of Marxists that shows that formulations of knowledge can reinforce disadvantage.  Another way is to describe the approach as too metaphysical by presupposing a universal Cartesian space in which students can be located. Realism is yet another word that comes to mind, an approach that assumes that what IRT measures actually exists in reality.  Again, as I have argued elsewhere [constellation and continuum], the continuum metaphor is only one way to describe learning progress. So the observation that ‘progression occurs in a somewhat lumpy way’, is more than likely a reflection of the IRT model or metaphor, and not a phenomenon from the underlying reality of learning.  This is not to discredit the validity of the IRT model or results derived from it for the purpose of international evaluation; it simply questions the universality of any claims made.

An alternative to presupposing universal realism across nations and cultures on matters such as reading and mathematics, is to develop a procedure for SGD countries to agree on what is common to all with respect to these content areas and to create a common scale around that agreed content and then report explicitly to that effect. That is, report that the scales represent what has agreed to be common, and not was is considered universal and enduring.  The claims to universality, along with the described content methodology, could be characterised as cultural appropriation followed by cultural imperialism. Such an approach is likely to meet with resistance from teachers and the like at some point. People are social and cultural beings who use language to express themselves socially and culturally.  Of course reading progress is important to these expressions and for prosperity, but these expressions are also specific to each cultural context and not a universal function of language.

French President Charles de Gaulle’s famous 1962 observation on “How can you govern a country which has two hundred and forty-six varieties of cheese?” provides a good example of where language equivalence does not mean cultural equivalence. Cheese (Australian), kaas (Dutch) and fromage (French) are language equivalents, but Australians have Cheddar and Tasty, the Dutch have Edam and Gouda, and the French have a much broader variety.  Claims to social and cultural equivalence based on the simple language equivalence of ‘cheese’ is therefore likely to meet resistance. Reporting with claims to universality based on assessments that are only linguistically equivalent could therefore be perceived through a hegemonic narrative instead of the emancipatory one that is being sought by the UN.

It is difficult to know the status of the paper on which I’m commenting (research or marketing). It describes a comprehensive and worthwhile exercise, but it will require comprehensive consultation and discourse among target countries to develop legitimate measures that are acceptable to all.

The transcendent educator and the curious case of educational boards

A recent attack by a professor on universities, including his own, led me to consider who educators are and what they stand for. There is a current trend for educators to talk outside of the institution they inhabit and to no apparent audience. This blog discusses how the transcendent disembodied educator could lead to adverse consequences.


The curious case of educational boards

Education is an institutionalised way of encountering the Other in body and spirit. Institutions create the time and space, and its people create collective wisdom and place. But do educators lose something by eschewing the collective ‘brand’. Does rejecting the neoliberal notion of ‘brand’ also ditch responsibility for presenting a coherent position to a community, as well as ditch loyalty to the teams that create those positions? What do disembodied educators stand for, and do their appeals to a generalised Other, in the form of some general goodness or badness, return us to a more primitive form of discourse.

For every unwanted ATAR is a private provider ready to sell a stairway to heaven

The Teese article that piqued my initial interest is a case in point. This article blames former governments and vice chancellors, as well as inequities in the resource distribution between Victorian public and private schools, for skewing VCE and ATAR results in favour of well-resourced schools. These are undifferentiated woes with many and varied historical antecedents. But who is Teese’s Other, who is he addressing. I can’t identify an embodied Other in Teese’s article, there is no course of action, there is no suggestion on how to make the VCE or ATAR fairer. His critique simply undermines public confidence in public institutions, thereby opening the door to the silent and opaque commercial sector. For every unwanted ATAR is a private provider ready to sell a stairway to heaven.

Victoria, as for the rest of Australia, has a proud public sector tradition in education, particularly of embracing the Other through its world class institutions including the VCE and VTAC (ATAR). Australians Ray Adams and Margaret Wu led the design and implementation of PISA and Andreas Schleicher, possibly the OECD’s most influential thinker on education, studied at Deakin University. There are many new and younger Australian talents, and along with its heritage Australia is well placed to lead the global education revolution, but leading will necessarily be complex and about doing and justifying (e.g. PISA) and not about wanton critique. In education, you can only ever ‘do’ in the presence of an Other.

In education, you can only ever ‘do’ in the presence of an Other

Of course many educators are disembodied. This article is written from a disembodied perspective without regard to an institutional loyalty. After decades of embodied educational experience this author is currently a commentator and not a player. But the notion of the transcendent disembodied educator is becoming more common. At the harmless level there is the social media profile views are my own and not my employer’s. There are young teachers unable to secure a permanent position who find it difficult to establish a sense of place. There are the teachers and bureaucrats I used to work with who would complain about the Department this and the Department that, oblivious to their sense of place and responsibility for creating organisational culture. Then there are academics like Teese that leave us to question which institution they represent and who they seek to address. Then there is the curious case of inter-connected educational boards.

There are a number of men who transcend and span organisations and for who it is difficult to ascertain a sense of place and audience. For example, Tony Mackay is Director at ACER, Council member at Swinburne University, Director of the Innovation Unit London, Board Member for Teach for Australia, on the Board for Foundation for Young Australia, CEO at the Centre for Strategic Education, past chair of AITSL, past deputy chair ACARA and has an association with ANZSOG. Another is Tony Cook, an esteemed public servant, also a Board Member at ACER and Director at AITSL. There’s also the ubiquitous John Hattie; staff member at Visible Learning, Chair at AITSL, Director of MERI at the University of Melbourne and an occasional blogger at Pearson.

The transparency (see below) of these board memberships and affiliations is testament to propriety and integrity, but what advantage do organisations gain from this level of connectedness. As an observer the range of roles illustrates an interweaving of interests and the potential for a loss of organisational agency. Networks across senior educators are of course a lot broader, deeper and opaque operating not only at the board level but also through conference attendance, keynote addresses and participation in consultative groups and workshops. A detailed study of these extended networks is beyond the scope of this blog, and also beyond the remit of overarching governance structures.

These men in some respects are the transcendent super heroes we aspire to in some of our tweets and posts, we crave the ability leap tall institutions in a single bound. But is something lost in the process? From keeping students back for detention to the awarding of contracts, educators at all levels make moral decisions. The consequences of financial and people decisions become progressively more profound up the bureaucratic hierarchy where established processes and highly refined judgement are generally required.

Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million

In some circumstances, informal coordination among peak bodies, administrators and consultants through intersecting board memberships and related affiliations is a poor substitute for structured consultation with parents, principals and teachers. Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million as was the case for Victoria’s Ultranet. Sometimes consultation is painful because it exposes ignorance; leading to witless outcomes (article on disgraced official Victoria). Victoria’s experience shows that poor governance can have disastrous consequences for education and erode proud traditions and honourable careers. Good governance and separation of responsibilities that avoid conflicts of interest remain important, particularly within the education sector that deals with large funds and people’s lives.

But educators at all levels are at times transcendent and disembodied whenever we engage the world without a clear sense of place and without a clear sense of audience. The luxury of transcendence becomes more available the further you are from the classroom. And it’s at the national level that adverse effects are detected through assessments such as PISA, recalling that PISA tests bureaucracies and systems, not teachers and students. It’s Australia’s declining PISA performance that makes this a conversation that has to be had.

It’s Australia’s declining PISA performance that makes this a conversation that has to be had. 

There are many woes in education, and while most teachers have a clear sense of place and of the Other, there are also times they feel at the bottom of the pile. The institutions available to teachers can be limited – school based committees, industrial unions, subject associations, consultation groups and political groups. Whatever the choice, and unless you are just letting of a bit of steam, it’s probably most effective to work with the Other through the institutions in front of you rather than take the transcendent disembodied stance. Then demand the same from leaders.

Cursory Web Search Results Illustrating inter-connectedness.

Tony Mackay – Director, ACER

Tony Mackay – Director, Teach For Australia

Tony Mackay – CEO, Centre for Strategic Education

Tony Mackay – Former Chair, AITSL

Tony Mackay – Former Deputy Chair, ACARA

Tony Mackay – Board of Directors, FYA

Tony Mackay – University Council, Swinburne

Tony Mackay – ANZSOG

Tony Mackay – Director, Innovation Unit

Tony Cook – Director, ACER

Tony Cook – Director, AITSL

Tony Cook – Associate Secretary, DET

John Hattie – Chair, AITSL

John Hattie – Staff member, Visible Learning

John Hattie – Director, MERI

John Hattie – Pearson writer (1)

John Hattie – Pearson blogger (2)