On Measuring Global Reading Progress

Reflections and comments on ACER’s Next steps: measuring reading progress

Strong governments and strong global institutions are important for defining, monitoring and addressing inequality in education.  These three policy activities are linked by policy narratives that need to be strong, coherent and consistent to garner global legitimacy. The efforts of ACER’s Centre for Global Education and Monitoring is to be commended on this front.

Work on the United Nation’s Sustainable Global Development (SGD) goals goes back to 1972, and its 2030 agenda focusing on Poverty, Food, Health, Education, Gender and Water is laudable and an agenda to which we could all agree.  However agreement on implementation requires the legitimation of a stronger narrative, and there are elements of the ACER approach that I would like to explore in this blog.

There are many things to like about ACER’s approach; the use of Item Response Theory (IRT) to develop a commonly agreed scale is one of them. IRT is a proven methodology for system and national evaluation, even though the methodology becomes suspect at the school, class and student levels.  The other welcome element is the use of pairwise comparisons in the development of content. The recent increase in the use of teacher-based pairwise comparison is welcome because it reengages the teaching profession with scale formation, an engagement that has atrophied over recent decades due to the use of IRT scaling methodologies. However, in the ACER proposal teacher engagement seems limited to pairwise comparisons in preliminary item selection, and does not seem to extend to international agreement on content.

Where the proposal is likely to encounter legitimation issues relate to the hypothesis that educational skills are universal across the target countries and able to be described on a common scale.  Sure, technically this can be done, I’ve rarely seen any test data that doesn’t scale, and where some items don’t scale properly these can be removed for ‘mysterious item reasons’. However there is bound to be concern around the legitimacy of claims about the universality of scales developed in this manner.

As I have argued elsewhere [on a unifying principle], the notion of being able to universally ‘identify where a student is’ is problematic. There are many ways of describing this issue. One way is to say that it’s too Kantian and ignores the work of Hegel in showing that knowledge is historically and socially located, and the work of Marxists that shows that formulations of knowledge can reinforce disadvantage.  Another way is to describe the approach as too metaphysical by presupposing a universal Cartesian space in which students can be located. Realism is yet another word that comes to mind, an approach that assumes that what IRT measures actually exists in reality.  Again, as I have argued elsewhere [constellation and continuum], the continuum metaphor is only one way to describe learning progress. So the observation that ‘progression occurs in a somewhat lumpy way’, is more than likely a reflection of the IRT model or metaphor, and not a phenomenon from the underlying reality of learning.  This is not to discredit the validity of the IRT model or results derived from it for the purpose of international evaluation; it simply questions the universality of any claims made.

An alternative to presupposing universal realism across nations and cultures on matters such as reading and mathematics, is to develop a procedure for SGD countries to agree on what is common to all with respect to these content areas and to create a common scale around that agreed content and then report explicitly to that effect. That is, report that the scales represent what has agreed to be common, and not was is considered universal and enduring.  The claims to universality, along with the described content methodology, could be characterised as cultural appropriation followed by cultural imperialism. Such an approach is likely to meet with resistance from teachers and the like at some point. People are social and cultural beings who use language to express themselves socially and culturally.  Of course reading progress is important to these expressions and for prosperity, but these expressions are also specific to each cultural context and not a universal function of language.

French President Charles de Gaulle’s famous 1962 observation on “How can you govern a country which has two hundred and forty-six varieties of cheese?” provides a good example of where language equivalence does not mean cultural equivalence. Cheese (Australian), kaas (Dutch) and fromage (French) are language equivalents, but Australians have Cheddar and Tasty, the Dutch have Edam and Gouda, and the French have a much broader variety.  Claims to social and cultural equivalence based on the simple language equivalence of ‘cheese’ is therefore likely to meet resistance. Reporting with claims to universality based on assessments that are only linguistically equivalent could therefore be perceived through a hegemonic narrative instead of the emancipatory one that is being sought by the UN.

It is difficult to know the status of the paper on which I’m commenting (research or marketing). It describes a comprehensive and worthwhile exercise, but it will require comprehensive consultation and discourse among target countries to develop legitimate measures that are acceptable to all.

Reflections on Rosie Batty as Australian of the Year

As Rosie Batty’s term as Australian of the Year comes to an end, I would briefly like to reflect on the impact and possible lessons we could learn from her experience and advocacy.

Like most people, the first time I saw Rosie Batty on television was in an interview shortly after Luke’s death. That interview had a profound effect. It wasn’t filled with anger, hatred or blame. Instead it was filled with sadness, compassion and understanding; including towards Luke’s father.  That she could have been anyone’s mum, daughter or friend, made me suddenly realize that this sort of domestic violence could happen to anyone.

Each time I saw Rosie on television I thought that in a better world we would never have known her.   As Australian of the Year, each one of us would have gained their own insights and inspiration from Rosie’s experience. Mine, mundane as it is, is that we should all make an effort to ensure we never put anyone in Rosie Batty’s position again.

Throughout the reporting a few key things stood out for me. The incident was not random and the justice and welfare systems had sufficient interventions to identify the problem. Further, Luke’s father had a number of active arrest warrants and intervention orders. The police even had the opportunity to arrest and detain him, but did not do so because of problems with the police database.

Problems with the Victorian police database go back a number of years.  In 2005, the director of Police Integrity, George Brouwer, called for the database to be replaced and that cost should not be a deterrent. Assistant Commissioner Kieran Walshe at time did not agree and did not consider it a priority. Successive Victorian governments have history of problems in public sector IT services includeing CenITex, LEAP, MYKI and Ultranet to name a few.

Large computer systems are not hard or impossible by their nature. What makes these projects hard are greed and fanatical desires for efficiency.  What I’ve learnt from Rosie Batty is that things that ensure the safety and welfare of our children run deep, and each of us can make a difference at every level of society.  To make a better world we could begin with a duty of care when developing IT systems, get that right and the balance sheet will look after itself.


The Age – a report on the inquest

7.30 Report story

ABC Report – 2005, on LEAP database

CenITex story

The Uncanny Progressive versus Traditional Debate

Meditation on the following blogs

Dr Beardface  On reading (part 1)  On defending shit work  On ideology

Linda J. Graham  On tax-payer funded research   Angry white men

Greg Ashman   Come, join the enlightenment    Loose ends    The disconnect

debsnet   Traditional Progressivity or Progressive Traditionalism: Ditch the dichotomy

Corinne Campbell  On TeachMeets, EduChats and Marketing

My recent twitter feed has had much discussion about traditional and enlightenment values. While some seem satisfied with their respective positions, to me it’s a manifestation of an underlying discontent that’s been brewing throughout my 30 career in education, and these blogs provide an opportunity to consider these issues propelled by real people with real emotions, not abstract ones.

While the debate had material for many tangential excursions, I will restrict this blog to a couple of key themes – post structuralism, sex and race, the Enlightenment, and the role of teachers in a post traditional landscape. Further, I haven’t addressed all the blogs related to this debate.

This blog is part of my public thinking for my PhD, and the references are as much for my research purposes as for any academic pretensions.  This topic is really too big for twitter and the blogosphere so this is more of an essay, in some ways proving the point that the topic is too dense and complex.

Post Structuralism

Issues with post-structuralism in education drove much of the twitter debate to which I’m responding, perhaps it’s best to quickly summarise my understanding of this endeavour. Structuralism was an attempt to identify underlying structures, codes and conventions that produce meaning and make meaning possible.  However early structuralists like Barthes, Lacan and Foucault recognized that meaning making is not independent of the person making the meaning; that is, a subject’s sex, social class and ethnic identity affect meaning making.  This led to post structuralism and in particular Deconstruction led by Derrida who critiqued hierarchical oppositions in Western thought. Derrida showed that notions such as inside/outside, mind/body, nature/culture were not natural but a construction.  While the work of Deconstruction sought to dismantle and reinscribe textual meaning, it did not seek to destroy meaning. However, in effect, Deconstruction did become a teasing out of warring forces of signification within a text and is therefore associated with broader movements such feminist theory, various psychoanalytic theories, Marxist thought, Post-Colonial Theory, and Minority discourses (Culler, 1997, pp. 125–131)

Posts-structuralism has generated much academic activity and material, and even if a small percentage of this material is dross, this seems to be sufficient to attract much ridicule from traditionalists. Nevertheless, post-structuralism remains a valid and useful endeavour, particularly for education which has a key interest in matters of content and representation. Drawing on the notion of education’s instructional core(Elmore, 1996), students, teachers and content are the three central concerns of education; from a post-structural perspective this translates into two meaning-making subjects and a collection of externally produced content signifiers.  It is the concern with signification and the subject that makes post-structuralism particularly relevant to education, more so than some of the other ‘posts’ related to economics, management, art and culture (e.g. Drucker, 1993; Jameson, 1991). Furthermore, all these ‘posts’ sits within broader changes within western societies sometimes described as a condition of postmodernity (e.g. Harvey, 1990; Lyotard, 1984).

Post-structuralism has failed in many respects to live up to its political promise, while it provides a range of social enquiries it seemed to have had little interest in concrete political issues such as justice, freedom, truth and autonomy (Eagleton, 2008, p. 199). One example is post-structuralism’s scepticism of Government  (see Governmentality e.g. Foucault, Burchell, Gordon, & Miller, 1991; Foucault, 2008). Without recourse to an effective government it becomes difficult to mount a case for emancipatory interests such as equality. A case for equality requires a government capable of both monitoring equality and implementing effective policy in response. Equality requires ‘governmentality’, and universal education is traditionally provided by government. So post-structuralism’s scepticism and critique of the role of government has, unwittingly or otherwise, weakened the position of the state to define, monitor and redress disadvantage in education. Furthermore, in diminishing the state’s role in defining and redressing disadvantage, post-structuralism has, again perhaps unwittingly, opened up the landscape for market forces to redefine and address perceived disadvantage.  Post-structuralism can redress this by either better scoping out its concerns to focus on signification, or by developing a stronger narrative in favour of systems and government.

Irrespective, post-structuralism will continue to have a strong role to play in education due to its concern for signification and representation, particularly when it’s able to take a ‘structuralist’ stance to inform how subject matter should be represented in the digital age. This will continue to be a highly contested area (Beavis, 2010; Kress, 2003). For this reason, post-structuralism is unlikely to be usurped by the more contemporary post-humanism  (Barad, 2003) within the field of education any time soon.

Sex, Race and Uncanny Australia

The trad-prog debate also involved sex and race through the invocation of Angry-White-Men, a reformulation of the post-modern Dead-White-Men. This invocation generated some offence as well as ironic amusement.  There is no doubting the phenomenon of the violent angry male, but there are also men who are angry about other things such as Australia’s treatment of asylum seekers and quality of universal education.  Conflating these forms of anger may not be useful.

Collins Street

Collins St, 5p.m.1955, John Brack © National Gallery of Victoria 

The distinct strata that once divided men and women has evaporated  

The distinct strata that once divided men and women in Australia have also evaporated.  Two of the protagonists propelling the twitter exchange, for example, included a senior female academic and a male student, a reversal of traditional power relations.  These inversions are no longer isolated, Australia’s richest person is now a woman, and we have had a woman prime minister. Further, for each Alan Jones and Andrew Bolt in the public sphere there are is now a corresponding – and arguably more articulate and successful – Clementine Ford and Jane Caro.  Nevertheless, while ‘social media’ power between men and women may have equalised in the public sphere, this has not necessarily translated into real economic or political equality.

Similar issues exist for race. White Australia once was able offer generosity to its Asian neighbours for failed colonisation practices in Vietnam for example. In education this led to a culture of inclusiveness. White Australia is no longer able to assert itself through either colonisation or generosity in the same way from a position of power. Chinese nationals now have the economic upper hand to purchase property within the catchment areas of some of Australia’s most sought after public schools (Chinese buyers flock to Glen Waverley). The Australian economy is no longer able to assert itself within an Occidental context of dominant white Anglo-Saxon men and women. Australia is becoming increasingly dependent on Oriental forces (see Said, 1994 for post-colonial framing of Oriental and Occidental).

So the traditional framings of feminism and post-colonialism are no longer able to provide a coherent narrative of power relations in Australia in a way that resonates with the lived experience of many Australians. Most Australians, particularly in education, now routinely report to by both men and women of both Occidental and Oriental backgrounds.  This is not to say that there are no systemic structural inequities based on sex or race, and that sex and race are no longer valid targets of public policy, but structural inequities can no longer be fully explained in terms of hegemonic white male power.   Nor can white male anger be dismissed as a contemporary manifestation of dead white men, it is likely to be more pernicious than that and involve female protagonists (e.g. Pauline Hanson)

There is therefore unfamiliarity and strangeness around the roles of sex and race in power relations, boundaries that once distinguished one from the other may no longer be tenable or recognisable.  Gelder and Jacobs, drawing on Freud and Kristeva, developed the notion of an uncanny Australia with respect to the sacredness in Aboriginal culture(Gelder & Jacobs, 1998, p. 26).  This notion of uncanniness could be extended to sex and race, an uncanniness that could itself be the root of anger.

A flight to Enlightenment

A flight to Enlightenment and towards the certainty of empiricism is one response to an uncanny Australia and a more complex environment.  While such a flight could be dismissed as a simple psychological defence, it also seems part of a broader trend and therefore worthy of exploration.  For example, Geoff Masters, CEO of Australia’s preeminent educational research organisation, considers the field of educational assessment as currently divided and in disarray due to fault lines occurring between competing philosophies, methods and approaches (Masters, 2013, p. 1). As I have argued elsewhere, Masters’ response to this disarray is a unifying principle that takes a Kantian metaphysical philosophical stance, or an early Enlightenment stance. A stance that presupposes a cognition (presumably white male) before another cognition that acts as a philosophical arbiter of practical reason, judgement, and theoretical reason (Habermas, 1996, p. 2).  Masters’ proposed principle also privileges the role of objective measurement and the Rasch Model (Masters, 1982) of which Masters is a world leading exponent. In doing so Masters also regresses to an early version of the Enlightenment that ignores Hegel’s work in showing that philosophy is not transcendental but historically located (Singer, 2001, p. 13)

Furthermore, education at heart is not a science but a social activity. Education does deal with facts, but mainly deals with norms that are socially constructed.  Facts and Norms should not be confused. Curriculum, for example, cannot be determined by empirical means. Instead, curriculum is developed by drawing on social norms and social reasoning and articulates the shared expectations of a broader community.  Even for those aspects of education that can be measured, the notion of causation is less well understood,  and the validity of meaning making is underdeveloped and under-theorized (Markus & Borsboom, 2013, p. 15). Blind experiments that are able to test some of the more contentious issues are also not possible in education due to ethical constraints, so for many of these issues effective social reasoning is required because empiricism is simply not an option.

While a retreat to the Enlightenment may be comforting and provide certainty in a time of uncanniness, even those dedicated to retrieving the Enlightenment, such as Habermas, emphasise the centrality of moral discourse and pragmatics (Habermas, 1985, 1987, 1996, 1998).

Teachers and Systems

From an effective teachers point of view the dichotomy between traditional and progressive, or any hierarchical oppositions, make little sense. Teachers are practical and pragmatic reasoners who, when given sufficient autonomy and support, use their educational expertise, their engagement with the broader educational community, and their knowledge of their students, to deliver lessons that effortlessly traverse oppositions. It is this skill and experience that makes teachers excellent social reasoners and moral agents. However articulating these skills with systems remains problematic.

Systems provide the resources and administrative authority for teachers to conduct their work, and the system-teacher relationship requires reciprocity.  Where this reciprocity is distorted it can lead to systems colonizing the world of teachers (see Habermas, 1987). One example of where the nature of reciprocity has changed relates to educational standards. Traditionally teachers, as moral agents, contributed significantly to standard setting exercises that reflected social expectations (Cizek, 2012). However, the social process of standard setting is increasingly being replaced by instrumentally defined cut-off points and levels (e.g. see OECD, 2012, pp. 258–263) which may be appropriate for system evaluation but perhaps less so for reporting to students and parents.  The diminishing role of subject associations is evidence of this transition which has led to a weakened relationship between teachers and systems.  System consultations with the teaching profession are being increasingly replaced by private discussions among board-level coteries. There is also the phenomenon of teachers in leadership positions being appropriated (bought, seduced, corrupted) by commercial interests.

So while there is reason to be sanguine about the capacity of teachers to navigate divides within their classrooms, systemic problems remain that require political action to generalise teacher experience across systems.

Looking Ahead

Julia Gillard prime ministership may provide a useful glimpse of what the future might look like. Germaine Greer describes it thus

it’s important to realise that Julia Gillard is part of a coalition. What that means is that she has to negotiate every single policy position. What that means is camel trading on the floor. It happens to be what she’s good at. You can say, ‘We want to know what she really, really believes.’ In fact, it’s irrelevant because whatever she really, really believes is not what’s going to happen.(“Q&A :Politics and porn in a post-feminist world,” 2012)

To me, this is the future. It doesn’t matter what any of us think, it is our capacity to engage and negotiate issues into action that makes us effective.  Who knows what the world would look like when men and women are equal, where the Occident and the Orient are equal, and where the Palestinian-Israeli conflict is resolved.  Nobody knows, and we will only find out if we take the necessary steps forward to engage and negotiate. There is a strong argument to be made that Gillard has been Australia’s most effective Prime Minister, not by way of being able to unify a majority around a single set of ideas in the manner of Bob Hawke, but by way of being able to effectively negotiate difference in a manner that delivered a more enlightened post-traditional society.  Further, Gillard’s post-traditional effectiveness was matched by a traditional hostility including that of Greer, who quickly followed up the above quote with comment that Gillard had a ‘big arse’; probably one of the most disappointing moments in Australia’s gender debate. Almost uncanny.

Barad, K. (2003). Posthumanist performativity : Toward an understanding of how matter comes to matter. Signs, 28(3), 801–831.

Beavis, C. A. (2010). English in the Digital Age: Making English Digital. English in Australia, 45(2), 21–30. Retrieved from http://www98.griffith.edu.au/dspace/handle/10072/37149

Cizek, G. J. (Ed.). (2012). Setting Performance Standards : Foundations, Methods, and Innovations. New York: Routledge.

Culler, J. (1997). Literary Theory: A Very Short Introduction. Oxford: Oxford University Press, UK.

Drucker, P. (1993). Post-Capitalist Society. Routledge.

Eagleton, T. (2008). Literary Theory : An Introduction, Anniversary Edition. Minneapolis: University of Minnesota Press.

Elmore, R. F. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–26.

Foucault, M. (2008). The Birth of Biopolitics: Lectures at the Collège de France, 1978–1979. (G. Burchell, Trans., A. I. Davidson, Ed.). New York: Palgrave Macmillan.

Foucault, M., Burchell, G., Gordon, C., & Miller, P. (1991). The Foucault Effect: Studies in Governmentality. Chicago: University of Chicago Press.

Gelder, K., & Jacobs, J. M. (1998). Uncanny Australia: Sacredness and Identity in a Postcolonial Nation. Carlton South: Melbourne University Press.

Habermas, J. (1985). The Theory of Communicative Action: Reason and the rationalization of society. (T. McCarthy, Trans.). Boston: Beacon Press.

Habermas, J. (1987). Lifeworld and system: a critique of functionalist reason. (T. McCarthy, Trans.). Boston: Beacon Press.

Habermas, J. (1996). Moral Consciousness and Communicative Action. (C. Lenhardt & S. W. Nicholsen, Trans.). Cambridge MA: MIT Press.

Habermas, J. (1998). Between Facts and Norms: Contributions to a Discourse Theory of Law and Democracy. (W. Rehg, Trans.). Cambridge MA: MIT Press.

Harvey, D. (1990). The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change. Cambridge MA: Blackwell.

Jameson, F. (1991). Postmodernism, Or, The Cultural Logic of Late Capitalism. Duke University Press.

Kress, G. (2003). Literacy in the New Media Age. London: Routledge.

Lyotard, J.-F. (1984). The Postmodern Condition: A Report on Knowledge. Minneapolis: University of Minnesota Press.

Markus, K. A., & Borsboom, D. (2013). Frontiers of Test Validity Theory : Measurement, Causation, and Meaning. New York: Routledge.

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Masters, G. N. (2013). Reforming Educational Assessment: Imperatives, principles and challenges. Australian Education Review. Retrieved from http://research.acer.edu.au/aer/12

OECD. (2012). PISA 2009 Technical Report. Paris: OECD Publishing. doi:10.1787/9789264167872-en

Q&A :Politics and porn in a post-feminist world. (2012). Australia: Australian Broadcasting Corporation. Retrieved from http://www.abc.net.au/tv/qanda/txt/s3451584.htm

Said, E. W. (1994). Orientalism. New York: Vintage Books.

Singer, P. (2001). Hegel: A Very Short Introduction. Oxford: OUP Oxford.


On Angry White Men

Relations between the sexes seem a lot more toxic now than I would have imagined when I was young.   During the sixties, the decade of my birth, the roles of men and women seemed to consist of distinct and persistent patterns that quickly evolved with the availability of the contraceptive pill and broader social movements. A convincing case for sexual equality was prosecuted throughout the seventies and as a teenager it pretty much seemed to me that the traditional male role of dominating nature had become redundant; bridges were easy, the Ford Falcon GTHO Phase 3 was the epitome in car production, and the world had enough bombs to destroy the earth several times over.  Man (sic) had reached the moon, and the next step to Mars seemed to involve solving social problems more than technical ones.  What was once considered a man’s traditional work had been done. So I entered the workforce quite prepared for a working life where gender roles would evolve significantly and where work was to become more socially and less technically focussed.  A grand narrative had been established, and Deborah Wardley’s victory against a silly Reg Ansett that allowed her to pilot a plane seemed a first step in a long trajectory of social progress.  Yet relations between the sexes seem more toxic now than then.

At the political level there have been mixed and sometimes troubling results. The highly admired Joan Kirner was education Minister and then State premier during the first years of my teaching career and her reforms to the VCE put Victoria in a very good place educationally.  At the time it seemed inevitable that Joan would be the first of many women premiers for Victoria, but over 20 years and 6 premiers later there has been no further progress.  Similarly for Western Australia, where there has been no progress since a concerted hatchet job was conducted against a very competent and intelligent Carmen Lawrence.  The current political environment continues to be toxic; women are welcomed as loyal deputies but shunned when they manifest any will to power.  There is the case of Julie Bishop, did she or didn’t she participate in a power play, while Truss’s subversion on Macfarlane is regarded as part of good sport.  Anthony Albanese continues to be regarded as good bloke having lost his leadership challenge, but had a woman contested and similarly lost it’s not hard to imagine that she might be characterised as spurned and brooding. How are women to be effectively socialised into leadership in such a toxic environment, and what effect does this role modelling have on relations between the sexes at work in general, school and daily life. My view is that history has not yet fully recognised the achievements of Julia Gillard, and the inevitability of  Australia having more women prime ministers in the near future is not so certain.

So here we are, things have moved significantly since the death of Barthes (1980) and Foucault (1984) yet things also seem more toxic. While the critiques of Barthes, Foucault and Derrida may still have relevance, the social conditions that they described no longer exist.  Women still encounter unfair structural and systemic hurdles to their expressions of identity and power, but these hurdles are no longer as universal and uniform as they once were. While women still encounter systemic disadvantages, there’s now also a sufficient critical mass of competent and powerful women to change the dynamic.   We now need to develop frameworks and approaches that remove toxicity from these swirling and evolving power relations.

Many men are angry but the reason for anger varies.  Some men are dangerously angry because they grieve a loss of control over women; lock them up. Some are angry because they can no longer use nature as their playground; educate them. Others are angry because objective instrumental reasoning is dead; these can be indulged a little and exposed as they’re no longer relevant. Yet others are angry because they consider humans behaving poorly towards each other and towards their environment; engage them for their energy. I draw my inspiration for anger and energy from the likes of Henry Rollins.

New theoretical positions need to be developed for the contemporary world. Foucault, Barthes and Derrida probably no longer cover it. Greer has inspired many but her contribution in calling out Julia Gillard for her ‘big arse’ has been less than useful and only contributes to toxicity.  From an educators perspective the work of Butler, Nodding and Gilligan continue to be informative.  My view is that there is further potential in integrating an ‘ethics of care’ within a grander narrative of justice. Gilligan and Kohlberg did work on this some time ago but it could be revisited.  Then there is Judy Wajcam’s work on technology and techno-feminism that challenges views towards technology.

Never mind the bickering, there’s work to be done.

The Demise of Teacher Professional Judgement

Follow up to Constellation or Continuum – metaphors for assessment

There are many ways in which teacher professional judgement can shape schooling.  Teachers can participate in the development of study designs, curriculum and syllabus, and they can also participate in exam setting, exam marking and standard setting.  In this way teachers perform sophisticated social roles in mediating between systems and the lifeworld of students as well as in setting and maintaining educational norms and expectations on behalf of the community. This kind of participation, where teachers both contribute to the creation of norms and learn how to teach them, is present in all systems to some extent, and highlights the important roles as moral agents and moral leaders that teachers can have.   However there are currently two developments working against teachers taking on system roles as moral agents:  1) instrumental reasoning of mathematical models and 2) the post-conventional/post-traditional nature of technology based education making teacher participation problematic.

Instrumental Reasoning

Where once curriculum and assessment were reflections of social expectation (including expectation of industry), this normative function has to some extent been superseded by uni-dimensional models of curriculum and assessment, mainly the Item Response Theory models (e.g. see Ayala, 2009; Embretson & Reise, 2000; Masters, 1982; Rasch, 1980) and its associated continuum metaphor.  In education systems where Item Response Theory models becomes prevalent learning progressions are less determined by social expectation and more determined by instrumentally defined scale progression, so that curriculum begins to comprise of ‘content that scales’ instead of content that meets social expectations.  Once curriculum content is comprised of ‘content that scales’, teachers’ participation in standard setting is no longer a requirement as instead of socially defined educational standards these standards can be set by way of cut-points, cut-scores and bands instrumentally and arbitrarily defined by application of Item Response Theory  based algorithms.

My thesis will argue that this phenomenon can lead to various outcomes including 1) alienation of teachers’ work, 2) curriculum and assessment not addressing social expectations, 3) students alienated from society and not fully socialised, and 4) a general loss of social capital across the system. It can also be seen as very efficient and cost saving as it doesn’t require expensive teacher engagement.

Post-conventional or post-traditional nature of education

The need to develop new educational norms and expectations during a time of developments in digital technology presents another issue for teacher engagement. Beavis (2010, p. 26) articulates this well when she states that factors such as cultural heritage and identity are at play for not only the student and teacher but also the subject itself.  The required moral reasoning of teachers is therefore far greater at a time where the system capacity of teachers has been greatly diminished through cutbacks etc. This leaves a vacated landscape that private sector can seek to fill (e.g Ultranet see Bajkowski, 2013), or other consortia (e.g. 21st Century Skills see Griffin, McGaw, & Care, 2012).


Not all contemporary assessments are grounded on mathematical models. For example the Victorian Certificate of Education (VCE) is one example of curriculum and assessment that is firmly socially grounded.  The study designs for the VCE (VCE study Designs)  reflect the social, cultural and economic activity of Victoria, and Victorian teachers are actively involved in its design and implementation, including exam setting and marking. The VCE also uses routine statistical techniques (standardization and normalization) to create a single score and then ATAR for students that can be used as currency in the future job and education market in Victoria and beyond. These features make VCE a highly regarded qualification but that it has such significant social buy-in will make it difficult to adapt to technology-based. Although this can be overcome with good management, good planning and sufficient resources for stakeholder engagement.

There is also some hope produced by the constellation metaphor and in the use of Bayesian techniques in the development of curriculum and assessment that is more comprehensive (e.g. Almond, Mislevy, Steinberg, Yan, & Williamson, 2015). However the establishment of good Bayesian belief networks also requires extensive experienced teacher participation, so the danger of the constellation metaphor is that instead of relying on teachers’ input for belief networks, these networks will instead by based on trawling through learning analytic data. Should this occur, my thesis is that this would also lead to alienating circumstances for teachers and students.

My thesis will develop with the view that sophisticated and social cohesive education systems have a sufficient base of morally competent teachers that are involved in the setting of curriculum and assessment, where the judgement of these teachers are informed and supported by sophisticated data systems (constellation and continuum). Of course this could potentiality bifurcate the other way, where teachers and students become increasingly alienated by technocratic systems.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. Tallahassee: Springer.

Ayala, R. J. De. (2009). The Theory and Practice of Item Response Theory. Guilford Press.

Bajkowski, B. J. (2013). News Review . Vic Auditor fails Ultranet, (March).

Beavis, C. A. (2010). English in the Digital Age: Making English Digital. English in Australia, 45(2), 21–30. Retrieved from http://www98.griffith.edu.au/dspace/handle/10072/37149

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. L. Erlbaum Associates.

Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and Teaching of 21st Century Skills. Dordrecht: Springer. doi:10.1007/978-94-007-2324-5

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA PRESS.

Constellation or Continuum – metaphors for assessment

In this post I want to lay ground work for a major shift in assessment methodology that education will experience in the coming decades. It will do so by discussing educational objectives, heuristic metaphors, and mathematical models.

To be clear, what we are talking about here are mathematical models and how they implement metaphors or ways of verbal reasoning about educational objectives.  These models inform how we think about and organise content, including assessment, at the system level. While this post will remain agnostic on the science of how the brain works, these models nevertheless inform how we approach students and organise schooling.

After discussing two metaphors, this post will discuss potential issues in their use with a view to informing teacher participation in a broader debate. While it may be unreasonable to expect teachers to understand the mathematics, it is reasonable to expect teachers to engage at the metaphorical and verbal levels.

Constellation or Continuum

The constellation and continuum metaphors have long and evolved histories in the academic and published literature. I will discuss these metaphors in terms of their main exponents and uses.

The continuum metaphor, or the ruler metaphor, is the one most Australians would be familiar with or have experienced.  It is the metaphor used by both NAPLAN and PISA as part of system evaluation. It is therefore also used in many derivative studies or by those who wish to align themselves with these methodologies.  Australia has many world leading exponents for the continuum metaphor with Geoff Masters the most well know due to his development of the Partial Credit Model (Masters, 1982), which was a development of the earlier Rasch Model (Rasch, 1980).  The mathematical models associated with this metaphor are generally called Rasch Models or Item Response Theory (e.g. see Ayala, 2009; Embretson & Reise, 2000) which are often described in terms of improvements to Classical Test Theory.

The constellation metaphor is not so well known in large scale assessment.  A well know exponent is Robert Mislevy who, while remaining pluralistic, opened up the field through his work with others in Evidence Centred Design (ECD) (Almond, Mislevy, Steinberg, Yan, & Williamson, 2015; Mislevy, Steinberg, Almond, Haertel, & Penuel, 2003). This metaphor can also be associated with diagnostic assessment or cognitive assessment (e.g. Leighton & Gierl, 2007, 2011; Rupp & Templin, 2008). The mathematical models associated with this metaphor include Bayesian Networks, Neural Networks and elaborations of Item Response Theory. The constellation metaphor is not as widely used as they are more difficult to implement, although they are often used in post-hoc analysis of learning data.

A simple example

The profound differences between the two metaphors can be illustrated through a simple example. Below is a diagram showing a simple test of 8 questions which tests four operations using smaller numbers then larger numbers.  Student A can do all operations but not with larger numbers. Student B can just do addition and subtraction.


The key issue here is that each student has quite a different state of proficiency yet the raw score for these two patterns cannot distinguish between them, so raw scores mathematical models as used by the continuum metaphor cannot readily detect this type of difference.  A deviant response pattern may be picked up in a misfit or bias analysis, but unless there is some additional treatment these two students will be reported the same.

The two ways of reporting these two response patterns under each metaphor is illustrated below.


It is clear that differences between the two students are lost under the continuum metaphor, but are captured under the constellation metaphor.

My hypothesis is that Australia is captured by the continuum metaphor due to the good fortune of it having the leading Item Response Theorists in the world (Masters, Adams, Andrich, Wu, Wilson etc), it is this circumstance that has also led to a neglect of the constellation metaphor and a concern about what individual Australian students are able to do; a neglect that has led to a decline in overall student performance and to a paradoxical situation where Australia is well placed to measure its decline.  This is a hypothesis only that cannot be empirically proved but which can be reasoned about.

Furthermore, I also contend that the continuum metaphor, with its focus on measurement, comparability and comparisons, is sometimes mistaken for neoliberal forces. It’s not really a conspiracy, but just a by-product of some smart people working very effectively in the endeavor of their interest.


The constellation and continuum metaphors have corresponding metaphors for how we talk about teaching.  Related to constellation metaphor is ‘who a student is’, ‘collection of knowledge’, ‘learning as growth’ and ‘depth and relation’. Related to the continuum metaphor is ‘where a student is’, ‘uni-dimensionality’, ‘teacher as conduit’, ‘learning as filling an empty vessel’.

A particularly effective use of the continuum metaphor is as a system evaluation tool, that’s why it’s used in PISA, NAPLAN and TIMSS.  As a system evaluation metaphor it is also very effective at detecting system biases and therefore it served both accountability and civil rights movements in the United States during last century (see Gordon, 2013), which in part has led to the dominance of the metaphor today.

What is clear from the example above is that the continuum metaphor, and by extension NAPLAN, is a poor diagnostic device and is able to provide little information about the student and on what to teach next, other than a vague location where a student may be in relation to other students.

While the constellation metaphor is better at providing diagnostic information to teachers, these sorts of assessments are also a lot more difficult to manage and implement and have therefore not been implemented at scale. Instead, the constellation metaphor is increasingly being used for post-hoc analysis and fishing exercises on causal relations in education; for example learning analytics (e.g. Behrens & DiCerbo, 2014).  For those who consider education as a purposeful activity, this type of post-hoc meaning making may be of concern.

I trust this may help some, writing it has helped clarify some of my thoughts.


Where both the constellation and continuum metaphors are driven by mathematical models, the determination of matters such as bands and cut-scores are largely arbitrary and determined by a choice of parameter. This contrasts to traditional standard setting procedures that are based on the professional judgements of groups of teachers (e.g. see Cizek, 2012) or holistic judgements in higher education (e.g. see Sadler, 2009).  The metaphors can of course be used to support teacher judgement, and some methods in Cizek’s book recommend this.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. Tallahassee: Springer.

Ayala, R. J. De. (2009). The Theory and Practice of Item Response Theory. Guilford Press.

Behrens, J. T., & DiCerbo, K. E. (2014). Harnessing the Currents of the Digital Ocean. In J. A. Larusson & B. White (Eds.), Learning Analytics:From Research to Practice (pp. 39–60). New York: Springer.

Cizek, G. J. (Ed.). (2012). Setting Performance Standards : Foundations, Methods, and Innovations. New York: Routledge.

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. L. Erlbaum Associates.

Gordon, E. W. (Ed.). (2013). To Assess, to Teach, to Learn: A Vision for the Future of Assessment : Technical Report. Retrieved from http://www.gordoncommission.org/rsc/pdfs/gordon_commission_technical_report.pdf

Leighton, J. P., & Gierl, M. J. (2007). Cognitive Diagnostic Assessment for Education: Theory and Applications. New York: Cambridge University Press.

Leighton, J. P., & Gierl, M. J. (2011). The Learning Sciences in Educational Assessment: The Role of Cognitive Models. Cambridge University Press.

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Mislevy, R. J., Steinberg, L. S., Almond, R. G., Haertel, G. D., & Penuel, W. R. (2003). Leverage points for improving educational assessment (PADI technical report 2). Menlo Park: SRI International.

Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA PRESS.

Rupp, A. A., & Templin, J. L. (2008). Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art. Measurement: Interdisciplinary Research & Perspective, 6(4), 219–262. doi:10.1080/15366360802490866

Sadler, D. R. (2009). Indeterminacy in the use of preset criteria for assessment and grading. Assessment & Evaluation in Higher Education.


Are Post-Structuralists too Dense and Complex?

Here’s a quick rejoinder to the blogging conversation between Greg Thompson, Greg Ashman and Naomi Barnes.

To recap, the initial Thompson post celebrated a couple of books on technology and culture and was explicitly unapologetic for the use of big words.  Ashman rejoined by questioning the value of certain types of scholarship, particularly various traditions of sociology that use big words, concluding that some sociological traditions need to do a better job of explaining themselves.  Barnes then rejoined from a post-structural, feminist, critical race perspective; saying good writing is important but this is hard in a genre dominated by white men of the Enlightenment.   While the arguments were well addressed in the various blogs, I feel the urgency of the underlying issue was not fully appreciated and I will attempt to redress that a little here.


white men

To begin, whatever the term, Occidental culture has experienced a seismic latent cultural shift since the sixties, a shift signified by many events. These events, including the 1967 summer of love in San Francisco, the 1968 Paris riots, the 1972 demolition of the Pruitt Igoe estate in St Louis, the 1973 oil crisis, the release of Koyaanisqatsi,  the art of Jeff Koons, among a range of other events, led to what some characterise as a condition of postmodernity (e.g. Harvey, 1990; Lyotard, 1984), a condition that needed to be described and theorised leading to various theoretical positions including postmodernism and post-structuralism.  These are in turn associated with the linguistic turn in philosophy, which focussed on the role of language in creating reality.

The decades of the linguistic turn of course produced much dross, and it would not be hard to find examples to sustain Ashman’s argument of too ‘Dense and Complex’.  Nevertheless such examples should not discount the worthiness of the endeavours to theorise the contemporary world. However post structuralism is not beyond criticism, skepticism or challenge. Eagleton (2008, pp. 199–200) for example, in an afterword to his 25th anniversary edition on literary theory, observes that as the 1980s wore on that post-structuralism failed to deliver on its political promise, with the German tradition including Habermas –  tenaciously clinging to topics such as discourse, justice, autonomy and ethics –  being better placed to provide a response to circumstances.  More recently others and in different traditions, such as Barad (2003), consider language as having been given too much power due to the linguistic turn and propose an alternative in posthumanism .  Given the coherence of Barad’s position, Ashman may find relief from dense and complex word smithing in her work.

The dichotomy between positivism and post-structuralism implicit in the blogs concerned me. This dichotomy does not seem appropriate for educational research or social science in general.  For example, while much quantitative educational research is dense with numerical methodology, all educational research and assessments are underpinned by educational content that is represented in some way.  It is within the brief exchange of signifiers between assessor (stimulus) and assessee (response) that post structuralism, with its tradition of addressing representation, has a role to play.  This is particularly the case as technology is restructuring the field of representation and communication leading to massive power struggles (Kress, 2003).   One current example is the big shift in how team work is being represented in schools, what used to be team sport, team dance, and team music is now appropriated thought proprietary technology (e.g. Griffin, McGaw, & Care, 2012). It is here that post-structuralists could insinuate themselves in the creation of new representational forms to be used within education to ensure that feminist and post-colonial concerns, as well as a bag of others, are addressed from the outset in the introduction of technology.  Rather than critiquing commercial implementations post-hoc, there remains an opportunity to influence given that many high stakes exams remain predominantly paper-based due to, from my perspective, a failure to agree on matters of subject matter representation using new technologies.  Post structuralist could definitely do more in asserting and explaining themselves here.

However, the real issue around which I would like to generate a sense of urgency is at the heart of Thompson’s initial post, the industrialization of time that sees humans being reconfigured for a fragmented world with disconnected events.  These observations resonate with my lived experience, amplify my own reading (e. g. Wajcman, 2015), and make me very concerned for future generations. It is towards addressing these issues, not just to identify them, that post-structuralists in education have their future work cut out.

Barad, K. (2003). Posthumanist performativity : Toward an understanding of how matter comes to matter. Signs, 28(3), 801–831.

Eagleton, T. (2008). Literary Theory : An Introduction, Anniversary Edition. Minneapolis: University of Minnesota Press.

Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and Teaching of 21st Century Skills. Dordrecht: Springer. doi:10.1007/978-94-007-2324-5

Harvey, D. (1990). The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change. Cambridge MA: Blackwell.

Kress, G. (2003). Literacy in the New Media Age. London: Routledge.

Lyotard, J.-F. (1984). The Postmodern Condition: A Report on Knowledge. Minneapolis: University of Minnesota Press.

Wajcman, J. (2015). Pressed for Time: The Acceleration of Life in Digital Capitalism. Chicago: University of Chicago Press.

The possibility of a unifying principle for assessment

[thank you to all those supporting me to date in much greater numbers than I had expected. It’s a bit difficult to stick with my longish blogs. I’m sharing my pre-confirmation PhD thinking for today so apologies for the dryness and density, but I feel that we need to go here to engage the neoliberal agenda, lighter material to come later]

Thought piece on Geoff N. Masters – Reforming Educational Assessment: Imperatives, principles and challenges


which metaphor for an assessment principle – constellation or continuum?

It is easy to agree with Geoff Masters (2013, p. 1) when he observes educational assessment as a field divided and in disarray.  Educational assessment began by providing simple reliable indicators to parents as well as to students for currency in the job and education markets.  Assessment has now grown to encompass school and system evaluation as well as scientific research, with elements of quality management and market research creeping in. Data collection is moving from research as event to embedded and ongoing research through ubiquitous and unobtrusive data collection  (Behrens & DiCerbo, 2014). This transition is blurring the demarcation between educational assessment and other forms of data collection.

While it’s easy to agree on the disarray, Masters’ unifying principle to address the chaos is problematic. Masters proposes that the fundamental purpose of assessment is to establish where learners are in their learning at the time of assessment (2013, p. 5), but this principle seems too attached to the objective measurement school and its philosophical stance.

The problem that Masters is sensibly trying to address is the divided approaches and paradigms in contemporary assessment practices such as quantitative, qualitative, formative, summative and the like. Masters addresses this problem by suggesting a universal transcendent principle to underwrite all assessment practices. But for his principle to be unifying, universal and useful it must be better than competing alternate ways of formulating a principle.  So is Masters’ principle something we could all agree to over other contenders for universal principles? While I do not propose to proffer an alternative at this stage, let’s explore Masters’ principle a little further.

Masters’ unifying principle is presaged by a learning space, either unidimensional (continuum) or multidimensional (continua), in which a student can be located at a particular point in time.  The language of the principle is about mathematical space and location, and by incorporating this metaphor into a principle he seeks to subsume all assessment practices.  His principle assumes that there is a true location at which each learner can be located at a point in time, and that once that location is determined that information can be used to fulfil all possible educational information purposes.   So there are two issues, is the location metaphor the best way to describe contemporary assessment practices, and is a location – should it be able to be determined – once determined be sufficient to meet all educational information needs.

The foundation for Masters’ principle appears to be the objective school of measurement with its Rasch-based and IRT-based models (e.g. see Embretson & Reise, 2000; Masters, 1982; Rasch, 1980). It is this school of measurement with its concerns for true score and measurement error that lends itself to the ‘where is the student’ metaphor.   However, there are increasing calls for the use of other measurement models for which the ‘who is the student’ metaphor is probably more appropriate. Notable examples of this work includes that of Mislevy as well as that of Leighton and Gierl (Almond, Mislevy, Steinberg, Yan, & Williamson, 2015; Leighton, Gierl, & Hunka, 2004; Leighton & Gierl, 2007, 2011). These alternative models, by moving away from the singular location metaphor, challenge the usefulness of Masters’ unifying principle.

There are several ways of describing and locating Masters’ unifying principle.  One that comes to mind is that Masters takes a Kantian approach with its focus on objective transcendence presupposing learning as moving from location to location. From the objective measurement school this is couched as ‘the idea of the variable must transcend any particular set of observations and the measure on the  variable must transcend the observed responses on which it is based’ (Wright & Stone, 1979, p. 141), where what is learning is seen as an a priori concept measured by the subject through empirical observation; along with appropriate application of measurement error.  By casting Masters’ approach as Kantian allows us to quickly sketch out a landscape of alternative foundations for a unifying principle.

Unlike Kant, Hegel took history into account. Where Kant thought he could say on purely philosophical grounds what human nature is and always must be, Hegel accepted that the Human condition could change from one historical era to another (Singer, 2001, p. 13). The Hegelian notion of a dynamic history challenges the stability of Masters’ notion of ‘establish where learners are’, because this location is dependent on historical context.  It’s then a fairly short leap to a Marxist critique of the principle, that any measure used to implement the principle could be biased against certain groups which of course could be mitigated by techniques such as DIF.  It is at this point that I find we can discard Masters’ principle from being universal, and that it’s at best a useful heuristic. This brief analysis points to the danger of basing principles on an instrumental technique, in this case the Rasch model. A principle should probably come before selecting a technical implementation.

When considering assessment from a Marxist perspective, and within the context of Lyotard’s (1984) analysis of knowledge , three further approaches become apparent. The first one is neo-liberalism and its concern for performativity (Ball, 2003) which Lyotard (1984, p. 54) describes as being defined by an input/output ratio. Masters’ Rasch model provides a particular advantage here over other models such as Bayesian networks .  As Masters has earlier stated, in order to enable quantitative comparisons, or make ratios, we need a linear scale that makes differences between persons the same wether through hard or easy items(Wright & Masters, 1982, p. 8). That is, the Rasch model’s ability to create linear scales dovetails neatly into neoliberalism’s need for ratios. Masters may therefore be inadvertently buttressing a neoliberal agenda with his unifying principle.

Returning to Lyotard(1984), the Marxist agenda bifurcated around the time his book was published into what I characterise as post-structuralists and neo-modernists. On assessment, the post-structuralist due to their incredulity of grand-narratives (in particular those that involve numbers) continue to take a suspicious stance towards systems and system assessment.  This stance has continued to grow since early days of the Frankfurt school in particular Marcuse and his notion of the Great Refusal (Marcuse, 1974, 2012). Post-structuralists therefore can find it difficult to engage with system assessment in a positive sense, but they have a lot to say about the lives of individuals within the lifeworld which continues to be valuable for system assessment.  Neo-modernists on the other hand, in the tradition of Habermas (1985, 1987), are simpatico with the petit narratives of the post-structuralists but engage more constructively with systems. Neo-modernists consider the system to have emancipatory potential while having a tendency to colonize the lifeworld of communities that needs to watched and mitigated through transparency and deliberate democratic processes. From a neo-modernist perspective, a principle should be based around what is sought to be achieved, what needs to be understood, or what needs to be coordinated across the system. A neo-modernist will continue to embrace the objective measurement school strongly however, because of objective measurement has a strong ability to determine DIF, bias, and fairness. But objective measurement would not presage a universal principle on assessment.

This author will continue to work in a modernist tradition towards one or more universal principles for assessment to provide alternative to Masters which I consider too close to the neoliberal agenda.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. Tallahassee: Springer.

Ball, S. J. (2003). The teacher’s soul and the terrors of performativity. Journal of Education Policy, 18(2), 215–228.

Behrens, J. T., & DiCerbo, K. E. (2014). Harnessing the Currents of the Digital Ocean. In J. A. Larusson & B. White (Eds.), Learning Analytics:From Research to Practice (pp. 39–60). New York: Springer.

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. L. Erlbaum Associates.

Habermas, J. (1985). The Theory of Communicative Action: Reason and the rationalization of society. (T. McCarthy, Trans.). Boston: Beacon Press.

Habermas, J. (1987). Lifeworld and system: a critique of functionalist reason. (T. McCarthy, Trans.). Boston: Beacon Press.

Leighton, J. P., & Gierl, M. J. (2007). Cognitive Diagnostic Assessment for Education: Theory and Applications. New York: Cambridge University Press.

Leighton, J. P., & Gierl, M. J. (2011). The Learning Sciences in Educational Assessment: The Role of Cognitive Models. Cambridge University Press.

Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka’s Rule-Space Approach. Journal of Educational Measurement, 41(3), 205–237. doi:10.1111/j.1745-3984.2004.tb01163.x

Lyotard, J.-F. (1984). The Postmodern Condition: A Report on Knowledge. Minneapolis: University of Minnesota Press.

Marcuse, H. (1974). Eros and Civilization: A Philosophical Inquiry Into Freud. Beacon.

Marcuse, H. (2012). One-Dimensional Man: Studies in the Ideology of Advanced Industrial Society (Vol. 8). Beacon Press.

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Masters, G. N. (2013). Reforming Educational Assessment: Imperatives, principles and challenges. Australian Education Review. Retrieved from http://research.acer.edu.au/aer/12

Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA PRESS.

Singer, P. (2001). Hegel: A Very Short Introduction. Oxford: OUP Oxford.

Wright, B. D., & Masters, G. N. (1982). Rating Scale Analysis. Chicago: MESA PRESS.

Wright, B. D., & Stone, M. H. (1979). Best Test Design. Chicago: MESA PRESS.

The transcendent educator and the curious case of educational boards

A recent attack by a professor on universities, including his own, led me to consider who educators are and what they stand for. There is a current trend for educators to talk outside of the institution they inhabit and to no apparent audience. This blog discusses how the transcendent disembodied educator could lead to adverse consequences.


The curious case of educational boards

Education is an institutionalised way of encountering the Other in body and spirit. Institutions create the time and space, and its people create collective wisdom and place. But do educators lose something by eschewing the collective ‘brand’. Does rejecting the neoliberal notion of ‘brand’ also ditch responsibility for presenting a coherent position to a community, as well as ditch loyalty to the teams that create those positions? What do disembodied educators stand for, and do their appeals to a generalised Other, in the form of some general goodness or badness, return us to a more primitive form of discourse.

For every unwanted ATAR is a private provider ready to sell a stairway to heaven

The Teese article that piqued my initial interest is a case in point. This article blames former governments and vice chancellors, as well as inequities in the resource distribution between Victorian public and private schools, for skewing VCE and ATAR results in favour of well-resourced schools. These are undifferentiated woes with many and varied historical antecedents. But who is Teese’s Other, who is he addressing. I can’t identify an embodied Other in Teese’s article, there is no course of action, there is no suggestion on how to make the VCE or ATAR fairer. His critique simply undermines public confidence in public institutions, thereby opening the door to the silent and opaque commercial sector. For every unwanted ATAR is a private provider ready to sell a stairway to heaven.

Victoria, as for the rest of Australia, has a proud public sector tradition in education, particularly of embracing the Other through its world class institutions including the VCE and VTAC (ATAR). Australians Ray Adams and Margaret Wu led the design and implementation of PISA and Andreas Schleicher, possibly the OECD’s most influential thinker on education, studied at Deakin University. There are many new and younger Australian talents, and along with its heritage Australia is well placed to lead the global education revolution, but leading will necessarily be complex and about doing and justifying (e.g. PISA) and not about wanton critique. In education, you can only ever ‘do’ in the presence of an Other.

In education, you can only ever ‘do’ in the presence of an Other

Of course many educators are disembodied. This article is written from a disembodied perspective without regard to an institutional loyalty. After decades of embodied educational experience this author is currently a commentator and not a player. But the notion of the transcendent disembodied educator is becoming more common. At the harmless level there is the social media profile views are my own and not my employer’s. There are young teachers unable to secure a permanent position who find it difficult to establish a sense of place. There are the teachers and bureaucrats I used to work with who would complain about the Department this and the Department that, oblivious to their sense of place and responsibility for creating organisational culture. Then there are academics like Teese that leave us to question which institution they represent and who they seek to address. Then there is the curious case of inter-connected educational boards.

There are a number of men who transcend and span organisations and for who it is difficult to ascertain a sense of place and audience. For example, Tony Mackay is Director at ACER, Council member at Swinburne University, Director of the Innovation Unit London, Board Member for Teach for Australia, on the Board for Foundation for Young Australia, CEO at the Centre for Strategic Education, past chair of AITSL, past deputy chair ACARA and has an association with ANZSOG. Another is Tony Cook, an esteemed public servant, also a Board Member at ACER and Director at AITSL. There’s also the ubiquitous John Hattie; staff member at Visible Learning, Chair at AITSL, Director of MERI at the University of Melbourne and an occasional blogger at Pearson.

The transparency (see below) of these board memberships and affiliations is testament to propriety and integrity, but what advantage do organisations gain from this level of connectedness. As an observer the range of roles illustrates an interweaving of interests and the potential for a loss of organisational agency. Networks across senior educators are of course a lot broader, deeper and opaque operating not only at the board level but also through conference attendance, keynote addresses and participation in consultative groups and workshops. A detailed study of these extended networks is beyond the scope of this blog, and also beyond the remit of overarching governance structures.

These men in some respects are the transcendent super heroes we aspire to in some of our tweets and posts, we crave the ability leap tall institutions in a single bound. But is something lost in the process? From keeping students back for detention to the awarding of contracts, educators at all levels make moral decisions. The consequences of financial and people decisions become progressively more profound up the bureaucratic hierarchy where established processes and highly refined judgement are generally required.

Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million

In some circumstances, informal coordination among peak bodies, administrators and consultants through intersecting board memberships and related affiliations is a poor substitute for structured consultation with parents, principals and teachers. Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million as was the case for Victoria’s Ultranet. Sometimes consultation is painful because it exposes ignorance; leading to witless outcomes (article on disgraced official Victoria). Victoria’s experience shows that poor governance can have disastrous consequences for education and erode proud traditions and honourable careers. Good governance and separation of responsibilities that avoid conflicts of interest remain important, particularly within the education sector that deals with large funds and people’s lives.

But educators at all levels are at times transcendent and disembodied whenever we engage the world without a clear sense of place and without a clear sense of audience. The luxury of transcendence becomes more available the further you are from the classroom. And it’s at the national level that adverse effects are detected through assessments such as PISA, recalling that PISA tests bureaucracies and systems, not teachers and students. It’s Australia’s declining PISA performance that makes this a conversation that has to be had.

It’s Australia’s declining PISA performance that makes this a conversation that has to be had. 

There are many woes in education, and while most teachers have a clear sense of place and of the Other, there are also times they feel at the bottom of the pile. The institutions available to teachers can be limited – school based committees, industrial unions, subject associations, consultation groups and political groups. Whatever the choice, and unless you are just letting of a bit of steam, it’s probably most effective to work with the Other through the institutions in front of you rather than take the transcendent disembodied stance. Then demand the same from leaders.

Cursory Web Search Results Illustrating inter-connectedness.

Tony Mackay – Director, ACER

Tony Mackay – Director, Teach For Australia

Tony Mackay – CEO, Centre for Strategic Education

Tony Mackay – Former Chair, AITSL

Tony Mackay – Former Deputy Chair, ACARA

Tony Mackay – Board of Directors, FYA

Tony Mackay – University Council, Swinburne

Tony Mackay – ANZSOG

Tony Mackay – Director, Innovation Unit

Tony Cook – Director, ACER

Tony Cook – Director, AITSL

Tony Cook – Associate Secretary, DET

John Hattie – Chair, AITSL

John Hattie – Staff member, Visible Learning

John Hattie – Director, MERI

John Hattie – Pearson writer (1)

John Hattie – Pearson blogger (2)



NAPLAN, Conflict of Interest and Research Ethics

Rejoinder to Timna Jacks – Company marking NAPLAN accused of conflict of interest

Timna Jacks’ article on Pearson Australia is a welcome reminder of Jean-François Lyotard’s (1979) seminal observations about data banks and the commercialization of knowledge.  The issues Lyotard identified have been emerging for decades and can be addressed through the tradition of research ethics.


Research ethics seeks to protect the vulnerable in data collection, in this case students.  The traditional ethical concerns of informed consent and conflict of interest are central to the issue brought to light by Jacks and are concerns that have been somewhat disregarded in the data frenzy currently capturing the education sector. On informed consent there are four key ethical issues: 1) do students have a choice about participation, 2) do students trust the data collection process, 3) are students confident that their results will be used fairly, and 4) are the interests of students, data agencies and third parties balanced.  The nature of students’ informed consent is unclear, there is a social compulsion to participate on the basis that there is a legislative mandate for students to attend school, but the mandate to attend school translates into a social expectation, and not compulsion, for students to participate in NAPLAN.  The key issue in Jacks’ article centers on the balanced interests of parties. Are the interests of students sufficiently balanced with the interests of others? The article suggests no.

By way of background, education is currently experiencing a clash of data collection traditions.  Traditionally educational assessment focused on providing a reliable indicator that teachers could use to report to parents and that systems could report to students for use in the broader education and job markets.    A second distinct data tradition relates to school evaluation and accountability such as the evaluative reports made available through the MySchool website – www.myschool.edu.au.  That NAPLAN provides a reliable indicator to parents and systems generates broad public support for the program – even if its curriculum coverage is somewhat limited. But there are now three other data traditions operating across education that may also be infiltrating NAPLAN and which may not be so transparent: scientific education research, quality management, and market research.

Public support for scientific education research tends to be high but this is a little more fraught. Scientific research is littered with disturbing episodes (e.g. Albert Neisser, Willowbrook, Tuskegee) but has been largely tamed through initiatives such as the Declaration of Helsinki and the work of university ethics committees. The extent to which NAPLAN data is used for scientific research and the ethical frameworks surrounding this research is unclear.  Therefore there is some justification for public concern on these matters.

Walter Shewhart in quality management provides yet another data tradition.  This tradition became prevalent during the industrial age to ensure quality and reproducibility of manufacturing. These techniques are now widely applied in the service sector and are increasingly being applied in education. In education, this tradition is used by educational administrators to influence the work of schools and teachers.

Finally, it is the tradition of market research that is the most pernicious in education and the issue at the heart of Jacks’ article; the possibility that data collected on the basis of creating a common understanding is being used for concealed strategic action.  While it is unlikely that this may be happening within such a large organisation, it is the possibility that it might be happening that is of concern, and it questions the social expectation that we as adults place on children to participate in NAPLAN.

The existing regulatory framework around data collection in Victoria, for example, is quite fragmented and patchwork. Children are mandated to attend school through the Education and Training Reform Act 2006 which is silent on participation in testing. There’s also the Privacy and Data Protection Act 2014, the Health Records Act 2001 and Public Records Act 1973. It is uncertain if this legislative and regulatory framework appropriately addresses the underlying issues first identified by Lyotard and which Jacks’ alerts us to in her article.

Jacks alerts us to a significant issue in education and we need to be thankful for her efforts. But it’s potentially only the tip of the iceberg in terms of ethical issues.  Two actions are required of government. First, a comprehensive review of the legislative and regulatory frameworks around data collection in education.  Should any shortcomings be identified these need to be addressed and new standards promulgated to bureaucrats, contractors, parents, teachers and students. The second action relates to conflict of interest. Government needs to centralize data collection in a new statutory agency independent from education administration and commercial education services.  That is, data collection, indicator production and reporting should reside in an authority independent from the Department responsible for the management of schools and teachers, and reside in an authority with no other responsibility but data, indicators and reporting. This would also mitigate the kind of ethical issues the Victorian department has recently experienced.