The possibility of a unifying principle for assessment

[thank you to all those supporting me to date in much greater numbers than I had expected. It’s a bit difficult to stick with my longish blogs. I’m sharing my pre-confirmation PhD thinking for today so apologies for the dryness and density, but I feel that we need to go here to engage the neoliberal agenda, lighter material to come later]

Thought piece on Geoff N. Masters – Reforming Educational Assessment: Imperatives, principles and challenges

ConstelationRuler_1

which metaphor for an assessment principle – constellation or continuum?

It is easy to agree with Geoff Masters (2013, p. 1) when he observes educational assessment as a field divided and in disarray.  Educational assessment began by providing simple reliable indicators to parents as well as to students for currency in the job and education markets.  Assessment has now grown to encompass school and system evaluation as well as scientific research, with elements of quality management and market research creeping in. Data collection is moving from research as event to embedded and ongoing research through ubiquitous and unobtrusive data collection  (Behrens & DiCerbo, 2014). This transition is blurring the demarcation between educational assessment and other forms of data collection.

While it’s easy to agree on the disarray, Masters’ unifying principle to address the chaos is problematic. Masters proposes that the fundamental purpose of assessment is to establish where learners are in their learning at the time of assessment (2013, p. 5), but this principle seems too attached to the objective measurement school and its philosophical stance.

The problem that Masters is sensibly trying to address is the divided approaches and paradigms in contemporary assessment practices such as quantitative, qualitative, formative, summative and the like. Masters addresses this problem by suggesting a universal transcendent principle to underwrite all assessment practices. But for his principle to be unifying, universal and useful it must be better than competing alternate ways of formulating a principle.  So is Masters’ principle something we could all agree to over other contenders for universal principles? While I do not propose to proffer an alternative at this stage, let’s explore Masters’ principle a little further.

Masters’ unifying principle is presaged by a learning space, either unidimensional (continuum) or multidimensional (continua), in which a student can be located at a particular point in time.  The language of the principle is about mathematical space and location, and by incorporating this metaphor into a principle he seeks to subsume all assessment practices.  His principle assumes that there is a true location at which each learner can be located at a point in time, and that once that location is determined that information can be used to fulfil all possible educational information purposes.   So there are two issues, is the location metaphor the best way to describe contemporary assessment practices, and is a location – should it be able to be determined – once determined be sufficient to meet all educational information needs.

The foundation for Masters’ principle appears to be the objective school of measurement with its Rasch-based and IRT-based models (e.g. see Embretson & Reise, 2000; Masters, 1982; Rasch, 1980). It is this school of measurement with its concerns for true score and measurement error that lends itself to the ‘where is the student’ metaphor.   However, there are increasing calls for the use of other measurement models for which the ‘who is the student’ metaphor is probably more appropriate. Notable examples of this work includes that of Mislevy as well as that of Leighton and Gierl (Almond, Mislevy, Steinberg, Yan, & Williamson, 2015; Leighton, Gierl, & Hunka, 2004; Leighton & Gierl, 2007, 2011). These alternative models, by moving away from the singular location metaphor, challenge the usefulness of Masters’ unifying principle.

There are several ways of describing and locating Masters’ unifying principle.  One that comes to mind is that Masters takes a Kantian approach with its focus on objective transcendence presupposing learning as moving from location to location. From the objective measurement school this is couched as ‘the idea of the variable must transcend any particular set of observations and the measure on the  variable must transcend the observed responses on which it is based’ (Wright & Stone, 1979, p. 141), where what is learning is seen as an a priori concept measured by the subject through empirical observation; along with appropriate application of measurement error.  By casting Masters’ approach as Kantian allows us to quickly sketch out a landscape of alternative foundations for a unifying principle.

Unlike Kant, Hegel took history into account. Where Kant thought he could say on purely philosophical grounds what human nature is and always must be, Hegel accepted that the Human condition could change from one historical era to another (Singer, 2001, p. 13). The Hegelian notion of a dynamic history challenges the stability of Masters’ notion of ‘establish where learners are’, because this location is dependent on historical context.  It’s then a fairly short leap to a Marxist critique of the principle, that any measure used to implement the principle could be biased against certain groups which of course could be mitigated by techniques such as DIF.  It is at this point that I find we can discard Masters’ principle from being universal, and that it’s at best a useful heuristic. This brief analysis points to the danger of basing principles on an instrumental technique, in this case the Rasch model. A principle should probably come before selecting a technical implementation.

When considering assessment from a Marxist perspective, and within the context of Lyotard’s (1984) analysis of knowledge , three further approaches become apparent. The first one is neo-liberalism and its concern for performativity (Ball, 2003) which Lyotard (1984, p. 54) describes as being defined by an input/output ratio. Masters’ Rasch model provides a particular advantage here over other models such as Bayesian networks .  As Masters has earlier stated, in order to enable quantitative comparisons, or make ratios, we need a linear scale that makes differences between persons the same wether through hard or easy items(Wright & Masters, 1982, p. 8). That is, the Rasch model’s ability to create linear scales dovetails neatly into neoliberalism’s need for ratios. Masters may therefore be inadvertently buttressing a neoliberal agenda with his unifying principle.

Returning to Lyotard(1984), the Marxist agenda bifurcated around the time his book was published into what I characterise as post-structuralists and neo-modernists. On assessment, the post-structuralist due to their incredulity of grand-narratives (in particular those that involve numbers) continue to take a suspicious stance towards systems and system assessment.  This stance has continued to grow since early days of the Frankfurt school in particular Marcuse and his notion of the Great Refusal (Marcuse, 1974, 2012). Post-structuralists therefore can find it difficult to engage with system assessment in a positive sense, but they have a lot to say about the lives of individuals within the lifeworld which continues to be valuable for system assessment.  Neo-modernists on the other hand, in the tradition of Habermas (1985, 1987), are simpatico with the petit narratives of the post-structuralists but engage more constructively with systems. Neo-modernists consider the system to have emancipatory potential while having a tendency to colonize the lifeworld of communities that needs to watched and mitigated through transparency and deliberate democratic processes. From a neo-modernist perspective, a principle should be based around what is sought to be achieved, what needs to be understood, or what needs to be coordinated across the system. A neo-modernist will continue to embrace the objective measurement school strongly however, because of objective measurement has a strong ability to determine DIF, bias, and fairness. But objective measurement would not presage a universal principle on assessment.

This author will continue to work in a modernist tradition towards one or more universal principles for assessment to provide alternative to Masters which I consider too close to the neoliberal agenda.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. Tallahassee: Springer.

Ball, S. J. (2003). The teacher’s soul and the terrors of performativity. Journal of Education Policy, 18(2), 215–228.

Behrens, J. T., & DiCerbo, K. E. (2014). Harnessing the Currents of the Digital Ocean. In J. A. Larusson & B. White (Eds.), Learning Analytics:From Research to Practice (pp. 39–60). New York: Springer.

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. L. Erlbaum Associates.

Habermas, J. (1985). The Theory of Communicative Action: Reason and the rationalization of society. (T. McCarthy, Trans.). Boston: Beacon Press.

Habermas, J. (1987). Lifeworld and system: a critique of functionalist reason. (T. McCarthy, Trans.). Boston: Beacon Press.

Leighton, J. P., & Gierl, M. J. (2007). Cognitive Diagnostic Assessment for Education: Theory and Applications. New York: Cambridge University Press.

Leighton, J. P., & Gierl, M. J. (2011). The Learning Sciences in Educational Assessment: The Role of Cognitive Models. Cambridge University Press.

Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka’s Rule-Space Approach. Journal of Educational Measurement, 41(3), 205–237. doi:10.1111/j.1745-3984.2004.tb01163.x

Lyotard, J.-F. (1984). The Postmodern Condition: A Report on Knowledge. Minneapolis: University of Minnesota Press.

Marcuse, H. (1974). Eros and Civilization: A Philosophical Inquiry Into Freud. Beacon.

Marcuse, H. (2012). One-Dimensional Man: Studies in the Ideology of Advanced Industrial Society (Vol. 8). Beacon Press.

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Masters, G. N. (2013). Reforming Educational Assessment: Imperatives, principles and challenges. Australian Education Review. Retrieved from http://research.acer.edu.au/aer/12

Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA PRESS.

Singer, P. (2001). Hegel: A Very Short Introduction. Oxford: OUP Oxford.

Wright, B. D., & Masters, G. N. (1982). Rating Scale Analysis. Chicago: MESA PRESS.

Wright, B. D., & Stone, M. H. (1979). Best Test Design. Chicago: MESA PRESS.

The transcendent educator and the curious case of educational boards

A recent attack by a professor on universities, including his own, led me to consider who educators are and what they stand for. There is a current trend for educators to talk outside of the institution they inhabit and to no apparent audience. This blog discusses how the transcendent disembodied educator could lead to adverse consequences.

BoardPic

The curious case of educational boards

Education is an institutionalised way of encountering the Other in body and spirit. Institutions create the time and space, and its people create collective wisdom and place. But do educators lose something by eschewing the collective ‘brand’. Does rejecting the neoliberal notion of ‘brand’ also ditch responsibility for presenting a coherent position to a community, as well as ditch loyalty to the teams that create those positions? What do disembodied educators stand for, and do their appeals to a generalised Other, in the form of some general goodness or badness, return us to a more primitive form of discourse.

For every unwanted ATAR is a private provider ready to sell a stairway to heaven

The Teese article that piqued my initial interest is a case in point. This article blames former governments and vice chancellors, as well as inequities in the resource distribution between Victorian public and private schools, for skewing VCE and ATAR results in favour of well-resourced schools. These are undifferentiated woes with many and varied historical antecedents. But who is Teese’s Other, who is he addressing. I can’t identify an embodied Other in Teese’s article, there is no course of action, there is no suggestion on how to make the VCE or ATAR fairer. His critique simply undermines public confidence in public institutions, thereby opening the door to the silent and opaque commercial sector. For every unwanted ATAR is a private provider ready to sell a stairway to heaven.

Victoria, as for the rest of Australia, has a proud public sector tradition in education, particularly of embracing the Other through its world class institutions including the VCE and VTAC (ATAR). Australians Ray Adams and Margaret Wu led the design and implementation of PISA and Andreas Schleicher, possibly the OECD’s most influential thinker on education, studied at Deakin University. There are many new and younger Australian talents, and along with its heritage Australia is well placed to lead the global education revolution, but leading will necessarily be complex and about doing and justifying (e.g. PISA) and not about wanton critique. In education, you can only ever ‘do’ in the presence of an Other.

In education, you can only ever ‘do’ in the presence of an Other

Of course many educators are disembodied. This article is written from a disembodied perspective without regard to an institutional loyalty. After decades of embodied educational experience this author is currently a commentator and not a player. But the notion of the transcendent disembodied educator is becoming more common. At the harmless level there is the social media profile views are my own and not my employer’s. There are young teachers unable to secure a permanent position who find it difficult to establish a sense of place. There are the teachers and bureaucrats I used to work with who would complain about the Department this and the Department that, oblivious to their sense of place and responsibility for creating organisational culture. Then there are academics like Teese that leave us to question which institution they represent and who they seek to address. Then there is the curious case of inter-connected educational boards.

There are a number of men who transcend and span organisations and for who it is difficult to ascertain a sense of place and audience. For example, Tony Mackay is Director at ACER, Council member at Swinburne University, Director of the Innovation Unit London, Board Member for Teach for Australia, on the Board for Foundation for Young Australia, CEO at the Centre for Strategic Education, past chair of AITSL, past deputy chair ACARA and has an association with ANZSOG. Another is Tony Cook, an esteemed public servant, also a Board Member at ACER and Director at AITSL. There’s also the ubiquitous John Hattie; staff member at Visible Learning, Chair at AITSL, Director of MERI at the University of Melbourne and an occasional blogger at Pearson.

The transparency (see below) of these board memberships and affiliations is testament to propriety and integrity, but what advantage do organisations gain from this level of connectedness. As an observer the range of roles illustrates an interweaving of interests and the potential for a loss of organisational agency. Networks across senior educators are of course a lot broader, deeper and opaque operating not only at the board level but also through conference attendance, keynote addresses and participation in consultative groups and workshops. A detailed study of these extended networks is beyond the scope of this blog, and also beyond the remit of overarching governance structures.

These men in some respects are the transcendent super heroes we aspire to in some of our tweets and posts, we crave the ability leap tall institutions in a single bound. But is something lost in the process? From keeping students back for detention to the awarding of contracts, educators at all levels make moral decisions. The consequences of financial and people decisions become progressively more profound up the bureaucratic hierarchy where established processes and highly refined judgement are generally required.

Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million

In some circumstances, informal coordination among peak bodies, administrators and consultants through intersecting board memberships and related affiliations is a poor substitute for structured consultation with parents, principals and teachers. Consulting with parents and teachers can be painful, but perhaps not as painful as wasting $180 million as was the case for Victoria’s Ultranet. Sometimes consultation is painful because it exposes ignorance; leading to witless outcomes (article on disgraced official Victoria). Victoria’s experience shows that poor governance can have disastrous consequences for education and erode proud traditions and honourable careers. Good governance and separation of responsibilities that avoid conflicts of interest remain important, particularly within the education sector that deals with large funds and people’s lives.

But educators at all levels are at times transcendent and disembodied whenever we engage the world without a clear sense of place and without a clear sense of audience. The luxury of transcendence becomes more available the further you are from the classroom. And it’s at the national level that adverse effects are detected through assessments such as PISA, recalling that PISA tests bureaucracies and systems, not teachers and students. It’s Australia’s declining PISA performance that makes this a conversation that has to be had.

It’s Australia’s declining PISA performance that makes this a conversation that has to be had. 

There are many woes in education, and while most teachers have a clear sense of place and of the Other, there are also times they feel at the bottom of the pile. The institutions available to teachers can be limited – school based committees, industrial unions, subject associations, consultation groups and political groups. Whatever the choice, and unless you are just letting of a bit of steam, it’s probably most effective to work with the Other through the institutions in front of you rather than take the transcendent disembodied stance. Then demand the same from leaders.

Cursory Web Search Results Illustrating inter-connectedness.

Tony Mackay – Director, ACER

Tony Mackay – Director, Teach For Australia

Tony Mackay – CEO, Centre for Strategic Education

Tony Mackay – Former Chair, AITSL

Tony Mackay – Former Deputy Chair, ACARA

Tony Mackay – Board of Directors, FYA

Tony Mackay – University Council, Swinburne

Tony Mackay – ANZSOG

Tony Mackay – Director, Innovation Unit

Tony Cook – Director, ACER

Tony Cook – Director, AITSL

Tony Cook – Associate Secretary, DET

John Hattie – Chair, AITSL

John Hattie – Staff member, Visible Learning

John Hattie – Director, MERI

John Hattie – Pearson writer (1)

John Hattie – Pearson blogger (2)

 

 

NAPLAN, Conflict of Interest and Research Ethics

Rejoinder to Timna Jacks – Company marking NAPLAN accused of conflict of interest

Timna Jacks’ article on Pearson Australia is a welcome reminder of Jean-François Lyotard’s (1979) seminal observations about data banks and the commercialization of knowledge.  The issues Lyotard identified have been emerging for decades and can be addressed through the tradition of research ethics.

Koomen_Poster_DataUses

Research ethics seeks to protect the vulnerable in data collection, in this case students.  The traditional ethical concerns of informed consent and conflict of interest are central to the issue brought to light by Jacks and are concerns that have been somewhat disregarded in the data frenzy currently capturing the education sector. On informed consent there are four key ethical issues: 1) do students have a choice about participation, 2) do students trust the data collection process, 3) are students confident that their results will be used fairly, and 4) are the interests of students, data agencies and third parties balanced.  The nature of students’ informed consent is unclear, there is a social compulsion to participate on the basis that there is a legislative mandate for students to attend school, but the mandate to attend school translates into a social expectation, and not compulsion, for students to participate in NAPLAN.  The key issue in Jacks’ article centers on the balanced interests of parties. Are the interests of students sufficiently balanced with the interests of others? The article suggests no.

By way of background, education is currently experiencing a clash of data collection traditions.  Traditionally educational assessment focused on providing a reliable indicator that teachers could use to report to parents and that systems could report to students for use in the broader education and job markets.    A second distinct data tradition relates to school evaluation and accountability such as the evaluative reports made available through the MySchool website – www.myschool.edu.au.  That NAPLAN provides a reliable indicator to parents and systems generates broad public support for the program – even if its curriculum coverage is somewhat limited. But there are now three other data traditions operating across education that may also be infiltrating NAPLAN and which may not be so transparent: scientific education research, quality management, and market research.

Public support for scientific education research tends to be high but this is a little more fraught. Scientific research is littered with disturbing episodes (e.g. Albert Neisser, Willowbrook, Tuskegee) but has been largely tamed through initiatives such as the Declaration of Helsinki and the work of university ethics committees. The extent to which NAPLAN data is used for scientific research and the ethical frameworks surrounding this research is unclear.  Therefore there is some justification for public concern on these matters.

Walter Shewhart in quality management provides yet another data tradition.  This tradition became prevalent during the industrial age to ensure quality and reproducibility of manufacturing. These techniques are now widely applied in the service sector and are increasingly being applied in education. In education, this tradition is used by educational administrators to influence the work of schools and teachers.

Finally, it is the tradition of market research that is the most pernicious in education and the issue at the heart of Jacks’ article; the possibility that data collected on the basis of creating a common understanding is being used for concealed strategic action.  While it is unlikely that this may be happening within such a large organisation, it is the possibility that it might be happening that is of concern, and it questions the social expectation that we as adults place on children to participate in NAPLAN.

The existing regulatory framework around data collection in Victoria, for example, is quite fragmented and patchwork. Children are mandated to attend school through the Education and Training Reform Act 2006 which is silent on participation in testing. There’s also the Privacy and Data Protection Act 2014, the Health Records Act 2001 and Public Records Act 1973. It is uncertain if this legislative and regulatory framework appropriately addresses the underlying issues first identified by Lyotard and which Jacks’ alerts us to in her article.

Jacks alerts us to a significant issue in education and we need to be thankful for her efforts. But it’s potentially only the tip of the iceberg in terms of ethical issues.  Two actions are required of government. First, a comprehensive review of the legislative and regulatory frameworks around data collection in education.  Should any shortcomings be identified these need to be addressed and new standards promulgated to bureaucrats, contractors, parents, teachers and students. The second action relates to conflict of interest. Government needs to centralize data collection in a new statutory agency independent from education administration and commercial education services.  That is, data collection, indicator production and reporting should reside in an authority independent from the Department responsible for the management of schools and teachers, and reside in an authority with no other responsibility but data, indicators and reporting. This would also mitigate the kind of ethical issues the Victorian department has recently experienced.

Time for the emeriti to stop taking pot shots at the establishment

A rejoinder to Richard Teese – How our elite unis ‘game’ the VCE and ATAR

Richard Teese’s article on universities, the ATAR and VCE shows that Hebert Marcuse’s Great Refusal is alive and well among Australia’s elite ‘progressive’ thinkers. The Great Refusal continues to provide the vacuum that allows neoliberalism to flourish. Teese’s article, with its Great Refusal tinged with a bit of ‘dead-white-male’ transcendence provides a caricature of authority that leaps established institutions at a single bound, swiping hard working teachers along the way. Teese confounds and conflates many distinct interests, institutions and dynamics in his overarching spray at public institutions; in particular, the role of universities, the ATAR and the VCE.

Teese’s authority on universities must be respected however. When he characterises their behaviour as bellicose, vain and market driven we need to accept that these are true even if somewhat localised and self-reflexive by way of Teese’s own university career.  Yet these characteristics may not be attributable to other universities with their autonomous mission and values.  Saying that universities ‘hide behind their students’, instead of taking ‘pride in their students’ could reflect Teese’s personal university experience, or may perhaps reflect a personal negative disposition.  Nevertheless, each university has its own autonomous body politic consisting of its own academics, so sweeping generalizations about their collective motivations are likely to be erroneous.

As to the ATAR, it is what it is. It provides a common meeting point for individuals and institutions to coordinate their actions.  The power of enfranchisement that the ATAR can bring is demonstrated by the celebration of Casimira Tipiloura’s achievement in being the first from the Tiwi Islands to attain an ATAR.  The ATAR provides students with a personal indicator that allows them to open up conversations with a large range of institutions on future educational options.   Of course how individual universities take up that conversation is up to them.  Some are indeed odious by priding themselves on the basis of QS rankings, research rankings, and world university rankings.  While the ATAR is a good facilitator of conversations between institutions and students, it remains a crude instrument and necessarily so given its academic and geographic scope as well as its focus on inclusiveness and fairness.  Furthermore, it is an annual ranking that’s not criterion referenced so substantive meaning of rank is not stable from year-to-year.  The ATAR therefore has no predictive validity and is not designed to be a predictive measure.  As Teese identifies the progress of students at the tertiary level is dependent on the preparedness of both students and academics as well the university’s balanced priority between teaching and research. Commentary on the ATAR’s predictive validity is therefore ill informed.

The most pernicious aspect of Teese’s commentary relates to the VCE.  The VCE remains a world leader as a broad based credential.  It has over one hundred subjects including community languages, has a wide range of assessment types, has a mix of internal and central assessment, has substantial teacher involvement in implementation at the school and central levels, and uses effective statistical techniques to articulate student VCE achievement with national and international institutions.  The VCE involves detailed processes conducted within tight timeframes on a shoe string budget.  Criticism in the vein of Teese’s can only damage these highly effective yet fragile processes operating within the public sphere. Teese’s criticism can only provide succour to neoliberal forces who would seek to apply proprietary uni-dimensional models to tertiary selection.

There are of course significant broader equity issues in education and these have been well identified by studies such as the OECD’s PISA.  It is interesting to note that the rearticulation and regurgitation of these same issues continues to be of interest to think tanks, research centres and academics alike. However, callous criticism of public and civil institutions such as the VCE and ATAR can only undermine the universal enfranchisement that these institutions seek to provide. It is a type of criticism that serves to dissolve publicly justified processes to have them replaced by opaque market based mechanisms.

The tradition of the Great Refusal is therefore alive and well and in some part explains why we have reached this current situation.  Victoria, a state with a university ranked among the best in the world for education, a university with academics that wantonly slag-off fellow state-based institutions, a state with a declining educational performance. I will not be as careless as Richard Teese with my inferences however.

What is clear is that as societies become more complex knowledge becomes increasingly differentiated, concurrent to this phenomenon are demands for broader enfranchisement for students of all backgrounds.  This requires enhanced technical and problem-solving skills across the educational industry to develop ever more comprehensive systems that are justifiably fair in ensuring enfranchisement at the national clearing house that is currently the ATAR. Unfortunately the current academic focus seems to be on cheap pot shots at the establishment, a stance that should have died in Paris in 1968.