The Demise of Teacher Professional Judgement

Follow up to Constellation or Continuum – metaphors for assessment

There are many ways in which teacher professional judgement can shape schooling.  Teachers can participate in the development of study designs, curriculum and syllabus, and they can also participate in exam setting, exam marking and standard setting.  In this way teachers perform sophisticated social roles in mediating between systems and the lifeworld of students as well as in setting and maintaining educational norms and expectations on behalf of the community. This kind of participation, where teachers both contribute to the creation of norms and learn how to teach them, is present in all systems to some extent, and highlights the important roles as moral agents and moral leaders that teachers can have.   However there are currently two developments working against teachers taking on system roles as moral agents:  1) instrumental reasoning of mathematical models and 2) the post-conventional/post-traditional nature of technology based education making teacher participation problematic.

Instrumental Reasoning

Where once curriculum and assessment were reflections of social expectation (including expectation of industry), this normative function has to some extent been superseded by uni-dimensional models of curriculum and assessment, mainly the Item Response Theory models (e.g. see Ayala, 2009; Embretson & Reise, 2000; Masters, 1982; Rasch, 1980) and its associated continuum metaphor.  In education systems where Item Response Theory models becomes prevalent learning progressions are less determined by social expectation and more determined by instrumentally defined scale progression, so that curriculum begins to comprise of ‘content that scales’ instead of content that meets social expectations.  Once curriculum content is comprised of ‘content that scales’, teachers’ participation in standard setting is no longer a requirement as instead of socially defined educational standards these standards can be set by way of cut-points, cut-scores and bands instrumentally and arbitrarily defined by application of Item Response Theory  based algorithms.

My thesis will argue that this phenomenon can lead to various outcomes including 1) alienation of teachers’ work, 2) curriculum and assessment not addressing social expectations, 3) students alienated from society and not fully socialised, and 4) a general loss of social capital across the system. It can also be seen as very efficient and cost saving as it doesn’t require expensive teacher engagement.

Post-conventional or post-traditional nature of education

The need to develop new educational norms and expectations during a time of developments in digital technology presents another issue for teacher engagement. Beavis (2010, p. 26) articulates this well when she states that factors such as cultural heritage and identity are at play for not only the student and teacher but also the subject itself.  The required moral reasoning of teachers is therefore far greater at a time where the system capacity of teachers has been greatly diminished through cutbacks etc. This leaves a vacated landscape that private sector can seek to fill (e.g Ultranet see Bajkowski, 2013), or other consortia (e.g. 21st Century Skills see Griffin, McGaw, & Care, 2012).

Discussion

Not all contemporary assessments are grounded on mathematical models. For example the Victorian Certificate of Education (VCE) is one example of curriculum and assessment that is firmly socially grounded.  The study designs for the VCE (VCE study Designs)  reflect the social, cultural and economic activity of Victoria, and Victorian teachers are actively involved in its design and implementation, including exam setting and marking. The VCE also uses routine statistical techniques (standardization and normalization) to create a single score and then ATAR for students that can be used as currency in the future job and education market in Victoria and beyond. These features make VCE a highly regarded qualification but that it has such significant social buy-in will make it difficult to adapt to technology-based. Although this can be overcome with good management, good planning and sufficient resources for stakeholder engagement.

There is also some hope produced by the constellation metaphor and in the use of Bayesian techniques in the development of curriculum and assessment that is more comprehensive (e.g. Almond, Mislevy, Steinberg, Yan, & Williamson, 2015). However the establishment of good Bayesian belief networks also requires extensive experienced teacher participation, so the danger of the constellation metaphor is that instead of relying on teachers’ input for belief networks, these networks will instead by based on trawling through learning analytic data. Should this occur, my thesis is that this would also lead to alienating circumstances for teachers and students.

My thesis will develop with the view that sophisticated and social cohesive education systems have a sufficient base of morally competent teachers that are involved in the setting of curriculum and assessment, where the judgement of these teachers are informed and supported by sophisticated data systems (constellation and continuum). Of course this could potentiality bifurcate the other way, where teachers and students become increasingly alienated by technocratic systems.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. Tallahassee: Springer.

Ayala, R. J. De. (2009). The Theory and Practice of Item Response Theory. Guilford Press.

Bajkowski, B. J. (2013). News Review . Vic Auditor fails Ultranet, (March).

Beavis, C. A. (2010). English in the Digital Age: Making English Digital. English in Australia, 45(2), 21–30. Retrieved from http://www98.griffith.edu.au/dspace/handle/10072/37149

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. L. Erlbaum Associates.

Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and Teaching of 21st Century Skills. Dordrecht: Springer. doi:10.1007/978-94-007-2324-5

Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. doi:10.1007/BF02296272

Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA PRESS.

Time for the emeriti to stop taking pot shots at the establishment

A rejoinder to Richard Teese – How our elite unis ‘game’ the VCE and ATAR

Richard Teese’s article on universities, the ATAR and VCE shows that Hebert Marcuse’s Great Refusal is alive and well among Australia’s elite ‘progressive’ thinkers. The Great Refusal continues to provide the vacuum that allows neoliberalism to flourish. Teese’s article, with its Great Refusal tinged with a bit of ‘dead-white-male’ transcendence provides a caricature of authority that leaps established institutions at a single bound, swiping hard working teachers along the way. Teese confounds and conflates many distinct interests, institutions and dynamics in his overarching spray at public institutions; in particular, the role of universities, the ATAR and the VCE.

Teese’s authority on universities must be respected however. When he characterises their behaviour as bellicose, vain and market driven we need to accept that these are true even if somewhat localised and self-reflexive by way of Teese’s own university career.  Yet these characteristics may not be attributable to other universities with their autonomous mission and values.  Saying that universities ‘hide behind their students’, instead of taking ‘pride in their students’ could reflect Teese’s personal university experience, or may perhaps reflect a personal negative disposition.  Nevertheless, each university has its own autonomous body politic consisting of its own academics, so sweeping generalizations about their collective motivations are likely to be erroneous.

As to the ATAR, it is what it is. It provides a common meeting point for individuals and institutions to coordinate their actions.  The power of enfranchisement that the ATAR can bring is demonstrated by the celebration of Casimira Tipiloura’s achievement in being the first from the Tiwi Islands to attain an ATAR.  The ATAR provides students with a personal indicator that allows them to open up conversations with a large range of institutions on future educational options.   Of course how individual universities take up that conversation is up to them.  Some are indeed odious by priding themselves on the basis of QS rankings, research rankings, and world university rankings.  While the ATAR is a good facilitator of conversations between institutions and students, it remains a crude instrument and necessarily so given its academic and geographic scope as well as its focus on inclusiveness and fairness.  Furthermore, it is an annual ranking that’s not criterion referenced so substantive meaning of rank is not stable from year-to-year.  The ATAR therefore has no predictive validity and is not designed to be a predictive measure.  As Teese identifies the progress of students at the tertiary level is dependent on the preparedness of both students and academics as well the university’s balanced priority between teaching and research. Commentary on the ATAR’s predictive validity is therefore ill informed.

The most pernicious aspect of Teese’s commentary relates to the VCE.  The VCE remains a world leader as a broad based credential.  It has over one hundred subjects including community languages, has a wide range of assessment types, has a mix of internal and central assessment, has substantial teacher involvement in implementation at the school and central levels, and uses effective statistical techniques to articulate student VCE achievement with national and international institutions.  The VCE involves detailed processes conducted within tight timeframes on a shoe string budget.  Criticism in the vein of Teese’s can only damage these highly effective yet fragile processes operating within the public sphere. Teese’s criticism can only provide succour to neoliberal forces who would seek to apply proprietary uni-dimensional models to tertiary selection.

There are of course significant broader equity issues in education and these have been well identified by studies such as the OECD’s PISA.  It is interesting to note that the rearticulation and regurgitation of these same issues continues to be of interest to think tanks, research centres and academics alike. However, callous criticism of public and civil institutions such as the VCE and ATAR can only undermine the universal enfranchisement that these institutions seek to provide. It is a type of criticism that serves to dissolve publicly justified processes to have them replaced by opaque market based mechanisms.

The tradition of the Great Refusal is therefore alive and well and in some part explains why we have reached this current situation.  Victoria, a state with a university ranked among the best in the world for education, a university with academics that wantonly slag-off fellow state-based institutions, a state with a declining educational performance. I will not be as careless as Richard Teese with my inferences however.

What is clear is that as societies become more complex knowledge becomes increasingly differentiated, concurrent to this phenomenon are demands for broader enfranchisement for students of all backgrounds.  This requires enhanced technical and problem-solving skills across the educational industry to develop ever more comprehensive systems that are justifiably fair in ensuring enfranchisement at the national clearing house that is currently the ATAR. Unfortunately the current academic focus seems to be on cheap pot shots at the establishment, a stance that should have died in Paris in 1968.