Dependents

Visiting aliens could easily be forgiven for thinking those precious little bundles we carry everywhere are just another form of demanding infant – one that screams at us at inopportune moments, making us drop everything else we’re doing, nestling them to our cheeks and reassuring them with gentle murmurings. We pour our hearts into them, dote on them and spend more money on them than we ever intended to. We even feed them regularly via their own umbilical cord. But despite all the long conversations, topups and upgrades they never seem to grow up, nor do we cease to be fascinated staring into their bright little eyes.

Assessors

TEACHER 1: So, before we get down to looking through this student’s work what's your gut feeling about their mark… a “B”?

TEACHER 2: Sorry, I don’t think this is appropriate – we haven’t even looked at the backup work yet.

TEACHER 1: Sure, but you must have an idea of where they stand in terms of mark.

TEACHER 2: No, I don’t know that that I do – that’s what this process is supposed to be about surely.

TEACHER 1: It is. I was just trying to get a sense of whether we were in the least bit in agreement about the overall achievement of this student.

TEACHER 2: How can we even know that without having looked at their work?

TEACHER 1: We have though, throughout the year. Okay, I haven’t gone through it in detail yet but we’ve known them for several years now and we’ve got a pretty good sense of their track record surely?

TEACHER 2: So you think that by plucking a mark out of the air that we will be more likely to award the correct mark?

TEACHER 1: Steady on! No, not in the least. I just thought it might be useful if we knew where we were starting from, that’s all.

TEACHER 2: Well I’d prefer if we didn’t start by pigeonholing anybody and I’d also prefer not to be influenced by suggested marks before I’ve even had a chance to go through the work and form my own opinion.

TEACHER 1: That’s totally fine by me. But you seem to be suggesting that starting with a provisional mark is wrong.

TEACHER 2: I think it is wrong.

TEACHER 1: Really, why?

TEACHER 2: Because you’re starting with a bias and the likelihood is that you’ll just end up confirming the bias.

TEACHER 1: I don’t think that’s true at all. I start with a sense of what the student has achieved in the past and how their current performance compares with that. I’m also comparing their work with other work I have seen over the years at this level. As I look through their work I’m obviously looking for evidence to contradict my initial assumptions. If I find it I adjust my evaluation. If I don’t find any, my evaluation stays the same. I don’t see how that’s wrong.

TEACHER 2: And how can you be sure that the mark you start with isn’t subtly influencing your evaluation?

TEACHER 1: I’m not using it to reinforce my opinions. I’m using it as a starting point. I’m quite happy for it to change – radically if necessary. But are you telling me that you never come to the assessment of work with a ballpark idea of a mark?

TEACHER 2: I think it’s important to try to be as objective as possible.

TEACHER 1: Sure, but that’s not really what I asked. Are you saying that you treat each student’s work as though you’ve never seen it before?

TEACHER 2: I try to divorce myself from any preferences I might have formed in the time that I have known the students and yes I try to approach each student’s work as if I’ve never marked it before. Their track record is totally irrelevant as far as I’m concerned. If they have done really well in the past or really badly it has no baring on what they have submitted here. I have to mark it in relation to what is expected at the level.

TEACHER 1: So it doesn’t matter if they are seriously struggling or have improved massively since their last assessment?

TEACHER 2: Of course it matters, and we should mention it in our feedback, but that’s not what we’re marking. Marks aren’t rewards or punishments for getting better or worse. They’re an indication of the attainment of a specific level of learning.

TEACHER 1: Yes yes I know that, but do you think that we should completely ignore their journey?

TEACHER 2: In the marks, yes.

TEACHER 1: Going back to something you said a moment ago - you said you gauge the students’ work in relation to the level that is expected?

TEACHER 2: Of course.

TEACHER 1: Then I don’t really see how that is significantly different from having a ballpark idea of the mark at the beginning. Surely as you look through their work you begin to form an idea of the mark and then you keep looking until the evidence becomes overwhelming one way or the other?

TEACHER 2: No, that’s not how I evaluate things at all. I start with what is expected at the level and as I go through the work I’m looking for how well it achieves the required level. The marks are secondary.

TEACHER 1: But that sounds almost identical to what I do except I’ve already converted the level outcomes to a grade in my mind beforehand, but I‘m still looking for what is expected at the level.

LATER:

TEACHER 1: So what mark would you award this work?

TEACHER 2: I think it’s a “C”.

TEACHER 1: Yeah, I think it’s a “C” too. So, why do you think it’s a C?...

Any resemblance in the preceding script to characters or situations real or imagined is purely coincidental.

Intuition and Prejudice

“Assessments are based, not on whether the decisions made are any good, but on whether they were made in accordance with what is deemed to be an appropriate process. We assume, not only that good procedure will give rise to good outcome, but also that the ability to articulate the procedure is the key to good outcomes.” -John Kay

It could be argued that one of the principle differences between a novice and an expert is the degree to which experts are able to explain the reasons for any judgement made and to supply these retrospectively. But this begs a question: if experts make intuitive judgements – which they often do - then how can we be sure that these are not simply preferences dressed up in the trappings of deft argumentation?

A standard reply would go something like this: if the justification for an intuitive judgement stands up to rigourous debate or close scrutiny then we have no alternative than to accept the justification as valid. If such valid judgements are performed repeatedly then we need no further proof of expertise. Novices, on the other hand, are less competent in teasing out or debating their reasoning process and their judgements tend to be more inconsistent. So, even if a novice makes the same choice as an expert, we cannot say that they have exercised judgement, since judgement - in order to be judgement - needs to be justifiable.

But is this not simply an intellectual sleight of hand that surreptitiously places the quality of the justification above the quality of the decision? And whilst we may accept that instantaneous decisions are justifiable post-hoc, this still doesn’t answer the question of how such decisions can be made without calling upon resources that require far more time-consuming and considered deliberation. In other words: what is actually going on when judgements are made in an intuitive instant?

Dylan Wiliam writes of experts having what he calls “scripts” that allow them to make rapid decisions and improvise within their domain of expertise. He cites the work of Chase and Simon and their studies comparing expert and novice chess players. Chase and Simon found that expert chess players were significantly more accurate at quickly memorizing the positions of chess pieces placed in positions of actual games whereas they were significantly worse at memorizing the positions of randomly arranged chess pieces:

“Chase and Simon suggest that the much better performance of experts with ‘real’ chessboards stems from an ability to ‘read’ a chessboard in terms of a series of standard configurations of pieces that they have learnt through their experience as chess players.” 

Such expertise is the product of prolonged experience within a domain and cannot be acquired through shortcuts, formulae or crash-courses. Nonetheless we are all experts in the art of everyday life and the vast majority of the decisions we make are instantaneous and automatic. If pressed, we would have no difficulty in providing reasons for our having stopped at a red light. But can it be said that the background familiarity amassed by experts - us included - constitutes a form of criteria, or are criteria simply a means to externalize and make explicit what is essentially a tacit process that works in an entirely different way and therefore allows many decisions to be made intuitively rather than demanding deeper reflection?

"It is fairly clear that teachers acquire notions of ‘standards’ much more effectively when presented with actual samples of students’ work that exemplify the standards being promulgated than when given criterion-based descriptions of the standards. … I would like to suggest that the explanation of this phenomenon is that the grading of pieces of work with respect to a set of internal standards involves a far greater use of unconscious processing than has previously been acknowledged." -Dylan Wiliam

Psychologists, like Daniel Kahneman, distinguish between two kinds of decision making systems: System 1 and System 2. System 1 is roughly what we would think of as intuition. System 2 could be described as deliberative thinking:

“System 2 is the one who believes that it's making the decisions. But in reality, most of the time, System 1 is acting on its own, without your being aware of it. It's System 1 that decides whether you like a person, which thoughts or associations come to mind, and what you feel about something. All of this happens automatically. You can't help it, and yet you often base your decisions on it. […] System 2, on the other hand, is lazy and only becomes active when necessary. Slow, deliberate thinking is hard work. It consumes chemical resources in the brain, and people usually don't like that.” [my emphasis]

From the discussion so far it would seem that experts do not have sufficient time to formulate let alone deliberate over criteria in their intuitive judgements. This is not to say that they never pause to contemplate things more closely or even to reverse their initial judgements. They can and sometimes do. But what it does suggest is that criteria are not the primary tools of experts: experience is. By experience I mean the repertoire of exemplars that have been acquired through studying something closely – through comparing different instances and considering the relative merits of various examples, both good and bad. One nectarine alone would never be sufficient to furnish anyone with a clear idea of what constitutes a good nectarine. We need to experience a range of qualities of an experience in order to formulate an awareness of what might enhance it or detract from it, of whether the juice running down our chin is a sign of succulence or overripeness.

The point I’m making is that experts do not use criteria or deliberative thinking to form their initial judgements, instead they draw on experience and the more extensive this experience, the more well informed their judgements. However, an intuitive judgement, no matter how well informed, is not - strictly speaking - a judgement at all, but a pre-judgement; a pre-judice: an evaluation made without due consideration of the specific issues or circumstances at hand. It is a shortcut based on an unconscious repository of prior examples and is therefore limited to the known and familiar.

If the prevailing conditions in any situation that urgently demands judgement bear a close resemblance to previously encountered circumstances then it is obviously appropriate not to waste valuable time and energy logically weighing up the options.

“Intuitive intelligence is the ability to know directly, to perceive and appreciate whole or hidden patterns beyond (or faster than) logic.” -Tom Atlee (The Co-Intelligence Institute) 

But where there is no urgency, and especially where decisions impact on other people – students for instance – it seems prudent to be wary of the tendency to cut cognitive corners. As the research of Daniel Kahneman, Philip Tetlock and others increasingly shows (although it is not impossible to find exceptions), even the judgements of experts can be far from reliable, especially when making predictions about intrinsically complex and unpredictable domains. In the arts (which are well known for their unpredictability), as in other creative fields, intuition is frequently revered as a unique and often indispensable route to problem solving and discovery. But in arts education, where the judgements of teachers are directed not just towards artworks but towards individuals, too great a reliance upon intuitive judgement or gut-feeling risks undermining the putative integrity of academic assessment. And there’s the rub: what works for the production of art may well be at odds with its evaluation.

You might also be interested in the following article (Thanks to Tor for the link)

Evolution Misunderstood

Evaluating the Sensuous

"Talking of Pleasure, this moment I was writing with one hand, and with the other holding to my Mouth a Nectarine – good God, how fine. It went down soft, pulpy, slushy, oozy – all its delicious embonpoint melted down my throat like a large beatified Strawberry." –John Keats

Meaning, or “content”, as it is often described, is certainly a primary dimension of artworks but it would be a mistake to consider it as the primary dimension. Whilst art is undoubtedly a realm of significations, meanings and associations it is also a realm of the sensuous: of the profound variety and variability of physical qualities and their exquisite influence upon our senses. Artworks are formed through physical engagement with the stuff of the world and this stuff gives texture, substance and colour to experience. Indeed it is impossible to conceive of experience as something divorced from its physical causes and forms. To be without our senses would be precisely to be denied experience, since experience is surely the very stuff of sense impulses triggered and stimulated by the world around us. Without it would be to exist as an entirely disembodied consciousness in the most profoundly isolated and terrifying emptiness imaginable, with nothing but our memories and thoughts for company.

An education in the arts, at its very best, seeks to sensitise students to the subtleties and nuances of process and material, to encourage and inspire a deep familiarity with the extraordinary potential of sensory experience and to instil a sensitivity to the importance of inflection and an appreciation of the pleasures of lingering with and savouring the vast spectacle of the sensuous.

Grand ideals indeed, but are students aware that these are amongst the ‘objectives’ that their education sets out to instil and where exactly are they, or their equivalents, explicitly stated? - certainly not in the Grading Criteria, Learning Outcomes or even in the benchmark statements of the Quality Assurance Agency. How then, are the sensuous qualities of artworks to be evaluated, or rather, how might the sensuous qualities of artworks be evaluated fairly and, in order to do so, might it be necessary - or even possible - to establish meaningful and applicable criteria?

“In addition to interpretation, critics almost always make judgements about the quality of the work. Such judgements are statements about the merits of what has been seen, not statements about matters of preference. Judgements, such as “I think this is a good piece of work,” can be debated. Preferences, such as “I like this painting,” are not debatable; they are matters of choice.” –Elliot Eisner

This aspect of debatability - and the scrutiny it implies - is crucial, but in order for debate to proceed the discussion must address itself to the reasons that led to any judgements made. The more consistent and widely applicable these reasons, the more they are likely to approach what we might consider to be criteria: standards or principles by which judgements are qualified. But when it comes to sensuous experience are criteria really of any use in forming opinions? Surely we simply respond instinctively to the things we like, dislike or hate; to the things and experiences that impact upon our senses and emotions?

Certainly we are primed with instinctive reactions towards various phenomena and these instinctive responses may be relatively universal but they are also relatively unsophisticated. Just as often, perhaps significantly more so, our responses are informed and conditioned by prior experience and this allows us to judge similar experiences directly and immediately without having to abstract ourselves from the immanent sensation of embodied experience. Keat’s didn’t apply criteria to his nectarine, at least, he didn’t consciously deliberate in-the-moment over the specific qualities of his experience. To have done so would have distracted him from the fullness of experience. His evaluation – if we can call it that - is retrospective and clearly seeks not simply to describe the experience but to celebrate it by inducing, as far as possible, a palpable evocation through the eloquent flow of precisely woven form.

Could we formulate criteria for the evaluation of nectarines? No doubt, and it seems likely that these criteria would probably capture much that Keats sought to distil (fruit growers and competition judges probably already use such criteria). Could we create criteria for fruit in general? Possibly, but this would pose a much more challenging task and the criteria formed would likely have little value to the individual nectarine grower.

So, what possible value could criteria have for that impossibly diverse and bizarre form of fruit known as art? And does this mean that we have no means to fairly evaluate the sensuous qualities of art? Moreover, might the relative lack of emphasis and acknowledgement of the sensuous in art school Assessment Criteria and Learning Outcomes be responsible for a growing indifference and lack of awareness of one of art’s most vital attributes?

“How should we develop students' unassessable qualities? Should we refrain from developing them because we can't measure them?” –Phil Race