Signification and Sensibility: Notes on the Photographic Index

Is smoke a representation of the fire that produces it? Do shadows represent the things that cast them. Do the rings of a tree represent its age? Is the inverted view seen in a dewdrop a representation? Is a footprint? Or an echo?

For Charles Sanders Peirce (1839 – 1914) each of the above would constitute what he defined as “Indices.” For Peirce "An Index [singular] is a sign which refers to the Object that it denotes by virtue of being really affected by that Object.” Indices “represent their objects independently of any resemblance to them, only by virtue of real connections with them.”

This notion of a causal connection between the index and its object has often been thought to confer a special status upon indexical signs, and amongst such signs none has figured more prominently, or generated more discussion, than that of the photograph. Photographs are the result of electrochemical changes manifested in a photosensitive surface in response to exposure to light, and this causal relationship is believed to contribute to, or constitute even, the unique - and some would say “objective” – identity or “truth” of photographic images.

Perhaps most notable of all commentators upon the indexical nature of photography has been the French literary theorist and philosopher Roland Barthes whose 1964 Essay “The Rhetoric of the Image” proclaimed that the photograph is a “message without a code.” (other theorists argue something similar when they claim that photography is “transparent” to its subject) For Barthes this unique feature of photography is a consequence of its “absolutely analogical nature.” Barthes argued that photography should thus be differentiated from “drawing which, even when denoted, is a coded message.”

Since 1964 many theorists have followed in Barthes’ footsteps, variously arguing for the unique status of photography, its “truth claims”, its essential “objectivity”, “immediacy” or “reality.” But rather than explore the intricacies of these theories I’d like instead to make a point that may well have been obvious right from the outset.

The smoke rising from a chimney is not a representation and nor is a shadow, an echo or the miniature view spied in a dewdrop. It is true that these things have direct causes but the fact that these are implied, suggested, connoted, denoted, indicated, signified or otherwise made evident does not bring about a transformation of their subsequent effects into representations. An obvious or inevitable connection between an effect and its cause does not yield a representation. Representations are the products of intentional acts of substitution.

And what then of the unique status of photography? Certainly it can’t be anything to do with its absolute analogical nature. And if it is, we are simply left with the very obvious question raised by digital photography. It might then be argued that the analogical nature of photography, to which Barthes was referring, was in fact closer to what C. S. Peirce meant when he wrote:

“Photographs, especially instantaneous photographs, are very instructive, because we know that they are in certain respects exactly like the objects they represent. But this resemblance is due to the photographs having been produced under such circumstances that they were physically forced to correspond point by point to nature.”

Notice though that Peirce states that photographs “are in certain respects exactly like the objects they represent.” Did he mean to say that they are exactly the same as the objects they represent? Presumably not. To be exactly like is precisely not to be exactly the same. If Peirce had been more exacting he should perhaps have written: photographs are in certain respects and in certain circumstances perceptually indiscriminable from the things they represent. In other words, if the circumstances of presentation are very tightly controlled (in a lab experiment for example), we can find that we are unable to discriminate the difference between a photograph and whatever it represents in a variety of respects. It is nonetheless likely that we could easily distinguish between the representation and its object but it is important to point out here that conceptual distinguishability and perceptual discriminability are distinctly different. We are capable of distinguishing between things as a consequence of our ability to use language, whereas our capacity to discriminate perceptually is genetically acquired. Infants don’t have to learn to see photographic representations any more than they have to learn to see. Distinguishing, on the other hand, requires the development conceptual understanding.

It should be apparent then, that our genetically acquired ability to perceptually discriminate or, more precisely, to fail to discriminate, is the decisive factor that enables photographic representation. And it should also be noted that this includes all photorealistic depictions, despite Barthes’ logically unsupportable contention that all drawings are coded.

It turns out then, that Peirce’s theory of the index – at least as it pertains to photographs – is incapable of throwing much light upon the nature of  photography. It both lacks analytical precision and places too great an emphasis on the immediate causal history of photographs. As every digital photograph clearly attests, the causal history – even the immediate causal history - of the photographic image is often little more than a mystery to most of us. The same can be said of analogue photography in fact. This isn’t to say that the causal history of photographs is unimportant. Of course, some makers of photographic images deliberately choose to emphasise the indexical aspects of their craft; to draw attention to the processes of production, to the methods, qualities and materiality of the print and so on. In this sense they rely on, or seek to encourage, our abilities to make distinctions, to recognise, to compare and to distinguish. We might also say that these often subtle qualities draw upon our sensibilities, but this word demands a little care. If by sensibilities we mean some kind of genetically inherited ability to sense the finer details and qualities of things then we might have reason to be sceptical. Are our sensibilities genetically acquired? To some small degree perhaps they are, but for the most part our sensibilities are developed and refined through experience and most especially through considered attention, assisted significantly by our abilities to use language. Sensibilities are cultivated.

It might also be argued that photographs function by way of reference, signification or meaning. There is no doubt at all that photographs indicate, connote, denote, signify and allude to all kinds of things but they are by no means reliant upon these forms of signification because meaning, like sensibility, derives from our skills as language users. Although photography and other forms of non-verbal representation are enormously influenced, assisted and informed by language – by our abilities to distinguish - they are made possible by circumstances that must certainly have preceded our capacity to use language. They are derived moreover from a felicitous vulnerability in our perceptual capacities, a systematic perceptual weakness that we have discovered how to exploit to our profound and extraordinary advantage.

Representing the Mind

Researchers in the fields of cognitive neuroscience and Artificial Intelligence are fond describing the brain as a highly sophisticated computational device in which data is encoded and processed. Prior to the invention and development of computers many of the most prominent theories of the "workings" of the brain assumed either that it was minutely mechanical in its functioning (tiny cogs and pulleys, weights and levers) or else – or also - that the physical matter of the brain was simply some form of interface between the brain and the non-physical mind. This “Dualistic” theory, as it came to be known, is most closely associated with the 17^th Century French philosopher Rene Descartes.

Since the 17^th Century, advances in scientific understanding have enabled us to be a great deal more certain that the mind is entirely physical in origin, yet vestiges of Cartesian Dualism continue to linger in many quarters despite the overwhelming evidence that consciousness – the mind - is a product of electrochemical processes occurring in – or largely in - the brain.

Throughout history, many differing conceptions of the origins of thought and intelligent agency have stocked our vocabulary with a rich variety of metaphors borrowed from a range of objects, activities and occupations. We talk of the mechanisms of the mind, of streams of consciousness and information flows, of brain cells lighting up, sparking and firing, of cascades, filters, causal chains and neural triggers.

Such terms are extremely useful and often unavoidable in formulating theories and characterising barely understood processes. But characterisations can also present significant difficulties, especially if taken too literally as conceptual models. When attempting to understand complex and currently unexplained phenomena we are inevitably restricted both by what we know and what we don't know. Our knowledge and the terminology we have to describe it frames our thinking in ways that do not always help to illuminate the issues or processes we seek to understand. The same was true for Descartes, and is no doubt the reason he rejected the mechanical model of mind in favour of a non-physical account. It is perhaps possible then, to see how the versatility and character of the analogies available to us can significantly influence the concepts we are likely to formulate and the conclusions we are likely to draw during our exploration of unfamiliar processes and phenomena.

Since the 1970's Deirdre Gentner’s research has focussed on the use of analogical reasoning in the domains of cognitive science and artificial intelligence in particular. Gentner’s findings show that “analogical access and analogical inference are governed by very different rules.” In other words, when we make comparisons we have a tendency to pick out superficially accessible analogies instead of analogies that might offer greater predictive potential. I think this is precisely what is happening when the functioning of the brain is explained by way of computational analogies. Even more worrying though is the assumption – frequently made by theorists and scientists - that brains necessarily utilise encoded representations. I would argue that the failure to recognise the incoherence of this analogy is having a significantly detrimental influence on the development of both cognitive neuroscience and artificial intelligence.

In their 1995 book “Foundational Issues in Artificial Intelligence and Cognitive Science” Mark Bickhard and Loren Terveen argue that the fields of Cognitive Science and Artificial Intelligence are “in the midst of a programmatic impasse.” They make the vital point that this impasse cannot be overcome by making changes – no matter how insightful - at the “project level.” That is to say; if the foundations aren’t sound, no amount of work above ground is ever likely to resolve the initial errors - in fact it is far more likely to compound them.

Bickhard and Terveen coin the term “Encodingism” to describe the assumption that cognition involves the use of representational encodings. They write:

“It is assumed that the symbol represents a particular thing, and that it - the symbol - somehow informs the system of what that symbol is supposed to represent. This is a fatal assumption in spite of its seeming obviousness.”

The point they are making is that it is by no means straightforward what any symbol is actually intended to symbolise. As C. S. Peirce theorised at the end of the 19^th Century, symbols represent their denoted objects as a result of rules or interpretive habits that are independent of any properties that may or may not be shared between symbols and the objects they represent. Symbols are things that require interpretation. They do not lead immediately and without complication to the things they denote. The rules and habits of symbol use have to be learned. If this is true, then it is extremely doubtful that an intelligence could emerge capable of interpreting symbols without a prior non-symbol-using evolution. And if intelligence is possible without symbol use, then no encoded symbols are required for cognition.

Knowing Absence

Towards an alternative theory of the role of representation in perception.

“You don't merely think that the tomato has a back, or judge or infer that it is there. You have a sense, a visual sense, of its presence. In what does the visual sense of the presence of the hidden parts of the thing consist, if it does not consist in the fact that we see them? This is the problem of perceptual presence—or better: the problem of presence in absence. The object shows up for visual consciousness precisely as unseen. […] The thing that needs explaining is not that we mistakenly take ourselves to see something that we don't see. The puzzle is that we take ourselves to have a sense of the presence in perception of something that is manifestly out of view." —Alva Noë

Here contemporary philosopher of perception, Alva Noë, outlines the the issue of what he terms "presence in absence." Whilst I do not agree with all of Noë’s theories, his avoidance of any reliance upon cognitive representational states is very much to his credit. Nonetheless, in rejecting these unsubstantiated phantoms, these spectral darlings of much scientific and philosophical presupposition, he pays little regard to the role of other forms of representation (the only kind for which proof exists) that are so vital for an explanation of our ability to conceive of absence and, arguably, of perception itself.

Noë is right, I think, to identify a significant philosophical issue with the question of absence, of our capacity to account for the parts of the things that are currently out of view. For him the answer to this paradox lies in our "sensorimotor understanding” and “conceptual knowledge" of the things that we see that "brings the world into focus for perceptual consciousness" and that "discloses the world to us." He writes:

"To perceive something, you must understand it, and to understand it you must, in a way, already know it, you must have already made its acquaintance.

There are no novel experiences. The conditions of novelty are, in effect, the conditions of invisibility. To experience something, you must comprehend it by the familiarizing work of the understanding. You must master it. Domesticated it. Know it."

But how should we interpret this “understanding”, this “perceiving”, this "knowing"? If I reach for what I know is a fresh tomato only to find that it is fake then quite evidently there was something I didn’t know that I now know. Moreover, if the new knowledge contradicts the assumed knowledge – the preconception - then I think it would be right to say that I have encountered an unexpected, if not novel, experience. If someone pops their head around my door to say hello, I know that I’m not looking at a disembodied head. But I’m not convinced that the best explanation for what I perceive is provided by reference to my sensorimotor understanding or even to my conceptual knowledge. What the explanation requires is my capacity to answer the question – should it ever arise – “To what is the head attached?”

For an organism to be surprised - for novelty even - there has to be a capacity for expectation. Without an expectation of presence it is inconceivable that an absence could be registered. A vixen doesn't register the absence of an inexistent fox cub any more than we might detect the absence of the future. Or, to paraphrase Noë: to expect something, you must have already made its acquaintance. But, as we have already seen, to be acquainted with something, does not necessitate knowing it in any substantive sense. To be acquainted with something may be to know nothing more than what to expect when becoming reacquainted with it – to be aware of what features, attributes or properties one would miss if they were absent. In order for this to be possible we need not assume representational mental states nor require sensorimotor understanding but merely a capacity for registering a disagreement between the current perception and whatever residual cognitive capacities (memories) remain of the thing as previously encountered.

But to conclude the explanation here would be to neglect the most important part of the story. What needs explaining is the nature of this capacity to register absence. Advocates of representational states will simply assert that we possess cognitive representations that allow us to perceive absence. But as Noë rightly argues, an absence is not something that can figure as a mental state, let alone a representational mental state because "What they [neurons] can’t do is fire in such a way as to signal that a hidden feature is present”

The reason that our capacity to produce representations is so vital to an understanding of absence is because perception is conditional upon our capacity to represent things. For sophisticated representation users like ourselves this capacity most often manifests itself as one or other of a vast repertoire of representational actions: an exclamation, a description, a raised eyebrow, a shrug, a song, a dance, a movie, a poem. But it need not be the case that we actually produce a representation. We need only be capable of doing so.

So, when Noë says that we have a “visual sense” of the hidden parts of the tomato he is not implying – although he might be mistaken as implying - that we literally have a sensory capacity to detect absence. Not even the Large Hadron Collider at CERN has sensors capable of detecting absence. Absence is undetectable. What we “sense”, or better still, what we are capable of doing when we look at the world is representing the things that we see. Furthermore, we know how to offer representational best guesses about the things that we can’t see. But this knowledge isn’t so much a knowledge of the fullness of things - it is a knowledge of how to represent them in whatever partial or limited form they are encountered. Very often the accuracy of our representational capacities is borne out by further investigation. Sometimes not. Sometimes the back of the tomato turns out to have a bite out of it.