Semiotics Denatured

The theory of the sign (Semiotics) is perhaps most closely associated with the work of the Swiss linguist Ferdinand de Saussure and the American philosopher Charles Sanders Peirce. Over the last century, semiotics has had a very significant influence in numerous fields of research from the arts and literary criticism to biology and cognitive neuroscience. This post is intended to expose what I think is a major flaw in the theory of the sign, a flaw that continues to beleaguer scientific research and philosophical enquiry often in quite far reaching ways.

The Wikipedia entry on signs distinguishes between “natural” signs (or what Peirce called “indexical signs”) and “conventional” signs (“symbols”). In an influential paper from 1955 H. P. Grice makes a similar distinction between what he calls "natural" and "non-natural" meanings. Grice's distinction can therefore be seen in the same light that I aim to shed upon the concept of natural signs.

On the subject of the sign, Wikipedia states: “A natural sign bears a causal relation to its object—for instance, thunder is a sign of storm, or medical symptoms signify a disease.” I hope it is already evident that something isn't quite right about this formulation. Symptoms are caused by disease but they are not signals produced by disease. Likewise, thunder is caused by storms but its influence upon the world is not a consequence of its possible status as a sign. Such a status is not a property of thunder but can only be ascribed to the sound of thunder in much the same way that the function of a tool is assigned to it through use. This is not to suggest that nonverbal creatures cannot be influenced by regularly occurring states of affairs and develop efficacious responses as a consequence. But what I do want to suggest is that Pavlov’s dogs, for example, did not salivate because they interpreted the bell as a sign for dinner but because they had developed an autonomic response to the sound of the bell. Autonomic responses do not function by way of interpretation, unconscious or otherwise. Such a suggestion would undermine the important distinction we typically assume between intentional behaviours (actions) and the many non-conscious processes and responses that support, enable and propagate the vast majority of life on Earth.

Discussing C.S. Peirce's theory of the sign, Noble and Davidson (1996) state: "A mouse rustling in the undergrowth is producing an indexical acoustic sign of itself." If this is true, then every effect would have to be a sign of its cause and the entire universe must be a teeming mass of communicating representations.  According to Semetsky (2005 p.232) this is precisely what Peirce believed: “Everything is a sign: the whole universe, for Peirce, is perfused with signs.” Interestingly, Semetsky also identifies a paradox in Peirce’s thinking since he also claimed that: “nothing is a sign unless it is interpreted as a sign.” Indeed it should be obvious that the universe is only composed of signs to the extent that we sign users are capable of interpreting it as such. So when someone states: "A footprint... can communicate a message." this is either just a handy metaphor or the attribution of communicative agency where none is warranted. Such marks are interpretable by someone capable of extrapolating from them in causal terms, but without a skilled interpreter—moreover a symbol user capable of making meaningful attributions — the marks are merely whatever they are: a cluster of properties. Interpretable things are not communicators, but become interpretable only by being treated as if they are part of practices of use  — most commonly as part of practices of communication. Treating things in this way has significant predictive and retrodictive efficacy, so much so in fact, that we regularly assume (mistakenly of course) that all life must be capable of the same skills of attribution.

An advocate of semiotics might wish to interject here by denying that natural signs are representations at all. This is the move that biosemiotician, Marcello Barbieri (2013) makes when he claims that a natural sign “cannot show or inform, it can only point to an object as if to say: ‘There it is!’” But this is misconceived. What is pointing after all if not a form of showing? Pointing is precisely equivalent to holding something up, presenting it or nodding towards it. Likewise, if nothing is pointed to or shown when we exclaim “There it is!” the utterance is unintelligible. Barbieri continues: “A thermometer and a footprint are natural indices.” The suggestion that a sophisticated instrument of numerical (i.e. symbolic) measurement is a natural sign is simply absurd. But it gets worse. Barbieri claims:

Any metabolic process presupposes a goal directed organism and as such it is a semiotic process, since the organism selects and evaluates environmental stimuli with respect to their adequacy or inadequacy for the purpose of the organism’s survival. (2013)

Barbieri is by no means alone in the attribution of goal directed action to the most simple of forms of life:

The discovery and use of natural signs is a required prerequisite of existence for any living system because they are indispensable to movement, the search for food, regulation, communication, and many other information-related activities. (Sukhoverkhov 2012)

Sukhoverkhov and Barbieri evidently agree that even the most simple organisms necessarily treat the world as if it were composed of signs that they use to direct their behaviour. More astonishingly still, Peirce agrees: “The microscopist looks to see whether the motions of a little creature show any purpose. If so, there is mind there.” On this view then, not only is the universe perfused with signs, but all life is perfused with mind.

The assumption that mind is a prerequisite for any living system should be rejected as both implausible and wildly extravagant. And the conclusion drawn by Peirce on the basis of the movements of a microorganism is in obvious need of revision. The point of error lies in the unjustified assumption that efficacious movements constitute purposeful (i.e. goal directed) behaviour: actions.

If the most simple organisms require minds to survive and propagate then there can be no explanation of how mindedness could ever have evolved. A scientifically parsimonious explanation of the evolution of intelligent life must therefore distinguish between efficacious behaviour on the one hand and its more highly evolved and genuinely purposeful cousin (action) on the other. When a microorganism moves along a food gradient it is not propelled by a goal; it is propelled by the causal influence of the food gradient. This sophisticated but nonetheless predictable behaviour has been honed by millions of years of evolution in which innumerable less well adapted creatures have perished. And whilst this behaviour may resemble action, one thing should be certain: microorganisms are not capable of producing representations of any sort, let alone goals.

There is nothing natural about so called “natural signs”. Nor is interpretation a prerequisite of life on Earth. Sensory discrimination is certainly a prerequisite of all life, but it is certainly not a sign of mind, not even incipient or rudimentary mind.

If we want to understand the evolutionary emergence of mindedness, we first need to be clear about what it takes for a creature to treat an object as if it has properties that it does not actually possess. Such skills are certainly not to be found amongst microorganisms. Mind is born of culture.

In the case of non-linguistic signs there is always the danger that their meanings will seem natural; one must view them with a certain detachment to see that their meanings are in fact the products of a culture, the result of shared assumptions and conventions. But in the case of linguistic signs the conventional or ‘arbitrary’ basis is obvious, and therefore by taking linguistics as a model one may avoid the familiar mistake of assuming that signs which appear natural to those who use them have an intrinsic meaning and require no explanation. (Culler 1975, p.6)

Mute Witnesses

"While photographs may not lie, liars may photograph." Lewis Hine (1909)

A few days ago I presented a paper at a conference at the University of California Berkeley on the subject to the image. One of the other speakers gave a presentation beginning with the above quote from the early 20th Century social documentary photographer Lewis Hine. The quote reminded me of Picasso's famous remark: "“Art is a lie that makes us realise the truth.” Like Picasso and Hein, many people hold the view that images — photographs in particular — are truth bearers, that they provide meaningful testimony and have what philosophers sometimes call "factive", as opposed to fictive, status. I aim to explain why such talk about images has the effect of misleadingly reducing them to linguistic tokens. Furthermore, doing so overlooks, misunderstands or worse still ignores, the essentially mute but nonetheless powerful effectiveness of images as substitutes for the things they represent.

As any linguist will confirm, all well formed sentences contain a subject and a predicate. Language is thus a system of procedures by which we ascribe attributes to things through the use of arbitrary symbols. Only the most intelligent creatures can do this because only the most intelligent creatures are capable of following the rules necessary to engage in practices of predication: of the socially negotiated attribution of abstract linguistic tokens to objects and states of affairs.

It should be clear to everyone that images are not linguistic entities, yet quite evidently it is not in the least clear. Almost all theories of representation refer to images as "signs" or "signifiers", as "readable" objects or "messages" that require "decoding", "deciphering" or "interpreting." In everyday use, we talk of how images "convey meaning", "have content" and are "about" the things to which they "refer." We also talk of what images "tell" us, what they "describe", "articulate", "suggest", "explain" and "imply."  And it is not impossible to find reference to images as oracles and chronicles or soothsayers or that they predict the future, commentate on the present and narrate the past. It might help to exemplify the absurdity of such thinking by noting that we can say exactly the same of tea leaves or the lines on one's hand. That we can do so, reveals far more about our infatuation with language than it does about the nature of images or the susceptibilities and skills that enable their use.

Any student wishing to understand the question of how images actually work (this was the title of my presentation at the conference by the way) will be met by an impenetrable thicket of confused and over complex theorisation about these profoundly simple but powerful tools. They will have to assimilate and understand numerous technical terms like "denotation", "connotation", "punctum", "studium", "icon", "index", "symbol", "sign", "referent", "veridicality", "verisimilitude" etc. And with each step along this path they will be no closer to the answer they seek. In fact, with each step, they will descend deeper into a convoluted labyrinth from which there is little hope of return.

Depictive images work because they can be mistaken for the things they represent in certain ways and in certain respects. It is as simple as that. There are ways to make images resemble the things they depict because there are ways and respects in which they can be made more or less indiscriminable from them, ways that fully exploit the potential for illusion. You simply cannot do this with words — words do not look anything like the things they stand in for.

So when we say that images "tell" "truths" or "lies" we ignore their essential nature and instead treat them as linguistic items. In ordinary usage this is fine, but strictly speaking (which is what we should require of all serious theories) lying and telling truths are the exclusive preserve of language users. Of course, images can depict things that never did, could or will ever happen. But nonverbal misrepresentation does not reduce to verbal misrepresentation: to lying. Images are not texts and the skills necessary to use them for communicative purposes are by no means reliant upon (although they are massively assisted by) our skills as language users.

There are two fundamental questions we can ask of any image: "What is it of?" and "What is it about?" The first is always more basic than the second because the second relies to a very significant degree on the first. If it were not a matter of some importance what images are actually of, then we could indeed replace them with abstractions, with symbolic tokens, with words. We can do this of course, but not without significant loss.

Recognising what an image is of, is usually effortless, whereas the answer to the question of what an image is about — what it means — is almost never so. In fact the answer to the question of meaning is about as straightforward as the answer to the question of the function of a length of string. If you do not know how to use a length of string, then it has no function. The same is true of meaning.

Images can neither lie nor tell the truth. They can be used in acts of lying and they can be used to corroborate truths, but just as a nonverbal human witness can point to the perpetrator of a crime with no recourse to language, so too do images gain their fundamental efficacy from factors that are entirely independent of linguistic competence. Images can be deceptive but they cannot deceive. They can mislead and misguide but they cannot cheat. They can be clear but they cannot be honest.  They can distort but they cannot feign. They can simulate but they cannot pretend.

Images are powerful because they trigger many of the same embodied responses as the things they represent — just as words do in fact. But, unlike language, they do not require elaborate skills in symbolic substitution and rule following to do this. So it is simply mistaken to suggest or conclude that images are bearers of truth, tellers of tales or descriptions of the world. If someone shows you a view through a window, they are not showing you a lie and nor are they showing you the truth. Likewise, a view of the moon through the distorting lens of a telescope is neither factive nor fictive. When we present evidence of the truth, the evidence does not constitute the truth. Truth is not something that can be perceived. When we say "I see the truth" we do not mean to suggest that the truth is something that can be seen. We mean that the truth is something that can be understood.

During the conference, another of the presenters mentioned something that struck me as relevant to this analysis. Apparently the root of the word "epiphany" is to be found in the Ancient Greek term: phanein, meaning "to show." Images are used in acts of showing. It is what we do with images, and more specifically, the communicative practices within which images are integrated, that transforms them into such extraordinary and useful tools. Language enables us to use images in extraordinarily sophisticated ways, but language also significantly obscures our understanding of these essentially mute witnesses.