Semiotics Denatured

The theory of the sign (Semiotics) is perhaps most closely associated with the work of the Swiss linguist Ferdinand de Saussure and the American philosopher Charles Sanders Peirce. Over the last century, semiotics has had a very significant influence in numerous fields of research from the arts and literary criticism to biology and cognitive neuroscience. This post is intended to expose what I think is a major flaw in the theory of the sign, a flaw that continues to beleaguer scientific research and philosophical enquiry often in quite far reaching ways.

The Wikipedia entry on signs distinguishes between “natural” signs (or what Peirce called “indexical signs”) and “conventional” signs (“symbols”). In an influential paper from 1955 H. P. Grice makes a similar distinction between what he calls "natural" and "non-natural" meanings. Grice's distinction can therefore be seen in the same light that I aim to shed upon the concept of natural signs.

On the subject of the sign, Wikipedia states: “A natural sign bears a causal relation to its object—for instance, thunder is a sign of storm, or medical symptoms signify a disease.” I hope it is already evident that something isn't quite right about this formulation. Symptoms are caused by disease but they are not signals produced by disease. Likewise, thunder is caused by storms but its influence upon the world is not a consequence of its possible status as a sign. Such a status is not a property of thunder but can only be ascribed to the sound of thunder in much the same way that the function of a tool is assigned to it through use. This is not to suggest that nonverbal creatures cannot be influenced by regularly occurring states of affairs and develop efficacious responses as a consequence. But what I do want to suggest is that Pavlov’s dogs, for example, did not salivate because they interpreted the bell as a sign for dinner but because they had developed an autonomic response to the sound of the bell. Autonomic responses do not function by way of interpretation, unconscious or otherwise. Such a suggestion would undermine the important distinction we typically assume between intentional behaviours (actions) and the many non-conscious processes and responses that support, enable and propagate the vast majority of life on Earth.

Discussing C.S. Peirce's theory of the sign, Noble and Davidson (1996) state: "A mouse rustling in the undergrowth is producing an indexical acoustic sign of itself." If this is true, then every effect would have to be a sign of its cause and the entire universe must be a teeming mass of communicating representations.  According to Semetsky (2005 p.232) this is precisely what Peirce believed: “Everything is a sign: the whole universe, for Peirce, is perfused with signs.” Interestingly, Semetsky also identifies a paradox in Peirce’s thinking since he also claimed that: “nothing is a sign unless it is interpreted as a sign.” Indeed it should be obvious that the universe is only composed of signs to the extent that we sign users are capable of interpreting it as such. So when someone states: "A footprint... can communicate a message." this is either just a handy metaphor or the attribution of communicative agency where none is warranted. Such marks are interpretable by someone capable of extrapolating from them in causal terms, but without a skilled interpreter—moreover a symbol user capable of making meaningful attributions — the marks are merely whatever they are: a cluster of properties. Interpretable things are not communicators, but become interpretable only by being treated as if they are part of practices of use  — most commonly as part of practices of communication. Treating things in this way has significant predictive and retrodictive efficacy, so much so in fact, that we regularly assume (mistakenly of course) that all life must be capable of the same skills of attribution.

An advocate of semiotics might wish to interject here by denying that natural signs are representations at all. This is the move that biosemiotician, Marcello Barbieri (2013) makes when he claims that a natural sign “cannot show or inform, it can only point to an object as if to say: ‘There it is!’” But this is misconceived. What is pointing after all if not a form of showing? Pointing is precisely equivalent to holding something up, presenting it or nodding towards it. Likewise, if nothing is pointed to or shown when we exclaim “There it is!” the utterance is unintelligible. Barbieri continues: “A thermometer and a footprint are natural indices.” The suggestion that a sophisticated instrument of numerical (i.e. symbolic) measurement is a natural sign is simply absurd. But it gets worse. Barbieri claims:

Any metabolic process presupposes a goal directed organism and as such it is a semiotic process, since the organism selects and evaluates environmental stimuli with respect to their adequacy or inadequacy for the purpose of the organism’s survival. (2013)

Barbieri is by no means alone in the attribution of goal directed action to the most simple of forms of life:

The discovery and use of natural signs is a required prerequisite of existence for any living system because they are indispensable to movement, the search for food, regulation, communication, and many other information-related activities. (Sukhoverkhov 2012)

Sukhoverkhov and Barbieri evidently agree that even the most simple organisms necessarily treat the world as if it were composed of signs that they use to direct their behaviour. More astonishingly still, Peirce agrees: “The microscopist looks to see whether the motions of a little creature show any purpose. If so, there is mind there.” On this view then, not only is the universe perfused with signs, but all life is perfused with mind.

The assumption that mind is a prerequisite for any living system should be rejected as both implausible and wildly extravagant. And the conclusion drawn by Peirce on the basis of the movements of a microorganism is in obvious need of revision. The point of error lies in the unjustified assumption that efficacious movements constitute purposeful (i.e. goal directed) behaviour: actions.

If the most simple organisms require minds to survive and propagate then there can be no explanation of how mindedness could ever have evolved. A scientifically parsimonious explanation of the evolution of intelligent life must therefore distinguish between efficacious behaviour on the one hand and its more highly evolved and genuinely purposeful cousin (action) on the other. When a microorganism moves along a food gradient it is not propelled by a goal; it is propelled by the causal influence of the food gradient. This sophisticated but nonetheless predictable behaviour has been honed by millions of years of evolution in which innumerable less well adapted creatures have perished. And whilst this behaviour may resemble action, one thing should be certain: microorganisms are not capable of producing representations of any sort, let alone goals.

There is nothing natural about so called “natural signs”. Nor is interpretation a prerequisite of life on Earth. Sensory discrimination is certainly a prerequisite of all life, but it is certainly not a sign of mind, not even incipient or rudimentary mind.

If we want to understand the evolutionary emergence of mindedness, we first need to be clear about what it takes for a creature to treat an object as if it has properties that it does not actually possess. Such skills are certainly not to be found amongst microorganisms. Mind is born of culture.

In the case of non-linguistic signs there is always the danger that their meanings will seem natural; one must view them with a certain detachment to see that their meanings are in fact the products of a culture, the result of shared assumptions and conventions. But in the case of linguistic signs the conventional or ‘arbitrary’ basis is obvious, and therefore by taking linguistics as a model one may avoid the familiar mistake of assuming that signs which appear natural to those who use them have an intrinsic meaning and require no explanation. (Culler 1975, p.6)

Mute Witnesses

"While photographs may not lie, liars may photograph." Lewis Hine (1909)

“Photography is truth. The cinema is truth twenty-four times per second."” — Jean-Luc Godard

Remarks of this kind are familiar. They suggest that images—and unedited photographs in particular—stand in some special relation to truth: that they testify, bear witness, or in some sense “tell” us how things are.

It is easy enough to speak this way, but less easy to say clearly what it amounts to.

We often describe images using terms borrowed from verbal communication. We say that they “convey meaning”, that they “tell” us something, that they “refer” to, “describe” or even “explain” the world. Photographs are said to document, to testify, to lie, or to reveal the truth. In everyday use, such talk is unproblematic. But it can begin to mislead if taken too literally, as though images function in the same way as sentences.

The difficulty becomes clearer if we attend, not to how we speak about images, but to how they are actually used. A photograph of a cup does not say anything. It does not assert, deny, or qualify. It does not exhibit anything like the structure of a sentence. Yet this does not make it empty or inert. On the contrary, its effectiveness lies precisely in the fact that it does not need to be read in this way. One simply sees what it is of. This recognition is typically immediate. No decoding is required, no translation into words. The question of what an image is about may be open, and often contested, but the question of what it is of is usually resolved without effort. Only after this does the further question—what it is about—begin to arise, and that question is rarely straightforward. We readily distinguish between uncertainty at these two levels. An ambiguous image may prompt disagreement about what it is of, but once that is settled, differences in interpretation do not ordinarily cast it back into doubt. What it shows remains stable, even where what it is taken to mean does not. This difference is important because if it made no difference what an image was of, then it could be replaced without loss by a description. In most cases, it cannot.

Part of what makes this possible is the use of illusionistic effects, with perspectival distortion as a particularly clear example. Images are made so that, under suitable conditions, they resemble what they depict closely enough for recognition to occur directly. Lines, tones, perspective, shading, focus—these are not arbitrary devices, but ways of exploiting regularities in how we see. A drawing can suggest depth; a photograph can stand in for what it shows, not completely, and not without remainder, but sufficiently for it to be taken, in certain respects, for the thing itself.

It is important to note that photographs are not, for the most part, illusions in any strong sense. We do not generally mistake them for the things they depict. But they employ illusionistic means, and under particular conditions—distance, blur, scale, lighting—they can produce effects that approach indistinguishability. The possibility of illusion is built into their operation, even where it is not realised.

Words do not work in this way. Words do not resemble the things they are about. Their use depends upon learned conventions, upon the ability to follow rules and to participate in practices of predication. Images make use of quite different capacities. They engage perception directly, relying not on symbolic substitution, but on visible similarities. For this reason, it is not obvious that images should be treated as bearers of truth or falsehood in the same way as sentences. A statement can be true or false because it asserts something about how things are. An image may depict something that did not occur, or present things in a misleading way, but it does not itself make an assertion. It shows, and what is shown can then be taken up in various ways. This is not to deny that images can be used in acts of deception, or that they can mislead. But the source of that deception lies in how images are used, combined, or interpreted—not in the image taken in isolation. Just as a gesture can be misleading without being a lie, so an image can mislead without asserting anything at all.

Two questions are often asked of an image: what is it of? and what is it about? The first is more basic. Without some answer to it, the second cannot get started. The temptation to reverse this order—to treat images primarily as vehicles of meaning—is understandable, but it risks obscuring the conditions under which they function.

This is perhaps most evident in theoretical discussions, where images are frequently assimilated to models drawn from language. They are described as signs, as messages, as objects to be decoded. Such frameworks can be useful, but they can also encourage the thought that images depend fundamentally on processes of interpretation, in the sense in which words do. What recedes in this picture is the extent to which images rely upon ordinary perceptual capacities—on the ability to recognise, to discriminate, and to respond to visual similarities. These capacities do not depend on linguistic competence, although they can be extended and organised by it.

In this respect, images might better be understood as things used in acts of showing. They make aspects of the world available in a way that can be shared. What follows from that—what they mean, how they are interpreted, what role they play—depends upon the practices in which they are embedded or inserted.

The appeal to truth, then, may be less fundamental than it appears. When we say that an image shows the truth, we are not attributing truth to the image itself so much as situating it within a wider context of use. A photograph may serve as evidence; it may support or undermine a claim. But the truth does not reside in the image as such. To recognise this is not to diminish the importance of images, but to clarify it. Their force lies not in their ability to state or to assert, but in their capacity to present—to place something before us in a way that can be taken up, examined, and discussed.

It is in that taking up, rather than in the image itself, that questions of truth and falsity arise.