Whatever knowledge we may have of the world is rooted in the changing states of our own bodies. As John Pollock puts it in his refutation of externalism: “Cognition can make use of any states to which it has direct access, but those are just the internal states” (Pollock and Cruz 1999 pp130-142). The beginnings of information about the environment are patterns of neural excitation in the special sensory probes — the retina at the back of the eye, the cochlea in the inner ear, the semicircular canals of the vestibule (also in the inner ear), the olfactory nerve endings in the nasal mucosae, the gustatory papillae in the back of the tongue, and the nerve terminals in the superficial layers of the skin. When an agent sees his Aunt Maud, the pattern of excitation in the retina bears a topological resemblance to the stimulus object (Aunt Maud). For example, the parts of the retinal image that correspond to Aunt Maud’s arms will be on either side of the part that corresponds to her torso, rather than being randomly distributed about the field of vision. Thus the earliest impression formed in a sensory probe is what C.S. Peirce called an iconic representation, a representation that resembles the stimulus object in much the way that a painting of a landscape resembles the landscape — the representation is reduced and simplified, detail has been omitted, but parts that are close together in the stimulus object are close together in the representation.
The nerve terminals in the special sensory probes send these patterns to circumscribed entry points in the brain — the early sensory cortices — where the iconic representations form topographically organised patterns of activity (Damasio 1994 p91, Damasio 2003 p197). Iconic representations form the raw material for two further processes. On the one hand, iconic representations are transformed into categorical representations, with which symbolic representations are linked. On the other hand certain changes are made on the basis of the iconic representations to a set of body maps, with consequences described in the next section.
There are two main explanations for the way in which iconic representations are transformed to categorical representations, and neither has yet been fully elucidated at the neural level. The conceptual spaces approach (Gärdenfors 2000) begins with quality dimensions such as temperature, weight, brightness, and pitch, corresponding to different ways in which an agent can judge stimuli to be similar or different. These dimensions form clusters in the sense that, for example, a stimulus object cannot have a hue without also having a brightness, nor can a sound have a pitch without at the same time having a loudness. Such clusters are called domains. A conceptual space is formed of one or more domains. A property such as “is red” corresponds to a convex region in a conceptual space. The transformation of iconic representations into categorical representations results from the partitioning of the conceptual space into convex regions by means of a Voronoi tessellation, which has a discretising effect and is compatible with Rosch’s prototype theory of categorisation (Rosch 1978). Roughly speaking, Voronoi tessellations are produced as follows. Each prototypical point in space determines a convex region by annexing for its region all points that lie closer to it than to any other prototype. Finally, the discretised categorical representation is linked with a symbolic representation: “a symbol just summarizes the information contained in a region of a domain of a conceptual space by refering [sic] to the prototypical element of the region” (Gärdenfors 2000 p258).
The alternative account of the iconic-to-categorical process relies on accomplishing an analog-to-discrete transformation in a manner that produces the phenomenon called categorical perception (Harnad 1987). Again the starting point is an iconic representation, described as “being an analog of the sensory input (more specifically, of the proximal projection of the distal stimulus object on the device’s transducer surfaces)”, and these analogs “faithfully preserve the iconic character of the input for such purposes as same-different judgments, stimulus-matching, and copying” (Harnad 1987 pp551/2). Some form of filtering that picks out invariant features now produces a categorical representation. The filtering mechanism may be any of several candidates that have been proposed, and a constraint is that the filter will have to produce categorical perception (at least in categories of colours and phonemes). Categorical perception is the name given to the phenomenon that “for certain perceptual categories, within-category differences look much smaller than between-category differences even when they are the same size physically. For example, on color perception, differences between reds and between yellows look much smaller than equal-sized differences that cross the red/yellow boundary; the same is true of the phoneme categories /ba/ and /da/. Indeed, the effect of the category boundary is not merely quantitative, but qualitative” (Harnad 1987 p535). Categorical perception too is compatible with Rosch’s prototype theory of categorisation. Finally, categorical representations are linked to symbolic representations: labels associated with categorical representations “provide the elementary terms for a third representational system, the symbolic descriptions of natural language” (Harnad 1987 p22).
Neurobiologically speaking, categorical representations are situated not in the early sensory cortices that house the topographically organised (iconic) representations but in the prefrontal sectors (Damasio 1994 p183). While the entire prefrontal region seems involved in categorising contingencies that involve personal relevance (the life experience of the organism), the bioregulatory and social domains appear to be aligned with the ventromedial sector, while knowledge of an ‘objective’ nature (mathematics, language, knowledge of the actions of objects in space-time) is aligned with the dorsolateral region. Categorical representations are special cases of what Damasio calls dispositional representations, “because what they do, quite literally, is order other neural patterns about, make neural activity happen elsewhere”. Dispositional representations generally occupy convergence zones located throughout the higher-order association cortices in occipital, temporal, parietal, and frontal regions, and in basal ganglia and limbic structures.
Among the processes that dispositional representations control is the activation of those iconic representations (topographically organised and situated in the early sensory cortices) that become images in the mind. Dispositional representations can activate more than one representation at a time, holding them in working memory so that they may be attended, side by side or in rapid alternation (Damasio 1994 pp240/3). This is how symbols are linked to categorical representations — symbolic representations are just special iconic representations, which may be activated as images (visual or auditory) together with an appropriate categorical representation, by acquired dispositional representations. Thus the symbol string ‘cat’ is a visual image (iconic representation) coupled in our minds with a representation of the category of cats, and the coupling is effected by a third kind of representation, a dispositional representation. Clearly, our excursion into neurobiology draws into serious question Dennett’s assumption that the brain processes only symbols.
A human brain does process symbols, of course, and this may indeed occur in the context of deduction. But an agent performing a deductive inference would normally (unless the situation were deliberately artificial) be dealing with symbolic representations that are grounded, i.e. associated with categorical representations that themselves rest upon multiple iconic representations (Harnad 1994 p386). The grounding provides the symbols with meaning, with semantics. This is in stark contrast to the way in which automated reasoning programs manipulate meaningless (to them) strings of symbols purely in accordance with syntactic rules, and to the behaviour of the man in the Chinese Room, who understands the rules written in English on the walls of the room but not the Chinese symbols that the rules mention. When humans reason deductively, they have at their disposal not only the rules of inference but also the meanings —metaphorically, human deduction involves not just the poor fellow in the Chinese Room but also the Chinese speaker outside.
It is important to realise that symbol manipulations are pointless unless meaning can be attached in some way, and that being able to interpret symbol manipulations, not just locally but globally, is not a trivial criterion — “It is easy to pick a bunch of arbitrary symbols and to formulate arbitrary yet systematic rules for manipulating them, but this does not guarantee that there will be any way to interpret it all so as to make sense” (Harnad 1994). All rule-based symbol manipulation acquires its significance from semantics. Gödel’s incompleteness theorem too contrasts deductive inference with the semantic process of moving from object-level to meta-level, i.e. moving from symbols to a universe of sets within which the meanings of symbols reside as particular sets, and where it can be verified essentially by inspection that the Gödel sentence (asserting, say, that Peano arithmetic is consistent) is true since the set of natural numbers satisfies the Peano axioms (Enderton 2001 pp266-270).
We claim, therefore, that human deduction itself is intrinsically more than the syntactic manipulation Symbolic AI would seek to simulate it by, and that human cognition involves much more than mere deduction. The grounding of a symbol, specifically its association with a categorical representation, provides in principle for a notion of relevance — after all, what is a category but a grouping together of items that share a family resemblance? From this perspective it is the lack of symbol-grounding, i.e. the lack of semantic involvement in the inferential process, that bedevils automated reasoning systems and that provides the Chinese Room thought experiment with its force. The Strong AI Hypothesis may turn out to be true, but this will not be achieved by Symbolic AI. A growing awareness of this fact has led to the emergence of an alternative to Symbolic AI, namely Embodied AI (Clancey 1997; Artificial Intelligence 149(1) 2003). As we shall see, embodiment has become important in psychology also. Let us explore the role it plays in human cognition.