But to me, synesthesia is the wrong word to describe overlapping, multimodal sensations. Overlapping means that there are seperable sensations, with different envelopes, that may co-occur. But the way I experience synesthesia is quite different. When I hear a loud, unexpected noise, it brings the sensation of bright color. Smells have texture. Flashing lights have an auditory rhythm. These combinations aren’t the result of exploration over time, and they don’t overflow. They aren’t like a coffee cup which can be seen, smelled, and touched at the same time. Synesthesia is self-contained. Synesthesia is integrated.

It’s pretty clear to me that Merleau-Ponty’s idea of synesthesia and the widely documented neurological condition by the same name are different phenomena. Therefore I think it’s important to use a different word to refer to each. So for the phenomenon that Merleau-Ponty is identifying, I say “intersthesia” is more apt. It better implies overlapping layers of sensation that come and go continuously, in multiple dimensions. With this new word, we can say that we are all intersthetic all the time, and some of us are synesthetic some of the time.

“Yes or no: do we have a body—that is, not a permanent object of thought, but a flesh that suffers when it is wounded, hands that touch?” — The Visible and the Invisible

Merleau-Ponty’s Philosophy by Lawrence Hass was the first full book I read on the great phenomenologist. If you’re fascinated by sensation, perception, synesthesia, metaphor, and flesh (and frankly, who isn’t?), please read it! It offers many wonderful revelations. I’ll briefly review the following topics from the book:

• Sensation/perception is a false dichotomy.
• Perception is “contact with otherness.”
• Synesthesia is a constant feature of experience.
• The concepts of “reversibility” and “flesh”

Sensation/perception is a false dichotomy

In the first section of the book, Hass lays out some pretty convincing evidence that Cartesian duality is patent nonsense. People have been putting down Descartes a lot lately, so it was helpful to get the nuts and bolts of these arguments all in one place.

In the course of this, Hass also shows that, for the same reasons that the mind/body dichotomy is false, so too is the sensation/perception dichotomy. This rocked my world. I was so sold on the idea that sensations were sense impressions which the brain used to build up knowledge of externalities that I didn’t even question who told it to me or why I thought it was true. But this theory quite simply does not survive Merleau-Ponty’s phenomenological analysis. Hass writes, “The pure, atomic sensation—the alleged building-block of complex perception—is not an empirical or experiential object.”

The implications of Merleau-Ponty’s argument are enormous. For one thing, it seems that the empiricist ontology of the sensation has it just backward: sensations do not build up to perceptions. On the contrary, empirical experience shows that a “sensation”… is always experienced amid perceptual complexity, always identified against a larger field… it is the same error that would be made if one cut a loaf of bread into slices and then assumed the loaf was originally composed of just those slices. (l. 509)

So we can define sensation in terms of perception. But if perception comes first, what is it?

Perception is “contact with otherness”

In perception I am opening onto things that transcend me, that go beyond me and my ideas. One aspect of this transcendence is that perception is a field of contact with otherness. I incessantly see and touch and smell coherent, holistic things that are distinct from me: tables, chairs, a beautiful piece of pottery, steaming hot food from the Chinese restaurant… And I open onto other perceivers perceiving me, touching me, and touching the things that I perceive. “Don’t think, look!” Look and see what you live and experience: below the intellectual concept of pure sense-data, basic perception is “knowledge of existences.”… It is not just strange, but contradictory to our knowledge by direct acquaintance to say that this contact, my caress, this sculpted piece you and I perceive together, my body as felt and experienced, your perceiving body as perceived, are “merely in my mind or brain.” Where precisely in my brain are you perceived and touched? Where precisely in my mind or brain is the chair, as I perceive it with my hands, body, and eyes? Fortunately we can be saved from these absurdities by remembering what we know from experience: perception is our opening onto things that are not oneself. Withdraw into our thoughts or imaginations as we might, withdraw into and reify our ideas or theoretical objects as we do, perception is our perpetual deliverance from narcissism. (l. 526)

There is an interesting paradox here which Merleau-Ponty never gets tired of trying to evoke: that perceptual experience is a field of contact with things, but it is a contact with things and a world that opens up, eludes, and limits our explorations. Further, it provides experiences of certain beings who demand a different kind of approach altogether: other perceiving creatures who perceive me and perceive things with me. Carnal contact, coherent things in an open context, radical otherness: once again, these are all experiences that emerge for one’s living body; they are all experiences we have of the world’s transcendence. (l. 910)

The passages above give you a taste of the argument against Descartes, but there’s much more.

Synesthesia is a constant feature of experience

Merleau-Ponty argues that, far from being a strange neurological condition, we all experience synesthesia when we use cross-sensory metaphors.

…Once we overturn the qualia view of sensation and return perceived things to their place in a symbiotic field, then we immediately open up to the inter-sensorial life of perception…. Seeing the coffee cup on the table before me brings with it the possibility of touching it and smelling its contents…. In virtually every experience of things in the world, there is an overlapping of at least some of our sense modalities, just as there is an overlapping of views in binocular vision. And just as binocular vision yields an experience of the world’s depth, the overlapping of sense modalities gives the things we perceive depth, consistency, and density… At this point it should be pretty clear how abstract it becomes to talk about “five separate senses.” This kind of talk derives from the analytic textbook, not life…. A great virtue of Merleau-Ponty’s account of inter-sensorial perception is that it coheres with phenomena we know perfectly well from the flow of life. For instance, it explains why seeing the blue water of the bay calms our bodies, and conversely, how red or pink brings arousal. It explains how one can see the brittleness of glass, the hardness of an airplane blade, or the dryness of linen from a certain fold in it…. Further, it explains why our everyday language is full of expressions where the senses crisscross: we say that sounds are “sharp” or “dull,” that tastes are “thick” or “dense”; we talk about music as “hard” or “soft,” “light” or “heavy.” There is no shortage of examples that demonstrate the overlapping of our sense-modalities: it is as common and constant in living experience as the beating of one’s heart. As as result, we need to embrace the fact that “synaesthesia” is not some weird laboratory condition, but quite the opposite: “Synaesthetic perception is the rule, and we are unaware of it only because scientific knowledge shifts the center of gravity of experience, so that we have unlearned how to see, hear, and generally speaking, feel.” (l. 1037)

This makes me wonder whether the neurological condition called synesthesia may be modeled as a spectrum that varies from person to person and experience to experience. My synesthesia certainly varies in intensity, but I don’t know why or what factors affect it. Other people I know experience strong synesthesia, all the time. Others do not, but they still use synesthetic metaphors. Maybe we are all using our innate capacity for synesthesia when we use these metaphors. In particular, spatial metaphors, because:

Spatial reasoning must have the body of a self at the center.

Merleau-Ponty offers the insight that spatial metaphor always assumes a perceiver at the center:

To be sure, one’s body is not experienced as “beside” things as, say, the chair is next to the couch. On the contrary, the body’s position on the room determines the “beside” relation in the first place… Up, down, beside, behind, inside, outside, left, right, concave, convex, vertical, horizontal, between, above, below: these are all spatial relations that draw their very meaning and intelligibility from my body lived as an organizing whole. (l. 1041)

This has some interesting implications for mathematics, Hass continues, because it indicates that geometry requires an embodied perceiver, and so does not exist before its truths are acquired by someone. Even algebra requires a body in time in order to make use of words such as “next,” “succession,” and “progression.” Rather than being heavenly and perfect, math is fundamentally perceptual.

Reversibility

Reversibility is a core part of Merleau-Ponty’s philosophy. My understanding of the word is that it refers to the feedback loops and singularities we are all familiar with. It’s the failure of causal attribution that’s just a part of life.

There is a relation of my body to itself that makes it the vinculum of the self and things. When my right hand touches my left, I am aware of it as a “physical thing.” But at the same moment, if I wish, an extraordinary event takes place: here is my left hand as well starting to perceive my right…. Thus I touch myself touching; my body accomplishes “a sort of reflection.” In it, through it, there is not just the unidirectional relationship of the one who perceives to what he perceives. The relationship is reversible, the touched hand becomes the touching hand, and I am obliged to say… that the body is a “perceiving thing,” a “subject-object.” (The Philosopher and His Shadow, quoted in Hass, l. 2008)

This passage makes clear how reversibility and synesthesia are closely tied:

We must habituate ourselves to think that every visible is cut out in the tangible, every tactile being in some manner promised to visibility, and that there is encroachment, infringement … between the tangible and the visible…. It is a marvel too little noticed that … every displacement of my body … has its place in the same visible universe that I itemize and explore with [my hands], as conversely, every vision takes place somewhere in tactile space. There is double and crossed situation of the visible in the tangible and of the tangible in the visible; the two maps are complete, and yet they do not merge into one. The two parts are total parts and yet are not superposable. (The Visible and the Invisible, quoted in Hass, l. 2031)

Flesh

Flesh is portrayed as one of the most difficult but crucial ideas in Merleau-Ponty’s writings. Hass devotes considerable effort to a close reading on flesh. It caught my eye because I really enjoyed the book Philosophy in the Flesh, and I understand now that nobody can use that word in the title of a philosophy book without alluding to Merleau-Ponty. Here are a couple of especially inspiring passages about flesh:

We can denounce [the living body] as the “prison of the soul” or the principle of sin. We can insist that our analyses exhaust it. We can dismiss it from the penumbra of “legitimate” philosophy as “not intellectual enough” or “touchy-feely.” We can, in our everyday lives, ignore, forget, or overlook it, and as we saw, these tendencies are supported by the outward structure of the body itself. Yet this carnality is what we live, and as Merleau-Ponty’s philosophy shows, we are in fact able to know it and articulate its features so that whole sectors of our experience become intelligible….To be sure, one’s living-conscious body is not a machine… It is, as Merleau-Ponty puts it in his final writings, neither substance nor matter nor machine, but flesh. (l. 1384)

Through our behaviors, we become paired—self and other. More precisely, since you “prolong” my intentions through your projective behavior, since you caress the things with hands not unlike mine, since you look at me with gleaming eyes, I extend my self to you. My corporeal schema and perceptual powers reflexively transfer; your resonant behavior means that you are “Another.” Indeed, Merleau-Ponty shows that our knowledge of other minds or selves is rooted in this experiential coupling of flesh-to-flesh that is lived through our behavior. (l. 3816)

Finally, a quote from Merleau-Ponty that I think both James Gibson and Alva Noë would approve of:

…Body consciousness is “in the first place not a matter of ‘I think that’ but of ‘I can’”. (l. 1290)

2.013
Each thing is, as it were, in a space of possible states of affairs. This space I can imagine empty, but I cannot imagine the thing without the space.
2.0131
A spatial object must be situated in infinite space. (A spatial point is an argument-place.)
A speck in the visual field, though it need not be red, must have some colour: it is, so to speak, surrounded by colour-space. Notes must have some pitch, objects of the sense of touch some degree of hardness, and so on.

This is a prime example of a sensory metaphor used throughout the book. Objects are described in a visual way, as being seen as situated within a possibility space or belief space. They themselves have extension, but we perceive them as taking up some amount of the visual field (surrounded by context, which here is represented as other possibilities for their position or physical attributes).

2.151
Pictorial form is the possibility that things are related to one another in the same way as the elements of the picture.
2.1511
That is how a picture is attached to reality; it reaches right out to it.
2.1512
It is laid against reality like a measure.
2.15121
Only the end-points of the graduating lines actually touch the object that is to be measured.

Again we get a visual metaphor described in terms of physicality. How is a picture “attached” to reality? It reaches out to it. And while it touches reality, it only just touches it, in a tangential way. Wittgenstein starts with a visual image and then writes “attached”, “reaches out”, “laid against”, and “touch”—all haptic metaphors.

2.1514
The pictorial relationship consists of the correlations of the picture’s elements with things.
2.1515
These correlations are, as it were, the feelers of the picture’s elements, with which the picture touches reality.

Now we have moved from the picture as a “measure,” a passive geometry tool, to a picture as an agent. Not just any agent, but an agent with a capacity for haptic perception. What is a “feeler”? To me that word means a mobile extremity with sense organs which can be used to find out about the world. So, what Wittgenstein seems to be saying here is that when we generate a picture in our mind, it’s as if we are extending our hand into the world.

4.002
…
Language disguises thought. So much so, that from the outward form of the clothing it is impossible to infer the form of the thought beneath it, because the outward form of the clothing is not designed to reveal the form of the body, but for entirely different purposes.

In other words, thoughts are like physical objects. A word envelops a thought. We have a thought and then we toss a word-robe over it and shove it onto the stage of discourse where it can interface with other enrobed thoughts.

4.411
It immediately strikes one as probable that the introduction of elementary propositions provides the basis for understanding all other kinds of proposition. Indeed the understanding of general propositions palpably depends on the understanding of elementary propositions.

Once again, a tactile metaphor (“palpably”) is used for emphasis and to indicate comprehensive understanding.

5.557
…
What belongs to its application, logic cannot anticipate.
It is clear that logic must not clash with its application.
But logic has to be in contact with its application.
Therefore, logic and its application must not overlap.

I.e.,

Is “overlap” a haptic metaphor or a visual one? It could be either, or both.

6.432
How things are in the world is a matter of complete indifference for what is higher. God does not reveal himself in the world.
6.4321
The facts all contribute only to setting the problem, not to its solution.
6.44
It is not how things are in the world that is mystical, but that it exists.
6.45
To view the world sub specie aeterni is to view it as a whole—a limited whole.
Feeling the world as a limited whole—it is this that is mystical.

To feel is to know, silently, mystically.

I was pretty surprised at how easy it seems to foresee Wittgenstein’s turn from the eye to the hand. He presents what he calls a picture theory of language, but it repeatedly leads to a description of solid objects in space, or bodies moving and feeling and contacting each other. Of course I’m reading with a very biased perspective. Not only is my goal to hunt for tactile metaphors but I also know how the story ends some 30 years later. Still, I can’t help but feel that the tactile language in the Tractatus may foreshadow the shift to come.

I just finished Action In Perception by Alva Noë. It’s a very readable introduction to the enactive view of perceptual consciousness, which argues that perception neither happens in us nor to us; rather, it’s something we do with our bodies, situated in the physical world, over time. Our knowledge of the way in which sensory stimulation varies as we control our bodies is what brings experience about. Without sensorimotor skill, a stimulus cannot constitute a percept. Noë presents empirical evidence for his claim, drawing on the phenomenology of change blindness as well as tactile vision substitution systems. I highly recommend the book.

The emphasis on embodied experience leads to the use of touch as a model for perception, rather than the traditional vision-based approach. Here’s an excerpt:

Touch acquires spatial content—comes to represent spatial qualities—as a result of the ways touch is linked to movement and to our implicit understanding of the relevant tactile-motor dependencies governing our interaction with objects. [Philosopher George Berkeley] is right that touch is, in fact, a kind of movement. When a blind person explores a room by walking about in it and probing with his or her hands, he or she is perceiving by touch. Crucially, it is not only the use of the hands, but also the movement in and through the space in which the tactile activity consists. Very fine movements of the fingers and very gross wanderings across a landscape can each constitute excercises of the sense of touch. Touch, in all such cases, is movement. (At the very least, it is movement of something relative to the perceiver.) These Berkeleyan ideas form a theme, more recently, in the work of [Brian O'Shaughnessy's book "Consciousness and World"]. He writes: “touch is in a certain respect the most important and certainly the most primordial of the senses. The reason is, that it is scarely to be distinguished from the having of a body that can act in physical space”…

But why hold that touch is the only active sense modality? As we have stressed, the visual world is not given all at once, as in a picture. The presence of detail consists not in its representation now in consciousness, but in our implicit knowledge now that we can represent it in consciousness if we want, by moving the eyes or by turning the head. Our perceptual contact with the world consists, in large part, in our access to the world thanks to our possession of sensorimotor knowledge.

Here, no less than in the case of touch, spatial properties are available due to links to movement. In the domain of vision, as in that of touch, spatial properties present themselves to us as “permanent possibilities of movement.” As you move around the rectangular object, its visible profile deforms and transforms itself. These deformations and transformations are reversible. Moreover, the rules governing the transformation are familiar, at least to someone who has learned the relevant laws of visuomotor contingency. How the item looks varies systematically as a function of your movements. Your experience of it as cubical consists in your implicit understanding of the fact that the relevant regularity is being observed.

Virtual and augmented reality interface design practices have already begun to demonstrate these concepts. Head mounted augmented reality displays sense the user’s eye and body movements to construct virtual percepts. Head related transfer functions (HRTFs) synthesize sound as it would be heard by an organism with certain physical characteristics, in a particular environment. It seems to me that an important implication for enactive interface design is that haptic sensory patterns can lead to perceptual experience in all sensory modes (vision, hearing, touch). Thus, all human-computer interaction/user experience can be viewed in a haptic context.