Image Image Image Image Image

Presented at CHI 2012, Touché is a capacitive system for pervasive, continuous sensing. Among other amazing capabilities, it can accurately sense gestures a user makes on his own body. “It is conceivable that one day mobile devices could have no screens or buttons, and rely exclusively on the body as the input surface.” Touché.

Noticing that many of the same sensors, silicon, and batteries used in smartphones are being used to create smarter artificial limbs, Fast Company draws the conclusion that the market for smartphones is driving technology development useful for bionics. While interesting enough, the article doesn’t continue to the next logical and far more interesting possibility: that phones themselves are becoming parts of our bodies. To what extent are smartphones already bionic organs, and how could we tell if they were? I’m actively researching design in this area – stay tuned for more about the body-incorporated phone.

A study provides evidence that talking into a person’s right ear can affect behavior more effectively than talking into the left.

One of the best known asymmetries in humans is the right ear dominance for listening to verbal stimuli, which is believed to reflect the brain’s left hemisphere superiority for processing verbal information.

I heavily prefer my left ear for phone calls. So much so that I have trouble understanding people on the phone when I use my right ear. Should I be concerned that my brain seems to be inverted?

Read on and it becomes clear that going beyond perceptual psychology, the scientists are terrifically shrewd:

Tommasi and Marzoli’s three studies specifically observed ear preference during social interactions in noisy night club environments. In the first study, 286 clubbers were observed while they were talking, with loud music in the background. In total, 72 percent of interactions occurred on the right side of the listener. These results are consistent with the right ear preference found in both laboratory studies and questionnaires and they demonstrate that the side bias is spontaneously displayed outside the laboratory.

In the second study, the researchers approached 160 clubbers and mumbled an inaudible, meaningless utterance and waited for the subjects to turn their head and offer either their left of their right ear. They then asked them for a cigarette. Overall, 58 percent offered their right ear for listening and 42 percent their left. Only women showed a consistent right-ear preference. In this study, there was no link between the number of cigarettes obtained and the ear receiving the request.

In the third study, the researchers intentionally addressed 176 clubbers in either their right or their left ear when asking for a cigarette. They obtained significantly more cigarettes when they spoke to the clubbers’ right ear compared with their left.

I’m picturing the scientists using their grant money to pay cover at dance clubs and try to obtain as many cigarettes as possible – carefully collecting, then smoking, their data – with the added bonus that their experiment happens to require striking up conversation with clubbers of the opposite sex who are dancing alone. One assumes that, if the test subject happened to be attractive, once the cigarette was obtained (or not) the subject was invited out onto the terrace so the scientist could explain the experiment and his interesting line of work. Well played!

Another MRI study, this time investigating how we learn parts of speech:

The test consisted of working out the meaning of a new term based on the context provided in two sentences. For example, in the phrase “The girl got a jat for Christmas” and “The best man was so nervous he forgot the jat,” the noun jat means “ring.” Similarly, with “The student is nising noodles for breakfast” and “The man nised a delicious meal for her” the hidden verb is “cook.”

“This task simulates, at an experimental level, how we acquire part of our vocabulary over the course of our lives, by discovering the meaning of new words in written contexts,” explains Rodríguez-Fornells. “This kind of vocabulary acquisition based on verbal contexts is one of the most important mechanisms for learning new words during childhood and later as adults, because we are constantly learning new terms.”

The participants had to learn 80 new nouns and 80 new verbs. By doing this, the brain imaging showed that new nouns primarily activate the left fusiform gyrus (the underside of the temporal lobe associated with visual and object processing), while the new verbs activated part of the left posterior medial temporal gyrus (associated with semantic and conceptual aspects) and the left inferior frontal gyrus (involved in processing grammar).

This last bit was unexpected, at first. I would have guessed that verbs would be learned in regions of the brain associated with motor action. But according to this study, verbs seem to be learned only as grammatical concepts. In other words, knowledge of what it means to run is quite different than knowing how to run. Which makes sense if verb meaning is accessed by representational memory rather than declarative memory.

Researchers at the University of Tampere in Finland found that,

Interfaces that vibrate soon after we click a virtual button (on the order of tens of milliseconds) and whose vibrations have short durations are preferred. This combination simulates a button with a “light touch” – one that depresses right after we touch it and offers little resistance.

Users also liked virtual buttons that vibrated after a longer delay and then for a longer subsequent duration. These buttons behaved like ones that require more force to depress.

This is very interesting. When we think of multimodal feedback needing to make cognitive sense, synchronization first comes to mind. But there are many more synesthesias in our experience that can only be uncovered through careful reflection. To make an interface feel real, we must first examine reality.

Researchers at the Army Research Office developed a vibrating belt with eight mini actuators — “tactors” — that signify all the cardinal directions. The belt is hooked up to a GPS navigation system, a digital compass and an accelerometer, so the system knows which way a soldier is headed even if he’s lying on his side or on his back.

The tactors vibrate at 250 hertz, which equates to a gentle nudge around the middle. Researchers developed a sort of tactile morse code to signify each direction, helping a soldier determine which way to go, New Scientist explains. A soldier moving in the right direction will feel the proper pattern across the front of his torso. A buzz from the front, side and back tactors means “halt,” a pulsating movement from back to front means “move out,” and so on.

A t-shirt design by Derek Eads.

Recent research reveals some fun facts about aural-tactile synesthesia:

Both hearing and touch, the scientists pointed out, rely on nerves set atwitter by vibration. A cell phone set to vibrate can be sensed by the skin of the hand, and the phone’s ring tone generates sound waves — vibrations of air — that move the eardrum…

A vibration that has a higher or lower frequency than a sound… tends to skew pitch perception up or down. Sounds can also bias whether a vibration is perceived.

The ability of skin and ears to confuse each other also extends to volume… A car radio may sound louder to a driver than his passengers because of the shaking of the steering wheel. “As you make a vibration more intense, what people hear seems louder,” says Yau. Sound, on the other hand, doesn’t seem to change how intense vibrations feel.

Max Mathews, electronic music pioneer, has died.

Though computer music is at the edge of the avant-garde today, its roots go back to 1957, when Mathews wrote the first version of “Music,” a program that allowed an IBM 704 mainframe computer to play a 17-second composition. He quickly realized, as he put it in a 1963 article in Science, “There are no theoretical limits to the performance of the computer as a source of musical sounds.”

Rest in peace, Max.

UPDATE: I haven’t updated this blog in a while, and I realized after posting this that my previous post was about the 2010 Modulations concert. Max Mathews played at Modulations too, and that was the last time I saw him.

I finally got around to recording and mastering the set I played at the CCRMA Modulations show a few months back. Though I’ve been a drum and bass fan for many years, this year’s Modulations was the first time I’d mixed it for others. Hope you like it!

Modulations 2010
Drum & Bass | 40:00 | May 2010

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Download (mp3, 82.7 MB)


1. Excision — System Check
2. Randomer — Synth Geek
3. Noisia — Deception
4. Bassnectar — Teleport Massive (Bassnectar Remix)
5. Moving Fusion, Shimon, Ant Miles — Underbelly
6. Brookes Brothers — Crackdown
7. The Ian Carey Project — Get Shaky (Matrix & Futurebound’s Nip & Tuck Mix)
8. Netsky — Eyes Closed
9. Camo & Krooked — Time Is Ticking Away feat. Shaz Sparks

Over the last few days this video has been so much bombshell to many of my music-prone friends.

It’s called the Multi-Touch Light Table and it was created by East Bay-based artist/fidget-house DJ Gregory Kaufman. The video is beautifully put together, highlighting the importance of presentation when documenting new ideas.

I really like some of the interaction ideas presented in the video. Others, I’m not so sure about. But that’s all right: the significance of the MTLT is that it’s the first surface-based DJ tool that systematically accounts for the needs of an expert user.

Interestingly, even though it looks futuristic and expensive to us, interfaces like this will eventually be the most accessible artistic tools. Once multi-touch surface are ubiquitous, the easiest way to gain some capability will be to use inexpensive or open-source software. The physical interfaces created for DJing, such as Technics 1200s, are prosthetic objects (as are musical instruments), and will remain more expensive because mechanical contraptions will always be. Now, that isn’t to say that in the future our interfaces won’t evolve to become digital, networked, and multi-touch sensitive, or even that their physicality will be replaced with a digital haptic display. But one of the initial draws of the MTLT—the fact of its perfectly flat, clean interactive surface—seems exotic to us right now, and in the near future it will be default.

Check out this flexible interface called impress. Flexible displays just look so organic and, well impressive. One day these kinds of surface materials will become viable displays and they’ll mark a milestone in touch computing.

It’s natural to stop dancing between songs. The beat changes, the sub-rhythms reorient themselves, a new hook is presented and a new statement is made. But stopping dancing between songs is undesirable. We wish to lose ourselves in as many consecutive moments as possible. The art of mixing music is to fulfill our desire to dance along to continuous excellent music, uninterrupted for many minutes (or, in the best case, many hours) at a time. (Even if we don’t explicitly move our bodies to the music, when we listen our minds are dancing; the same rules apply.)

I don’t remember what prompted me to take that note, but it was probably not that the mixing was especially smooth.



A tomato hailing from Capay, California.

LHCSound is a site where you can listen to sonified data from the Large Hadron Collider. Some thoughts:

  • That’s one untidy heap of a website. Is this how it feels to be inside the mind of a brilliant physicist?
  • The name “LHCSound” refers to “Csound”, a programming language for audio synthesis and music composition. But how many of their readers will make the connection?
  • If they are expecting their readers to know what Csound is, then their explanation of the process they used for sonification falls way short. I want to know the details of how they mapped their data to synthesis parameters.
  • What great sampling material this will make. I wonder how long before we hear electronic music incorporating these sounds.

The Immersive Pinball demo I created for Fortune’s Brainstorm:Tech conference was featured in a BBC special on haptics.

I keep watching the HTC Sense unveiling video from Mobile World Congress 2010. The content is pretty cool, but I’m more fascinated by the presentation itself. Chief marketing officer John Wang gives a simply electrifying performance. It almost feels like an Apple keynote.

The iFeel_IM haptic interface has been making rounds on the internet lately. I tried it at CHI 2010 a few weeks ago and liked it a lot. Affective (emotional haptic) interfaces are full of potential. IFeel_IM mashes together three separate innovations:

  • Touch feedback in several different places on the body: spine, tummy, waist.
  • Touch effects that are generated from emotional language.
  • Synchronization to visuals from Second Life

All are very interesting. The spine haptics seemed a stretch to me, but the butterfly-in-the-tummy was surprisingly effective. The hug was good, but a bit sterile. Hug interfaces need nuance to bring them to the next level of realism.

The fact that the feedback is generated from the emotional language of another person seemed to be one of the major challenges—the software is built to extract emotionally-charged sentences using linguistic models. For example, if someone writes “I love you” to you, your the haptic device on your tummy will react by creating a butterflies-like sensation. As an enaction devotee I would rather actuate a hug with a hug sensor. Something about the translation of words to haptics is difficult for me to accept. But it could certainly be a lot of fun in some scenarios!

I’ve re-recorded my techno mix Awake with significantly higher sound quality. So if you downloaded a copy be sure to replace it with the new file!

Awake

Awake
Techno | 46:01 | October 2009

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Download (mp3, 92 MB)


1. District One (a.k.a. Bart Skils & Anton Pieete) — Dubcrystal
2. Saeed Younan — Kumbalha (Sergio Fernandez Remix)
3. Pete Grove — I Don’t Buy It
4. DBN — Asteroidz featuring Madita (D-Unity Remix)
5. Wehbba & Ryo Peres — El Masnou
6. Broombeck — The Clapper
7. Luca & Paul — Dinamicro (Karotte by Gregor Tresher Remix)
8. Martin Worner — Full Tilt
9. Joris Voorn — The Deep

I recently started using Eclipse on OS X and it was so unresponsive, it was almost unusable. Switching tabs was slow, switching perspectives was hella slow. I searched around the web for a solid hour for how to make it faster and finally found the solution. Maybe someone can use it.

My machine is running OS X 10.5, and I have 2 GB of RAM. (This is important because the solution requires messing with how Eclipse handles memory. If you have a different amount of RAM, these numbers may not work and you’ll need to fiddle with them.)

  • Save your work and quit Eclipse.
  • Open the Eclipse application package by right-clicking (or Control-clicking) on Eclipse.app and select “Show Package Contents.”
  • Navigate to Contents→MacOS→, and open “eclipse.ini” in your favorite text editor.
  • Edit the line that starts with -”XX:MaxPermSize” to say “-XX:MaxPermSize=128m”.
  • Before that line, add a line that says “-XX:PermSize=64m”.
  • Edit the line that starts with “-Xms” to say “-Xms40m”.
  • Edit the line that starts ith “-Xmx” to say “-Xmx768m”.
  • Save & relaunch Eclipse.

Worked like a charm for me.

Scroll to Top

To Top

books

24

Sep
2009

3 Comments

In books

By David Birnbaum

Merleau-Ponty’s philosophy

On 24, Sep 2009 | 3 Comments | In books | By David Birnbaum

9780253219732_lrg

“Yes or no: do we have a body—that is, not a permanent object of thought, but a flesh that suffers when it is wounded, hands that touch?” — The Visible and the Invisible

Merleau-Ponty’s Philosophy by Lawrence Hass was the first full book I read on the great phenomenologist. If you’re fascinated by sensation, perception, synesthesia, metaphor, and flesh (and frankly, who isn’t?), please read it! It offers many wonderful revelations. I’ll briefly review the following topics from the book:

  • Sensation/perception is a false dichotomy.
  • Perception is “contact with otherness.”
  • Synesthesia is a constant feature of experience.
  • The concepts of “reversibility” and “flesh”

Read more…

Tags | , , , , , , , , , , , , ,

01

Jul
2009

No Comments

In books

By David Birnbaum

Tactility in the Tractatus Logico-Philisophicus

On 01, Jul 2009 | No Comments | In books | By David Birnbaum

I’ve written before about the later writings of Wittgenstein and the metaphor of the word as a manual tool. However, in Ludwig’s first published work, the Tractatus Logico-Philosophicus, his theory of language is that sentences represent states of affairs, the so-called picture theory of language. Although he later abandoned that viewpoint for the tool-based one, I was intrigued by this historically significant switch-up, so I read through the Tractatus with special attention to its visual and tactile metaphors. Here are a few examples.

2.013
Each thing is, as it were, in a space of possible states of affairs. This space I can imagine empty, but I cannot imagine the thing without the space.
2.0131
A spatial object must be situated in infinite space. (A spatial point is an argument-place.)
A speck in the visual field, though it need not be red, must have some colour: it is, so to speak, surrounded by colour-space. Notes must have some pitch, objects of the sense of touch some degree of hardness, and so on.

This is a prime example of a sensory metaphor used throughout the book. Objects are described in a visual way, as being seen as situated within a possibility space or belief space. They themselves have extension, but we perceive them as taking up some amount of the visual field (surrounded by context, which here is represented as other possibilities for their position or physical attributes).

2.151
Pictorial form is the possibility that things are related to one another in the same way as the elements of the picture.
2.1511
That is how a picture is attached to reality; it reaches right out to it.
2.1512
It is laid against reality like a measure.
2.15121
Only the end-points of the graduating lines actually touch the object that is to be measured.

Again we get a visual metaphor described in terms of physicality. How is a picture “attached” to reality? It reaches out to it. And while it touches reality, it only just touches it, in a tangential way. Wittgenstein starts with a visual image and then writes “attached”, “reaches out”, “laid against”, and “touch”—all haptic metaphors.

2.1514
The pictorial relationship consists of the correlations of the picture’s elements with things.
2.1515
These correlations are, as it were, the feelers of the picture’s elements, with which the picture touches reality.

Now we have moved from the picture as a “measure,” a passive geometry tool, to a picture as an agent. Not just any agent, but an agent with a capacity for haptic perception. What is a “feeler”? To me that word means a mobile extremity with sense organs which can be used to find out about the world. So, what Wittgenstein seems to be saying here is that when we generate a picture in our mind, it’s as if we are extending our hand into the world.

4.002

Language disguises thought. So much so, that from the outward form of the clothing it is impossible to infer the form of the thought beneath it, because the outward form of the clothing is not designed to reveal the form of the body, but for entirely different purposes.

In other words, thoughts are like physical objects. A word envelops a thought. We have a thought and then we toss a word-robe over it and shove it onto the stage of discourse where it can interface with other enrobed thoughts.

4.411
It immediately strikes one as probable that the introduction of elementary propositions provides the basis for understanding all other kinds of proposition. Indeed the understanding of general propositions palpably depends on the understanding of elementary propositions.

Once again, a tactile metaphor (“palpably”) is used for emphasis and to indicate comprehensive understanding.

5.557

What belongs to its application, logic cannot anticipate.
It is clear that logic must not clash with its application.
But logic has to be in contact with its application.
Therefore, logic and its application must not overlap.

I.e.,
logicapplication

Is “overlap” a haptic metaphor or a visual one? It could be either, or both.

6.432
How things are in the world is a matter of complete indifference for what is higher. God does not reveal himself in the world.
6.4321
The facts all contribute only to setting the problem, not to its solution.
6.44
It is not how things are in the world that is mystical, but that it exists.
6.45
To view the world sub specie aeterni is to view it as a whole—a limited whole.
Feeling the world as a limited whole—it is this that is mystical.

To feel is to know, silently, mystically.

I was pretty surprised at how easy it seems to foresee Wittgenstein’s turn from the eye to the hand. He presents what he calls a picture theory of language, but it repeatedly leads to a description of solid objects in space, or bodies moving and feeling and contacting each other. Of course I’m reading with a very biased perspective. Not only is my goal to hunt for tactile metaphors but I also know how the story ends some 30 years later. Still, I can’t help but feel that the tactile language in the Tractatus may foreshadow the shift to come.

Tags | , ,

The Hand

On 04, Jun 2009 | One Comment | In art, books, cognition, neuroscience, physiology, tactility | By David Birnbaum

0679740473The Hand by Frank Wilson is a rare treat. It runs the gamut from anthropology (both the cultural and evolutionary varieties), to psychology, to biography. Wilson interviews an auto mechanic, a pupeteer, a surgeon, a physical therapist, a rock climber, a magician, and others—all with the goal of understanding the extent to which the human hand defines humanness.

Wilson is a neurologist who works with musicians who have been afflicted with debilitating chronic hand pain. As he writes about his many interviews, a few themes emerge that are especially relevant to our interests here.

Incorporation
Incorporation is the phenomenon of internalizing external objects; it’s the feeling that we all get that a tool has become one with our body.

The idea of “becoming one” with a backhoe is no more exotic than the idea of a rider becoming one with a horse or a carpenter becoming one with a hammer, and this phenomenon itself may take its origin from countless monkeys who spent countless eons becoming one with tree branches. The mystical feel comes from the combination of a good mechanical marriage and something in the nervous system that can make an object external to the body feel as if it had sprouted from the hand, foot, or (rarely) some other place on the body where your skin makes contact with it…

The contexts in which this bonding occurs are so varied that there is no single word that adequately conveys either the process or the many variants of its final form. One term that might qualify is “incorporation”—bringing something into, or making it part of, the body. It is a commonplace experience, familiar to anyone who has ever played a musical instrument, eaten with a fork or chopsticks, ridden a bicycle, or driven a car. (p. 63)

Projection
Projection is the ability to use the hand as a bridge for projecting consciousness from one location to another. (Wilson did not use the word “projection” in the book.) In some ways, projection can be seen as the opposite of incorporation. Master puppeteer Anton Bachleitner:

It takes at least three years of work to say you are a puppeteer. The most difficult job technically is to be able to feel the foot contact the floor as it actually happens. The only way to make the puppet look as though it is actually walking is by feeling what is happening through your hands. The other thing which I think you cannot really train for, but only can discover with very long practice and experience, is a change in your own vision.

The best puppeteer after some years will actually see what is happening on the stage as if he himself was located in the head of the puppet, looking out through the puppet’s eyes—he must learn to be in the puppet. This is true not only in the traditional actor’s sense, but in an unusual perceptual sense. The puppeteer stands two meters above the puppet and must be able to see what is on the stage and to move from the puppet’s perspective. Moving is a special problem because of this distance, because the puppet does not move at the same time your hand does. Also, there can be several puppets on the stage at the same time, and to appear realistic they must react to each other as they would in real life. So again the puppeteer must himself be mentally on the stage and able to react as a stage actor would react. This is something I cannot explain, but it is very imprortant for a puppeteer to be able to do this. (pp. 92–93)

Serge Percelly, professional juggler:

[An act is successful] not because you put something in the act that’s really difficult, but because you put something in the act in exactly the right way—in a way that makes it more interesting, not only for me but for the audience as well. I’m just trying somehow to do the act that I would have loved to see. (p. 111)

Skill
Wilson is a musician and a doctor to musicians, so he has special insight into the neurology of musical skill—which he recognizes as special case of manual skill that involves gesture, communication, and emotion.

Musical skill provides the clearest example and the cleanest proof of the existence of a whole class of self-defined, personally distinctive motor skills with an extended training and experience base, strong ties to the individual’s emotional and cognitive development, strong communicative intent, and very high performance standards. Musical skill, in other words, is more than simply praxis, ordinary manual dexterity, or expertness in pantomime. (p. 207)

The upper-limb (or “output”) requirements for an instrumentalist are not unique either; they depend upon the possession of arms, fingers, and thumbs, specific but idiosyncratic limits on the rage of motion at the shoulder, elbow, wrist, hand, and finger joints, variable abilities to achieve repetition rates and forces with specific digital configurations in sequence at multiple contact points on a sound-making device, and so on. Peculiarities in the physical configuration and movement capabisities of the musician’s limbs can be an advantage or disadvantage but are reflected in (and in adverse cases can be overcome through) instrument design: How wide can you make the neck of a guitar? How far apart should the keys be on a piano? Where should the keys be placed on a flute—in general? and for Susan and Peter? (p. 225)

Awareness
Touch experience can be a gateway to awareness, which can in turn heal both the mind and the body. Moshe Feldenkrais invented a form of physical therapy that focuses on stimulating an awareness of touch and movement sensations in order to relieve pain.

Most people slouch, tilt, shuffle, twist, stumble, and hobble along. Why should that be? Was there something wrong with their brains? After considering what dancers and musicians go through to improve control of their movements, [Feldenkrais] guessed that people must either be ignorant of the possibilities or refuse to act on them. So they just heave themselves around, lurching from parking place to office to parking place, utterly oblivious to what they are doing, to their appearance, and even to the sensations that arise from bodily movement. He suspected that people just lose contact with their own bodies. If and when they do notice, it is because they are so stiff that they can’t get out of bed or are in so much pain that they can barely get out of a chair. Then they start noticing…

What [Feldenkrais] was doing did not seem complicated. The goal of the guided movements was not to learn how to move, in the sense of learning to do a new dance step. The goal was not to stretch ligaments or muscles. It was not to increase strength. The goal, as he saw it, was to get the messages moving again and to encourage the brain to pay attention to them. (p. 244)

And his student, Anat Baniel, on the deep psychological roots of movement disorders:

I think working with children has given me this idea, which isn’t often discussed in medicine: a lot of disease—medical disease and emotional “dis-ease”—is an outcome of a lack of full development. It’s not something we can get to just by removing a psychological block…

Of course there are problems due to traumatic events in childhood, or disease—you name it. Feldenkrais said that ideal development would happen if the child was not opposed by a force too big for its strength. When you say to a small child, “Don’t touch that, it’s dangerous!” you create such a forceful inhibition that you actually distort the child’s movement, and growth, in a certain way.

Feldenkrais taught us to look for what isn’t there. Why doesn’t movement happen in the way that it should, given gravity, given the structure of the body, given the brain? For all of us there is a sort of sphere, or range, of movement that should be possible. Some people get only five or ten percent of that sphere, and you have to ask, “What explains the difference between those who get very little and those who get a lot?” Feldenkrais said that the difference is that in the process of development, the body encountered forces that were disproportionate to what the nervous system could absorb without becoming overinhibited—or overly excited, which is a manifestation of the same thing. (p. 252)

Feldenkrais’s approach is fascinating, but there is scant discussion in Wilson’s book about the role of the therapist’s hand in this process. After all, this kind of therapy is wholly reliant on an accidental discovery: that the patient can be made aware of her own body through an external, expert hand radiating pressure and heat. How is this possible? The topic isn’t explored.

There are many, many wonderful things to learn from this book for anyone with an interest in biology, art, music, history, or sports. You can find Frank Wilson on the web at Handoc.com

Tags | , , ,

08

Oct
2008

No Comments

In books
cognition
music
neuroscience

By David Birnbaum

Embodied music cognition

On 08, Oct 2008 | No Comments | In books, cognition, music, neuroscience | By David Birnbaum

6a00c11413d7d0819d00fae8d7c728000b-500piThis is Your Brain on Music is a great introductory book on the neuroscience of music. Although I found it weighted a bit too much toward popular science for my liking, that was its stated purpose, and there was still plenty of good information in it.

Here we have an explanation of musical timing as an analogy for a moving body:

Virtually every culture and civilization considers movement to be an integral part of music making and listening. Rhythm is what we dance to, sway our bodies to, and tap our feet to… It is no coincidence that making music requires the coordinated, rhythmic use of our bodies, and that energy be transmitted from body movements to a musical instrument. (57)

‘Tempo’ refers to the pace of a musical piece—how quickly or slowly it goes by. If you tap your foot or snap your fingers in time to a piece of music, the tempo of the piece will be directly related to how fast or slow you are tapping. If a song is a living, breathing entity, you might think of the tempo as its gait—the rate at which it walks by—or its pulse—the rate at which the heart of the song is beating. The word ‘beat’ indicates the basic unit of measurement in a musical piece; this is also called the ‘tactus’. Most often, this is the natural point at which you would tap your foot or clap your hands or snap your fingers. (59)

Levitin also delves into the possible evolutionary reasons for music, noting that music seems to always go with dance, and that the concept of the expert musical performer is very recent:

When we ask about the evolutionary basis for music, it does no good to think about Britney or Bach. We have to think about what music was like around fifty thousand years ago. The instruments recovered from archeological sites can help us understand what our ancestors used to make music, and what kinds of melodies they listened to. Cave paintings, paintings on stoneware, and other pictorial artifacts can tell us something about the role that music played in daily life. We can also study contemporary societies that have been cut off from civilization as we know it, groups of people who are living in hunter-gatherer lifestyles that have remained unchanged for thousands of years. One striking find is that in every society of which we’re aware, music and dance are inseparable.

The arguments against music as an adaptation consider music only as disembodied sound, and moreover, as performed by an expert class for an audience. But it is only in the last five hundred years that music has become a spectator activity—the thought of a musical concert in which a class of “experts” performed for an appreciative audience was virtually unknown throughout our history as a species. And it has only been in the last hundred years or so that the ties between musical sound and human movement have been minimized. The embodied nature of of music, the indivisibility of movement and sound, the anthropologist John Blacking writes, characterizes music across cultures and across times. (257)

I agree. Even though we may use modern technology to exploit musical cognitive faculties for maximum effect, the idea that music/dance is a counter-evolutionary accident seems wrong to me.

You can find the website that accompanies the book at yourbrainonmusic.com.

Tags | ,

07

Sep
2008

No Comments

In books
language
tactility

By David Birnbaum

Philosopher deathmatch, and how words are like tools

On 07, Sep 2008 | No Comments | In books, language, tactility | By David Birnbaum

9780060936648I just finished reading Wittgenstein’s Poker. From the jacket:

In October 1946, philosopher Karl Popper arrived at Cambridge to lecture at a seminar hosted by his legendary colleague Ludwig Wittgenstein. It did not go well: the men began arguing, and eventually, Wittgenstein began waving a fire poker toward Popper. It lasted scarcely 10 minutes, yet the debate has turned into perhaps modern philosophy’s most contentious encounter, largely because none of the eyewitnesses could agree on what happened. Did Wittgenstein physically threaten Popper with the poker? Did Popper lie about it afterward?

The authors provide a comprehensive biographical and historical context for the incident, and use it as a springboard into the two men’s respective philosophies. It’s an enjoyable look at two self-important, short-tempered intellectuals and their rivalry.

As I mentioned in this post, I find Wittgenstein’s philosophy of language often invokes touch themes. In the following excerpt from Poker (originating from one of his lectures), Wittgenstein makes a point about a colleague’s statement, “Good is what is right to admire,” utilizing a haptic metaphor:

The definition throws no light. There are three concepts, all of them vague. Imagine three solid pieces of stone. You pick them up, fit them together and you now get a ball. What you’ve now got tells you something about the three shapes. Now consider you have three balls of soft mud or putty. Now you put the three together and mold out of them a ball. Ewing makes a soft ball out of three pieces of mud. (68)

Another example stems from Wittgenstein’s midlife change in philosophical outlook. In his first publication, the Tractatus Logicio-Philosophicus, he was preoccupied with the “picture theory of language”—the idea that sentences describe “states of affairs” that can be likened to the contents of a picture. Later, he developed a theory of language based on words as tools for conveying meaning. In my reading, he shifted from a vision-based to a haptic-based (in fact, a distinctly physical-interaction-based) understanding of how language works.

The metaphor of language as a picture is replaced by the metaphor of language as a tool. If we want to know the meaning of a term, we should not ask what it stands for: we should instead examine how it is actually used. If we do so, we will soon recognize that there is no underlying single structure. Some words, which at first glance look as if they perform similar functions, actually operate to distinct sets of rules. (229)

Here’s the relevant passage directly from Philisophical Investigations:

It is like looking into the cabin of a locomotive. We see handles all looking more or less alike. (Naturally, since they are all supposed to be handled.) But one is the handle of a crank which can be moved continuously (it regulates the opening of a valve); another is the handle of a switch, which has only two effective positions, it is either off or on; a third is the handle of a brake-lever, the harder one pulls on it, the harder it brakes; a fourth, the handle of the pump: it has an effect only so long as it is moved to and fro. (PI, I, par. 12)

Words as the physical interface to meaning. Love it!

Tags | , , ,

28

May
2008

One Comment

In books
tactility

By David Birnbaum

Wittgenstein

On 28, May 2008 | One Comment | In books, tactility | By David Birnbaum

The philosophy of Ludwig Wittgenstein seems to come up often, so I decided to read up on what all the hoopla is about. This semi-biographical introduction to his major works is fascinating and easy to read. It seemed pretty comprehensive as well, though I’m no expert.

Wittgenstein was primarily concerned with logic and language, but I found his emphasis on know-how as opposed to know-that, and his view that skill supersedes knowledge of rules, to have a certain ‘embodiment’ quality to it. Excerpts:

So, language can indeed be said to be governed by rules; but those rules are for the most part only implicit in native speakers’ common usage. They can be derived from common usage by anyone who pays attention to it, but they are rarely operative in it: we do not normally use rules to work out what is correct. Rather, we have a fairly reliable ‘feeling’ for what sounds right in a given case. Rules can be formulated to codify our usage, but our usage is not ultimately based on such rules. (191)

Our traditional concept of pain, however, is not in competition with concepts that classify the same phenomena in terms of their underlying conditions. No physiologist could convince me that what in my own case I call ‘pain’ may not in fact be pain, or that my pain was in truth not where I felt it but in the brain. Why is this so?

Concepts are an expression of our interests. We group things together and call them by a common name according to those resemblances we find striking or important. And in different contexts we may be interested in different aspects of things. To classify phenomena scientifically, by their underlying structures or causes, is not always what we want. For instance, when taking an aesthetic attitude towards things, we are concerned entirely with their appearances. Invisible micro-structures become wholly irrelevant. And another area in which the scientific urge to leave behind the surface for underlying causes is often out of place is the realm of feeling: where our primary interest is in people’s conscious experience. Their suffering and well-being is important to us in its own right, and not merely as an indication of some underlying physiological conditions. Therefore physiological concepts like ‘lesion of tissue’ — whatever their importance for diagnosis and therapy — can never be in competition with, or act as substitutes for, our traditional concepts of feelings and emotions that are taught and understood through their links with natural expressive behaviour and characterized by the special authority we have in their first-person use. (254)

Tags |

18

Apr
2008

No Comments

In books

By David Birnbaum

Singulatarianism

On 18, Apr 2008 | No Comments | In books | By David Birnbaum

TSiN

Ray Kurzweil’s The Singularity Is Near is worth reading. It’s a great introduction to the “emerging technologies” meme. If you’ve already begun wandering down the diabolical path of contemplating the obsolescence of biology and the societal transformation that will result, Singularity doesn’t reveal much. However it’s an easy read and a cultural milestone, so pushing through it is still probably a good idea.

That said, I had a few gripes with the book. Firstly, the author’s tone doesn’t sound like the cool scientific theorist he claims to be. Virtually every discussion of progress in biotechnology lends particular weight to the synthesizing of artificial pancreas cells, which Kurzweil readily acknowledges he needs to reverse the course of his diabetes. He repeats over and over that people (like him) who take full advantage of today’s knowledge about longevity will live to see the Singularity (and by corollary, live indefinitely), and that those who do not will needlessly pay the price of annihilation. I don’t remember even one acknowledgment that his own efforts may fail (or be misguided!) despite his dogged adherence to health guidelines derived from cutting edge research. This doesn’t seem to reflect the attitude of a detached intellectual.

Maybe I’m being too harsh. It’s pretty clear that Kurzweil tries to make his subject singularly exciting, and the book seems to be written at least partially for the journalists who might like to capitalize on the author’s theory by writing articles to the effect of, “This man says you’ll socialize with robots one day. Isn’t that insane plus totally interesting?”

Still, I found his discussions of sociology and morality hugely lacking. Kurzweil falls far short in his brief examination of the coming collisions between the Singularity and virtually every modern social structure. He might counter that his goal for the book was to cultivate optimism and open minds to the grand potential of the coming transcendence. But with its foreboding title, one would expect the book to not only open minds but also put them on guard against perversions of his grand vision. And for his slight allowance that catastrophe may wipe us all out before (or after) the Singularity, he does not explicitly acknowledge that our place in the universe is so absurdly minute that, for instance, even after our entire solar system becomes a giant computer the collision of two unknown (or even known) neutron stars halfway across our galaxy could fry it all, and human technology might be powerless to stop it.

I extracted some passages that were relevant to haptics:

The current disadvantages of Web-based commerce (for example, limitations in the ability to directly interact with products and the frequent frustrations of interacting with inflexible menus and forms instead of human personnel) will gradually dissolve as the trends move robustly in favor of the electronic world. By the end of this decade, computers will disappear as distinct physical objects, with displays built in our eyeglasses and electronics woven in our clothing, providing full-immersion visual virtual reality. Thus, “going to a Web site” will mean entering a virtual-reality environment—at least for the visual and auditory senses—where we can directly interact with products and people, both real and simulated… Haptic (tactile) interfaces will enable us to touch products and people. It is difficult to identify any lasting advantage of the old brick-and-mortar world that will not ultimately be overcome by the rich interactive interfaces that are soon to come. (104–105)

I don’t like “haptic (tactile)” (they’re not equivalent), but the point is clear. There were also some references to musical interaction research:

Edward Taub at the University of Alabama studied the region of the cortex responsible for evaluating the tactile input from the fingers. Comparing non-musicians to experienced players of stringed instruments, he found no difference in the brain regions devoted to the fingers of the right hand but a huge difference for the fingers of the left hand. (174)

Dr. Alvaro Pascual-Leone at Harvard University scanned the brains of volunteers before and after they practiced a simple piano exercise. The brain motor cortex of the volunteers changed as a direct result of their practice. He then had a second group just think about doing the piano exercise but without actually moving any muscles. This produced an equally pronounced change in the motor-cortex network. (175)

Finally, I think the use of the word “mastering” here brings up an important point:

Machines have exacting memories. Contemporary computers can master billions of facts accurately, a capability that its doubling every year. (261)

Haptic metaphors like this are used generously throughout the book, no doubt to underscore Kurzweil’s intense belief that humanity is ingrained in nonbiological intelligent technology. However, as a terminology freak I would argue that computers memorize facts, but don’t master them. Mastery comes with an ability to manipulate freely. For mastery, you need a body.

Tags | , , , ,

27

Feb
2008

No Comments

In books

By David Birnbaum

Enactive perception

On 27, Feb 2008 | No Comments | In books | By David Birnbaum

Action In Perception

I just finished Action In Perception by Alva Noë. It’s a very readable introduction to the enactive view of perceptual consciousness, which argues that perception neither happens in us nor to us; rather, it’s something we do with our bodies, situated in the physical world, over time. Our knowledge of the way in which sensory stimulation varies as we control our bodies is what brings experience about. Without sensorimotor skill, a stimulus cannot constitute a percept. Noë presents empirical evidence for his claim, drawing on the phenomenology of change blindness as well as tactile vision substitution systems. I highly recommend the book.

The emphasis on embodied experience leads to the use of touch as a model for perception, rather than the traditional vision-based approach. Here’s an excerpt:

Touch acquires spatial content—comes to represent spatial qualities—as a result of the ways touch is linked to movement and to our implicit understanding of the relevant tactile-motor dependencies governing our interaction with objects. [Philosopher George Berkeley] is right that touch is, in fact, a kind of movement. When a blind person explores a room by walking about in it and probing with his or her hands, he or she is perceiving by touch. Crucially, it is not only the use of the hands, but also the movement in and through the space in which the tactile activity consists. Very fine movements of the fingers and very gross wanderings across a landscape can each constitute excercises of the sense of touch. Touch, in all such cases, is movement. (At the very least, it is movement of something relative to the perceiver.) These Berkeleyan ideas form a theme, more recently, in the work of [Brian O'Shaughnessy's book "Consciousness and World"]. He writes: “touch is in a certain respect the most important and certainly the most primordial of the senses. The reason is, that it is scarely to be distinguished from the having of a body that can act in physical space”…

But why hold that touch is the only active sense modality? As we have stressed, the visual world is not given all at once, as in a picture. The presence of detail consists not in its representation now in consciousness, but in our implicit knowledge now that we can represent it in consciousness if we want, by moving the eyes or by turning the head. Our perceptual contact with the world consists, in large part, in our access to the world thanks to our possession of sensorimotor knowledge.

Here, no less than in the case of touch, spatial properties are available due to links to movement. In the domain of vision, as in that of touch, spatial properties present themselves to us as “permanent possibilities of movement.” As you move around the rectangular object, its visible profile deforms and transforms itself. These deformations and transformations are reversible. Moreover, the rules governing the transformation are familiar, at least to someone who has learned the relevant laws of visuomotor contingency. How the item looks varies systematically as a function of your movements. Your experience of it as cubical consists in your implicit understanding of the fact that the relevant regularity is being observed.

Virtual and augmented reality interface design practices have already begun to demonstrate these concepts. Head mounted augmented reality displays sense the user’s eye and body movements to construct virtual percepts. Head related transfer functions (HRTFs) synthesize sound as it would be heard by an organism with certain physical characteristics, in a particular environment. It seems to me that an important implication for enactive interface design is that haptic sensory patterns can lead to perceptual experience in all sensory modes (vision, hearing, touch). Thus, all human-computer interaction/user experience can be viewed in a haptic context.

Tags | , , , , , , ,