Aero-tactile resesearch

home

Aero-tactile integration in speech perception

In November, 2009, Bryan Gick and I published an article entitled "Aero-tactile integration in speech perception" in Nature.

The abstract reads (references found in article):

Visual information from a speaker’s face can enhance1 or interfere with2 accurate auditory perception. This integration of information across auditory and visual streams has been observed in functional imaging studies3,4, and has typically been attributed to the frequency and robustness with which perceivers jointly encounter event-specific information from these two modalities5. Adding the tactile modality has long been considered a crucial next step in understanding multisensory integration. However, previous studies have found an influence of tactile input on speech perception only under limited circumstances, either where perceivers were aware of the task6,7 or where they had received training to establish a cross-modal mapping8–10. Here we show that perceivers integrate naturalistic tactile information during auditory speech perception without previous training. Drawing on the observation that some speech sounds produce tiny bursts of aspiration (such as English ‘p’)11, we applied slight, inaudible air puffs on participants’ skin at one of two locations: the right hand or the neck. Syllables heard simultaneously with cutaneous air puffs were more likely to be heard as aspirated (for example, causing participants to mishear ‘b’ as ‘p’). These results demonstrate that perceivers integrate event-relevant tactile information in auditory perception in much the same way as they do visual information.

Characteristics of Air Puffs Produced in English 'pa'

In May, 2009, I published an article, with Peter Anderson, Bryan Gick, and Sheldon Green, entitled "Characteristics of Air Puffs Produced in English 'pa': Experiments and Simulations" in the Journal of the Acoustical Society of America.

The abstract reads:

Three dimensional large eddy simulations, microphone 'pop' measurements, and high-speed videos of the airflow and lip opening associated with the syllable 'pa' are presented. In the simulations, the mouth is represented by a narrow static ellipse with a back pressure dropping to 1/10th of its initial value within 60 ms of the release. The simulations show a jet penetration rate that falls within range of the pressure front of microphone pop. The simulations and high-speed video experiments were within 20% agreement after 40 ms, with the video experiments showing a slower penetration rate than the simulations during the first 40 ms. Kinematic measurements indicate that rapid changes in lip geometry during the first 40 ms underlie this discrepancy. These findings will be useful for microphone manufacturers, sound engineers, and researchers in speech aerodynamics modeling and articulatory speech synthesis.

Full body aero-tactile integration in speech perception

In September 2010, I published an article with Bryan Gick entitled "Full body aero-tactile integration in speech perception" in the INTERSPEECH 2010 proceedings.

The abstract reads (references found in article):

We follow up on our research demonstrating that aero-tactile information can enhance or interfere with accurate auditory perception, even among uninformed and untrained perceivers [1]. Mimicking aspiration, we applied slight, inaudible air puffs on participants’ skin at the ankle, simultaneously with syllables beginning with aspirated (‘pa’, ‘ta’) and unaspirated (‘ba’, ‘da’) stops, dividing the participants into two groups, those with hairy, and those with hairless ankles. Since hair follicle endings (mechanoreceptors) are used to detect air turbulence [2] we expected, and observed, that syllables heard simultaneously with cutaneous air puffs would be more likely to be heard as aspirated, but only among those with hairy ankles. These results demonstrate that information from any part of the body can be integrated in speech perception, but the stimuli must be unambiguously relatable to the speech event in order to be integrated into speech perception.

The temporal window of audio-tactile integration in speech perception

In November, 2010, Bryan Gick, Yoko Ikegami and I published an article entitled "The temporal window of audio-tactile integration in speech perception" in JASA-EL.

The abstract reads:

Asynchronous cross-modal information is integrated asymmetrically in audio-visual perception. To test whether this asymmetry generalizes across modalities, auditory (aspirated “pa” and unaspirated “ba” stops) and tactile (slight, inaudible, cutaneous air puffs) signals were presented synchronously and asynchronously. Results were similar to previous AV studies: the temporal window of integration for the enhancement effect (but not the interference effect) was asymmetrical, allowing up to 200 ms of asynchrony when the puff followed the audio signal, but only up to 50 ms when the puff preceded the audio signal. These findings suggest that perceivers accommodate differences in physical transmission speed of different multimodal signals.