"Perception of sounds from impacted materials:
virtual reality and neuro-acoustic perspectives"       


Related articles:

Journal articles:
  • J.A. Micoulaud-Franchi, M. Aramaki, A. Merer, M. Cermolacce, S. Ystad, R. Kronland-Martinet, and J. Vion-Dury. Categorization and timbre perception of environmental sounds in schizophrenia, Psychiatry research, 2011, doi:10.1016/j.psychres.2011.03.001. direct link to sounds
  • M. Aramaki, M. Besson, R. Kronland-Martinet, and S. Ystad. Controlling the perceived material in an impact sound synthesizer (pdf). IEEE Transactions on Audio, Speech and Language Processing, Vol. 19(2), pp. 301-314, 2011.
  • M. Aramaki, C. Marie, R. Kronland-Martinet, S. Ystad, and M. Besson. Sound categorization and conceptual priming for non linguistic and linguistic sounds (pdf). Journal of Cognitive Neuroscience, Vol. 22(11), pp. 2555-2569, 2010.
Conference Presentations and Papers:
  • M. Aramaki, L. Brancheriau, R. Kronland-Martinet and S. Ystad. Perception of impacted materials : sound retrieval and synthesis control perspectives. In 5th International Symposium on Computer Music Modeling and Retrieval, CMMR 2008 - Genesis of Meaning in Sound and Music, pp. 1–8, Copenhagen, Denmark, 19-23 May 2008.
  • M. Aramaki, R. Kronland-Martinet, Th. Voinier and S. Ystad. Timbre control of a real-time percussive synthesizer. In Proceedings of the International Congress on Acoustics (ICA2007), Eds : Antonio Calvo-Manzano, Antonio Pérez-Lòpez, Salvador Santiago, ISBN 84-87985-12-2, Madrid, Espagne, 2-7 Septembre 2007.
Book Chapters:
  • M. Aramaki, Ch. Gondre, R. Kronland-Martinet, T. Voinier, and S. Ystad. “Imagine the sounds : an intuitive control of an impact sound synthesizer”, in Computer Music Modeling and Retrieval - Auditory Display, ser. LNCS, S. Ystad, R. Kronland-Martinet, and K. Jensen, Eds. Springer-Verlag Berlin Heidelberg, 2010, vol. 5954.
  • M. Aramaki, M. Besson, R. Kronland-Martinet, and S. Ystad, “Timbre perception of sounds from impacted materials: behavioral, electrophysiological and acoustic approaches”, in Computer Music Modeling and Retrieval - Genesis of Meaning of Sound and Music, ser. LNCS, S. Ystad, R. Kronland-Martinet, and K. Jensen, Eds. Springer-Verlag Berlin Heidelberg, 2009, vol. 5493, pp. 1–17.
  • M. Aramaki, L. Brancheriau, R. Kronland-Martinet, and S. Ystad, “Perception of impacted materials: sound retrieval and synthesis control perspectives”, in Computer Music Modeling and Retrieval - Genesis of Meaning of Sound and Music, ser. LNCS, S. Ystad, R. Kronland-Martinet, and K. Jensen, Eds. Springer-Verlag Berlin Heidelberg, 2009, vol. 5493, pp. 134–146.

                                                                                                                                                                  Updated 29/04/2012


Sound categorization experiment

In this study, we focused on the identification of the perceptual properties of impacted materials to provide an intuitive control of an impact sound synthesizer. To investigate such properties, impact sounds from everyday life objects, made of different materials (Wood, Metal and Glass), were recorded and analyzed. These sounds were synthesized using an analysis-synthesis technique and tuned to the same chroma. Sound continua were created to simulate progressive transitions between materials. Sounds from these continua were then used in a categorization experiment to determine sound categories representative of each material (called typical sounds). We also examined changes in electrical brain activity (using Event Related Potentials -ERPs- method) associated with the categorization of these typical sounds. Moreover, acoustic analysis was conducted to investigate the relevance of acoustic descriptors known to be relevant for both timbre perception and material identification. Both acoustic and electrophysiological data confirmed the importance of damping and highlighted the relevance of spectral content for material perception. Based on these findings, controls for damping and spectral shaping were tested in synthesis applications. A global control strategy, with a three-layer architecture, was proposed for the synthesizer allowing the user to intuitively navigate in a "Material space" and defining impact sounds directly from the material label. 


Stimuli

Analysis-synthesis and tuning of natural sounds:

Impact sounds from everyday life objects made of Wood, Metal and Glass materials were recorded, analyzed, synthesized and tuned to the same chroma.

WOOD:
Sound 1 original synthesis tuned
Sound 2 original synthesis tuned
Sound 3 original synthesis tuned
Sound 4 original synthesis tuned
Sound 5 original synthesis tuned

METAL:     

Sound 1 original synthesis tuned
Sound 2 original synthesis tuned
Sound 3 original synthesis tuned
Sound 4 original synthesis tuned
Sound 5 original synthesis tuned


GLASS:
 
Sound 1 original synthesis tuned
Sound 2 original synthesis tuned
Sound 3 original synthesis tuned
Sound 4 original synthesis tuned
Sound 5 original synthesis tuned


Sound continua:

Then, we created sound continua that simulate progressive transitions between two sounds of different materials by mixing the spectra and by morphing the damping. 
An example of such a continuum between a wood and a metal sound is given  here.

The whole set of sounds used for the categorization experiment is available below. 

1- Transition between wood and metal
       Continuum #1:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #2:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #3:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #4:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #5:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20

2- Transition between wood and glass
       Continuum #1:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #2:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #3:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #4:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #5:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20

3- Transition between glass and metal
       Continuum #1:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #2:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #3:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #4:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20
       Continuum #5:            1    2    3    4    5    6    7    8    9    10    11    12    13    14    15    16    17    18    19    20

Stimuli for sound categorization in schizophrenia

Sound continua were used to investigate disturbances of sound categorization in patients with schizophrenia.

A same experimental protocol based on a categorization task of the perceived material was conducted. 

To shorten the duration of the experiment for patients, we reduced the previous sound corpus and selected the 3 following continua (one per transition):

1- Transition between wood and metal: Continuum #2

2- Transition between wood and glass: Continuum #1

3- Transition between glass and metal: Continuum #2



Importance of the spectral centroid (CSG ; related to the brighness) and the spectral bandwidth (SB) in sound categorization

Both CSG and SB have shown to be of little influence on the material categorization. This can be illustrated by the following examples:

Sounds with identical CSG obtained by impacting objets of different materials: GLASS    WOOD    METAL
Sounds with identical SB obtained by impacting objets of different materials: GLASS    WOOD    METAL


Influence of damping and spectral complexity on material perception


The perception of materials mainly correlates with the damping of the spectral components (sound examples can be heard here or here). 
Nevertheless, damping cannot explain by itself the material categorization.

We constructed 9 stimuli by independently combining the spectra of typical sounds (of Wood, Metal or Glass) with the damping characteristics of these sounds. 
These stimuli correspond to the 9 possible combinations between the 3 spectra and the 3 damping law characteristics of each material category.

  Wood damping Metal damping Glass damping
Wood spectrum
sound
sound
sound
Metal spectrum
sound
sound
sound
Glass spectrum
sound
sound
sound

Participants were asked to categorize these 9 sounds as Wood, Metal or Glass. 
Results confirmed that damping is not sufficient to characterize the material categories and that both damping and the spectral content of sounds are relevant cues for sound categorization.

Control of the perceived material in a sound synthesizer: "The Material Space"


Control of sound dynamics (damping)

For impact sounds, sound dynamics is exclusively governed by damping. 

A control of damping is proposed through the control of 2 parameters characterizing the frequency-dependent damping law.

The first parameter reflects global damping. The second one reflects the frequency-relative damping, i.e., difference between high frequency component damping and low frequency component damping. 

The manipulation of these 2 parameters gives a refined control of the sound dynamics, allowing the synthesis of a wide variety of realistic sounds.

Control of spectral shaping


Two strategies are proposed to control the spectral shaping:

- spectral dilation. This control is provided by using 2 parameters. The first parameter reflects global shaping. The second one reflects the relative shaping.

- amplitude and frequency modulations. This control is provided by acting on modulation index I and the value of the modulating frequency.

The spectral shaping control can also be applied locally on a specific frequency range of the original spectrum by using a window function W.     


Control strategy based on damping and spectral dilation:

Sounds corresponding to the numbered position illustrate the navigation in the  Material space:

- Sound Metal (M)

- Sound Wood (W)

- Sound Glass (G)

- Sound at the center (C)

1- Transition between Wood and Metal
       along the circle :          W    20    19    18    17    16    15    14    13    12    11    M
       straight :                        W    40    39    38    37    36    M
       through the center :     W    51    50     C    48    49     M

2- Transition between Wood and Glass
       along the circle :          W    21    22    23    24    25    26    27    28    29    30    G
       straight :                        W    41    42    43    44    45    G
       through the center :     W    51    50     C    46    47    G

3- Transition between Glass and Metal
       along the circle :          G    1    2    3    4    5    6    7    8    9    10    M
       straight :                        G    31    32    33    34    35     M
       through the center :     G    47    46     C    48    49    M


Control strategy based on damping and amplitude modulation:

Transition between Glass and Metal:   Glass    »    Step 1    »    Step 2    »    Step 3    »    Step 4    »    Metal

Sound effects:     Glass    »    Cracked glass