The development of audiovisual speech perception in Mandarin-speaking children: Evidence from the McGurk paradigm.

Publisher: Blackwell Publishers Country of Publication: United States NLM ID: 0372725 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1467-8624 (Electronic) Linking ISSN: 00093920 NLM ISO Abbreviation: Child Dev Subsets: MEDLINE

Publication: Malden, MA : Blackwell Publishers
Original Publication: [Chicago, etc.] : Published by the University of Chicago Press for the Society for Research in Child Development [etc.]

Speech Perception*/physiology; Male ; Adult ; Child ; Humans ; Child, Preschool ; Cross-Sectional Studies ; Acoustic Stimulation ; Visual Perception/physiology ; China ; Auditory Perception/physiology ; Photic Stimulation ; Speech/physiology

The developmental trajectory of audiovisual speech perception in Mandarin-speaking children remains understudied. This cross-sectional study in Mandarin-speaking 3- to 4-year-old, 5- to 6-year-old, 7- to 8-year-old children, and adults from Xiamen, China (n = 87, 44 males) investigated this issue using the McGurk paradigm with three levels of auditory noise. For the identification of congruent stimuli, 3- to 4-year-olds underperformed older groups whose performances were comparable. For the perception of the incongruent stimuli, a developmental shift was observed as 3- to 4-year-olds made significantly more audio-dominant but fewer audiovisual-integrated responses to incongruent stimuli than older groups. With increasing auditory noise, the difference between children and adults widened in identifying congruent stimuli but narrowed in perceiving incongruent ones. The findings regarding noise effects agree with the statistically optimal hypothesis.
(© 2023 The Authors. Child Development © 2023 Society for Research in Child Development.)

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near‐optimal bimodal integration. Current Biology, 14, 257–262. https://doi.org/10.1016/j.cub.2004.01.029.
Alsius, A., Paré, M., & Munhall, K. G. (2017). Forty years after hearing lips and seeing voices: The McGurk effect revisited. Multisensory Research, 31, 111–144. https://doi.org/10.1163/22134808‐00002565.
Bernstein, L. E. (2012). Visual speech perception. In G. Bailly, P. Perrier, & E. Vatikiotis‐Bateson (Eds.), Audiovisual speech processing (pp. 21–39). Cambridge University Press.
Burnham, D., & Dodd, B. (2004). Auditory‐visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 45, 204–220. https://doi.org/10.1002/dev.20032.
Burr, D., & Gori, M. (2012). Multisensory integration develops late in humans. In M. M. Murray & M. T. Wallace (Eds.), The neural bases of multisensory processes (pp. 345–362). CRC Press.
Chen, Y., & Hazan, V. (2009). Developmental factors and the non‐native speaker effect in auditory‐visual speech perception. The Journal of the Acoustical Society of America, 126, 858–865. https://doi.org/10.1121/1.3158823.
de Boysson‐Bardies, B., & Vihman, M. M. (1991). Adaptation to language: Evidence from babbling and first words in four languages. Language, 67, 297–319. https://doi.org/10.1353/lan.1991.0045.
Denes, P. B., & Pinson, E. N. (1993). The speech chain: The physics and biology of spoken language. W.H. Freeman.
Desjardins, R. N., & Werker, J. F. (2004). Is the integration of heard and seen speech mandatory for infants? Developmental Psychobiology, 45, 187–203. https://doi.org/10.1002/dev.20033.
Dupont, S., Aubin, J., & Ménard, L. (2005). Study of the McGurk effect in 4 and 5‐year‐old French Canadian children. ZAS Papers in Linguistics, 40, 1–17. https://doi.org/10.21248/zaspil.40.2005.254.
Elliott, L. L. (1979). Performance of children aged 9 to 17 years on a test of speech intelligibility in noise using sentence material with controlled word predictability. The Journal of the Acoustical Society of America, 66, 651–653. https://doi.org/10.1121/1.383691.
Eramudugolla, R., Henderson, R., & Mattingley, J. B. (2011). Effects of audio–visual integration on the detection of masked speech and non‐speech sounds. Brain and Cognition, 75, 60–66. https://doi.org/10.1016/j.bandc.2010.09.005.
Ernst, M. O. (2008). Multisensory integration: A late bloomer. Current Biology, 18, R519–R521. https://doi.org/10.1016/j.cub.2008.05.002.
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. https://doi.org/10.1038/415429a.
Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169. https://doi.org/10.1016/J.TICS.2004.02.002.
Fetsch, C. R., Pouget, A., DeAngelis, G. C., & Angelaki, D. E. (2012). Neural correlates of reliability‐based cue weighting during multisensory integration. Nature Neuroscience, 15, 146–154. https://doi.org/10.1038/nn.2983.
Frossard, J., & Renaud, O. (2021). Permutation tests for regression, ANOVA, and comparison of signals: The permuco package. Journal of Statistical Software, 99, 1–32. https://doi.org/10.18637/jss.v099.i15.
Gijbels, L., Yeatman, J. D., Lalonde, K., & Lee, A. K. C. (2021). Audiovisual speech processing in relationship to phonological and vocabulary skills in first graders. Journal of Speech, Language, and Hearing Research, 64, 5022–5040. https://doi.org/10.1044/2021_JSLHR‐21‐00196.
Gori, M., Campus, C., & Cappagli, G. (2021). Late development of audio‐visual integration in the vertical plane. Current Research in Behavioral Sciences, 2, 100043. https://doi.org/10.1016/j.crbeha.2021.100043.
Gori, M., Del Viva, M., Sandini, G., & Burr, D. C. (2008). Young children do not integrate visual and haptic form information. Current Biology, 18, 694–698. https://doi.org/10.1016/j.cub.2008.04.036.
Hazan, V., & Li, E. (2008). The effect of auditory and visual degradation on audiovisual perception of native and non‐native speakers. Proceedings of the annual conference of the International Speech Communication Association, INTERSPEECH (pp. 1191–1194). https://doi.org/10.21437/interspeech.2008‐359.
Heikkilä, J., Lonka, E., Ahola, S., Meronen, A., & Tiippana, K. (2017). Lipreading ability and its cognitive correlates in typically developing children and children with specific language impairment. Journal of Speech, Language, and Hearing Research, 60, 485–493. https://doi.org/10.1044/2016_JSLHR‐S‐15‐0071.
Hirst, R. J., Stacey, J. E., Cragg, L., Stacey, P. C., & Allen, H. A. (2018). The threshold for the McGurk effect in audio‐visual noise decreases with development. Scientific Reports, 8, 1–18. https://doi.org/10.1038/s41598‐018‐30798‐8.
Hu, B. Y., Fan, X., Yang, Y., & Neitzel, J. (2017). Chinese preschool teachers' knowledge and practice of teacher–child interactions: The mediating role of teachers' beliefs about children. Teaching and Teacher Education, 63, 137–147. https://doi.org/10.1016/j.tate.2016.12.014.
Jerger, S., Damian, M. F., Spence, M. J., Tye‐Murray, N., & Abdi, H. (2009). Developmental shifts in children's sensitivity to visual speech: A new multimodal picture–word task. Journal of Experimental Child Psychology, 102, 40–59. https://doi.org/10.1016/j.jecp.2008.08.002.
Johnson, C. E. (2000). Childrens' phoneme identification in reverberation and noise. Journal of Speech, Language, and Hearing Research, 43, 144–157. https://doi.org/10.1044/jslhr.4301.144.
Kishon‐Rabin, L., & Henkin, Y. (2000). Age‐related changes in the visual perception of phonologically significant contrasts. British Journal of Audiology, 34, 363–374. https://doi.org/10.3109/03005364000000152.
Knowland, V. C. P., Evans, S., Snell, C., & Rosen, S. (2016). Visual speech perception in children with language learning impairments. Journal of Speech, Language, and Hearing Research, 59, 1–14. https://doi.org/10.1044/2015_JSLHR‐S‐14‐0269.
Lalonde, K., & Holt, R. F. (2015). Preschoolers benefit from visually salient speech cues. Journal of Speech, Language, and Hearing Research, 58, 135–150. https://doi.org/10.1044/2014_JSLHR‐H‐13‐0343.
Lalonde, K., & Werner, L. A. (2021). Development of the mechanisms underlying audiovisual speech perception benefit. Brain Sciences, 11, 49. https://doi.org/10.3390/brainsci11010049.
Lewkowicz, D. J., & Flom, R. (2014). The audiovisual temporal binding window narrows in early childhood. Child Development, 85, 685–694. https://doi.org/10.1111/cdev.12142.
Li, M., Chen, X., Zhu, J., & Chen, F. (2022). Audiovisual mandarin lexical tone perception in quiet and noisy contexts: The influence of visual cues and speech rate. Journal of Speech, Language, and Hearing Research, 65, 4385–4403. https://doi.org/10.1044/2022_JSLHR‐22‐00024.
Li, Y., Mei, L., & Dong, Q. (2008). The characteristics and development of audiovisual speech perception in native Chinese speakers. Psychological Development and Education, 3, 43–47.
Li, Y., & Zhu, J. (2014). Instruction manual of chinese version of the Wechsler Preschool and primary scale of intelligence (4th ed.). King‐May Psychological Measurement Technology Development Co., Ltd.
Liu, M., Du, X., & Liu, Q. (2020). The features of audiovisual speech perception in noise of children with autism Spectrum disorder. Chinese Journal of Applied Psychology, 3, 232–239.
Macdonald, J., & McGurk, H. (1978). Visual influences on speech perception processes. Perception & Psychophysics, 24, 253–257. https://doi.org/10.3758/BF03206096.
Magnotti, J. F., Basu Mallick, D., Feng, G., Zhou, B., Zhou, W., & Beauchamp, M. S. (2015). Similar frequency of the McGurk effect in large samples of native mandarin Chinese and American English speakers. Experimental Brain Research, 233, 2581–2586. https://doi.org/10.1007/s00221‐015‐4324‐7.
Massaro, D. W. (1989). Testing between the TRACE model and the fuzzy logical model of speech perception. Cognitive Psychology, 21, 398–421. https://doi.org/10.1016/0010‐0285(89)90014‐5.
Massaro, D. W., Thompson, L. A., Barron, B., & Laren, E. (1986). Developmental changes in visual and auditory contributions to speech perception. Journal of Experimental Child Psychology, 41, 93–113. https://doi.org/10.1016/0022‐0965(86)90053‐6.
McGurk, H., & Macdonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. https://doi.org/10.1038/264746a0.
McLeod, S., & Crowe, K. (2018). Children's consonant acquisition in 27 languages: A cross‐linguistic review. American Journal of Speech‐Language Pathology, 27, 1546–1571. https://doi.org/10.1044/2018_AJSLP‐17‐0100.
Ménard, L., Leclerc, A., & Tiede, M. (2014). Articulatory and acoustic correlates of contrastive focus in congenitally blind adults and sighted adults. Journal of Speech, Language, and Hearing Research, 57, 793–804. https://doi.org/10.1044/2014_JSLHR‐S‐12‐0395.
Nardini, M., Jones, P., Bedford, R., & Braddick, O. (2008). Development of cue integration in human navigation. Current Biology, 18, 689–693. https://doi.org/10.1016/j.cub.2008.04.021.
R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R‐project.org/.
Robinson, C. W., & Sloutsky, V. M. (2004). Auditory dominance and its change in the course of development. Child Development, 75, 1387–1401. https://doi.org/10.1111/j.1467‐8624.2004.00747.x.
Robinson, C. W., & Sloutsky, V. M. (2010). Development of cross‐modal processing. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 135–141. https://doi.org/10.1002/WCS.12.
Rosenblum, L. D., & Dorsi, J. (2021). Primacy of multimodal speech perception for the brain and science. In The handbook of speech perception (pp. 28–57). Wiley. https://doi.org/10.1002/9781119184096.ch2.
Sato, M., Troille, E., Ménard, L., Cathiard, M.‐A., & Gracco, V. (2013). Silent articulation modulates auditory and audiovisual speech perception. Experimental Brain Research, 227, 275–288. https://doi.org/10.1007/s00221‐013‐3510‐8.
Schorr, E. A., Fox, N. A., van Wassenhove, V., & Knudsen, E. I. (2005). Auditory‐visual fusion in speech perception in children with cochlear implants. Proceedings of the National Academy of Sciences of the United States of America, 102, 18748–18750. https://doi.org/10.1073/pnas.0508862102.
Sekiyama, K., & Burnham, D. (2008). Impact of language on development of auditory‐visual speech perception. Developmental Science, 11, 306–320. https://doi.org/10.1111/j.1467‐7687.2008.00677.x.
Sekiyama, K., Burnham, D., Tam, H., & Erdener, D. (2003). ISCA archive auditory‐visual speech perception development in Japanese and English speakers. Future University Hakodate.
Sekiyama, K., Soshi, T., & Sakamoto, S. (2014). Enhanced audiovisual integration with aging in speech perception: A heightened McGurk effect in older adults. Frontiers in Psychology, 5, 323. https://doi.org/10.3389/fpsyg.2014.00323.
Stacey, J. E., Howard, C. J., Mitra, S., & Stacey, P. C. (2020). Audio‐visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect. Attention, Perception, & Psychophysics, 82, 3544–3557. https://doi.org/10.3758/S13414‐020‐02042‐X.
Thompson, L. A., & Massaro, D. W. (1994). Children's integration of speech and pointing gestures in comprehension. Journal of Experimental Child Psychology, 57, 327–354. https://doi.org/10.1006/jecp.1994.1016.
Tremblay, C., Champoux, F., Voss, P., Bacon, B. A., Lepore, F., & Théoret, H. (2007). Speech and non‐speech audio‐visual illusions: A developmental study. PLoS One, 2, e742. https://doi.org/10.1371/journal.pone.0000742.
Tye‐Murray, N., Hale, S., Spehar, B., Myerson, J., & Sommers, M. S. (2014). Lipreading in school‐age children: The roles of age, hearing status, and cognitive ability. Journal of Speech, Language, and Hearing Research, 57, 556–565. https://doi.org/10.1044/2013_JSLHR‐H‐12‐0273.
Tye‐Murray, N., Sommers, M., & Spehar, B. (2007). Auditory and visual lexical neighborhoods in audiovisual speech perception. Trends in Amplification, 11, 233–241. https://doi.org/10.1177/1084713807307409.
Tye‐Murray, N., Spehar, B., Myerson, J., Hale, S., & Sommers, M. (2016). Lipreading and audiovisual speech recognition across the adult lifespan: Implications for audiovisual integration. Psychology and Aging, 31, 380–389. https://doi.org/10.1037/pag0000094.
Van Engen, K. J., Dey, A., Sommers, M. S., & Peelle, J. E. (2022). Audiovisual speech perception: Moving beyond McGurk. The Journal of the Acoustical Society of America, 152, 3216–3225. https://doi.org/10.1121/10.0015262.
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America, 102, 1181–1186. https://doi.org/10.1073/pnas.0408949102.
Vihman, M. M. (1996). Phonological development: The origins of language in the child. Blackwell Publishing.
Vihman, M. M., Macken, M. A., Miller, R., Simmons, H., & Miller, J. (1985). From babbling to speech: A re‐assessment of the continuity issue. Language, 61, 397–445. https://doi.org/10.2307/414151.
Walley, A. C., & Carrell, T. D. (1983). Onset spectra and formant transitions in the adult's and child's perception of place of articulation in stop consonants. The Journal of the Acoustical Society of America, 73, 1011–1022. https://doi.org/10.1121/1.389149.
Witten, I. B., & Knudsen, E. I. (2005). Why seeing is believing: Merging auditory and visual worlds. Neuron, 48, 489–496. https://doi.org/10.1016/j.neuron.2005.10.020.
Zhang, H. (2008). Instruction manual of chinese version of the Wechsler Intelligence Scale for Children (4th ed.). King‐May Psychological Measurement Technology Development Co., Ltd.
Zhang, J., Meng, Y., McBride, C., Fan, X., & Yuan, Z. (2018). Combining behavioral and ERP methodologies to investigate the differences between McGurk effects demonstrated by Cantonese and mandarin speakers. Frontiers in Human Neuroscience, 12, 181. https://doi.org/10.3389/fnhum.2018.00181.

PolyU/RFS2122-5H01 Research Grants Council, University Grants Committee

Date Created: 20231016 Date Completed: 20240418 Latest Revision: 20240418

20240418

10.1111/cdev.14022

37843038