Exploration of factors affecting webcam-based automated gaze coding.

Publisher: Springer Country of Publication: United States NLM ID: 101244316 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1554-3528 (Electronic) Linking ISSN: 1554351X NLM ISO Abbreviation: Behav Res Methods Subsets: MEDLINE

Publication: 2010- : New York : Springer
Original Publication: Austin, Tex. : Psychonomic Society, c2005-

Fixation, Ocular*/physiology; Humans ; Female ; Adult ; Male ; Eye-Tracking Technology ; Young Adult ; Algorithms ; Infant ; Reproducibility of Results ; Eye Movements/physiology ; Behavioral Research/methods

Online experiments have been transforming the field of behavioral research, enabling researchers to increase sample sizes, access diverse populations, lower the costs of data collection, and promote reproducibility. The field of developmental psychology increasingly exploits such online testing approaches. Since infants cannot give explicit behavioral responses, one key outcome measure is infants' gaze behavior. In the absence of automated eyetrackers in participants' homes, automatic gaze classification from webcam data would make it possible to avoid painstaking manual coding. However, the lack of a controlled experimental environment may lead to various noise factors impeding automatic face detection or gaze classification. We created an adult webcam dataset that systematically reproduced noise factors from infant webcam studies which might affect automated gaze coding accuracy. We varied participants' left-right offset, distance to the camera, facial rotation, and the direction of the lighting source. Running two state-of-the-art classification algorithms (iCatcher+ and OWLET) revealed that facial detection performance was particularly affected by the lighting source, while gaze coding accuracy was consistently affected by the distance to the camera and lighting source. Morphing participants' faces to be unidentifiable did not generally affect the results, suggesting facial anonymization could be used when making online video data publicly available, for purposes of further study and transparency. Our findings will guide improving study design for infant and adult participants during online experiments. Moreover, training algorithms using our dataset will allow researchers to improve robustness and allow developmental psychologists to leverage online testing more efficiently.
(© 2024. The Author(s).)

Amazon. (n.d.). Amazon rekognition: Developer guide. https://docs.aws.amazon.com/rekognition/latest/dg/what-is.html . Accessed 22 Apr 2024.
Aslin, R. N. (2007). What’s in a look? Developmental Science, 10(1), 48–53. https://doi.org/10.1111/j.1467-7687.2007.00563.x. (PMID: 10.1111/j.1467-7687.2007.00563.x171816992493049)
Bacon, D., Weaver, H., & Saffran, J. (2021). A framework for online experimenter-moderated looking-time studies assessing infants’ linguistic knowledge. Frontiers in Psychology, 12, 703839. https://doi.org/10.3389/fpsyg.2021.703839. (PMID: 10.3389/fpsyg.2021.703839346302118497712)
Bailey, T. M., & Plunkett, K. (2002). Phonological specificity in early words. Cognitive Development, 17(2), 12651282. https://doi.org/10.1016/S0885-2014(02)00116-8. (PMID: 10.1016/S0885-2014(02)00116-8)
Baltrušaitis, T., Zadeh, A., Lim, Y. C., & Morency, L. P. (2018). Openface 2.0: Facial behavior analysis toolkit. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (pp. 59–66). IEEE. https://doi.org/10.1109/FG.2018.00019. (PMID: 10.1109/FG.2018.00019)
Bánki, A., de Eccher, M., Falschlehner, L., Hoehl, S., & Markova, G. (2022). Comparing online webcam-and laboratory-based eye-tracking for the assessment of infants’ audio-visual synchrony perception. Frontiers in Psychology, 12, 733933. https://doi.org/10.3389/fpsyg.2021.733933. (PMID: 10.3389/fpsyg.2021.733933350874428787048)
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01. (PMID: 10.18637/jss.v067.i01)
Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristia, A. (2018). Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Development, 89(6), 1996–2009. https://doi.org/10.1111/cdev.13079. (PMID: 10.1111/cdev.13079297369626282795)
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com’s Mechanical Turk. Political Analysis, 20(3), 351–368. https://doi.org/10.1093/pan/mpr057. (PMID: 10.1093/pan/mpr057)
Bradski, G. (2000). The openCV library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer, 25(11), 120–123.
BrighterAI. (n.d.). Brighter redact: Image & video redaction. https://azuremarketplace.microsoft.com/en-us/marketplace/apps/brighteraitechnologiesgmbh1596709907398.ips-online?tab=overview&exp=ubp8 . Accessed 22 Apr 2024.
Byers‐Heinlein, K., Bergmann, C., & Savalei, V. (2022). Six solutions for more reliable infant research. Infant and Child Development, 31(5), e2296. https://doi.org/10.1002/icd.2296.
Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160. https://doi.org/10.1016/j.chb.2013.05.009. (PMID: 10.1016/j.chb.2013.05.009)
Chouinard, B., Scott, K., & Cusack, R. (2019). Using automatic face analysis to score infant behaviour from video collected online. Infant Behavior and Development, 54, 1–12. https://doi.org/10.1016/j.infbeh.2018.11.004. (PMID: 10.1016/j.infbeh.2018.11.00430508782)
Chuey, A., Asaba, M., Bridgers, S., Carrillo, B., Dietz, G., Garcia, T., Leonard, J. A., Liu, S., Merrick, M., Radwan, S., Stegall, J., Velez, N., Woo, B., Wu, Y., Zhou, X. J., Frank, M. C., & Gweon, H. (2021). Moderated online data-collection for developmental research: Methods and replications. Frontiers in Psychology, 12, 734398. https://doi.org/10.3389/fpsyg.2021.734398. (PMID: 10.3389/fpsyg.2021.734398348038138595939)
Dalrymple, K. A., Manner, M. D., Harmelink, K. A., Teska, E. P., & Elison, J. T. (2018). An examination of recording accuracy and precision from eye tracking data from toddlerhood to adulthood. Frontiers in Psychology, 9, 803. https://doi.org/10.3389/fpsyg.2018.00803. (PMID: 10.3389/fpsyg.2018.00803298757275974590)
Davis-Kean, P. E., & Ellis, A. (2019). An overview of issues in infant and developmental research for the creation of robust and replicable science. Infant Behavior and Development, 57, 101339. https://doi.org/10.1016/j.infbeh.2019.101339. (PMID: 10.1016/j.infbeh.2019.10133931351250)
DeBolt, M. C., Rhemtulla, M., & Oakes, L. M. (2020). Robust data and power in infant research: A case study of the effect of number of infants and number of trials in visual preference procedures. Infancy, 25(4), 393–419. https://doi.org/10.1111/infa.12337. (PMID: 10.1111/infa.1233732744759)
Erel, Y., Potter, C. E., Jaffe-Dax, S., Lew-Williams, C., & Bermano, A. H. (2022). iCatcher: A neural network approach for automated coding of young children’s eye movements. Infancy, 27(4), 765–779. https://doi.org/10.1111/infa.12468. (PMID: 10.1111/infa.12468354163789320879)
Erel, Y., Shannon, K. A., Chu, J., Scott, K. M., Kline Struhl, M., Cao, P., …, Liu, S. (2023). iCatcher+: Robust and automated annotation of infant’s and young children's gaze direction from videos collected in laboratory, field, and online studies. Advances in Methods and Practices in Psychological Science, 6(2), 1–23. https://doi.org/10.1177/25152459221147250.
Fantz, R. L. (1964). Visual experience in infants: Decreased attention to familiar patterns relative to novel ones. Science, 146(3644), 668–670. https://doi.org/10.1126/science.146.3644.668. (PMID: 10.1126/science.146.3644.66814191712)
Fernald, A., Pinto, J. P., Swingley, D., Weinbergy, A., & McRoberts, G. W. (1998). Rapid gains in speed of verbal processing by infants in the 2nd year. Psychological Science, 9(3), 228–231. https://doi.org/10.1111/1467-9280.00044. (PMID: 10.1111/1467-9280.00044)
Fernald, A., Zangl, R., Portillo, A. L., & Marchman, V. A. (2008). Looking while listening: Using eye movements to monitor spoken language comprehension by infants and young children. In I. A. Sekerina, E. M. Fernandez, & H. Clahsen (Eds.), Developmental psycholinguistics: On-line methods in children’s language processing (pp. 97–135). John Benjamins. (PMID: 10.1075/lald.44.06fer)
Fischer, T., Chang, H. J., & Demiris, Y. (2018). RT-GENE: Real-time eye gaze estimation in natural environments. Proceedings of the European Conference on Computer Vision (ECCV) (pp. 334–352). Springer. Retrieved from https://openaccess.thecvf.com/content_ECCV_2018/html/Tobias_Fischer_RT-GENE_Real-Time_Eye_ECCV_2018_paper.html . Accessed 22 Apr 2024.
Friend, M., & Keplinger, M. (2008). Reliability and validity of the Computerized Comprehension Task (CCT): Data from American English and Mexican Spanish infants. Journal of Child Language, 35(1), 77–98. https://doi.org/10.1017/S0305000907008264. (PMID: 10.1017/S0305000907008264183004305501698)
Golinkoff, R. M., Hirsh-Pasek, K., Cauley, K. M., & Gordon, L. (1987). The eyes have it: Lexical and syntactic comprehension in a new paradigm. Journal of Child Language, 14(1), 23–45. https://doi.org/10.1017/s030500090001271x. (PMID: 10.1017/s030500090001271x3558524)
Golinkoff, R. M., Ma, W., Song, L., & Hirsh-Pasek, K. (2013). Twenty-five years using the intermodal preferential looking paradigm to study language acquisition: What have we learned? Perspectives on Psychological Science, 8(3), 316–339. https://doi.org/10.1177/1745691613484936. (PMID: 10.1177/174569161348493626172975)
Hagihara, H., Ienaga, N., Terayama, K., Moriguchi, Y., & Sakagami, M. (2021). Looking represents choosing in toddlers: Exploring the equivalence between multimodal measures in forced-choice tasks. Infancy, 26(1), 148–167. https://doi.org/10.1111/infa.12377. (PMID: 10.1111/infa.1237733341103)
Hagihara, H., Zaadnoordijk, L., Cusack, R., & Tsuji, S. (2022, September 23). A video dataset for the exploration of factors affecting webcam-based automated gaze coding [Paper presentation]. Innovations in Online Research 2022, Online.
Hamlin, J. K., Wynn, K., & Bloom, P. (2007). Social evaluation by preverbal infants. Nature, 450(7169), 557–559. https://doi.org/10.1038/nature06288. (PMID: 10.1038/nature0628818033298)
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X. (PMID: 10.1017/S0140525X0999152X20550733)
Hessels, R. S., & Hooge, I. T. (2019). Eye tracking in developmental cognitive neuroscience–The good, the bad and the ugly. Developmental Cognitive Neuroscience, 40, 100710. https://doi.org/10.1016/j.dcn.2019.100710. (PMID: 10.1016/j.dcn.2019.100710315939096974897)
Hessels, R. S., Andersson, R., Hooge, I. T. C., Nyström, M., & Kemner, C. (2015). Consequences of eye color, positioning, and head movement for eye-tracking data quality in infant research. Infancy, 20(6), 601–633. https://doi.org/10.1111/infa.12093. (PMID: 10.1111/infa.12093)
Hessels, R. S., Cornelissen, T. H. W., Kemner, C., & Hooge, I. T. C. (2015). Qualitative tests of remote eyetracker recovery and performance during head rotation. Behavior Research Methods, 47(3), 848–859. https://doi.org/10.3758/s13428-014-0507-6. (PMID: 10.3758/s13428-014-0507-625033759)
King, D. E. (2009). Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research: JMLR, 10(60), 1755–1758. Retrieved from https://www.jmlr.org/papers/v10/king09a.html . Accessed 22 Apr 2024.
Kominsky, J. F., Begus, K., Bass, I., Colantonio, J., Leonard, J. A., Mackey, A. P., & Bonawitz, E. (2021). Organizing the methodological toolbox: Lessons learned from implementing developmental methods online. Frontiers in Psychology, 12, 702710. https://doi.org/10.3389/fpsyg.2021.702710. (PMID: 10.3389/fpsyg.2021.702710345890238473607)
Li, L., Bao, J., Yang, H., Chen, D., & Wen, F. (2019). Faceshifter: Towards high fidelity and occlusion aware face swapping. arXiv. https://doi.org/10.48550/arXiv.1912.13457.
Lo, C., Mani, N., Kartushina, N., Mayor, J., & Hermes, J. (2021). e-Babylab: An open-source browser-based tool for unmoderated online developmental studies. PsyArXiv. https://doi.org/10.31234/osf.io/u73sy.
Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584. https://doi.org/10.1016/j.tics.2020.05.003. (PMID: 10.1016/j.tics.2020.05.003324512397211671)
Maye, J., Werker, J. F., & Gerken, L. (2002). Infant sensitivity to distributional information can affect phonetic discrimination. Cognition, 82(3), B101–B111. https://doi.org/10.1016/S0010-0277(01)00157-3. (PMID: 10.1016/S0010-0277(01)00157-311747867)
Montague, D. P. F., & Walker-Andrews, A. S. (2001). Peekaboo: A new look at infants’ perception of emotion expressions. Developmental Psychology, 37(6), 826–838. https://doi.org/10.1037/0012-1649.37.6.826. (PMID: 10.1037/0012-1649.37.6.82611699756)
Niehorster, D. C., Cornelissen, T. H. W., Holmqvist, K., Hooge, I. T. C., & Hessels, R. S. (2018). What to expect from your remote eye-tracker when participants are unrestrained. Behavior Research Methods, 50(1), 213–227. https://doi.org/10.3758/s13428-017-0863-0. (PMID: 10.3758/s13428-017-0863-028205131)
Nirkin, Y., Keller, Y., & Hassner; T. (2019). FSGAN: Subject agnostic face swapping and reenactment. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, 7184–7193. https://openaccess.thecvf.com/content_ICCV_2019/html/Nirkin_FSGAN_Subject_Agnostic_Face_Swapping_and_Reenactment_ICCV_2019_paper.html . Accessed 22 Apr 2024.
Oakes, L. M. (2017). Sample size, statistical power, and false conclusions in infant looking-time research. Infancy, 22(4), 436–469. https://doi.org/10.1111/infa.12186. (PMID: 10.1111/infa.1218628966558)
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716.
Papoutsaki, A., Sangkloy, P., Laskey, J., Daskalova, N., Huang, J., & Hays, J. (2016). WebGazer: Scalable webcam eye tracking by learning from user interactions. Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), 3839–3845.
R Core Team. (2022). R: A language and environment for statistical computing. https://www.R-project.org . Accessed 22 Apr 2024.
Rhodes, M., Rizzo, M. T., Foster-Hanson, E., Moty, K., Leshin, R. A., Wang, M., ..., & Ocampo, J. D. (2020). Advancing developmental science via unmoderated remote research with children. Journal of Cognition and Development, 21(4), 477–493. https://doi.org/10.1080/15248372.2020.1797751.
Schlegelmilch, K., & Wertz, A. E. (2019). The effects of calibration target, screen location, and movement type on infant eye-tracking data quality. Infancy, 24(4), 636–662. https://doi.org/10.1111/infa.12294. (PMID: 10.1111/infa.1229432677249)
Scott, K., & Schulz, L. (2017). Lookit (part 1): A new online platform for developmental research. Open Mind: Discoveries in Cognitive Science, 1(1), 4–14. https://doi.org/10.1162/OPMI_a_00002. (PMID: 10.1162/OPMI_a_00002)
Scott, K., Chu, J., & Schulz, L. (2017). Lookit (Part 2): Assessing the viability of online developmental research, results from three case studies. Open Mind: Discoveries in Cognitive Science, 1(1), 15–29. https://doi.org/10.1162/opmi_a_00001. (PMID: 10.1162/opmi_a_00001)
Semmelmann, K., & Weigelt, S. (2018). Online webcam-based eye tracking in cognitive science: A first look. Behavior Research Methods, 50, 451–465. https://doi.org/10.3758/s13428-017-0913-7. (PMID: 10.3758/s13428-017-0913-728593605)
Singh, L., Cristia, A., Karasik, L. B., Rajendra, S. J., & Oakes, L. M. (2021, November 24). Diversity and representation in infant research: Barriers and bridges towards a globalized science of infant development. PsyArXiv. https://doi.org/10.31234/osf.io/hgukc.
Starkey, P., Spelke, E. S., & Gelman, R. (1983). Detection of intermodal numerical correspondences by human infants. Science, 222(4620), 179–181. https://doi.org/10.1126/science.6623069. (PMID: 10.1126/science.66230696623069)
Steffan, A., Zimmer, L., Arias‐Trejo, N., Bohn, M., Dal Ben, R., Flores‐Coronado, M. A., Franchin, L., Garbisch, I., Grosse Wiesmann, C., Hamlin, J. K., Havron, N., Hay, J. F., Hermansen, T. K., Jakobsen, K. V., Kalinke, S., Ko, E., Kulke, L., Mayor, J., Meristo, M., ..., & Schuwerk, T. (2024). Validation of an open source, remote web‐based eye‐tracking method (WebGazer) for research in early childhood. Infancy, 29(1), 31–55. https://doi.org/10.1111/infa.12564.
Tran, M., Cabral, L., Patel, R., & Cusack, R. (2017). Online recruitment and testing of infants with Mechanical Turk. Journal of Experimental Child Psychology, 156, 168–178. https://doi.org/10.1016/j.jecp.2016.12.003. (PMID: 10.1016/j.jecp.2016.12.00328088051)
Tsuji, S., Amso, D., Cusack, R., Kirkham, N., & Oakes, L. M. (2022). Empirical research at a distance: New methods for developmental science. Frontiers in Psychology, 13, 938995. https://doi.org/10.3389/fpsyg.2022.938995. (PMID: 10.3389/fpsyg.2022.938995356934949175027)
Valtakari, N. V., Hessels, R. S., Niehorster, D. C., Viktorsson, C., Nyström, P., Falck-Ytter, T., Kemner, C., & Hooge, I. T. C. (2023). A field test of computer-vision-based gaze estimation in psychology. Behavior Research Methods. https://doi.org/10.3758/s13428-023-02125-1. (PMID: 10.3758/s13428-023-02125-13710110010990994)
Venker, C. E., Pomper, R., Mahr, T., Edwards, J., Saffran, J., & Ellis Weismer, S. (2020). Comparing automatic eye tracking and manual gaze coding methods in young children with autism spectrum disorder. Autism Research, 13(2), 271–283. https://doi.org/10.1002/aur.2225. (PMID: 10.1002/aur.222531622050)
Wass, S. V., Forssman, L., & Leppänen, J. (2014). Robustness and precision: How data quality may influence key dependent variables in infant eye-tracker analyses. Infancy, 19(5), 427–460. https://doi.org/10.1111/infa.12055. (PMID: 10.1111/infa.12055)
Werchan, D. M., Thomason, M. E., & Brito, N. H. (2022). OWLET: An automated, open-source method for infant gaze tracking using smartphone and webcam recordings. Behavior Research Methods, 55, 3149–4163. https://doi.org/10.3758/s13428-022-01962-w. (PMID: 10.3758/s13428-022-01962-w36070130)
Yuan, S., & Fisher, C. (2009). “Really? She blicked the baby?” Two-year-olds learn combinatorial facts about verbs by listening. Psychological Science, 20(5), 619–626. https://doi.org/10.1111/j.1467-9280.2009.02341.x. (PMID: 10.1111/j.1467-9280.2009.02341.x19476591)
Zaadnoordijk, L., & Cusack, R. (2022). Online Testing in Developmental Science: A Guide to Design and Implementation. In R. O. Gilmore & J. J. Lockman (Eds.), Advances in Child Development and Behavior: New Methods and Approaches for Studying Child Development (62nd ed., pp. 93–125). Academic Press.
Zaadnoordijk, L., Buckler, H., Cusack, R., Tsuji, S., & Bergmann, C. (2021). A global perspective on testing infants online: Introducing ManyBabies-AtHome. Frontiers in Psychology, 12, 703234. https://doi.org/10.3389/fpsyg.2021.703234. (PMID: 10.3389/fpsyg.2021.703234345667818458619)
Zhang, X., Sugano, Y., & Bulling, A. (2019). Evaluation of appearance-based methods and implications for gaze-based applications. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, (pp. 1–13). https://doi.org/10.1145/3290605.3300646.

20H05919 Japan Society for the Promotion of Science; JP21J00750 Japan Society for the Promotion of Science; JP22K13664 Japan Society for the Promotion of Science; AdG 787981 ERC; 891535 HORIZON EUROPE Marie Sklodowska-Curie Actions

Keywords: Automated gaze coding; Data quality; Online experiment; Open dataset; Webcam video data

Date Created: 20240501 Date Completed: 20240829 Latest Revision: 20241031

20241031

PMC11362184

10.3758/s13428-024-02424-1

38693440