Short and sweet: Comparing strategies for the reduction of questionnaires on self-criticism and social safeness while preserving construct validity.

Ahead of Print

Publisher: Wiley Country of Publication: England NLM ID: 0107305 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1464-066X (Electronic) Linking ISSN: 00207594 NLM ISO Abbreviation: Int J Psychol Subsets: MEDLINE

Publication: 2014- : Chichester : Wiley
Original Publication: 1966-<1978>: Paris : Dunod

Measuring psychological traits with standardised questionnaires is an essential component of clinical practice and research; however, patients and participants risk fatigue from overly long and repetitive measures. When developing the short form of a questionnaire, the most widely used method for selecting an item subset uses factor analysis loadings to identify the items most closely related to the psychological construct being measured. However, this approach will tend to select highly correlated, homogeneous items and might therefore restrict the breadth of the construct examined. In this study, we will present Yarkoni's genetic algorithm for scale reduction and compare it with the classical scale reduction method. The algorithm will be applied to the shortening of three instruments for measuring self-compassion and social safeness (two unidimensional measures and a three-factor measure). We evaluated the shortened scales using correlation with long-form scores, internal reliability and the change in the correlations observed with other related constructs. Findings suggested that the classical method preserves internal reliability, but Yarkoni's genetic algorithm better maintained correlations with other constructs. An additional qualitative assessment of item content showed that the latter method led to a more heterogeneous selection of items, better preserving the full complexity of the constructs being measured.
(© 2024 The Author(s). International Journal of Psychology published by John Wiley & Sons Ltd on behalf of International Union of Psychological Science.)

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57, 289–300. https://doi.org/10.1111/j.2517‐6161.1995.tb02031.x.
Bowling, N. A., Gibson, A. M., Houpt, J. W., & Brower, C. K. (2021). Will the questions ever end? Person‐level increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718–738. https://doi.org/10.1177/1094428120947794.
Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.
Brunell, A. B., Davis, M. S., Schley, D. R., Eng, A. L., van Dulmen, M. H., Wester, K. L., & Flannery, D. J. (2013). A new measure of interpersonal exploitativeness. Frontiers in Psychology, 4, 299. https://doi.org/10.3389/fpsyg.2013.00299.
Camilleri, J. A., Eickhoff, S. B., Weis, S., Chen, J., Amunts, J., Sotiras, A., & Genon, S. (2021). A machine learning approach for the factorization of psychometric data with application to the Delis Kaplan executive function system. Scientific Reports, 11(1), 16896. https://doi.org/10.1038/s41598‐021‐96342‐3.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555.
Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin‐Germeys, I., & Viechtbauer, W. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151. https://doi.org/10.1177/1073191120957102.
Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360. https://doi.org/10.1093/poq/nfp031.
Gilbert, P., Clarke, M., Hempel, S., Miles, J. N., & Irons, C. (2004). Criticizing and reassuring oneself: An exploration of forms, styles and reasons in female students. British Journal of Clinical Psychology, 43(1), 31–50. https://doi.org/10.1348/014466504772812959.
Gilbert, P., McEwan, K., Matos, M., & Rivis, A. (2011). Fears of compassion: Development of three self‐report measures. Psychology and Psychotherapy: Theory, Research and Practice, 84(3), 239–255. https://doi.org/10.1348/147608310X526511.
Gilbert, P., McEwan, K., Mitra, R., Richter, A., Franks, L., Mills, A., Bellew, R., & Gale, C. (2009). An exploration of different types of positive affect in students and patients with a bipolar disorder. Clinical Neuropsychiatry, 6(4), 135–143.
Goretzko, D. (2022). Factor retention in exploratory factor analysis with missing data. Educational and Psychological Measurement, 82(3), 444–464. https://doi.org/10.1177/00131644211022031.
Goretzko, D., & Bühner, M. (2020). One model to rule them all? Using machine learning algorithms to determine the number of factors in exploratory factor analysis. Psychological Methods, 25(6), 776–786. https://doi.org/10.1037/met0000262.
Iglesias, C., & Torgerson, D. (2000). Does length of questionnaire matter? A randomised trial of response rates to a mailed questionnaire. Journal of Health Services Research & Policy, 5(4), 219–221. https://doi.org/10.1177/135581960000500406.
Kleka, P., & Soroko, E. (2018). How to avoid the sins of questionnaires abridgement. Survey Research Methods, 12(2), 147–160.
Kruyen, P. M., Emons, W. H., & Sijtsma, K. (2013). On the shortcomings of shortened tests: A literature review. International Journal of Testing, 13(3), 223–248. https://doi.org/10.1080/15305058.2012.703734.
Meng, X. L., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin, 111, 172–175. https://doi.org/10.1037//0033‐2909.111.1.172.
Neff, K. D. (2003). The development and validation of a scale to measure self‐compassion. Self and Identity, 2(3), 223–250. https://doi.org/10.1080/15298860309027.
Neff, K. D., & McGehee, P. (2010). Self‐compassion and psychological resilience among adolescents and young adults. Self and Identity, 9(3), 225–240. https://doi.org/10.1080/15298860902979307.
Nunnally, J. C. (1978). An overview of psychological measurement. In B. B. Wolman (Ed.), Clinical diagnosis of mental disorders. Springer. https://doi.org/10.1007/978‐1‐4684‐2490‐4_4.
Raykov, T., & Pohl, S. (2013). Essential unidimensionality examination for multicomponent scales: An interrelationship decomposition approach. Educational and Psychological Measurement, 73(4), 581–600. https://doi.org/10.1177/0013164412470451.
Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437.
Richter, A., Gilbert, P., & McEwan, K. (2009). Development of an early memories of warmth and safeness scale and its relationship to psychopathology. Psychology and Psychotherapy: Theory, Research and Practice, 82(2), 171–184. https://doi.org/10.1348/147608308X395213.
Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237. https://doi.org/10.1080/00223891.2015.1089249.
Sahdra, B. K., Ciarrochi, J., Parker, P., & Scrucca, L. (2016). Using genetic algorithms in a large nationally representative American sample to abbreviate the Multidimensional Experiential Avoidance Questionnaire. Frontiers in Psychology, 7, 167967. https://doi.org/10.3389/fpsyg.2016.00189.
Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3(3), 271–295.
Yarkoni, T. (2010). The abbreviation of personality, or how to measure 200 personality scales with 200 items. Journal of Research in Personality, 44(2), 180–198. https://doi.org/10.1016/j.jrp.2010.01.002.
Ziegler, M., Poropat, A., & Mell, J. (2014). Does the length of a questionnaire matter? Journal of Individual Differences, 35(4), 250–261. https://doi.org/10.1027/1614‐0001/a000147.

Keywords: Construct validity; Scale abbreviation; Scale reduction; Self‐criticism; Social safeness

Date Created: 20240928 Latest Revision: 20240928

20240929

10.1002/ijop.13249

39340159