Traditional and Computer-Based Assessment of Executive Functions
DOI:
https://doi.org/10.19090/pp.v18i1.2601Keywords:
Computer-based assessments, Executive functions, Psychometric properties, Cognitive flexibility, InhibitionAbstract
This study aimed to assess the psychometric properties of two newly developed computer-based tasks (i.e., Mental Shifting/Flexibility Task and Auditory-Visual Go/No-Go Task) for measuring two key domains of executive functions (EF) - inhibition and mental shifting (flexibility) - in healthy adults. Together with these tasks, traditional paper-and-pencil tests were used for assessing construct validity (Wisconsin Card Sorting Test - WCST, Trail Making Test - TMT, Verbal Fluency Tests, and Advanced Progressive Matrices- APM). The sample consisted of 468 adult twins (70.7% female, mean age 24.06 years) or 234 twin pairs. Results revealed low to moderate correlations between the reaction times and the number of errors in the computer-based tasks and traditional tests. Specifically, the Mental Shifting/Flexibility Task showed significant correlations with the TMT and the WCST. The Auditory-Visual Go/No-Go Task was significantly related to TMT and APM, suggesting shared cognitive processes linked to inhibition, cognitive flexibility, and processing speed. The computer-based tasks demonstrated moderate to good ICC reliability, especially in reaction time measures, while error rates showed poorer reliability. It was concluded that computer-based tasks are useful for measuring executive functions. However, further validation, development of standardized norms, and optimization of these tools are needed. Future research should explore how these tools can be integrated into existing cognitive assessment batteries for more accurate measurement of executive functions across diverse populations and clinical contexts.
Metrics
References
Allport, A., & Wylie, G. (1999). Task-switching: Positive and negative priming of task-set. In G. W. Humphreys, J. Duncan, & A. Treisman (Eds.), Attention, space, and action: Studies in cognitive neuroscience (pp. 273–296). Oxford University Press. DOI: https://doi.org/10.1093/oso/9780198524694.003.0016
Alloway, T. P., & Carpenter, R. K. (2020). The relationship among children’s learning disabilities, working memory, and problem behaviours in a classroom setting: Three case studies. The Educational and Developmental Psychologist, 37(1), 4–10. https://doi.org/10.1017/edp.2020.1 DOI: https://doi.org/10.1017/edp.2020.1
Baron, I. S. (2004). Neuropsychological evaluation of the child. Oxford University Press.
Capovilla, A. G. S., Montiel, J. M., Macedo, E. C., & Charin, S. (2005). Computerized Stroop Test. University São Francisco.
Cianchetti, C., Corona, S., Foscoliano, M., Scalas, F., & Sannio–Fancello, G. (2005). Modified Wisconsin Card Sorting Test: Proposal of a supplementary sorting method. Archives of Clinical Neuropsychology, 20, 555–558. https://doi.org/10.1016/j.acn.2004.12.002 DOI: https://doi.org/10.1016/j.acn.2004.12.002
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284 DOI: https://doi.org/10.1037//1040-3590.6.4.284
Collerton, J., Collerton, D., Yasumichi, A., Barrass, K., Eccles, M., Jagger, C., … Kirkwood, T. (2007). A comparison of computerized and pencil-and-paper tasks in assessing cognitive function in community-dwelling older people in the Newcastle 85+ study. Journal of the American Geriatrics Society, 55, 1630–1635. https://doi.org/10.1111/j.1532-5415.2007.01379.x DOI: https://doi.org/10.1111/j.1532-5415.2007.01379.x
Dancy, C. L., & Ritter, F. E. (2017). IGT-Open: An open-source, computerized version of the Iowa Gambling Task. Behavior Research Methods, 49(3), 972–978. https://doi.org/10.3758/s13428-016-0759-4 DOI: https://doi.org/10.3758/s13428-016-0759-4
Diamond A. (2013). Executive functions. Annual Review of Psychology, 64, 135–168. https://doi.org/10.1146/annurev-psych-113011-143750 DOI: https://doi.org/10.1146/annurev-psych-113011-143750
Feenstra, H. E. M., Vermeulen, I. E., Murre, J. M. J., & Schagen, S. B. (2017). Online cognition: Factors facilitating reliable online neuropsychological test results. Clinical Neuropsychologist, 31(1), 59–84. https://doi.org/10.1080/13854046.2016.1190405 DOI: https://doi.org/10.1080/13854046.2016.1190405
Friedman, N. P., Miyake, A., Young, S. E., DeFries, J. C., Corley, R. P., & Hewitt, J. K. (2008). Individual differences in executive functions are almost entirely genetic in origin. Journal of Experimental Psychology: General, 137(2), 201–225. https://doi.org/10.1037/0096-3445.137.2.201 DOI: https://doi.org/10.1037/0096-3445.137.2.201
Friedman, N. P., & Miyake, A. (2004). The relations among inhibition and interference control functions: A latent-variable analysis. Journal of Experimental Psychology: General, 133(1), 101–135. https://doi.org/10.1037/0096-3445.133.1.101 DOI: https://doi.org/10.1037/0096-3445.133.1.101
Gajewski, P. D., Hanisch, E., Falkenstein, M., Thönes, S., & Wascher, E. (2018). What does the n-back task measure as we get older? Relations between working-memory measures and other cognitive functions across the lifespan. Frontiers in Psychology, 9, 2208. https://doi.org/10.3389/fpsyg.2018.02208 DOI: https://doi.org/10.3389/fpsyg.2018.02208
Garavan, H., Ross, T. J., & Stein, E. A. (1999). Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proceedings of the National Academy of Sciences of the United States of America, 96(14), 8301–8306. https://doi.org/10.1073/pnas.96.14.8301 DOI: https://doi.org/10.1073/pnas.96.14.8301
Golden, C. J. (1978). Stroop color and word test: a manual for clinical and experimental uses. Stoelting Company.
Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders. Lea Febiger.
Heaton, R. K. (1981). Wisconsin card sorting test manual. Psychological assessment resources.
Heaton, R. K., Chelune, G. J., Tallei, J. L., Key, G. G. & Curtiss, G. (1993). Wisconsin Card Sorting Test manual: Revised and expanded. Psychological Assessment Resources.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83. https://doi.org/10.1017/S0140525X0999152X DOI: https://doi.org/10.1017/S0140525X0999152X
Hinkin, C. H., Castellon, S. A., Hardy, D. J., Granholm, E., & Siegle, G. (1999). Computerized and traditional Stroop Task dysfunction in HIV-1 infection. Neuropsychology, 13(2), 306–316. https://doi.org/10.1037//0894-4105.13.2.306 DOI: https://doi.org/10.1037//0894-4105.13.2.306
Horn, J. L., & Noll, J. (1997). Human cognitive capabilities: Gf-Gc theory. In D. P. Flanagan, J. L. Genshaft, & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 53–91). The Guilford Press.
IBM Corp. (2017). IBM SPSS Statistics for Windows, Version 25.0. IBM Corp.
Jacola, L. M., Willard, V. W., Ashford, J. M., Ogg, R. J., Scoggins, M. A., Jones, M. M., Wu, S., & Conklin, H. M. (2014). Clinical utility of the N-back task in functional neuroimaging studies of working memory. Journal of Clinical and Experimental Neuropsychology, 36(8), 875–886. https://doi.org/10.1080/13803395.2014.953039 DOI: https://doi.org/10.1080/13803395.2014.953039
Jurado, M. B., & Rosselli, M. (2007). The elusive nature of executive functions: A review of our current understanding. Neuropsychology Review, 17, 213–233. https://doi.org/10.1007/s11065-007-9040-z DOI: https://doi.org/10.1007/s11065-007-9040-z
Kane, R. L., & Kay, G. G. (1992). Computerized assessment in neuropsychology: a review of tests and test batteries. Neuropsychology Review, 3(1), 1–117. https://doi.org/10.1007/BF01108787 DOI: https://doi.org/10.1007/BF01108787
Kessels, R. P. (2019). Improving precision in neuropsychological assessment: Bridging the gap between classic paper-and-pencil tests and paradigms from cognitive neuroscience. The Clinical Neuropsychologist, 33(2), 357-368. https://doi.org/10.1080/13854046.2018.1518489 DOI: https://doi.org/10.1080/13854046.2018.1518489
Khng, K. H., & Lee, K. (2014). The relationship between Stroop and stop-signal measures of inhibition in adolescents: influences from variations in context and measure estimation. PloS one, 9(7), e101356. https://doi.org/10.1371/journal.pone.0101356 DOI: https://doi.org/10.1371/journal.pone.0101356
Latendorf, A., Runde, L. M., Salminen, T., & Steinert, A. (2021). Digitization of neuropsychological diagnostics: a pilot study to compare three paper-based and digitized cognitive assessments. Aging Clinical and Experimental Research, 33, 1585-1597. https://doi.org/10.1007/s40520-020-01668-z DOI: https://doi.org/10.1007/s40520-020-01668-z
Lezak, M. D., Howieson, D. B., Loring, D. W., Hannay, H. J., & Fischer, J. S. (2004). Neuropsychological assessment (4th ed.). Oxford University Press.
Luciano, M., Wright, M., Smith, G. A., Geffen, G. M., Geffen, L. B., & Martin, N. G. (2001). Genetic covariance among measures of information processing speed, working memory, and IQ. Behavior Genetics, 31(6), 581–592. https://doi.org/10.1023/a:1013397428612 DOI: https://doi.org/10.1023/A:1013397428612
McDonald, A. S. (2002). The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments. Computers & Education, 39, 299–312. https://doi.org/10.1016/S0360-1315(02)00032-5 DOI: https://doi.org/10.1016/S0360-1315(02)00032-5
McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman & Hall. https://doi.org/10.1201/9780429029608 DOI: https://doi.org/10.1201/9780429029608
Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114(3), 449–458. https://doi.org/10.1037/0033-2909.114.3.449 DOI: https://doi.org/10.1037//0033-2909.114.3.449
Miller, J. B., & Barr, W. B. (2017). The technology crisis in neuropsychology. Archives of Clinical Neuropsychology, 32(5), 541–554. https://doi.org/10.1093/arclin/acx050 DOI: https://doi.org/10.1093/arclin/acx050
Mitrović, D., Smederevac, S., Delgado-Cruzata, L., Sadiković, S., Pajić, D., Prinz, M., Budimlija, Z., Oljača, M., Kušić-Tišma, J., Vučinić, N., & Milutinović, A. (2024). Personality and COMT gene: Molecular-genetic and epigenetic associations with NEO-PI-R personality domains and facets in monozygotic twins. Frontiers in Genetics, 15, Article 1455872. https://doi.org/10.3389/fgene.2024.1455872 DOI: https://doi.org/10.3389/fgene.2024.1455872
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. https://doi.org/10.1006/cogp.1999.0734 DOI: https://doi.org/10.1006/cogp.1999.0734
Morris, R. G., Miotto, E. C., Feigenbaum, J. D., Bullock, P., & Polkey, C. E. (1997). The effect of goal-subgoal conflict on planning ability after frontal- and temporal-lobe lesions in humans. Neuropsychologia, 35(8), 1147-1157. https://doi.org/10.1016/S0028-3932(97)00009-2 DOI: https://doi.org/10.1016/S0028-3932(97)00009-2
Neubauer, A. C., Riemann, R., Mayer, R., & Angleitner, A. (1997). Intelligence and reaction times in the Hick, Sternberg and Posner paradigms. Personality and Individual Differences, 22(6), 885–894. https://doi.org/10.1016/S0191-8869(97)00003-2 DOI: https://doi.org/10.1016/S0191-8869(97)00003-2
Park, S. Y., & Schott, N. (2022). The trail-making-test: Comparison between paper-and-pencil and computerized versions in young and healthy older adults. Applied Neuropsychology. Adult, 29(5), 1208–1220. https://doi.org/10.1080/23279095.2020.1864374 DOI: https://doi.org/10.1080/23279095.2020.1864374
Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & R Core Team. (2021). Nlme: Linear and Nonlinear Mixed Effects Models (Version 3.1-153). R package. https://CRAN.R-project.org/package=nlme
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
Raven, J., Raven, J. C., & Court, J. H. (1998). Raven manual: Section 1, general overview, 1998 edition. Oxford Psychologists Press Ltd.
Reitan, R. M. (1955). The relation of the trail making test to organic brain damage. Journal of Consulting Psychology, 19, 393–394. https://doi.org/10.1037/h0044509 DOI: https://doi.org/10.1037/h0044509
Riordan, P., Lombardo, T., & Schulenberg, S. E. (2013). Evaluation of a computer-based administration of the Rey complex figure test. Applied Neuropsychology, 20(3), 169–178. https://doi.org/10.1080/09084282.2012.670171 DOI: https://doi.org/10.1080/09084282.2012.670171
RStudio Team. (2023). RStudio: Integrated Development Environment for R (Version 2023.12.0 Build 369). RStudio, PBC. https://posit.co/
Sanders, L. M. J., Hortobágyi, T., Balasingham, M., Van der Zee, E. A., & van Heuvelen, M. J. G. (2018). Psychometric properties of a Flanker Task in a sample of patients with dementia: A pilot study. Dementia and Geriatric Cognitive Disorders Extra, 8(3), 382–392. https://doi.org/10.1159/000493750 DOI: https://doi.org/10.1159/000493750
Schatz, P., & Browndyke, J. (2002). Applications of computer-based neuropsychological assessment. The Journal of Head Trauma Rehabilitation, 17(5), 395–410. https://doi.org/10.1097/00001199-200210000-00003 DOI: https://doi.org/10.1097/00001199-200210000-00003
Schwartz, S., Baldo, J., Graves, R. E., & Brugger, P. (2003). Pervasive influence of semantics in letter and category fluency: A multidimensional approach. Brain and Language, 87(3), 400–411. https://doi.org/10.1016/S0093-934X(03)00141-X DOI: https://doi.org/10.1016/S0093-934X(03)00141-X
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420 DOI: https://doi.org/10.1037//0033-2909.86.2.420
Smederevac, S., Mitrović, D., Sadiković, S., Milovanović, I., Branovački, B., Dinić, B. M., & others. (2019). Serbian twin registry. Twin Research and Human Genetics, 22(6), 660–666. https://doi.org/10.1017/thg.2019.114 DOI: https://doi.org/10.1017/thg.2019.114
Spreen, O., & Strauss, E. (1991). A compendium of neuropsychological tests. Administration, norms and commentary. Oxford University Press.
Steinmetz, J. P., Brunner, M., Loarer, E., & Houssemand, C. (2010). Incomplete psychometric equivalence of scores obtained on the manual and the computer version of the Wisconsin Card Sorting test? Psychological Assessment, 22(1), 199–202. https://psycnet.apa.org/doi/10.1037/a0017661 DOI: https://doi.org/10.1037/a0017661
Swan, G. E., & Carmelli, D. (2002). Evidence for genetic mediation of executive control: a study of aging male twins. The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences, 57, 133–143. https://doi.org/10.1093/geronb/57.2.P133 DOI: https://doi.org/10.1093/geronb/57.2.P133
Tien, A. Y., Spevack, T. V., Jones, D. W., Pearlson, G. D., Schlaepfer, T. E., & Strauss, M. E. (1996). Computerized Wisconsin Card Sorting Test: Comparison with manual administration. Kaohsiung Journal of Medical Sciences, 12, 479–485.
Troyer, A. K., Moscovitch, M., & Winocur, G. (1997). Clustering and switching as two components of verbal fluency: Evidence from younger and older healthy adults. Neuropsychology, 11(1), 138–146. https://doi.org/10.1037/0894-4105.11.1.138 DOI: https://doi.org/10.1037//0894-4105.11.1.138
Tyburski, E., Kerestey, M., Kerestey, P., Radoń, S., & Mueller, S. T. (2021). Assessment of motor planning and inhibition performance in non-clinical sample—reliability and factor structure of the Tower of London and Go/No Go computerized tasks. Brain Sciences, 11(11), 1420. https://doi.org/10.3390/brainsci11111420 DOI: https://doi.org/10.3390/brainsci11111420
Vernon, P. A., & Jensen, A. R. (1984). Individual and group differences in intelligence and speed of information processing. Personality and Individual Differences, 5(4), 411–423. https://doi.org/10.1016/0191-8869(84)90006-0 DOI: https://doi.org/10.1016/0191-8869(84)90006-0
Welsh, M. C., Cicerello, A., Cuneo, K., & Brennan, M. (1995). Error and temporal patterns in Tower of Hanoi performance: Cognitive mechanisms and individual differences. Journal of General Psychology, 122(1), 69-81. https://doi.org/10.1080/00221309.1995.9914919 DOI: https://doi.org/10.1080/00221309.1995.9921223
Vermeent, S., Dotsch, R., Schmand, B., Klaming, L., Miller, J. B., & van Elswijk, G. (2020). Evidence of validity for a newly developed digital cognitive test battery. Frontiers in Psychology, 11, 770. https://doi.org/10.3389/fpsyg.2020.00770 DOI: https://doi.org/10.3389/fpsyg.2020.00770
Wager, T. D., Sylvester, C. Y., Lacey, S. C., Nee, D. E., Franklin, M., & Jonides, J. (2005). Common and unique components of response inhibition revealed by fMRI. NeuroImage, 27(2), 323–340. https://doi.org/10.1016/j.neuroimage.2005.01.054 DOI: https://doi.org/10.1016/j.neuroimage.2005.01.054
Wagner, G. P., & Trentini, C. M. (2009). Assessing executive functions in older adults: a comparison between the manual and the computer-based versions of the Wisconsin Card Sorting Test. Psychology & Neuroscience, 2, 195-198. https://doi.org/10.3922/j.psns.2009.2.011 DOI: https://doi.org/10.3922/j.psns.2009.2.011
Welsh, M. C., Pennington, B. F., & Groisser, D. B. (1991). A normative developmental study of executive function: A window on prefrontal function in children. Developmental Neuropsychology, 7(2), 131-149. https://doi.org/10.1080/87565649109540577 DOI: https://doi.org/10.1080/87565649109540483
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Željka Nikolašević, Bojana M. Dinić, Milan Oljača, Ilija Milovanović, Jasmina Kodžopeljić, Vojislava Bugarski Ignjatović
This work is licensed under a Creative Commons Attribution 4.0 International License.
