Changing the Success Probability in Computerized Adaptive Testing: A Monte-Carlo Simultion on the Open Matrices Item Bank

International Society for Technology, Education, and Science. 944 Maysey Drive, San Antonio, TX 78227. Tel: 515-294-1075; Fax: 515-294-1003; email: [email protected]; Web site: http://www.istes.org

Y

10

Success; Probability; Computer Assisted Testing; Adaptive Testing; Monte Carlo Methods; Matrices; Test Items; Item Banks; Algorithms; Item Response Theory; Test Wiseness

For efficiency, Computerized Adaptive Test (CAT) algorithm selects items with the maximum information, typically with a 50% probability of being answered correctly. However, examinees may not be satisfied if they only correctly answer 50% of the items. Researchers discovered that changing the item selection algorithms to choose easier items (i.e., success probability > 50%), albeit not optimum from a measurement efficiency standpoint, would provide a better experience. The current study aims to investigate the impact of changing the success probability on measurement efficiency. A Monte-Carlo simulation was performed on the Open Matrices Item Bank and simulated item bank. A total of 1500 examinees were generated. We modified the item selection algorithm with the expected success probability of 60%, 70%, and 80%. Each examinee was assigned to five item selection methods: maximum-information, random, p=0.6, p=0.7, and p=0.8. The results indicated that traditional CAT was 60-70% shorter than random item selection. Altering the success probability did not affect the estimation of the examinee's ability. Increasing the probability of success in CAT increased the number of items required to achieve specified levels of precision. Practical considerations on how to maximize the trade-off between examinees' experiences and measurement efficiency are mentioned in the discussion. [For the full proceedings, see ED654100.]

As Provided

2024

ED654460