Name: Yan Yan

Ph.D. Dissertation Defense Meeting

Date: Wednesday, April 26, 2023

Time: 10:00 AM

Location: Zoom click here 

Meeting ID: 604 352 6671

Passcode: 512856  

Advisor: Susan Embretson, Ph.D. (Georgia Tech)

Dissertation Committee Members:

James Roberts, Ph.D. (Georgia Tech)

Rickey Thomas, Ph.D. (Georgia Tech)

Michael Hunter, Ph.D. (Penn State University)

Kirk Becker, Ph.D. (Pearson VUE)

Title: Modeling The Variability of Automatically Generated Items: The Impact of Incidental Effect on Ability Estimation

Abstract: The advent of computerized testing and more frequent test administration call for a large number of items that can be used to supply item banks. Automatic item generation (AIG) is a comparatively innovative area in assessment and measurement research, where specific cognitive and psychometric propositions are applied to test construction practices to efficiently produce test items using technology (Gierl & Haladyna, 2013). The current work reviewed the historical background of AIG, the development approaches of AIG, and the statistical modeling and evaluation of AIG items. A simulation study was then conducted to:: 1.) investigate the performances of various models for AIG item calibration when different levels of variability lie in item families; 2.) investigate the impact of the incidental effects on candidate ability estimation; and 3.) evaluate test outcomes when item variants are used with known item parameters vs. when recalibration happens on AIG items (estimated item parameters). The results show that, first, the level of similarity within the item family plays a role in the effectiveness of ways to model generated items. Second, adding the incidental effects in item difficulty modeling can improve the model fit statistics and accuracy of the ability measure, especially when the variability of item difficulty within the family is relatively large. Third, the recovery of ability parameters can be impacted by different conditions, including the levels of variability within the item family, the estimation models used, and whether item parameters are estimated or known. Furthermore, it is confirmed that with high-quality pre-calibrated parameters (including known incidental effects) for item models, the item calibration procedure for the automatically generated items can be simplified.