Modeling Multilevel Dependence Structures for Responses and Process Data in a Bayesian Framework (PR 19-03)

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group. In practice, test takers are often part of nested groups, such as schools, classrooms, and workgroups. Applied research has shown that ignoring group membership can result in biased
inferences about the latent constructs of interest. In this report, the BCSM methodology is extended to model, and control for, within- and between-group variations in the BCSM dependence structures. A simulation study illustrates the advantages of multilevel BCSM over the one-level BCSM. Limitations and future prospects of multilevel BCSM are discussed.