About this srcd poster session

---

Panel information
Panel 8. Education, Schooling

Abstract

---

Early math skills are essential to short- and long-term achievement with performance gaps emerging in preschool (Watts et al., 2014). Preschoolers with risks tend to show slower growth over time in early math learning, suggesting urgency for early math instruction (Hojnoski et al., 2018; Lambert et al., 2014).

However, few multidimensional mathematics assessments that inform instruction are readily available to PreK educators (National Research Council, 2009). LLAMA, a tablet-based preK math assessment inclusive of items from numeracy, geometry, pre-algebraic thinking, and measurement domains was developed to address these concerns to support educators’ data-based decision making and instructional planning for all students.

Our research questions across the 4-year project were informed by standards for psychometric quality, including fairness in testing; social validity, and anti-bias in measurement (Randall, 2021). Phase 1: To what extent do empirical key math skills translate to task models and items according to teacher and expert input? Phase 2: How do facets of administration (e.g., context, child choice, audio v. in-person administration) influence child performance? Phases 3&4: 1) Do LLAMA items meet high psychometric standards? and 2) Do LLAMA item banks provide comprehensive coverage for each learning progression?

Participants. Over 700 preschoolers from 4 states participated. Children were diverse in language, race, ethnicity, disability, and socioeconomic status and were enrolled in a range of preschool settings (e.g., public and state-funded preschool, Head Start, ECSE).

Measures and Procedures. We used an iterative development process with mixed methods and phases to develop LLAMA. In Year 1, we created construct maps; developed items; and gathered child, teacher, and expert feedback. In Year 2, we revised items; examined administration procedures for equity and access; and gathered parent feedback. In Year 3, we piloted LLAMA with over 300 preschool children in 4 states, and evaluated item difficulty, discrimination, model fit, and differential item functioning. In Year 4, we tested LLAMA with another 300 preschool children in 4 states to inform the computer adaptive-testing platform that delivers LLAMA items.

This design allowed us to incorporate participant feedback into each phase. We built LLAMA following Wilson’s (2005) framework for constructing measures, with item design guided by construct maps that represent developmental theory, and with results evaluated through psychometric analysis and Rasch modeling. Results from each phase informed the next, including feedback from practitioner and expert partners.

Results. Phase 1: data showed clear learning pathways for 4 preK math domains, and that learning pathways translate to task models for item development and to ordered items for each task model. Phase 2: items tested with diverse groups of preK children in four contexts (i.e., outer space, jungle, ocean) based on child choice of item context showed that context was not related to child performance based on race/ethnicity or ability. Phases 3&4: findings showed evidence of item reliability across item difficulty, leading to a fully developed computer-adaptive prototyped test for all learners. We evaluated item difficulty level, discrimination, model fit, and differential item functioning for subpopulations.

Author information

Author	Role
Kristen Missall, Ph.D., University of Washington	Presenting author
Robin Hojnoski, PhD, Lehigh University	Non-presenting author
Anthony Albano, PhD, University of California-Davis	Non-presenting author
David Purpura, PhD, Purdue University	Non-presenting author

⇦ Back to session