Times are displayed in (UTC-05:00) Central Time (US & Canada) Change
About this paper symposium
| Panel information |
|---|
| Panel 16. Prevention and Interventions |
| Paper #1 | |
|---|---|
| Explaining Fade-Out and Persistence of Intervention Effects: Meta-Analytic Evidence | |
| Author information | Role |
| Jens Dietrichson, VIVE- The Danish Center for Social Research, Denmark | Presenting author |
| Abstract | |
|
Introduction The effects of preschool and school interventions are typically larger at the end-of-intervention than at follow-up, i.e., effect sizes fade out over time (e.g., Abenavoli, 2019; Bailey, Duncan, et al., 2020; Hart et al., 2024). Why fade-out occurs and sometimes co-exists with persistent effects is not fully understood (e.g., Bailey et al., 2017; Pages et al., 2022). Learning why is important because a) assessments of the returns to intervention depend on why fade-out occurs, b) some explanations of fade-out are compatible with leading theories of skill development and others are not, and, in turn, c) knowledge about which interventions have high returns and which theories best explain skill development may lead to better interventions. Hypotheses This study uses meta-analytic methods to examine a range of hypotheses for which explanations of fade-out make different predictions. Some examples are (with example explanations mentioned in parentheses): is there fade-out in samples that are identical at post and follow-up (for which the sample composition is not the reason for fadeout); if fade-out is related to the duration between the post and follow-up (against the prediction of regression to the mean); if there is more fade-out when the domains tested at post and follow-up are less overlapping (which is the prediction if the control group catches up due to lack of transfer); if the treatment group’s scores decreases from post to follow-up on identical test items (which they should if treated students forget the intervention content); if there is fade-out of effect sizes based on identical test items (against the prediction of an explanation based on vertical scaling of tests); and if the treatment and control group develops in the same pace as groups outside of the intervention (which explanations based on negative (positive) side-effects on treatment (control) group investments say they should not). Study Population The current analysis sample include 42 studies and 677 effect sizes from school interventions. All include a majority of at-risk students and outcomes are standardized tests in mathematics and reading. Median follow-up length is 12 months, and the minimum and maximum is 1 month and 10 years. We are in the process of substantially expanding this data set through a systematic search of studies of developmental and educational interventions with a longitudinal follow-up. Methods We use random effects meta-regression models to test the hypotheses. All models are versions of the correlated-hierarchical effects models developed by Pustejovsky and Tipton (2022). Preliminary Results We find substantial fade-out on average (around 50%) and most support for, and no robust evidence against, explanations of fade-out based on negative side effects on investments for the treatment group, and control group catch-up due to the treatment group being unable to build on the skills learned in the intervention. Our results indicate that some, but far from all, fade-out is a statistical artifact and that treatment group skills growing more slowly than the control group’s skills after the end-of-intervention is an important factor. |
|
| Paper #2 | |||
|---|---|---|---|
| Examining whether intervention characteristics affect follow-up impacts above and beyond short-term impacts | |||
| Author information | Role | ||
| Tyler W. Watts, Teachers College, Columbia University, United States | Presenting author | ||
| Emma R. Hart, Teachers College, Columbia University, United States | Non-presenting author | ||
| Drew H. Bailey, University of California, Irvine, United States | Non-presenting author | ||
| Abstract | |||
|
Introduction Systematic reviews have found that short-term intervention impacts do not necessarily lead to persistent skill advantages (Bailey et al., 2020). As researchers have attempted to account for the factors that could explain why some interventions produce more durable impacts than others, many have pointed to the reasonable possibility that intervention characteristics could provide crucial explanatory power. Myriad factors have been implicated as possibilities in the literature. Plausible examples include the child's age during the intervention, intervention intensity, and involvement of parents. These possibilities raise a crucial question for researchers: if two programs produced the same impact on a given skill at the end of an intervention (e.g., two programs produce identical impacts of 0.30 SDs on a measure of reading achievement), but they produced these impacts by affecting highly varying developmental processes, can we predict which intervention will produce longer-lasting effects? Method We compiled the Meta-Analysis of Educational RCTs with Follow-up (MERF; see Hart et al., in press) dataset to examine whether observable intervention factors could explain the persistence or fadeout of intervention effects. Our procedures included a substantial screening and searching process, which led to 85 RCTs that reported impacts at post-test and follow-up on measures of child skills and behaviors. The sample included a broad set of interventions, such as early childhood programs, adolescent substance use programs, social-emotional interventions, and reading tutoring. Every study was dual-coded for intervention features, study design elements, and impact magnitudes. Our primary analyses relied on a straightforward meta-regression approach. We first examined the extent to which follow-up impacts on child skills were explained by post-test impacts, as we hypothesized that post-test impact magnitude would be the best predictor of follow-up impact magnitude. For this meta-regression, we used a random effects model with study-level with random intercepts and slopes included. Next, we introduced a set of measures capturing intervention characteristics to examine if observable features of interventions predicted the magnitude of the follow-up impact over and above the post-test impact. Results The average post-test impact in our dataset was 0.28 SDs (p < 0.001), and this fell to 0.20 SDs (p < 0.001) at 6-months to 1-year follow-up. As expected, we found that post-test impacts were strong predictors of follow-up impacts at one year (b = .43, p< .001). Importantly, we observed substantial unexplained heterogeneity in follow-up effects after controlling for the post-test effect size (SD = 0.11). However, we found that our large set of intervention features, which included variables measuring intervention intensity, child developmental period, and intervention skills targeted, did not further explain this heterogeneity (see Table 1). Discussion Our findings provide two interesting takeaways for researchers and intervention developers. First, when forecasting follow-up effects, it appears that two interventions producing similar post-test impacts might have similar longer-term trajectories, even if these interventions were qualitatively dissimilar. Second, although studies do differ from one another in their predicted follow-up impact, we have limited ability to predict a priori which intervention features will matter most for influencing longer-term outcomes. |
|||
| Paper #3 | |
|---|---|
| Can we forecast when educational interventions will find impacts on adult outcomes? | |
| Author information | Role |
| Emma R. Hart, Teachers College, Columbia University, United States | Presenting author |
| Casey Moran, Teachers College, Columbia University, United States | Non-presenting author |
| Drew H. Bailey, University of California-Irvine, United States | Non-presenting author |
| Tyler W. Watts, Teachers College, Columbia University, United States | Non-presenting author |
| Abstract | |
|
Two patterns have emerged from educational intervention evaluations that have complicated developmental theory. First, although many evaluations find initial intervention impacts on child skills, such impacts commonly fade in the years following program end (Bailey et al., 2017; Hart et al., in press). Second, a small set of rigorous intervention evaluations have found long-run intervention effects on adult outcomes despite short-term fadeout (Chetty et al., 2011; Deming, 2009; Gray-Lobe et al., 2023). Together, evidence of fadeout and emergence raises many questions, namely: If sustained skill impacts are not a necessary condition for long-run effects, what information can be used to forecast when interventions will have long-run effects? Indeed, although interventions are hypothesized to have long-run effects through initial and sustained boosts to child skills, it is unclear from the current literature whether short-term impacts on skills provide the theoretical basis for longer-term impacts on adult outcomes. Understanding this process is crucial for the accurate explication of developmental processes that link early life competencies to later adult thriving. The current study aimed to bring clarity to this area by meta-analyzing intervention impacts from all existing educational intervention RCTs with adult follow-up. We set out to identify: 1) whether the ‘fadeout-emergence’ paradox was observed beyond the small set of well-known cases; 2) whether short-term intervention impacts forecast when interventions will find long-run impacts on adult outcomes, providing a causal link between early skills and later adult functioning. Building on the original Meta-Analysis of Educational RCTs with Follow-up (MERF; see Hart et al., in press) sample, we systematically identified all educational RCTs targeting child skills prior to high school with impacts assessed at post-test and in adulthood. Our review of 7,201 abstracts yielded 29 treatment-control contrasts from 25 studies. We double-coded and discrepancy-checked 104 papers reporting impacts on these interventions with 89% reliability. In a meta-analytic framework with study-level random effects and weighting by inverse sampling variances, we observed an average adult impact of .11 SD (p < .001) that varied slightly by outcome domain (Figure 1). We also observed fadeout on child skills measured consistently at post-test and at short-term follow-up, with post-test impacts persisting at a rate of 61% at one-year follow-up and 23% at two-year follow-up. Preliminary analyses using the average post-test and adult follow-up impact from each study suggested that larger post-test effects appeared to be predictive of larger adult impacts (β = .13), though this association was not statistically significant (Figure 2). We found evidence for fading intervention impacts on child skills followed by meaningful long-run impact on adult outcomes, suggesting some empirical basis for the fadeout/emergence paradox. We also found suggestive evidence that post-test impacts may forecast long-run intervention effects, though additional analysis is needed prior to SRCD to probe our confidence in this association across intervention types and models. This work has implications for shaping investments in educational programs and policies and refining developmental theory. Note: We recently presented a preliminary version of this work at SREE, but will significantly expand our analyses for SRCD. |
|
⇦ Back to session
Do Educational Interventions Produce Long-Term Effects? Using Meta-Analytic Data to Examine Persistence and Fadeout
Submission Type
Paper Symposium
Description
| Session Title | Do Educational Interventions Produce Long-Term Effects? Using Meta-Analytic Data to Examine Persistence and Fadeout |