Why Curriculum Decision Makers Should Look at All the Evidence

Developing guidance on what works in early education is always challenging and that certainly applies to the difficult business of evaluating and selecting a curriculum. Whenever specific early education curricula are evaluated, judgment calls have to be made on the strengths and weaknesses of the evaluation including such issues as the duration and quality of training in the curriculum prior to the evaluation, how well the measures used actually measure children’s learning and development (are they broad enough and deep enough?), and how well any given curriculum is implemented in the classroom at the time the research was conducted.

When the final results are published, these become the go-to issues when people scratch their heads about why a curriculum did or didn’t do well in the review. Recent efforts to summarize evidence regarding the effects of various curricula have brought these issues front and center. The 2008 federal Preschool Curriculum Evaluation Research (PCER) report found most curricula in the study provided little or no advantage over existing practice. On an ongoing basis, the Institute for Educational Sciences’ What Works Clearinghouse (WWC) provides reviews that yield lackluster ratings for a number of programs.

Among the findings WWC reports is that the Tools of the Mind Curriculum had no significant effects. What may not be clear to readers is that the study WWC reviewed was designed to determine whether Tools of the Mind could produce equivalent academic results while improving results for self-regulation and social behavior compared to a more traditional curriculum that was also expected to produce strong academic gains. Indeed, this is exactly what was found—strong reductions in behavior problems and improvements in self-regulation with academic gains at least as strong as from the other curriculum. However, WWC does not take into account any effects of early childhood curriculum on executive function, self-regulation, or any aspect of social and emotional development.

The latest effort in this arena comes by way of Johns Hopkins University where Bette Chambers and a team of colleagues have taken another look at many of the same preschool programs appearing in the federal efforts. The team conducting the review says 40 studies evaluating 28 different programs met its research standards. Without getting bogged down in details, suffice to say that they made different judgments than did WWC about what studies to include and how to evaluate their results. It should come as no surprise that the results of the Johns Hopkins review differ substantially from those of the federal efforts. The Johns Hopkins team review awards favorable ratings to 11 of the 28 programs and reports that six show “strong” evidence of effectiveness.

Many will be asking whether the Johns Hopkins review provides a better basis for choosing a curriculum than the previous reviews. Not really. Their criterion for “strong evidence of effectiveness” is two findings of effect sizes larger than .20 on any measure regardless of the importance of the outcome. They ignore the nature of the comparison made in the study (was it to another strong curriculum or poor practices?), and they also ignore non-academic outcomes. I respectfully suggest that neither the Johns Hopkins review nor the federal efforts provides an adequate basis for choosing a curriculum. Together they provide useful information but remain incredibly narrow. They still do not completely survey the relevant studies of curricula (for example, rigorous studies of the High/Scope approach), consider many of the outcomes we in early education are trying to produce, or take into account much of the field’s knowledge about what works. Much of this comes from basic and applied research on the education and development of young children that does not follow the “horse race” model for curriculum comparison studies.

On a practical level, the question for curriculum decision makers becomes what resources to consult in deciding on a curriculum for their programs. The best advice is to take a very broad look at the evidence that includes wide-ranging analyses such as NIEER’s meta-analysis of 120 studies; even if this does not point to specific curricula it does help identify characteristics of more effective early education from the broadest set of comparative studies collected to date.

Readers interested in more detailed, concrete advice on how to choose a highly effective curriculum should consult NIEER’s policy brief Preschool Curriculum Decision-Making: Dimensions to Consider.

Steve Barnett
Co-director, NIEER

2 Responses to Why Curriculum Decision Makers Should Look at All the Evidence

  1. Dr. Barnett,

    Thank you for the wonderful comments.

    In reflecting on the issues and supposed ineffectual nature of curricula in early childhood education, I think we should carefully consider the expertise and evaluative skills of the teachers who use them.

    Many teachers of young children across America prefer to use a professionally made curriculum or a curriculum modeled (or “cookie-cut”) from someone else’s plan and implement it in the classroom without any evaluative plan on hand from the beginning. I think how teachers evaluate curricula is the primary factor that will influence which curriculums are chosen, modified, overhauled, or trashed.

    Unfortunately, there are still many teachers who fail to take this vital reflective step, thus their curriculum is ineffective.

  2. Aleida Siles says:

    Wonderful website. A lot of useful information here. I am sending it to some friends ans also sharing in delicious. And of course, thanks for your sweat!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: