Education Week - July 18, 2018 - 9
Gates Teacher-Effectiveness Program Shows No Payoff
By Madeline Will
The Bill & Melinda Gates Foundation's multimillion-dollar, multiyear
effort aimed at making teachers
more effective largely fell short of
its goal to increase student achievement-including among low-income
and minority students, a recent
This conclusion to an expensive
chapter of teacher-evaluation reform underscores the difficulty of
making sweeping, lasting changes
to teacher performance. The evaluation of the program was conducted
by the RAND Corporation with the
American Institutes for Research
and funded by the Gates Foundation.
Under its Intensive Partnerships
for Effective Teaching initiative, the
Gates Foundation gave grants to
three large school districts-Memphis, Tenn. (which merged with
Shelby County during the course
of the initiative); Pittsburgh; and
Hillsborough County, Fla.-and to
one charter school consortium in
California starting in the 2009-10
school year. The foundation poured
$212 million into these partnerships over about six years, and the
districts put up matching funds.
The total cost of the initiative was
The school sites agreed to design
new teacher-evaluation systems
that incorporated classroom-observation rubrics and a measure
of growth in student achievement.
They also agreed to offer individualized professional development
based on teachers' evaluation results, and to revamp recruitment,
hiring, and placement. Schools also
implemented new career pathways
for effective teachers and awarded
teachers with bonuses for good performance.
"The initiative itself tried to pull a
bunch of levers to have a big impact
on student performance," said Brian
Stecher, a RAND researcher and
the lead author of the report. "The
sites did in fact modify all of these
levers, some more than others, but
in the end, there were no big payoffs
in terms of improved graduation
[rates] or achievement of students
in general, and low-income and minority students in particular."
By the end of the 2014-15 school
year, the study found, student outcomes were not significantly better
than outcomes in similar school
sites that did not participate in the
initiative. Researchers also found
no evidence that low-income and
minority students had greater access to effective teachers than their
white, more-affluent peers, which
had been another stated goal of the
Gates Foundation. (Researchers
also collected student outcome data
for the 2015-16 and 2016-17 school
years, and will update the conclusions this fall or next spring.)
A caveat to these results is that
while the initiative was taking
place, high-stakes teacher-evaluation measures were also being
enacted across the country. The research looked at the extent to which
the Gates partnerships improved
student outcomes over and above
the statewide reforms.
Still, at the end of the research
period, very few teachers in participating districts were classified
as ineffective, which researchers
believe is in part due to an unwillingness among school leaders to
give low ratings based on classroom
observations. Also, the sites did
not retain more effective teachers,
although researchers did find declines in the retention of ineffective
"We believe that this work, which
originated in ideas that came from
the field, led to critical conversations and drove change and partnerships across the country," said
Allan Golston, the president of the
U.S. program at the Gates Foundation, in a statement. "We have
taken these lessons to heart, and
they are reflected in the work that
we're doing moving forward."
Last October, the Gates Foundation had announced a major
shift in its investment strategy
for education, pivoting away from
teacher-evaluation efforts entirely.
The foundation plans to pump $1.7
billion into K-12 education, with a
focus on improved curricula that
match state standards for learning
and helping networks of middle and
high schools scale up best practices.
(Education Week receives financial support from the Gates Foundation for coverage of continuous
improvement strategies in education. Education Week retains sole
editorial control of its content.)
'Pushback and Disharmony'
Before this latest pivot, Gates had
devoted more than $700 million to
its teacher-quality agenda. However, the singular focus on teacher
effectiveness in this work might be
one reason student achievement
didn't improve, Stecher said, adding
that maybe factors like early-childhood education, family support, and
child nutrition also need to be addressed.
For that reason, many educators
weren't surprised by the results,
said Ted Dwyer, the chief of data,
research, accountability, and assessment for the Pittsburgh schools.
"A lot of people in districts felt
like there was a disconnect, and
[the initiative] created an enormous amount of focus on the adults
in the system, rather than what really matters in the system, which is
our kids," said Dwyer, who was the
manager of evaluations at the Hillsborough district when the Gates
work first started.
Still, years of research show
that teacher effectiveness is important for student growth, said
Daniel Goldhaber, the director of
the Center for Education Data and
Research at the University of Washington, whose center receives Gates
funding. (Goldhaber is also employed by AIR, but was not involved
in this research.)
"These findings don't undermine
any of the papers that this [initiative] was built on," he said. "It undermines the notion that we have
the political will to do this."
Indeed, the RAND study found
that while all sites initially had approval from most involved parties
to adapt their teacher-evaluation
systems, teachers' unions began to
object a few years into the process.
"When the results started being
used to give cash rewards or to
identify teachers for required planning and ultimately, perhaps, termination, the teacher organizations
reacted defensively," Stecher said.
"[Districts] had to suffer through
a lot of pushback and disharmony."
And that ill will might have influenced evaluation scores, the study
suggests. Over time, fewer and
fewer teachers were identified as
low-performing in most of the sites.
The study found some evidence that
this shift may have been due to increasingly generous ratings on subjective parts of the evaluation like
classroom observations, rather than
an actual improvement in teaching.
Past independent research has
shown that principals rate nearly
all teachers as "effective," but when
principals are asked their opinions
of teachers in confidence, they're
much more likely to give harsh ratings.
The RAND study suggests that
because the initiative had sites use
evaluation results as the basis for
tenure and dismissal decisions,
principals might have avoided giving low observation ratings. Dwyer
said Hillsborough schools, at least,
had safeguards in place to prevent
that scenario from happening.
Another challenge was the rigorous nature of the observation rubric recommended by Gates: If the
evaluation scores were to be used in
personnel decisions, Stecher said,
a principal might need to observe
a teacher for an hour, four times a
year. But shorter classroom dropins might provide helpful, more
immediate feedback for a teacher,
which school leaders preferred.
Over time, some of the sites reduced the length and frequency of
the observations to free up more
time for administrators and to better support teacher improvement.
Districts also didn't have successful models on which to base some of
their reforms, particularly evaluation-linked, professional-development systems, Stecher said. That
made it harder for sites to develop
new, innovative practices.
Although student performance
largely did not improve enough to
meet the initiative's goals, the study
did find some positive effects of the
reforms. For instance, most teachers surveyed in all the districts said
they had become more reflective
about their teaching and had made
changes to their instruction as a result of the evaluation system.
School sites will also keep some
of the practices they used during
the initiative, even without ongoing
Gates support. (Some of the reforms
are now required by law.)
The Gates partnerships will inform future research and initiatives,
researchers and analysts said.
"One of the things philanthropy
can and should do is experiment
and let us learn about what works,"
Hess said. "It was an expensive
experiment, but it was a reasonable hypothesis. ... For good or bad,
we've learned a lot. Not only about
teacher evaluation, but about this
approach to trying to change how
school systems work."
DO TEACHERS THINK STUDENTS WILL
BENEFIT FROM THE EVALUATION REFORM?
Researchers tracked participating teachers' agreement that in the long run,
students will benefit from the teacher-evaluation system. The proportion of
teachers who thought it would help students declined since 2011 in the three
public school districts-most dramatically in Pittsburgh. Enthusiasm is
higher among teachers in the charter schools.
Alliance Public Schools
Aspire Public Schools
The Legacy of Reform
"The big takeaway for me from
this work is that maybe it might
be even harder to go into existing
systems ... and just change them in
terms of their approach to evaluation and professional practice than
we understood," said Frederick M.
Hess, the director of education policy studies at the American Enterprise Institute. (Hess also writes an
opinion blog at edweek.org.)
"It's not just about applying big
gobs of money and consulting and
encouragement and even policy
changes, it's about execution," he
added. "Execution is about dozens
of very small decisions made everyday."
Green Dot Public Schools
*Researchers did not survey teachers in 2012.
SOURCE: RAND Corporation and American Institutes for Research
EDUCATION WEEK | July 18, 2018 | www.edweek.org | 9