Education Week - April 24, 2013 - Special Report - S10

S10

EDUCATION WEEK APRIL 24, 2013
Industry & Innovation > www.edweek.org/go/i&ireport
n

Vetting Product Research
To Determine What Works

E
Educators are
struggling to
balance a
desire for
evidence of
effectiveness
with the need
to try new
approaches

very day, Kenneth Grover, the principal of the
175-student Innovations High School in Salt
Lake City, wades through printed ads and
emails pushing everything from computers to
lighted pens.
“If you read the brochures with beautiful and
happy kids on them, you’re thinking, ‘Wow, this
is what I’ve been looking for,’ ” Mr. Grover said.
From his experience, though, vendors cite
research in their promotions only 20 percent
of the time—and, upon investigation, only about half that research
is conducted by an independent party or self-administered under
strict guidelines.
School districts are bombarded with marketing materials from
companies claiming their products can help change the face of education and raise student achievement. Yet a common complaint is
the lack of significant data to back up the slick slogans.
As educators try to balance a desire for evidence with the need
for innovation, at a time when standards are rising and technologies are advancing at breakneck speed, many find themselves undergoing rapid-fire pilot projects to determine which products and
services best suit their districts.
“If you’re waiting for all the evidence to be fully baked, you’re
going to be waiting a long time,” said Kenneth Zeff, the chief strategy and innovation officer for the 95,000-student Fulton County,
Ga., school system in the Atlanta metropolitan area.
Independent research is often seen as the gold standard for
authenticating effectiveness claims, and the U.S. Department of
Education’s What Works Clearinghouse is considered the leading
source of scientific evidence of what works in education.
But many small companies simply don’t have the budgets to pay
for such research, and the largest, most respected studies cost a
lot and can take years to complete. (Mathematica Policy Research,
for instance, recently completed a seven-year, $4 million study for
the Knowledge Is Power Program, or kipp, charter school network,
which runs 125 schools in 20 states, serving 41,000 students.)
But even small companies with modest resources can demonstrate efficacy, according to the Education Industry Association, a
trade group based in Vienna, Va.
The association has begun encouraging members to seek out
independent validation to set themselves apart from their competitors during the relatively recent “explosion of entrepreneurship into the K-12 marketplace,” said Steve Pines, the association’s
executive director.
“What’s missing,” he said, “is some third-party documentation
that can separate the wheat from the chaff.”

A Discriminating Consumer

BY
ROBIN L. FLANIGAN

That absence of independent research about certain products can
be particularly difficult for smaller districts with even fewer resources.
The 7,000-student Henry County school system in Collinsville,
Va., deals with that situation, in part, by empowering teachers to experiment with free apps, about 10 to 15 per month, which helps the
district do its own research before purchasing a product or service.
Administrators there also look for positive results from districts of
comparable size and demographics before deciding to implement a
program, according to Janet Copenhaver, the district’s director of technology and innovation.
Even large education providers struggle with technological advances
often outpacing the speed of rigorous research.
“That becomes a real challenge for school leaders, because being
able to move quickly and accurately to make decisions in real time
is critical,” said Joseph Olchefske, the president of Mosaica Online at
New York City-based Mosaica Education Inc. The private company
manages 75 schools in the United States and overseas, serving 19,000
students.
Instead, Mr. Olchefske directs his company to look at research for
broad direction, then commit to its own continuous review, analysis,

and evaluation of student performance. He referred to that commitment when addressing criticism for lower-than-average student
achievement data and higher-than-average disciplinary problems,
among other issues, that Mosaica faced recently.
Mosaica examines its results quarterly and makes midcourse corrections, Mr. Olchefske said, which allows for improvements to be made
even without waiting for the golden seal of approval from top-quality
research.
“You can be a discriminating consumer, but what you can’t really
do is get to a place where there’s a definitive conclusion,” he said. “At
some point, you have to take a risk. Research never does away with
the need for judgment.”
The 39,000-student Cherokee County school district in Canton, Ga.,
puts more weight on its own standards than on independent studies.
“It really is about our own research,” said Bobby Blount, the district’s assistant superintendent for accountability, technology, and
strategic planning.
That philosophy was more out of necessity than choice nearly 10
years ago, when the district wanted to begin using new interactive
whiteboards—a market that had not yet been scrutinized by research.
“All we had to go on at that point were sales people telling us how
great and wonderful their product was,” Mr. Blount recalled.
The Cherokee County system decided to perform its own test, installing whiteboards in about a dozen classrooms in various grades.
The whiteboards proved to be effective, improving both teaching and
learning.
When it was time to make the larger investment in a districtwide
rollout, “we had two vendors come in and pretty much do a dog-andpony show,” Mr. Blount said, which led district officials at the time
to choose one company’s product for elementary classrooms and the
other company’s product for middle and high school classrooms.
Cherokee is now piloting five types of math software this school
year, while examining findings about the software from other districts.
“We rely on each other quite a bit,” Mr. Blount said. “That lends
more credence than anything else.”

‘Practical Considerations’
When the 28,000-student Colorado Springs School District 11,
in Colorado, was considering a couple of years ago whether to buy
st Math software from the mind Research Institute, a nonprofit
education research company based in Irvine, Calif., the institute’s
own data-collection process gave administrators a good first impression.
Randomized studies conducted in collaboration with the University of California, Irvine, sweetened the pot. Then visits to see the
program in action at schools in Anaheim, Calif., and Chicago sealed
the deal.
David Sawtelle, the math facilitator for the Colorado Springs district, has learned over time to press vendors who claim little more
than that their products are “research-based.”
“What that turns out to mean is that a product is designed in
accordance with research around best practices, and then there’s a
citation of a study that was done in which that practice was potentially effective,” he said. “We’ve become more discriminating. We ask,
‘If you’re research-based, how is your research validated?’ ”
The right answer to that and other critical questions—such as
how the program was implemented, what kind of professional development is needed, and what is the right environment for it to be
successful—depends on each district’s needs.
“Context is very important,” said Steven M. Ross, the evaluation
director for the Center for Research and Reform in Education at
Johns Hopkins University’s school of education in Baltimore. “It’s
not like picking a prescription out of a box. You have to be much
more nuanced in your selection.”
Smaller companies without the budgets to perform independent
studies can take heart from the fact that research only goes so far,
PAGE S12 >

http://www.edweek.org/go/i&ireport

Education Week - April 24, 2013 - Special Report

Table of Contents for the Digital Edition of Education Week - April 24, 2013 - Special Report