Education Week - April 24, 2013 - Special Report - S5

EDUCATION WEEK APRIL 24, 2013
Industry & Innovation > www.edweek.org/go/i&ireport
n

Beta Testing Ed. Products
Can Get Tricky for Schools
City College of Technology (which is part of the cuny system) and others
joined Mr. Davis in crafting the school’s academic focus and structure.
The model they established created a school that offers students a curriculum in core academic subjects, but also provides them with an associate degree in applied science, in either computer information systems or
electromechanical engineering technology, which is awarded by the College of Technology. The college had a major role, along with ibm officials, in
drawing up the curriculum and courses.
Students at the school pay no costs, either in tuition or fees, in taking
courses in pursuit of the degree. Organizers expect that students typically
will complete the program in six years, though they could finish sooner.
And p-tech officials believe graduates will be prepared to enter the
workforce. School leaders also say students will be ready to pursue bachelor’s degrees, if they choose that route.
P-tech is housed on the campus of Paul Robeson High School, a public
school that has struggled academically and is scheduled to be closed by
the city. About 85 percent of p-tech’s students receive free or reduced-price
lunches, and the same percentage is African-American.
P-tech is part of the School Improvement Grant program, a federal effort aimed at turning around low-performing, economically disadvantaged
schools. That program has a mixed record of success, despite a substantial
federal investment. (See Education Week, Dec. 5, 2012.)
The school has an extended day and an extended school year, features
that have become increasingly common in other public school environments, including at turnaround sites and in charter schools.

Company Connections
Mayor Bloomberg, who controls the New York City school system, has
given principals considerable autonomy in personnel decisions and other
policies. P-tech’s Mr. Davis says that authority has helped him direct
school resources to where they’re needed most—such as by increasing
the teaching time devoted to certain subjects (and the compensation for
educators putting in extra time in those classes), particularly for students
who need extra help.
Ibm and university officials worked with p-tech administrators to map
the skills needed for an entry-level job in the technology industry onto the
school’s curriculum. Each student is also expected to follow a personalized
academic pathway, based on his or her academic strengths, weaknesses,
and interests.
Faculty members from the school’s university partners teach collegelevel classes. More than 70 students at the school are taking a college
course now, a number that will increase over time, said Mr. Davis.
Ibm also has a more direct involvement in students’ lives. The mentors
provided by the company are expected to meet with students two or three
times a year, and they also interact with them through an online platform
established by the school.
Mentors can help students with individual assignments and projects,
but the hope is that they will also shape students’ thoughts about their
careers, said Temeca Simpson, the ibm liaison at the school and a full-time
employee of the company who works on campus.
The mentors help students understand “where they could potentially be
in four, five, six, seven years,” said Ms. Simpson, a former teacher.
For one of those mentors, Karen Thompson, the pairing with a student
named Nyaisa Galloway, 16, gave her the chance to tutor a teenager on
the skills needed to land a good job, and to succeed in it—knowledge that
many young people too often lack, she said.
Ms. Thompson, a project manager at ibm, said she’s given the p-tech student advice on issues ranging from test preparation to the importance of
getting a college degree. She’s also determined to convey a more personal
message to the student, who, like Ms. Thompson, is African-American.
“I want her to understand that there’s nothing off limits to her,” said
Ms. Thompson. And if the teenager wants to pursue a career in a math,
science, or technology field, she will know “what that really entails, and
‘these are the steps I should take.’ ”
The student she’s paired with said the idea of obtaining a college degree
through p-tech was a major attraction to her, as was the opportunity to
get experience in hands-on, technical fields. Ms. Galloway says her adult
mentor has urged her to prepare academically—and to persist.
“The first thing she says is never give up,” said Ms. Galloway, who is in
her second year at the school. “Always work toward your goals.”
When students complete the program, they will be first in line for jobs
PAGE S6 >

eta testing is one of the most
basic steps on the path to getting education products and
ideas into the classroom—and,
researchers and developers say,
one of the trickiest to get right.
As applied to K-12, the term
generally refers to the early
stage of testing almost any product in schools—
a game, an assessment, a software package, a
personal computer—and then refining it, based
on how well it works.
School districts typically agree to take part in
beta testing because they will get access to a
product or service they believe will help them—
which could be a new learning tool for their students or a professional-development program
for their staff members.
Yet beta testing can also pose challenges for
both schools and product developers. When
superintendents and school boards agree to
beta-test a product, they know that they may
be creating extra work for teachers and administrators who need to be trained in how to use
it, and that their schools may be forced to carve
out class time for trying it out.
Those concerns, among others, sometimes lead
districts to reject offers to stage beta tests, to
the frustration of developers desperate to take
products for test flights in schools.
To complicate matters, beta testing can mean
very different things within the research community, and within schools.
As in other fields, like medicine, education
beta tests are often defined as efforts to test
something through a series of pilots or experiments to see if a product does what it’s supposed
to before it goes live on the market, said Grover
J. “Russ” Whitehurst, a former director of the Institute of Education Sciences, the main research
arm of the U.S. Department of Education.
In other cases, a beta test can include a far
more structured process to evaluate something in a scientifically rigorous fashion, such
as through a randomized control trial, said Mr.
Whitehurst, who now directs the Brown Center,
a research division at the Brookings Institution
in Washington focused on school improvement.
In those experiments, one group of students
might be exposed to a product, while a second
one is not, and results are compared between
the two groups.
In any case, the process is “critical for any
product that has any degree of innovation in it,”
said Mr. Whitehurst, who conducted beta tests
on reading and assessment products he helped
develop before joining the federal government.
“If the developer intends for it to accomplish
something that hasn’t been accomplished before, it’s absolutely necessary,” he said.

Strengths and Shortcomings
Finding a district willing to act as a partner
in beta testing is one thing; staging a successful
pilot test that leads to a product breaking into
the K-12 market is something else.
Elliot Soloway, a professor of computer science
and engineering at the University of Michigan
in Ann Arbor, who has conducted several beta
tests as a product developer, has experienced
that reality firsthand.

BY SEAN CAVANAGH
Several years ago, Mr. Soloway said, he
worked with a company that was beta-testing
pocket personal computers—meant to serve
as “mobile learning environments”—among a
relatively small group of teachers and about
150 students in a school district in an Eastern
state. (He declined to name the school system
or the state.) The teachers in the group quickly
became adept at using the tool. Students’ test
scores improved. So did their record in turning
in homework on time.
“It went swimmingly,” Mr. Soloway recalled.
The trouble started when the product was
scaled up for use by more than 50 teachers and
1,500 students. Costs rose. While the system’s
software and hardware worked well in the original test group, unexpected problems emerged
when teachers in the larger group used the devices, probably because those educators lacked
the tech-savvy to resolve glitches on their own,
Mr. Soloway speculated. Teachers confused by
the technology tended to stop using the devices
“dead in their tracks,” without providing developers with enough information to diagnose the
problem, he said.
Another complication: While teachers in the
original test group were able to craft learning
activities to make use of the technology’s features, teachers in the larger group were unable
to figure out how to make the technology work
for their curricular needs—and believed doing
so simply added to their workload, he added.
The pocket personal computer system never
made it to the marketplace.
The project unraveled because of a computer
“bug rebellion” and a “teacher rebellion,” Mr.
Soloway said. “It was horrible. ... We got lured
into thinking we had a successful pilot; we can
roll it into the rest of the school. Not a chance.”
The most obvious temptation among developers is to structure the beta test in a way that
will heighten the chances a product will succeed, which can undermine the process, Mr.
Soloway said.
In those circumstances, “of course it’s going to
succeed,” he said, but the developer finds “you
don’t learn enough” about the product’s true
strengths and shortcomings.
Mr. Soloway still consults with education
companies, and he still does beta testing, but
he says he has learned from earlier setbacks.
Today, he says he pays more attention to providing teachers with tech support, and help crafting lessons that mesh with the technology. He
adds that experience has taught him that developers and district officials need to work closely
together to choose a range of teachers and
students with different backgrounds, who are
likely to have different degrees of willingness
to accept the technology, rather than cherrypicking participants.
Michael Casserly, the executive director of the
Washington-based Council of the Great City
Schools, said that when district leaders resist
allowing beta testing in their schools, it’s often
because they worry that process isn’t transparent, and that the results may not be portrayed
accurately. Concerns about creating new administrative burdens and training tend to be less
important, he said.
The council will agree to help coordinate or
recommend beta tests among its 67 member
districts only if developers agree to make the rePAGE S7 >

http://www.edweek.org/go/i&ireport

Education Week - April 24, 2013 - Special Report

Table of Contents for the Digital Edition of Education Week - April 24, 2013 - Special Report