Education Week - April 24, 2013 - Special Report - (Page S5)

| S5 EDUCATION WEEK APRIL 24, 2013 Industry & Innovation > n Beta Testing Ed. Products Can Get Tricky for Schools City College of Technology (which is part of the cuny system) and others joined Mr. Davis in crafting the school’s academic focus and structure. The model they established created a school that offers students a curriculum in core academic subjects, but also provides them with an associate degree in applied science, in either computer information systems or electromechanical engineering technology, which is awarded by the College of Technology. The college had a major role, along with ibm officials, in drawing up the curriculum and courses. Students at the school pay no costs, either in tuition or fees, in taking courses in pursuit of the degree. Organizers expect that students typically will complete the program in six years, though they could finish sooner. And p-tech officials believe graduates will be prepared to enter the workforce. School leaders also say students will be ready to pursue bachelor’s degrees, if they choose that route. P-tech is housed on the campus of Paul Robeson High School, a public school that has struggled academically and is scheduled to be closed by the city. About 85 percent of p-tech’s students receive free or reduced-price lunches, and the same percentage is African-American. P-tech is part of the School Improvement Grant program, a federal effort aimed at turning around low-performing, economically disadvantaged schools. That program has a mixed record of success, despite a substantial federal investment. (See Education Week, Dec. 5, 2012.) The school has an extended day and an extended school year, features that have become increasingly common in other public school environments, including at turnaround sites and in charter schools. Company Connections Mayor Bloomberg, who controls the New York City school system, has given principals considerable autonomy in personnel decisions and other policies. P-tech’s Mr. Davis says that authority has helped him direct school resources to where they’re needed most—such as by increasing the teaching time devoted to certain subjects (and the compensation for educators putting in extra time in those classes), particularly for students who need extra help. Ibm and university officials worked with p-tech administrators to map the skills needed for an entry-level job in the technology industry onto the school’s curriculum. Each student is also expected to follow a personalized academic pathway, based on his or her academic strengths, weaknesses, and interests. Faculty members from the school’s university partners teach collegelevel classes. More than 70 students at the school are taking a college course now, a number that will increase over time, said Mr. Davis. Ibm also has a more direct involvement in students’ lives. The mentors provided by the company are expected to meet with students two or three times a year, and they also interact with them through an online platform established by the school. Mentors can help students with individual assignments and projects, but the hope is that they will also shape students’ thoughts about their careers, said Temeca Simpson, the ibm liaison at the school and a full-time employee of the company who works on campus. The mentors help students understand “where they could potentially be in four, five, six, seven years,” said Ms. Simpson, a former teacher. For one of those mentors, Karen Thompson, the pairing with a student named Nyaisa Galloway, 16, gave her the chance to tutor a teenager on the skills needed to land a good job, and to succeed in it—knowledge that many young people too often lack, she said. Ms. Thompson, a project manager at ibm, said she’s given the p-tech student advice on issues ranging from test preparation to the importance of getting a college degree. She’s also determined to convey a more personal message to the student, who, like Ms. Thompson, is African-American. “I want her to understand that there’s nothing off limits to her,” said Ms. Thompson. And if the teenager wants to pursue a career in a math, science, or technology field, she will know “what that really entails, and ‘these are the steps I should take.’ ” The student she’s paired with said the idea of obtaining a college degree through p-tech was a major attraction to her, as was the opportunity to get experience in hands-on, technical fields. Ms. Galloway says her adult mentor has urged her to prepare academically—and to persist. “The first thing she says is never give up,” said Ms. Galloway, who is in her second year at the school. “Always work toward your goals.” When students complete the program, they will be first in line for jobs PAGE S6 > B eta testing is one of the most basic steps on the path to getting education products and ideas into the classroom—and, researchers and developers say, one of the trickiest to get right. As applied to K-12, the term generally refers to the early stage of testing almost any product in schools— a game, an assessment, a software package, a personal computer—and then refining it, based on how well it works. School districts typically agree to take part in beta testing because they will get access to a product or service they believe will help them— which could be a new learning tool for their students or a professional-development program for their staff members. Yet beta testing can also pose challenges for both schools and product developers. When superintendents and school boards agree to beta-test a product, they know that they may be creating extra work for teachers and administrators who need to be trained in how to use it, and that their schools may be forced to carve out class time for trying it out. Those concerns, among others, sometimes lead districts to reject offers to stage beta tests, to the frustration of developers desperate to take products for test flights in schools. To complicate matters, beta testing can mean very different things within the research community, and within schools. As in other fields, like medicine, education beta tests are often defined as efforts to test something through a series of pilots or experiments to see if a product does what it’s supposed to before it goes live on the market, said Grover J. “Russ” Whitehurst, a former director of the Institute of Education Sciences, the main research arm of the U.S. Department of Education. In other cases, a beta test can include a far more structured process to evaluate something in a scientifically rigorous fashion, such as through a randomized control trial, said Mr. Whitehurst, who now directs the Brown Center, a research division at the Brookings Institution in Washington focused on school improvement. In those experiments, one group of students might be exposed to a product, while a second one is not, and results are compared between the two groups. In any case, the process is “critical for any product that has any degree of innovation in it,” said Mr. Whitehurst, who conducted beta tests on reading and assessment products he helped develop before joining the federal government. “If the developer intends for it to accomplish something that hasn’t been accomplished before, it’s absolutely necessary,” he said. Strengths and Shortcomings Finding a district willing to act as a partner in beta testing is one thing; staging a successful pilot test that leads to a product breaking into the K-12 market is something else. Elliot Soloway, a professor of computer science and engineering at the University of Michigan in Ann Arbor, who has conducted several beta tests as a product developer, has experienced that reality firsthand. BY SEAN CAVANAGH Several years ago, Mr. Soloway said, he worked with a company that was beta-testing pocket personal computers—meant to serve as “mobile learning environments”—among a relatively small group of teachers and about 150 students in a school district in an Eastern state. (He declined to name the school system or the state.) The teachers in the group quickly became adept at using the tool. Students’ test scores improved. So did their record in turning in homework on time. “It went swimmingly,” Mr. Soloway recalled. The trouble started when the product was scaled up for use by more than 50 teachers and 1,500 students. Costs rose. While the system’s software and hardware worked well in the original test group, unexpected problems emerged when teachers in the larger group used the devices, probably because those educators lacked the tech-savvy to resolve glitches on their own, Mr. Soloway speculated. Teachers confused by the technology tended to stop using the devices “dead in their tracks,” without providing developers with enough information to diagnose the problem, he said. Another complication: While teachers in the original test group were able to craft learning activities to make use of the technology’s features, teachers in the larger group were unable to figure out how to make the technology work for their curricular needs—and believed doing so simply added to their workload, he added. The pocket personal computer system never made it to the marketplace. The project unraveled because of a computer “bug rebellion” and a “teacher rebellion,” Mr. Soloway said. “It was horrible. ... We got lured into thinking we had a successful pilot; we can roll it into the rest of the school. Not a chance.” The most obvious temptation among developers is to structure the beta test in a way that will heighten the chances a product will succeed, which can undermine the process, Mr. Soloway said. In those circumstances, “of course it’s going to succeed,” he said, but the developer finds “you don’t learn enough” about the product’s true strengths and shortcomings. Mr. Soloway still consults with education companies, and he still does beta testing, but he says he has learned from earlier setbacks. Today, he says he pays more attention to providing teachers with tech support, and help crafting lessons that mesh with the technology. He adds that experience has taught him that developers and district officials need to work closely together to choose a range of teachers and students with different backgrounds, who are likely to have different degrees of willingness to accept the technology, rather than cherrypicking participants. Michael Casserly, the executive director of the Washington-based Council of the Great City Schools, said that when district leaders resist allowing beta testing in their schools, it’s often because they worry that process isn’t transparent, and that the results may not be portrayed accurately. Concerns about creating new administrative burdens and training tend to be less important, he said. The council will agree to help coordinate or recommend beta tests among its 67 member districts only if developers agree to make the rePAGE S7 >

Table of Contents for the Digital Edition of Education Week - April 24, 2013 - Special Report

Education Week - April 24, 2013
Education Industry Players Exert Public- Policy Influence
Companies, Policymakers Look For Common Ground
Industry Shapes Goals And Tech Focus at N.Y.C. School
Beta Testing Ed. Products Can Get Tricky for Schools
Vetting Product Research to Determine What Works
Big-Name Companies Feature Larger-Impact Research Efforts
What to Ask About Research
Privatization Choices
À la Carte Purchasing Tactics Signal Districts’ Unique Needs
Big Companies Face K-12 Shift

Education Week - April 24, 2013 - Special Report