Education Week - April 24, 2013 - Special Report - (Page S10)

S10 | EDUCATION WEEK APRIL 24, 2013 Industry & Innovation > www.edweek.org/go/i&ireport n Vetting Product Research To Determine What Works E Educators are struggling to balance a desire for evidence of effectiveness with the need to try new approaches very day, Kenneth Grover, the principal of the 175-student Innovations High School in Salt Lake City, wades through printed ads and emails pushing everything from computers to lighted pens. “If you read the brochures with beautiful and happy kids on them, you’re thinking, ‘Wow, this is what I’ve been looking for,’ ” Mr. Grover said. From his experience, though, vendors cite research in their promotions only 20 percent of the time—and, upon investigation, only about half that research is conducted by an independent party or self-administered under strict guidelines. School districts are bombarded with marketing materials from companies claiming their products can help change the face of education and raise student achievement. Yet a common complaint is the lack of significant data to back up the slick slogans. As educators try to balance a desire for evidence with the need for innovation, at a time when standards are rising and technologies are advancing at breakneck speed, many find themselves undergoing rapid-fire pilot projects to determine which products and services best suit their districts. “If you’re waiting for all the evidence to be fully baked, you’re going to be waiting a long time,” said Kenneth Zeff, the chief strategy and innovation officer for the 95,000-student Fulton County, Ga., school system in the Atlanta metropolitan area. Independent research is often seen as the gold standard for authenticating effectiveness claims, and the U.S. Department of Education’s What Works Clearinghouse is considered the leading source of scientific evidence of what works in education. But many small companies simply don’t have the budgets to pay for such research, and the largest, most respected studies cost a lot and can take years to complete. (Mathematica Policy Research, for instance, recently completed a seven-year, $4 million study for the Knowledge Is Power Program, or kipp, charter school network, which runs 125 schools in 20 states, serving 41,000 students.) But even small companies with modest resources can demonstrate efficacy, according to the Education Industry Association, a trade group based in Vienna, Va. The association has begun encouraging members to seek out independent validation to set themselves apart from their competitors during the relatively recent “explosion of entrepreneurship into the K-12 marketplace,” said Steve Pines, the association’s executive director. “What’s missing,” he said, “is some third-party documentation that can separate the wheat from the chaff.” A Discriminating Consumer BY ROBIN L. FLANIGAN That absence of independent research about certain products can be particularly difficult for smaller districts with even fewer resources. The 7,000-student Henry County school system in Collinsville, Va., deals with that situation, in part, by empowering teachers to experiment with free apps, about 10 to 15 per month, which helps the district do its own research before purchasing a product or service. Administrators there also look for positive results from districts of comparable size and demographics before deciding to implement a program, according to Janet Copenhaver, the district’s director of technology and innovation. Even large education providers struggle with technological advances often outpacing the speed of rigorous research. “That becomes a real challenge for school leaders, because being able to move quickly and accurately to make decisions in real time is critical,” said Joseph Olchefske, the president of Mosaica Online at New York City-based Mosaica Education Inc. The private company manages 75 schools in the United States and overseas, serving 19,000 students. Instead, Mr. Olchefske directs his company to look at research for broad direction, then commit to its own continuous review, analysis, and evaluation of student performance. He referred to that commitment when addressing criticism for lower-than-average student achievement data and higher-than-average disciplinary problems, among other issues, that Mosaica faced recently. Mosaica examines its results quarterly and makes midcourse corrections, Mr. Olchefske said, which allows for improvements to be made even without waiting for the golden seal of approval from top-quality research. “You can be a discriminating consumer, but what you can’t really do is get to a place where there’s a definitive conclusion,” he said. “At some point, you have to take a risk. Research never does away with the need for judgment.” The 39,000-student Cherokee County school district in Canton, Ga., puts more weight on its own standards than on independent studies. “It really is about our own research,” said Bobby Blount, the district’s assistant superintendent for accountability, technology, and strategic planning. That philosophy was more out of necessity than choice nearly 10 years ago, when the district wanted to begin using new interactive whiteboards—a market that had not yet been scrutinized by research. “All we had to go on at that point were sales people telling us how great and wonderful their product was,” Mr. Blount recalled. The Cherokee County system decided to perform its own test, installing whiteboards in about a dozen classrooms in various grades. The whiteboards proved to be effective, improving both teaching and learning. When it was time to make the larger investment in a districtwide rollout, “we had two vendors come in and pretty much do a dog-andpony show,” Mr. Blount said, which led district officials at the time to choose one company’s product for elementary classrooms and the other company’s product for middle and high school classrooms. Cherokee is now piloting five types of math software this school year, while examining findings about the software from other districts. “We rely on each other quite a bit,” Mr. Blount said. “That lends more credence than anything else.” ‘Practical Considerations’ When the 28,000-student Colorado Springs School District 11, in Colorado, was considering a couple of years ago whether to buy st Math software from the mind Research Institute, a nonprofit education research company based in Irvine, Calif., the institute’s own data-collection process gave administrators a good first impression. Randomized studies conducted in collaboration with the University of California, Irvine, sweetened the pot. Then visits to see the program in action at schools in Anaheim, Calif., and Chicago sealed the deal. David Sawtelle, the math facilitator for the Colorado Springs district, has learned over time to press vendors who claim little more than that their products are “research-based.” “What that turns out to mean is that a product is designed in accordance with research around best practices, and then there’s a citation of a study that was done in which that practice was potentially effective,” he said. “We’ve become more discriminating. We ask, ‘If you’re research-based, how is your research validated?’ ” The right answer to that and other critical questions—such as how the program was implemented, what kind of professional development is needed, and what is the right environment for it to be successful—depends on each district’s needs. “Context is very important,” said Steven M. Ross, the evaluation director for the Center for Research and Reform in Education at Johns Hopkins University’s school of education in Baltimore. “It’s not like picking a prescription out of a box. You have to be much more nuanced in your selection.” Smaller companies without the budgets to perform independent studies can take heart from the fact that research only goes so far, PAGE S12 > http://www.edweek.org/go/i&ireport

Table of Contents for the Digital Edition of Education Week - April 24, 2013 - Special Report

Education Week - April 24, 2013
Contents
Education Industry Players Exert Public- Policy Influence
Companies, Policymakers Look For Common Ground
Industry Shapes Goals And Tech Focus at N.Y.C. School
Beta Testing Ed. Products Can Get Tricky for Schools
Vetting Product Research to Determine What Works
Big-Name Companies Feature Larger-Impact Research Efforts
What to Ask About Research
Privatization Choices
À la Carte Purchasing Tactics Signal Districts’ Unique Needs
Big Companies Face K-12 Shift

Education Week - April 24, 2013 - Special Report

http://ew.edweek.org/nxtbooks/epe/ew_sr_03072018
http://ew.edweek.org/nxtbooks/epe/ew_02282018
http://ew.edweek.org/nxtbooks/epe/ew_02212018
http://ew.edweek.org/nxtbooks/epe/ew_02142018
http://ew.edweek.org/nxtbooks/epe/ew_02072018
http://ew.edweek.org/nxtbooks/epe/ew_01242018
http://ew.edweek.org/nxtbooks/epe/ew_01172018
http://ew.edweek.org/nxtbooks/epe/ew_01102018
http://ew.edweek.org/nxtbooks/epe/ew_12132017
http://ew.edweek.org/nxtbooks/epe/ew_11292017
http://ew.edweek.org/nxtbooks/epe/ew_11152017
http://ew.edweek.org/nxtbooks/epe/ew_11082017
http://ew.edweek.org/nxtbooks/epe/ew_11012017
http://ew.edweek.org/nxtbooks/epe/ew_10252017
http://ew.edweek.org/nxtbooks/epe/ew_10182017
http://ew.edweek.org/nxtbooks/epe/ew_10112017
http://ew.edweek.org/nxtbooks/epe/ew_10042017
http://ew.edweek.org/nxtbooks/epe/ew_09272017
http://ew.edweek.org/nxtbooks/epe/ew_09202017
http://ew.edweek.org/nxtbooks/epe/ew_09132017
http://ew.edweek.org/nxtbooks/epe/ew_09062017
http://ew.edweek.org/nxtbooks/epe/ew_08302017
http://ew.edweek.org/nxtbooks/epe/ew_08232017
http://ew.edweek.org/nxtbooks/epe/ew_07192017
http://ew.edweek.org/nxtbooks/epe/ew_06212017
http://ew.edweek.org/nxtbooks/epe/ew_06142017
http://ew.edweek.org/nxtbooks/epe/ew_06072017
http://ew.edweek.org/nxtbooks/epe/ew_05312017
http://ew.edweek.org/nxtbooks/epe/ew_05242017
http://ew.edweek.org/nxtbooks/epe/ew_05172017
http://ew.edweek.org/nxtbooks/epe/ew_05102017
http://ew.edweek.org/nxtbooks/epe/ew_04262017specialreport
http://ew.edweek.org/nxtbooks/epe/ew_04262017
http://ew.edweek.org/nxtbooks/epe/ew_04192017
http://ew.edweek.org/nxtbooks/epe/ew_04052017
http://ew.edweek.org/nxtbooks/epe/ew_03292017
http://ew.edweek.org/nxtbooks/epe/ew_03222017
http://ew.edweek.org/nxtbooks/epe/ew_03082017
http://ew.edweek.org/nxtbooks/epe/ew_03012017
http://ew.edweek.org/nxtbooks/epe/ew_02222017
http://ew.edweek.org/nxtbooks/epe/ew_02152017
http://ew.edweek.org/nxtbooks/epe/ew_02082017
http://ew.edweek.org/nxtbooks/epe/ew_01252017
http://ew.edweek.org/nxtbooks/epe/ew_01252017specialreport
http://ew.edweek.org/nxtbooks/epe/ew_01182017
http://ew.edweek.org/nxtbooks/epe/ew_01042017
http://ew.edweek.org/nxtbooks/epe/ew_12142016
http://ew.edweek.org/nxtbooks/epe/ew_12142016v2
http://ew.edweek.org/nxtbooks/epe/ew_11302016
http://ew.edweek.org/nxtbooks/epe/ew_11162016
http://ew.edweek.org/nxtbooks/epe/ew_11092016
http://ew.edweek.org/nxtbooks/epe/ew_11022016
http://ew.edweek.org/nxtbooks/epe/ew_10262016
http://ew.edweek.org/nxtbooks/epe/ew_10192016
http://ew.edweek.org/nxtbooks/epe/ew_10122016
http://ew.edweek.org/nxtbooks/epe/ew_10052016
http://ew.edweek.org/nxtbooks/epe/ew_09282016
http://ew.edweek.org/nxtbooks/epe/ew_09212016
http://ew.edweek.org/nxtbooks/epe/ew_09142016
http://ew.edweek.org/nxtbooks/epe/ew_09072016
http://ew.edweek.org/nxtbooks/epe/ew_08312016
http://ew.edweek.org/nxtbooks/epe/ew_08242016
http://ew.edweek.org/nxtbooks/epe/ew_08032016
http://ew.edweek.org/nxtbooks/epe/ew_08032016
http://ew.edweek.org/nxtbooks/epe/ew_07202016
http://ew.edweek.org/nxtbooks/epe/ew_01112017
http://ew.edweek.org/nxtbooks/epe/ew_06082016
http://ew.edweek.org/nxtbooks/epe/tc_06092016
http://ew.edweek.org/nxtbooks/epe/dc_06022016
http://ew.edweek.org/nxtbooks/epe/ew_06012016
http://ew.edweek.org/nxtbooks/epe/ew_05182016
http://ew.edweek.org/nxtbooks/epe/ew_05112016
http://ew.edweek.org/nxtbooks/epe/ew_sr_05112016
http://ew.edweek.org/nxtbooks/epe/ew_04272016
http://ew.edweek.org/nxtbooks/epe/ew_04202016
http://ew.edweek.org/nxtbooks/epe/ew_04132016
http://ew.edweek.org/nxtbooks/epe/ew_03302016
http://ew.edweek.org/nxtbooks/epe/ew_sr_03302016
http://ew.edweek.org/nxtbooks/epe/ew_03232016
http://ew.edweek.org/nxtbooks/epe/ew_03162016
http://ew.edweek.org/nxtbooks/epe/ew_03092016
http://ew.edweek.org/nxtbooks/epe/ew_02242016
http://ew.edweek.org/nxtbooks/epe/ew_sr_02242016
http://ew.edweek.org/nxtbooks/epe/ew_02172016
http://ew.edweek.org/nxtbooks/epe/ew_02102016
http://ew.edweek.org/nxtbooks/epe/ew_sr_01272016
http://ew.edweek.org/nxtbooks/epe/ew_01272016
http://ew.edweek.org/nxtbooks/epe/ew_01202016
http://ew.edweek.org/nxtbooks/epe/ew_sr_01132016
http://ew.edweek.org/nxtbooks/epe/ew_01132016
http://ew.edweek.org/nxtbooks/epe/qc_01072016
http://ew.edweek.org/nxtbooks/epe/ew_01062016
http://ew.edweek.org/nxtbooks/epe/ew_12092015
http://ew.edweek.org/nxtbooks/epe/ew_12022015
http://ew.edweek.org/nxtbooks/epe/ew_sr_11112015
http://ew.edweek.org/nxtbooks/epe/ew_11112015
http://ew.edweek.org/nxtbooks/epe/ew_11042015
http://ew.edweek.org/nxtbooks/epe/ew_10282015
http://ew.edweek.org/nxtbooks/epe/ew_10212015
http://ew.edweek.org/nxtbooks/epe/ew_sr_10212015
http://ew.edweek.org/nxtbooks/epe/ew_10142015
http://ew.edweek.org/nxtbooks/epe/ew_10072015
http://ew.edweek.org/nxtbooks/epe/ew_09302015
http://ew.edweek.org/nxtbooks/epe/ew_sr_09302015
http://ew.edweek.org/nxtbooks/epe/ew_09232015
http://ew.edweek.org/nxtbooks/epe/ew_09162015
http://ew.edweek.org/nxtbooks/epe/ew_09092015
http://ew.edweek.org/nxtbooks/epe/ew_08262015
http://ew.edweek.org/nxtbooks/epe/ew_08192015
http://ew.edweek.org/nxtbooks/epe/ew_08052015
http://ew.edweek.org/nxtbooks/epe/ew_07082015
http://ew.edweek.org/nxtbooks/epe/ew_06102015
http://ew.edweek.org/nxtbooks/epe/tc_06112015
http://ew.edweek.org/nxtbooks/epe/dc_06042015
http://ew.edweek.org/nxtbooks/epe/ew_06032015
http://ew.edweek.org/nxtbooks/epe/ew_05202015
http://ew.edweek.org/nxtbooks/epe/ew_sr_05132015
http://ew.edweek.org/nxtbooks/epe/ew_05132015
http://ew.edweek.org/nxtbooks/epe/ew_05062015
http://ew.edweek.org/nxtbooks/epe/ew_04222015
http://ew.edweek.org/nxtbooks/epe/ew_sr_04152015
http://ew.edweek.org/nxtbooks/epe/ew_04152015
http://ew.edweek.org/nxtbooks/epe/ew_04012015
http://ew.edweek.org/nxtbooks/epe/ew_03252015
http://ew.edweek.org/nxtbooks/epe/ew_sr_03182015
http://ew.edweek.org/nxtbooks/epe/ew_03182015
http://ew.edweek.org/nxtbooks/epe/ew_03042015
http://ew.edweek.org/nxtbooks/epe/ew_02252015
http://ew.edweek.org/nxtbooks/epe/ew_sr_02252015
http://ew.edweek.org/nxtbooks/epe/ew_sr_02182015
http://ew.edweek.org/nxtbooks/epe/ew_02182015
http://ew.edweek.org/nxtbooks/epe/ew_02042015
http://ew.edweek.org/nxtbooks/epe/ew_01282015
http://ew.edweek.org/nxtbooks/epe/ew_01212015
http://ew.edweek.org/nxtbooks/epe/ew_sr_01212015
http://ew.edweek.org/nxtbooks/epe/ew_01142015
http://ew.edweek.org/nxtbooks/epe/qc_01082015
http://ew.edweek.org/nxtbooks/epe/ew_01072015
http://ew.edweek.org/nxtbooks/epe/ew_12102014
http://ew.edweek.org/nxtbooks/epe/ew_12032014
http://ew.edweek.org/nxtbooks/epe/ew_11122014
http://ew.edweek.org/nxtbooks/epe/ew_sr_11122014
http://ew.edweek.org/nxtbooks/epe/ew_11052014
http://ew.edweek.org/nxtbooks/epe/ew_10292014
http://ew.edweek.org/nxtbooks/epe/ew_10222014
http://ew.edweek.org/nxtbooks/epe/ew_sr_10222014
http://ew.edweek.org/nxtbooks/epe/ew_10152014
http://ew.edweek.org/nxtbooks/epe/ew_10082014
http://ew.edweek.org/nxtbooks/epe/ew_sr_10012014
http://ew.edweek.org/nxtbooks/epe/ew_10012014
http://ew.edweek.org/nxtbooks/epe/ew_09242014
http://ew.edweek.org/nxtbooks/epe/ew_09172014
http://ew.edweek.org/nxtbooks/epe/ew_09102014
http://ew.edweek.org/nxtbooks/epe/ew_08272014
http://ew.edweek.org/nxtbooks/epe/ew_08202014
http://ew.edweek.org/nxtbooks/epe/ew_sr_08202014
http://ew.edweek.org/nxtbooks/epe/ew_08062014
http://ew.edweek.org/nxtbooks/epe/ew_07092014
http://ew.edweek.org/nxtbooks/epe/ew_06112014_v2
http://ew.edweek.org/nxtbooks/epe/ew_06112014
http://ew.edweek.org/nxtbooks/epe/dc_06052014
http://ew.edweek.org/nxtbooks/epe/ew_06042014
http://ew.edweek.org/nxtbooks/epe/ew_05212014
http://ew.edweek.org/nxtbooks/epe/ew_05142014
http://ew.edweek.org/nxtbooks/epe/ew_05072014
http://ew.edweek.org/nxtbooks/epe/ew_sr_04232014
http://ew.edweek.org/nxtbooks/epe/ew_04232014
http://ew.edweek.org/nxtbooks/epe/ew_04162014
http://ew.edweek.org/nxtbooks/epe/ew_04022014
http://ew.edweek.org/nxtbooks/epe/ew_03262014
http://ew.edweek.org/nxtbooks/epe/tc_20140313
http://ew.edweek.org/nxtbooks/epe/ew_03122014
http://ew.edweek.org/nxtbooks/epe/ew_sr_03052014
http://ew.edweek.org/nxtbooks/epe/ew_03052014
http://ew.edweek.org/nxtbooks/epe/ew_02262014
http://ew.edweek.org/nxtbooks/epe/ew_sr_02192014
http://ew.edweek.org/nxtbooks/epe/ew_02192014
http://ew.edweek.org/nxtbooks/epe/ew_01292014_v2
http://ew.edweek.org/nxtbooks/epe/ew_02052014
http://ew.edweek.org/nxtbooks/epe/ew_01292014
http://ew.edweek.org/nxtbooks/epe/ew_01222014
http://ew.edweek.org/nxtbooks/epe/ew_01152014
http://ew.edweek.org/nxtbooks/epe/qc_01092014
http://ew.edweek.org/nxtbooks/epe/ew_01082014
http://ew.edweek.org/nxtbooks/epe/ew_12112013
http://ew.edweek.org/nxtbooks/epe/ew_12042013
http://ew.edweek.org/nxtbooks/epe/ew_11132013
http://ew.edweek.org/nxtbooks/epe/ew_11062013
http://ew.edweek.org/nxtbooks/epe/ew_sr_10302013
http://ew.edweek.org/nxtbooks/epe/ew_10302013
http://ew.edweek.org/nxtbooks/epe/ew_10232013
http://ew.edweek.org/nxtbooks/epe/ew_10162013
http://ew.edweek.org/nxtbooks/epe/ew_10092013
http://ew.edweek.org/nxtbooks/epe/ew_sr_10022013
http://ew.edweek.org/nxtbooks/epe/ew_10022013
http://ew.edweek.org/nxtbooks/epe/ew_09252013
http://ew.edweek.org/nxtbooks/epe/ew_09182013
http://ew.edweek.org/nxtbooks/epe/ew_09112013
http://ew.edweek.org/nxtbooks/epe/ew_08282013
http://ew.edweek.org/nxtbooks/epe/ew_sr_08212013
http://ew.edweek.org/nxtbooks/epe/ew_08212013
http://ew.edweek.org/nxtbooks/epe/ew_08072013
http://ew.edweek.org/nxtbooks/epe/ew_07102013
http://ew.edweek.org/nxtbooks/epe/ew_06122013
http://ew.edweek.org/nxtbooks/epe/dc_06062013
http://ew.edweek.org/nxtbooks/epe/ew_06052013
http://ew.edweek.org/nxtbooks/epe/ew_sr_05222013
http://ew.edweek.org/nxtbooks/epe/ew_05222013
http://ew.edweek.org/nxtbooks/epe/ew_05152013
http://ew.edweek.org/nxtbooks/epe/ew_05082013
http://ew.edweek.org/nxtbooks/epe/ew_sr_04242013
http://ew.edweek.org/nxtbooks/epe/ew_04242013
http://ew.edweek.org/nxtbooks/epe/ew_04172013
http://ew.edweek.org/nxtbooks/epe/ew_04032013
http://ew.edweek.org/nxtbooks/epe/ew_03272013
http://ew.edweek.org/nxtbooks/epe/ew_03132013
http://ew.edweek.org/nxtbooks/epe/tc_20130314
http://ew.edweek.org/nxtbooks/epe/ew_03062013
http://ew.edweek.org/nxtbooks/epe/ew_02272013
http://ew.edweek.org/nxtbooks/epe/ew_sr_02202013
http://ew.edweek.org/nxtbooks/epe/ew_02202013
http://ew.edweek.org/nxtbooks/epe/ew_sr_02062013
http://ew.edweek.org/nxtbooks/epe/ew_02062013
http://ew.edweek.org/nxtbooks/epe/ew_01302013
http://ew.edweek.org/nxtbooks/epe/ew_01232013
http://ew.edweek.org/nxtbooks/epe/ew_01162013
http://ew.edweek.org/nxtbooks/epe/qc_01102013
http://ew.edweek.org/nxtbooks/epe/ew_01092013
http://ew.edweek.org/nxtbooks/epe/ew_12122012
http://ew.edweek.org/nxtbooks/epe/ew_12052012
http://ew.edweek.org/nxtbooks/epe/ew_sr_11142012
http://ew.edweek.org/nxtbooks/epe/ew_11142012
http://ew.edweek.org/nxtbooks/epe/ew_11072012
http://ew.edweek.org/nxtbooks/epe/ew_10312012
http://ew.edweek.org/nxtbooks/epe/ew_sr_10242012
http://ew.edweek.org/nxtbooks/epe/ew_10242012
http://ew.edweek.org/nxtbooks/epe/ew_10172012
http://ew.edweek.org/nxtbooks/epe/ew_10102012
http://ew.edweek.org/nxtbooks/epe/ew_10032012
http://ew.edweek.org/nxtbooks/epe/ew_09262012
http://ew.edweek.org/nxtbooks/epe/ew_09192012
http://ew.edweek.org/nxtbooks/epe/ew_09122012
http://ew.edweek.org/nxtbooks/epe/ew_08292012
http://ew.edweek.org/nxtbooks/epe/ew_sr_08222012
http://ew.edweek.org/nxtbooks/epe/ew_08222012
http://ew.edweek.org/nxtbooks/epe/ew_20120829
http://ew.edweek.org/nxtbooks/epe/ew_sr_08292012
http://ew.edweek.org/nxtbooks/epe/ew_20120822_v2
http://ew.edweek.org/nxtbooks/epe/ew_20120822
http://ew.edweek.org/nxtbooks/epe/ew_test
http://ew.edweek.org/nxtbooks/epe/diplomascount_2012issue34
https://www.nxtbookmedia.com