PORTLAND -- Does the use of laptop computers improve the writing skills of Maine students? A recent highly-publicized report from the Maine Education Policy Research Institute claims to have identified data showing an increase in writing performance. However, a closer examination of that data by The Maine Heritage Policy Center reveals that study's conclusion is simply not supported.
In the new report titled Maine's laptop program is improving student writing? Not so fast, Stephen L. Bowen, education policy analyst, reviewed the writing data sources used by the Maine Education Policy Research Institute in their October 24, 2007 report. He found that the report over-relied on highly subjective data, made selective use of the more substantive data it did analyze, and provided virtually no meaningful evidence in support of its claims.
The first data source used by the report's authors was a survey which was unscientific and subjective in nature. This so-called "self-reporting" data used survey instruments to ask students and teachers if they thoughtthat the laptops had made the students into better writers. The report concluded from that data that "70 percent or more of the students think the laptops have facilitated their learning."
However Bowen found that the report's authors were reaching. Only half of the students surveyed said they "agreed" or "strongly agreed" that they "do more work with the laptops" or that the "quality of their work has improved" since using laptops, and less than 40 percent "agreed" or "strongly agreed" that they were "better able to understand their school work" when using laptops.
"Using unscientific survey data is problematic in itself," said Bowen, "But for the authors to stretch those findings significantly undercuts the credibility of the report."
Maine Education Assessment scores were used as the second data source. While a scientific source, Bowen found that the selective nature in which the data was used resulted in a skewed result. The report's authors looked at MEA testing data from 2000, before laptops were implemented, and then from 2005, after the program had been in place for a couple of years. What they found was that the state average score on the eighth grade MEA writing test grew from 534.1 to 536.5 over that period, and that the percent of students rated "proficient" in their writing on the exams rose from 29 percent to 41 percent.
In selecting test scores from 2000 to use as a baseline, the report's authors helped their case immeasurably because scores and achievement percentages were both down that year. But by the very next year, 2001, the average scale score had jumped from 534 to 536--higher than the jump from 2004 to 2005--and the percent of students meeting the proficiency standard rose from the 29 percent cited in the report to 39 percent. This single-year ten-point jump occurred three years beforethe laptops were put into use, though the report's authors would have readers believe that the laptops were "at least in part" responsible. The fact is that the 39 percent of students meeting the proficient writing standard in 2001 and 2002 outperformed the first generation of laptop users, who took the test in 2004.
"There was indeed writing score improvement from 2000 to 2005, but it occurred well before the laptops were put into the hands of a single student," said Bowen. "However, the Maine Education Policy Research Institute's selective use of the data would make you believe that the laptops were responsible for the improvement."
The last piece of data was more "self-reporting" by students regarding how they have been using laptops in their writing. The report showed that students reporting the most thorough use of the laptops had an average MEA writing score of 538.8, while the "non-users" had a score of 532.
Bowen found that this was an especially glaring oversight given how small the non-user group was. Out of the 15,881 students surveyed, only 642 responded that they did not use the laptops for their writing, yet it was the average score of those 642 students, only four percent of the total, that was used to make the case that the use of the laptops, in and of itself, resulted in higher test scores.
"Again, the use of subjective 'self-reporting' data is suspect in itself," said Bowen. "For researchers to avoid fully analyzing the data to ensure its validity really makes this report's findings questionable.
"It may very well be that the laptop program is having the kind profound impact on student learning that its backers hope for, but their cause is done a great disservice by this highly suspect report on writing achievement. Whether the laptops are a great innovation or an enormous waste of resources remains an open question. In order to find an answer to this question policymakers must demand far better analysis of the program."
The report is available at: www.mainepolicy.org.
The Maine Heritage Policy Center is a 501 (c) 3 nonprofit, nonpartisan research and educational organization based in Portland, Maine. The Center formulates and promotes free market, conservative public policies in the areas of economic growth, fiscal matters, health care, and education - providing solutions that will benefit all the people of Maine. Contributions to MHPC are tax deductible to the extent allowed by law.
