Dog Eats AFT Homework

A teachers union's dishonest study of charter schools.

BY WILLIAM G. HOWELL, PAUL E. PETERSON AND MARTIN R. WEST

Wednesday, August 18, 2004 12:01 a.m. EDT

It is not unusual for interest groups to issue misleading reports that further their political agenda. And for this reason, newspapers generally ignore them, treat them with great skepticism, or make sure they vet the study with independent observers.

Not so in the case of the recently released study of charter schools issued by the American Federation of Teachers, which, after receiving top billing in the right-hand corner of the front page of yesterday's New York Times, was picked up by news media across the country. According to the Times, the AFT had unearthed an apparent coverup by the Department of Education, which had buried key findings in "mountains of data . . . released without public announcement." The department, it seems, is taking a long time to issue its report on charter schooling in America.

So the AFT took matters into its own hands, attributing its success in conducting the study to "a combination of intuition, prior knowledge, considerable digging, and luck." Perhaps. But within a few hours of its release, we were able to replicate its results--and conduct a similar "study" of religious schools besides.

The AFT's conclusion: "Charter schools are underperforming."

Data from the National Assessment of Educational Progress (NAEP), often called the nation's report card, show students in charter schools doing less well than the nationwide public-school average, which includes middle class students from well-heeled suburbs. Similar results are obtained within selected states.

Big deal. These results could easily indicate nothing other than the simple fact that charter schools are typically asked to serve problematic students in low-performing districts with many poor, minority children.

Indeed, if the AFT believes these findings, it must also concede that religious schools excel. According to the same NAEP data from which the AFT study is taken, religious schools outperformed the public schools nationwide by nine points, a gap that is as large as the public school-charter school difference AFT is trumpeting.

On other occasions, the AFT has objected to interpreting such findings as evidence that religious schools are superior, on the grounds that they attract an especially able group of students. But for charter schools, apparently, similar student differences are less important.

"To enhance the fairness of the analysis," the AFT study makes comparisons among students eligible for free lunch and in various kinds of communities, which again shows public school students doing better than those in charter schools. But if these simple comparisons prove the AFT case, then they also prove that religious schools are better than public ones--for within these same categories, religious schools outperform public ones.

Indeed, the AFT's most telling comparisons-- the ones within ethnic groups--cut against the case it is trying to make. This comparison is vital, precisely because prior research has found ethnic differences to be large. Yet when the authors look just at African-American or Hispanic children, they find no statistically significant difference between public school students and those in charter schools.

But do any of these findings--within ethnic groups or otherwise--say anything meaningful about the quality of charter schools? Not a bit. For starters, one must do much more than look separately at students grouped by free lunch status, ethnicity or school location, in order to take into account family influences on a child's learning capacity. All of these factors--and many other considerations--must be combined into a sophisticated analysis in order to begin to gauge how well students perform.

What makes such a task essential is the simple fact that charter schools are usually placed in challenging situations. Most states allow charter schools to form only where students are having difficulties, and charter schools are, in many cases, then asked to accept the most challenging of students. Any credible analysis of their effectiveness must account for these facts on the ground.

But this just touches the surface. The AFT study only looks at student performance at a single moment in time. One needs to track student progress within a school over multiple years in order to ascertain how much the child is learning. Moreover, nothing in these data accounts for the length of time that a charter school has been in place--a factor known to have an impact on a school's performance. First-year schools usually have difficulties. Having just hired new staff and teachers, implemented new curricula, and acquired a building facility to use, new schools often face considerable start-up problems. Almost one-third of the charter schools nationwide were less than two years old when the NAEP was administered, raising doubts about whether even a sophisticated analysis of NAEP data would be relevant once charter schools have had time to become well established.

According to AFT official Bella Rosenberg, "Analyses are always welcome, but first things first. . . . Surely the interests of children are better served by timely and straightforward information about whether charter school performance measures up to the claims made for it."

Of course timeliness is important, but bad information is worse than none. To know whether charter schools are doing better, careful analyses are essential. For all of the reasons outlined above, the Department of Education is well advised to prepare its report on charter schools carefully, taking as much information into account as possible. If this explains the official report's delay, this can hardly be called a coverup.

The limited information currently available prevents anyone, including the AFT, from taking even the most modest steps toward addressing these issues. In short, the AFT's report tells us hardly anything about the relative effectiveness of charter schools. But one thing is sure: Charter schools do not appear to be bastions of privilege. What remains unclear is how much they can do for the underprivileged. Sadly, the AFT report tells us nothing about that.

Messrs. Howell, Peterson and West are researchers in the Program on Education Policy and Governance at Harvard. Mr. Peterson is also a senior fellow at the Hoover Institution and a member of the Koret Task Force on K-12 Education.