Where’s the Data?
Except, they don’t.
Nobody actually knows whether small classes are better than large. Pascarella and Terenzini’s How College Affects Students, the bible of such matters, says “We uncovered 10 studies that focus on the effects of class size on course learning. All of the investigations are quasi-experimental or correlational in design …. Unfortunately, five of the studies used course grade as the measure of learning … the conflicting evidence and continuing methodological problems surrounding this small body of research make it difficult to form a firm conclusion.”
The text of How College Affects Students occupies 650 pages; class size consumes just over one of them. In the long history of higher education up to 2005, when the latest edition of HCAS was published, there had never been a single truly experimental study of college class size. Not one.
Similarly, the American Association of University Professors recently released a report criticizing the big regional accreditors for failing to enforce standards related to the growing use of part-time professors, which the AAUP regards with much dismay. Like class size, the full-time / part-time issue is generally treated as a given. U.S. News & World Report uses both measures in its methodology to rank colleges, and they’re among the few such ranking measures that don’t cause college leaders to erupt in fits of consternation. Yet the AAUP report contains no references to research proving the underlying premise of full-time professorial superiority — because, I’m guessing, it more or less doesn’t exist. A few studies have examined full-time / part-time status and completion rates, but when it comes to actual student learning — basically nothing. This is not a standard of evidence that university professors would tolerate in their own research.
In other words, when it comes to the central enterprise of higher education — teaching students — we don’t know if the reigning professional qualification system works, or how many professors we actually need. And this is true for all kinds of other basic elements of college teaching and learning — curricula, training, pedagogy, and much more.
It’s not like these questions couldn’t be answered. Millions of students attend college every day in classes that vary greatly in size — much more so than in K–12 schools, where the issue has been studied exhaustively. Many college courses are taught by full-time professors, many not. It wouldn’t take the world’s greatest social scientist to design an experiment to get at what those differences mean.
To be sure, there are sui generis courses at every college that wouldn’t lend themselves to such analysis because they combine the unique perspective of a particular scholar with his or her subject expertise and time-honed approach to teaching. But that’s often not the case. Transcript studies indicate that 20 percent of all course credits earned by college graduates come in just 13 introductory courses like English Composition, Calculus, and Introduction to Economics. Seventy-one percent of all college graduates take some version of Psychology 101. Calculus is pretty much calculus wherever you go (or should be). And even in cases where curricula vary between institutions, larger universities routinely teach many sections of the same course every semester. Why doesn’t anyone ever study how much learning varies between them, and why?
In part, because that would require some kind of objective measure of how much students learn in different sections and/or institutions, i.e. some kind of standardized test. Such assessments are often considered anathema to what the AAUP somewhat breathlessly terms the “sacred principle” of academic freedom. But I don’t really understand why. There’s a great deal of logic behind academic freedom for scholarship. When scholars are bound by conventional thinking and political pressure, their research can suffer immensely. But it’s not clear why giving professors license to say and think what they like necessarily translates into an absolute license to teach how they like, at a level of quality that’s more or less up to them. Academic freedom shouldn’t immunize anyone from scrutiny of how much their students learn between the beginning of the semester and the end. Yes, there are student evaluations, which mean more than they once did. But they are poor substitutes for objective measures of student achievement. Basing education research and instructor performance assessment entirely on student evaluations is like basing clinical drug trials entirely on patient reports of how they feel.
And there’s one type of class that’s certainly amenable to standardized testing: developmental courses. If colleges choose of their own volition to give all incoming students a test to determine whether they need remediation, it seems reasonable to give developmental students a similar test once they’ve completed the course, and use evidence of (what one hopes is) increased learning to judge which kinds of developmental approaches — and instructors — are best.
The underlying cause of this remarkable information deficit is pretty clear: Colleges and universities don’t really need to know — or want to know — the answers to these questions. They don’t need to know because student learning results are peripheral to the core incentive system in which they operate. University success is measured in terms of dollars raised, high-achieving students recruited, and prestigious scholarship produced—period. Even less selective institutions are highly influenced by these values. They may not have the research mission of the academic giants, but they share organizational models, practices, and ways of thinking, all of which cut against rigorous self-evaluation of teaching and learning.
And they don’t want to know because, well, what then? I’ll wager dollars to donuts that any well-designed study of the relative effectiveness of full-time vs. part-time instructors would find far more variation within those populations than between them. That would have profound implications for the way college teaching is supervised and evaluated — like, for example, that it should be supervised and evaluated in a meaningful way. And what if someone actually proved that low-paid part-time professors and large, highly profitable freshman lecture courses had a demonstrable negative impact on learning? The money to support financially hemorrhaging sports programs, rapidly expanding administrative budgets, and light teaching loads for senior faculty doesn’t exactly grow on trees.
In fairness, there are some positive signs. The popularity of surveys like the National Survey of Student Engagement, the Community College Survey of Student Engagement, and the Collegiate Learning Assessment indicate a desire for self-study (as long, in the case of most four-year institutions, the results are safely hidden from public view).
But the overall lack of needed information about the core educational mission remains profound. The American higher education system is, we are often reminded, unmatched in diversity. It contains thousands of institutions and hundreds of thousands of college instructors, each granted significant autonomy to teach in different ways. Yet instead of learning from these differences we often ignore them. In our colleges and universities we’ve constructed the greatest engine of inquiry the world has ever known. That penetrating gaze is too infrequently directed inward, to know itself.
Kevin Carey is the research and policy manager of Education Sector. He blogs about K–12 and higher education policy issues at The Quick and the Ed. An archive of his Outside the Circle columns may be found here.
How do we assess students?