bias in bias studies
http://www.insidehighered.com/news/2007/01/22/bias
Bias Seen in Bias Studies
Professors are all Democrats, except those who are communists.
Professors all hate Bush. Professors favor like-minded students and
love converting those who love God, country and the president. You’ve
read all the claims and more, in right-leaning blogs and columns.
Frequently, these claims are based on studies — many have been
released in the last two years — of professors. Party registration is
documented, or professors respond to surveys, or syllabus content is
rated.
A new study being released today aims to debunk all of those studies.
“The ‘Faculty Bias’ Studies: Science or Propaganda,” takes eight of
the recent studies on faculty politics and judges them by five
general tests of social science research. Today’s study finds that
the eight all come up short in adhering to research standards. The
new study was sponsored by the American Federation of Teachers and
the work was conducted by John B. Lee, an education researcher and
consultant who said that once the AFT commissioned the work, it did
not restrict his approach or findings in any way.
The various studies analyzed are by no means identical, but they tend
to have two major themes (although some stress just one of the
themes): that faculty members are liberal and that their liberal
inclinations are significant in considering their performance.
Lee’s analysis finds some support for the first theme. “Taken
together, these studies at best suggest that college faculty members
are more likely to be Democrats than Republicans,” he writes.
However, even on this theme, he notes that the studies tend to
exclude community college faculty members and to focus on faculty at
elite institutions — probably skewing the results.
The second theme takes a more thorough beating in the study. “Among
the most serious claims the authors make is that this liberal
dominance results in systematic exclusion of conservative ideas,
limited promotion opportunities for conservative faculty, and
expression in the classroom of liberal perspectives that damage
student leaning,” Lee writes. “These claims, however, are not
supported by the research. Basic methodological flaws keep a critical
reader from accepting the conclusions suggested by the authors.”
The flaw Lee identifies most frequently with this theme is one in
which researchers note a correlation and — in Lee’s opinion — then
see a causal relationship without sufficient evidence that one exists.
AFT officials said that they commissioned their study out of concern
that the drumbeat of reports on political bias were suggesting to the
public and politicians that faculty members are unprofessionally
injecting politics into the curriculum, hiring and grading. Some of
those whose work is criticized in the AFT’s report, however, said
that it was the faculty group’s report that was guilty of bias, and
they questioned the legitimacy of the new study, which they termed
propaganda.
The New Research
The new AFT study looks at eight studies, including some that have
attracted substantial attention (both praise and criticism), such as
work published in 2005 in The Forum that analyzed faculty attitudes
at four-year institutions and concluded that conservatives,
practicing Christians and women are less likely than others to get
faculty jobs at top colleges. That study was based on a survey of
1,643 faculty members. Other studies looked at faculty attitudes in
certain disciplines or at certain institutions.
Some of the studies were prompted by specific events, such as the
American Council of Trustees and Alumni’s “How Many Ward
Churchills?,” which analyzed class materials online at top
institutions and found that the controversial Colorado professor’s
ideas — which have been in the news while his university has
considered whether to fire him — are shared by many professors. Some
of the reports are by social scientists, published in peer-reviewed
journals. Others were issued by associations that are players in the
culture wars of academe.
Lee said that to test the validity of the studies, he wanted
standards that could not be considered partisan, so he used a 2006
statement by the White House Office of Management and Budget about
objectivity in research. Based on that statement, he asked five
questions about each of the faculty bias studies:
Can another researcher with a different perspective replicate the
results using the information provided by the author? Are the
definitions used in the studies clear enough? Does the research
eliminate alternative explanations for the results? Do the
conclusions follow logically from the evidence? Has the author
guarded against assumptions that could introduce systematic bias into
the study? Using this framework, Lee gives the studies failing
grades. Four studies had data that could be replicated, and he gave
three studies acceptable reviews on clarity of terms, but it was
downhill from there, and he argues that none of the reports can truly
back up their contentions.
Besides offering that general rubric, Lee goes through each study,
summarizing problems he found with it. For example, “How Many Ward
Churchills?” was based on a review of online materials at various
colleges. Lee notes that the researchers for the study appeared to
focus on syllabuses or courses that had certain key words: activism,
discrimination, gay issues, Marxism, oppression, pornography,
radical, women’s studies, among others. Lee writes that selecting 65
courses at 48 colleges “does not allow for the sweeping
generalizations the authors make.”
Even for those courses, he notes, the authors of “How Many Ward
Churchills?” didn’t actually observe the courses, so while they may
know that certain topics or perspectives are covered, they have no
way of judging the intellectual character of a classroom. While that
report’s authors wrote that Americans should be “outraged by the one-
sided doctrinaire perspective” of their courses, Lee writes that they
had no evidence to assert much of anything about the courses.
In several of the studies, Lee notes that relatively small subgroups
of college faculty were surveyed, generally professors at elite and/
or four-year institutions. Because community college professors, on
average, are more centrist and more religious than colleagues at four-
year institution, Lee questions whether their exclusion limits the
ability of the studies’ authors to make statements about academe as a
whole.
Another theme he returns to over and over again is one of
demonstrating (or not) causal relationships. He notes that there are
many explanations for political trends and demographics among the
professoriate, so it is unfair to assume that a liberal tilt
(assuming one exists) reflects bias. He notes, for example, that the
studies do not explore whether there could be non-political
explanations.
Many have questioned, for example, the lack of data on applicant
pools for faculty positions, and compared the disparity in political
inclinations to that of Wall Street, where there are not suggestions
that any Republican tilt is the result of bias or results in any
discrimination against Democratic investors. Lee also compares the
military, where recent polls have found a Republican tilt in
opinions, but no evidence that soldiers service to their country is
affected by whether they are seeking to protect members of one party
or another.
While Lee finds flaws in all of the studies, he says that they have
had influence, and notes that the studies have been widely cited by
conservative pundits. Looking at the studies together, he says that
it is clear that the authors “have a clear agenda” of charging
professors with unprofessional conduct, and yet lack the evidence to
make their case. Not a single study, he says, shows political bias in
the classroom or hiring decisions.
“Until credible studies are conducted to provide a more grounded and
systematic approach to understanding the subtle relationship between
political beliefs and professional responsibilities, it is
irresponsible to suggest that the conclusions reached in these
reports represent a scientifically derived set of facts. They do
not,” Lee writes. “Passing off personal opinions as facts is not
science; it is the antithesis of what serious researchers try to do,
regardless of whether they are conservative or liberal.”
Two of the authors whose work is criticized by the AFT took issue
with the conclusions, and questioned whether the organization could
fairly look at these issues.
“Critical commentary is always socially useful, and this new report
is no exception. Even just a cursory reading will teach us much about
the moral and intellectual character of its sponsors — the AFT, the
AFL-CIO, and Free Exchange on Campus,” said Daniel Klein, a professor
of economics at George Mason University and the co-author of two of
the studies reviewed in the report.
Anne Neal, president of the American Council of Trustees and Alumni
(which issued two of the reports reviewed), criticized the AFT for
commissioning the study. Via e-mail, she said: “Faced with mountains
of evidence from ACTA and others documenting a troubling lack of
professionalism in the academy, AFT chooses, instead, to shoot the
messenger. In doing so, far from undermining ACTA, it discredits
itself. AFT’s study is severely flawed. It is filled with inaccurate
and tendentious interpretations — for instance, framing the debate in
terms of politics rather than professional standards outlined by
ACTA; applying irrelevant ’scientific’ standards to textual analysis;
and offering such shoddy research that the sections on ACTA totally
confuse and conflate two different reports, rendering the critique
invalid, even laughable.”
She added: “In the face of troubling evidence of a politicized
classroom, has AFT conducted any studies of its own to see if there
is problem? Taken concrete steps to explore the atmosphere in the
classroom? The answer, of course, is no. AFT’s report is not science
— it’s propaganda.”
With more studies of the sort the AFT criticized in the works now,
stay tuned for more debate.
— Scott Jaschik