"There are more methodological problems in regards to the study of cognitive sex differences, and sex differences in general, than there are actual sex differences."

--Carolyn Jacklin

          This is both a profound and rather remarkable statement to be made by one of the leading researchers in the field of sex-difference research. Yet, it is the principal point made by Carolyn Jacklin during a lecture she presented on sex-related research.  Throughout her presentation Jacklin discusses one methodological problem after another, including small and inconsistent findings, publication bias, definitional problems and inappropriate conclusions drawn from available data; and at the end of her talk Jacklin concludes that "much past research has not been done very well."  Jacklin does not suggest, however, that sex-difference research should be discontinued; rather, she urges her audience to do better research in the future.  Given all the fundamental methodological problems that Jacklin raises, it might be equally valid to ask why such research should be continued.  Whose interest does it serve?  In what way does this research enhance our understanding of human behavior?

          The goal of scientific research is to develop theories and models that lead to better and better predictions of behavior, be it the behavior of physical objects in space, molecules in living organisms, or individuals in society.  Predictability is, thus, fundamental to doing science.  Yet, Jacklin clearly states that predictability is one of the things that is severely lacking in sex-difference research.  According to Jacklin, "we gain little in prediction by knowing the sex of a child."  Indeed, the largest sex difference found concerns "rough and tumble" play among 4-year olds, and this amounted in one study to only 15%.  This means that with regard to this "trait", 85% of boys are indistinguishable from 85% of girls.  This is not a very big difference, certainly not the clear typological difference pictured by many people, including many professional social and behavioral scientists.  Furthermore, this specific differences did not even concern a cognitive difference; When discussing cognitive traits, the differences are much less.  Caplan and Caplan in Thinking Critically About Research on Sex and Gender (1998) state that in those studies that show cognitive differences, those differences range between 1-5%.  At the same time, there are many studies --perhaps the majority-- that show no differences at all.  This means that between 95 and 100% of men and women show no differences in cognitive abilities.  How valid is it then to make claims of significant sex difference when such small differences are routinely found, and when in a large proportion of the cases no differences are found?

          According to Jacklin, we need even to be skeptical of those studies that purport to find sex differences.  She points out that many of the differences found are simply a function of small sample size.  One important study suggesting sex differences in chemical sensitivity, for example, was based on only 3 men and 3 women.  Another study claiming sex differences in brain lateralization was based on the analysis of only 9 male brains and 5 female brains (Caplan & Caplan).  How valid is it to claim the existence of sex differences based on research using such small samples?  How valid are claims made for studies of even 40 or 50 individuals?  The tendency to combine the research of numerous independent studies into a single statistical analysis (in order to obtain the large numbers required to determine statistical significance) simply compounds the problem. As Caplan & Caplan point out, by combining many individual studies into a "meta-analysis," small insignificant differences obtained through separate research (whose methodologies have no connection to one another) are magically transformed into a pseudo-significant "single" study, the conclusions of which are questionable.

          The composition of the samples used in studies of sex differences also raises problems.  Such research is mostly based on American subjects.  According to Jacklin, very little cross-cultural research exists, and what little research has been done outside the U.S. has been done in England among comparable social groups.  Indeed, according to a recent article in the American Anthropologist, 80% of the articles published in social psychology journals are based on research conducted on students in Introduction to Psychology classes.  How are such studies (and by extension sex-difference studies) biased by the composition of their sample populations?  Might not the results be different if the studies were based on students in Anthropology, Chemistry or Art courses, let alone on working-class individuals who did not go to college, individuals in minority communities, or those living on the Navajo Reservation or in African villages?

          Another critical problem raised by both Jacklin, and Caplan & Caplan, is the problem of defining the variables under investigation.  Fundamental to the scientific process is the use of clearly definable concepts amenable to testing and, therefore to either verification or falsification.  While such concepts as "verbal ability", 'math ability" and "spatial visualization" are treated in the literature as unitary concepts, as with "intelligence" they are a composite of a variety of specific characteristics and skills that may or may not vary together.  An individual that is good at one type of computation may not be good at another; similarly, individuals may vary in which specific verbal skills they excel and in which they do poorly.  To combine these individual skills together into a single unitary concept called verbal ability, math ability or spatial visualization amounts to a reification of those concepts  --in the same way that combining the vast array of individual skills into a unitary concept called "intelligence" represents a reification of that concept. It then becomes highly problematic to claim that different studies using different operational definitions of verbal ability are all, in fact, measuring verbal ability, and thus verifying sex differences for verbal ability.

          Given the fundamental methodological problems associated with sex-difference research together with the small and insignificant conclusions obtained from this research, it is reasonable to ask why such research continues and why the concept of sex difference persists.  Both Jacklin and Caplan & Caplan provide the same explanation:  publication bias.  According to Jacklin, there is "a tremendous bias towards the indexing, publishing, citing and remembering of a positive instance."  She suggests that claims for a difference are likely to be based on one study in 20  --to her a chance phenomenon.  Caplan & Caplan cite one study that showed a sex difference in spatial abilities among white children but not among black children. According to them, the white children's difference was cited in subsequent publications far more frequently than the Black children's non-difference.  To what extent does such selective bias towards the publication and subsequent referencing of research purporting to show a positive sex difference (to the near exclusion of research showing no differences) create a false image not only among the public, but also within the social and behavioral science community of larger and more significant differences than actually exist?  Can this be likened to the "Abuse Bowl" phenomenon?

          While Jacklin only scratches the surface of the problems associated with sex-difference research, the methodological problems she addresses raise important issues regarding that research.  Given all of the methodological problems associated with such studies and the consistently small (and frequently non-existent) differences found, it is reasonable to ask why such research persists?  Would we continue to conduct such inconclusive research in physics, chemistry, biology or medicine?  Would we continue to fund research that after 30 years could not point to conclusive results?  Whose interests are being served by such research? 

          To what extent might social and political factors contribute to this persistence? The philosopher of science, Phillip Frank, proposed two reasons for the acceptance and persistence of scientific theories: their technical superiority and/or their social acceptability.  Clearly, if both Jacklin and Caplan & Caplan are correct, sex-difference research has produced precious little suggestion of sex-related differences in cognitive behavior.  The concept of sex-difference provides little explanatory power.   How might the social and political developments that have been taking place in higher education account for the persistence of this research?  Might the continuation of such research serve the interests of college and university social and behavioral science departments, as well as professional psychology journals?  Could the continued interest in the topic of sex-difference reflect the transformation, for example, of psychology from a discipline that was 67% male in 1975 to one that is 67% female in 1995?  If so, could this persistence simply be a product of the Science of Self-Preservation as suggested by Seligman?

 

Other Sources:

Christina Hoff Sommers, The War Against Boys. The Atlantic Monthly (May 2000).

Emily Eakin, Listening to the Voice of Women. New York Times (March 30, 2002).