Let’s say you want to argue that women watch more TV than men do. You find a sample that measures gender and time watching TV. You look for the relationship and it’s not there—men and women are watching about the same overall amount of TV. But don’t fret, here’s where researcher degrees of freedom can save you!
For example, there might be variables that cover what kinds of programs the sample members are watching and there might be a category for sports. You can eliminate from the sample all the time spent watching sports! You really can. Who’s going to stop you? Just argue that, you know, watching sports isn’t really watching TV. Talk about how watching sports is some special phenomenon. Cite a few books about sports. Anyway, your problem is solved. Now you can present an analysis showing that women watch more TV than men do, proving that you were right all along.
I’ve been reading the new book “Democracy for Realists” by Achen and Bartels and came across a claim made in the intro that I was surprised by: “As the Democratic and Republican parties took increasingly clear, opposing stands on [abortion], partisan identities came into conflict with gender identities. We show that this conflict was resolved in quite different ways for women and for men. A substantial number of women gravitated to the party sharing their view on abortion, reflecting the deep significance of the issue for women. Men, on the other hand, more often changed their view about abortion to comport with their partisanship—in effect, letting their party tell them what to think about one of the most contentious moral issues in contemporary American politics.”
Yeah, well, you know where this is going.
Turns out that the later section with the abortion analysis suddenly starts taking about how Catholics “differ” and are “unique and complex, requiring separate study.” And then it happens—the researcher degree of freedom: “Hence we focus here on non-Catholics only.” They threw out the Catholics. Move along; nothing to see here. ¯\_(ツ)_/¯
They’re using the Youth-Parent Socialization Panel Study (YPSPS), which I’ve got as well. So I did some checks. And, sure enough, the pattern they find among non-Catholics is basically the reverse of the pattern among Catholics (where now men more than women are picking parties based on their abortion views). And in neither group (non-Catholics or Catholics) is the gender difference significant at a level that you ought to put much faith in. And if you use the whole sample, it’s totally clear that there is no significant general gender difference of the kind announced in the intro of the book.
The real story, as far as I can see from my quick check, is that from 1982 to 1997 in the YPSPS there’s a decent portion of people switching-abortion-opinions-to-match-prior-party-affiliations and perhaps even more switching-party-affiliations-to-match-prior-abortion-views. And, if you believe in 3-way interactions (and it’s not clear that you should), there may be something going on where churchgoing and Catholic men are relatively likely to switch parties while non-churchgoing and non-Catholic men are relatively likely to switch abortion views. But there is no general gender difference of the sort claimed by the intro text.
Achen and Bartels have done and continue to do plenty of good work. In this instance, though, they were kicking around a p-hacky sack and ultimately hid the ball. The gender-difference-in-abortion-and-party-switching tale will be one of those things—yet another one of those goddamn things—that will be believed and repeated for years to come, even though it was never true.