Nate Silver from FiveThirtyEight.com Breaks Down Prop. 19 Polls

The Broadus Effect? Social Desirability Bias and California Proposition 19

by Nate Silver

This November, Californians will vote on Proposition 19, which would legalize, tax and regulate marijuana for recreational consumption. Six polls have been conducted since the measure made the ballot in March. Three — one from Public Policy Polling and two from SurveyUSA — used automated scripts (“robopolls”) to conduct their interviews. The other three, from Field, PPIC, and Reuters/Ipsos — used live human operators.

The methodologies split in the support they show for the initiative. The three automated surveys all have Prop 19 passing by a double-digit margin. The human-operator polls, meanwhile, each show it trailing narrowly.

Although some of these polls contain incomplete demographic information, the split appears to be driven more by minority voters than by whites. The three automated polls each show the initiative leading by between 28 and 38 among black voters, for instance. But the one traditional poll to break out numbers among African-Americans had it trailing by 12.

Likewise, the traditional polls show Proposition 19 trailing by about 25 points among Hispanics. But the robopolls show support among Hispanics being about even. Even though the margins of error associated with these subgroups are quite large — especially for blacks, who constitute a relatively small portion of California’s population — the differences are highly statistically significant.

These effects are also evident in two recent national polls on marijuana legalization, fromRasmussen (automated) and Pew (human), respectively. Although Rasmussen’s robots posit higher support for marijuana legalization among all racial groups, the differences are much larger for blacks and Hispanics (which Rasmussen lumps into “other” along with groups like Asians).

There are a couple of reasons why these discrepancies might have arisen. One might be that the automated surveys are having difficulty getting a representative sample of minorities. Automated surveys generally have lower response rates, and that impact may be most felt among minorities, who are usually harder to get on the phone.

Nevertheless, this is a relatively highly-rated group of automated surveys, particularly SurveyUSA and PPP, which don’t take as many of the shortcuts that some of their competitors do. And so it raises another possibility:

What if voters are more likely to admit their tolerance for marijuana to an automated script, which may create the feeling of greater anonymity? Marijuana usage remains fairly stigmatized in polite society in America, enough so that even liberal politicians like Barbara Boxer, Dianne Feinstein, Jerry Brown and Barack Obama have refused to state their support for legalizing the drug. But as most Americans between ages 20 and 55 have smoked marijuana, they may not consider it such a big deal in the privacy of their homes — or the privacy of the ballot booth.

This might also explain why the split is larger among black and Hispanic voters. Marijuana usage is almost certainly more stigmatized when associated with minorities, and drug possession arrests occur much more frequently in minority communities. This is in spite of the fact that rates of marijuana consumption are only a smidgen higher among blacks than among whites, and are somewhat lower among Hispanics. (Although, note that the link I just pointed you to is also based on survey data, and so could be subject to some of the same biases.)

Perhaps this hypothesis is overstated, and drug use does not carry the same stigma in California that it does elsewhere in the country. When I visited San Francisco for four days last winter, I twice saw people quite nonchalantly smoking joints in bars, something you’ll only see once in a blue moon in New York or Chicago (or maybe I’ve just been going out to the wrong places).

Nevertheless, it’s possible that we’re seeing some sort of Bradley effect in reverse, which I’ve reluctantly dubbed the “Broadus Effect” after the given name of the rapper Snoop Dogg, himself a frequent consumer of cannabinoid-rich products.

The original Bradley Effect, named for former Los Angeles mayor Tom Bradley, occurs when respondents in surveys are asked about socially desirable behaviors, such as being free from racial prejudice. Although the racial version of Bradley effect itself isprobably a thing of the past, social desirability bias may manifest itself in other ways. Automated polls have sometimes shown relatively lower levels support for gay marriage initiatives, for instance, in states like Maine and California. Homophobia is fairly common, but has become socially undesirable; the purveyors of the automated polls have sometimes claimed that their respondents are free to be more honest when there’s not another human being on the line. If the theory holds, automated polls might also provide a setting for voters to be more honest about their feelings on marijuana use, another behavior that is probably more widespread (and privately tolerated) than it is socially acceptable. If so, that would be good news for Prop 19.