PNAS study questions – social studies’ replication

 

Or, to rephrase, once a social  study is done, the findings aren’t able to be reached twice in a row.  How much would you trust a study which concluded 2+5=6? Or 7? Or 4? depending on who looked at the data. Probably not much.

A new paper published in the Proceedings of the National Academy of Sciences (PNAS) sheds light on the scale of the problem and calls into question the veracity of social science research in general, including that which anti-gun advocates use to push for gun control.

This started in 2015, when UVa Professor Brian Nosek and a team of 270 researchers undertook to replicate the results of 98 studies for Science magazine.  And they were successful – 39% of the time. So 3 out of 5 times, 2+5 equalled 25, or 12, or whatever. That ain’t science.

In an article on the team’s findings, the journal Nature noted, “John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek’s study suggests.”

At the time, the New York Times explained how researchers’ incentives that can lead to the perversion of science, noting,

The report appears at a time when the number of retractions of published papers is rising sharply in a wide variety of disciplines. Scientists have pointed to a hypercompetitive culture across science that favors novel, sexy results and provides little incentive for researchers to replicate the findings of others, or for journals to publish studies that fail to find a splashy result.NY Times

“Splashy”… meaning newsworthy. Or, more likely, grant-worthy from the folk who control partisan purse-strings.

A 2020 analysis by the RAND Corporation, a nonprofit research organization, parsed the results of 27,900 research publications on the effectiveness of gun control laws. From this vast body of work, the RAND authors found only 123 studies, or 0.4 percent, that tested the effects rigorously.

We took a look at the significance of the 123 rigorous empirical studies and what they actually say about the efficacy of gun control laws.

The answer: nothing. The 123 studies that met RAND’s criteria may have been the best of the 27,900 that were analyzed, but they still had serious statistical defects, such as a lack of controls, too many parameters or hypotheses for the data, undisclosed data, erroneous data, misspecified models, and other problems.

Moreover, the authors noted that there appears to be something of an inverse relationship between the most rigorously conducted “gun violence” studies and those that receive media attention.

The PNAS paper further undermines the validity of social science research – even in cases where attempts are made to control for bias. Titled “Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty,” the paper shows that researchers given the exact same data and hypothesis come to wildly different conclusions as a result of the researchers’ idiosyncratic decisions.

To construct their experiment, the authors assembled 161 researchers in 73 teams and provided them with the same data and hypothesis to be tested. In this case, the researchers were asked to determine from the data whether “greater immigration reduces support for social policies among the public.” To attempt to control for the bias towards “splashy” findings, the researchers were promised co-authorship of a final paper on the topic regardless of their conclusions.

Explaining the results of the experiment, the authors reported,

Results from our controlled research design in a large-scale crowdsourced research effort involving 73 teams demonstrate that analyzing the same hypothesis with the same data can lead to substantial differences in statistical estimates and substantive conclusions. In fact, no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.

PNAS

The researchers decide which bits of data, and types of data to include, as well as how they interlock – and in doing so, whether they want to or not, they influence the conclusions.

But all that social research – being transgender is OK! Gunz are eevul! Everyone wants to be a gay socialist! and so on….well, seems that no matter what, at the end of the day a variation of what Mr. Twain (“There are three kinds of lies: lies, damned lies, and statistics”) should be amended to add “social research studies”. Well, 60-80% of the time.

 

Great summary article (from which most of the above quotes are lifted) is at Buckeye Firearms. Plug – especially if you care about the law and firearms, a great site.

Category: "Teh Stoopid", It's science!