PNAS study questions – social studies’ replication

| December 28, 2022

Nothing to do with the article. You try and find a picture of a social study.


Or, to rephrase, once a social  study is done, the findings aren’t able to be reached twice in a row.  How much would you trust a study which concluded 2+5=6? Or 7? Or 4? depending on who looked at the data. Probably not much.

A new paper published in the Proceedings of the National Academy of Sciences (PNAS) sheds light on the scale of the problem and calls into question the veracity of social science research in general, including that which anti-gun advocates use to push for gun control.

This started in 2015, when UVa Professor Brian Nosek and a team of 270 researchers undertook to replicate the results of 98 studies for Science magazine.  And they were successful – 39% of the time. So 3 out of 5 times, 2+5 equalled 25, or 12, or whatever. That ain’t science.

In an article on the team’s findings, the journal Nature noted, “John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek’s study suggests.”

At the time, the New York Times explained how researchers’ incentives that can lead to the perversion of science, noting,

The report appears at a time when the number of retractions of published papers is rising sharply in a wide variety of disciplines. Scientists have pointed to a hypercompetitive culture across science that favors novel, sexy results and provides little incentive for researchers to replicate the findings of others, or for journals to publish studies that fail to find a splashy result.NY Times

“Splashy”… meaning newsworthy. Or, more likely, grant-worthy from the folk who control partisan purse-strings.

A 2020 analysis by the RAND Corporation, a nonprofit research organization, parsed the results of 27,900 research publications on the effectiveness of gun control laws. From this vast body of work, the RAND authors found only 123 studies, or 0.4 percent, that tested the effects rigorously.

We took a look at the significance of the 123 rigorous empirical studies and what they actually say about the efficacy of gun control laws.

The answer: nothing. The 123 studies that met RAND’s criteria may have been the best of the 27,900 that were analyzed, but they still had serious statistical defects, such as a lack of controls, too many parameters or hypotheses for the data, undisclosed data, erroneous data, misspecified models, and other problems.

Moreover, the authors noted that there appears to be something of an inverse relationship between the most rigorously conducted “gun violence” studies and those that receive media attention.

The PNAS paper further undermines the validity of social science research – even in cases where attempts are made to control for bias. Titled “Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty,” the paper shows that researchers given the exact same data and hypothesis come to wildly different conclusions as a result of the researchers’ idiosyncratic decisions.

To construct their experiment, the authors assembled 161 researchers in 73 teams and provided them with the same data and hypothesis to be tested. In this case, the researchers were asked to determine from the data whether “greater immigration reduces support for social policies among the public.” To attempt to control for the bias towards “splashy” findings, the researchers were promised co-authorship of a final paper on the topic regardless of their conclusions.

Explaining the results of the experiment, the authors reported,

Results from our controlled research design in a large-scale crowdsourced research effort involving 73 teams demonstrate that analyzing the same hypothesis with the same data can lead to substantial differences in statistical estimates and substantive conclusions. In fact, no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.


The researchers decide which bits of data, and types of data to include, as well as how they interlock – and in doing so, whether they want to or not, they influence the conclusions.

But all that social research – being transgender is OK! Gunz are eevul! Everyone wants to be a gay socialist! and so on….well, seems that no matter what, at the end of the day a variation of what Mr. Twain (“There are three kinds of lies: lies, damned lies, and statistics”) should be amended to add “social research studies”. Well, 60-80% of the time.


Great summary article (from which most of the above quotes are lifted) is at Buckeye Firearms. Plug – especially if you care about the law and firearms, a great site.

Category: "Teh Stoopid", It's science!

Inline Feedbacks
View all comments

Why I don’t Trust the Scientists or their pseudo-science.


Social Science Ain’t


I’m surrounded by a team of about 40 PhD scientists/engineers who, in addition to their actual work, publish/present annually over 70 peer-reviewed, juried, and refereed papers/journal articles (we are a research center after all) … 30 papers so far for FY23. If their data results were as goofy as these social science papers, their tenure in this agency would be pretty short.

Only a couple of us have lowly master degrees — I joke with the PhDs that I’m a scientist as well: a political scientist!
New PhD: “Call be by my earned title, Doctor.”

Me: “Okay, but only when you call me by my earned title, Master.”


How much would you trust a study which concluded 2+5=6?”

This jackass would.

Hack Stone

Attention students, this is your Principal, PC Principal. It has come to my attention that math is inherently racist and reeks of white supremacist patriarchy, resulting in students of color being unable to comprehend the intricacies of math. As a result, in the pursuit of racial equity, we will no longer be teaching any math. Additionally, this afternoon at 1:00 PM, all students are to report to the auditorium for a screening of the 2016 film Hidden Figures, the true story of a team of female African-American mathematicians who served a vital role in NASA during the early years of the U.S. space program.


Good intentions and a mastery of pronouns and the 72 genders puts people in space and builds bridges, dumbasses!


With unicorn farts and self-esteem!
comment image


I would also note that, according to the movie anyway, they taught themselves to operate and program a computer from the accompanying manual(s). I would allude to Kipling’s “You’re a better man than I am, Gunga Din” but that would probably be construed as racist.

RGR 4-78

“The idea that math (or data) is culturally neutral or in any way objective is a MYTH. i’m ready to move on with that understanding. who’s coming with me?”

Hey lady, is that what you truly want?
Is that what you want when an MD, RN or Paramedic is mathing your dose rate by your weight with drugs to alleviate your cardiac pain without overdosing you?



Once more, Leftists gonna Leftist:
comment image


Yeah, try that with me and said leftist would get laughed at, punched in the mouth, and then laughed at again.


“A math education professor…”

Now if she was an actual math professor she would know that 2+2 does indeed =4 for everybody. Obviously she is neither a good educator nor a good math professor.


Now, now, now David. How many times must the spapos seagull tell you that the empirical data reaches the conclusions that proves that the empirical data is ALWAYS empirical, empirically speaking, and is ALWAYS the empirical conclusions that your betters insist that are empirically correct.

It is empirically imperative that the data seekers prove that the empirical data shows that all of you deplorable MAGA Terrorists must be disarmed because the Imperial Emperors data seekers are planning on doing things that will make you want to shoot them.

Molon Labe, mofos.


And let us not forget that “scientific” studies are commonly weaponized by the left to support their political positions or ignored by the same people to defeat opposing views.

Remember how Lars and LC refused to accept inexpensive, long-established drugs like hydroxychloroquine and Ivermectin as safe, efficacious treatments for Covid without their safety and efficacy being uselessly reestablished with new, controlled, double-blind studies?

Yet those same two jumped on the government’s vaccine bandwagon despite the clearly questionable vaccines being implemented without undergoing the lengthy and rigorous trials that had previously been required of all vaccines.

Just as their political forebears did in the Soviet Union, the contemporary left has co-opted and mobilized science to underpin their political doctrine and stifle their political enemies.

So-called “scientific” studies are the weapons they use…


The irony of you talking about Ivermectin as a wonder-drug for COVID on a post about shitty science is just … wow, absolutely amazing. Thank you, Poe, for making me laugh out loud today!




“Teh Science, c’est moi!” –Dr. Fauci


With current party line desired results are always replicable, comrade!
comment image

Last edited 1 year ago by Anonymous

I’ve followed pretend science studies, results, and ultimate retractions for many years. It’s a hobby of mine. From the Guardian – yes, that Guardian, in 2015…

Last edited 1 year ago by JustALurkinAround
Old tanker

When I was a kid I heard this statement. “There are 3 kinds of liars. Liars, damned liars and statisticians.”

When I took statistics in college back in the early 70’s I learned the truth of that statement. While basic mathematics told the truth (at least back then when logical thinking was in vogue), it is childishly easy to manipulate a statistical study to provide the data you wish it to do so. It ranges from where you collect it (study location demographics) to what questions you use and how they are worded, to how you tabulate the data including what raw data you determine to be dubious (at least to your desired conclusion). There ar e many others but as we see the CDC doing now, when the master speaks, the data reeks of the master’s design.

RGR 4-78

when the master speaks, the data reeks”

Well said.


I think I’ve shared this before, but when I was employed as a petty bureaucrat computer programmer for the State, it was not uncommon for me to be tasked with running a report for some State legislator/senator.

It was also not uncommon for me to include/exclude segments of the cadre so that the results of the report showed what that legislator/senator wanted it to show.

As the old sayin’ goes: “Figures don’t lie, but liars figure.”
I’ve had a long-standing cynicism concerning these “studies” ever since.


When I was a new employee at a local government agency it was undergoing some sort of efficiency study conducted by an outside consulting firm. Everyone had to fill out forms describing the tasks I was performing and how long I spent doing each. I was amazed at the number of tasks I was performing, and the time needed to perform them. Time does fly when you are having fun, I guess. That week or two was the hardest and busiest period of my employment there (sarc).

One of our tasks was to collect air, noise, and water samples.
I am still willing to bet good money that much of the “data” collected by my fellow workers was garbage.

Green Thumb

A study was run on a The False Commander “Phony” Phil Monkress (CEO of All-Points Logistics) to see if he was a turd.

And the study found that he was indeed a turd.

1,000 additional replication studies were run to verify the rigor of the research and veracity of the conclusions and claims of the original study and they too found that Phil Monkress was a Turd (but with a capitol T).

Hack Stone

A follow up DNA analysis showed no traces of US Navy SEAL in Phil Monkress.

Green Thumb

Being around academia, I can tell you first hand that this is becoming more and more common in the social sciences.

It is not about the rigor of research or integrity of results. It is about chosen study pools, limited investigation, pre-determined results to advance a college or departments agenda and less-than-stringent peer review process, particularity in qualitative or mixed method data.

Furthermore, the entire replication (as mentioned above) and translation / generalization of results into another area of interest or investigation is but an afterthought. One might say they have a novel concept that contains justice and beneficence for all, but the application process and feasibility are, shall we say, zero.


Lysenkoism, comrade– has been done before!

Boiling Mad CPO

ack in my university days, I was assigned as a group project a survey to find out what the random person in the street thought about a certain subject – to wit – sex on TV.

Because I was in the navy, most of my respondents surveyed a certain way. The other group members naturally surveyed their own respondents. The phone operators answered a certain way, the young adults in the group answered a certain way, etc etc etc.

So all in all the survey showed nothing that was useful in any type of conclusion for our project.

I guess what I am getting at, is your input is only as good as your guestions/repondents. In other words, the outcome can be what you want it to be – full of shit


Especially with the advent of flatscreens.

Hack Stone

Maybe today, but forty years ago a 19” television would come in a 48” Genuine Faux Wood Cabinet. It helped lend credibility when you told the girl “If you have sex with me, I promise to get you on television.”


comment image
(Ones with an added stereo were bigger.)

Last edited 1 year ago by Anonymous
President Elect Toxic Deplorable Racist SAH Neande

GIGO – Garbage In, Garbage Out


I’ve read several studies that say that creating job programs and reducing police presence will reduce crime.

Strange thing though. Today every single person in the US that wants a job can have one for a good wage and has been the case for the last few years. Police are at record low numbers.

Crime continues to.rise steadily the last few years.


You guys, come on.
I am disappoint.


The Proceedings of the National Academy of Sciences (PNAS), 


And that pricked your interest…


You said a mouthfull!


We had to leave a little “meat” on the ‘bone(r)” for ya Chippy. It helps show OAM what a dick you can be.

Mission Accomplished!

A Proud Infidel®™

You gonna shaft us with that joke?


Impulsive sex, violence, drug use, etc.? That’s okay?


German Euthanasia Clinics??? I thought Aktion T4 and Action 14f13 were done away with in 1944???


Euro-progressives live for that sh*t… waitin’ since ’45:

every prog dream.png

Man, fuck those clock-making sons of bitches. We should just let Vlad have the place.


“…calls into question the veracity of social science research in general,…”

There has been no question in my mind for decades. “Science” my ass.