Brady Score – Meaningful Metric, or Misleading BS?

This article by Jonn and the comments to same got me to thinking about the subject of gun control again.  It also reminded me of something I originally wrote a couple of years ago for a site that no longer exists and which wasn’t published before the site folded.  And  I also never got around to sending it elsewhere for publication.  So here goes.

Fair warning:  this article is a bit longish, and there’s some math involved.  (smile)

Introduction

Fairly recently (late 2009/early 2010) the Brady Campaign to Prevent Gun Violence (hereafter referred to the “Brady Campaign”) published its evaluation of US state firearms laws. It defined in this evaluation a measure it called the “Brady State Scorecard.”  This Brady State Scorecard yields a single numerical value for the state’s firearms laws – the state’s “Brady Score”.  The higher a state’s Brady Score, the more restrictive that state’s firearms laws.

The Brady Campaign’s thesis is that laws restricting gun and ammunition purchase and ownership promote public safety, presumably by reducing gun-related crime.  They’ve been working to promote more restrictive firearms laws for literally decades.

However, with the introduction of the Brady Score the Brady Campaign has allowed a test of their thesis. This article will do exactly that.

Specifically, this article will provide a statistical test indicating whether there is reasonable evidence for a direct cause and effect relationship between restrictive gun laws and a state’s overall murder rate, a state’s  firearm murder rate, and that state’s percentage of murders committed using firearms – or, in plain terms, whether gun control works to reduce gun violence.  If there is indeed a strong a cause and effect relationship between restrictive firearms laws (as measured by the Brady Score) and lowered gun violence, that should be both apparent and obvious on examination of the data.

The Brady Campaign – Background

The history and mission the Brady Campaign to Prevent Gun Violence is illustrative. Here is the Brady Campaign’s history:

The Brady Campaign and the Brady Center to Prevent Gun Violence has a long and rich history of working to save lives.

Mark Borinsky, who had been robbed and nearly killed at gunpoint, founded the organization in 1974 as the National Council to Control Handguns. Pete Shields became Chairman in 1978 following the murder of his twenty-three-year-old son, Nick, in 1974.

The organization was renamed Handgun Control, Inc (HCI) in 1980. In 1983, the Center to Prevent Handgun Violence (CPHV) was founded as an education outreach organization dedicated to reducing gun violence. In 1989, CPHV establishes the Legal Action Project to take the fight against gun violence to the courts.

In 2001, HCI was renamed the Brady Campaign to Prevent Gun Violence and CPHV was renamed Brady Center to Prevent Gun Violence in honor of Jim and Sarah Brady for their commitment and courage to make America safer.

Need to see more?  Here is the Brady Campaign’s mission – again, in their own words:

We are devoted to creating an America free from gun violence, where all Americans are safe at home, at school, at work, and in our communities.

The Brady Campaign works to pass and enforce sensible federal and state gun laws, regulations, and public policies through grassroots activism, electing public officials who support common sense gun laws, and increasing public awareness of gun violence. Through our Million Mom March and Brady Chapters, we work locally to educate, remember victims, and pass sensible gun laws, believing that children have the right to grow up in environments free from the threat of gun violence.

The Brady Center works to reform the gun industry by enacting and enforcing sensible regulations to reduce gun violence, including regulations governing the gun industry. In addition, we represent victims of gun violence in the courts. We educate the public about gun violence through litigation, grassroots mobilization, and outreach to affected communities.

Given the above, one would reasonably expect the Brady Campaign to be in favor of restrictive gun laws.  That is indeed the case. Indeed, from the above the Brady Campaign’s philosophy can be simply and succinctly summarized: “Guns baaaaad . . . . gun control gooooooood!”

The Brady State Scorecard

The Brady State Scorecard is the Brady Campaign’s metric to quantitatively “rate” state laws relating to firearm and ammunition purchase and ownership. To do so, the Brady Campaign has defined five major categories, each having multiple elements. Each of these categories has reasonably innocuous-sounding names: “Curb Firearm Trafficking”, “Strengthen Brady Background Checks”, “Child Safety”, “Ban Military-Style Assault Weapons”, and “Guns in Public Places and Local Control”.  State laws and policies relating to each major category are rated and given a numerical score; the results are summed.  The output is a single number – a state’s “Brady Score” – and ranges from a minimum possible of 0 to a maximum possible of 100. A complete description of how a state’s Brady Score is calculated, along with 2009 Brady Scores, for all states may be found on the Brady Campaign’s website here.

However, as with many such things, the “devil is in the details”. For example: “Curb Firearm Trafficking” sounds innocuous enough. However, this major category includes the subcategory “Crime Gun Identification”. “Crime Gun Identification” has two elements: “Ballistic Fingerprinting” and “Require microstamping on semi-auto handguns”. To achieve a perfect score, this means all guns would need to be fired, their ballistic signatures recorded and kept on file, and all semi-automatic handguns would require microstamping.

Similarly, under “Strengthen Brady Background Checks”, the subcategory “Permit to Purchase”includes “Fingerprinting required”. This means a perfect Brady Score requires a firearm purchaser’s fingerprints to be on file with the state. This major category also includes the subcategory “Ammunition Regulations” – and yes, that means exactly what you might think. For a perfect Brady Score, ammunition purchase would require a permit (or a point-of-sale Brady Check), and keeping records (presumably by-name) of all ammunition purchases would be mandatory.

Privacy advocates will simply love those provisions!

Finally, even the major category of “Child Safety” includes some absurd provisions. It includes the subcategory “Childproof Handguns”, with the single element “Only authorized users are able to operate new handguns”. Theoretically possible, perhaps – and maybe that will be a routine feature when Captain James T. Kirk actually commands the starship USS Enterprise some year in the 23d century.   But for now, that’s pretty much a pipe dream.  Requiring that by law would make most if not all current handgun designs unlawful.

Moreover, the category “Child Safety” also includes the subcategory “Juvenile Handgun Purchases”. The Brady State Scorecard defines this simply as “Must be 21”. I guess in the Brady Campaign’s view a 19 or 20 year old military combat veteran isn’t trustworthy enough to own a firearm.

In short: the Brady State Scorecard is biased as hell in favor of legal restrictions on firearms and ammunition ownership. Given the Brady Campaign’s background, that’s exactly what one would have expected.

However, regardless of it’s obviously biased origin, the Brady Score could still be a useful metric. If the Brady Campaign is correct, increasing restrictions on lawful gun ownership (and therefore legal gun availability) should lower firearm-related crime. Therefore, a higher Brady Score should be associated with a lower rate of gun-related crime. And if this effect is direct and unambiguous, a linear model (the simplest mathematical model for a cause and effect relationship) should be fairly descriptive of that effect – that is, it should show significant correlation.

Linear Models and Correlation

A model may be defined as “a simplified representation of a system or phenomenon, as in the sciences or economics, with any hypotheses required to describe the system or explain the phenomenon, often mathematically.” If such a model is to be used to predict future behavior, a mathematical basis is necessary.

The simplest mathematical models are based on linear (direct) relationships.  That is, they can be expressed as a simple linear equation of the form “y = mx + b” that we all remember (and love!) from high-school algebra.  Nonlinear models, while generally better at describing reality accurately, are often extremely difficult to discern, develop, or test. Moreover, for many real-world purposes, linear models suffice – particularly when there is a strong cause and effect relationship between the variable causing the observed behavior (the independent variable) and the variable showing the effect (the dependent variable).

Indeed, modern science and engineering is full of useful linear models that are simplifications of more complex nonlinear ones.  Examples include Newton’s famous relationship between force, mass, and acceleration (F = MA); the well-known relationship between average speed, distance, and time (D = RT); the energy required to lift an object vertically (W = FH); and Ohm’s law for DC circuits (V = IR).  Each of these neglects effects predicted by more accurate nonlinear models, but which are negligible under most conditions.  In each case, a linear model is more than sufficient in daily life.

Further, linear models have been extensively studied.  The problem of deriving a linear model from a set of real-world data – and of testing how well such a derived model actually describes that data – has also been extensively studied. The process of deriving such a linear model is called linear regression; the measure that describes how “well” such a model describes observed real-world data is called the correlation coefficient.

Describing the details of linear regression and the calculation of the correlation coefficient is well beyond the scope of this article. However, the calculations – while tedious – are also fairly straightforward, and are now standard  functions in many spreadsheet and/or other software packages.

In plain English, the correlation coefficient describes – in virtually all cases – how well a linear equation can be used to represent observed data.  The correlation coefficient ranges from -1.0 to +1.0. A value of -1.0 means all observed data lies exactly on a line with negative slope; a value of +1.0 means all observed data lies exactly along a line with positive slope. (The correlation of data lying exactly on a horizontal line is mathematically undefined.)  As an example:  data set with a correlation coefficient having absolute value of approximately |0.8| or more means the observed data is scattered reasonably near – but not directly on – a line.

Real life data will rarely if ever exhibit perfect correlation (e.g., +/-1.0) to a derived linear model.  But if that model is a reasonably accurate representation of reality – e.g., if the cause and effect connection is real and substantial – it may well be fairly close to unity.

A correlation of zero, in contrast, usually indicates that the observed data cannot be accurately modeled by a linear equation.  This figure shows examples of correlation coefficients for various data sets plotted in what is called  a scatter plot – e.g., on a Cartesian X-Y axis.

A couple of cautions regarding interpreting correlation. Though suggestive, a high absolute value for correlation (e.g., one with an absolute value close to one) does not conclusively prove cause and effect – though it can be a fairly strong indicator.  There could always be another underlying process unrelated to the independent variable (or on which the assumed independent variable is actually dependent) that is instead causing the observed behavior.  Similarly, lack of correlation does not prove a lack of relationship – though it does indicate that a linear model doesn’t work well to represent any relationship which may exist. This is apparent from looking at the example scatter plots in the lower row here, all of which have a correlation coefficient of zero. Even a cursory look shows each scatter plot with a correlation coefficient of zero has discernible structure – but none of these structures are linear in the variables of interest and  the correlation coefficient for each is zero.

Finally, one might wonder how to test linear correlation for significance. There are various methods to test for the significance of the correlation coefficient for a model determined via linear regression. A simple test, used in the Six Sigma methodology for statistical process control,  is to multiply the correlation coefficient by the square root of the number of (x,y) pairs used to calculate the correlation coefficient. If this value is greater than 3, the correlation can be regarded as significant.

Now, regarding the Brady Score:  a possible test now suggests itself. The Brady Campaign’s longstanding thesis is that restrictive gun control laws (which result in high state Brady Scores) result in lower gun crime.  Therefore, if restrictive gun laws indeed lower gun crime, a significant negative correlation between Brady Score and measures of gun crime should be observed. All that remains is to select those measures, collect the appropriate data, do the math, and analyze the results.

Collecting the Data

Thankfully, suitable data is readily available. The 2009 Brady Score for each state is available in consolidated form at the Brady Campaign’s website.  As the Brady Campaign’s basic thesis is that more restrictive gun laws lead to less gun crime, the Brady Score will be the independent variable for correlation studies.

Moreover:  the UK Guardian newspaper fairly recently (October 2009) collected and made public data – obtained from US government sources – for the year 2008 regarding murders in all US states other than Florida.  (The District of Columbia was also excluded.)  Significantly, this data includes more than the overall murder rate per 100,000 residents. It also includes the firearm murder fraction – e.g., the percentage of murders committed in each state using a firearm. From this, it’s simple arithmetic to determine each state’s firearm murder rate per 100,000 residents.

From these data sources, we can obtain three sets of 49 x-y pairs, perform linear regression, and test the results for significance.  If we use Brady Score as the independent variable, doing this will give an indication as to whether or not a linear, direct cause and effect relationship exists between Brady Score and three different measures of the relative frequency of gun violence.

Use of 2008 crime data is appropriate for this comparison.  While 2009 and later data is available, the fact is that the Brady State Scorecard was published in October 2009 – so the data used regarding state laws to calculate the Brady State Scorecard was very likely 2008 or early-2009 data. (If the Brady Campaign indicated the cutoff date for their Brady State Scorecard’s data, I didn’t find it.)

We thus now have three good metrics against which to perform linear regression vis-à-vis the Brady Score and test the resulting correlation coefficient for significance. Total murder rate is the first.  If the Brady Campaign is correct, a rising Brady Score should be expected to reduce the overall murder rate by reducing firearm murders. For the same reason, firearm murder rate (naturally) is the second. Finally, the firearm murder fraction is the third; a higher Brady Score should be expected to lower the proportion of murders committed using firearms by making them less available.  Using rate data vice raw numbers of murders accounts for varying state populations.

All three of these metrics should show declines vis-à-vis rising Brady Score – if the Brady Campaign’s thesis that more restrictive firearms law leads to less gun violence is correct.  And if the cause and effect relationship is direct and significant, a linear model should describe that fairly accurately – with a correlation coefficient that is significant.  Conversely, if there isn’t a cause and effect relationship, a linear model won’t work – and the correlation coefficient computed will be insignificant.

Here’s the raw data, along with scatter plots of same.  Florida and DC are omitted as firearm murder fraction and overall murder rate were not included in the UK Guardian’s data for those jurisdictions.  The file is a MicroSoft Excel spreadsheet (2003 format), so you’ll need something that can read and display that file format to view the data and scatter plots.

Methodology

The methodology used in performing these tests was simple.  A linear model was assumed representing cause-and-effect relationships between restrictive gun laws (as measured by a state’s Brady Score) and that state’s overall Murder Rate, Firearm Murder Rate, and Firearm Murder Fraction.  The Brady Score was used as a numerical measure of the restrictiveness of a state’s firearms laws and was assumed to be the independent variable in each case.  Linear regression was then performed to determine the correlation coefficient.  If the Brady Campaign’s thesis is correct,  the expected result is a high negative correlation in each case (e.g., a higher Brady Score would be associated with a lower rate of firearm murders, overall murders, and a lower fraction of firearm murders).  Data was obtained from the sources indicated above.  Each linear regression model’s coefficient correlation was calculated, and whether these correlation coefficients were significant was determined.  The overall results were then analyzed and conclusions determined.

These specific steps were as followed:

1. Obtained Brady Score for all 50 US states.

2. Obtained Murder Rate and Firearm Murder Fraction for all US states except Florida and the District of Columbia.

3. Entered above data into an Excel spreadsheet.

4. Verified the data entered into Excel against data sources listed above.

5. Used built in MicroSoft Excel arithmetic functions to calculate state Firearm Murder Rate from Murder Rate and Firearm Murder Fraction.

6. Used the built in Excel function “CORREL” to calculate the correlation coefficient between Brady Score and Murder Rate.

7. Used the built in Excel function “CORREL” to calculate the correlation coefficient between Brady Score and Firearm Murder Rate.

8. Used the built in Excel function “CORREL” to calculate the correlation coefficient between Brady Score and Firearm Murder Fraction.

9. Analyzed resulting correlation coefficients for significance.

10. Examined results and determined conclusions.

Results

The Brady Campaign will not like the results presented below.

1. The correlation coefficient between Brady Score and Murder Rate was  near zero and positive:  +0.042418.

2. The correlation coefficient between Brady Score and Firearm Murder Rate was also near zero and positive:   +0.045577.

3. The correlation coefficient between Brady Score and Firearm Murder Fraction was less than .15 and positive:   +0.141732.

4. All correlation coefficients are positive. If a higher Brady Score was linked to a lower level of gun violence, a negative correlation would be expected in all three cases.

5. None of the calculated correlations are significant. In each case, the correlation coefficient multiplied by 7 (the square root of the 49 pairs used to calculate each correlation) was less than 1.0. A value greater than 3 for this test is required for a correlation to be deemed significant. This indicates lack of evidence of any direct cause and effect relationship between Brady Score and Murder Rate, Firearm Murder Rate, or Firearm Murder Fraction.

6. Scatter plots of the data reveal no obvious nonlinear structure, thus implying no easily-discerned nonlinear relationship between Brady Score and Murder Rate, Firearm Murder Rate, or Firearm Murder Fraction.  If anything, the scatterplots look more like 3 random noise bursts contained in a superimposed decaying sinusoidal envelope centered around a positive  constant and with the envelope decaying with increasing Brady Score.

Conclusions

1. There is no significant correlation between a state’s Brady Score and that state’s Murder Rate, Firearm Murder Rate, or Firearm Murder Fraction.

2.There is no linear relationship between restrictive gun laws (Brady Score) and a state’s rate of gun murders;  between a state’s Brady Score and it’s overall murder rate; or between a state’s Brady Score and the fraction of murders committed using guns.  This strongly implies that there is no direct cause and effect relationship between restrictive gun laws and either the overall murder rate, the firearm murder rate, or the fraction of murders committed by firearms. If there was such a direct cause and effect relationship, we would have expected to have observed a strong negative correlation (e.g., a correlation coefficient between -0.80 or so and -1.0) in each case above. Instead, a small positive correlation relatively close to zero (e.g., between 0.0 and 0.15) was observed in each case.

3. As a quantification of how restrictive a given state’s firearms laws are, the Brady Score appears meaningful.  States with high Brady Scores indeed have highly restrictive firearms laws.

4. However, as an indicator of how laws restricting firearms affect public safety the Brady Score can be described by its initials – BS.  As measured by Brady Score, restrictive firearms laws appear essentially unrelated to a state’s rate of firearms crime.  Something else is the cause of the variation.

In short:  restrictive gun laws don’t seem to be conclusively linked to reduced rates of gun violence.  In fact, there appears to be little if any linkage at all.

 

Updated  (22 June 2012) –  A downloadable version of this article is now available here.  Download and use it yourself as you see fit.  However, please ask any acquaintances to download it themselves vice sending it to them – and ask them to also click a few ads here at TAH before they leave.   If nothing else, that will help Jonn cover the hosting fees for TAH.

Category: Guns, Legal, Politics