In the wake of the Maine mass shootings that left 18 people dead, Republican politicians are returning to a familiar talking point: the way to prevent these frequent massacres is not changing how easy it is for Americans to access guns – it’s locking up more people with mental health issues.

The Florida governor, Ron DeSantis, said after the shooting that he did not support expanding the “red flag” laws that make it easier to temporarily take guns away from people who have threatened to hurt themselves or others. Instead, he argued on CNN, an “involuntary commitment” of the shooter would have “done the trick”. In 2018, then-president Donald Trump responded to a school shooting in Parkland, Florida, which left 17 people dead, in part by tweeting about why “a sicko” like the shooter should have been brought to a “mental institution”.

Mental health experts have spent years trying to fact-check politicians who blame mass shootings on mental illness, an equation they have called misleading, counterproductive and cruel. Among them is Dr Paul Appelbaum, a professor of psychiatry at Columbia University and one of the authors of a landmark study challenging assumptions about the connections between mental illness and violence risk.

“The primary misconception that exists among many members of the public is that gun violence, particularly mass shootings, results primarily from mental illness, and in fact that’s not true,” Appelbaum said. This widespread belief has “diverted attention from ways of making it harder for people to get guns”, he said, and instead focused attention on mental health interventions unlikely to prevent much violence.

The conversation has been lightly edited for length and clarity.

What’s the best estimate for how much violence in the United States is linked to mental illness?

Jeff Swanson at Duke published a study that estimated somewhere around 4% of violence was attributable to mental illness, which meant if you could get rid of all the violence that was caused by or related to mental illness, we’d still be left with 96% of the violence in this country. There are lots of reasons to want to provide good mental healthcare, but violence reduction is, in my view, pretty low on the list.

What do we know about the perpetrators of mass shootings in terms of mental illness?

We know two things: the vast majority of mass shooting perpetrators are not suffering from a serious mental illness. We also know that people with serious mental illness are somewhat overrepresented among mass shooters. Another way of saying that is: there’s an increased risk of mass violence by people with serious mental disorders, but nonetheless the vast majority of such incidents are not linked to serious mental illness.

What about other risk factors for violence?

What’s important to say, before we narrow the focus to mental illness, is that none of those factors are as powerfully related to violence as some common characteristics that exist across people with mental illness and without mental illness. Age: young people are much more likely to be violent than older people. Men are much more likely to be violent than women. People who use alcohol and other disinhibiting substances are much more likely to be violent than people who don’t. Prior violence is strongly associated with future violence.

Then there are things like unemployment or job instability, which is associated with violence as well.

In a major study that I was involved in, there were over 130 variables that we found to be significantly associated with violence, and the most strongly associated ones were substance use and antisocial personality disorders, a formal diagnosis that involves a lifelong pattern of disregard for the interests of others: exploitation, manipulation of others, lack of concern for others, not being sensitive to rules.

“Sociopathy”, “psychopathy”, antisocial personality, more or less refer to the same cluster of symptoms, the same groups of people who disregard legal and social norms and have difficulty restraining their impulses. It’s not what most people think of as a serious mental illness.

You were one of the authors of the 1998 MacArthur Violence Risk Assessment study, which followed 951 patients after they were discharged from a psychiatric hospital, and tracked how many committed any kind of violence. Only 6% committed a violent act against a stranger, 2% committed violence with a gun, and only 1% committed a violent act against a stranger with a gun. Can you say more about what this study revealed?

Among the most important takeaways were that the targets of violence were usually friends or relatives of that person, which is to say that stranger violence was relatively uncommon. The fear of the average person, that someone with a mental illness might come out of nowhere and attack them as they’re walking down the street is an extremely unlikely event. Most violence occurred in the apartments or living spaces of either the person with mental illness themselves, or one of their friends or relatives. Most of the violence was minor – it counted as violence because someone got shoved or kicked or punched, but serious injury was rare, and gun use was very rare.

Republican politicians have often argued, as Mike Pence put it: “The answer to mass shootings is not fewer guns; it’s more institutions for the mentally ill.” Would that actually reduce violence?

Well, insofar as most violence and most gun violence has absolutely no relationship to mental illness, and insofar as most people with mental illness who would benefit from treatment could be treated successfully in the community, or with short-term hospitalisation followed by community-based treatment, more asylums, or facilities for long-term hospitalisation, are not the answer here. They’re not the answer to the mental health problem that we have, and they’re certainly not the answer to the violence problem.

What are the effects of this perception that mental illness is linked to violence?

There’s always been stigma. That’s exacerbated by the drumbeat of stories about a link between mental illness and violence. After high-profile mass violence, when for a short period of time there seems to be a window of opportunity to achieve some sort of policy change, you will notice that among the groups that most vociferously point to mental illness is the National Rifle Association. A little bit of money is appropriated for some sort of expansion of mental health services. But people are not talking about universal background checks or getting rid of assault weapons or restricting the size of magazines. They’re not talking about things that would really be effective in terms of reducing the number of deaths that would occur every year.

Does the extra money that US lawmakers dedicate to mental illness after mass shootings actually make a difference?

It would be hard to say the mental health system has improved significantly. It’s chronically underfunded, so any additional funding is a good thing, but the funding that’s provided is often targeted to specific programs for a limited period of time. You get an infusion of funds for a few years for school-based counseling, or the development of emergency services, and then they’re sort of left on their own. It’s a haphazard approach.

You’ve noted that the US, despite being a wealthy country, lacks an adequate mental health system. What are some of the consequences of US lawmakers “being completely unwilling to fund” adequate treatment for people with serious mental disorders?

It’s become commonplace to say that there are more people with serious mental illness confined in our correctional system today than are treated in our country’s hospitals.

The Bureau of Justice Statistics in a recent report noted that about 15% of people who are in prisons and 26% of people in jails in this country meet the criteria for what they call “serious psychological distress”. They were, at that point, having serious mental symptoms. A much larger proportion, 37% in prisons and 44% in jails, reported previous diagnoses of mental disorders. So there’s a huge overrepresentation of people with serious mental disorders.

If people struggling with mental health problems end up incarcerated, does that mean they get the help they need?

There’s a general agreement that jails and prisons are terrible places for people with mental illnesses. Correctional facilities are not designed as treatment facilities, and they often lack the level of services that are necessary for effective psychiatric treatment – staff, medications, other forms of therapy. In many respects, they’re counter-therapeutic. They’re often brutal places where people with mental illnesses are subject to exploitation by fellow prisoners and staff. If you have a serious mental illness, that’s exactly the wrong place for you to be.

Why are so many people with mental health problems in prison, if that’s not an appropriate place for treatment?

Part of the answer is that there’s nowhere else for them to go. Another part of the answer, though, is that to the extent that people with mental illnesses are perceived as dangerous, they may be dealt with punitively.

After the 2012 school shooting in Newtown, Connecticut, you argued that it could be a good moment to fundamentally overhaul the US mental health system. What were you envisioning then? What still needs to be done?

We haven’t had a truly coordinated mental health policy in this country since the early 1960s, when the community health system was enacted in 1963. It had an extremely reasonable goal, which was to create a nationwide system of community mental health centers that covered every community in the country and provided a single point of contact for entry to the mental health system. Rather than Googling names of psychiatrists and trying to get on their waiting list, you had one place to call. The centers would be responsible for a given area, so they also had an incentive to do outreach, case finding, preventive services, community-based education. It was a very progressive vision for what a coordinated mental health system would look like, but it was never realized.

Only about half of the community mental health centers that were envisioned were ever built. Although there was federal funding for construction, and a little bit of startup money, there was never federal money appropriated for operations. That was supposed to be something the states would pick up, and in many cases the states did not. Federal funding went away in 1980 when the Reagan administration converted all federal mental health programs into a block grant states could use any way they wanted. That meant the clinics were on their own. Many of them turned to a model of serving insured people, which meant that the uninsured people had nowhere to go again, and they stopped the services that weren’t money makers. That was not a happy ending to what had seemed like a very promising experiment.

More than a decade since the Newtown shooting, has there been any progress in how we deal with mental health issues?

I’m not sure we’re any better off than we were 10 years ago. There has been an ongoing contraction of acute psychiatric hospital beds, because psychiatric units are not as profitable for hospitals as medical surgical units are. When hospitals get squeezed, the first place they look for savings is in their psychiatric units. They look at the space, and they say: “That space could be used for a more profitable purpose.”

For steps forward: there’s a greater recognition that crisis stabilisation services can help people stay out of hospitals, and reduce other negative consequences. The data suggest that roughly 250 people a year with serious mental illnesses are being killed by police.

You’re seeing more efforts to replace police as first responders to people with mental illnesses – to replace them with mental health professionals, who stand a greater likelihood of being able to help and reduced likelihood of causing harm.

This is part of an ongoing series on the connections between mental health and gun violence