Social psychology addresses many of the important questions that concern us as human beings. It’s also the subject of newspaper editorials on most days: why is there conflict between groups? How can it be reduced? Why do we trust some people and not others? What makes us angry? What are the consequences for our judgement and decision making?

A century of experimental research has applied itself to the task of addressing these and other fascinating questions. Yet this research is not routinely discussed publicly in any depth, and few people are aware it exists.

But there is one significant exception to this rule: work that addresses the question of why decent people sometimes commit appalling acts. This is a question that needs to be asked with alarming regularity — whether we are discussing the behaviour of bureaucrats in the Holocaust or torture at Abu Ghraib, corruption in Enron or phone-hacking at News International.

In all of these examples, academics and other commentators have one important body of social psychological research that they can (and do) turn to in order to make sense of the abuses they observe. Informed by Hannah Arendt’s analysis of the 1961 trial of Adolf Eichmann, this is generally referred to as the banality of evil thesis. It suggests humans have a natural tendency to conform to the rules and roles that make up group life and to the authorities that represent and enforce them.

If there is only one idea you’ve learnt from social psychology, this is probably it.

And if you know the details of any social psychological studies, then these probably relate to the two programs of research that are usually taken to support the banality of evil thesis: Stanley Milgram’s Yale-based research on Obedience to Authority and Philip Zimbardo’s Stanford Prison Experiment.

In the former, members of the general public were shown to be willing to administer apparently lethal electrical shocks to a stranger because they were asked to by an experimenter.

In the latter, college students were assigned to roles as prisoners and guards in a mock prison but the study had to be terminated after six days on account of the damage done to prisoners by increasingly cruel guards. As Zimbardo and colleagues put it, the lesson of these studies is that such abuse is a “natural consequence” of humans’ propensity to conform — even when that conformity has horrific consequences.

As scientific stories go, this is easy and dramatic to recount, and it certainly improves upon the suggestion that abuse is only ever a manifestation of the toxic personalities of wrong-doers. Indeed, for thirty years it was the best analysis on offer.

Over the course of the last decade, though, the banality of evil thesis has been subjected to increasing challenge — first from historians looking closely at the life and crimes of Nazis such as Eichmann and then from social psychologists like ourselves looking ever more closely at the specifics of the work of Milgram and Zimbardo.

Revisiting the classic experiments

In our case, a particular stimulus for our efforts was work we did together on the 2001 BBC Prison Study. The goal was to revisit the issues raised by Zimbardo’s work in a study that had a similar structure but where we had no formal role and experimental interventions were designed to test pre-specified theoretical principles.

We made the former change because we were concerned Zimbardo’s own analysis was clouded by the fact he took on the role of prison superintendent in his own study. Telling his guards, “You can create in the prisoners … a sense of fear to some degree, you can create a notion of arbitrariness. … What all this leads to is a sense of powerlessness,” seems to conflict with his claims that the ensuing violence was an instinctive expression of participants’ conformity to pre-learned societal scripts.

For us, Zimbardo’s leadership was a big part of the analytic story (as it is in all cases of tyranny), and we were struck not by the passive conformity of his guards but by their apparent enthusiasm for their task. We came to understand the guards were not merely conforming but instead displaying active followership. Furthermore, this followership appeared to be predicated upon identification with Zimbardo’s scientific enterprise.

Supporting this analysis, the guards in our study did not resort spontaneously to violence. Towards the end of the study there was, however, a move to introduce a more brutal regime. Critically, though, this only emerged once a subgroup of participants — under the influence of a charismatic leader — had come to believe that this was the best solution to problems the prison system was facing.

Tyranny, then, was not something participants slipped carelessly into. It was the result of conviction and hard work.

Spurred on by other data that support this argument, our most recent work has sought to see whether behaviour in Milgram’s paradigm could be subject to the same reanalysis.

The short answer is it can – and we outline the details in an article published today in the open-access journal PLoS Biology. We discuss evidence that the destructive behaviour of Milgram’s participants can be predicted by the degree to which they identified with his scientific goals. Moreover, the behaviour seems to have reflected a willingness to work towards goals that sprang from genuine enthusiasm for science and for Yale — enthusiasms that Milgram went to some length to cultivate.

The simple message of our research is that tyranny arises not from zombie-like conformity but from the twin processes of motivated leadership and engaged followership. What’s more, people proceed down the path to tyranny not because they are ignorant of the harm they are doing, but because they know full well what they are doing and believe it to be justified by ends that they perceive to be noble.

In these terms, what we need to be afraid of is not a nature that turns us into mindless automatons. Instead, it’s our acceptance of a particular model of “us” and “them” that commits us to the unthinkable, together with leadership that mobilises us to act on that commitment.

At the very least, 50 years on from Milgram’s research, it’s time to question the one thing we all thought we knew about social psychology.