Trust in randomness?

Just over a decade ago, Egypt’s Coptic Christians chose their new pope. The names of three favoured candidates were placed in a glass bowl, then a blindfolded boy selected from the trio at random. Religious people can appeal to the idea that the outcome wasn’t truly random; God himself decided on Tawadros II. Yet it is a seemingly unsettling way to deal with a serious decision.

In secular settings, randomness is usually reserved for gambling and games. The words “postcode lottery” are not uttered in joyous celebration. With the notable exception of jury service, we do not usually draw lots to allocate duties, jobs or privileges.

Perhaps that is a mistake. Why not — bear with me here — allocate academic funding by lottery? Traditionally, a grant-maker would have a pot of money, invite applications, then rank them all and give grants to the best. But an alternative is to deploy a simple cut-off: every application that seems credible enough to take seriously goes into the pot and the grants are distributed at random.

Ten years ago, the Health Research Council of New Zealand began awarding funding along these lines. Several other grant-makers have followed suit, including the British Academy, which now awards about 500 grants each year using a lottery.

One benefit of this approach is efficiency. The British Academy grants are not large, £10,000 at most, and a thorough evaluation might cost nearly as much as the grant itself.

Another attraction is diversity. Hetan Shah, chief executive of the British Academy, has been pleased to see more grants go to researchers from ethnic minorities and to researchers from institutions that previously hadn’t been funded. This is partly because such researchers have been more willing to apply under the randomised process.

While a quick, transparent and even-handed process is simpler, randomisation can offer us much more than that. Whenever there is an idea, policy, treatment or procedure of uncertain value, randomly giving it to some and not to others is the ideal way to figure out what its effects truly are.

Again and again, we have assumed that expert judgment is enough, only to find that the experts didn’t really know. That is the lesson of medical history, where doctors would confidently prescribe a course of treatment that turned out to be harmful. That was true in the time of bloodletting and is still true in the modern age.

For example, antiarrhythmic drugs were widely deployed in the 1970s and 1980s in the belief that they calmed errant heartbeats and therefore saved lives. That belief was only properly tested in 1987, when a large five-year randomised trial began. It was stopped halfway through when it became clear that, while the drugs did indeed stop the errant heartbeats, they had a tendency to stop the regular heartbeats too. According to Druin Burch’s Taking the Medicine, these drugs killed 50,000 people in the US alone. It took a proper randomised trial to put a stop to the well-meaning but fatal error.

The stakes are lower at the British Academy, and the variables that might be studied are less stark than the death rate. But the principle is the same: once you randomly allocate anything, you can compare the recipients with those who missed out and start to gauge the impact.

Philip Clarke, a professor of health economics at the University of Oxford, was part of a team evaluating the New Zealand grants and will also be assessing the new approach at the British Academy. He hopes to be able to figure out, for example, whether receiving a grant enables a researcher to stay in academia, to publish more, to be cited more by other researchers, to secure other grants or to win media coverage in their research.

Without randomisation, all of these impacts are nearly impossible to gauge. Did being selected for a grant help you to publish a widely cited article? Or was the grant itself irrelevant, and you received it because you were the kind of person who publishes good work anyway? With randomisation, the impact of the grants can be measured, at least in principle.

We shouldn’t stop there. Randomisation presents a golden opportunity to learn. And once you start looking for those opportunities, you see them everywhere. Not long ago, Ben Goldacre and his colleagues at the OpenPrescribing project analysed the prescription behaviour of clinics around the NHS, figuring out who was quick to follow the latest prescription guidelines and who was prescribing expensive or outdated treatments.

When Keith Ridge, then chief pharmacist of the NHS, saw the results, he asked for a list of the worst offenders, planning to upbraid each of them personally. Goldacre had another suggestion: conduct a randomised trial of Keith Ridge, by giving him a random assortment of the worst offenders to see whether those berated actually improved as a result.

I’ve written before about researchers who used random allocations to study the impact of substantial business development grants to Nigerian entrepreneurs, or small grants to tiny Sri Lankan businesses rebuilding after the terrible tsunami of 2004. Since there is a limited amount of cash, and many deserving recipients, and since everyone can see the fairness of drawing lots, why not turn scarce resources into insight?

Perhaps it is a stretch from the Coptic pope to Keith Ridge, but it should not be a stretch to use more lotteries — and to learn from them.