In 2006, after meeting poverty-stricken villagers outside Buenos Aires, serial entrepreneur Blake Mycoskie established TOMS shoes. For every pair of shoes you bought, a needy child in a poor country would get a pair too.
Six years in, Mycoskie agreed to a rigorous evaluation of free shoe distribution. Across eighteen communities in El Salvador, some children were randomly selected to get free canvas loafers, while others served as a control group. The approach was like a drug trial. In effect, the toss of a coin decided who got the shoes.
The study showed that the canvas loafers didn’t go to waste: most children wore their new shoes most of the time. But the children’s health wasn’t any better, as the TOMS shoes were generally replacing older footwear. Free shoes didn’t improve children’s self-esteem, and made them feel more dependent on outsiders.
These were shocking findings. Corporate philanthropy wasn’t an add-on for TOMS — it was the firm’s founding credo. At the time the study was released, TOMS had given away more than 50 million pairs of shoes. Now a randomized trial showed that among recipients in El Salvador, free shoes weren’t doing much to improve child outcomes, and may even have been fostering a sense of dependency.
Yet rather than trying to discredit the evaluation, TOMS responded promptly. They switched from canvas loafers to sports shoes, and are addressing the dependency issue by giving the shoes to kids as rewards for regularly attending school. “TOMS is perhaps the most nimble organization any of us has ever worked with,” wrote Bruce Wydick, the University of San Francisco economist who led the evaluation.
Since the 1990s, there has been an eightfold increase in the number of randomized studies in development economics. Like the TOMS evaluation, they have often surprised us. Microcredit doesn’t seem as transformative as its early proponents had claimed. Cash grants to promising entrepreneurs work surprisingly well. Randomized trials have been run at a massive scale: one experiment in India involved 19 million people. An experiment in Mexico even randomized the building of local roads.
Randomized trials have their origins in medical research. In 1945, US military doctors attempting to treat tuberculosis gave an experimental drug, streptomycin, to three patients. One died, one went blind, and the third — Bob Dole — made a full recovery. One in three wasn’t a great success rate, so a large-scale randomized trial was conducted. The drug worked. For the first time, we had an effective treatment for a disease that has killed over one billion people.
In 1954, a trial randomly injected 600,000 US children with either polio vaccine or salt water. The treatment worked, and mass immunization began the following year. Clinical trials have also led to the rejection of ineffective treatments, such as antibiotics for the common cold, steroid injections for severe head injuries, and lung volume reduction surgery for emphysema.
If you participate in the modern economy, you’re probably part of multiple randomized trials. Among the nation’s largest restaurants, retailers and financial institutions, at least a third are running randomized experiments. The colors on the Google home page were selected based on a randomized trial. Netflix hones its algorithms based on experiments. Money transfer firm Western Union used randomized experiments to decide what combination of fixed fees and foreign exchange mark-ups to charge consumers. CVS once tested the impact of an in-store promotion by pausing it in 400 randomly selected stores. To the surprise of management, profits rose in those stores, and the promotion was axed entirely.
Randomized trials have their limits. As a wag once noted, just because parachutes haven’t been tested in a clinical trial, would you forego one when you jumped from a plane? And randomized trials won’t tell us the impact of denuclearization of the Korean peninsula, raising interest rates or permitting Sprint to merge with T-Mobile.
Additionally, even where we have randomized evidence, we should be wary about extrapolating results into very different contexts. Just because quality pre-school worked in 1960s Ypsilanti, it doesn’t mean it’s the perfect answer in modern-day Yemen.
Randomized trials can’t answer every question, but they’re a surprisingly useful tool for understanding what works. From medicine to crime, politics to business, randomized trials are changing our world.
