Should I trust economists?

Where are the airplanes?

When I was in college it was traditional to applaud the professor on the last day of class. In my junior year I took a game theory course where the professor announced on the last day that he was ending his lecture 15 minutes early, and the last person to stop clapping would get $20. The second to last would get $10, and the third to last would get $5. After most of the students politely clapped for bit and then left, there were eight of us remaining, each talking a big game about how we didn’t have anywhere to be for a long time. Eventually, we all agreed to stop clapping at the same time, and split the money. So we counted down together….3…..2…..1……and then five of my classmates stopped clapping! The remaining three of us were shocked that any of our classmates would cooperate with strangers in a one-shot multi-person prisoners dilemma. After having a good laugh, we ended up making a similar agreement to stop, but in our case it was enforceable; we got in a tight circle and each of us grabbed the arm of our neighbor, so that nobody could renege on the agreement. The professor was nice enough to throw in an extra dollar so that we could split the pot evenly.

I tell this story mostly because I like it, but also to establish my credentials as someone for whom the economist’s way of looking at the world comes pretty naturally. I’ve co-taught a class called “Rational Choice Theory and its Critics” several times now with a colleague of mine in the economics department, but both of us were more sympathetic to rational choice theory than to the critics. And my book on idealization in epistemology was probably influenced more by Dani Rodrik’s book on economic methodology than by any other single work. So I don’t come to this topic as a skeptic of economics. Rather, in the spirit of my earlier post on Mill’s Trident, I aim to raise awkward questions for a position I’m ultimately sympathetic to.

I read the IGM surveys with interest, and tend to trust the collective judgment of the economists they poll over my own on the sorts of questions they’re asked.¹ But not everybody does; you won’t have the scour the internet for all that long to find commentators who are much less disposed to defer to the judgment of economists than I am. My sense is that in the past, skepticism about the reliability of economists was more likely to come from the political left. Perhaps the edifice of academic economics is best understood as part of the ideology of the ruling class, whose function is to provide rationalizations for maintaining the status quo rather than to be a genuine, knowledge-producing social science. Then economists’ opposition to policies like rent control, or laws against price-gouging, can be safely dismissed. Today, distrust of economists seems about as likely to come from the political right, who need some strategy for explaining away the fact that economists overwhelmingly favor free trade. But I aim to raise questions about what grounds outsiders have for trust in the discipline without any particular political agenda, though I admit it’s their contemporary political relevance that has me thinking about them.

What distinguishes the economist’s approach to economic questions—at least in its contemporary, academic form—is its commitment to mathematical modeling. Starting around the middle of the twentieth century, with figures like Paul Samuelson, mathematical formalism became the defining feature of serious economic work. Today, an aspiring academic economist needs to take real analysis; it’s often math majors, rather than economics majors, who are best prepared for graduate study in the field. This modeling orientation is part of what gives economists their distinctive claim to authority. It also creates barriers to entry. Many people have intuitions about markets, prices, and trade. But the economists’ claim to special insight rests in part on the idea that informal reasoning is unreliable in this domain—that it leads to fallacies or overlooks important equilibrium effects that only formal models can capture.

But why should we trust that mathematical modeling in economics is reliable? Even in the natural sciences, which provided the inspiration for the turn towards mathematical modeling in economics, the practice can seem “unreasonably effective.” There’s no clear a priori reason why the highly abstract structures studied by pure mathematicians—e.g., in group theory, or non-Euclidean geometry—should turn out to have such important applications in physics. Paul Dirac put it memorably:

“The mathematician plays a game in which he himself invents the rules while the physicist plays a game in which the rules are provided by nature, but as time goes on it becomes increasingly evident that the rules which the mathematician finds interesting are the same as those which nature has chosen”

The economic theorist is playing a game that can look a lot like the game of the pure mathematician, studying and proving things about the behavior of idealized populations of perfectly rational, utility maximizing, computationally unbounded agents. But it’s at least not obvious that economic theorists are as lucky as pure mathematicians in having an aesthetic preference for abstract structures that turn out to be useful in predicting and manipulating real-world phenomena.

In the wake of the financial crisis, Paul Krugman wrote a famously pessimistic essay, lamenting that “economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth.” Now I’m sure that even then, Krugman did not think the Samuelsonian revolution was a wrong turn. He did not recommend that economists return to the 19th century and trade in their equations for prose. But for outsiders less inclined to trust, and less invested in the toolkit that Krugman and others are so specialized in, what assurances can be offered that the methods of contemporary economics are fit for purpose?

With the physical sciences, engineering is what lets outsiders know that the models are appropriately tethered to reality. Most people don’t understand mechanics, but they can also see that people using models based on ideas from physics are able to build working airplanes. It would be miraculous for people relying on thoroughly erroneous principles to nevertheless succeed in getting roughly a million pounds of metal, luggage, and flesh from New York to LA in under six hours.² So the best case that could be made, to a skeptical outsider, that economists’ models are similarly tethered to economic reality would be to point to economic analogues of airplanes. That is, we’d like to see economic artifacts that obviously perform a useful function, and whose design essentially relied on methods peculiar to economics.

It’s a miracle!

When I’ve put this question to economists, and since then to AI, the first example that always comes up is auctions—especially spectrum auctions—usually followed by other examples of mechanism design (e.g., the medical school residency match system). Governments are often in a position where it makes sense to sell goods or rights via an auction. It makes sense for different parties to have rights to different part of the broadcast spectrum—so that signals won’t interfere with each other—so you need some way of determining who gets to broadcast on which part of the spectrum. An auction is a natural way to do this. But governments designing such auctions face lots of pitfalls. Bidders may collude to keep prices down: e.g., “if you promise not to outbid me on this lot, I’ll promise not to outbid you on that one”. And because the goods being sold are heterogenous—different parts of the broadcast spectrum are best for different purposes—it’s important for them to go to the parties that will make the best use of them. Different ways of designing auctions will be more or less likely to efficiently allocate the goods being sold.

My understanding is that spectrum auctions have gone much better, by uncontroversial metrics, when economists have played a central role in designing them.³ There were auctions before auction theory, and auctions afterwards, and the ones afterwards raised much more revenue than the ones before. Auction design is also important in the private sector; e.g., Google hires economists to design the auctions it uses to sell advertising.⁴ So suppose our skeptic is willing to grant this much: auction design, and perhaps mechanism design more broadly, enables economists to reliably achieve uncontroversially better results than would be possible without the use of highly abstract and sophisticated game theory.

How far does this get us towards establishing a general trust in economics, of the sort that would cut ice in discussions of rent control, or tariffs?

Not all that far, I fear. First, have you ever participated in a spectrum auction? Didn’t think so. Have you ever flown on an airplane? (or used a computer, or a gps system, or…) If mechanism design is the economic analog of airplanes…it’s a lot less important and impressive than airplanes!

While that may seem glib, the fact is that mechanism design is fundamentally simpler than most economic policy. To borrow an analogy from the physical sciences, mechanism design is like studying a closed, highly controlled system—a chemostat in a lab, where the experimenter can set pH levels, temperature, the concentration of key nutrients, and pretty much everything else relevant to bacterial growth. In such conditions, the effects of interventions are highly predictable, and precise manipulation of the system is possible. But most economic policy is more like trying to intervene in a real-world pond: an open system potentially influenced by all manner of of unobserved variables.

In an auction, the economist gets to set the rules and define the terms of engagement. It’s a system built from scratch, with limited variables and clear constraints. But broader economic policymaking rarely affords that kind of control. Policymakers step into a living system already shaped by history, politics, and culture—one they don’t fully understand and can’t easily steer. The agents involved may not behave as models predict and the effects of policy are filtered through layers of institutional and social complexity.

Once we move past the best cases of mechanism design, while it’s easy enough to find institutions whose workings depend crucially on esoteric economic modeling—central banking being an obvious example—it’s much harder to make a persuasive case that the modeling is essential to the institution’s functioning well. What would happen if central banks used simple, “set-and-forget” rules that didn’t require sophisticated modeling of the economy to apply?⁵ Or if central banks had never been established in the first place? Would we be worse off? These are extremely difficult, opaque counterfactuals; they’re nowhere near as obvious as questions about what would happen if amateurs tried to design rockets.

A natural objection to the pessimism I’ve been offering up so far is that for the last several decades the real action in economics hasn’t been in abstract theory but in empirical work. The “credibility revolution” has shifted prestige toward studies that isolate causal effects using natural experiments, or in the best case actual randomized controlled trials. For instance, if a policy is implemented in one state but not another, economists can compare outcomes in similar neighboring counties that are newly subject to different laws. Or if admission to some specialized education program is conditioned on achieving a certain score on a test, economists can compare students who just barely scored high enough and were admitted to the program to those who scored just below the threshold, on the theory that they’ll be otherwise quite similar, and thus that the comparison will isolate the effect of the program. Or in the best case, aid programs with limited budgets can randomize how their aid is dispersed, and aid recipients can be compared to a control group.

Maybe my lack of skepticism is showing, but I think it’s hard to read about these methods and not conclude that, when used properly, they can tell us a lot about what effects policies had in a particular time and place. But even the best economic experiments—natural or otherwise—get us a lot less than we’d like, as they typically face difficult questions of generalizability. You can be pretty sure that electrons in China behave much the same as electrons in Mexico. But the causal effect of an economic policy in one setting may not travel well to another, because it depends on cultural norms, informal institutions, or political dynamics—factors that vary from place to place in ways that economists lack good theories of. So even if I become convinced that, e.g., giving people lentils as an incentive worked to increase rates of vaccination in rural India, I might remain just as agnostic as before about what would work in Idaho. Moreover, many of the most pressing economic policy questions can’t really be studied by natural experiments, as they’re never implemented in random or quasi-random ways in otherwise similar places and times.

So far I’ve been trying to steel man the position of someone who supports rent control, or laws against price gouging, or steep tariffs, and isn’t much bothered by the fact that economists overwhelmingly doubt that such policies will produce their intended effects.⁶ But I do want to say a bit about why I still put such stock in the collective judgment of economists, despite basically agreeing with the criticisms above. To be clear, this is offered in the spirit of autobiography rather than argument; from here on out I’m not trying to be persuasive, but just reporting my own state of mind.

There’s an old joke about two hikers who run into a grizzly bear. One kneels down to tie his sneakers. The other says, “What are you doing? You can’t outrun a bear.” The first replies, “I don’t need to outrun the bear—I just need to outrun you.”

That’s more or less how I feel about trusting economists. I don’t have to think they’re especially good at predicting the effects of economic policies in absolute terms. I just have to think that the alternatives—relying on my own intuitions, partisan punditry, or critics armed with cherry-picked case studies—are even worse. The most serious critiques of economics rarely offer better methods. At best, they point to places where mainstream assumptions failed. That’s important. But pointing out that the quarterback threw an interception doesn’t mean you could’ve read the defense better, let alone that the team would be wise to put you in for the rest of the game.

For all its limitations, economics as a discipline at least tries to learn from its mistakes. While economics is not physics and never could be, imitating the hard sciences has some real benefits: economists care about falsification. They are expected to test their models against data and to revise or abandon them when the fit is poor. That’s an enormously valuable epistemic practice—and one that shouldn’t be taken for granted. Absent such norms, the natural human tendency—and the one I see in much criticism of economics—is to present your views in the best possible light, rather than to subject them to severe tests. And while the limitations I’ve discussed are real, I haven’t dwelled on the various counterintuitive phenomena that economics gets right; none of these successes are as impressive as airplanes, but they’re also nothing to sneeze at. So I’ll keep reading the IGM surveys.

1. What questions are they asked? It’s clear the organizers attempt to ask questions that are positive rather than normative, and my sense is they prefer questions they expect will have consensus answers, rather than ones on which there will be significant disagreement. But neither generalization holds universally.

2. You’re so lucky to be alive right now! Imagine telling this to someone in the 19th century, let alone the 10th.

3. How do I know this? I suppose I don’t. I’ve heard it, and read it, and haven’t seen anybody contest it, but I haven’t dug into it myself.

4. How powerful a consideration is this? I fear it may be a bit question begging. Because I’m already inclined to think markets tend to do a good job of finding efficient allocations of resources, including labor, I tend to think Google wouldn’t pay economists so much money to design their auctions unless Google was getting a lot of bang for its buck. But if you’re more antecedently skeptical of the sorts of modeling assumptions economists make, you also may be less impressed by the fact that knowledge of economics is rewarded by the private sector; maybe you’ll favor a broadly sociocultural explanation for the wage premium enjoyed by economics PhDs, rather than a narrowly economic one.

5. Various such rules have been proposed, though admittedly it’s still economists proposing them.

6. Of course, I’m not getting into the substantive arguments against these policies. One could argue that the argument for trusting economists is really best made case-by-case: explain the standard models of these policies, and show why they’re persuasive! I’ve got some sympathy to that response, but for now I’m interested in how far we can get just by reasoning about what sorts of methods economists are using, without getting into the details of the models those methods produce. I’m trying to stick to what Bay Area Rationalists call the “outside view”, and what epistemologists call “higher-order” evidence.