You have seen enough of "studies show". The sentence worth stopping at is a different one: "A survey finds interpersonal trust twelve percentage points lower in country A than in country B."
It looks sturdier, because it comes with a number. But for it to hold, two gates have to be passed, and both sit before the statistics: when people in the two countries tick that box, are they holding the same ruler; and even if the number is real, is the "lower" something the country did, or were those two sets of people different to begin with?
When those gates fail, no number of decimal places matters. The research below is not here to hand you "so social science is untrustworthy" — it is the discipline measuring where it is brittle with its own tools. Once it has been measured, you can tell which kind of conclusion you can act on and which kind is only a lead.
A health form has a line on it: your overall health these past three months — very good, good, fair, poor. You tick one. Does the person beside you, ticking the same box, mean the same thing?
The economist Amartya Sen laid out a comparison in the British Medical Journal in 2002. Kerala, in India, has the country's highest literacy and highest life expectancy (around seventy-four years at the time), and the most self-reported illness of any state. Bihar has the worst medical and educational provision and the lowest rate of self-reported illness. For the same set of complaints, Americans report more than Keralans do. Measure health by whether people say they are ill, and the ranking comes out: the United States least healthy, Bihar healthiest.
Sen's explanation is simple: reporting that you are ill requires recognizing the state you are in as illness. Where the people around you live with pain as a standing condition, a great many states get counted as simply what being alive is like; someone who has seen a doctor and knows the name of the condition will report more problems from the same body. Nobody is lying. The ruler is local, and it drifts with what you have seen.
This gate has a name. Turning something hard to pin down into a number you can record is operationalization; asking afterwards whether that number is still the thing you started with is construct validity. The second question is the dangerous one.
The most widely circulated measure of social trust is stuck on it. The wording has barely changed in decades: generally speaking, would you say most people can be trusted, or that you need to be careful in dealing with people? In 2000 Glaeser and three other economists had a hundred and eighty-nine students both fill in that questionnaire and play for real money: hand cash to a stranger who may return it or keep it. The results came out crossed — people who answered "most can be trusted" were more likely to send money back, but no more willing to hand it over first. The measure called "trust" is capturing something closer to "do I keep my word". One sample, all undergraduates, so not a verdict; but from then on the question of whether the name fits was on the table.
The repair is plain: have everyone rate the same few invented people on the same scale, and the difference in their ratings is the difference in their rulers (King and colleagues, 2004). Whether a number can be compared depends not on its decimal places but on whether the people producing it held the same ruler.
"If you move to a better neighbourhood, will your children do better?" — the hard part is not the answer but how anyone could know. Families who can move are already different from families who cannot: more money, more initiative. If the children do better later, you cannot separate the neighbourhood's contribution from what the family already was. This rock is called selection.
From 1994, five American cities did something rare: housing vouchers handed out by lottery to families in high-poverty public housing, some of them restricted to moves into low-poverty neighbourhoods. The programme was Moving to Opportunity. The lottery is the whole point — who got a voucher had nothing to do with what kind of family they were, so the difference between the two groups afterwards can be put down to getting the voucher.
In 2007 Kling, Liebman and Katz reported the first round: four to seven years on, voucher families were indeed living in safer neighbourhoods, but there was no detectable overall effect on adults' employment or earnings; mental health improved substantially for adults and girls, while the gains for girls were partly offset by effects running the other way for boys. What many people read off this was that neighbourhood effects had been overstated.
In 2016 Chetty, Hendren and Katz used tax records to follow these children into their twenties. Those under thirteen when the family moved were earning, in their mid-twenties, an average of three thousand four hundred and seventy-seven dollars more a year, about thirty-one per cent above the control-group mean, and were more likely to have attended college. For those already adolescent at the move, the estimated effect was slightly negative; the authors suspect the cost of disrupted schooling.
Same experiment, same people, randomization untouched. All that changed was what was measured, on whom, and how long afterwards. The lesson is not that somebody misread it, but that the word "effect" is incomplete on its own: without "for whom, how long afterwards, and which outcome", it is not even the kind of statement that can be right or wrong.
A further spread comes from the analysts. In 2018 one study handed the same dataset and the same question, specified in advance, to twenty-nine teams; the effect sizes that came back ran from 0.89 to 2.93, with twenty teams finding a significant positive effect and nine not. So "what is the result" is often not a number but an interval, part of it coming from the world and part from whoever sits in front of the data.
You have probably seen the figure: two lines of equal length, one with arrowheads pointing out and one pointing in, and the outward one looks longer. It is the Müller-Lyer illusion.
In 1966 Segall, Campbell and Herskovits published a measurement across sixteen groups and nearly two thousand people: the mean misjudgement ran from 1.4 per cent up to 20.3 per cent, with the three European-descended samples the most susceptible and the San foragers of the Kalahari the least. The explanation was the carpentered-world hypothesis: people raised among right angles and square rooms have visual systems trained by those lines.
In 2010 Henrich, Heine and Norenzayan put this example first and raised the question that has since been cited to death: in the top journals of six subfields of psychology, 68 per cent of participants came from the United States and 96 per cent from Western industrialized countries, and an undergraduate at a Western university is some four thousand times more likely to be studied by a psychologist than someone picked at random outside. Their acronym for these populations — western, educated, industrialized, rich, democratic — spells WEIRD. The word "people" in the literature refers to a rather untypical handful of them.
But in 2025 Dorsa Amir and Chaz Firestone went back through that flagship example in Psychological Review and concluded that the carpentered-world hypothesis is almost certainly false: the illusion shows up in non-human animals living in wildly different environments, in sense modalities other than vision, and in congenitally blind subjects. More pointedly, on a rereading of the 1966 data the evidence for cultural variation, and its correlation with the cultural variables in question, turns out to be highly inconsistent.
How should that be read? Not as "the WEIRD problem was fake", but: when a discipline examines itself it uses the same fallible evidence as everything else, so the self-examination has to be examined too — which is proof that the machinery is still running. The harder evidence for the WEIRD problem lies elsewhere.
Henrich and colleagues also organized something else: taking the ultimatum game into fifteen small-scale societies. One person splits a sum of real money, the other can only accept all of it or reject all of it, and a rejection leaves both with nothing. Among European and American undergraduates offers usually come close to half, and low offers get rejected. Across these fifteen groups mean offers ran from 26 to 58 per cent: the Machiguenga of the Amazon offered 26 per cent on average and almost never rejected anything; two groups in Papua New Guinea went the other way, with some people offering more than half, and these "hyper-fair" offers were frequently turned down.
This is where the quantitative-versus-qualitative argument comes down to earth. From the numbers alone, "hyper-fair offers rejected" can only be recorded as noise. It became readable because the people running the games had lived in those villages: under those rules of gift-giving, accepting an oversized gift means taking on an obligation you cannot discharge. Numbers tell you where something differs; only someone who has lived there knows what the difference is.
And the damage a narrow sample does is never that the conclusion is wrong. It is that nobody knows how far the conclusion reaches — and on the page those two look identical, both written as "people tend to…".