At least, that is what they answered!
In 2022 the Pew Research Center slipped a deliberately absurd question into one of its questionnaires: it asked respondents whether they were licensed to operate an SSGN-class nuclear submarine. Among those under thirty, 12% said yes. In reality the share of Americans holding that qualification rounds to zero, and were the figure genuine it would single-handedly rewrite the history of the United States Navy: nearly one young American in eight would be a qualified submariner.
This was not a study about submariners, and it was not an accident either. The question was a trap, planted on purpose. Pew was not measuring the Navy, it was testing its own instruments. That is precisely what trap questions are for, because an extremely rare characteristic makes an ideal thermometer. Their advantage lies entirely in their sensitivity: when the expected answer is zero, there is no need to establish which deviation counts as significant, since any percentage above zero is already a signal. And once that signal appears, the researcher knows to stop and look more closely before publishing anything else from that questionnaire.
Pew did not stop at a single instrument. In the same set of checks it put sixteen yes-or-no questions to participants about personal conditions and circumstances, from smoking and high blood pressure to drawing a pension or receiving compensation for a workplace injury. Some ticked ten or more of them, and those answers, taken together, added up to biographies that are all but impossible to find in the real population. Two different instruments, built independently of one another, pointed to the same conclusion.
That is how a methodological check becomes solid: not through one spectacular clue, but through controls that corroborate each other.
And the thermometer worked. That 12% did not go unnoticed: it was detected, measured and published as a methodological finding. This last point deserves a pause, because it is the most instructive of all. A research institute put into print the outcome of one of its own quality checks — the part of the work that normally stays in internal files. That is not an admission of weakness. It is how a discipline ordinarily behaves when it demands of its own instruments the same evidence it demands of its data.
The logic, in the end, is the same as that of the non-existent law we wrote about in August, the Public Affairs Act of 1975, invented out of thin air by the Cincinnati researchers in order to understand how opinions take shape on matters nobody could possibly know anything about. In both cases the question is not there to gather information, because the right answer is known in advance. It is there to measure the instrument rather than the opinion. It belongs to a family of items that almost never surfaces in a questionnaire, and that says a great deal about whoever built it.
The figure, after all, was true, in the sense that 12% really did click “yes”. It was the respondent who was not.
Asking who is really behind a percentage is not the reader’s suspicion; it is the craft of whoever builds that percentage. And it is a craft practised before fieldwork begins: trap questions on extremely rare characteristics, consistency checks between answers that cannot coexist, verification of completion times, call-backs on the least plausible cases. Things that never appear in the tables, and that separate a finding from a number.