Podcasts Technology Astral Codex Ten Podcast Selection Bias Is A Fact Of Life, Not An Excuse For Rejecting Internet Surveys

Selection Bias Is A Fact Of Life, Not An Excuse For Rejecting Internet Surveys

Published: Jan. 2, 2023, 11:08 a.m.

b'

https://astralcodexten.substack.com/p/selection-bias-is-a-fact-of-life

Sometimes people do amateur research through online surveys. Then they find interesting things. Then their commenters say it doesn\\u2019t count, because \\u201cselection bias!\\u201d This has been happening to Aella for years, but people try it sometimes on me too.

I think these people are operating off some model where amateur surveys necessarily have selection bias, because they only capture the survey-maker\\u2019s Twitter followers, or blog readers, or some other weird highly-selected snapshot of the Internet-using public. But real studies by professional scientists don\\u2019t have selection bias, because . . . sorry, I don\\u2019t know how their model would end this sentence.

The real studies by professional scientists usually use Psych 101 students at the professional scientists\\u2019 university. Or sometimes they will put up a flyer on a bulletin board in town, saying \\u201cEarn $10 By Participating In A Study!\\u201d in which case their population will be selected for people who want $10 (poor people, bored people, etc). Sometimes the scientists will get really into cross-cultural research, and retest their hypothesis on various primitive tribes - in which case their population will be selected for the primitive tribes that don\\u2019t murder scientists who try to study them. As far as I know, nobody in history has ever done a psychology study on a truly representative sample of the world population.

This is fine. Why?

'