BROADCASTS.com

b' https://astralcodexten.substack.com/p/willpower-human-and-machine

Two paragraphs from the mesa-optimizers post, which I quoted again in the adaptation-executors post:

Consider evolution, optimizing the fitness of animals. For a long time, it did so very mechanically, inserting behaviors like \\u201cuse this cell to detect light, then grow toward the light\\u201d or \\u201cif something has a red dot on its back, it might be a female of your species, you should mate with it\\u201d. As animals became more complicated, they started to do some of the work themselves. Evolution gave them drives, like hunger and lust, and the animals figured out ways to achieve those drives in their current situation. Evolution didn\\u2019t mechanically instill the behavior of opening my fridge and eating a Swiss Cheese slice. It instilled the hunger drive, and I figured out that the best way to satisfy it was to open my fridge and eat cheese.

And:

Mesa-optimizers would have an objective which is closely correlated with their base optimizer, but it might not be perfectly correlated. The classic example, again, is evolution. Evolution \\u201cwants\\u201d us to reproduce and pass on our genes. But my sex drive is just that: a sex drive. In the ancestral environment, where there was no porn or contraceptives, sex was a reliable proxy for reproduction; there was no reason for evolution to make me mesa-optimize for anything other than \\u201chave sex\\u201d. Now in the modern world, evolution\\u2019s proxy seems myopic - sex is a poor proxy for reproduction. I know this and I am pretty smart and that doesn\\u2019t matter. That is, just because I\\u2019m smart enough to know that evolution gave me a sex drive so I would reproduce - and not so I would have protected sex with somebody on the Pill - doesn\\u2019t mean I immediately change to wanting to reproduce instead. Evolution got one chance to set my value function when it created