Podcasts Technology Astral Codex Ten Podcast Constitutional AI: RLHF On Steroids

Constitutional AI: RLHF On Steroids

Published: May 12, 2023, 11:51 p.m.

b'

https://astralcodexten.substack.com/p/constitutional-ai-rlhf-on-steroids

A Machine Alignment Monday post, 5/8/23

What Is Constitutional AI?

AIs like GPT-4 go through several different

1

types of training. First, they train on giant text corpuses in order to work at all. Later, they go through a process called \\u201creinforcement learning through human feedback\\u201d (RLHF) which trains them to be \\u201cnice\\u201d. RLHF is why they (usually) won\\u2019t make up fake answers to your questions, tell you how to make a bomb, or rank all human races from best to worst.

RLHF is hard. The usual method is to make human crowdworkers rate thousands of AI responses as good or bad, then train the AI towards the good answers and away from the bad answers. But having thousands of crowdworkers rate thousands of answers is expensive and time-consuming. And it puts the AI\\u2019s ethics in the hands of random crowdworkers. Companies train these crowdworkers in what responses they want, but they\\u2019re limited by the crowdworkers\\u2019 ability to follow their rules.f

'