BROADCASTS.com

How should we think about artificial general intelligence (AGI), and the risks it might pose? What constraints exist on technical solutions to the problem of aligning superhuman AI systems with human intentions? In this episode, I talk to Richard Ngo about his report analyzing AGI safety from first principles, and recent conversations he had with Eliezer Yudkowsky about the difficulty of AI alignment.

\xa0

Topics we discuss, and timestamps:

\xa0- 00:00:40 - The nature of intelligence and AGI

\xa0\xa0 - 00:01:18 - The nature of intelligence

\xa0\xa0 - 00:06:09 - AGI: what and how

\xa0\xa0 - 00:13:30 - Single vs collective AI minds

\xa0- 00:18:57 - AGI in practice

\xa0\xa0 - 00:18:57 - Impact

\xa0\xa0 - 00:20:49 - Timing

\xa0\xa0 - 00:25:38 - Creation

\xa0\xa0 - 00:28:45 - Risks and benefits

\xa0- 00:35:54 - Making AGI safe

\xa0\xa0 - 00:35:54 - Robustness of the agency abstraction

\xa0\xa0 - 00:43:15 - Pivotal acts

\xa0- 00:50:05 - AGI safety concepts

\xa0\xa0 - 00:50:05 - Alignment

\xa0\xa0 - 00:56:14 - Transparency

\xa0\xa0 - 00:59:25 - Cooperation

\xa0- 01:01:40 - Optima and selection processes

\xa0- 01:13:33 - The AI alignment research community

\xa0\xa0 - 01:13:33 - Updates from the Yudkowsky conversation

\xa0\xa0 - 01:17:18 - Corrections to the community

\xa0\xa0 - 01:23:57 - Why others don't join

\xa0- 01:26:38 - Richard Ngo as a researcher

\xa0- 01:28:26 - The world approaching AGI

\xa0- 01:30:41 - Following Richard's work

\xa0

The transcript:\xa0axrp.net/episode/2022/03/31/episode-13-first-principles-agi-safety-richard-ngo.html

\xa0

Richard on the Alignment Forum: alignmentforum.org/users/ricraz

Richard on Twitter: twitter.com/RichardMCNgo

The AGI Safety Fundamentals course: eacambridge.org/agi-safety-fundamentals

\xa0

Materials that we mention:

\xa0- AGI Safety from First Principles: alignmentforum.org/s/mzgtmmTKKn5MuCzFJ

\xa0- Conversations with Eliezer Yudkowsky: alignmentforum.org/s/n945eovrA3oDueqtq

\xa0- The Bitter Lesson: incompleteideas.net/IncIdeas/BitterLesson.html

\xa0- Metaphors We Live By: en.wikipedia.org/wiki/Metaphors_We_Live_By

\xa0- The Enigma of Reason: hup.harvard.edu/catalog.php?isbn=9780674237827

\xa0-\xa0Draft report on AI timelines, by Ajeya Cotra: alignmentforum.org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines

\xa0- More is Different for AI: bounded-regret.ghost.io/more-is-different-for-ai/

\xa0- The Windfall Clause: fhi.ox.ac.uk/windfallclause

\xa0- Cooperative Inverse Reinforcement Learning: arxiv.org/abs/1606.03137

\xa0- Imitative Generalisation: alignmentforum.org/posts/JKj5Krff5oKMb8TjT/imitative-generalisation-aka-learning-the-prior-1

\xa0-\xa0Eliciting Latent Knowledge: docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit

\xa0-\xa0Draft report on existential risk from power-seeking AI, by Joseph Carlsmith: alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-ai

\xa0- The Most Important Century: cold-takes.com/most-important-century