Language agents reduce the risk of existential catastrophe

Journal article

Goldstein, Simon and Kirk-Giannini, Cameron Domenico. (2023). Language agents reduce the risk of existential catastrophe. AI & Society. pp. 1-11.
AuthorsGoldstein, Simon and Kirk-Giannini, Cameron Domenico

Recent advances in natural-language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability.

Keywordslanguage agents; existential risk; reward misspecification; goal misgeneralization; interpretable AI
JournalAI & Society
Journal citationpp. 1-11
Digital Object Identifier (DOI)
Scopus EID2-s2.0-85168370049
Page range1-11
FunderThe Center for AI Safety
Publisher's version
All rights reserved
File Access Level
Output statusIn press
Publication dates
Online19 Aug 2023
Publication process dates
Accepted03 Aug 2023
Deposited08 Nov 2023
Permalink -

Restricted files

Publisher's version

  • 25
    total views
  • 0
    total downloads
  • 2
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

A question-sensitive theory of intention
Beddor, Bob and Goldstein, Simon. (2023). A question-sensitive theory of intention. The Philosophical Quarterly. 73(2), pp. 346-378.
Getting accurate about knowledge
Carter, Sam and Goldstein, Simon. (2023). Getting accurate about knowledge. Mind. 132(525), pp. 158-191.
Attitude verbs’ local context
Blumberg, Kyle and Goldstein, Simon. (2023). Attitude verbs’ local context. Linguistics and Philosophy. 46(3), pp. 483-507.
Fragile knowledge
Goldstein, Simon. (2022). Fragile knowledge. Mind. 131(522), pp. 487-515.
Goldstein, Simon and Kirk-Giannini, Cameron Domenico. (2022). Contextology. Philosophical Studies. 179(11), pp. 3187-3209.
Sly Pete in dynamic semantics
Goldstein, Simon David. (2022). Sly Pete in dynamic semantics. Journal of Philosophical Logic. 51(5), pp. 1103-1117.
Knowledge from multiple experiences
Goldstein, Simon and Hawthorne, John. (2022). Knowledge from multiple experiences. Philosophical Studies. 179(4), pp. 1341-1372.
Counterfactual contamination
Goldstein, Simon and Hawthorne, John. (2022). Counterfactual contamination. Australasian Journal of Philosophy. 100(2), pp. 262-278.
Probability for epistemic modalities
Goldstein, Simon and Santorio, Paolo. (2021). Probability for epistemic modalities. Philosophers' Imprint. 21(33), pp. 1-37.
Mighty knowledge
Beddor, Bob and Goldstein, Simon. (2021). Mighty knowledge. Journal of Philosophy. 118(5), pp. 229-269.
The normality of error
Carter, Sam and Goldstein, Simon. (2021). The normality of error. Philosophical Studies. 178, pp. 2509-2533.
Losing confidence in luminosity
Goldstein, Simon and Waxman, Daniel. (2021). Losing confidence in luminosity. Noûs. 55(4), pp. 962-991.
Epistemic modal credence
Goldstein, Simon. (2021). Epistemic modal credence. Philosophers' Imprint. 21(26), pp. 1-24.
The counterfactual direct argument
Goldstein, Simon. (2020). The counterfactual direct argument. Linguistics and Philosophy. 43(2), pp. 193-232.
Free choice impossibilty results
Goldstein, Simon. (2020). Free choice impossibilty results. Journal of Philosophical Logic. 49(2), pp. 249-282.
Conditional heresies
Cariani, Fabrizio and Goldstein, Simon. (2020). Conditional heresies. Philosophy and Phenomenological Research. 101(2), pp. 251-282.
A theory of conditional assertion
Goldstein, Simon. (2019). A theory of conditional assertion. Journal of Philosophy. 116(6), pp. 293-318.
Generalized update semantics
Goldstein, Simon. (2019). Generalized update semantics. Mind: A Quarterly review of philosophy. 128(511), pp. 795-835.
Free choice and homogeneity
Goldstein, Simon. (2019). Free choice and homogeneity. Semantics and Pragmatics. 12, pp. 1-47.
Triviality results for probabilistic modals
Goldstein, Simon. (2019). Triviality results for probabilistic modals. Philosophy and Phenomenological Research. 99(1), pp. 188-222.
A stronger doctrine of double effect
Bronner, Ben and Goldstein, Simon. (2018). A stronger doctrine of double effect. Australasian Journal of Philosophy. 96(4), pp. 793 - 805.
Believing epistemic contradictions
Beddor, Bob and Goldstein, Simon. (2018). Believing epistemic contradictions. The Review of Symbolic Logic. 11(1), pp. 87-114.
A preface paradox for intention
Goldstein, Simon. (2016). A preface paradox for intention. Philosophers' Imprint. 16(14), pp. 1-20.