Language agents reduce the risk of existential catastrophe

Journal article


Goldstein, Simon and Kirk-Giannini, Cameron Domenico. (2023). Language agents reduce the risk of existential catastrophe. AI & Society. pp. 1-11. https://doi.org/10.1007/s00146-023-01748-4
AuthorsGoldstein, Simon and Kirk-Giannini, Cameron Domenico
Abstract

Recent advances in natural-language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability.

Keywordslanguage agents; existential risk; reward misspecification; goal misgeneralization; interpretable AI
Year2023
JournalAI & Society
Journal citationpp. 1-11
PublisherSpringer
ISSN0951-5666
Digital Object Identifier (DOI)https://doi.org/10.1007/s00146-023-01748-4
Scopus EID2-s2.0-85168370049
Page range1-11
FunderThe Center for AI Safety
Publisher's version
License
All rights reserved
File Access Level
Controlled
Output statusIn press
Publication dates
Online19 Aug 2023
Publication process dates
Accepted03 Aug 2023
Deposited08 Nov 2023
Permalink -

https://acuresearchbank.acu.edu.au/item/8zy79/language-agents-reduce-the-risk-of-existential-catastrophe

Restricted files

Publisher's version

  • 45
    total views
  • 0
    total downloads
  • 0
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

AI deception : A survey of examples, risks, and potential solutions
Park, Peter S., Goldstein, Simon, O'Gara, Aidan, Chen, Michael and Hendrycks, Dan. (2024). AI deception : A survey of examples, risks, and potential solutions. Patterns. 5(5), pp. 1-16. https://doi.org/10.1016/j.patter.2024.100988
Shutdown-seeking AI
Goldstein, Simon David and Robinson, Pamela. (2024). Shutdown-seeking AI. Philosophical Studies : an international journal for philosophy in the analytic tradition. pp. 1-13. https://doi.org/10.1007/s11098-024-02099-6
A question-sensitive theory of intention
Beddor, Bob and Goldstein, Simon. (2023). A question-sensitive theory of intention. The Philosophical Quarterly. 73(2), pp. 346-378. https://doi.org/10.1093/pq/pqac031
Getting accurate about knowledge
Carter, Sam and Goldstein, Simon. (2023). Getting accurate about knowledge. Mind. 132(525), pp. 158-191. https://doi.org/10.1093/mind/fzac009
Attitude verbs’ local context
Blumberg, Kyle and Goldstein, Simon. (2023). Attitude verbs’ local context. Linguistics and Philosophy. 46(3), pp. 483-507. https://doi.org/10.1007/s10988-022-09373-y
Fragile knowledge
Goldstein, Simon. (2022). Fragile knowledge. Mind. 131(522), pp. 487-515. https://doi.org/10.1093/mind/fzab040
Contextology
Goldstein, Simon and Kirk-Giannini, Cameron Domenico. (2022). Contextology. Philosophical Studies. 179(11), pp. 3187-3209. https://doi.org/10.1007/s11098-022-01820-7
Sly Pete in dynamic semantics
Goldstein, Simon David. (2022). Sly Pete in dynamic semantics. Journal of Philosophical Logic. 51(5), pp. 1103-1117. https://doi.org/10.1007/s10992-022-09660-w
Knowledge from multiple experiences
Goldstein, Simon and Hawthorne, John. (2022). Knowledge from multiple experiences. Philosophical Studies. 179(4), pp. 1341-1372. https://doi.org/10.1007/s11098-021-01710-4
Counterfactual contamination
Goldstein, Simon and Hawthorne, John. (2022). Counterfactual contamination. Australasian Journal of Philosophy. 100(2), pp. 262-278. https://doi.org/10.1080/00048402.2021.1886129
Probability for epistemic modalities
Goldstein, Simon and Santorio, Paolo. (2021). Probability for epistemic modalities. Philosophers' Imprint. 21(33), pp. 1-37.
Mighty knowledge
Beddor, Bob and Goldstein, Simon. (2021). Mighty knowledge. Journal of Philosophy. 118(5), pp. 229-269. https://doi.org/10.5840/jphil2021118518
The normality of error
Carter, Sam and Goldstein, Simon. (2021). The normality of error. Philosophical Studies. 178, pp. 2509-2533. https://doi.org/10.1007/s11098-020-01560-6
Losing confidence in luminosity
Goldstein, Simon and Waxman, Daniel. (2021). Losing confidence in luminosity. Noûs. 55(4), pp. 962-991. https://doi.org/10.1111/nous.12348
Epistemic modal credence
Goldstein, Simon. (2021). Epistemic modal credence. Philosophers' Imprint. 21(26), pp. 1-24.
The counterfactual direct argument
Goldstein, Simon. (2020). The counterfactual direct argument. Linguistics and Philosophy. 43(2), pp. 193-232. https://doi.org/10.1007/s10988-019-09272-9
Free choice impossibilty results
Goldstein, Simon. (2020). Free choice impossibilty results. Journal of Philosophical Logic. 49(2), pp. 249-282. https://doi.org/10.1007/s10992-019-09517-9
Conditional heresies
Cariani, Fabrizio and Goldstein, Simon. (2020). Conditional heresies. Philosophy and Phenomenological Research. 101(2), pp. 251-282. https://doi.org/10.1111/phpr.12565
A theory of conditional assertion
Goldstein, Simon. (2019). A theory of conditional assertion. Journal of Philosophy. 116(6), pp. 293-318. https://doi.org/10.5840/jphil2019116620
Generalized update semantics
Goldstein, Simon. (2019). Generalized update semantics. Mind: A Quarterly review of philosophy. 128(511), pp. 795-835. https://doi.org/10.1093/mind/fzy076
Free choice and homogeneity
Goldstein, Simon. (2019). Free choice and homogeneity. Semantics and Pragmatics. 12, pp. 1-47. https://doi.org/10.3765/sp.12.23
Triviality results for probabilistic modals
Goldstein, Simon. (2019). Triviality results for probabilistic modals. Philosophy and Phenomenological Research. 99(1), pp. 188-222. https://doi.org/10.1111/phpr.12477
A stronger doctrine of double effect
Bronner, Ben and Goldstein, Simon. (2018). A stronger doctrine of double effect. Australasian Journal of Philosophy. 96(4), pp. 793 - 805. https://doi.org/10.1080/00048402.2017.1400572
Believing epistemic contradictions
Beddor, Bob and Goldstein, Simon. (2018). Believing epistemic contradictions. The Review of Symbolic Logic. 11(1), pp. 87-114. https://doi.org/10.1017/S1755020316000514
A preface paradox for intention
Goldstein, Simon. (2016). A preface paradox for intention. Philosophers' Imprint. 16(14), pp. 1-20.