更改

According to [[Eliezer Yudkowsky]], a significant problem in AI safety is that unfriendly artificial intelligence is likely to be much easier to create than friendly AI. While both require large advances in recursive optimisation process design, friendly AI also requires the ability to make goal structures invariant under self-improvement (or the AI could transform itself into something unfriendly) and a goal structure that aligns with human values and does not automatically destroy the human race. An unfriendly AI, on the other hand, can optimize for an arbitrary goal structure, which does not need to be invariant under self-modification.<ref name="singinst12">[http://singinst.org/upload/CEV.html Coherent Extrapolated Volition, Eliezer S. Yudkowsky, May 2004 ] {{webarchive|url=https://web.archive.org/web/20100815055725/http://singinst.org/upload/CEV.html |date=2010-08-15 }}</ref> {{harvtxt|Bill Hibbard|2014}} proposes an AI design that avoids several dangers including self-delusion,<ref name="JAGI2012">{{Citation| journal=Journal of Artificial General Intelligence| year=2012| volume=3| issue=1| title=Model-Based Utility Functions| first=Bill| last=Hibbard| postscript=.| doi=10.2478/v10229-011-0013-5| page=1|arxiv = 1111.3934 |bibcode = 2012JAGI....3....1H | s2cid=8434596}}</ref> unintended instrumental actions,<ref name="selfawaresystems"/><ref name="AGI-12a">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_56.pdf  Avoiding Unintended AI Behaviors.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle. [http://intelligence.org/2012/12/19/december-2012-newsletter/ This paper won the Machine Intelligence Research Institute's 2012 Turing Prize for the Best AGI Safety Paper].</ref> and corruption of the reward generator.<ref name="AGI-12a"/> He also discusses social impacts of AI<ref name="JET2008">{{Citation| url=http://jetpress.org/v17/hibbard.htm| journal=Journal of Evolution and Technology| year=2008| volume=17| title=The Technology of Mind and a New Social Contract| first=Bill| last=Hibbard| postscript=.}}</ref> and testing AI.<ref name="AGI-12b">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_57.pdf  Decision Support for Safe AI Design|.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle.</ref> His 2001 book ''[[Super-Intelligent Machines]]'' advocates the need for public education about AI and public control over AI. It also proposed a simple design that was vulnerable to corruption of the reward generator.

According to [[Eliezer Yudkowsky]], a significant problem in AI safety is that unfriendly artificial intelligence is likely to be much easier to create than friendly AI. While both require large advances in recursive optimisation process design, friendly AI also requires the ability to make goal structures invariant under self-improvement (or the AI could transform itself into something unfriendly) and a goal structure that aligns with human values and does not automatically destroy the human race. An unfriendly AI, on the other hand, can optimize for an arbitrary goal structure, which does not need to be invariant under self-modification.<ref name="singinst12">[http://singinst.org/upload/CEV.html Coherent Extrapolated Volition, Eliezer S. Yudkowsky, May 2004 ] {{webarchive|url=https://web.archive.org/web/20100815055725/http://singinst.org/upload/CEV.html |date=2010-08-15 }}</ref> {{harvtxt|Bill Hibbard|2014}} proposes an AI design that avoids several dangers including self-delusion,<ref name="JAGI2012">{{Citation| journal=Journal of Artificial General Intelligence| year=2012| volume=3| issue=1| title=Model-Based Utility Functions| first=Bill| last=Hibbard| postscript=.| doi=10.2478/v10229-011-0013-5| page=1|arxiv = 1111.3934 |bibcode = 2012JAGI....3....1H | s2cid=8434596}}</ref> unintended instrumental actions,<ref name="selfawaresystems"/><ref name="AGI-12a">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_56.pdf  Avoiding Unintended AI Behaviors.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle. [http://intelligence.org/2012/12/19/december-2012-newsletter/ This paper won the Machine Intelligence Research Institute's 2012 Turing Prize for the Best AGI Safety Paper].</ref> and corruption of the reward generator.<ref name="AGI-12a"/> He also discusses social impacts of AI<ref name="JET2008">{{Citation| url=http://jetpress.org/v17/hibbard.htm| journal=Journal of Evolution and Technology| year=2008| volume=17| title=The Technology of Mind and a New Social Contract| first=Bill| last=Hibbard| postscript=.}}</ref> and testing AI.<ref name="AGI-12b">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_57.pdf  Decision Support for Safe AI Design|.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle.</ref> His 2001 book ''[[Super-Intelligent Machines]]'' advocates the need for public education about AI and public control over AI. It also proposed a simple design that was vulnerable to corruption of the reward generator.

按照[[Eliezer Yudkowsky]]的观点，人工智能安全的一个重要问题是，不友好的人工智能可能比友好的人工智能更容易创建。虽然两者都需要递归优化过程的进步，但友好的人工智能还需要目标结构在自我改进过程中保持不变（否则人工智能可以将自己转变成不友好的东西），以及一个与人类价值观相一致且不会自动毁灭人类的目标结构。另一方面，一个不友好的人工智能可以针对任意的目标结构进行优化，而目标结构不需要在自我改进过程中保持不变。Bill Hibbard (2014)提出了一种人工智能设计，可以避免包括自欺欺人、无意的工具性行为和奖励机制的腐败等一些危险。他还讨论了人工智能和人工智能测试的社会影响。他在2001年出版的“超级智能机器Super-Intelligent Machines”一书中提倡对人工智能的公共教育和公众控制。该书还提出了一个简单的易受奖励机制的腐败影响的设计。It also proposed a simple design that was vulnerable to corruption of the reward generator.

按照[[Eliezer Yudkowsky]]的观点，人工智能安全的一个重要问题是，不友好的人工智能可能比友好的人工智能更容易创建。虽然两者都需要递归优化过程的进步，但友好的人工智能还需要目标结构在自我改进过程中保持不变（否则人工智能可以将自己转变成不友好的东西），以及一个与人类价值观相一致且不会自动毁灭人类的目标结构。另一方面，一个不友好的人工智能可以针对任意的目标结构进行优化，<ref name="singinst12">[http://singinst.org/upload/CEV.html Coherent Extrapolated Volition, Eliezer S. Yudkowsky, May 2004 ] {{webarchive|url=https://web.archive.org/web/20100815055725/http://singinst.org/upload/CEV.html |date=2010-08-15 }}</ref> 而目标结构不需要在自我改进过程中保持不变。Bill Hibbard (2014)提出了一种人工智能设计<ref name="JAGI2012">{{Citation| journal=Journal of Artificial General Intelligence| year=2012| volume=3| issue=1| title=Model-Based Utility Functions| first=Bill| last=Hibbard| postscript=.| doi=10.2478/v10229-011-0013-5| page=1|arxiv = 1111.3934 |bibcode = 2012JAGI....3....1H | s2cid=8434596}}</ref> ，可以避免包括自欺欺人<ref name="selfawaresystems"/><ref name="AGI-12a">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_56.pdf  Avoiding Unintended AI Behaviors.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle. [http://intelligence.org/2012/12/19/december-2012-newsletter/ This paper won the Machine Intelligence Research Institute's 2012 Turing Prize for the Best AGI Safety Paper].</ref> 、无意的工具性行为和奖励机制<ref name="AGI-12a"/>的腐败等一些危险。他还讨论了人工智能<ref name="JET2008">{{Citation| url=http://jetpress.org/v17/hibbard.htm| journal=Journal of Evolution and Technology| year=2008| volume=17| title=The Technology of Mind and a New Social Contract| first=Bill| last=Hibbard| postscript=.}}</ref>和人工智能测试的社会影响<ref name="AGI-12b">[http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_57.pdf  Decision Support for Safe AI Design|.] Bill Hibbard. 2012 proceedings of the Fifth Conference on Artificial General Intelligence, eds. Joscha Bach, Ben Goertzel and Matthew Ikle.</ref> 。他在2001年出版的“超级智能机器Super-Intelligent Machines”一书中提倡对人工智能的公共教育和公众控制。该书还提出了一个简单的易受奖励机制的腐败影响的设计。It also proposed a simple design that was vulnerable to corruption of the reward generator.

===社会生物进化的下一步===

===社会生物进化的下一步===