Superintelligence

<h2 id="feasibility-of-artificial-superintelligence">Feasibility of artificial superintelligence</h2>

The creation of artificial superintelligence (ASI) has been a topic of increasing discussion in recent years, particularly with the rapid advancements in <a href="/facts/Artificial_intelligence/lJGauQwX">artificial intelligence</a> (AI) technologies.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a><a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>

<h3>Progress in AI and claims of AGI</h3>
Recent developments in AI, particularly in <a href="/facts/Large_language_model/WnogWVJY">large language models</a> (LLMs) based on the <a href="/facts/Transformer_(machine_learning_model)/cDbjx6a8">transformer</a> architecture, have led to significant improvements in various tasks. Models like <a href="/facts/GPT-3/MlKhkbSe">GPT-3</a>, <a href="/facts/GPT-4/MuXdd4sW">GPT-4</a>, <a href="/facts/Claude_3.5/9Dnt4VJH">Claude 3.5</a> and others have demonstrated capabilities that some researchers argue approach or even exhibit aspects of <a href="/facts/Artificial_general_intelligence/vbxwaq8t">artificial general intelligence</a> (AGI).<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a>
However, the claim that current LLMs constitute AGI is controversial. Critics argue that these models, while impressive, still lack true understanding and are primarily sophisticated pattern matching systems.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a>

<h3>Pathways to superintelligence</h3>
Philosopher <a href="/facts/David_Chalmers/poKbqEWc">David Chalmers</a> argues that AGI is a likely path to ASI. He posits that AI can achieve equivalence to <a href="/facts/Human_intelligence/vgHXwLNL">human intelligence</a>, be extended to surpass it, and then be amplified to dominate humans across arbitrary tasks.<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a>
More recent research has explored various potential pathways to superintelligence:

<ol><li>Scaling current AI systems – Some researchers argue that continued scaling of existing AI architectures, particularly transformer-based models, could lead to AGI and potentially ASI.<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a></li>
<li>Novel architectures – Others suggest that new AI architectures, potentially inspired by neuroscience, may be necessary to achieve AGI and ASI.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a></li>
<li>Hybrid systems – Combining different AI approaches, including symbolic AI and neural networks, could potentially lead to more robust and capable systems.<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a></li></ol>
<h3>Computational advantages</h3>
Artificial systems have several potential advantages over biological intelligence:

<ol><li>Speed – Computer components operate much faster than biological neurons. Modern microprocessors (~2 GHz) are seven orders of magnitude faster than neurons (~200 Hz).<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a></li>
<li>Scalability – AI systems can potentially be scaled up in size and computational capacity more easily than biological brains.</li>
<li>Modularity – Different components of AI systems can be improved or replaced independently.</li>
<li>Memory – AI systems can have perfect recall and vast knowledge bases. It is also much less constrained than humans when it comes to working memory.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a></li>
<li>Multitasking – AI can perform multiple tasks simultaneously in ways not possible for biological entities.</li></ol>
<h3>Potential path through transformer models</h3>
Recent advancements in transformer-based models have led some researchers to speculate that the path to ASI might lie in scaling up and improving these architectures. This view suggests that continued improvements in transformer models or similar architectures could lead directly to ASI.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>
Some experts even argue that current large language models like GPT-4 may already exhibit early signs of AGI or ASI capabilities.<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a> This perspective suggests that the transition from current AI to ASI might be more continuous and rapid than previously thought, blurring the lines between narrow AI, AGI, and ASI.
However, this view remains controversial. Critics argue that current models, while impressive, still lack crucial aspects of general intelligence such as true understanding, reasoning, and adaptability across diverse domains.<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a>
The debate over whether the path to ASI will involve a distinct AGI phase or a more direct scaling of current technologies remains ongoing, with significant implications for AI development strategies and safety considerations.

<h3>Challenges and uncertainties</h3>
Despite these potential advantages, there are significant challenges and uncertainties in achieving ASI:

<ol><li><a href="/facts/Ethics_of_artificial_intelligence/u7H0AbEV">Ethical</a> and <a href="/facts/AI_safety/Blgof0uy">safety</a> concerns – The development of ASI raises numerous ethical questions and potential risks that need to be addressed.<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a></li>
<li>Computational requirements – The computational resources required for ASI might be far beyond current capabilities.</li>
<li>Fundamental limitations – There may be fundamental limitations to intelligence that apply to both artificial and biological systems.</li>
<li>Unpredictability – The path to ASI and its consequences are highly uncertain and difficult to predict.</li></ol>
As research in AI continues to advance rapidly, the question of the feasibility of ASI remains a topic of intense debate and study in the scientific community.

<h2 id="feasibility-of-biological-superintelligence">Feasibility of biological superintelligence</h2>
<a href="/facts/Carl_Sagan/HhdhTggo">Carl Sagan</a> suggested that the advent of <a href="/facts/Caesarean_section/GJ623ers">Caesarean sections</a> and <a href="/facts/In_vitro_fertilisation/kQNMtaBu">in vitro fertilization</a> may permit humans to evolve larger heads, resulting in improvements via <a href="/facts/Natural_selection/JXTte6en">natural selection</a> in the <a href="/facts/Adaptive_evolution_in_the_human_genome/0NS4m7BP">heritable</a> component of <a href="/facts/Human_intelligence/vgHXwLNL">human intelligence</a>.<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a> By contrast, <a href="/facts/Gerald_Crabtree/ePRDPbM1">Gerald Crabtree</a> has argued that decreased selection pressure is resulting in a slow, centuries-long <a href="/facts/Fertility_and_intelligence/3HnVetzA">reduction in human intelligence</a> and that this process instead is likely to continue. There is no scientific consensus concerning either possibility and in both cases, the biological change would be slow, especially relative to rates of cultural change.
<a href="/facts/Selective_breeding/CAkkM2UG">Selective breeding</a>, <a href="/facts/Nootropics/XRMFvK7L">nootropics</a>, <a href="/facts/Epigenetics/UfAAERcn">epigenetic modulation</a>, and <a href="/facts/Genetic_engineering/vwdlDR61">genetic engineering</a> could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude improvement. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process rapidly.<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a> A well-organized society of high-intelligence humans of this sort could potentially achieve <a href="/facts/Collective_intelligence/fXxGyLtI">collective</a> superintelligence.<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a>
Alternatively, collective intelligence might be constructional by better organizing humans at present levels of individual intelligence. Several writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a <a href="/facts/Global_brain/FHg0eJA2">global brain</a> with capacities far exceeding its component agents. If this systemic superintelligence relies heavily on artificial components, however, it may qualify as an AI rather than as a biology-based <a href="/facts/Superorganism/A6LJ2NDA">superorganism</a>.<a class="footnote-ref" id="fnref:25" href="#fn:25">25</a> A <a href="/facts/Prediction_market/D9JGo7RP">prediction market</a> is sometimes considered as an example of a working collective intelligence system, consisting of humans only (assuming algorithms are not used to inform decisions).<a class="footnote-ref" id="fnref:26" href="#fn:26">26</a>
A final method of intelligence amplification would be to directly <a href="/facts/Neuroenhancement/nAQBsXP6">enhance</a> individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using <a href="/facts/Nootropics/XRMFvK7L">nootropics</a>, somatic <a href="/facts/Gene_therapy/CaxRLfKM">gene therapy</a>, or <a href="/facts/Brain%25E2%2580%2593computer_interface/wdXWqFVr">brain−computer interfaces</a>. However, Bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent <a href="/facts/Cyborg/Ih2KzUEw">cyborg</a> interface is an <a href="/facts/AI-complete/Ik4mAVRy">AI-complete</a> problem.<a class="footnote-ref" id="fnref:27" href="#fn:27">27</a>

<h2 id="forecasts">Forecasts</h2>
Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen. At the 2006 <a href="/facts/AI%4050/fEt32DIu">AI@50</a> conference, 18% of attendees reported expecting machines to be able "to simulate learning and every other aspect of human intelligence" by 2056; 41% of attendees expected this to happen sometime after 2056; and 41% expected machines to never reach that milestone.<a class="footnote-ref" id="fnref:28" href="#fn:28">28</a>
In a survey of the 100 most cited authors in AI (as of May 2013, according to Microsoft academic search), the median year by which respondents expected machines "that can carry out most human professions at least as well as a typical human" (assuming no <a href="/facts/Global_catastrophic_risks/P74aJr4H">global catastrophe</a> occurs) with 10% confidence is 2024 (mean 2034, st. dev. 33 years), with 50% confidence is 2050 (mean 2072, st. dev. 110 years), and with 90% confidence is 2070 (mean 2168, st. dev. 342 years). These estimates exclude the 1.2% of respondents who said no year would ever reach 10% confidence, the 4.1% who said 'never' for 50% confidence, and the 16.5% who said 'never' for 90% confidence. Respondents assigned a median 50% probability to the possibility that machine superintelligence will be invented within 30 years of the invention of approximately human-level machine intelligence.<a class="footnote-ref" id="fnref:29" href="#fn:29">29</a>
In a 2022 survey, the median year by which respondents expected "High-level machine intelligence" with 50% confidence is 2061. The survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human workers.<a class="footnote-ref" id="fnref:30" href="#fn:30">30</a>
In 2023, <a href="/facts/OpenAI/V7WVK1t4">OpenAI</a> leaders <a href="/facts/Sam_Altman/jhstWT0n">Sam Altman</a>, <a href="/facts/Greg_Brockman/gEsrDf4h">Greg Brockman</a> and <a href="/facts/Ilya_Sutskever/dBXbUQ39">Ilya Sutskever</a> published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.<a class="footnote-ref" id="fnref:31" href="#fn:31">31</a> In 2024, Ilya Sutskever left OpenAI to cofound the startup <a href="/facts/Safe_Superintelligence_Inc./sJYCLsCG">Safe Superintelligence</a>, which focuses solely on creating a superintelligence that is <a href="/facts/AI_safety/Blgof0uy">safe</a> by design, while avoiding "distraction by management overhead or product cycles".<a class="footnote-ref" id="fnref:32" href="#fn:32">32</a> Despite still offering no product, the startup became valued at $30 billion in February 2025.<a class="footnote-ref" id="fnref:33" href="#fn:33">33</a> In 2025, the forecast scenario "AI 2027" led by <a href="/facts/Daniel_Kokotajlo_(researcher)/okVMs0E8">Daniel Kokotakjlo</a> predicted rapid progress in the automation of coding and AI research, followed by ASI.<a class="footnote-ref" id="fnref:34" href="#fn:34">34</a>

<h2 id="design-considerations">Design considerations</h2>
The design of superintelligent AI systems raises critical questions about what values and goals these systems should have. Several proposals have been put forward:<a class="footnote-ref" id="fnref:35" href="#fn:35">35</a>

<h3>Value alignment proposals</h3>
<ul><li><a href="/facts/Coherent_extrapolated_volition/R58PN5ID">Coherent extrapolated volition</a> (CEV) – The AI should have the values upon which humans would converge if they were more knowledgeable and rational.</li>
<li>Moral rightness (MR) – The AI should be programmed to do what is morally right, relying on its superior cognitive abilities to determine ethical actions.</li>
<li>Moral permissibility (MP) – The AI should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to CEV).</li></ul>
Bostrom elaborates on these concepts:

<blockquote>instead of implementing humanity's coherent extrapolated volition, one could try to build an AI to do what is morally right, relying on the AI's superior cognitive capacities to figure out just which actions fit that description. We can call this proposal "moral rightness" (MR) ...
MR would also appear to have some disadvantages. It relies on the notion of "morally right", a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong ...

One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on moral permissibility: the idea being that we could let the AI pursue humanity's CEV so long as it did not act in morally impermissible ways.<a class="footnote-ref" id="fnref:36" href="#fn:36">36</a></blockquote>
<h3>Recent developments</h3>
Since Bostrom's analysis, new approaches to AI value alignment have emerged:

<ul><li>Inverse Reinforcement Learning (IRL) – This technique aims to infer human preferences from observed behavior, potentially offering a more robust approach to value alignment.<a class="footnote-ref" id="fnref:37" href="#fn:37">37</a></li>
<li><a href="/facts/Constitutional_AI/9Dnt4VJH">Constitutional AI</a> – Proposed by Anthropic, this involves training AI systems with explicit ethical principles and constraints.<a class="footnote-ref" id="fnref:38" href="#fn:38">38</a></li>
<li>Debate and amplification – These techniques, explored by OpenAI, use AI-assisted debate and iterative processes to better understand and align with human values.<a class="footnote-ref" id="fnref:39" href="#fn:39">39</a></li></ul>
<h3>Transformer LLMs and ASI</h3>
The rapid advancement of transformer-based LLMs has led to speculation about their potential path to ASI. Some researchers argue that scaled-up versions of these models could exhibit ASI-like capabilities:<a class="footnote-ref" id="fnref:40" href="#fn:40">40</a>

<ul><li>Emergent abilities – As LLMs increase in size and complexity, they demonstrate unexpected capabilities not present in smaller models.<a class="footnote-ref" id="fnref:41" href="#fn:41">41</a></li>
<li>In-context learning – LLMs show the ability to adapt to new tasks without fine-tuning, potentially mimicking general intelligence.<a class="footnote-ref" id="fnref:42" href="#fn:42">42</a></li>
<li>Multi-modal integration – Recent models can process and generate various types of data, including text, images, and audio.<a class="footnote-ref" id="fnref:43" href="#fn:43">43</a></li></ul>
However, critics argue that current LLMs lack true understanding and are merely sophisticated pattern matchers, raising questions about their suitability as a path to ASI.<a class="footnote-ref" id="fnref:44" href="#fn:44">44</a>

<h3>Other perspectives on artificial superintelligence</h3>
Additional viewpoints on the development and implications of superintelligence include:

<ul><li><a href="/facts/Recursive_self-improvement/xcg4M6AF">Recursive self-improvement</a> – <a href="/facts/I._J._Good/nRWnP9KP">I. J. Good</a> proposed the concept of an "intelligence explosion", where an AI system could rapidly improve its own intelligence, potentially leading to superintelligence.<a class="footnote-ref" id="fnref:45" href="#fn:45">45</a></li>
<li>Orthogonality thesis – Bostrom argues that an AI's level of intelligence is orthogonal to its final goals, meaning a superintelligent AI could have any set of motivations.<a class="footnote-ref" id="fnref:46" href="#fn:46">46</a></li>
<li><a href="/facts/Instrumental_convergence/4SfNTeJE">Instrumental convergence</a> – Certain instrumental goals (e.g., self-preservation, resource acquisition) might be pursued by a wide range of AI systems, regardless of their final goals.<a class="footnote-ref" id="fnref:47" href="#fn:47">47</a></li></ul>
<h3>Challenges and ongoing research</h3>
The pursuit of value-aligned AI faces several challenges:

<ul><li>Philosophical uncertainty in defining concepts like "moral rightness"</li>
<li>Technical complexity in translating ethical principles into precise algorithms</li>
<li>Potential for unintended consequences even with well-intentioned approaches</li></ul>
Current research directions include multi-stakeholder approaches to incorporate diverse perspectives, developing methods for scalable oversight of AI systems, and improving techniques for robust value learning.<a class="footnote-ref" id="fnref:48" href="#fn:48">48</a><a class="footnote-ref" id="fnref:49" href="#fn:49">49</a>
Al research is rapidly progressing towards superintelligence. Addressing these design challenges remains crucial for creating ASI systems that are both powerful and aligned with human interests.

<h2 id="potential-threat-to-humanity">Potential threat to humanity</h2>
Main articles: <a href="/facts/Existential_risk_from_artificial_general_intelligence/j7oxWDYY">Existential risk from artificial general intelligence</a>, <a href="/facts/AI_alignment/Blgof0uy">AI alignment</a>, and <a href="/facts/AI_safety/Blgof0uy">AI safety</a>
The development of artificial superintelligence (ASI) has raised concerns about potential existential risks to humanity. Researchers have proposed various scenarios in which an ASI could pose a significant threat:

<h3>Intelligence explosion and control problem</h3>
Some researchers argue that through recursive self-improvement, an ASI could rapidly become so powerful as to be beyond human control. This concept, known as an "intelligence explosion", was first proposed by I. J. Good in 1965:

<blockquote>Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.<a class="footnote-ref" id="fnref:50" href="#fn:50">50</a></blockquote>
This scenario presents the AI control problem: how to create an ASI that will benefit humanity while avoiding unintended harmful consequences.<a class="footnote-ref" id="fnref:51" href="#fn:51">51</a> Eliezer Yudkowsky argues that solving this problem is crucial before ASI is developed, as a superintelligent system might be able to thwart any subsequent attempts at control.<a class="footnote-ref" id="fnref:52" href="#fn:52">52</a>

<h3>Unintended consequences and goal misalignment</h3>
Even with benign intentions, an ASI could potentially cause harm due to misaligned goals or unexpected interpretations of its objectives. Nick Bostrom provides a stark example of this risk:

<blockquote>When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.<a class="footnote-ref" id="fnref:53" href="#fn:53">53</a></blockquote>
Stuart Russell offers another illustrative scenario:

<blockquote>A system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances, rather than to improve the external world.<a class="footnote-ref" id="fnref:54" href="#fn:54">54</a></blockquote>
These examples highlight the potential for catastrophic outcomes even when an ASI is not explicitly designed to be harmful, underscoring the critical importance of precise goal specification and alignment.

<h3>Potential mitigation strategies</h3>
Researchers have proposed various approaches to mitigate risks associated with ASI:

<ul><li><a href="/facts/AI_capability_control/A7SMMkAq">Capability control</a> – Limiting an ASI's ability to influence the world, such as through physical isolation or restricted access to resources.<a class="footnote-ref" id="fnref:55" href="#fn:55">55</a></li>
<li>Motivational control – Designing ASIs with goals that are fundamentally aligned with human values.<a class="footnote-ref" id="fnref:56" href="#fn:56">56</a></li>
<li>Ethical AI – Incorporating ethical principles and decision-making frameworks into ASI systems.<a class="footnote-ref" id="fnref:57" href="#fn:57">57</a></li>
<li>Oversight and governance – Developing robust international frameworks for the development and deployment of ASI technologies.<a class="footnote-ref" id="fnref:58" href="#fn:58">58</a></li></ul>
Despite these proposed strategies, some experts, such as Roman Yampolskiy, argue that the challenge of controlling a superintelligent AI might be fundamentally unsolvable, emphasizing the need for extreme caution in ASI development.<a class="footnote-ref" id="fnref:59" href="#fn:59">59</a>

<h3>Debate and skepticism</h3>
Not all researchers agree on the likelihood or severity of ASI-related existential risks. Some, like <a href="/facts/Rodney_Brooks/JcAvm7Wp">Rodney Brooks</a>, argue that fears of superintelligent AI are overblown and based on unrealistic assumptions about the nature of intelligence and technological progress.<a class="footnote-ref" id="fnref:60" href="#fn:60">60</a> Others, such as <a href="/facts/Joanna_Bryson/gIdKF6bB">Joanna Bryson</a>, contend that <a href="/facts/Anthropomorphism/foTf9q24">anthropomorphizing</a> AI systems leads to misplaced concerns about their potential threats.<a class="footnote-ref" id="fnref:61" href="#fn:61">61</a>

<h3>Recent developments and current perspectives</h3>
The rapid advancement of LLMs and other AI technologies has intensified debates about the proximity and potential risks of ASI. While there is no scientific consensus, some researchers and AI practitioners argue that current AI systems may already be approaching AGI or even ASI capabilities.

<ul><li>LLM capabilities – Recent LLMs like GPT-4 have demonstrated unexpected abilities in areas such as reasoning, problem-solving, and multi-modal understanding, leading some to speculate about their potential path to ASI.<a class="footnote-ref" id="fnref:62" href="#fn:62">62</a></li>
<li>Emergent behaviors – Studies have shown that as AI models increase in size and complexity, they can exhibit emergent capabilities not present in smaller models, potentially indicating a trend towards more general intelligence.<a class="footnote-ref" id="fnref:63" href="#fn:63">63</a></li>
<li>Rapid progress – The pace of AI advancement has led some to argue that we may be closer to ASI than previously thought, with potential implications for existential risk.<a class="footnote-ref" id="fnref:64" href="#fn:64">64</a></li></ul>
A minority of researchers and observers, including some in the AI development community, believe that current AI systems may already be at or near AGI levels, with ASI potentially following in the near future. This view, while not widely accepted in the scientific community, is based on observations of rapid progress in AI capabilities and unexpected emergent behaviors in large models.<a class="footnote-ref" id="fnref:65" href="#fn:65">65</a>
However, many experts caution against premature claims of AGI or ASI, arguing that current AI systems, despite their impressive capabilities, still lack true understanding and general intelligence.<a class="footnote-ref" id="fnref:66" href="#fn:66">66</a> They emphasize the significant challenges that remain in achieving human-level intelligence, let alone superintelligence.
The debate surrounding the current state and trajectory of AI development underscores the importance of continued research into AI safety and ethics, as well as the need for robust governance frameworks to manage potential risks as AI capabilities continue to advance.<a class="footnote-ref" id="fnref:67" href="#fn:67">67</a>

<h2 id="see-also">See also</h2>

<ul><li><a href="/facts/Artificial_general_intelligence/vbxwaq8t">Artificial general intelligence</a></li>
<li><a href="/facts/AI_safety/Blgof0uy">AI safety</a></li>
<li><a href="/facts/AI_takeover/xE6EAuTD">AI takeover</a></li>
<li><a href="/facts/Artificial_brain/ohpdSjWe">Artificial brain</a></li>
<li><a href="/facts/Artificial_intelligence_arms_race/UFAz5cfM">Artificial intelligence arms race</a></li>
<li><a href="/facts/Effective_altruism/dIIN7SrC">Effective altruism</a></li>
<li><a href="/facts/Ethics_of_artificial_intelligence/u7H0AbEV">Ethics of artificial intelligence</a></li>
<li><a href="/facts/Existential_risk/P74aJr4H">Existential risk</a></li>
<li><a href="/facts/Friendly_artificial_intelligence/R58PN5ID">Friendly artificial intelligence</a></li>
<li><a href="/facts/Future_of_Humanity_Institute/552hPF9e">Future of Humanity Institute</a></li>
<li><a href="/facts/Intelligent_agent/8C2Q93Vw">Intelligent agent</a></li>
<li><a href="/facts/Machine_ethics/c7ndkqCC">Machine ethics</a></li>
<li><a href="/facts/Machine_Intelligence_Research_Institute/LfTdXVp9">Machine Intelligence Research Institute</a></li>
<li><a href="/facts/Machine_learning/e0w0XJTu">Machine learning</a></li>
<li><a href="/facts/Neural_scaling_law/B372rC8f">Neural scaling law</a> – Statistical law in machine learning</li>
<li><a href="/facts/Noosphere/zCetV7d7">Noosphere</a> – Philosophical concept of biosphere successor via humankind's rational activities</li>
<li><a href="/facts/Outline_of_artificial_intelligence/gpySbMRe">Outline of artificial intelligence</a></li>
<li><a href="/facts/Posthumanism/xylaD2tQ">Posthumanism</a></li>
<li><a href="/facts/Robotics/wkmvodJz">Robotics</a></li>
<li><a href="/facts/Self-replication/NbsfzT4l">Self-replication</a></li>
<li><a href="/facts/Self-replicating_machine/c95FbumY">Self-replicating machine</a></li>
<li><a href="/facts/Superintelligence%3a_Paths%2c_Dangers%2c_Strategies/mQkwHpAL">Superintelligence: Paths, Dangers, Strategies</a></li></ul>

<h3>Papers</h3>
<ul><li><a href="/facts/Nick_Bostrom/kyNalABj">Bostrom, Nick</a> (2002), <a href="http://www.nickbostrom.com/existential/risks.html">"Existential Risks"</a>, <a href="/facts/Journal_of_Evolution_and_Technology/ckgVJAk5">Journal of Evolution and Technology</a>, 9, retrieved 2007-08-07.</li>
<li><a href="/facts/David_Chalmers/poKbqEWc">Chalmers, David</a> (2010). <a href="http://consc.net/papers/singularity.pdf">"The Singularity: A Philosophical Analysis"</a> (PDF). Journal of Consciousness Studies. 17: 7–65.</li>
<li>Legg, Shane (2008). <a href="http://www.vetta.org/documents/Machine_Super_Intelligence.pdf">Machine Super Intelligence</a> (PDF) (PhD). Department of Informatics, University of Lugano. Retrieved September 19, 2014.</li>
<li><a href="/facts/Vincent_C._M%25C3%25BCller/JGrMIne3">Müller, Vincent C.</a>; <a href="/facts/Nick_Bostrom/kyNalABj">Bostrom, Nick</a> (2016). <a href="http://www.philpapers.org/archive/MLLFPI">"Future Progress in Artificial Intelligence: A Survey of Expert Opinion"</a>. In Müller, Vincent C. (ed.). Fundamental Issues of Artificial Intelligence. Springer. pp. 553–571.</li>
<li>Santos-Lang, Christopher (2014). <a href="https://web.archive.org/web/20140729094949/http://grinfree.com/Responsibility.pdf">"Our responsibility to manage evaluative diversity"</a> (PDF). ACM SIGCAS Computers and Society. 44 (2): 16–19. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1145%2F2656870.2656874">10.1145/2656870.2656874</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:5649158">5649158</a>. Archived from the original on July 29, 2014.</li></ul>
<h3>Books</h3>
<ul><li><a href="/facts/Bill_Hibbard/7oIAFsrG">Hibbard, Bill</a> (2002). Super-Intelligent Machines. Kluwer Academic/Plenum Publishers.</li>
<li><a href="/facts/Nick_Bostrom/kyNalABj">Bostrom, Nick</a> (2014). <a href="/facts/Superintelligence%3a_Paths%2c_Dangers%2c_Strategies/mQkwHpAL">Superintelligence: Paths, Dangers, Strategies</a>. Oxford University Press.</li>
<li><a href="/facts/Max_Tegmark/4Y3DefCk">Tegmark, Max</a> (2018). <a href="/facts/Life_3.0/dmIIVjpC">Life 3.0: being human in the age of artificial intelligence</a>. London, England. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-14-198180-2. <a href="/facts/OCLC_(identifier)/Yqad8waq">OCLC</a> <a href="https://search.worldcat.org/oclc/1018461467">1018461467</a>.{{cite book}}: CS1 maint: location missing publisher (link)</li>
<li>Russell, Stuart J. (2019). <a href="/facts/Human_Compatible/UgJetTxz">Human compatible: artificial intelligence and the problem of control</a>. New York. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-525-55861-3. <a href="/facts/OCLC_(identifier)/Yqad8waq">OCLC</a> <a href="https://search.worldcat.org/oclc/1113410915">1113410915</a>.{{cite book}}: CS1 maint: location missing publisher (link)</li>
<li><a href="/facts/Nada_Sanders/q2nvc1IX">Sanders, Nada R.</a> (2020). The humachine: humankind, machines, and the future of enterprise. John D. Wood (First ed.). New York, New York. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-429-00117-8. <a href="/facts/OCLC_(identifier)/Yqad8waq">OCLC</a> <a href="https://search.worldcat.org/oclc/1119391268">1119391268</a>.{{cite book}}: CS1 maint: location missing publisher (link)</li></ul>
<h2 id="external-links">External links</h2>
<ul><li><a href="https://evolutionnews.org/2015/02/bill_gates_join/">Bill Gates Joins Stephen Hawking in Fears of a Coming Threat from "Superintelligence"</a></li>
<li><a href="https://reason.com/2014/09/12/will-superintelligent-machines-destroy-h/">Will Superintelligent Machines Destroy Humanity?</a></li>
<li><a href="https://www.ign.com/articles/2015/03/25/apple-co-founder-has-sense-of-foreboding-about-artificial-superintelligence">Apple Co-founder Has Sense of Foreboding About Artificial Superintelligence</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Mucci, Tim; Stryker, Cole (2023-12-14). "What Is Artificial Superintelligence? | IBM". www.ibm.com. Retrieved 2025-04-17. <a href="https://www.ibm.com/think/topics/artificial-superintelligence" target="_blank">https://www.ibm.com/think/topics/artificial-superintelligence</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Bostrom 2014, Chapter 2. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Bostrom 2014, p. 22. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Pearce, David (2012), Eden, Amnon H.; Moor, James H.; Søraker, Johnny H.; Steinhart, Eric (eds.), "The Biointelligence Explosion: How Recursively Self-Improving Organic Robots will Modify their Own Source Code and Bootstrap Our Way to Full-Spectrum Superintelligence", Singularity Hypotheses, The Frontiers Collection, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 199–238, doi:10.1007/978-3-642-32560-1_11, ISBN 978-3-642-32559-5, retrieved 2022-01-16 <a href="978-3-642-32559-5" target="_blank">978-3-642-32559-5</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Gouveia, Steven S., ed. (2020). "ch. 4, "Humans and Intelligent Machines: Co-evolution, Fusion or Replacement?", David Pearce". The Age of Artificial Intelligence: An Exploration. Vernon Press. ISBN 978-1-62273-872-4. <a href="978-1-62273-872-4" target="_blank">978-1-62273-872-4</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Bostrom 2014, Chapter 2. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Legg 2008, pp. 135–137. - Legg, Shane (2008). Machine Super Intelligence (PDF) (PhD). Department of Informatics, University of Lugano. Retrieved September 19, 2014. <a href="http://www.vetta.org/documents/Machine_Super_Intelligence.pdf" target="_blank">http://www.vetta.org/documents/Machine_Super_Intelligence.pdf</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">"'Superintelligence' is the next big thing for OpenAI: Sam Altman". The Economic Times. 2025-01-06. ISSN 0013-0389. Retrieved 2025-02-01. <a href="https://economictimes.indiatimes.com/tech/artificial-intelligence/superintelligence-is-the-next-big-thing-for-openai-sam-altman/articleshow/116993208.cms?from=mdr" target="_blank">https://economictimes.indiatimes.com/tech/artificial-intelligence/superintelligence-is-the-next-big-thing-for-openai-sam-altman/articleshow/116993208.cms?from=mdr</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">"OpenAI co-founder Sutskever sets up new AI company devoted to 'safe superintelligence'". AP News. 2024-06-20. Retrieved 2025-02-01. <a href="https://apnews.com/article/openai-sutskever-altman-artificial-intelligence-safety-c6b48a3675fb3fb459859dece2b45499" target="_blank">https://apnews.com/article/openai-sutskever-altman-artificial-intelligence-safety-c6b48a3675fb3fb459859dece2b45499</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">"Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI". Futurism. 23 March 2023. Retrieved 2023-12-13. <a href="https://futurism.com/gpt-4-sparks-of-agi" target="_blank">https://futurism.com/gpt-4-sparks-of-agi</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Marcus, Gary; Davis, Ernest (2023). "GPT-4 and Beyond: The Future of Artificial Intelligence". arXiv:2303.10130 [econ.GN]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Chalmers 2010, p. 7. - Chalmers, David (2010). "The Singularity: A Philosophical Analysis" (PDF). Journal of Consciousness Studies. 17: 7–65. <a href="http://consc.net/papers/singularity.pdf" target="_blank">http://consc.net/papers/singularity.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Kaplan, Jared; McCandlish, Sam; Henighan, Tom; Brown, Tom B.; Chess, Benjamin; Child, Rewon; Gray, Scott; Radford, Alec; Wu, Jeffrey; Amodei, Dario (2020). "Scaling Laws for Neural Language Models". arXiv:2001.08361 [cs.LG]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Hassabis, Demis; Kumaran, Dharshan; Summerfield, Christopher; Botvinick, Matthew (2017). "Neuroscience-Inspired Artificial Intelligence". Neuron. 95 (2): 245–258. doi:10.1016/j.neuron.2017.06.011. PMID 28728020. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Garcez, Artur d'Avila; Lamb, Luis C. (2020). "Neurosymbolic AI: The 3rd Wave". arXiv:2012.05876 [cs.AI]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Bostrom 2014, p. 59. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Bostrom 2014, p. 59. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Sutskever, Ilya (2023). "A Brief History of Scaling". ACM Queue. 21 (4): 31–43. doi:10.1145/3595878.3605016 (inactive 5 June 2025).{{cite journal}}: CS1 maint: DOI inactive as of June 2025 (link) <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Lee, Yin Tat; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Precup, Doina; Sountsov, Pavel; Srivastava, Sanjana; Tessler, Catherine; Tian, Jianfeng; Zaheer, Manzil (22 March 2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Marcus, Gary (2020). "The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence". arXiv:2002.06177 [cs.AI]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Russell 2019. - Russell, Stuart J. (2019). Human compatible: artificial intelligence and the problem of control. New York. ISBN 978-0-525-55861-3. OCLC 1113410915. <a href="https://search.worldcat.org/oclc/1113410915" target="_blank">https://search.worldcat.org/oclc/1113410915</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Sagan, Carl (1977). The Dragons of Eden. Random House. <a href="/wiki/Carl_Sagan" target="_blank">/wiki/Carl_Sagan</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Bostrom 2014, pp. 37–39. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Bostrom 2014, p. 39. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">Bostrom 2014, pp. 48–49. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
<li id="fn:26">Watkins, Jennifer H. (2007), Prediction Markets as an Aggregation Mechanism for Collective Intelligence <a href="https://escholarship.org/uc/item/8mg0p0zc" target="_blank">https://escholarship.org/uc/item/8mg0p0zc</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></li>
<li id="fn:27">Bostrom 2014, pp. 36–37, 42, 47. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:27" class="footnote-back-ref">↩</a></li>
<li id="fn:28">Maker, Meg Houston (July 13, 2006). "AI@50: First Poll". Archived from the original on 2014-05-13. <a href="https://web.archive.org/web/20140513052243/http://www.megmaker.com/2006/07/ai50_first_poll.html" target="_blank">https://web.archive.org/web/20140513052243/http://www.megmaker.com/2006/07/ai50_first_poll.html</a> <a href="#fnref:28" class="footnote-back-ref">↩</a></li>
<li id="fn:29">Müller & Bostrom 2016, pp. 3–4, 6, 9–12. - Müller, Vincent C.; Bostrom, Nick (2016). "Future Progress in Artificial Intelligence: A Survey of Expert Opinion". In Müller, Vincent C. (ed.). Fundamental Issues of Artificial Intelligence. Springer. pp. 553–571. <a href="http://www.philpapers.org/archive/MLLFPI" target="_blank">http://www.philpapers.org/archive/MLLFPI</a> <a href="#fnref:29" class="footnote-back-ref">↩</a></li>
<li id="fn:30">Roser, Max (7 February 2023). "AI timelines: What do experts in artificial intelligence expect for the future?". Our World in Data. Retrieved 2023-08-09. <a href="https://ourworldindata.org/ai-timelines" target="_blank">https://ourworldindata.org/ai-timelines</a> <a href="#fnref:30" class="footnote-back-ref">↩</a></li>
<li id="fn:31">"Governance of superintelligence". openai.com. Retrieved 2023-05-30. <a href="https://openai.com/blog/governance-of-superintelligence" target="_blank">https://openai.com/blog/governance-of-superintelligence</a> <a href="#fnref:31" class="footnote-back-ref">↩</a></li>
<li id="fn:32">Vance, Ashlee (June 19, 2024). "Ilya Sutskever Has a New Plan for Safe Superintelligence". Bloomberg. Retrieved 2024-06-19. <a href="https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab" target="_blank">https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab</a> <a href="#fnref:32" class="footnote-back-ref">↩</a></li>
<li id="fn:33">"There's Something Very Weird About This $30 Billion AI Startup by a Man Who Said Neural Networks May Already Be Conscious". Futurism. 2025-02-24. Retrieved 2025-04-27. <a href="https://futurism.com/ilya-sutskever-safe-superintelligence-product" target="_blank">https://futurism.com/ilya-sutskever-safe-superintelligence-product</a> <a href="#fnref:33" class="footnote-back-ref">↩</a></li>
<li id="fn:34">Roose, Kevin (2025-04-03). "This A.I. Forecast Predicts Storms Ahead". The New York Times. ISSN 0362-4331. Retrieved 2025-04-27. <a href="https://www.nytimes.com/2025/04/03/technology/ai-futures-project-ai-2027.html" target="_blank">https://www.nytimes.com/2025/04/03/technology/ai-futures-project-ai-2027.html</a> <a href="#fnref:34" class="footnote-back-ref">↩</a></li>
<li id="fn:35">Bostrom 2014, pp. 209–221. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:35" class="footnote-back-ref">↩</a></li>
<li id="fn:36">Bostrom 2014, pp. 209–221. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:36" class="footnote-back-ref">↩</a></li>
<li id="fn:37">Christiano, Paul; Leike, Jan; Brown, Tom B.; Martic, Miljan; Legg, Shane; Amodei, Dario (2017). "Deep Reinforcement Learning from Human Preferences" (PDF). NeurIPS. arXiv:1706.03741. <a href="https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf" target="_blank">https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf</a> <a href="#fnref:37" class="footnote-back-ref">↩</a></li>
<li id="fn:38">"Constitutional AI: Harmlessness from AI Feedback". Anthropic. December 15, 2022. <a href="https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback" target="_blank">https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback</a> <a href="#fnref:38" class="footnote-back-ref">↩</a></li>
<li id="fn:39">"Learning complex goals with iterated amplification". OpenAI. October 22, 2018. <a href="https://openai.com/index/learning-complex-goals-with-iterated-amplification/" target="_blank">https://openai.com/index/learning-complex-goals-with-iterated-amplification/</a> <a href="#fnref:39" class="footnote-back-ref">↩</a></li>
<li id="fn:40">Bommasani, Rishi; et al. (2021). "On the Opportunities and Risks of Foundation Models". Stanford University. arXiv:2108.07258. <a href="https://crfm.stanford.edu/report.html" target="_blank">https://crfm.stanford.edu/report.html</a> <a href="#fnref:40" class="footnote-back-ref">↩</a></li>
<li id="fn:41">Wei, Jason; Tay, Yi; Bommasani, Rishi; Raffel, Colin; Zoph, Barret; Borgeaud, Sebastian; Yogatama, Dani; Bosma, Maarten; Zhou, Denny; Metzler, Donald; Chi, Ed H.; Hashimoto, Tatsunori; Vinyals, Oriol; Liang, Percy; Dean, Jeff; Fedus, William (2022-06-26). "Emergent Abilities of Large Language Models". Transactions on Machine Learning Research. arXiv:2206.07682. ISSN 2835-8856. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:41" class="footnote-back-ref">↩</a></li>
<li id="fn:42">Brown, Tom B.; et al. (2020). "Language Models are Few-Shot Learners". NeurIPS. arXiv:2005.14165. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:42" class="footnote-back-ref">↩</a></li>
<li id="fn:43">Alayrac, Jean-Baptiste; Donahue, Jeff; Luc, Pauline; Miech, Antoine; Barr, Iain; Hasson, Yana; Lenc, Karel; Mensch, Arthur; Millican, Katie; Reynolds, Malcolm; Ring, Roman; Rutherford, Eliza; Cabi, Serkan; Han, Tengda; Gong, Zhitao; Samangooei, Sina; Monteiro, Marianne; Menick, Jacob; Borgeaud, Sebastian; Brock, Andrew; Nematzadeh, Aida; Sharifzadeh, Sahand; Binkowski, Mikolaj; Barreira, Ricardo; Vinyals, Oriol; Zisserman, Andrew; Simonyan, Karen (2022). "Flamingo: a Visual Language Model for Few-Shot Learning". NeurIPS. arXiv:2204.14198. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:43" class="footnote-back-ref">↩</a></li>
<li id="fn:44">Marcus, Gary (August 11, 2022). "Deep Learning Alone Isn't Getting Us To Human-Like AI". Noema. <a href="https://www.noemamag.com/deep-learning-alone-isnt-getting-us-to-human-like-ai/" target="_blank">https://www.noemamag.com/deep-learning-alone-isnt-getting-us-to-human-like-ai/</a> <a href="#fnref:44" class="footnote-back-ref">↩</a></li>
<li id="fn:45">"The AI apocalypse: will the human race soon be terminated?". The Irish Times. March 30, 2017. <a href="https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220" target="_blank">https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220</a> <a href="#fnref:45" class="footnote-back-ref">↩</a></li>
<li id="fn:46">Bostrom, Nick (2012). "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents" (PDF). Minds and Machines. 22 (2): 71–85. doi:10.1007/s11023-012-9281-3. <a href="https://nickbostrom.com/superintelligentwill.pdf" target="_blank">https://nickbostrom.com/superintelligentwill.pdf</a> <a href="#fnref:46" class="footnote-back-ref">↩</a></li>
<li id="fn:47">Omohundro, Stephen M. (January 2008). "The basic AI drives" (PDF). Frontiers in Artificial Intelligence and Applications. <a href="https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf" target="_blank">https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf</a> <a href="#fnref:47" class="footnote-back-ref">↩</a></li>
<li id="fn:48">Gabriel, Iason (2020-09-01). "Artificial Intelligence, Values, and Alignment". Minds and Machines. 30 (3): 411–437. arXiv:2001.09768. doi:10.1007/s11023-020-09539-2. ISSN 1572-8641. <a href="https://doi.org/10.1007%2Fs11023-020-09539-2" target="_blank">https://doi.org/10.1007%2Fs11023-020-09539-2</a> <a href="#fnref:48" class="footnote-back-ref">↩</a></li>
<li id="fn:49">Russell 2019. - Russell, Stuart J. (2019). Human compatible: artificial intelligence and the problem of control. New York. ISBN 978-0-525-55861-3. OCLC 1113410915. <a href="https://search.worldcat.org/oclc/1113410915" target="_blank">https://search.worldcat.org/oclc/1113410915</a> <a href="#fnref:49" class="footnote-back-ref">↩</a></li>
<li id="fn:50">Good, I. J. (1965). "Speculations Concerning the First Ultraintelligent Machine". Advances in Computers. <a href="#fnref:50" class="footnote-back-ref">↩</a></li>
<li id="fn:51">Russell 2019, pp. 137–160. - Russell, Stuart J. (2019). Human compatible: artificial intelligence and the problem of control. New York. ISBN 978-0-525-55861-3. OCLC 1113410915. <a href="https://search.worldcat.org/oclc/1113410915" target="_blank">https://search.worldcat.org/oclc/1113410915</a> <a href="#fnref:51" class="footnote-back-ref">↩</a></li>
<li id="fn:52">Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (PDF). Global Catastrophic Risks. doi:10.1093/oso/9780198570509.003.0021. ISBN 978-0-19-857050-9. <a href="978-0-19-857050-9" target="_blank">978-0-19-857050-9</a> <a href="#fnref:52" class="footnote-back-ref">↩</a></li>
<li id="fn:53">Bostrom 2002. - Bostrom, Nick (2002), "Existential Risks", Journal of Evolution and Technology, 9, retrieved 2007-08-07 <a href="http://www.nickbostrom.com/existential/risks.html" target="_blank">http://www.nickbostrom.com/existential/risks.html</a> <a href="#fnref:53" class="footnote-back-ref">↩</a></li>
<li id="fn:54">Russell 2019, p. 136. - Russell, Stuart J. (2019). Human compatible: artificial intelligence and the problem of control. New York. ISBN 978-0-525-55861-3. OCLC 1113410915. <a href="https://search.worldcat.org/oclc/1113410915" target="_blank">https://search.worldcat.org/oclc/1113410915</a> <a href="#fnref:54" class="footnote-back-ref">↩</a></li>
<li id="fn:55">Bostrom 2014, pp. 129–136. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:55" class="footnote-back-ref">↩</a></li>
<li id="fn:56">Bostrom 2014, pp. 136–143. - Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. <a href="#fnref:56" class="footnote-back-ref">↩</a></li>
<li id="fn:57">Wallach, Wendell; Allen, Colin (2008-11-19). Moral Machines: Teaching Robots Right from Wrong. Oxford University Press. ISBN 978-0-19-970596-2. <a href="978-0-19-970596-2" target="_blank">978-0-19-970596-2</a> <a href="#fnref:57" class="footnote-back-ref">↩</a></li>
<li id="fn:58">Dafoe, Allan (August 27, 2018). "AI Governance: A Research Agenda" (PDF). Center for the Governance of AI. <a href="https://cdn.governance.ai/GovAI-Research-Agenda.pdf" target="_blank">https://cdn.governance.ai/GovAI-Research-Agenda.pdf</a> <a href="#fnref:58" class="footnote-back-ref">↩</a></li>
<li id="fn:59">Yampolskiy, Roman V. (July 18, 2020). "On Controllability of Artificial Intelligence" (PDF). arXiv:2008.04071. <a href="https://philpapers.org/archive/YAMOCO.pdf" target="_blank">https://philpapers.org/archive/YAMOCO.pdf</a> <a href="#fnref:59" class="footnote-back-ref">↩</a></li>
<li id="fn:60">Brooks, Rodney (October 6, 2017). "The Seven Deadly Sins of AI Predictions". MIT Technology Review. Retrieved 2024-10-23. <a href="https://www.technologyreview.com/2017/10/06/241837/the-seven-deadly-sins-of-ai-predictions/" target="_blank">https://www.technologyreview.com/2017/10/06/241837/the-seven-deadly-sins-of-ai-predictions/</a> <a href="#fnref:60" class="footnote-back-ref">↩</a></li>
<li id="fn:61">Bryson, Joanna J (2019). "The Past Decade and Future of AI's Impact on Society". Towards a New Enlightenment? A Transcendent Decade. 11. ISBN 978-84-17141-21-9. <a href="978-84-17141-21-9" target="_blank">978-84-17141-21-9</a> <a href="#fnref:61" class="footnote-back-ref">↩</a></li>
<li id="fn:62">Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Yin Tat Lee; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Marco Tulio Ribeiro; Zhang, Yi (April 2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712 [cs.CL]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:62" class="footnote-back-ref">↩</a></li>
<li id="fn:63">Wei, Jason; Tay, Yi; Bommasani, Rishi; Raffel, Colin; Zoph, Barret; Borgeaud, Sebastian; Yogatama, Dani; Bosma, Maarten; Zhou, Denny; Metzler, Donald; Chi, Ed H.; Hashimoto, Tatsunori; Vinyals, Oriol; Liang, Percy; Dean, Jeff; Fedus, William (2022-06-26). "Emergent Abilities of Large Language Models". Transactions on Machine Learning Research. arXiv:2206.07682. ISSN 2835-8856. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:63" class="footnote-back-ref">↩</a></li>
<li id="fn:64">Ord, Toby (2020). The precipice: existential risk and the future of humanity. london New York (N.Y.): Bloomsbury academic. ISBN 978-1-5266-0023-3. <a href="978-1-5266-0023-3" target="_blank">978-1-5266-0023-3</a> <a href="#fnref:64" class="footnote-back-ref">↩</a></li>
<li id="fn:65">Ngo, Richard; Chan, Lawrence; Mindermann, Sören (2022). "The Alignment Problem from a Deep Learning Perspective". ICLR. arXiv:2209.00626. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:65" class="footnote-back-ref">↩</a></li>
<li id="fn:66">"How and Why Gary Marcus Became AI's Leading Critic > Marcus says generative AI like ChatGPT poses immediate dangers". IEEE Spectrum. 17 September 2024. <a href="https://spectrum.ieee.org/ai-inventions" target="_blank">https://spectrum.ieee.org/ai-inventions</a> <a href="#fnref:66" class="footnote-back-ref">↩</a></li>
<li id="fn:67">Dafoe, Allan (August 27, 2018). "AI Governance: A Research Agenda" (PDF). Center for the Governance of AI. <a href="https://cdn.governance.ai/GovAI-Research-Agenda.pdf" target="_blank">https://cdn.governance.ai/GovAI-Research-Agenda.pdf</a> <a href="#fnref:67" class="footnote-back-ref">↩</a></li>
</ol>

Superintelligence open-in-new

Superintelligence