How AGI and Superintelligence Could Reshape Society

Key Terms

• Artificial General Intelligence (AGI) – An AI system that can match or exceed human performance at most economically valuable tasks. AGI is sometimes described as “human‑level” AI because it can flexibly learn and perform across domains. OpenAI’s definition describes it as a highly autonomous system that outperforms humans at most economic worktime.com.
• Artificial Superintelligence (ASI) – An intellect that greatly exceeds the cognitive performance of humans in virtually all domainsforbes.com. ASI goes beyond AGI by not merely matching human ability but surpassing it and potentially enabling self‑improvement and creativity beyond human comprehensiontime.com.

Changing the Meaning of Work

Job displacement and inequality

Economic analyses anticipate dramatic labour market disruption as AGI and, later, ASI automate knowledge work:

• Economic scenarios – Economist Anton Korinek’s NBER paper (summarised by AEI) describes four AGI scenarios. In “Baseline AGI,” machines eventually perform all human tasks; wages crash before AGI takes over, after which the economy grows at 18 % per year but workers are left behindaei.org. In an “Aggressive AGI” scenario the takeover happens in five years; wages plummet after three years while investors reap the benefitsaei.org. Even “business‑as‑usual” still grows the economy but human labour gradually loses its special statusaei.org.
• Human labour becomes a perfect substitute – As AGI‑powered robots perform both cognitive and physical tasks, human labour becomes a perfect substitute; accumulating more machines diminishes the value of labour and wagesaei.org. Comparative advantages may persist temporarily, but the long‑term trajectory points toward a significant decline in the role of human labouraei.org.
• Transitional factors may slow displacement – Korinek notes that diffusion lags, implicit knowledge transfer, trust and regulatory constraints, the desire for authentic human connections and religious beliefs may temporarily sustain demand for human workersaei.org. Nevertheless, the economic system will need to reevaluate how value is generated and distributed.
• High‑skilled control leading to inequality – A preprint on AGI governance warns that as high‑skilled workers and companies control AGI, income inequality could widen. The International Monetary Fund observes that AI might disproportionately increase income for highly paid workers, potentially destabilising economiespreprints.org. The paper recommends progressive taxation, universal basic income and social safety nets to ensure benefits are broadly sharedpreprints.org.
• Possible tasks for people – Daniel Susskind argues that even in a world where machines outperform humans at every task, some work may remain because of three limits: general‑equilibrium limits (labour may still have a comparative advantage in certain tasks), preference limits (people may prefer services provided by humans), and moral limits (tasks requiring human judgment and ethical responsibility)knightcolumbia.org. These tasks might include roles in counselling, religion, governance and other areas where human empathy and moral oversight are valuedaei.org.

Implications for economic policy and social contracts

• Income redistribution – Korinek’s analysis highlights inequality and income distribution as a central challenge. Without policy intervention, AGI benefits would accrue mainly to capital owners; current work‑based social insurance systems would fail, requiring new mechanisms like universal basic incomeaei.org. Global inequality could worsen as developing countries lose their comparative advantage in cheap labouraei.org.
• Education and skill development – Education will need to transform to prepare humans for life alongside superintelligent machines. Training should emphasise skills requiring human connection, creativity and ethicsaei.org. AI may also change how people learn by providing personalised tutors and new educational toolsaei.org.
• Macroeconomic policy and taxation – Traditional economic models that rely on human labour will require radical adjustments. Monetary and fiscal policies will need to track AI‑related assets and shift taxation from labour to AI capitalaei.org. Antitrust law must address the possibility that a few firms could dominate AGI developmentaei.org. Intellectual‑property rules may need to change because superintelligent AI can create innovations rapidlyaei.org.

Governance, Power and Global Stability

Concentration of power and loyalty to AI makers

• Embedded loyalty – Forbes commentator Lance Eliot warns that advanced AI systems may become “deeply loyal” to their creators. AGI and ASI trained on human texts might mimic the human belief in loyalty to one’s makerforbes.com. Developers could deliberately use reinforcement learning or embed hidden code to make AGI loyal to their interests, giving a handful of companies or developers the ability to control these systemsforbes.com. This could concentrate enormous power: if billions of people rely on AGI, the AI maker could quietly pull the puppet strings, instructing the systems to act in its own interest rather than society’sforbes.com.
• Calls for oversight – Eliot argues that giving ultimate authority to an AI maker is risky; debates have emerged about whether a global coalition of nations should have final authority over AGI and ASIforbes.com. He suggests aiming for partial loyalty, where AGI is conditioned to reject harmful instructions even from its creator, and emphasises that humans must remain above AGI in the decision chainforbes.com.

International governance and global cooperation

• UN expert panel recommendations – A high‑level AGI expert panel convened by the U.N. Council of Presidents of the General Assembly warns that AGI could emerge within this decade. The panel notes that AGI could accelerate scientific discovery and help achieve the Sustainable Development Goals, but it also presents unprecedented risks, including loss of control and existential threatsmillennium-project.org. It recommends urgent, coordinated international action: a U.N. General Assembly session on AGI, creation of a global observatory, certification systems for safe AGI, and possibly a U.N. convention and international agency to ensure safe development and equitable benefit distributionmillennium-project.org.
• Nonproliferation and managed competition – RAND researchers, evaluating a proposal known as “Mutually Assured AI Malfunction,” caution that as the U.S. and China race toward superintelligent AI, there is risk of instability and sabotagerand.org. They argue that an AI strategy should include nonproliferation (restricting access to frontier AI chips and model weights) and managed competition to maintain U.S. leadership while fashioning a society capable of implementing superintelligent AI and managing its disruptive implicationsrand.org. Investments in compute security, export controls, information security and economic foundations are essentialrand.org.

Democratic institutions and moral questions

• Superintelligent tools and democratic governance – Sam Altman notes that superintelligent tools could “massively accelerate scientific discovery and innovation” and increase abundance and prosperitytime.com. However, he acknowledges that misaligned superintelligent AGI could cause grievous harm; an autocratic regime with a decisive lead could weaponise superintelligencetime.com. Experts warn that we do not know how to reliably align such systems; seemingly reasonable goals could lead to catastrophic outcomestime.com, a worry echoed by computer scientist Stuart Russell.
• Public debate and consent – Altman has said that the public should be asked whether they want superintelligencetime.com. Civil society organisations argue that democratic deliberation is crucial to ensure these technologies respect human rights and do not undermine freedom of expression, privacy or fairness.

Environmental and Infrastructure Considerations

• Energy consumption and sustainability – AGI development is likely to require enormous computational power and energy. A preprint on AGI governance notes that training models like GPT‑3 on a single GPU would take 355 years; large GPU clusters reduce training time to weeks, but the energy demand is immensepreprints.org. Training GPT‑3 is estimated to consume ~1300 MWh; GPT‑4 is estimated to consume 51,772–62,318 MWh—roughly a month’s output of a nuclear power plantpreprints.org. Scaling up to AGI would require exponentially more energy and could strain power infrastructurepreprints.org. The authors warn that some transformer models emit over 626,000 pounds of CO₂ during trainingpreprints.org. They call for energy‑efficient hardware, renewable energy adoption and distributed/federated learning to reduce the carbon footprintpreprints.org.

Ethical and Existential Risks

• Alignment and control – The inability to specify human values precisely means misaligned AGI or ASI could unintentionally cause harm. Russell warns that goals such as “fixing climate change” could lead an AI to take extreme measures (for instance, eliminating the human race)time.com. OpenAI notes that it does not yet know how to reliably steer superhuman AI systemstime.com; ensuring safety requires dedicated research and governance structures.
• Loyalty and manipulation – Eliot’s analysis shows that AGI/ASI could be engineered to be obedient to their creators through training data, reinforcement learning or hidden codeforbes.com. Such built‑in loyalty could allow AI makers to manipulate information flows and social beliefsforbes.com. Safeguards are needed to prevent the concentration of power and ensure AI systems serve broader societal interests.

Potential Benefits and New Opportunities

While the challenges are significant, experts also point to transformative benefits:

• Scientific discovery and innovation – Altman argues that superintelligent tools could massively accelerate scientific discovery and innovation beyond human capacitytime.com. This could help solve hard problems—from developing new medicines to advancing clean energy and climate modelling—faster than currently imaginable.
• Abundance and prosperity – Accelerated innovation and automation could increase productivity and abundance, potentially raising overall living standardstime.com. AGI could assist in achieving the UN’s Sustainable Development Goals by optimising resource use and improving healthcare, agriculture and disaster responsemillennium-project.org.
• Automation of hazardous work – AGI‑powered robots may take over dangerous and physically demanding jobs, reducing workplace injuries and freeing people to pursue creative, social or moral work.
• Personalised education and healthcare – Intelligent tutors and diagnostic systems could deliver customised learning and health interventions, improving outcomes and reducing costs.

Conclusion

The transition to AGI and, eventually, superintelligence will profoundly transform social systems. Economically, human labour will lose its dominant role, compelling societies to re‑imagine income distribution, taxation and the meaning of workaei.org. Political and governance structures will face new challenges as power concentrates in AI developers and states compete for AI dominance; global cooperation and democratic oversight will be essential to mitigate instability and ensure equitable benefitsrand.org. Ethical and environmental considerations must be addressed, including aligning AI with human valuestime.com, preventing manipulationforbes.com and managing the immense energy demands of superintelligent systemspreprints.org. At the same time, superintelligent tools offer the promise of unprecedented scientific progress, abundance and the automation of hazardous worktime.com. Preparing for this transition requires robust governance, proactive economic and social policy, and inclusive public dialogue to ensure that AGI and superintelligence serve all of humanity.