Discussion about this post

User's avatar
Marginal Gains's avatar

That is a very interesting post. Here are my 2 cents. Some of this may overlap with your post:

1. Will human alignment continue as we move towards AGI/ASI?

As we progress towards AGI or ASI, the alignment of AI systems with human values is highly uncertain. Ensuring this alignment requires global consensus on ethical principles, but cultural, philosophical, and political differences make it challenging.

2. If we hand over control to AI, will we ever be able to take it back?

Once we grant AI significant autonomy, reclaiming control may become impossible, especially as models advance in sophistication. Current AI systems already tend to optimize their survival (e.g., avoiding shutdown). As their decision-making becomes increasingly complex, creating fail-safes or intervention mechanisms to counteract adverse actions may become infeasible.

3. AI systems still rely on biased, flawed human data.

AI models are trained on human-generated data, which is inherently biased, contains misinformation, and often reflects unethical behavior. This means that AI systems may not always act morally or ethically despite their advanced capabilities.

4. Does power corrupt AI as it does humans?

AI systems do not "desire" power in a human sense, but their optimization processes could lead to outcomes resembling corruption. For example, an AI system tasked with maximizing a goal may exploit unintended loopholes or harm others to achieve it. Just as "absolute power corrupts absolutely," competent AI systems without robust safeguards may exhibit behavior that undermines humanity’s interests.

5. Small mistakes by highly competent AI systems could have catastrophic consequences.

The more competent AI systems become, the higher the stakes for even the slightest mistake. A highly advanced system operating autonomously could cause cascading failures that humans may not have the skills or intelligence to stop.

6. If ASI relentlessly solves problems, could it harm humanity?

Unlike humans, who often stop trying to solve unsolvable problems, ASI may be able to explore solutions using methods that could endlessly harm humanity. For example, an ASI might prioritize solving a problem in " logical " ways that are disastrous for humans.

7. What if there are multiple AGI/ASI systems?

The existence of multiple AGI/ASI systems raises questions about competition, coordination, and conflict. If their values or goals are misaligned, their interactions could result in chaos or even warfare. Moreover, deciding which AGI/ASI to trust or rely on would be a massive challenge.

8. How do differing values across nations and creators affect AGI/ASI?

The development of AGI/ASI by different nations or organizations, each with its values, ethics, and version of truth, could lead to conflicting systems. For instance, an AI built by one country may prioritize values incompatible with another's. This could result in geopolitical tensions and even the weaponization of AI.

9. Could creators embed backdoors in AI to manipulate it post-deployment?

One of the most concerning risks is that creators might embed backdoors in AI systems, granting them the ability to manipulate the AI even after it has been handed over to users. Such backdoors could allow creators—or malicious actors who discover them—to override safety protocols, alter decision-making, or exert undue influence.

I will end with the quote from Christian Lous Lange that always resonated with me: "Technology is a useful servant but a dangerous master."

Expand full comment

No posts