Intelligence, the ability to get a right answer on hard problems, is not a threat.
Treating AIs as having rights is a threat. Presuming governments have a right to regulate computation is a threat. Effective Altruism cultists acting on the belief that the interests of 10^100 hypothetical shrimp outweigh the interests of humanity is a threat.
This is your best essay yet on this topic, directly addressing several of my dissatisfactions with previous discussions. Thank you!
I wish you'd more consistently use conditional language: 'an intelligence explosion would...' rather than 'the intelligence explosion will...'.
What would you say if I said I still think there's a good chance that meaningful intelligence explosion (accelerating absent compute growth) is difficult or impossible *prior* to roughly-human sample-efficiency? (Maintaining research taste over a moving frontier requires sample-efficient taste accumulation, and that taste is a critical factor.) Perhaps you also think something like this yourself? But perhaps you think the jagged bonuses of AI mean parity is achievable *substantially* before this point?
It's believable to me that sample efficiency undergoes a meaningful jump at this (slightly indeterminate) stage. Still unclear if that's an *accelerating* SIE/singularity. Do you consider your and others' previous work on estimating the returns to R&D to be the key summary on this question?
I'm with you on learning algorithms generalising further than learned models. I possibly put more weight on bottlenecks to integration and in-domain data gathering than you. I remain unsure where this ends up cashing out in terms of a) hard power, b) social influence, and c) broad economic impact.
The "gathering real world data" bottleneck doesn't work because of the size of current AI revenues - The big AI companies will simply buy a company using AI for automation just to get its data - With revenues in the billions this is just common sense - there are no effective moats if your "enemy" can just buy the castle, and it doesn't have to be the biggest and best castle either.
There are quite aggressive measures to overcome some existing bottlenecks, happening right now. It is worth wondering if broader efforts to curtail them would withstand the current force of data/IP pillaging.
Intelligence, the ability to get a right answer on hard problems, is not a threat.
Treating AIs as having rights is a threat. Presuming governments have a right to regulate computation is a threat. Effective Altruism cultists acting on the belief that the interests of 10^100 hypothetical shrimp outweigh the interests of humanity is a threat.
This is your best essay yet on this topic, directly addressing several of my dissatisfactions with previous discussions. Thank you!
I wish you'd more consistently use conditional language: 'an intelligence explosion would...' rather than 'the intelligence explosion will...'.
What would you say if I said I still think there's a good chance that meaningful intelligence explosion (accelerating absent compute growth) is difficult or impossible *prior* to roughly-human sample-efficiency? (Maintaining research taste over a moving frontier requires sample-efficient taste accumulation, and that taste is a critical factor.) Perhaps you also think something like this yourself? But perhaps you think the jagged bonuses of AI mean parity is achievable *substantially* before this point?
It's believable to me that sample efficiency undergoes a meaningful jump at this (slightly indeterminate) stage. Still unclear if that's an *accelerating* SIE/singularity. Do you consider your and others' previous work on estimating the returns to R&D to be the key summary on this question?
I'm with you on learning algorithms generalising further than learned models. I possibly put more weight on bottlenecks to integration and in-domain data gathering than you. I remain unsure where this ends up cashing out in terms of a) hard power, b) social influence, and c) broad economic impact.
The "gathering real world data" bottleneck doesn't work because of the size of current AI revenues - The big AI companies will simply buy a company using AI for automation just to get its data - With revenues in the billions this is just common sense - there are no effective moats if your "enemy" can just buy the castle, and it doesn't have to be the biggest and best castle either.
There are quite aggressive measures to overcome some existing bottlenecks, happening right now. It is worth wondering if broader efforts to curtail them would withstand the current force of data/IP pillaging.