If moral realism is true, should we still expect that the blackmailing scenarios are bad in expectation? If so, this seems like it assumes people will be badly-calibrated with respect to their confidence of their moral views; but wouldn't poor-calibration fade given intelligence increases?
Also, at the end when you mention "expected value of 1%", what is this in reference to (i.e. what would be 100%)?
I do think blackmail can remain an issue into the future. We discuss in the full piece why, even if moral realism is true, many or most people might still not act in line with the right kind of moral views (in sections 2.3 and 2.4.1).
I wonder about the assumption that our post-AGI Earth will host one unitary AGI. Why won’t there be multiple AGIs with different characteristics and goals? If there are, clashes over values will be much more complex than is considered here
Hmm, I certainly agree that we shouldn't be assuming ‘one unitary AI’! Can you point to where we (implicitly) made that assumption, or say how things would be more complex than we've suggested? It's possible we've overlooked important considerations here, but we weren't specifically imagining a single AI when writing this.
I re-read and think that I read the entire piece as analyzing what might happen in "the post-AGI situation" with that "situation" being singular, a reality that would be much the same for everyone across Earth.
Obviously that will not be true, just as "the post-wheel" or "post-writing" or "post-gun powder" situation was not the same for everyone across the world. Diffusion, cultural differences, etc.
But I think there is point to be made that a post-AGI Earth will be more complex than your piece suggests as different AGI systems will be offering different advice, knowledge, tricks, advances, etc. They will not converge or agree on very much, I think.
For example, multiple general intelligences will not likely converge on some common artistic, ethical, political, or economic advice--there seem not to be one correct art, ethics, politics, or economics. These are not domains that simply must be better, more rationally figured out. Even if systems are given common goals, they could interpret those goals differently, obviously, and even if they interpret them "the same" (whatever that means, viz. Wittgenstein), they will likely offer different action paths to achieve those goals.
So a post-AGI situation could see amplified differences in worldviews, with many factions working with their own AGI to grow their own views. We may need a United Nations for AGIs to meet and work out agreements and treaties to dial down their conflicts and keep them from becoming hot.
I appreciate your pressing me on my comment; I got to figure out what I trying to figure out.
Ok, thanks. Note that we give explicit reasons in the piece for expecting that AI advice won't cause everyone to converge to the same moral views, even in the long-run. So in that respect I'm not sure what the disagreement is.
I also agree the world after much AI progress will, like the world today, will include a jumble of different beliefs, views, material circumstances etc. Again I'm not sure where we assume or imply it must be very (or relatively much more) homegenous in those respects; that is indeed something you need to argue for.
Thanks for your patience—I think I projected rather than read carefully enough. Your claims for heterogeneity are there—and very important to counter the assumption of increasing homogeneity I incorrectly projected onto you and that is widely found other places. Mea culpa.
If moral realism is true, should we still expect that the blackmailing scenarios are bad in expectation? If so, this seems like it assumes people will be badly-calibrated with respect to their confidence of their moral views; but wouldn't poor-calibration fade given intelligence increases?
Also, at the end when you mention "expected value of 1%", what is this in reference to (i.e. what would be 100%)?
I do think blackmail can remain an issue into the future. We discuss in the full piece why, even if moral realism is true, many or most people might still not act in line with the right kind of moral views (in sections 2.3 and 2.4.1).
“Expected value of 1%” is as a fraction of the value of a ‘best feasible future’ — sorry that was unclear. We give more detail in section 3.1 here: https://www.forethought.org/research/no-easy-eutopia#31-valuing-the-future
I wonder about the assumption that our post-AGI Earth will host one unitary AGI. Why won’t there be multiple AGIs with different characteristics and goals? If there are, clashes over values will be much more complex than is considered here
Hmm, I certainly agree that we shouldn't be assuming ‘one unitary AI’! Can you point to where we (implicitly) made that assumption, or say how things would be more complex than we've suggested? It's possible we've overlooked important considerations here, but we weren't specifically imagining a single AI when writing this.
I re-read and think that I read the entire piece as analyzing what might happen in "the post-AGI situation" with that "situation" being singular, a reality that would be much the same for everyone across Earth.
Obviously that will not be true, just as "the post-wheel" or "post-writing" or "post-gun powder" situation was not the same for everyone across the world. Diffusion, cultural differences, etc.
But I think there is point to be made that a post-AGI Earth will be more complex than your piece suggests as different AGI systems will be offering different advice, knowledge, tricks, advances, etc. They will not converge or agree on very much, I think.
For example, multiple general intelligences will not likely converge on some common artistic, ethical, political, or economic advice--there seem not to be one correct art, ethics, politics, or economics. These are not domains that simply must be better, more rationally figured out. Even if systems are given common goals, they could interpret those goals differently, obviously, and even if they interpret them "the same" (whatever that means, viz. Wittgenstein), they will likely offer different action paths to achieve those goals.
So a post-AGI situation could see amplified differences in worldviews, with many factions working with their own AGI to grow their own views. We may need a United Nations for AGIs to meet and work out agreements and treaties to dial down their conflicts and keep them from becoming hot.
I appreciate your pressing me on my comment; I got to figure out what I trying to figure out.
Ok, thanks. Note that we give explicit reasons in the piece for expecting that AI advice won't cause everyone to converge to the same moral views, even in the long-run. So in that respect I'm not sure what the disagreement is.
I also agree the world after much AI progress will, like the world today, will include a jumble of different beliefs, views, material circumstances etc. Again I'm not sure where we assume or imply it must be very (or relatively much more) homegenous in those respects; that is indeed something you need to argue for.
Thanks for your patience—I think I projected rather than read carefully enough. Your claims for heterogeneity are there—and very important to counter the assumption of increasing homogeneity I incorrectly projected onto you and that is widely found other places. Mea culpa.