This all becomes irrelevant long, long, long before singularity. As the famous quote goes, "any sufficiently advanced technology is indistinguishable from magic". Unless the fundamental limit on intelligence is very, very low, once ASI exceeds this threshold "M" we are no longer able to remain in the game and the only way to assess the level of an ASI's intelligence would be to ask it.
I would like to see the effects of evolution considered more strongly. Instead of a model training its successor in a sequential loop, I think it is much more interesting to consider a population of models that speciate into different lineages which diversify and recombine over time. A Cambrian explosion of diversity rather than a lone genius that trains the next model.
The evolutionary pressure might come from competition for compute time - smarter models who make better improvements survive and get to run more experiments. Weaker models who fail to improve will 'die out' as they do not make valuable use of the GPU time.
It still might not lead to a singularity but I think that a branching, evolving, multi-agent system may grow faster and saturate later than a singleton iterating on itself.
A single lineage might run into a ceiling or bottleneck, but a diverse population is likely to have some species with higher ceilings than others, and with multiplication and recombination, the frontier will keep rising even when most species saturate and 'die out'.
Jesus I’d never considered this as an approach to growing intelligences. It takes advantage of hard-to-predict-in-advance emergent phenomena, using evolutionary principles to improve upon the EV of reason alone.
Great article! Would be really interested in an even deeper analysis of likely RSI phases and various limitations.
I noticed a few missed subscripts and superscripts:
- "If every cycle of the feedback loop takes the same time, we’d have Tn = k and tn = kn." (tn should be t_n)
- "For example, if the generation time shrinks exponentially as Tn = 1/(2n), then t∞ = 1." (2n should be 2^n)
- "Tn = 1/n is too slow, while Tn = 1/(n1+ε) is fast enough." (Tn should be T_n)
(Subscripts and superscripts are flattened into normal characters in this comment.)
There is also a double dot in the table appendix at the end: "The rates of growth of A over time produced by different rates of return f(A) in the differential equation Ȧ˙ = f(A)."
It seems to me that if the time-embedded difference model is capturing the shape of the underlying dynamics, the universe's Planck time would imply a definite "No" on singular growth since the generation time could not shrink below the Planck time.
Of course this growth upper bound is extremely weak; it implies a maximum of ~50 OoMs of headroom. But it does seem like if binding, it could rule out truly "singular" growth definitively.
Why should we care?
This all becomes irrelevant long, long, long before singularity. As the famous quote goes, "any sufficiently advanced technology is indistinguishable from magic". Unless the fundamental limit on intelligence is very, very low, once ASI exceeds this threshold "M" we are no longer able to remain in the game and the only way to assess the level of an ASI's intelligence would be to ask it.
I would like to see the effects of evolution considered more strongly. Instead of a model training its successor in a sequential loop, I think it is much more interesting to consider a population of models that speciate into different lineages which diversify and recombine over time. A Cambrian explosion of diversity rather than a lone genius that trains the next model.
The evolutionary pressure might come from competition for compute time - smarter models who make better improvements survive and get to run more experiments. Weaker models who fail to improve will 'die out' as they do not make valuable use of the GPU time.
It still might not lead to a singularity but I think that a branching, evolving, multi-agent system may grow faster and saturate later than a singleton iterating on itself.
A single lineage might run into a ceiling or bottleneck, but a diverse population is likely to have some species with higher ceilings than others, and with multiplication and recombination, the frontier will keep rising even when most species saturate and 'die out'.
Jesus I’d never considered this as an approach to growing intelligences. It takes advantage of hard-to-predict-in-advance emergent phenomena, using evolutionary principles to improve upon the EV of reason alone.
Interpreting "the singularity" as a literal mathematical singularity imposes some strong theological assumptions...
Great article! Would be really interested in an even deeper analysis of likely RSI phases and various limitations.
I noticed a few missed subscripts and superscripts:
- "If every cycle of the feedback loop takes the same time, we’d have Tn = k and tn = kn." (tn should be t_n)
- "For example, if the generation time shrinks exponentially as Tn = 1/(2n), then t∞ = 1." (2n should be 2^n)
- "Tn = 1/n is too slow, while Tn = 1/(n1+ε) is fast enough." (Tn should be T_n)
(Subscripts and superscripts are flattened into normal characters in this comment.)
There is also a double dot in the table appendix at the end: "The rates of growth of A over time produced by different rates of return f(A) in the differential equation Ȧ˙ = f(A)."
Very insightful, thank you.
It seems to me that if the time-embedded difference model is capturing the shape of the underlying dynamics, the universe's Planck time would imply a definite "No" on singular growth since the generation time could not shrink below the Planck time.
Of course this growth upper bound is extremely weak; it implies a maximum of ~50 OoMs of headroom. But it does seem like if binding, it could rule out truly "singular" growth definitively.