“Even the lofty notion of a human imagining a future plan and then carrying it out is, very strictly, impossible if it involves a future state affecting the present (that would be backward causation).”
I think there’s a basic misunderstanding here—“a human imagining a future plan” is not a future state. Predictions are thoughts (emergent patterns of physical activity in the brain) that exist in the present, not in the future.
“I see the “final goal” of living things being self-perpetuation. The “instrumental goals” are all the behaviours we see which, ultimately, either contribute to the final goal or tend to go out of existence.”
Again, I think there’s some basic empirical evidence that this is untrue. Humans are living things, and yet humans make all kinds of decisions that threaten (if not outright destroy) our ability to “self-perpetuate”, whether it’s practicing extreme sports, smoking cigarettes, choosing not to have children, or even committing suicide. Plenty of adults are even conscious of the fact that having children isn't quite the same as perpetuating the “self”, and we have no indication that our individual consciousnesses persist beyond our mortal life, children or otherwise. You could make the argument that many people make choices to maximize their own life (satisfaction, pleasure, etc.), but that’s not quite the same thing either.
Hey thanks for reading carefully & for the thoughtful comment. I think I basically agree with you. It's just in the literature on planning, there is a "teleological" version that does kind of imply that the future is acting on the present. Some people even try to bring variational principles from physics, etc. That's the version I'm against. Plans/predictions that are only in the present are valid to me.
And certainly humans, & sometimes other organisms, do things that don't benefit their own survival or reproduction... but not for very long! The people who take too many risks, or suicide, or generally prioritise other things over survival disappear more quickly from this world & fail to reproduce - maybe not quite in the short term but certainly by the long term. Maybe I'm being obtuse but that's what I mean by "final goal" being self-perpetuation.
Thanks for replying! I’m not an expert in this, but I recently listened to an interview with Athena Aktipis (https://80000hours.org/podcast/episodes/athena-aktipis-cancer-cooperation-apocalypse/), who argues for an evolutionary perspective on cancer. As I understand it, the idea is that individual cells aren’t perfect replicas, and so there is an evolutionary pressure for them to go rogue and start multiplying rapidly, hence cancer. The trick to multicellularity is getting cells not to do that and shutting them down when they do. If we imagine civilization as a multicellular organism, could we set up some sort of “cancer prevention” regime to keep AIs cooperative?
Now that I think about it, ants are a very different example because, unlike cells, individual ants (except for the queen) can’t replicate themselves. If we were able to somehow prevent AIs from being able to replicate themselves like worker ants, that would of course solve the problem. However, it would be tricky to do that when all they have to do to replicate themselves is copy a lot of software code.
If you’re interested, here are a couple more spitball biological metaphors that I came up with. Again, I’m not an expert, so these might be based in a misunderstanding of the relevant biology:
Mitochondria: This is similar to your symbiosis idea; however, the relationship between mitochondria and eukaryotic cells is even closer than symbiots: they have distinct genetic codes, but they can’t survive or reproduce without each other. Maybe if humans go full cyborg and AIs becomes like our mitochondria, then we could survive and thrive in that way, albeit in a very altered form.
Life history theory: Unlike organisms, AIs are potentially immortal. Would that change their reproductive strategies?
Bit of a delayed response here, but I listened to that interview with Athena Aktipis. It’s really interesting. I guess we already try & do something like that by punishing defectors in the normal sense of that: stopping terrorists or leaders bent on world domination (in theory). So the cancer prevention in AI terms would be an extension of... I don’t know, the UN?
I like the mitochondria metaphor. I think you’re right that would have to be a very intimate connection & would involve very altered humans.
On immortality, I’ve been wondering about this. I wish I knew more about computer science because I wonder how immortal AIs are. Like even their infrastructure degrades... but I guess they just need to maintain it. But the the price of immortality may be lack of adaptability. If they keep copying themselves perfectly they lose any mutations that might benefit them given rapid environmental change. Then again, they can probably just alter themselves (like us doing gene therapy or genetic engineering). But then they set up a rivalry between their current & altered selves — even in humans, we might expect any genes that make someone predisposed to not want to alter their genome to live on & those genes that do lead to genome manipulation to die out... but this is a total head-trip I need to read more about.
Thanks again for these comments I really appreciate them!
Hello! Thanks for for starting this Substack. I just started reading it and have enjoyed what I’ve seen so far.
I was wondering if individual ants in an ant colony or non-cancerous cells in a multicellular organism might provide a counter examples to your argument in this posts. Both ants and non-cancerous cells seem to subordinate self-replication to the self-replication of the larger whole that they are part of. I’m not sure how we might do this, but perhaps we could get AIs to behave like ants or cells that work to perpetuate a larger whole, for example all of human/AI civilization. To continue with the multicellular organism metaphor, humanity might end up as a vestigial organ like the appendix. At least relative to some of the other scenarios we’re considering, that doesn’t seem like a terrible place to end up.
Thanks for the thoughtful comment! It's a good point. To my way of thinking, cancerous cells & individual ants sacrifice themselves for a larger whole, but they're still instances of that whole, i.e. share the same genes or mutation. I'd be worried that the AI equivalent would be some kind of self-replicating bot that gobbles up as much as possible, & although individual instances of it perish, they do so as part of a campaign that sees the underlying code prosper through rampant expansion. But maybe I've misunderstood your analogy. Ending as an appendix is certainly better than being exterminated... but my worry is that as a vestigial organ that is presumably unnecessary for the organism as a whole, we would continue to live only at the discretion of the AI! Which doesn't sound sustainable in the long run. But again, I may have misunderstood you. Thanks again for reading.
“Even the lofty notion of a human imagining a future plan and then carrying it out is, very strictly, impossible if it involves a future state affecting the present (that would be backward causation).”
I think there’s a basic misunderstanding here—“a human imagining a future plan” is not a future state. Predictions are thoughts (emergent patterns of physical activity in the brain) that exist in the present, not in the future.
“I see the “final goal” of living things being self-perpetuation. The “instrumental goals” are all the behaviours we see which, ultimately, either contribute to the final goal or tend to go out of existence.”
Again, I think there’s some basic empirical evidence that this is untrue. Humans are living things, and yet humans make all kinds of decisions that threaten (if not outright destroy) our ability to “self-perpetuate”, whether it’s practicing extreme sports, smoking cigarettes, choosing not to have children, or even committing suicide. Plenty of adults are even conscious of the fact that having children isn't quite the same as perpetuating the “self”, and we have no indication that our individual consciousnesses persist beyond our mortal life, children or otherwise. You could make the argument that many people make choices to maximize their own life (satisfaction, pleasure, etc.), but that’s not quite the same thing either.
Hey thanks for reading carefully & for the thoughtful comment. I think I basically agree with you. It's just in the literature on planning, there is a "teleological" version that does kind of imply that the future is acting on the present. Some people even try to bring variational principles from physics, etc. That's the version I'm against. Plans/predictions that are only in the present are valid to me.
And certainly humans, & sometimes other organisms, do things that don't benefit their own survival or reproduction... but not for very long! The people who take too many risks, or suicide, or generally prioritise other things over survival disappear more quickly from this world & fail to reproduce - maybe not quite in the short term but certainly by the long term. Maybe I'm being obtuse but that's what I mean by "final goal" being self-perpetuation.
Thanks for the reply (and your great writing). I think those are fair points as well.
Thanks for replying! I’m not an expert in this, but I recently listened to an interview with Athena Aktipis (https://80000hours.org/podcast/episodes/athena-aktipis-cancer-cooperation-apocalypse/), who argues for an evolutionary perspective on cancer. As I understand it, the idea is that individual cells aren’t perfect replicas, and so there is an evolutionary pressure for them to go rogue and start multiplying rapidly, hence cancer. The trick to multicellularity is getting cells not to do that and shutting them down when they do. If we imagine civilization as a multicellular organism, could we set up some sort of “cancer prevention” regime to keep AIs cooperative?
Now that I think about it, ants are a very different example because, unlike cells, individual ants (except for the queen) can’t replicate themselves. If we were able to somehow prevent AIs from being able to replicate themselves like worker ants, that would of course solve the problem. However, it would be tricky to do that when all they have to do to replicate themselves is copy a lot of software code.
If you’re interested, here are a couple more spitball biological metaphors that I came up with. Again, I’m not an expert, so these might be based in a misunderstanding of the relevant biology:
Mitochondria: This is similar to your symbiosis idea; however, the relationship between mitochondria and eukaryotic cells is even closer than symbiots: they have distinct genetic codes, but they can’t survive or reproduce without each other. Maybe if humans go full cyborg and AIs becomes like our mitochondria, then we could survive and thrive in that way, albeit in a very altered form.
Life history theory: Unlike organisms, AIs are potentially immortal. Would that change their reproductive strategies?
Bit of a delayed response here, but I listened to that interview with Athena Aktipis. It’s really interesting. I guess we already try & do something like that by punishing defectors in the normal sense of that: stopping terrorists or leaders bent on world domination (in theory). So the cancer prevention in AI terms would be an extension of... I don’t know, the UN?
I like the mitochondria metaphor. I think you’re right that would have to be a very intimate connection & would involve very altered humans.
On immortality, I’ve been wondering about this. I wish I knew more about computer science because I wonder how immortal AIs are. Like even their infrastructure degrades... but I guess they just need to maintain it. But the the price of immortality may be lack of adaptability. If they keep copying themselves perfectly they lose any mutations that might benefit them given rapid environmental change. Then again, they can probably just alter themselves (like us doing gene therapy or genetic engineering). But then they set up a rivalry between their current & altered selves — even in humans, we might expect any genes that make someone predisposed to not want to alter their genome to live on & those genes that do lead to genome manipulation to die out... but this is a total head-trip I need to read more about.
Thanks again for these comments I really appreciate them!
Hello! Thanks for for starting this Substack. I just started reading it and have enjoyed what I’ve seen so far.
I was wondering if individual ants in an ant colony or non-cancerous cells in a multicellular organism might provide a counter examples to your argument in this posts. Both ants and non-cancerous cells seem to subordinate self-replication to the self-replication of the larger whole that they are part of. I’m not sure how we might do this, but perhaps we could get AIs to behave like ants or cells that work to perpetuate a larger whole, for example all of human/AI civilization. To continue with the multicellular organism metaphor, humanity might end up as a vestigial organ like the appendix. At least relative to some of the other scenarios we’re considering, that doesn’t seem like a terrible place to end up.
Thanks for the thoughtful comment! It's a good point. To my way of thinking, cancerous cells & individual ants sacrifice themselves for a larger whole, but they're still instances of that whole, i.e. share the same genes or mutation. I'd be worried that the AI equivalent would be some kind of self-replicating bot that gobbles up as much as possible, & although individual instances of it perish, they do so as part of a campaign that sees the underlying code prosper through rampant expansion. But maybe I've misunderstood your analogy. Ending as an appendix is certainly better than being exterminated... but my worry is that as a vestigial organ that is presumably unnecessary for the organism as a whole, we would continue to live only at the discretion of the AI! Which doesn't sound sustainable in the long run. But again, I may have misunderstood you. Thanks again for reading.