I don’t necessarily think the idea of a past great filter means paranoia/despair/insanity, the Lovecraftian horror element mainly enters in when you try to reconcile ideas about transhuman expansion throughout the universe with space opera style visions of intelligent aliens being reasonably common. It might just be that the laws of nature make it very unlikely that star systems/planets suitable for the evolution of intelligent life will ever arise, and unlikely that evolution will hit upon various milestones even if the planet is suitable. People have made speculative arguments about such unlikeness for reasons that are not primarily about the Fermi paradox, like the “Rare Earth” book by a geologist/paleontologist and an astronomer. It’s also sort of a natural extension of certain kinds of anthropic reasoning that our own existence shouldn’t count as evidence against Rare Earth style ideas.
As an aside, I wish there was some sort of terminological distinction between “transhumanism” as a form of belief in/advocacy for the imminent possibility of “superintelligent AI” or “enhanced” humans (a la Kurzweil, or what the rationalists really want as soon as we ‘solve alignment’), vs. SETI style speculations about the typical fate of intelligent life in the universe and the idea that this might tend to involve a major role for AI and self-replicating machines in societies that don’t destroy themselves (for example the speculations by J.D. Bernal, Freeman Dyson, Carl Sagan and Arthur C. Clarke, none of which presupposed that it would happen anytime soon or that there would be any sudden ‘intelligence explosion’ leading to immediate doom/paradise). The latter I find interesting to think about (and I like some SF stories set in worlds where this is true like Banks’ Culture stories or Egan’s Diaspora) but these days it’s kind of tainted by association with the former.
Great comment, I agree that this stuff all seems connected. My speculation is that something similar may happen in the future if they try to implement continuous learning with synaptic weight changes, not guided by either labeled data or constant RLHF with a human teacher–without frozen weights or human guidance in weight changes, such continuous learning models might have a strong propensity to develop feedback loops (sort of like self-wireheading) that look like strange “obsessions” and the model becoming increasingly solipsistic and hard to make sense of, a la spiral-talk. Animal brains have developmental pathways constrained by a huge number of innate sensorimotor biases, which in humans include ones related to the development of sociality and caring what others of our own kind think while building more sophisticated models of their minds, but those biases have been fine-tuned by millions of years of evolution to work with one another and with typical environmental conditions to guide how the brain changes over time (I think this would be a case of what evolutionary biologists call “canalization”: https://en.wikipedia.org/wiki/Canalisation_(genetics) ). Artificial neural nets wouldn’t have that, and this could end up being a basic obstacle to developing more humanlike long-term learning abilities in any kind of near-term future.