Justice Thomas too forecasted that “technology may evolve to the point where it becomes impossible to enforce actual child pornography laws because the Government cannot prove that certain pornographic images are of real children.” In his view, “if technological advances thwart prosecution of ‘unlawful speech,’ the Government may well have a compelling interest in barring or otherwise regulating some narrow category of ‘lawful speech’ in order to enforce effectively laws against pornography made through the abuse of real children.”
This hints at another issue: most nudity is not pornographic. A model could, for example be trained on medical and scientific imagery, or just innocent family photos; it’s my impression photos containing nude young children were more acceptable in the pre-digital era. It’s certainly distasteful that AI companies are training on such material without explicit permission from the subjects, but that’s true of almost all AI training material.
I share your unwillingness to test how good image generation models are at making a realistic facsimile of CSAM. Legal or not, that’s yucky and I don’t want to look at it.