Molly White makes a point that I’ve also been trying to articulate about AI crawling, but I think her post is better than my first attempts. Some level of open access is an important part of what makes the web special:

…“only allow humans to access” is just not an achievable goal. Any attempt at limiting bot access will inevitably allow some bots through and prevent some humans from accessing the site, and it’s about deciding where you want to set the cutoff.