Jeremy Keith has a good collection of links and quotes about AI crawling. On this specific part of the commentary I continue to disagree, though:

If you’re using the products powered by these attacks, you’re part of the problem. Don’t pretend it’s cute to ask ChatGPT for something. Don’t pretend it’s somehow being technologically open-minded to continuously search for nails to hit with the latest “AI” hammers.

I don’t think we should paint all AI tools with the same brush. Some tools might be well-behaved crawlers and some might not be. Note that this is a separate question from the legality of AI training. The context is mostly Wikipedia which is not under copyright.

Simon Willison adds about the Wikipedia data:

There’s really no excuse for crawling Wikipedia (“65% of our most expensive traffic comes from bots”) when they offer a comprehensive collection of bulk download options.

Ben Werdmuller also sees these as bad actors:

Here the issue is vendors being bad actors: creating an enormous amount of traffic for resource-strapped services without any of the benefits they might see from a real user’s financial support.

The argument I’m hearing from some folks is that because they consider AI to be bad, everything it touches must also be bad. All crawling, whether it respects robots.txt or not. All tools, because using them contributes to the success of LLMs.

I’d like to have more concrete answers, such as: do ChatGPT and Claude respect robots.txt or not? I assume they do, because they document their user agent strings. If they do, it doesn’t seem fair to punish ChatGPT because there is some other rogue AI crawler that is misbehaving.

AI is powerful and potentially dangerous. Because of this, most users will gravitate toward “brands” that are respected and accountable. In other words, users will prefer Apple Intelligence, ChatGPT, or Claude, where we know there has been some level of safety work, with only fringe users downloading and running models from other sources.

These mainstream AI tools should be contributing back. We know ChatGPT has a deal with Reddit, but they should also be making a recurring donation to Wikipedia. This would further differentiate the well-behaved bots from the ones skirting the edges of fairness.

Meta appears to have used their old move fast and break things playbook to training Llama, using pirated books. From The Atlantic:

Meta employees turned their attention to Library Genesis, or LibGen, one of the largest of the pirated libraries that circulate online. It currently contains more than 7.5 million books and 81 million research papers. Eventually, the team at Meta got permission from “MZ”—an apparent reference to Meta CEO Mark Zuckerberg—to download and use the data set.

Another thing that’s puzzling to me is if AI bots are so abusive, why haven’t I felt this in Micro.blog? We host blogs. If bots were destroying the open web, I would expect to notice it on my own servers.

If you dislike generative AI on philosophical grounds, or because of specific negative side effects such as energy use, that is totally fine. But then let’s stick to those arguments. I’m not yet convinced that legitimate AI crawling is going to destroy blogs or even Wikipedia.

David Brooks in The Atlantic starts by echoing what I quoted of his last month, then moves on to history lesson and how we eventually rebuild after Trump. A long, good read:

Humility, prudence, and honesty are not just nice virtues to have—they are practical tools that produce good outcomes. When you replace them with greed, lust, hypocrisy, and dishonesty, terrible things happen.

And:

When the time comes to build a new paradigm, progressives talk about economic redistribution; conservatives talk about cultural and civic repair. History shows that you need both: Recovery from national crisis demands comprehensive reinvention at all levels of society.

Tobias Lütke, in a memo to employees at Shopify, re-posted on Twitter / X:

Using Al effectively is now a fundamental expectation of everyone at Shopify. It’s a tool of all trades today, and will only grow in importance. Frankly, I don’t think it’s feasible to opt out of learning the skill of applying Al in your craft; you are welcome to try, but I want to be honest I cannot see this working out today, and definitely not tomorrow.

I just blogged a few days ago that I think generative AI gives programmers an advantage, but I wouldn’t mandate it, just like I wouldn’t mandate auto-complete or Stack Overflow. Programmers can be productive and happy in a variety of ways. Requiring AI (or forbidding it) strikes me as extremist for little benefit.

Marco Arment posted to Mastodon yesterday that he doesn’t use AI in Overcast beyond Xcode’s smarter auto-complete.

Maybe that’ll change in the future, but I see myself only ever using autocomplete-style speedups, not “write it for me”. I usually won’t even use code that other humans write!

Hard to argue with the results. Overcast is one of my favorite apps.

Also I know from my own experience that some programmers love the craft more than others. AI is a major disruption to the craft, so everyone will have different opinions on when it should be integrated into the process. It is an advancement, but how it’s used matters too.

This week’s ATP has a members-only special segment about vibe coding. One snippet from John Siracusa, encapsulating much of the conversation:

As programming tools have gotten better, the demand for programmers has gone up, not down.

When Steve Jobs returned to Apple, he gave a demo of NeXTSTEP at Macworld. I was lucky enough to be in the audience. Steve said that moving from traditional programming frameworks where you had to start with almost nothing, to Objective-C and what would become the Cocoa frameworks, was like starting a building on the 20th floor, with all the foundational pieces built for you.

A small team of only a couple developers could now build complete apps that before would’ve required a much larger team. Solo developers were of course already building apps on their own, but Steve was right that Cocoa would greatly increase productivity, opening up the Mac to more developers.

Vibe coding is different than a new system framework. With a framework, you mostly trust that it is well-tested and understood by the developers who wrote it. With AI slop, no human has tested the code before you literally just saw it unfold on screen. You have to understand it and tweak it before shipping a real product. Vibe coding instead is purposefully giving in to the machine and just letting code fly without any review.

Still, there’s a similarity to previous programming advances just because of the productivity gains. With AI, the step forward is more extreme than it has ever been.

Vibe coding will mostly prop up small projects and experiments, less likely to replace actual full-time development work. But if you’re programming without any assistance from AI, you are at a disadvantage, in the same way that programming in pure C with the Mac toolbox was much slower and error-prone for building a modern Mac app than using Objective-C when the Cocoa frameworks matured.

If after considering that productivity difference, a developer on principle wants to avoid AI, I respect that. Everyone should choose the programming language and frameworks that best suit their work. But there’s no sense in pretending that AI won’t change development, or that we won’t need to adapt along with it.