University libraries are being swarmed by AI bots, putting public access at risk

[ownedbycats]

https://www.cbc.ca/news/world/univer...bots-9.7408287

Eric Hellman, executive director of Project Gutenberg — a U.S.-based volunteer organization that digitizes and archives cultural works and distributes eBooks — wrote in a blog last year that these bots, which he characterizes as poorly programmed "monstrosities," are "destroying open access" on the internet.

He tells CBC News that "everything's gotten worse" since he wrote that.

Hellman says the problem comes down to good bots and bad bots.

He explains that search engines like Google, and archiving sites like Internet Archive, use bots that follow robots.txt — voluntary sets of instructions on websites that tell them what they can and can't do, so they don't disrupt the site's functioning while gathering information.

AI scraping bots, on the other hand, sometimes ignore these rules and overwhelm websites — in some cases requesting thousands of documents a second and downloading them in every available format. [...]

He says Project Gutenberg is happy for its materials to be used as AI training data and even offers a one-click download of its entire library of books, "but the bots are too stupid to use it."

[Gormagon]

The bots just do what some coders told them to do...or if LLMs were used the LLMs produced rubbish when bots aren't following the restrictions of the robots.txt. There's also the possibilty that the bots were designed the way they act, i.e., not to obey any restrictions on websites.