|
The bots just do what some coders told them to do...or if LLMs were used the LLMs produced rubbish when bots aren't following the restrictions of the robots.txt. There's also the possibilty that the bots were designed the way they act, i.e., not to obey any restrictions on websites.
|