|
The ethical dubiousness comes from the fact that if everyone starts spoofing search engine crawlers to bypass paywalls, then the maintainers of those paywalls will have to close that loophole, which means their content is no longer generally indexable, which is a net loss for everyone.
And it is fragile because replacing it by a secured protocol is trivial, for example using TLS client verification. All google would have to do is turn it on in their crawler and publish their public certificate. And this in turn would make it much harder for a google competitor to ever emerge, since now every paywall would have to actively add a new public certificate to its configuration to allow access to a new crawler.
|