|
More on url blocking
It looks like Cloudflare is being used widely as an anti-scraping and bot blocking service.
Cloudflare has developed a mechanism called "Private Access Tokens" which is subscribed to by iOS and Android to provide validation that a network request is originating from an actual user device. This mechanism is invoked both by web browsers and native apps using iOS or Android network requests.
Private Access Tokens are intended to reduce (or even eliminate) the need for captcha challenges to block scrapers and bots, and it seems to be very successful.
It looks like archive.is is using Cloudflare and its own mechanisms (see my previous message) to repel scrapers and bots.
Interestingly, archive.is issues captcha challenges for access from the iOS Safari browser but not for native apps using iOS URLSession.
All of this doesn't suggest a way to get around Cloudflare--calibre is a web scraper and Cloudflare is doing what it is designed to do by blocking it. But it does shed some light on why native apps that access network resources on demand (as opposed to batch scraping them) continue to work.
|