New/Updated: WSJ Wall Street Journal recipe

[TechnoCat]

Attached is a modified Wall Street Journal recipe. This retrieves the article text without requiring an WSJ account. (That probably means it replaces wsj_free.recipe on github.)

[kovidgoyal]

Bypassing the paywall by pretending to be the googlebot is IMO both fragile and ethically dubious.

So I think I will not update the builtin recipe with those changes, you are of course welcome to keep using your customized recipes.

[TechnoCat]

It's your product, so it's your call, but I don't agree with either statement. It's not fragile; it would take them a while to negotiate and implement a new behavior in common with Google. And I don't consider it an ethical connundrum either, given that it's all documented and is merely setting a User-agent.

But, as I said, it's your product and your call.

[kovidgoyal]

The ethical dubiousness comes from the fact that if everyone starts spoofing search engine crawlers to bypass paywalls, then the maintainers of those paywalls will have to close that loophole, which means their content is no longer generally indexable, which is a net loss for everyone.

And it is fragile because replacing it by a secured protocol is trivial, for example using TLS client verification. All google would have to do is turn it on in their crawler and publish their public certificate. And this in turn would make it much harder for a google competitor to ever emerge, since now every paywall would have to actively add a new public certificate to its configuration to allow access to a new crawler.

[NSILMike]

Originally Posted by kovidgoyal

The ethical dubiousness comes from the fact that if everyone starts spoofing search engine crawlers to bypass paywalls, then the maintainers of those paywalls will have to close that loophole, which means their content is no longer generally indexable, which is a net loss for everyone.

And it is fragile because replacing it by a secured protocol is trivial, for example using TLS client verification. All google would have to do is turn it on in their crawler and publish their public certificate. And this in turn would make it much harder for a google competitor to ever emerge, since now every paywall would have to actively add a new public certificate to its configuration to allow access to a new crawler.

I agree, completely. Also, though, there is easily researchable evidence that Paywalls are the wrong strategy for many sites. For example, the NY Times was highly dependent upon views for setting advertising rates for the online edition. When they added the paywall, their views, and therefore their advertising rates fell precipitously. They came close to bankruptcy until they loosened their requirements...