Quote:
Originally Posted by bobbysteel
Same here. It seems to be a case of the page being denied on the server from viewing via paywall.
|
If so, there is not much we can do about it, since the NYT requires a captcha to login, so we cannot log in in the recipe. You could try using delay = 1 which might avoid it (though it will make downloads very slow). Or if you want to get more sophisticated you can detect the paywall markup in preprocess_raw_html() and re-request the article.