The Spectator - only title and synopsis

[nano5]

The last two "The Spectator" only fetch title and synopsis, since October. The article body content is missing.

[unkn0wn]

you can use the attached recipe, it will load all articles but is still a temporary solution (Might fail due to too many requests).

time for someone to figure out and add login code to the recipe.

[unkn0wn]

@kovidgoyal how can I make use of wayback machine? is it nytimes exclusive?

[kovidgoyal]

yes i would need to add support for spectator to it. what is the url scheme for spectator? if it has a decent url scheme I might be able to do it.

[unkn0wn]

https://web.archive.org/web/20231013...support-hamas/
looks like wayback machine doesn't have access to these articles.

https://archive.today/ works but has different url and captcha checks. Can we do something for archive.today?
https://archive.ph/K6f5r

[kovidgoyal]

the archive.org entries are paywalled as well, so no point there. As for archive.today no idea never used it.

[kovidgoyal]

I took a brief look at archive.is changing the recipe to use it should be as simple as replacing the article urls with urls of the form

https://archive.is/latest/original_url

I dont know what their rate limiting and captcha policies are that will require experimentation.

[unkn0wn]

although it loads content in browser.. theres no response in calibre for these urls

Traceback (most recent call last):
  File "mechanize\_urllib2_fork.py", line 1238, in do_open
  File "http\client.py", line 1374, in getresponse
  File "http\client.py", line 318, in begin
  File "http\client.py", line 287, in _read_status
http.client.RemoteDisconnected: Remote end closed connection without response

I tried using print_version from 'https://archive.is/latest/' + url

if we can get response.. we can also fix WSJ recipe.

[kovidgoyal]

Does it work if you use the read_url() function from calibre.scraper.simple

[unkn0wn]

from calibre.scraper.simple import read_url
from calibre.ptempfile import PersistentTemporaryFile
...
    storage = []

    articles_are_obfuscated = True
    def get_obfuscated_article(self, url):
        raw = read_url(self.storage, 'https://archive.is/latest/' + url)
        pt = PersistentTemporaryFile('.html')
        pt.write(raw.encode('utf-8'))
        pt.close()
        return pt.name

I used get_obfuscated article method.

it works, but is there a simpler way?

[kovidgoyal]

Easier in what sense?

[unkn0wn]

idk, is this the right method though? I am noob here.

can we do it without writing into a temp file through get_obfuscated?

[kovidgoyal]

The cost of creating a temp file is insignificant compared to actually downloading so it doesnt matter, but I added some code to allow avoiding the temp file: https://github.com/kovidgoyal/calibr...6689de07213fbe

[unkn0wn]

will be able to use this in the next update i guess. Thanks.