Recipe Le Monde by Rémi Vanicat

[Muller]

Hello!

The recipe "Le Monde: Edition abonnés papier" written years ago by Rémi Vanicat doesn't seem to work since last monday. Am I the only one having this problem?

Thanks

[Micmacpec]

Hello,
not alone ...
Le Monde staysfor ma at january 29 th.

it seems there is an issue in the recipe at " zipurl_format = 'http://medias.lemonde.fr/abonnes/editionelectronique/%Y%m%d/html/%y%m%d.zip'"

But i'm not a javascript expert ...

Anyone has an idea ?

thanks

[kovidgoyal]

well if you can ficgure out what the correct url should be, I can modify the recipe accordingly.

[duluoz]

If anyone can PM me a username and password I can try to figure out the correct URL

[Akeras]

Hi Everyone,

I was also looking into that. The problem is that "Le Monde" changed their website and that now the content is requested through javascript.
I could build the recipe adapted to the content but the problem I did not solve is to access the html content after the javascript request (similar to webscraping). I downloaded the content once manually through the "inspect content" feature of Mozilla, I am not sure how to do that with the python already embedded in calibre.
I forked Sylvain Durand recipe: https://github.com/sylvaindurand/calibre if someone wants to take over....

Cheers,

[kovidgoyal]

You need to open the dev tools in your browser, check what query is used to fetch the content and send that query manually. Or alternately if the content is already present as encoded JSON in the html you can decode it from there.

[Akeras]

Hello Kovidgoyal,

Thanks for your reply.
Is there anyway to use selenium or equivalent inside the recipe?
It looks like finding the exact request is not that easy....

Regards,

[kovidgoyal]

You can use QWebEnginePage it's on my TODO list to create an API to make it easy, but as you can imagine, my TODO list is rather long.

[Akeras]

Thanks again for your reply!
I did explore the QWebEnginePage idea but when I want to import it

from PyQt5.QtWebEngineWidgets import  QWebEnginePage

I get:
calibre, version 4.8.0
ERROR: Invalid recipe: Failed to compile the recipe, with syntax error: QtWebEngineWidgets must be imported before a QCoreApplication instance is created

So I tried so trick found on internet which is to call first

from PyQt5.QtWidgets import QApplication
app = QApplication.instance()
if app is not None:
    import sip
    app.quit()
    sip.delete(app)

but app.quit() is actually killing Calibre....

Any thoughts?

Thanks a lot!

[kovidgoyal]

Dont perform the import at the top level of the recipe. Instead import it inside the get_browser() or similar method of your recipe.

[Akeras]

Thanks Kovidgoyal! Your solution works.

I worked a bit on the recipe and managed to have a parser giving some results on my own python kernel. It is not straightforward since the website is sending the "finish loading" signal before the javascript is loaded and not after.

I designed a function that wait until some given elements are found into the pages before running further. This cannot be done with

time.sleep()

since the webpage becomes unresponsive and the page is not loading anymore. So it seems to work if I use

from PyQt5 import QtTest
QtTest.QTest.qWait(XXX)

I basically loop over QTest until I find specific keys in the html of the page.

But QTest is not part of the modules one can import in Calibre...
Any idea how to go around that? I could not make QTimer works since the code outside of the timer continues to run: QTimer seems to open a new thread when running the code.

[kovidgoyal]

You dont use time.sleep() the way to wait while javascript is running is to use

QApplication.instance().processEvents()

in a loop periodically checking for the finish condition.