![]() |
#1 |
Member
![]() Posts: 23
Karma: 10
Join Date: Mar 2018
Device: Kindle oasis
|
Recipe Le Monde by Rémi Vanicat
Hello!
The recipe "Le Monde: Edition abonnés papier" written years ago by Rémi Vanicat doesn't seem to work since last monday. Am I the only one having this problem? Thanks |
![]() |
![]() |
![]() |
#2 |
Junior Member
![]() Posts: 1
Karma: 10
Join Date: Feb 2019
Device: Kobo
|
Hello,
not alone ... Le Monde staysfor ma at january 29 th. it seems there is an issue in the recipe at " zipurl_format = 'http://medias.lemonde.fr/abonnes/editionelectronique/%Y%m%d/html/%y%m%d.zip'" But i'm not a javascript expert ... Anyone has an idea ? thanks |
![]() |
![]() |
Advert | |
|
![]() |
#3 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
well if you can ficgure out what the correct url should be, I can modify the recipe accordingly.
|
![]() |
![]() |
![]() |
#4 |
Newsbeamer dev
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 123
Karma: 1000
Join Date: Dec 2011
Device: Kindle Voyage
|
If anyone can PM me a username and password I can try to figure out the correct URL
|
![]() |
![]() |
![]() |
#5 |
Junior Member
![]() Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
|
Hi Everyone,
I was also looking into that. The problem is that "Le Monde" changed their website and that now the content is requested through javascript. I could build the recipe adapted to the content but the problem I did not solve is to access the html content after the javascript request (similar to webscraping). I downloaded the content once manually through the "inspect content" feature of Mozilla, I am not sure how to do that with the python already embedded in calibre. I forked Sylvain Durand recipe: https://github.com/sylvaindurand/calibre if someone wants to take over.... Cheers, |
![]() |
![]() |
Advert | |
|
![]() |
#6 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You need to open the dev tools in your browser, check what query is used to fetch the content and send that query manually. Or alternately if the content is already present as encoded JSON in the html you can decode it from there.
|
![]() |
![]() |
![]() |
#7 |
Junior Member
![]() Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
|
Selenium with Calibre
Hello Kovidgoyal,
Thanks for your reply. Is there anyway to use selenium or equivalent inside the recipe? It looks like finding the exact request is not that easy.... Regards, |
![]() |
![]() |
![]() |
#8 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You can use QWebEnginePage it's on my TODO list to create an API to make it easy, but as you can imagine, my TODO list is rather long.
|
![]() |
![]() |
![]() |
#9 |
Junior Member
![]() Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
|
Thanks again for your reply!
I did explore the QWebEnginePage idea but when I want to import it Code:
from PyQt5.QtWebEngineWidgets import QWebEnginePage calibre, version 4.8.0 ERROR: Invalid recipe: Failed to compile the recipe, with syntax error: QtWebEngineWidgets must be imported before a QCoreApplication instance is created So I tried so trick found on internet which is to call first Code:
from PyQt5.QtWidgets import QApplication app = QApplication.instance() if app is not None: import sip app.quit() sip.delete(app) Any thoughts? Thanks a lot! |
![]() |
![]() |
![]() |
#10 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Dont perform the import at the top level of the recipe. Instead import it inside the get_browser() or similar method of your recipe.
|
![]() |
![]() |
![]() |
#11 |
Junior Member
![]() Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
|
QTest for Calibre?
Thanks Kovidgoyal! Your solution works.
I worked a bit on the recipe and managed to have a parser giving some results on my own python kernel. It is not straightforward since the website is sending the "finish loading" signal before the javascript is loaded and not after. I designed a function that wait until some given elements are found into the pages before running further. This cannot be done with Code:
time.sleep() Code:
from PyQt5 import QtTest QtTest.QTest.qWait(XXX) But QTest is not part of the modules one can import in Calibre... Any idea how to go around that? I could not make QTimer works since the code outside of the timer continues to run: QTimer seems to open a new thread when running the code. |
![]() |
![]() |
![]() |
#12 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You dont use time.sleep() the way to wait while javascript is running is to use
QApplication.instance().processEvents() in a loop periodically checking for the finish condition. |
![]() |
![]() |
![]() |
#13 |
Member
![]() Posts: 22
Karma: 10
Join Date: Mar 2014
Device: Kindle paperwhite 2
|
Hi Akeras,
Could you please publish the updated recipe or link ? Tried the Link to Github to Sylvaindurand but it seems to be broken ? Thanks ! |
![]() |
![]() |
![]() |
#14 |
Enthusiast
![]() Posts: 36
Karma: 10
Join Date: May 2021
Location: Europe
Device: Kobo Forma
|
Any news on a working recipe for the French newspaper Le Monde?
TIA |
![]() |
![]() |
![]() |
#15 |
Junior Member
![]() Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
|
Recipe / python
Hi all,
After spending much time in trying to figure out how to do it with calibre, I switch to creating a "quick and dirty" Python program with Selenium that scrap the website of LeMonde. Please find it here if it can be useful to someone. https://github.com/MartinHugg/LeMondeSelenium With python I only found a way to create epub ebooks so calibre is used to convert to mobi before the file is sent to the kindle. If you want another format, feel free to change the code accordingly. Cheers |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Le monde recipe's issue | Acryde | Recipes | 1 | 09-14-2017 08:53 AM |
Le Monde: updated recipe | veezh | Recipes | 0 | 03-27-2012 12:49 PM |
Recipe for Le Monde subscribers? | Thomas92 | Recipes | 0 | 12-27-2011 04:50 AM |
Improved recipe for Le Monde | veezh | Recipes | 0 | 02-25-2011 04:14 AM |
Updated recipe for Le Monde? | veezh | Recipes | 5 | 01-20-2011 09:06 PM |