Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 02-01-2019, 06:51 AM   #1
Muller
Member
Muller began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Mar 2018
Device: Kindle oasis
Recipe Le Monde by Rémi Vanicat

Hello!

The recipe "Le Monde: Edition abonnés papier" written years ago by Rémi Vanicat doesn't seem to work since last monday. Am I the only one having this problem?

Thanks
Muller is offline   Reply With Quote
Old 02-01-2019, 05:18 PM   #2
Micmacpec
Junior Member
Micmacpec began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Feb 2019
Device: Kobo
Hello,
not alone ...
Le Monde staysfor ma at january 29 th.

it seems there is an issue in the recipe at " zipurl_format = 'http://medias.lemonde.fr/abonnes/editionelectronique/%Y%m%d/html/%y%m%d.zip'"

But i'm not a javascript expert ...

Anyone has an idea ?

thanks
Micmacpec is offline   Reply With Quote
Advert
Old 02-03-2019, 09:14 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
well if you can ficgure out what the correct url should be, I can modify the recipe accordingly.
kovidgoyal is offline   Reply With Quote
Old 02-13-2019, 08:57 AM   #4
duluoz
Newsbeamer dev
duluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheese
 
Posts: 123
Karma: 1000
Join Date: Dec 2011
Device: Kindle Voyage
If anyone can PM me a username and password I can try to figure out the correct URL
duluoz is offline   Reply With Quote
Old 02-04-2020, 06:01 AM   #5
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
Hi Everyone,

I was also looking into that. The problem is that "Le Monde" changed their website and that now the content is requested through javascript.
I could build the recipe adapted to the content but the problem I did not solve is to access the html content after the javascript request (similar to webscraping). I downloaded the content once manually through the "inspect content" feature of Mozilla, I am not sure how to do that with the python already embedded in calibre.
I forked Sylvain Durand recipe: https://github.com/sylvaindurand/calibre if someone wants to take over....

Cheers,
Akeras is offline   Reply With Quote
Advert
Old 02-04-2020, 08:12 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You need to open the dev tools in your browser, check what query is used to fetch the content and send that query manually. Or alternately if the content is already present as encoded JSON in the html you can decode it from there.
kovidgoyal is offline   Reply With Quote
Old 02-06-2020, 07:39 AM   #7
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
Selenium with Calibre

Hello Kovidgoyal,

Thanks for your reply.
Is there anyway to use selenium or equivalent inside the recipe?
It looks like finding the exact request is not that easy....

Regards,
Akeras is offline   Reply With Quote
Old 02-06-2020, 09:51 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You can use QWebEnginePage it's on my TODO list to create an API to make it easy, but as you can imagine, my TODO list is rather long.
kovidgoyal is offline   Reply With Quote
Old 02-07-2020, 11:39 AM   #9
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
Thanks again for your reply!
I did explore the QWebEnginePage idea but when I want to import it

Code:
from PyQt5.QtWebEngineWidgets import  QWebEnginePage
I get:
calibre, version 4.8.0
ERROR: Invalid recipe: Failed to compile the recipe, with syntax error: QtWebEngineWidgets must be imported before a QCoreApplication instance is created

So I tried so trick found on internet which is to call first
Code:
from PyQt5.QtWidgets import QApplication
app = QApplication.instance()
if app is not None:
    import sip
    app.quit()
    sip.delete(app)
but app.quit() is actually killing Calibre....

Any thoughts?

Thanks a lot!
Akeras is offline   Reply With Quote
Old 02-07-2020, 12:06 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Dont perform the import at the top level of the recipe. Instead import it inside the get_browser() or similar method of your recipe.
kovidgoyal is offline   Reply With Quote
Old 02-14-2020, 02:37 AM   #11
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
QTest for Calibre?

Thanks Kovidgoyal! Your solution works.

I worked a bit on the recipe and managed to have a parser giving some results on my own python kernel. It is not straightforward since the website is sending the "finish loading" signal before the javascript is loaded and not after.

I designed a function that wait until some given elements are found into the pages before running further. This cannot be done with
Code:
time.sleep()
since the webpage becomes unresponsive and the page is not loading anymore. So it seems to work if I use

Code:
from PyQt5 import QtTest
QtTest.QTest.qWait(XXX)
I basically loop over QTest until I find specific keys in the html of the page.

But QTest is not part of the modules one can import in Calibre...
Any idea how to go around that? I could not make QTimer works since the code outside of the timer continues to run: QTimer seems to open a new thread when running the code.
Akeras is offline   Reply With Quote
Old 02-15-2020, 10:23 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,342
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You dont use time.sleep() the way to wait while javascript is running is to use

QApplication.instance().processEvents()

in a loop periodically checking for the finish condition.
kovidgoyal is offline   Reply With Quote
Old 04-08-2020, 05:14 AM   #13
xav
Member
xav began at the beginning.
 
Posts: 22
Karma: 10
Join Date: Mar 2014
Device: Kindle paperwhite 2
Hi Akeras,
Could you please publish the updated recipe or link ?
Tried the Link to Github to Sylvaindurand but it seems to be broken ?
Thanks !
xav is offline   Reply With Quote
Old 06-08-2021, 03:31 AM   #14
frereroy
Enthusiast
frereroy began at the beginning.
 
frereroy's Avatar
 
Posts: 36
Karma: 10
Join Date: May 2021
Location: Europe
Device: Kobo Forma
Any news on a working recipe for the French newspaper Le Monde?
TIA
frereroy is offline   Reply With Quote
Old 06-12-2021, 04:04 AM   #15
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 5
Karma: 12
Join Date: Feb 2020
Device: None
Recipe / python

Hi all,

After spending much time in trying to figure out how to do it with calibre, I switch to creating a "quick and dirty" Python program with Selenium that scrap the website of LeMonde. Please find it here if it can be useful to someone.

https://github.com/MartinHugg/LeMondeSelenium

With python I only found a way to create epub ebooks so calibre is used to convert to mobi before the file is sent to the kindle. If you want another format, feel free to change the code accordingly.

Cheers
Akeras is offline   Reply With Quote
Reply


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Le monde recipe's issue Acryde Recipes 1 09-14-2017 08:53 AM
Le Monde: updated recipe veezh Recipes 0 03-27-2012 12:49 PM
Recipe for Le Monde subscribers? Thomas92 Recipes 0 12-27-2011 04:50 AM
Improved recipe for Le Monde veezh Recipes 0 02-25-2011 04:14 AM
Updated recipe for Le Monde? veezh Recipes 5 01-20-2011 09:06 PM


All times are GMT -4. The time now is 03:07 PM.


MobileRead.com is a privately owned, operated and funded community.