Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 02-01-2019, 07:51 AM   #1
Muller
Member
Muller began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Mar 2018
Device: Kindle oasis
Recipe Le Monde by Rémi Vanicat

Hello!

The recipe "Le Monde: Edition abonnés papier" written years ago by Rémi Vanicat doesn't seem to work since last monday. Am I the only one having this problem?

Thanks
Muller is offline   Reply With Quote
Old 02-01-2019, 06:18 PM   #2
Micmacpec
Junior Member
Micmacpec began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Feb 2019
Device: Kobo
Hello,
not alone ...
Le Monde staysfor ma at january 29 th.

it seems there is an issue in the recipe at " zipurl_format = 'http://medias.lemonde.fr/abonnes/editionelectronique/%Y%m%d/html/%y%m%d.zip'"

But i'm not a javascript expert ...

Anyone has an idea ?

thanks
Micmacpec is offline   Reply With Quote
Old 02-03-2019, 10:14 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,109
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
well if you can ficgure out what the correct url should be, I can modify the recipe accordingly.
kovidgoyal is online now   Reply With Quote
Old 02-13-2019, 09:57 AM   #4
duluoz
Newsbeamer dev
duluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheeseduluoz can extract oil from cheese
 
Posts: 65
Karma: 1000
Join Date: Dec 2011
Device: Kindle Voyage
If anyone can PM me a username and password I can try to figure out the correct URL
duluoz is offline   Reply With Quote
Old 02-04-2020, 07:01 AM   #5
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2020
Device: None
Hi Everyone,

I was also looking into that. The problem is that "Le Monde" changed their website and that now the content is requested through javascript.
I could build the recipe adapted to the content but the problem I did not solve is to access the html content after the javascript request (similar to webscraping). I downloaded the content once manually through the "inspect content" feature of Mozilla, I am not sure how to do that with the python already embedded in calibre.
I forked Sylvain Durand recipe: https://github.com/sylvaindurand/calibre if someone wants to take over....

Cheers,
Akeras is offline   Reply With Quote
Old 02-04-2020, 09:12 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,109
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You need to open the dev tools in your browser, check what query is used to fetch the content and send that query manually. Or alternately if the content is already present as encoded JSON in the html you can decode it from there.
kovidgoyal is online now   Reply With Quote
Old 02-06-2020, 08:39 AM   #7
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2020
Device: None
Selenium with Calibre

Hello Kovidgoyal,

Thanks for your reply.
Is there anyway to use selenium or equivalent inside the recipe?
It looks like finding the exact request is not that easy....

Regards,
Akeras is offline   Reply With Quote
Old 02-06-2020, 10:51 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,109
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You can use QWebEnginePage it's on my TODO list to create an API to make it easy, but as you can imagine, my TODO list is rather long.
kovidgoyal is online now   Reply With Quote
Old 02-07-2020, 12:39 PM   #9
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2020
Device: None
Thanks again for your reply!
I did explore the QWebEnginePage idea but when I want to import it

Code:
from PyQt5.QtWebEngineWidgets import  QWebEnginePage
I get:
calibre, version 4.8.0
ERROR: Invalid recipe: Failed to compile the recipe, with syntax error: QtWebEngineWidgets must be imported before a QCoreApplication instance is created

So I tried so trick found on internet which is to call first
Code:
from PyQt5.QtWidgets import QApplication
app = QApplication.instance()
if app is not None:
    import sip
    app.quit()
    sip.delete(app)
but app.quit() is actually killing Calibre....

Any thoughts?

Thanks a lot!
Akeras is offline   Reply With Quote
Old 02-07-2020, 01:06 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,109
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Dont perform the import at the top level of the recipe. Instead import it inside the get_browser() or similar method of your recipe.
kovidgoyal is online now   Reply With Quote
Old 02-14-2020, 03:37 AM   #11
Akeras
Junior Member
Akeras began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2020
Device: None
QTest for Calibre?

Thanks Kovidgoyal! Your solution works.

I worked a bit on the recipe and managed to have a parser giving some results on my own python kernel. It is not straightforward since the website is sending the "finish loading" signal before the javascript is loaded and not after.

I designed a function that wait until some given elements are found into the pages before running further. This cannot be done with
Code:
time.sleep()
since the webpage becomes unresponsive and the page is not loading anymore. So it seems to work if I use

Code:
from PyQt5 import QtTest
QtTest.QTest.qWait(XXX)
I basically loop over QTest until I find specific keys in the html of the page.

But QTest is not part of the modules one can import in Calibre...
Any idea how to go around that? I could not make QTimer works since the code outside of the timer continues to run: QTimer seems to open a new thread when running the code.
Akeras is offline   Reply With Quote
Old 02-15-2020, 11:23 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,109
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You dont use time.sleep() the way to wait while javascript is running is to use

QApplication.instance().processEvents()

in a loop periodically checking for the finish condition.
kovidgoyal is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Le monde recipe's issue Acryde Recipes 1 09-14-2017 09:53 AM
Le Monde: updated recipe veezh Recipes 0 03-27-2012 01:49 PM
Recipe for Le Monde subscribers? Thomas92 Recipes 0 12-27-2011 05:50 AM
Improved recipe for Le Monde veezh Recipes 0 02-25-2011 05:14 AM
Updated recipe for Le Monde? veezh Recipes 5 01-20-2011 10:06 PM


All times are GMT -4. The time now is 06:55 AM.


MobileRead.com is a privately owned, operated and funded community.