Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 05-21-2022, 11:56 PM   #16
tpkindle
Junior Member
tpkindle began at the beginning.
 
Posts: 4
Karma: 10
Join Date: May 2022
Device: pw 10
Exactly the same issue here, im using my credentials but all the articles are cut short. Cheers!
tpkindle is offline   Reply With Quote
Old 05-24-2022, 01:25 AM   #17
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 41,248
Karma: 18286997
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I looked into this, WSJ has started delivering the paywalled content encrypted even for logged in subscribers. The content is then decrypted via javascript and displayed. I could in theory reverse engineer their javascript and decrypt the content in the recipe, but this is too much effort, at least for me. If anybody wants to have a stab at it, feel free, the decryption code is in https://www.wsj.com/_next/static/chu...f08f3742e3f.js

You can get the encrypted data from any wsj paywalled article page by searching for encryptedDataHash in the HTML.
kovidgoyal is offline   Reply With Quote
Old 05-24-2022, 10:05 AM   #18
tpkindle
Junior Member
tpkindle began at the beginning.
 
Posts: 4
Karma: 10
Join Date: May 2022
Device: pw 10
Thanks for the heads up Kovid.

That's a shame. If there is anyone willing to give it a try and see if we can work around this issue feel free to pm!
tpkindle is offline   Reply With Quote
Old 07-12-2022, 08:40 PM   #19
gforgiraffe39935
Enthusiast
gforgiraffe39935 began at the beginning.
 
gforgiraffe39935's Avatar
 
Posts: 26
Karma: 10
Join Date: Aug 2021
Device: Nook, Kobo, Kindle
delete
gforgiraffe39935 is offline   Reply With Quote
Old 08-02-2022, 11:19 AM   #20
Hellman101
Junior Member
Hellman101 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2017
Device: Kindle Voyage
Instead of trying to reverse engineer the javascript to do the decryption, would it be possible to leverage something like webdriverio to drive Chrome? This would allow the browser to render the page properly and then the recipe would just pull the rendered DOM which has already been decrypted. Main advantage here is if the decryption javascript changed in the future it wouldn't matter - at the end of the day the content has to render in a browser.

No idea if it's possible for the recipe to use something like webdriverio so might be barking up the wrong tree.
Hellman101 is offline   Reply With Quote
Old 08-02-2022, 11:23 AM   #21
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 41,248
Karma: 18286997
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
then you have to fight all the bot/scraping protections the website has implemented in javascript. But by all means feel free to try, calibre comes with Qt WebEngine you can use for this.
kovidgoyal is offline   Reply With Quote
Old 08-05-2022, 04:15 PM   #22
Hellman101
Junior Member
Hellman101 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Feb 2017
Device: Kindle Voyage
ahh gotcha. I assumed there was a downside to my suggestion but couldn't think of one. I'm not a python developer but might give it a go once the kids are back in school. Could someone point me to an example of a recipe calling JS? That would save me some time trying to just figure that part out.
Hellman101 is offline   Reply With Quote
Old 08-05-2022, 08:42 PM   #23
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 41,248
Karma: 18286997
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Recipes dont call JS at all. One would have to develop a new recipe type that uses Qt WebEngine in order to use JS in recipes.
kovidgoyal is offline   Reply With Quote
Reply

Tags
calibre, wsj, wsj.com

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Saving articles from news downloads sparks56 Calibre 1 02-17-2017 09:46 PM
How to insert links between articles? oecherprinte Recipes 3 11-27-2013 04:37 AM
ReadItLater recipe only downloads 10 saved articles? usuario74 Recipes 1 02-20-2011 04:24 PM
calibre only downloads some articles from FT St28 Recipes 0 01-21-2011 09:25 AM
Sharing/saving articles in news downloads for Kindle f1nkster Calibre 4 07-28-2010 01:53 PM


All times are GMT -4. The time now is 07:45 PM.


MobileRead.com is a privately owned, operated and funded community.