WSJ downloads all links, but not all articles - Page 2

tpkindle · 05-21-2022, 11:56 PM

Exactly the same issue here, im using my credentials but all the articles are cut short. Cheers!

kovidgoyal · 05-24-2022, 01:25 AM

I looked into this, WSJ has started delivering the paywalled content encrypted even for logged in subscribers. The content is then decrypted via javascript and displayed. I could in theory reverse engineer their javascript and decrypt the content in the recipe, but this is too much effort, at least for me. If anybody wants to have a stab at it, feel free, the decryption code is in https://www.wsj.com/_next/static/chu...f08f3742e3f.js

You can get the encrypted data from any wsj paywalled article page by searching for encryptedDataHash in the HTML.

tpkindle · 05-24-2022, 10:05 AM

Thanks for the heads up Kovid.

That's a shame. If there is anyone willing to give it a try and see if we can work around this issue feel free to pm!

gforgiraffe39935 · 07-12-2022, 08:40 PM

delete

Hellman101 · 08-02-2022, 11:19 AM

Instead of trying to reverse engineer the javascript to do the decryption, would it be possible to leverage something like webdriverio to drive Chrome? This would allow the browser to render the page properly and then the recipe would just pull the rendered DOM which has already been decrypted. Main advantage here is if the decryption javascript changed in the future it wouldn't matter - at the end of the day the content has to render in a browser.

No idea if it's possible for the recipe to use something like webdriverio so might be barking up the wrong tree.

kovidgoyal · 08-02-2022, 11:23 AM

then you have to fight all the bot/scraping protections the website has implemented in javascript. But by all means feel free to try, calibre comes with Qt WebEngine you can use for this.

Hellman101 · 08-05-2022, 04:15 PM

ahh gotcha. I assumed there was a downside to my suggestion but couldn't think of one. I'm not a python developer but might give it a go once the kids are back in school. Could someone point me to an example of a recipe calling JS? That would save me some time trying to just figure that part out.

kovidgoyal · 08-05-2022, 08:42 PM

Recipes dont call JS at all. One would have to develop a new recipe type that uses Qt WebEngine in order to use JS in recipes.

etmurphy · 03-17-2023, 06:31 PM

Thank you for fixing WSJ. I gather that it was fixed with this commit in September: https://github.com/kovidgoyal/calibr...928bd9aa21cb5b

jrobbin1 · 09-19-2023, 02:06 PM

Last few days WSJ recipe downloads section headers but no articles, both the free and the subscribed versions.

kovidgoyal · 09-21-2023, 03:23 AM

See https://www.mobileread.com/forums/sh...d.php?t=356129

mjfriedman · 10-03-2023, 07:52 AM

It’s working again!

mjfriedman · 10-03-2023, 07:53 AM

Whoops! Take it back. Now receiving all headlines plus first paragraph only of each story

kovidgoyal · 10-03-2023, 09:26 AM

yes that's the unencrypted part

mjfriedman · 10-24-2023, 09:20 PM

Today it really is working again!

09-19-2023, 02:06 PM	#25
jrobbin1 Junior Member Posts: 1 Karma: 10 Join Date: Sep 2023 Device: kindle	WSJ recipe not downloading articles Last few days WSJ recipe downloads section headers but no articles, both the free and the subscribed versions.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Saving articles from news downloads	sparks56	Calibre	1	02-17-2017 09:46 PM
How to insert links between articles?	oecherprinte	Recipes	3	11-27-2013 04:37 AM
ReadItLater recipe only downloads 10 saved articles?	usuario74	Recipes	1	02-20-2011 04:24 PM
calibre only downloads some articles from FT	St28	Recipes	0	01-21-2011 09:25 AM
Sharing/saving articles in news downloads for Kindle	f1nkster	Calibre	4	07-28-2010 01:53 PM

05-21-2022, 11:56 PM	#16
tpkindle Junior Member Posts: 4 Karma: 10 Join Date: May 2022 Device: pw 10	Exactly the same issue here, im using my credentials but all the articles are cut short. Cheers!

05-24-2022, 01:25 AM	#17
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I looked into this, WSJ has started delivering the paywalled content encrypted even for logged in subscribers. The content is then decrypted via javascript and displayed. I could in theory reverse engineer their javascript and decrypt the content in the recipe, but this is too much effort, at least for me. If anybody wants to have a stab at it, feel free, the decryption code is in https://www.wsj.com/_next/static/chu...f08f3742e3f.js You can get the encrypted data from any wsj paywalled article page by searching for encryptedDataHash in the HTML.

05-24-2022, 10:05 AM	#18
tpkindle Junior Member Posts: 4 Karma: 10 Join Date: May 2022 Device: pw 10	Thanks for the heads up Kovid. That's a shame. If there is anyone willing to give it a try and see if we can work around this issue feel free to pm!

07-12-2022, 08:40 PM	#19
gforgiraffe39935 Enthusiast Posts: 45 Karma: 1000 Join Date: Aug 2021 Device: Nook, Kobo, Kindle	delete

08-02-2022, 11:19 AM	#20
Hellman101 Junior Member Posts: 5 Karma: 10 Join Date: Feb 2017 Device: Kindle Voyage	Instead of trying to reverse engineer the javascript to do the decryption, would it be possible to leverage something like webdriverio to drive Chrome? This would allow the browser to render the page properly and then the recipe would just pull the rendered DOM which has already been decrypted. Main advantage here is if the decryption javascript changed in the future it wouldn't matter - at the end of the day the content has to render in a browser. No idea if it's possible for the recipe to use something like webdriverio so might be barking up the wrong tree.

08-02-2022, 11:23 AM	#21
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	then you have to fight all the bot/scraping protections the website has implemented in javascript. But by all means feel free to try, calibre comes with Qt WebEngine you can use for this.

08-05-2022, 04:15 PM	#22
Hellman101 Junior Member Posts: 5 Karma: 10 Join Date: Feb 2017 Device: Kindle Voyage	ahh gotcha. I assumed there was a downside to my suggestion but couldn't think of one. I'm not a python developer but might give it a go once the kids are back in school. Could someone point me to an example of a recipe calling JS? That would save me some time trying to just figure that part out.

08-05-2022, 08:42 PM	#23
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Recipes dont call JS at all. One would have to develop a new recipe type that uses Qt WebEngine in order to use JS in recipes.

03-17-2023, 06:31 PM	#24
etmurphy Emmet Murphy Posts: 5 Karma: 10 Join Date: Apr 2022 Location: San Francisco, CA, USA Device: Kindle Oasis	Thank you for fixing WSJ. I gather that it was fixed with this commit in September: https://github.com/kovidgoyal/calibr...928bd9aa21cb5b

09-21-2023, 03:23 AM	#26
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	See https://www.mobileread.com/forums/sh...d.php?t=356129

10-03-2023, 07:52 AM	#27
mjfriedman Connoisseur Posts: 64 Karma: 10 Join Date: Dec 2010 Device: Kindle Oasis	It’s working again!

10-03-2023, 07:53 AM	#28
mjfriedman Connoisseur Posts: 64 Karma: 10 Join Date: Dec 2010 Device: Kindle Oasis	Whoops! Take it back. Now receiving all headlines plus first paragraph only of each story

10-03-2023, 09:26 AM	#29
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	yes that's the unencrypted part

10-24-2023, 09:20 PM	#30
mjfriedman Connoisseur Posts: 64 Karma: 10 Join Date: Dec 2010 Device: Kindle Oasis	Today it really is working again!