MobileRead Forums
Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes
Reload this Page New York Times

Notices

Reply
 
Thread Tools Search this Thread
Old 06-14-2025, 10:26 AM   #1
jazzbox
Member
jazzbox doesn't litterjazzbox doesn't litter
 
Posts: 21
Karma: 190
Join Date: Nov 2017
Device: Kindle paperwhite
New York Times
For the last couple of days, recipes retrieve only the section headers. I checked print and web versions, book review, tech beat, sports beat. TIA
jazzbox is offline   Reply With Quote
Old 06-14-2025, 11:14 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,353
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Looks like the nytimes has started requiring javascript and captchas. Sigh. Probably will have to get it from the wayback machine instead, although the wayback machine doesnt seem to be storing nytimes articles in a timely fashion.

Another possibility is using the graphql api though that will require some reverse engineering.
kovidgoyal is online now   Reply With Quote
Old 06-15-2025, 03:02 AM   #3
unkn0wn
Guru
unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.
 
Posts: 616
Karma: 85520
Join Date: May 2021
Device: kindle
Worked until now because of google bot headers.. looks like the bot itself got blocked.

the graphql api one looks hard.
unkn0wn is offline   Reply With Quote
Old 06-15-2025, 07:17 AM   #4
scottsan1
Junior Member
scottsan1 began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Feb 2021
Device: iPad mini
I downloaded the Sunday NY Times today multiple times, only to retrieve section headers, as well. This is terrible.
scottsan1 is offline   Reply With Quote
Old 06-15-2025, 12:14 PM   #5
bllittle
Junior Member
bllittle began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2021
Device: Kindle app on android tablet
Following thread as I'm having the same issue. Hope to see a solution soon.
bllittle is offline   Reply With Quote
Old 06-18-2025, 09:52 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,353
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0
kovidgoyal is online now   Reply With Quote
Old 06-18-2025, 03:08 PM   #7
nickredding
onlinenewsreader.net
nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'
 
Posts: 327
Karma: 10143
Join Date: Dec 2009
Location: Phoenix, AZ & Victoria, BC
Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire
New York Times
The New York Times and New York Times (Web) recipes already use graphql and work fine.
nickredding is offline   Reply With Quote
Old 06-18-2025, 10:01 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,353
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by nickredding View Post
The New York Times and New York Times (Web) recipes already use graphql and work fine.
they use graphql for loading the index not article content.
kovidgoyal is online now   Reply With Quote
Old 06-19-2025, 12:04 AM   #9
unkn0wn
Guru
unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.
 
Posts: 616
Karma: 85520
Join Date: May 2021
Device: kindle
Quote:
Originally Posted by kovidgoyal View Post
This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0
wow. great find... wasnt expecting this to be fixed so easily.
unkn0wn is offline   Reply With Quote
Old 06-19-2025, 10:25 AM   #10
nickredding
onlinenewsreader.net
nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'nickredding knows the difference between 'who' and 'whom'
 
Posts: 327
Karma: 10143
Join Date: Dec 2009
Location: Phoenix, AZ & Victoria, BC
Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire
Quote:
Originally Posted by kovidgoyal View Post
they use graphql for loading the index not article content.
True, but the article content is encapsulated in JSON within the article content so fiddling the UA is not necessary--the JSON is there with a standard UA header.

It seems that major publications are moving to a Content Management System that does this JSON encapsulation as a method of defeating simple screen scrapers. It also seems that RSS feeds are going out of style and relying on RSS in the future for indexes will be increasingly unreliable.
nickredding is offline   Reply With Quote
Old 06-19-2025, 11:03 PM   #11
unkn0wn
Guru
unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.unkn0wn understands the Henderson-Hasselbalch Equation.
 
Posts: 616
Karma: 85520
Join Date: May 2021
Device: kindle
Quote:
Originally Posted by nickredding View Post
fiddling the UA is not necessary--the JSON is there with a standard UA header.
yea but it requires js enabled browser, and then there's captha page for bots to fight.
with these headers it doesnt require so.
unkn0wn is offline   Reply With Quote
Old 06-20-2025, 12:20 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,353
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by nickredding View Post
True, but the article content is encapsulated in JSON within the article content so fiddling the UA is not necessary--the JSON is there with a standard UA header.

It seems that major publications are moving to a Content Management System that does this JSON encapsulation as a method of defeating simple screen scrapers. It also seems that RSS feeds are going out of style and relying on RSS in the future for indexes will be increasingly unreliable.
that's irrelevant, you will get a captcha page if you try to download without an appropriate user-agent. nytimes uses captcha-delivery.com on its article pages.
kovidgoyal is online now   Reply With Quote
Old 06-21-2025, 10:58 AM   #13
bllittle
Junior Member
bllittle began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2021
Device: Kindle app on android tablet
Quote:
Originally Posted by kovidgoyal View Post
This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0
Thanks so much for the update! Its working again for me, too (at least for the moment).
bllittle is offline   Reply With Quote
Reply

« Previous Thread | Next Thread »

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
NYT Spanish New York Times Español El Times Recipe compa Recipes 0 03-24-2022 02:40 PM
The New York Times Vikas Chahal Recipes 0 05-09-2021 03:55 AM
New York Times Paywall? Starson17 Recipes 7 04-03-2011 08:33 PM
Help - New York Times Recipe brutalist Recipes 6 03-20-2011 10:17 PM
New York Times recipe madrone26 Calibre 4 04-02-2009 01:13 PM


All times are GMT -4. The time now is 06:05 AM.

Contact Us - FAQ - Guidelines - Privacy - Top

MobileRead.com is a privately owned, operated and funded community.