UK Guardian failing to download articles

paddyrm · 10-15-2020, 02:46 PM

For some time now the Guardian recipe has on some days failed to download some articles: in previous versions of Calibre this just meant that selecting the article from the index just jumped to the next available article.
From version 5, a placeholder "This article was downloaded by cailbre from https:" and the relevant URL (which if I load the Guardian onto my Android device lets my click on and read the article on the web) represents the missing download - there were 86 missing in today's edition!
I suspect it is a problem at the Guardian website rather than the recipe: but can I upload any data for you to look at?
Cheers
Paddy

kovidgoyal · 10-16-2020, 02:57 AM

Works fine for me, see attached, you sure you are using the builtin recipe?

paddyrm · 10-16-2020, 04:52 PM

Quote:

Originally Posted by kovidgoyal

Works fine for me, see attached, you sure you are using the builtin recipe?

Yes, 100% using your recipe: I search for it in Fetch News, Guardian and Observer under English (United Kingdom). Attached is my view of an item from today's edition if I click on under Headlines "QAnon President refuses..."
You will see it drops that and the next two articles, next viewable article is "Coronavirus PM's Covid plan in turmoil...". But as I say, the URL links work.
This happens almost daily, in variable amounts. I can tell how bad it is by the size of the file. Very occasionally, no problem. Have tried downloading at varied times, makes no difference. But it seems significant that you do not have the same problem.
All the best
Paddy

kovidgoyal · 10-16-2020, 11:53 PM

This kind of variable result is typically because the server is malfunctioning. Sending different markup for an A/B test or just being buggy. Not much I can do about it, I'm afraid.

paddyrm · 10-17-2020, 06:15 AM

Thanks for checking it for me Kovid, and for confirming a likely server problem.

And thanks for calibre!

Regards

Paddy

paddyrm · 10-24-2020, 04:18 PM

Just for information, in case other Guardian readers have the same problem:
Today (24/10/2020, 9am) the download was particularly thin, most articles were missing with just URLs rather than the text I was wanting to read. It came in at 3.1 Mb.
I decided to retry at 8pm, and the download was 7.2 Mb, with no articles missing!
Maybe the server had stabilised, perhaps less traffic and news updates? Who knows, but worth a try if you have similar problems.
Paddy

paddyrm · 12-01-2020, 04:36 PM

The Guardian download is getting worse by the day, filesize getting smaller, more and more articles blank with just the URL. In today's downlaod details I see

Fetched https://www.theguardian.com/politics...rade-deal-made in 3.153734 seconds

and yet the article is not there, even though it appears to have been fetched!

Is the middle line of these three responsible for removing the article from the "plumber" folders?

Removing duplicate article: Elliot Page: star of Juno and X-Men announces he is transgender from section: Most viewed
Removing duplicate article: UK likely to axe finance bill clauses if Brexit trade deal made from section: Most viewed
Removing duplicate article: Live Barr says no evidence of fraud that would change US election outcome – live from section: Most viewed

Paddy

paddyrm · 12-01-2020, 04:48 PM

The Guardian gets thinner and thinner by the day!

Today I followed this article, which had not arrived, just showed the URL at the top of the View screen. From the job details:

Fetched ...theguardian.com/politics/2020/dec/01/uk-likely-to-axe-finance-bill-clauses-if-brexit-trade-deal-made in 3.153734 seconds. Then
uk-likely-to-axe-finance-bill-clauses-if-brexit-trade-deal-made saved to C:\Users\Paddy\AppData\Local\Temp\calibre_aj8tl5od \v6584tae_plumber\feed_0\article_4\index.xhtml

So it seems to have downloaded and saved.

And yet I cannot see it!

Paddy

paddyrm · 12-01-2020, 04:49 PM

Sorry for duplicate posts! -- Paddy

kovidgoyal · 12-01-2020, 08:55 PM

No, its just that this website is in the middle of a transition to a react based article markup, with some articles using the old markup and some the new. Bloody pain. This should take care of it: https://github.com/kovidgoyal/calibr...97275cba2e6275

paddyrm · 12-03-2020, 06:04 AM

Cheers Kovid, bloody pain is right! It seemed to know all the articles I wanted to read and miss them off! A special implementation of AI...

But this morning was a marvellous download, the most complete and largest for some time. My jaw hit the floor! Thank you very much for examining this, locating and solving the problem. You're the best!

Paddy in Wales

10-15-2020, 02:46 PM	#1
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	UK Guardian failing to download articles For some time now the Guardian recipe has on some days failed to download some articles: in previous versions of Calibre this just meant that selecting the article from the index just jumped to the next available article. From version 5, a placeholder "This article was downloaded by cailbre from https:" and the relevant URL (which if I load the Guardian onto my Android device lets my click on and read the article on the web) represents the missing download - there were 86 missing in today's edition! I suspect it is a problem at the Guardian website rather than the recipe: but can I upload any data for you to look at? Cheers Paddy

10-16-2020, 02:57 AM	#2
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Works fine for me, see attached, you sure you are using the builtin recipe? Attached Thumbnails

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Guardian download	Snowboaardvicky	Recipes	32	12-21-2015 06:10 AM
download of wsj failing	amritsari	Calibre	4	09-06-2012 05:53 PM
Psychology Today recipe is recently failing to pull articles.	Shuichiro	Recipes	1	08-06-2011 05:23 PM
Guardian scheduled download failing	nickd	Recipes	2	04-10-2011 04:35 AM
Failing to download Slate	shoukyd	Calibre	2	04-02-2010 11:46 PM

10-16-2020, 11:53 PM	#4
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	This kind of variable result is typically because the server is malfunctioning. Sending different markup for an A/B test or just being buggy. Not much I can do about it, I'm afraid.

10-17-2020, 06:15 AM	#5
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	Thanks for checking it for me Kovid, and for confirming a likely server problem. And thanks for calibre! Regards Paddy

10-24-2020, 04:18 PM	#6
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	Just for information, in case other Guardian readers have the same problem: Today (24/10/2020, 9am) the download was particularly thin, most articles were missing with just URLs rather than the text I was wanting to read. It came in at 3.1 Mb. I decided to retry at 8pm, and the download was 7.2 Mb, with no articles missing! Maybe the server had stabilised, perhaps less traffic and news updates? Who knows, but worth a try if you have similar problems. Paddy

12-01-2020, 04:36 PM	#7
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	The Guardian download is getting worse by the day, filesize getting smaller, more and more articles blank with just the URL. In today's downlaod details I see Fetched https://www.theguardian.com/politics...rade-deal-made in 3.153734 seconds and yet the article is not there, even though it appears to have been fetched! Is the middle line of these three responsible for removing the article from the "plumber" folders? Removing duplicate article: Elliot Page: star of Juno and X-Men announces he is transgender from section: Most viewed Removing duplicate article: UK likely to axe finance bill clauses if Brexit trade deal made from section: Most viewed Removing duplicate article: Live Barr says no evidence of fraud that would change US election outcome – live from section: Most viewed Paddy

12-01-2020, 04:48 PM	#8
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	The Guardian gets thinner and thinner by the day! Today I followed this article, which had not arrived, just showed the URL at the top of the View screen. From the job details: Fetched ...theguardian.com/politics/2020/dec/01/uk-likely-to-axe-finance-bill-clauses-if-brexit-trade-deal-made in 3.153734 seconds. Then uk-likely-to-axe-finance-bill-clauses-if-brexit-trade-deal-made saved to C:\Users\Paddy\AppData\Local\Temp\calibre_aj8tl5od \v6584tae_plumber\feed_0\article_4\index.xhtml So it seems to have downloaded and saved. And yet I cannot see it! Paddy

12-01-2020, 04:49 PM	#9
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	Sorry for duplicate posts! -- Paddy

12-01-2020, 08:55 PM	#10
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, its just that this website is in the middle of a transition to a react based article markup, with some articles using the old markup and some the new. Bloody pain. This should take care of it: https://github.com/kovidgoyal/calibr...97275cba2e6275

12-03-2020, 06:04 AM	#11
paddyrm Connoisseur Posts: 67 Karma: 10 Join Date: Oct 2012 Device: Kindle 3	Cheers Kovid, bloody pain is right! It seemed to know all the articles I wanted to read and miss them off! A special implementation of AI... But this morning was a marvellous download, the most complete and largest for some time. My jaw hit the floor! Thank you very much for examining this, locating and solving the problem. You're the best! Paddy in Wales

Advert

Advert